Affecting Sound (with FMOD) | Slow Rush Studios

No, that's not a typo - last week we were effecting sound, but this week we're affecting sound!

That is: I replaced the existing sound playing from last week with playing sound through a thing called "FMOD".

And FMOD has cool goodies to mess with sound on the fly.

Effects on Sound

Have you ever noticed...

In shooter games when your health gets low, you hear a heartbeat that gets louder and louder?
Sometimes when you go underwater or a door closes off ("occludes") a room, sound gets muffled?
When you start a fight, music smoothly transitions to "combat music"? ¹
A projectile's sound changes pitch² as it flies past you?

All that helps immensely with setting the mood, but I barely know what FFT³ stands for, so it'd take me months to program each of those.

But fortunately you can plug a thing called "Audio Middleware" - like FMOD - into your game to make 'em a lot simpler to pull off.

Optional assignment for extra credit: if you've got ten minutes to kill, this is a really nice overview of FMOD (watch at 2x speed).

Sound Shortcomings

On top of that, last week I had to work around some major shortcomings in my game framework's sound support. ⁴

If just plugging in a Pro Level sound library like FMOD could address those, then so much the better.

Plug and No Play

I chose⁵ FMOD because they claimed to work on the web by (checks notes) Compiling To Web Assembly.

This was excellent, because my game also works on the web by Compiling To Web Assembly:

Detailed engineering diagram showing how Web Assembly libraries talk to each other

2 Web Assembly pieces? You just plug them together, and get sound, right?

Buuut, turns out that there's Compiling To Web Assembly and Compiling To Web Assembly and they are Not The Same, no sirree. ⁶

Detailed engineering diagram showing how Web Assembly libraries actually don't talk to each other

They're so different that code from one just won't talk to code from the other.

I had to resort to a drastic measure:

Detailed engineering diagram showing how Web Assembly libraries actually can talk to each other after all if you sacrifice your firstborn to one 'Brendan Eich'

You *can* speak to FMOD *from JavaScript*, and I can speak to JavaScript *from my game*, so I wrote some dirty Javascript code to glue FMOD into my game. Yuck.

There's a performance cost to this approach, but the only cost I noticed is having to spend hours of my time debugging why the hell FMOD wasn't happy with my shoddy JavaScript code. ⁷

So I guess it's fine!

The Upside

After all that (and rewriting the desktop⁸ audio integration to add FMOD too), where are we at?

All the workaround hacks from last week are gone!
Since FMOD handles audio decompression itself, sounds take way less network bandwidth - so the game loads much faster.
- And I can support music in the web build! (off by default to avoid being annoying)
And I hooked up some basic effects, like randomly varying the pitch of your footsteps to make them less repetitive.

Better dynamic effects - like when you're underwater - will have to wait for later.

Playable web build‎

Here we go, proof that this whole shebang can actually play sound:

Aside: if it doesn't play sound for you, please let me know!

If you want music, turn it on using the Sound menu in the top left.
- This specific music track is just a placeholder (although it is growing on me!)
You can compare with last week to notice that (for example) the footsteps are different.

Press F1 for help, including to see keyboard/mouse controls. Mobile devices probably won't work! By playing you agree to our Privacy Policy.

Known issues:

I still need to "mix" the audio properly: right now some sounds are way too loud.
Spatial positioning is not working yet, so sounds aren't getting softer from being further away (or being panned based on position).

Or even cooler (but much rarer): during a fight, your kill animations are sync'd to the beat of the music?

Via the Doppler effect

Fast Fourier Transform, turns out. All I know is that it's an audio thing - and that if I'd studied an engineering undergrad degree instead of Computer Science then I'd probably actually know what it was.

⁴

I wrote about them a bit in this footnote from last week.

But to give a concrete example, if there are 2 rockets flying making a pssshhhhhh rocket-thrusting noise and one explodes, I could stop all the pssshhhhhh sounds at once, or none of them - and not just the one from the rocket that exploded.

So I put in a hack where I play a single pssshhhhhh sound if at least one rockets is flying... it worked, but I didn't want to take that approach for all other looping sounds too!

⁵

The other major option is Wwise and they don't support the web at all.

Also Noita uses FMOD - and if you've been following along, you'll have realised that I am contractually required to copy everything Noita does.

⁶

I mean, obviously one of them is Italicized and the other is Bolded.

But technically, FMOD uses the Emscripten compiler to compile to Web Assembly, and I use Rust's (LLVM's) native support for compiling to Web Assembly - and the two compilers don't agree on how functions should be called - so you can't mix compiled code from each.

⁷

For those curious: I write Rust code that pretends to call functions defined in an external C library. Then there's a tool called wasm-bindgen that takes those external function declarations and rewrites them to actually call some Javascript functions - which are functions that I wrote myself, that proxy to FMOD's official javascript functions to call them.

The performance cost comes from having to copy data from Rust to JavaScript and then again to FMOD.

You might think that you can skip writing your own JS functions to wrap FMOD's JS API, but a) that doesn't stop the performance issue because all the same copying still takes place, and b) FMOD's JS API is very "what if we forced JavaScript to be like C, with pointer-based out-parameters and returning error codes instead of returning values and throwing exceptions for errors" (exhibit A). So it really helps to have vanilla Javascript interacting with FMOD when you need to debug.

⁸

Somehow there are at least 5 different people who decided to make their own library for "let's make it easy to use FMOD from Rust" and all those libraries seem somewhat abandoned?

I picked fmod-oxide because it is somewhat recent, its codebase seemed sane, and it actually worked when I tried its examples.

(And somehow zero people decided to try and support web, so maybe this is the first web-based Rust game ever to use FMOD?)