FMOD Unity: Jerk-Reactive Audio
What is it, and why did I use it for our game jam character?
This past weekend marked the conclusion of the Seattle Indies Game Jam 2020.
This year, the jam was held online, and was a two week long “slow” jam, rather than the typical 48–72 hour jams they usually hold.
A few positive outcomes resulted from this change.
For one, it allowed broader access to people who generally wouldn’t attend jams due to full work schedules.
Additionally, it provided room for people to work at their own pace, and take care of themselves during the jam — or push themselves and pick up new skills along the way.
Due to having a little more time to research, our team decided to make a networked multiplayer game. What we ended up with was this goofy physics-based king-of-the-hill style party game, with procedural character animation & moderately dynamic/reactive music.
None of this is what I’m here to talk about.
I’m here to talk about physics-reactive audio systems; specifically, jerk-reactive systems (yes, that’s a real thing).
If you’re anything like me and barely spent any time studying academic physics, you’re probably asking: “Colin, what the hell is jerk”?
In mathematical terms, jerk is the first time-derivative of acceleration.
To unpack that for anyone without a background in calculus, when you measure the change in an object’s position over time, you’re measuring the velocity.
When you measure the change in an objects velocity over time, you’re measuring the acceleration.
And when you measure the change in an object’s acceleration over time, you’re measuring the jerk.
To put it in more concrete terms, when you step on the gas in a car, and get pushed back into your seat, what you’re feeling is the acceleration. But the moment where it feels like you’re being pushed back the hardest would be the moment with the greatest jerk.
So why all the physics for a game jam game?
@applesauce2u, our gameplay programmer built out a velocity based character movement controller, and as I hinted at earlier, our artist @ZappForThat accented this with the goofy procedural creature animations for the character above.
This character is modeled after a pot, but has a separate upper and lower half. The lower half follows the motion of the character controller (and as such, is velocity based), but the upper half reacts to this. It’s motion is, at least indirectly based around acceleration.
My job was to make this character ‘feel’ like a goofy, rattly ceramic pot, using sound.
A more traditional method might have been to work some clanky little sound effects that trigger when certain parts of the animations, walk cycles, etc… are reached. But the animation system was already entirely physics-reactive, and I wanted to lean even harder into that feeling.
I wanted the clinks and clanks to happen when the top half of the pot knocked around.
Again, I had an option here, to measure internal collisions, and play a sound based upon those, but that felt A: pretty expensive on the CPU, and B: pretty difficult to control when accounting for Unity physics oddities.
So the solution I was left with was to measure the acceleration of the pot — or more specifically, the change in the acceleration over time, and use that as my input parameter for the sound effects.
The greater the change in acceleration over time, the louder a clank.
Now that we’re through with the ‘what’ and the ‘why,’ let’s move onto the ‘how.’
This section will all be about Unity & FMOD, as those are the tools we were using to make this game.
The first step was to put together a script and measure the jerk values we’re getting for various actions.
In Unity, a Rigidbody component already contains an accurate measurement of the object’s velocity. We had a Rigidbody on our character controller, so we used that as our starting point.
From there, we grabbed the velocity magnitude & the magnitude from the previous physics tick (lastVelocity — we’re using FixedUpdate & fixedDeltaTime to calculate against Unity Physics tick for more accurate, frame-independent data), and use those values to capture the acceleration.
We then, repeat this process with the acceleration, to capture the jerk value.
Finally, we store the velocity & acceleration for use in this process on the next physics update.
Using the Debug Log statement, we could check our jerk values to find a pretty solid estimate of the range of values it’s outputting.
With those values in mind, and the image of the character model in my head, the next step was to put together the sounds & FMOD event structure for the character’s clinks and clanks.
Each event call represents a single jerk peak, and takes a parameter value between 0 and 1 (we’ll be scaling the jerk value later, to match this).
I built this event using three independent layers.
The first layer consists of randomized containers that are scaled and blended together. The containers hold tonal ceramic hits that build in transient intensity and resonance as the jerk value increases.
The second is similar, but more heavily skewed towards accentuating percussive transients, and less on the tonal elements.
The final layer is made of short, grindy ceramic layers, adding some “slipping” texture to the impacts and emphasizing the feeling that things are moving around, rather than just clashing together (yes, I could have probably put this in it’s own event and triggered it from acceleration instead, but for the sake of game jam simplicity, this is where I left off).
Now, with the event built, the final step was to hook it up in Unity.
Fortunately, our debug code already handles most of the tricky parts of the implementation. All that’s left was to assign our audio attributes, and transform our impact (jerk) values into usable parameters for the FMOD event.
To do this, we created an adjustable field for the parameter scalar value (in case we adjusted the character controller in the future). By default, the scalar value of .001 worked pretty well for our physics implementation.
From there, we also added a pretty rough simulation of inertia, via the jerkAudioPlayThreshold parameter — also as an adjustable value.
All that remained was to hook up the audio callbacks.
For this, we first checked to make sure the jerk is greater than the jerkAudioPlayThreshold (to make sure acceleration is changing enough for us to care about playing a sound).
Next, we create an FMOD audio instance. We need one instance per sound event, to avoid having sounds cut each-other short.
Following this, we assign our jerk value, scaled down by our jerkParamScalar, to our audio instance (setting the value on the one relevant instance only — the current instance).
We then, assign the position of the sound to the position of the character in 3D space, start the audio playback, and free up our reference to the audio instance, as we won’t need to talk to it any more after this.
And with that, we have jerk-reactive audio playing on our characters!
Video credit: @ZappForThat