-
-
Notifications
You must be signed in to change notification settings - Fork 18.8k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add sample playback support #91382
Open
adamscott
wants to merge
41
commits into
godotengine:master
Choose a base branch
from
adamscott:sample-player
base: master
Could not load branches
Branch not found: {{ refName }}
Could not load tags
Nothing to show
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
Open
Add sample playback support #91382
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
adamscott
force-pushed
the
sample-player
branch
8 times, most recently
from
May 1, 2024 20:20
6115ee4
to
4ff18d6
Compare
This comment was marked as resolved.
This comment was marked as resolved.
adamscott
force-pushed
the
sample-player
branch
2 times, most recently
from
May 2, 2024 14:49
2371978
to
3de8154
Compare
…y renderer" This reverts commit 2584a92.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
When this PR will be merged for 4.3, this feature will be marked as experimental and could change in 4.4.
Based on these PRs
tl;dr
This PR adds (back) the concept of samples to the Godot Engine. Currently, the PR enables only the web platform to play samples.
It principally fixes #87329, as that issue would plague any non-threaded web releases with crackling audio.
Example
(old way)
(this PR)
Introduction
Godot uses streaming to mix game audio. Each active stream is registered and then the engine mix on-the-fly the needed audio frames together to output audio based on the audio latency parameter. It works very well on modern platforms.
Samples are another way to handle sound instead of mixing streams. Instead of handling mixing sound and music by the game processes, it relies on off-loading it to the host system. While it doesn't permit full access to the mixing apparatus, it's super useful on systems that don't have a lot of processing power.
To use samples, you register a sample, and then tell the system to play it when needed. And to stop it. It's like a music player, you set the file, then you click on play. You don't control how the software do it, but you know it does.
Godot used to have samples back in Godot 1 and 2, especially to support platforms like the PSP, and the web (thanks to the Web Audio API).
As newer console platforms let developers handle their own mixing logic and that
SharedArrayBuffer
s were introduced in browsers (permitting WebWorkers (web threads) to share memory with the game), samples support was dropped from Godot. Everything was fine.Anyway, the implementation was somewhat lacking. You had to specifically want to play samples, you couldn't use common nodes to play both streams and samples.
The problem
But on the web platform, Spectre and Meltdown happened. And it completely changed where
SharedArrayBuffer
s were able to be used. Enter "cross-origin isolated" websites, where it's impossible to contact other websites or display ads, and complicating hosting of simple games, greatly reducing the appeal for our web builds.Hence the work on #85939 in order to compile Godot to run on the main thread. This enables exporting Godot games on the browser without having to cross-origin isolate your website. Unfortunately, this brought an unexpected issue: software mixing is pretty much incompatible with single-threaded games. Especially running on older/less powerful hardware.
Wanna hear for yourself? Try the single-threaded platformer demo (without this PR applied) on your phone or on a computer that doesn't have a great CPU.
The investigation
My colleague @Faless and I considered every solution imaginable: augmenting latency for the web, traced the processes on the web and on mobile and refactor the
AudioWorklet
processing the audio. But alas. Nothing substantial could have been done.The only solution we found was to resort to Web Audio samples.
And it's not uncommon for web game engines. We were the uncommon ones not using web audio samples. So, a few weeks ago, I began work on this PR.
The sole requirement: seamlessness
My main focus was to reuse as possible as many features that already exist. It means that in order to play samples, I wanted the UX to keep as close as possible to existing tools.
Godot strives itself to offer the same experience for every target that it exports to. Imagine making the developer choose between having samples for the web export and streaming nodes for the rest. And having to manually add or remove nodes based on the platform with scripts.
This has such a big impact that it's a clear no go for us. We don't want that poor UX.
The solution
My solution is a big hack. (But it does works wonderfully.)
The idea is to reuse all the existing stream nodes and systems. And make the stream elements capable of producing samples.
This story begins with the new project setting
audio/general/default_playback_type
(hidden currently in the advanced options). Usually, it should stay with the value "Stream", as normally, that's how Godot works currently. But the magic happens withaudio/general/default_playback_type.web
set as "Sample".That's because
AudioStreamPlayer
,AudioStreamPlayer2D
andAudioStreamPlayer3D
now have a new property calledplayback_type
, which is set by default to... "Default". That's where the magic happens! On standard exports, the nodes will be defined as "Stream", but on web exports, "Sample" will be used instead!The magic operates behind the scenes though.
The man behind the curtain
Essentially, when a stream is considered a "sample", it doesn't get mixed at all in the mixing phase. Instead, it relies on callbacks by the
StreamPlayer
nodes.The
StreamPlayer
nodes, when theirplay()
method is called, are calling internallyAudioServer::start_sample_playback()
. All theAudioServer
does is to callAudioDriver::start_sample_playback()
. If the driver doesn't implement that function, it just doesn't play any sound. But if it does, the driver can now tell the backend to play that sound.The same thing happens for stop, pause, etc. You can even update the sample, like when the position of the node changes!
Isn't this fascinating?
Registering samples
Before playing the samples, it's important to register them first.
If played without previous registration, the player will make sure to register it first. Though, it's recommended to register manually streams. That's because, on single threaded games, memory transfer is synchronous, so it may make your game stutter. You register a stream as a sample by calling this method:
Under the hood, Godot will call the
mix()
method of the stream playback for the entire duration of the clip. This makes it so that it's possible to play any type of sound media that Godot supports (WAV, mp3, ogg vorbis).Is it really seamless, though?
These demos were exported to the web (single-threaded) using samples without ever touching the project nodes, resources, nor files.
Bugs yet to fix before merge
Buses don't chain properlyAutoplay doesn't work right now.Fix issues with sample rate.AnimationPlayer cannot play samples.Advanced audio importers fail to show / infinite loop(the problem came and go without any of my input)Only forward loop is supported (fix may not make it to the final release) looping is kinda a little broken right nowKnown limitations
Technical diagrams
Registering and playing samples
Samples and streams
Fixes
Fixes #87329