Please read https://github.com/UltraStar-Deluxe/Play/blob/master/CONTRIBUTING.md
I am currently diving through the code a bit and I am kinda curious how the Pitch Tracking exactly works
So far I saw that Dynamic wavelet does the audio processing to get the right pitch
the part I am not 100% sure about:
The mic buffer is checked once per frame and if it is above 1024 samples it is processed and the result of the pitch detection is stored
Then I guess whenever a new note needs to be checked if its correctly sung the last available value will be taken?
And for how the latency is accounted for I have no idea
MicSampleRecorder collects the raw samples of a mic. It uses a buffer of up to 1 second I think. Unity fills this buffer in a separate thread such that no sample is missed.
Then it depends.
The settings for example uses the simple MicPitchTracker. It waits for a certain number of samples to arrive and then analyzes these. I guess this is what you found.
In the game however, the pitchof every beat is analyzed (UltraStar song format kind of defines a grid of beats via the #BPM tag).
PlayerPitchTracker knows which beat has to be analyzed next. It waits until the Mic buffer contains the samples that correspond to that beat. This is where the mic-delay is considered. If the mic-delay is increased, then the PlayerPitchTracker will wait longer for the samples to arrive. If there are enough samples, then these are analyzed and a PitchEvent for the beat is fired.
Pitch detection passes different stages.
MicSampleRecorder collects the raw samples of a mic. It uses a buffer of up to 1 second I think. Unity fills this buffer in a separate thread such that no sample is missed.
Then it depends.
The settings for example uses the simple MicPitchTracker. It waits for a certain number of samples to arrive and then analyzes these. I guess this is what you found.
In the game however, the pitchof every beat is analyzed (UltraStar song format kind of defines a grid of beats via the #BPM tag).
PlayerPitchTracker knows which beat has to be analyzed next. It waits until the Mic buffer contains the samples that correspond to that beat. This is where the mic-delay is considered. If the mic-delay is increased, then the PlayerPitchTracker will wait longer for the samples to arrive. If there are enough samples, then these are analyzed and a PitchEvent for the beat is fired.
Thanks a lot for the detailed information I'll look further into it!
Really appreciated
because the buffer has a limited size the new samples are added to the back and if they hit the end they start at 0 again
Actually no. I just checked this. The new samples come into the buffer (highest index has newst samples, index 0 has oldest sample). When the buffer is full, the old samples just "fall out" of the buffer.
I am working on a Karaoke app as a website/webapp for fun atm
Thought you might be interested to see what I have atm
https://streamable.com/84pny0
It's really only visual atm tho, I've tested some stuff and it definitely is possible to do proper pitch recognition in a browser but it's gonna be a lot of work
@ZerNico there already exists a karaoke game as website, but because of legal issues and licensing fees being astronomically high if you are not a big Sony Music licensor or similar, no one did continue to put any more time into such games. code: https://github.com/Katharine/ponytone Website with very niche indie-music content: https://ponytone.online/
Thanks, it looks pretty interesting!
My plan was to make it for self hosting and not for one big page so legal issues shouldn't be an issue
Also audio processing changed a lot recently, audioworklets and web assembly should improve the pitch detection quite a bit