In August 2021, a co-founder from an Icelandic startup, OverTune, reached out to me and asked for help porting Auto Tune to iOS for their app. Pétur, the co-founder, told me the app was in a messed-up state and they needed help fixing it. I tried recording myself to see what the app currently was supposed to do, but all I got back were crackling noises resembling bits and pieces of my voice.

While debugging, I remember a developer friend pointing me to this article which left me inspired: Four Common Mistakes In Audio Development.

After recalling this example, I remembered one of the rules brought up in this article: never allocate memory on an audio thread. Calls like malloc() have to go through the kernel and result in a non-deterministic run-time.

After I removed the similar Objective-C + alloc calls I found near the audio thread, the crackling was gone. This left the team very happy, and we were able to proceed to making even cooler stuff—such as the initial voice recording interface I designed in SwiftUI shown above.

Another challenge to tackle was the case when performers were not recording with headphones.

This isn't uncommon in a non-performance scenario, such as taking a call on speakerphone. Phones have special processing, such as adaptive filters—preventing the far-end from hearing their own echo.

With the feedback in this recording, it was a little trickier to address since the speaker is near the microphone. One solution I came up with was creating a noise gate in C++, similar to the one used in Discord and other voice chat apps. The result turned out decent, especially in between the parts where the vocals weren't happening.

This noise gate proved to be useful for another module which I developed for voice effects: a vocoder to make people sound like robots.

To fuel the vocoder's carrier signal (the one which alters the voice), I was using some of the classic periodic signals—such as sawtooth, square, and triangle waves.

Although, the carrier also needed a way to track the pitch of the voice to make it sound similar to the Auto Tune. Pitch tracking was challenging because the sibilant sounds—as well as background noise—are high in frequency and throw the pitch tracking algorithm off a bit. This is where the noise gate and a couple of filters came into play.