Building Songora: Rendering a Full Song on a Phone Without Sending Audio to the Cloud

I shipped Songora — a beginner-friendly AI music studio. Describe an idea (or drop your own lyrics), pick a genre and voice, and listen.

The product looks like a prompt. The engineering is a render pipeline.

Two generators, one honest product

By default, Songora can call a cloud path for original lyrics and a spoken vocal preview. That’s useful. It is also not a full band.

When the network is gone — or you force local mode — the app falls back to an on-device sandbox renderer: device TTS + a wavetable instrumental, mixed on the phone.

Quality differs on purpose. I didn’t want offline mode to pretend it was the same studio as the cloud preview.

The interesting problem became:

Can a phone produce a complete song — melody, vocals, drums, bass, mix — without locking the UI, and without lying when a stage fails?

The local pipeline

SongAudioEngine runs in visible stages:

1. Melody — an ArrangementBuilder turns lyrics + genre/mood/tempo/energy into sections, phrase slots, and notes. Creativity shifts the seed so a re-roll actually re-rolls the tune.
2. Vocals — each lyric line is synthesized on the main isolate (platform TTS channels). Melodic processing happens later, off-thread.
3. Instrumental — wavetable layers (pads, chords, bass, arps, drums, per-genre sound design) inside a Dart isolate.
4. Mixing / finalizing — one master buffer. If RMS is essentially silence, throw. Don’t export a blank file and call it done.

A 90-second song is meant to render in a few seconds on a mid-range phone. That’s why synthesis stays wavetable, not a sample library, and why the heavy DSP never sits on the UI isolate.

Vocals: stretch, don’t autotune-by-default

Raw TTS is spoken, not sung. VocalEngine resamples, trims silence, then either:

• fits the line into its rhythmic slot (WSOLA time-stretch) so delivery stays human, or
• splits across melody notes and pitch-shifts when melodic mode is on

Choir is doubling and tiny pitch offsets. Voice styles are gentle formant/pitch colour — not a fake gender swap on a device that only ships one system voice.

The rule that mattered more than the DSP:

If more than half the phrases fail to synthesize, throw VocalSynthesisException. Do not ship an instrumental and label it as having vocals.

That matches a pattern I keep shipping: VidWeave doesn’t “almost” honour a crop. iRun doesn’t invent a finish time. Songora doesn’t invent a singer.

Jobs that survive a kill

Generation is a job with a persisted stage. Kill the app mid-render and it resumes on next launch. Credit spend/refund is keyed by job id — a song can’t be double-charged; a failed generation auto-refunds.

Moderation runs before generation (harmful prompts, artist impersonation). The local ledger is the sandbox source of truth.

What I shipped

Flutter, Riverpod, Hive, just_audio for playback (background + lock screen), a Lyrics Studio (rewrite, shorten, expand, rhyme, translate), occasion templates, and a library with retry/download.

No AI keys in the binary. Cloud credentials stay on a proxy. Local auth and purchases are labeled sandbox until the store backends are wired.

Store links:

• App Store
• Google Play

What I’d still harden

Cloud audio is still a vocal preview, not a produced track with a real band. Owning the proxy (keys, quotas, a fuller music backend) is the next product leap. iOS TTS quality needs a dedicated device pass. And the local renderer will always sound like a sandbox next to a studio model — the job is to keep that gap visible, not papered over.

The bet: offline should still make a song, and failure should still sound like failure.

If you’re generating media on mobile: isolate the DSP, persist the job, and fail the stage that actually broke.