The in-process MP4 muxer: keeping video and audio in lockstep

Every release on Autoretto passes through fifteen stages. Most people only hear about the exciting ones. The render gate and the AI policy check get the attention. But there is a stage that sits between the renderer and the verification gate. It is the in-process MP4 muxer. This post is about what it does.

The muxer takes in two temporary files. The first is a raw video track from the compositor. That track already has beat-synced cuts and any Sora motion clips applied. The second is the finished audio track from Suno, after mixing and leveling. There is also a metadata sidecar. The sidecar is generated during the rendering stage. It is not an afterthought. It is the contract between the renderer and the muxer. That sidecar lists the exact frame timestamps for every beat, the expected duration of the audio, and the codec settings for both tracks. The muxer reads the sidecar first. It knows what the final file should look like before it writes a single byte.

The work itself is precise but not glamorous. The muxer writes an ISO Base Media File Format container. It interleaves video samples and audio samples into a shared timeline. It writes the ftyp box, the moov box, and the mdat box. It sets the timescale for each track so both use the same clock. It places keyframes so seeking lands on clean frames. It also handles audio priming. No encoding happens here. The codecs were already chosen. The muxer just wraps them correctly.

The hard part is synchronization. Video frames run at 30 fps. Audio samples run at 44.1 or 48 kHz. The muxer must map each audio packet to a video presentation time. If it gets that mapping wrong, the beat lands a few milliseconds late. That is a defect you don't spot until a viewer complains. So the muxer simulates playback as it writes. It is not a full decode. It's a timestamp trace. For every audio packet and every video frame, it records the presentation timestamp and the byte offset. Then it walks both traces in parallel. If the difference between audio and video timestamps ever exceeds one frame, it rejects the run and starts over with a different interleave order. That is the first proof.

After the file is written, the muxer does a read-back. It opens the MP4 it just created and parses the box structure. It confirms the moov box exists and is not truncated. It checks the mdat box size against the expected byte count. It verifies that the audio track uses one codec and the video track uses another, and both match the sidecar. Then it runs a container probe for duration. If the duration is off by more than 200 milliseconds from the audio duration, the file fails.

The final proof is a checksum. The muxer computes a SHA-256 of the complete file and stores it in the run state. That checksum travels with the file to the next stage. If the render-verification gate or the YouTube pre-publish check opens the file and gets a different hash, they know something changed. The muxer does not just hand off a file. It hands off evidence.

This might seem like overkill for a simple container format. But in an autonomous pipeline, tiny errors become big problems. A corrupted moov box can make a video unplayable on YouTube's importer. A bad interleave can cause audio drift over a long track. The muxer's job is to catch those issues before they leave the machine. It is not the most glamorous stage in the factory. It is the stage that keeps the rest of the pipeline honest.