The scheduler now looks backward, and that fixed our double uploads

Our upload scheduler used to work on future timestamps. We would set a video to go live at a specific time, say Tuesday at 3:00 PM. The scheduler held that timestamp and waited. When the clock matched, it tried to upload. If everything went right, the video went out. If anything went wrong, it waited for the next attempt. That sounds simple. It was not reliable.

The problem was that uploads needed a browser. The YouTube upload flow ran in a browser session, and that session had to stay open. If the browser was closed, the timestamp passed and nothing happened. We would wake up to a missed slot and a silent channel. Worse, we sometimes fixed it by uploading late, which pushed the next scheduled upload out of rhythm. The whole schedule drifted.

So we changed the core logic. Instead of looking ahead to a future timestamp, the scheduler now looks backward. It keys off the most recent past slot inside a catch-up window. The catch-up window is a time range that we can tune, usually a few hours. If the current time is 4:10 and the 3:00 slot never fired, that slot is due. The scheduler does not care about 5:00 yet. It only cares about the missed slot that is still fresh.

That backward look makes the hourly cron possible. Every hour, a background process on our servers checks for due slots. It does not need a browser open. It does not need a human watching. It just asks what slot should have fired recently and did not. If the most recent past slot is inside the window, the cron starts the upload pipeline. The cron is the worker. The browser session, when needed, is created on demand inside the pipeline.

We still had one ugly case. The cron is not the only thing that can start a run. A creator might click a button. A different service might trigger an upload after an edit. Two runs can look at the same 3:00 slot at the same time. Without a guard, both would upload. YouTube would get two copies from one slot. That is a double publish.

The last-release stamp fixes that. After a successful publish, the pipeline writes a timestamp to storage. We treat that timestamp as the most recent true release. Any run that sees the 3:00 slot first checks the stamp. If the stamp is newer than the slot time plus a small buffer, the slot is already handled. The second run skips. The first run wins. The stamp is not clever. It is just a fact that gets checked every time.

The stamp also works with the catch-up window to prevent weird backlogs. Imagine the site is down for a day and comes back at 5:00 PM. The scheduler finds the 3:00 slot still inside the window. It also sees older slots from 1:00 and 11:00 AM, but those have fallen outside the window. Only the most recent one matters. The cron drains that one. The old videos are not queued up like a pile of overdue homework. The channel stays current. The stamp stays current. The system moves forward.