Local Player: transcode fallback and Whisper in a P0-P3 queue

Play can never depend on AI. Transcode and subtitles here are supporting actors — if they’re missing, the video plays just the same.

Context

In parts 1 and 2 I told you about the foundation and the progress system. Now the heavy part: fallback transcode (server/transcode/jobs.js) and local Whisper subtitles (server/subtitles/pipeline.js at 1300+ lines).

My rule from the start: a compatible file plays directly; everything else only steps in if the <video> fires error. And subtitles never block playback.

The problem I wanted to solve

My library mixes .mp4 with .mkv, .avi, and .mov that the browser can’t decode, plus lots of lessons with no subtitles and quiet audio. Converting everything by extension would burn CPU and disk space for nothing. And waiting on AI before pressing play would kill the experience.

So I split it: codec-based decisions with ffprobe (never by extension), progressive transcode in the same <video>, and a priority subtitle queue that never tries to do the whole library at once.

The most important decision

Two queues, one shared semaphore.

Transcode: jobs with dedup + FIFO, one at a time. Whisper: P0 what I opened, P1 the next lesson, P2 the first of each course after the scan, P3 the rest in batches of 20. Both share heavySlots (1 by default) — one waits for the other on purpose; it’s not a deadlock. The LLM stays out of this.

flowchart TD
  Erro["<video> error"] --> Fallback["/api/video/fallback"]
  Fallback --> Probe["ffprobe por codec"]
  Probe --> Direto["compatível: serve direto"]
  Probe --> Transcode["incompatível: job FIFO"]
  Aula["abrir aula"] --> P0["P0 demanda"]
  P0 --> P1["P1 próxima aula"]
  P1 --> P2["P2 primeiras pós-scan"]
  P2 --> P3["P3 background"]
  Transcode --> Heavy["heavySlots (1)"]
  P0 --> Heavy

I ruled out transcoding everything at scan time: on a USB stick it would be unfeasible in time, CPU, and space — and it would touch files that already played fine.

How I used AI in development

It helped me a lot with the text extractors for the tutor (PDF, Office, code, no dependencies), web search with anti-SSRF, and the token-saving skills (server/ai/skills.js: Caveman, RTK, Headroom, ADHD).

But it tried three slip-ups: using +faststart on progressive MP4 (breaks the fragmented one), using shell:true with paths containing spaces, and queueing the entire library. I corrected to -movflags frag_keyframe+empty_moov+default_base_moof, spawn with an args array, and P0-P3 with preemption of cheap jobs only.

What I discovered implementing it

• Subtitles: ffmpeg extracts 16kHz mono WAV → whisper.cpp → deterministic post-processing → canonical VTT in <lib>/.courseplayer/subtitles/<hash>.vtt + mirror in data/subtitles/. Raw is never overwritten; it’s reused if mtime+size match; force=1 regenerates.
• Transcode: cache in data/transcoded/sha1(...).mp4, invalidated by mtime; .tmp only becomes final via rename after exit 0; orphan .tmp returns 404.
• Seeking past what’s converted so far waits up to 60s or answers 416. Progress in the log every 25%.
• On the frontend I swap src on the same <video> (so I don’t lose the GainNode and listeners); position and volume come back once the segment is in buffered; a “Preparing compatibility…” badge appears with “Try again”.
• Subtitles are a custom overlay, never <track>; status via /api/subtitles/status; on open, it generates P0 and already queues the next one as P1. With no binary, model, or internet, the status says available:false and it plays normally.
• Audio: volume up to 100% on the control, beyond that via GainNode, with a normalizer, 0.5–2x speed saved.

What worked

• Without ffmpeg: a clear message, not a raw 500.
• Subtitle editor switched off on the frontend (no button, ?editSubtitles=1 ignored), but routes kept; saving with a diverged version returns 409, with a backup before regenerating.
• Tutor with lesson context (transcript + materials + timestamps) via SSE, key only on the backend, never in logs.

What didn’t work

• Whisper doesn’t run on FAT/vfat — I documented it and the temporary workspace goes to os.tmpdir().
• My first subtitle version with <track> fought styling and geometry; the overlay measured on the real video area fixed it.

What I’d do differently today

I would have created heavySlots as its own module from day one, instead of spreading the logic between transcode and whisper. It was born as a flag and became an invariant.

Result

Validated by docs/VALIDATION.md: .mkv shows the badge, plays in seconds, logs [TRANSCODE], saves to data/transcoded/; subtitles generate raw → canonical VTT + mirror; duplicate POST dedups; P0–P3 without sweeping everything; clearing transcode doesn’t touch progress.

What I learned

1. Fallback, not default

Deciding by codec after the error saves more CPU than any ffmpeg preset.

2. Priority is product

P0–P3 says what’s on my screen right now; pure FIFO would treat background as urgent.

3. Add-ons must degrade

Every AI path has its available:false tested. Without that, vibecoding becomes coupling.

Next steps

• Part 1: vibecoding with invariants and zero build step
• Part 2: atomic progress that travels with the drive

Conclusion

Transcode and Whisper are the flashiest part and the easiest to get wrong. The right call was treating both as second-class to playback — with a queue, a semaphore, and explicit degradation.

References

• server/transcode/jobs.js, server/subtitles/pipeline.js, server/subtitles/vtt.js, server/ai/skills.js
• docs/SUBTITLES.md, docs/whisper.md, docs/VALIDATION.md sections 3 and 6