Transcripts and text-based editing

Turn speech into words with times, in the browser, then read, search and cut the edit like text: in the Transcript tab, or from an agent with fablecut_transcribe.

FableCut turns the speech in a clip into words with times, so an edit can be read, searched and cut like text. Transcription runs in the browser: Whisper (OpenAI's speech model, through transformers.js) in a background worker of the editor tab. The library and the model are fetched from a CDN the first time and kept by the browser after that; the audio never leaves the machine.

Whisper detects the language by itself; give it one (en, es, hi, ja, … any Whisper code) when a clip is short or mixes languages. Long files are split at their quietest moments into pieces of at most 28 s, and stretches of silence are skipped (Whisper invents words in silence).

One transcript per file, kept by the server in analysis/transcripts/: {v, src, model, language, duration, created, words:[[text, start, end], …]}, times in seconds of the file. Like a proxy it belongs to the file's size and modification time, so a replaced or re-rendered file has none until it is transcribed again. It is not stored in project.json, so it never bloats the timeline, and every clip and sequence that uses the file reads the same one.

On the timeline a clip shows the words its window of the file holds, placed through its in point and speed (a word counts when its middle is inside the clip). A clip with a speed ramp is left out, and so are disabled clips and pictures whose sound is off.

Cutting words extracts on the targeted tracks, from just before the first word to just before the word after the last one: the pause after the cut goes too, the pause before it stays. Cuts land on whole frames, never in a neighbouring word or past the clip. Several cuts go last first, as one undo step (one revision for an agent): later material moves up, linked stems stay in step, and locked clips stay put - lock the music bed first if it should keep running under the tightened edit. Find matches whole words and ignores case, accents, punctuation and spacing: "ai powered" finds "A.I.-powered", and "cat" doesn't find "category".

In the editor, the Transcript tab beside the Inspector:

• Transcribe works on the selected clips' files, else on every file on the timeline that has no transcript yet. The model and the language are beside it. Progress shows below (the first time, the model download), with Cancel.
• The timeline's words, in paragraphs with timecodes. Click a word to go there; the word under the playhead is underlined and followed while playing. ⇧-click or drag to select a run of words, then press Delete (or Cut) to extract it.
• Find words marks every match, Enter steps through them, and Cut N extracts them all. Copy copies the text with its timecodes.

Agents use fablecut_transcribe. It transcribes in an open editor tab, else in headless Chrome / Edge, whose profile is kept in analysis/browser-profile/ so the model downloads once:

• {} transcribes every clip with sound that has no transcript yet, then returns the timeline's words as timed lines: [0:12.08] Open the demo, nothing to install. clipIds limits it to some clips; from / to limit the time (the reply stops at 300 lines); mediaIds reads whole files in file seconds instead.
• model:"small", language:"es", and again:true (transcribe again, with another model or language).
• find:"phrase" (or a list) reports each place it is said, with its span, the words around it and how much cutting them would remove; cut:true extracts them (tracks and force as for fablecut_silence). A version of the project is kept before the cut.

Whisper writes clean text: it often leaves out fillers ("um", "uh") and false starts, so find those by ear (fablecut_silence for the pauses), not by text. Word times are good to about a tenth of a second; listen to a cut that matters, or check it with fablecut_frame.