How AI Watches Cooking Videos: Inside Pluck's Multi-Modal Extraction Pipeline

Imagine a 30-second Instagram Reel where someone makes shakshuka. Text flashes on screen for two seconds — “1 can crushed tomatoes” — while the creator says “I like to add a little cumin here, maybe half a teaspoon.” There’s no written recipe in the caption and no blog post. The entire recipe exists as a tangle of audio, video, and fleeting on-screen text.

How does an AI turn that into a structured recipe with an ingredient list and numbered steps?

Our post on whether AI can actually extract recipes from video gives the overview. This one goes deeper, into the engineering behind each step.

What “watching” a video actually means for an AI

A person takes in a cooking video all at once. An AI has to do the same thing deliberately and in stages.

Pluck’s extraction pipeline gathers five kinds of signal: video frames, the audio (transcribed to text), on-screen text, the caption and description, and the post’s metadata. It hands them to a multi-modal AI model that reads them together, resolves conflicts and fills gaps.

Video frame sampling: seeing the kitchen

A 10-minute YouTube video at 30 frames per second contains 18,000 frames, most of them nearly identical. Processing them all would waste compute and add no signal.

Instead, Pluck picks a small set of key frames, around 8 to 10 per video. Scene detection finds the moments where the picture changes and the information density is highest: a cut from a prep shot to a stovetop shot, a title card appearing. Pluck also always keeps a frame from the first few seconds, where title cards usually sit, and one from the end, where many creators show a recipe summary.

In each frame the AI looks for ingredients, measurements visible on cups, spoons or kitchen scales, packaging labels, text overlays, and the state of the food, from raw to plated.

Frame analysis is indispensable for silent, ASMR-style cooking videos with no narration and sometimes no text. From the frames alone, the AI can identify most of the ingredients and reconstruct the preparation sequence.

Why frame rate and resolution matter

A 4K YouTube upload gives the AI crisp, legible text overlays and identifiable packaging. A low-resolution TikTok repost compressed through three platforms might have text that’s barely readable even to a human. Then the extraction has to lean on the audio and the caption instead, and the confidence score reflects that.

Audio transcription: listening to the cook

Speech-to-text is the single richest extraction channel for most cooking videos. Creators narrate as they cook, and that narration contains the core recipe: ingredients, quantities, timing, temperatures, and technique.

Modern speech recognition handles the chaos of a real kitchen surprisingly well. The models filter out kitchen noise and background music, and reliably handle the accents, fast speech and mumbling that would have killed older systems.

The AI also parses cooking language. It understands that “throw in a knob of butter” means butter is an ingredient, and that “season to taste” means salt and pepper are ingredients, even though no quantity was stated. And it catches asides: “my grandmother always used lard here but I’m using butter” makes butter the ingredient in this recipe, with the substitution noted.

On-screen text recognition: reading what creators write

A huge portion of short-form recipe video (TikTok, Instagram Reels, YouTube Shorts) uses text overlays as the primary communication method. A trending audio track plays while the creator layers text on screen, and the recipe lives entirely in that text layer.

Pluck doesn’t run a separate character-recognition step. The vision model reads the text in each key frame the way a person would, with layout and context. Large text at the top of the frame at the start of a video is probably a recipe title. Text that flashes briefly while an ingredient is added is probably a measurement. Text that persists across multiple cuts might be a running ingredient list.

Timing matters too. A text overlay that appears at the moment the creator pours milk into a bowl gets associated with that action, so a specific “1 cup whole milk” on screen completes a vague “add some milk” in the audio.

The 300-millisecond problem

Some creators use editing styles where text appears for only 300 milliseconds. Scene detection catches many overlay changes, because new text often arrives with a cut, but a set of key frames can still miss a flash between them. When the recipe comes out incomplete, the confidence score drops so you know to double-check it.

Caption parsing and metadata: the cross-reference layer

The caption, the video description and the post’s metadata form the fourth and fifth signals: a cross-reference layer that validates and enriches what the video and audio say.

They also come first. Before downloading anything, Pluck reads the caption. If the creator already wrote out the full recipe there, Pluck extracts from it directly and skips the slower download, frame and transcription steps. Only when the caption is missing the recipe, or only has part of it, does the full video analysis run.

Pluck does its own transcription of the audio rather than relying on platform auto-captions, which are notorious for mangling cooking terms. YouTube descriptions frequently contain partial or full ingredient lists, timestamps, or links to the written recipe, and TikTok captions might mention key ingredients or the recipe name.

The AI model sees the key frames, the transcript, the caption and the metadata together, and resolves conflicts between them for every ingredient, every measurement and every instruction. Take a YouTube creator making chicken tikka masala:

Frame analysis identifies chicken, tomatoes, cream, onions, and several spices on the counter.
Audio transcription captures “about a tablespoon of garam masala.”
OCR reads a text overlay that says “1 tbsp garam masala.”
Description lists “garam masala” in a partial ingredient list without a quantity.

The model resolves this to “1 tablespoon garam masala,” taking the quantity from the on-screen text and the audio (which agree) and the ingredient identity from audio, on-screen text and description (three-way agreement). If the transcript had misheard “garam masala” as something else, the other signals would outvote it.

Confidence scoring: knowing what you don’t know

Not every extraction is equally reliable, so Pluck assigns each one a confidence score, shown as High, Medium or Low confidence on the review screen. High-confidence extractions, where the signals agreed and the recipe is complete, can be saved quickly with minimal review. A low score comes with a prompt to review before saving and check the ingredient list against the video. Every field is editable.

The score also accounts for completeness and specificity. A recipe with a title, 12 well-specified ingredients, and 8 numbered steps scores higher than one with a title, 4 vague ingredients, and a single paragraph of instructions.

Why different platforms need different strategies

YouTube videos tend to be longer, better narrated, and often include description text. Audio transcription is usually the dominant channel. One caveat: Pluck extracts one recipe per video today, so a “5 easy weeknight dinners” compilation gives you one recipe, not five. Pluck handles videos up to 30 minutes long. Our guide on saving YouTube cooking videos as recipes has the detailed walkthrough.

TikTok videos are short, often loud, and heavily reliant on text overlays, so OCR becomes more important. The audio track is frequently a trending song rather than narration, which means audio transcription produces less signal, and every frame carries proportionally more information. And as we’ve covered extensively, TikTok’s own save feature is unreliable for long-term recipe storage.

Instagram Reels sit somewhere in between. They often combine narration with text overlays, and Instagram’s caption field regularly contains the full recipe in text form. When it does, the caption becomes the highest-confidence source, and the video analysis serves as validation rather than the primary extraction channel.

AI model routing: picking the right tool for the job

Pluck doesn’t rely on a single AI model. The pipeline routes each extraction by the kind of input it is, using models from Anthropic (Claude), OpenAI (GPT, plus Whisper for transcribing audio) and Google (Gemini). A web page, a photo and a video are different jobs, and simple, well-structured pages can go to a faster model while harder sources get a more capable one. Each job also has backup models from other providers, so if one provider is slow or down, the extraction falls back instead of failing.

Why any of this matters when you’re cooking dinner

All of this engineering serves a single practical purpose: you saw a recipe in a video, and you want to cook it without rewinding 40 times, screenshotting it and losing it in your camera roll, or transcribing it by hand.

What comes out the other end is a clean recipe card: title, ingredients, steps, times, servings. The format is the same whether the source was a 30-minute YouTube masterclass, a 15-second TikTok, or an Instagram photo carousel with the recipe spread across slides. Our How Pluck Works page covers all five extraction modes.

A structured recipe is also what makes the rest of the app work. A saved video can go into your meal plan, onto a grocery list where duplicate ingredients are merged and sorted by aisle, and into cook mode, where each step shows the ingredients it uses. The same structure is what lets Fill my week pick from your saved recipes and a pantry scan match a photo of your fridge against their ingredients. Long videos are processed in the background, and you get a notification when the recipe is ready.

Pluck isn’t the only app reading cooking videos anymore. ReciMe, Honeydew and Deglaze also transcribe a video’s audio, and Deglaze says it reads on-screen text frame by frame as well. Many others, including Mela and Pestle, read only the caption. If you’re comparing, the recipe app comparison hub lays out who does what.

Two years ago, none of this was possible at a quality level that was useful for real cooking. And it’s still getting better. The models improve, the confidence calibration tightens, and the category of videos that produce incomplete extractions shrinks with every update.

Want to see multi-modal extraction in action? Pluck is available on iOS and Android. Get it on Google Play. Also on iOS.