Can I search my video files by what is in them?

Not natively, no operating system indexes the picture content of a video. It becomes possible once each clip has a text description attached, generated by a vision model. You then search the text, which your operating system does index. The description is the bridge between pixels and search.

How accurate is object recognition on video footage?

Reliable for common categories: people, vehicles, animals, buildings, food, tools, water, vegetation, indoor versus outdoor. Less reliable for fine-grained identification, specific species, brands, or named places, and unreliable for anything small, blurred or briefly on screen. Treat it as a strong first filter, not a guarantee.

Can I search for a specific moment inside a long clip?

That needs timecode-level indexing, which is a much heavier job than describing a clip once. For spoken content it is solved, since transcripts carry timecodes. For visual content, per-clip descriptions plus a quick scrub within the found clip is the practical compromise for most workflows.

Does object search work offline?

Yes. Vision models small enough to run on an Apple Silicon Mac produce usable descriptions with nothing leaving the machine. Expect broader categories than a cloud model gives you, so fish rather than pufferfish, which is often enough to find the clip.

Search videos by objects: what actually works, and what does not

How object recognition on footage really performs, the difference between tags and descriptions, and why timecode-level search is a different problem.

21 July 2026 6 min read

“I know I have a shot of a red car driving past.” That sentence is the entire problem of video search, and until recently the only answer was to open clips until you found it.

Object search changes that, but not in the way the demos suggest. Here is what it actually does, how precise it really is, and how to get it working on your own drives.

Why your Mac cannot do this already

Spotlight is a good index. It knows every filename, every folder, the full text of your documents, the EXIF of your photos, the duration and dimensions of your videos.

It does not know what a video looks like. No operating system indexes picture content, because until recently that meant decoding every frame of every file and running it through a model, which was not remotely affordable.

So the honest framing of “search videos by objects” is a two-step job:

Generate text that describes what is in each clip
Search that text with the tools you already have

Step two has been solved since 2005. Step one is the new part, and it is where all the interesting behaviour is.

How the description step works

A vision model does not watch your clip. It samples frames, typically somewhere between eight and thirty depending on the clip’s length, and looks at them.

That sampling has consequences worth understanding:

Things that persist get found. A car parked in shot for ten seconds will appear in a sampled frame.
Things that flash by get missed. A bird crossing the frame for half a second probably falls between samples.
The description reflects the whole clip, not a moment. You get “a coastal road with traffic”, not a per-second log.

For most footage this is exactly right, because you are looking for the clip, not the frame. You want to get from 400 files to three, and then use your eyes.

What it gets right

After a few thousand clips, the reliable categories are consistent:

People and their arrangement. One person, a group, a crowd. Sitting, standing, walking, talking to camera. Close-up, wide.

Vehicles and transport. Car, truck, bicycle, boat, plane, train. Colour usually, model rarely.

Places, by type. Kitchen, office, street, forest, beach, mountain, stadium, warehouse. This is one of the strongest categories and the most useful in practice.

Common objects. Food, laptops, tools, instruments, furniture, signage, animals at the level of “dog” and “bird”.

Conditions. Day or night, indoor or outdoor, sunny or overcast, golden hour. For matching shots in an edit these are worth as much as the subjects.

Camera and framing. Static versus handheld, drone shot versus ground, close-up versus wide, obvious camera moves.

Combine two or three of those and you have a real search: “kitchen, two people, daylight” narrows 400 clips to four.

What it gets wrong

Equally consistent, and worth knowing before you rely on it:

Fine-grained identification. “Fish” is dependable. “Pufferfish” depends heavily on the model. A cloud model will often get species, breeds and plant types right. A small on-device model usually gives you the category.

Named places and people. A model may recognize a famous skyline. It will not know your client’s building, and it does not know that the person in frame is Marcus unless you told the system so. Worse, it may confidently name the wrong city, which is more expensive than saying nothing.

Text in frame. Cloud models read signage and whiteboards surprisingly well. Small local models often do not.

Anything small, fast or partially hidden. The prop in the corner, the logo on the mug, the thing that only appears at 0:42.

Counting. “Three people” is often “several people”. Do not build a search that depends on exact counts.

The practical stance: use generated descriptions as the filter that gets you from hundreds of clips to a handful, and use your own eyes for the last step. That is a 95 percent reduction in scrubbing, which is the whole point.

Tags or descriptions? Both, for different searches

Two forms of generated text, and they serve different queries.

Tags are a controlled, short list: kitchen, interview, daylight, handheld. Good for filtering and grouping, because the same word is always used for the same thing. This is what makes a B-roll tag system work.

Descriptions are a sentence: “two people at a kitchen counter, one talking to camera, morning light through the window”. Good for full-text search, because you can find them with any of the words in them, including ones you did not think to define as a tag.

Use both. Tags in the file’s Finder tags, description in the filename and in a sidecar text file. Then a broad search hits the description and a precise filter hits the tags.

Where to store it so the search works

Generating the text is only useful if the text lands where your search tools look. On a Mac:

Filename: Kitchen-Interview-Two-People-Morning_16x9_4K_25FPS.mp4. Visible everywhere, searchable everywhere.
Finder tags: the controlled vocabulary. Indexed by Spotlight, filterable from any open dialog.
Sidecar text file: the longer description and the transcript. Spotlight indexes the contents.

Notice what is not on the list: a proprietary catalog. If the description only exists inside one application, then “search my footage” means “open that application first”, and on an archive drive from two years ago, that application may not be installed any more. The longer argument is in where footage metadata should live.

The timecode question

The obvious next request: not “which clip has the red car” but “at what second does the red car appear”.

That is a genuinely harder problem. It means indexing at frame or second granularity, which multiplies the analysis cost by one or two orders of magnitude and produces an index that no file system can hold in a filename.

Where it stands today:

Spoken content: solved. Transcripts carry word-level timecodes as a matter of course, so searching for a phrase gives you the exact moment. If your material is interviews, you already have moment-level search. See transcribing video on a Mac.
Visual content: partially, at a price. Systems that do this exist, mostly server-side and priced per hour of footage.

For most independent work, clip-level visual search plus moment-level spoken search covers the real queries. You find the clip in seconds and then scrub thirty seconds of it, instead of scrubbing four hours.

Cloud or on-device

Both work. They fail differently.

Cloud gives you the fine-grained answers: species, landmarks, readable text, specific objects. Frames leave your machine and you pay per clip.

On-device on Apple Silicon gives you the categories: fish, coast, kitchen, crowd. Nothing leaves the machine, no per-clip cost, works with no connection. For NDA material this is not a preference, it is the only version that passes a contract review. Local AI video search covers what to expect in detail.

The useful default: on-device for bulk, cloud for the material where a wrong or vague description would cost you time later.

Cliptag does the description step on macOS. It analyzes video, photos and audio, writes a descriptive filename, applies subject tags as Finder tags, and saves transcripts as text next to the clip, so your search runs in Spotlight rather than in another application. Free plan with 25 cloud analyses a month, and the on-device mode is free and unlimited.