Playlist ingestion with parent URL provenance

A “mover” job is rarely glamorous. It is forty episodes of a podcast, or two hundred Shorts from a brand channel, or three years of a YouTube series. Editors will eventually do something interesting with the files; your job is to land them, label them, and make sure no one has to ask “wait, which channel did this come from?” three months later.

Provenance is the part nobody pays attention to until they need it. So they pay attention to it once, then forever.

Here is what trips most people up

Operators feed individual episode URLs into the queue and lose the playlist context. Most provenance failures happen at ingest, not at archive. If you queue 187 individual videos, you have 187 orphaned rows; if you queue the parent playlist URL and let the queue explode it, every row carries the parent URL as metadata and you can answer “where did this come from” without git-blame on a folder.

The parent URL rule

Whenever a platform exposes a playlist, channel, or series URL, send the parent URL to the queue, not the leaf URLs. The queue should:

1. Resolve the playlist into its current children at ingest time.
2. Stamp each child with the parent URL, the playlist title, and the position in the playlist.
3. Persist that metadata as a sidecar JSON next to the file.

This metadata is small and boring. It pays off the day someone asks “is this episode 7 or episode 8?” — the sidecar says "playlist_index": 7, "playlist_title": "Series 03" and you stop guessing.

The Clipr ingest writes this sidecar by default; under features the Movers section calls it out per audience. If you are using a different tool, write the sidecar by hand. It is worth the disk space.

Deduplication across runs

The second time a producer asks for “the same channel, this week’s episodes only,” your queue should refuse to re-download the 14 episodes you grabbed last week. The check is simple but easy to get wrong:

• Hash-based dedup on the source URL canonical form (strip tracking params, normalize protocol).
• Fingerprint-based dedup for files that already landed (size + first 4 KB checksum is sufficient for 99% of cases).
• Library-aware dedup when a Team library exists — check the shared catalog before queuing, not after.

Skip dedup based on title alone. Titles get edited. URLs and bytes do not.

Resume after VPN drops

Long ingests fail in the middle. The recoverable shape:

• Range-request resume: most CDNs support Range headers; partials should resume, not restart, when the queue comes back online.
• Checkpoint per N items: when ingesting a 200-episode playlist, persist progress every 5–10 items. A crash at episode 137 should resume at 138, not at 1.
• Detect manifest expiry: the manifest URL for a single video typically expires in 6–12 hours. If you resume the next morning, the queue should re-request the manifest, not retry against a stale URL.

The third one is the subtle one. A naive queue retries the same manifest URL after a 12-hour pause and gets 403s; a polite queue notices the gap and re-resolves the manifest first. Watch for the “stale manifest” badge on aged jobs in the UI.

Chapter markers and metadata that survive

YouTube exposes chapters; many podcast episodes have them. They survive an ingest only if you keep them. The Clipr ingest writes chapters into the container metadata when present, plus a sidecar chapters.json. If you re-encode in a downstream step, copy the chapter atom forward — most NLE pipelines drop it silently.

The same goes for:

• Original publish date (not the download date — keep both).
• Source title (the title at the time of capture, even if the channel renames it later).
• Original description (legal sometimes wants this for fair-use claims).
• Thumbnail URL (often expires; cache the bytes, not the URL).

If you only carry the file forward and drop the metadata, you have a video. If you carry both, you have an asset.

Folder layout that stays sane

For mover work, a flat-by-channel layout beats a nested-by-show layout:

/library/raw/2026-04/
  channel-a__series-03__ep01.mp4
  channel-a__series-03__ep01.json   # parent URL + chapter sidecar
  channel-a__series-03__ep02.mp4
  channel-a__series-03__ep02.json
  ...

Two delimiters (__ and -) separate channel / show / episode. The sidecar JSON stays adjacent. When an editor ingests, they grab the .mp4 and the sidecar in one shot. When archive happens later, the layout maps cleanly to a channel-a/series-03/ tree without parsing.

Filename schemes are a religion; pick one and stop debating. The principle that matters is: the filename should never be the only carrier of metadata. Filenames get renamed. Sidecars are harder to lose.

What if it does not work?

• The playlist returns fewer items than the page shows: the source platform paginates or hides items behind login. Re-queue from the signed-in tab; check whether the playlist has a ?si= or pagination parameter.
• Resume restarts from byte 0: the source CDN does not honor Range. Pre-allocate disk and accept the re-download; do not loop.
• Chapters disappear after re-encode: the encoder did not preserve the chapter atom. Add -map_chapters 0 (or your toolchain’s equivalent) to the encoder args.
• Two operators ingest the same channel and produce two libraries: missing dedup at the team level. Promote the channel to a “watched playlist” in the shared library so further ingests check the catalog first.
• Sidecars exist but downstream tools ignore them: standardize on one sidecar shape (Clipr writes a flat JSON; many studios use info.json à la yt-dlp). Convert at the boundary, not in the middle.

A throwaway script you should not throw away

If you ingest weekly, you want a one-line audit:

find /library/raw -name "*.mp4" | while read f; do
  s="${f%.*}.json"
  [ -f "$s" ] || echo "ORPHAN: $f"
done

Run it after every ingest. The first week it finds 12 orphans, you go fix the ingest config. The second week it finds zero. From then on it is a smoke test.

Closing

Mover work rewards boring discipline. Send parent URLs, write sidecars, dedup before queueing, resume manifests after gaps, and never let a filename be the only carrier of provenance. Do those five things and the question “where did this file come from” stops being answered with a Slack scrollback.

For the per-feature description of playlist ingestion and chapter handling, see features. For tier limits on shared libraries, see pricing. The ingest UI ships in the download.