docs: document collab naming mismatch, cross-profile dedup gap, and VP9

Three findings from the 2026-09-01 collab-detection/VP9 investigation in
Cosmo-Live-Downloads, all rooted in gallery-dl's own scraping output:

- Collab posts' JSON is filed under the true owner's identity but the
  media file under the scraped account's — plus coauthors is a native,
  uncaptured signal that already excludes the owner for us.
- The shared archive-db dedups by media_id across all profiles, not per
  profile, so a post reachable from two profiles' listings only gets its
  media written into whichever profile's crawl reached it first (510 such
  posts found archive-wide).
- Some video_versions entries are VP9; format selection is codec-blind,
  and Safari's VP9 decode only wires into its WebM demuxer, never MP4.

Also commits the two saved raw API response examples and the missing-
captions CSV generated during that investigation.
This commit is contained in:
2026-09-01 23:05:19 -04:00
parent 7638fa4a9a
commit d92fed1934
4 changed files with 660 additions and 0 deletions
+74
View File
@@ -276,6 +276,64 @@ viewer retire the lone-video heuristic in `src/lib/post-tabs.ts` — see
`/p/<shortcode>/` URL leaves it `null`. Sync always uses listing URLs, so this
only matters when testing by hand.
### Collab posts: the JSON and the media disagree about whose post it is
Verified directly against two saved raw API responses (`docs/example-api-
response-DcU3dM-gVgB.json`, a 2-way collab, and `docs/example-api-response-
DcLDme7zoyo.json`, a 5-way collab owned by an external account), 2026-09-01.
For an Instagram Collab, `username`/`fullname`/`owner_id` in the listing
response are always the **original poster's**, never the scraped account's —
even when the scraped account is one of the collaborators, not the owner. The
metadata `.json` sidecar (built from `include`, above) is filed under that
same original-poster identity, since its filename template uses
`{username}`. But the **media file** for that same post is written into
whichever profile's own crawl directory triggered the download, and its
filename's `{username}` slot took the *scraped* account's name, not the true
owner's. Net effect: one physical post produces a JSON named for the real
owner and a media file (or files, for a carousel) named for whoever we were
crawling — two different identities for one post, in the same directory.
Downstream (`cosmo_normalize_instagram.py` in `Cosmo-Live-Downloads`) had to
stop matching media to its JSON by username and match on shortcode/`{num}`
only, and separately built a cross-profile `collab_with` pass so every
participant's page shows the post, not just the one whose directory JD2/
gallery-dl happened to land the JSON in. See that repo's `NOTES.md`,
2026-09-01 entries, for the full fix.
**`coauthors` is a native, richer signal for this that we don't currently
capture.** The raw API response carries a `coauthors` array
(`{"full_name", "id", "username"}` per collaborator) that **excludes the
post's own owner** — confirmed on both saved examples, including one where
the owner (`dailyfashion_news`) is an external account with no ARTMS members
in her own name, present only via `coauthors`. It is not in the metadata
sidecar's `include` list today. Adding it would give collab detection a
direct field instead of inferring it from filename/directory identity
mismatches, and it never needs the "is this the owner" branch `coauthors`
already excludes for us.
### The shared archive-db dedups media across profiles too, not just within one
The skip-archive DB (see "Incremental sync" above) keys purely on
`instagram_<media_id>`, with no per-profile scoping. When the *same* media_id
is reachable from more than one profile's listing — a Collab post, or a
repost — whichever profile's crawl reaches it **first** downloads the file;
every other profile that later lists the same media_id sees it as
already-in-the-archive-db and skips the download, even though that file has
never actually landed in *that profile's own* directory tree. The metadata
`.json`/`.txt` sidecars are written regardless (they aren't gated by the
download-archive), so the symptom is a JSON with zero matching media files
in its own directory — 510 such posts were found across the real archive on
2026-09-01, entirely from this mechanism, not from anything actually missing
from Instagram.
This isn't fixable on the gallery-dl side without per-profile archive DBs
(which would defeat the point of skip-archive — re-downloading anything a
sibling profile already fetched). The fix lives downstream instead: `cosmo_
normalize_instagram.py` builds one archive-wide `shortcode → path` index once
and falls back to it when a post's own directory has no matching media. Worth
knowing before assuming a JSON-with-no-media post reflects a real scrape gap.
## Cadence, and the budget that enforces it
**Monthly for everything, daily for stories only.** Stories expire in 24h and
@@ -559,6 +617,22 @@ Verify by asking gallery-dl's own interpreter, not the shell:
- **Highlights already have two naming styles on disk**, and every undated file
has a dated twin. The scanner dedupes by index so they render once; it is
wasted disk, not a display bug.
- **Some `video_versions` entries are VP9, and format selection is
codec-blind.** The extractor picks `max(video_versions, key=lambda x:
(x["width"], x["height"], x["type"]))` — resolution only, no codec check
(`instagram.py`). Instagram appears to have started serving VP9-encoded
highest-resolution variants for some posts around when the new gdl-based
workflow started (2026-08); 87 such files were found archive-wide on
2026-09-01. VP9-in-MP4 plays fine everywhere gallery-dl was tested from
except **Safari/WebKit**, which wires VP9 decode only into its WebM
demuxer, never its MP4/ISOBMFF path — confirmed via WebKit bug trackers,
not guessed. Fetching a lower-resolution non-VP9 variant instead was
considered and rejected (quality loss); the fix is downstream, a one-time
`-c:v copy -c:a libopus` remux to `.webm` (`cosmo_remux_instagram_vp9.py`
in `Cosmo-Live-Downloads`) that keeps VP9 losslessly and only re-encodes
audio (WebM disallows AAC). Confirmed live via gallery-dl/yt-dlp that
Instagram never offers a native WebM option to request instead — this has
to be done locally, there's no source-side fix.
## Scanner work — done