docs: document collab naming mismatch, cross-profile dedup gap, and VP9
Three findings from the 2026-09-01 collab-detection/VP9 investigation in Cosmo-Live-Downloads, all rooted in gallery-dl's own scraping output: - Collab posts' JSON is filed under the true owner's identity but the media file under the scraped account's — plus coauthors is a native, uncaptured signal that already excludes the owner for us. - The shared archive-db dedups by media_id across all profiles, not per profile, so a post reachable from two profiles' listings only gets its media written into whichever profile's crawl reached it first (510 such posts found archive-wide). - Some video_versions entries are VP9; format selection is codec-blind, and Safari's VP9 decode only wires into its WebM demuxer, never MP4. Also commits the two saved raw API response examples and the missing- captions CSV generated during that investigation.
This commit is contained in:
@@ -276,6 +276,64 @@ viewer retire the lone-video heuristic in `src/lib/post-tabs.ts` — see
|
||||
`/p/<shortcode>/` URL leaves it `null`. Sync always uses listing URLs, so this
|
||||
only matters when testing by hand.
|
||||
|
||||
### Collab posts: the JSON and the media disagree about whose post it is
|
||||
|
||||
Verified directly against two saved raw API responses (`docs/example-api-
|
||||
response-DcU3dM-gVgB.json`, a 2-way collab, and `docs/example-api-response-
|
||||
DcLDme7zoyo.json`, a 5-way collab owned by an external account), 2026-09-01.
|
||||
|
||||
For an Instagram Collab, `username`/`fullname`/`owner_id` in the listing
|
||||
response are always the **original poster's**, never the scraped account's —
|
||||
even when the scraped account is one of the collaborators, not the owner. The
|
||||
metadata `.json` sidecar (built from `include`, above) is filed under that
|
||||
same original-poster identity, since its filename template uses
|
||||
`{username}`. But the **media file** for that same post is written into
|
||||
whichever profile's own crawl directory triggered the download, and its
|
||||
filename's `{username}` slot took the *scraped* account's name, not the true
|
||||
owner's. Net effect: one physical post produces a JSON named for the real
|
||||
owner and a media file (or files, for a carousel) named for whoever we were
|
||||
crawling — two different identities for one post, in the same directory.
|
||||
|
||||
Downstream (`cosmo_normalize_instagram.py` in `Cosmo-Live-Downloads`) had to
|
||||
stop matching media to its JSON by username and match on shortcode/`{num}`
|
||||
only, and separately built a cross-profile `collab_with` pass so every
|
||||
participant's page shows the post, not just the one whose directory JD2/
|
||||
gallery-dl happened to land the JSON in. See that repo's `NOTES.md`,
|
||||
2026-09-01 entries, for the full fix.
|
||||
|
||||
**`coauthors` is a native, richer signal for this that we don't currently
|
||||
capture.** The raw API response carries a `coauthors` array
|
||||
(`{"full_name", "id", "username"}` per collaborator) that **excludes the
|
||||
post's own owner** — confirmed on both saved examples, including one where
|
||||
the owner (`dailyfashion_news`) is an external account with no ARTMS members
|
||||
in her own name, present only via `coauthors`. It is not in the metadata
|
||||
sidecar's `include` list today. Adding it would give collab detection a
|
||||
direct field instead of inferring it from filename/directory identity
|
||||
mismatches, and it never needs the "is this the owner" branch `coauthors`
|
||||
already excludes for us.
|
||||
|
||||
### The shared archive-db dedups media across profiles too, not just within one
|
||||
|
||||
The skip-archive DB (see "Incremental sync" above) keys purely on
|
||||
`instagram_<media_id>`, with no per-profile scoping. When the *same* media_id
|
||||
is reachable from more than one profile's listing — a Collab post, or a
|
||||
repost — whichever profile's crawl reaches it **first** downloads the file;
|
||||
every other profile that later lists the same media_id sees it as
|
||||
already-in-the-archive-db and skips the download, even though that file has
|
||||
never actually landed in *that profile's own* directory tree. The metadata
|
||||
`.json`/`.txt` sidecars are written regardless (they aren't gated by the
|
||||
download-archive), so the symptom is a JSON with zero matching media files
|
||||
in its own directory — 510 such posts were found across the real archive on
|
||||
2026-09-01, entirely from this mechanism, not from anything actually missing
|
||||
from Instagram.
|
||||
|
||||
This isn't fixable on the gallery-dl side without per-profile archive DBs
|
||||
(which would defeat the point of skip-archive — re-downloading anything a
|
||||
sibling profile already fetched). The fix lives downstream instead: `cosmo_
|
||||
normalize_instagram.py` builds one archive-wide `shortcode → path` index once
|
||||
and falls back to it when a post's own directory has no matching media. Worth
|
||||
knowing before assuming a JSON-with-no-media post reflects a real scrape gap.
|
||||
|
||||
## Cadence, and the budget that enforces it
|
||||
|
||||
**Monthly for everything, daily for stories only.** Stories expire in 24h and
|
||||
@@ -559,6 +617,22 @@ Verify by asking gallery-dl's own interpreter, not the shell:
|
||||
- **Highlights already have two naming styles on disk**, and every undated file
|
||||
has a dated twin. The scanner dedupes by index so they render once; it is
|
||||
wasted disk, not a display bug.
|
||||
- **Some `video_versions` entries are VP9, and format selection is
|
||||
codec-blind.** The extractor picks `max(video_versions, key=lambda x:
|
||||
(x["width"], x["height"], x["type"]))` — resolution only, no codec check
|
||||
(`instagram.py`). Instagram appears to have started serving VP9-encoded
|
||||
highest-resolution variants for some posts around when the new gdl-based
|
||||
workflow started (2026-08); 87 such files were found archive-wide on
|
||||
2026-09-01. VP9-in-MP4 plays fine everywhere gallery-dl was tested from
|
||||
except **Safari/WebKit**, which wires VP9 decode only into its WebM
|
||||
demuxer, never its MP4/ISOBMFF path — confirmed via WebKit bug trackers,
|
||||
not guessed. Fetching a lower-resolution non-VP9 variant instead was
|
||||
considered and rejected (quality loss); the fix is downstream, a one-time
|
||||
`-c:v copy -c:a libopus` remux to `.webm` (`cosmo_remux_instagram_vp9.py`
|
||||
in `Cosmo-Live-Downloads`) that keeps VP9 losslessly and only re-encodes
|
||||
audio (WebM disallows AAC). Confirmed live via gallery-dl/yt-dlp that
|
||||
Instagram never offers a native WebM option to request instead — this has
|
||||
to be done locally, there's no source-side fix.
|
||||
|
||||
## Scanner work — done
|
||||
|
||||
|
||||
Reference in New Issue
Block a user