instaarchive-viewer/docs/video-upgrade-plan.md

Video upgrade pass — plan and findings (2026-10-14)

Status: designed and partly measured, NOT built. Nothing in the archive has been changed. Written so this can be picked up cold. Account context: the scraping account is the fresh erg05teur (see docs/gallery-dl.md, top of "The account was suspended…") — every request in this plan is spent on a new account, so the budget rules there apply in full.

Why this exists

The question was whether a logged-out fetch had given us lower-resolution media and whether a logged-in run could upgrade it. Measured result:

• Images: no upgrade needed. Header-scan of 38.9k archived images, plus live metadata for 8 posts across 2023–2026 (including the oddly small ones): archived sizes match exactly what a logged-in fetch offers (1440×1800, 3277×4096, 2976×3720; 400×400 / 742×928 items are small at the source, sitting next to 1440 px siblings in the same carousel).
• Logged-out gdl does not download smaller files — it fails (login redirect). No logged-out media ever entered the archive.
• Videos: this is where there is something to gain. Of ~1,300 archived videos in the six scraped profiles, ~1,200 are 720p (min side ≤ 720) and ~100 (2026 reels) are 1080p. A 2026 reel (DTpycSlDKHs, archived 720×1280) offers 1080×1920 VP9 video-only DASH + separate AAC audio. A 2024 post (C7RmlCcSWNR) tops out at 1276×720 — same as the archive — so older videos probably gain nothing. Expected beneficiaries: 2025–26 videos, roughly 700 posts. Not verified how far back 1080p exists.

Decisions already made (by the owner)

1. Videos in a separate pass from anything else (images need none).
2. Keep the replaced files for review in a _superseded/ area — never overwrite/delete.
3. Output .webm, not .mp4 (video stream copied, audio re-encoded AAC → Opus 96 kbps; the audio stream is ~43 kbps HE-AAC so the loss is negligible). Tested on real hardware: iPhone, iOS 18.7 Safari plays the VP9+Opus .webm at 1080×1920 and fails the VP9+AAC .mp4 (MediaError code 4, no video track) — even though canPlayType reports "probably" for it, so never trust canPlayType here. VP9-in-MP4 (the lossless remux) is therefore out. Re-encoding VP9 → H.264 was rejected (bigger, softer, no gain over the 720p original).

How the sample was produced (reproducible, one post)

# portrait video: 1080 is the WIDTH — `height=1080` selects nothing and fails "format not available"
yt-dlp --cookies-from-browser chrome:/home/matt/.config/google-chrome-devtools \
  -f "bv*[width>=1080]+ba" --merge-output-format mp4 -o 'X.%(ext)s' https://www.instagram.com/p/<shortcode>/
ffmpeg -i X.mp4 -c:v copy -c:a libopus -b:a 96k X.webm        # ffmpeg 7.1.5 on mattellite

Sizes for that post: 6.46 MB .mp4 / 6.53 MB .webm. For a browser test you need HTTP Range support (Python's http.server has none; Safari requires it) — the throwaway server used is not kept. gallery-dl needs $HOME/.local/bin on PATH over plain ssh.

Open decisions / work before building

item	notes
Viewer MIME type	src/lib/archive-cache.ts:127 and src/hooks/useArchiveScanner.ts:272,344,391 mint every video blob URL as video/mp4. Derive the type from the extension (.webm → video/webm) or Safari/iOS may refuse a WebM labelled mp4. MEDIA_RE / isVideo already recognise .webm.
Filename / dedup	The archive names YYYY-MM-DD_<user> - <shortcode>[ - <n>].mp4. Swapping to .webm must not leave both in archives/ (the viewer would show duplicates) — the old .mp4 has to be moved out, not just joined by a .webm. Check parseArchiveFilename (src/lib/archive-patterns.ts) for the extension handling and the JD2-vs-gdl same-stem duplicate logic.
Skip-archive	artms.db keys on instagram<media_id>, so a normal run will never re-fetch these. The upgrade tool must fetch targeted items deliberately and must not touch/poison artms.db semantics.
_superseded/ location	Must be outside /volume1/rslsync/sync/Instagram-archive/archives/ — that tree is scanned by the viewer and inside the Resilio-synced share (it would sync every replaced file to devices). Not decided. Also unchecked: whether agentapi can write outside that share. Use rsync --backup --backup-dir=<dir>/<date>/ so the old copy is kept.
Carousel videos	yt-dlp on a carousel post URL fails ("No video formats found") — see DIa5NpWRnIP. gdl's own metadata gives per-child ytdl: URLs for videos in carousels; the tool needs to handle child items, not post URLs.
Request budget	One yt-dlp -F-style request per post to learn what 1080p exists (plus the download). ~700 candidates ⇒ suggested ≤ 40 posts/day, resumable, behind check-session.py, hand-started or a slow timer — never a burst. Candidates can be pre-ranked from the archive alone (720p, 2025–26) at zero request cost.
Storage	1080p VP9 ≈ 1.6× the 720p file (6 MB vs ~4 MB for that sample, plus the kept original). Estimate before running at scale.

Planned design

1. Candidate list, zero requests: scan the NAS archive (read-only, ssh agentapi@<nas> from mattellite; the scans in this doc used header parsing in plain Python over stdin because the NAS has no ffprobe) for videos ≤ 720 min-side from 2025-01 onward, newest first.
2. Dry-run mode (default): for each candidate, one request to learn the best offered stream, record {shortcode, archived WxH, offered WxH} in a state file; print counts + estimated GB. Downloads nothing.
3. Execute mode: only where offered width > archived width; download video+audio, ffmpeg -c:v copy -c:a libopus -b:a 96k, verify with ffprobe (dimensions, duration within 0.5 s of the original, both streams present), publish the .webm, move the old file to _superseded/. Cap per run (--max-posts), pace with the same 12–20 s gaps, stop on any 429 / login redirect.
4. First step when resuming: do ONE reel end to end by hand (DTpycSlDKHs, ~3 requests) to prove download → remux → verify → publish → supersede → viewer playback, then the dry-run, and only after the weekly profiles timer has run cleanly on the new account (first fire ≈ 2026-10-23).

Things measured along the way (for reference)

• Image header scan by year (six scraped profiles): ≥ 1440 px for the vast majority; ~75 images < 1000 px, all checked samples genuinely small at the source. official_artms 2023–24 has a cluster < 1440 px (~1080-class originals) — not probed.
• Video min-side classes (six profiles): 2023 ≤720: 304, 2024 ≤720: 248, 2025 ≤720: 368 (+1 at 1080), 2026 ≤720: 302 and ≤1080: 100 (+1 at 1440×2560). A handful ≤540.
• The scan tools (read-only, stdlib only, run on the NAS user over ssh stdin) were throwaway and are not in the repo; recreate from this description if needed.