docs: video upgrade pass — findings, decisions, open items (not built)

TOOLING.md

@@ -895,6 +895,12 @@ and three systemd timers now run the sync unattended.
 State as of 2026-08-20, after the sync run and the repo split. Nothing here is
 broken; these are decisions not yet made and cleanups not yet done.
 
+> **Update 2026-10-14:** the "Fetching" notes below predate the timers being enabled — all three
+> `gdl-sync@*` timers are installed and enabled on `mattellite` (see "Scheduling" and
+> "2026-10-14"). Newer open work: **the video-upgrade pass**
+> ([`docs/video-upgrade-plan.md`](docs/video-upgrade-plan.md) — measured and decided, not built),
+> and watching the fresh scraping account through its first weekly profiles run (~2026-10-23).
+
 ### Fetching
 
 - **The timers are DISABLED** after the 2026-08-21 scraping warning; see that

docs/video-upgrade-plan.md

@@ -0,0 +1,85 @@
+# Video upgrade pass — plan and findings (2026-10-14)
+
+**Status: designed and partly measured, NOT built. Nothing in the archive has been changed.**
+Written so this can be picked up cold. Account context: the scraping account is the
+fresh `erg05teur` (see `docs/gallery-dl.md`, top of "The account was suspended…") — every
+request in this plan is spent on a new account, so the budget rules there apply in full.
+
+## Why this exists
+
+The question was whether a logged-out fetch had given us lower-resolution media and whether a
+logged-in run could upgrade it. Measured result:
+
+- **Images: no upgrade needed.** Header-scan of 38.9k archived images, plus live metadata for 8
+  posts across 2023–2026 (including the oddly small ones): archived sizes match exactly what a
+  logged-in fetch offers (1440×1800, 3277×4096, 2976×3720; 400×400 / 742×928 items are small at
+  the source, sitting next to 1440 px siblings in the same carousel).
+- **Logged-out gdl does not download smaller files — it fails** (login redirect). No logged-out
+  media ever entered the archive.
+- **Videos: this is where there is something to gain.** Of ~1,300 archived videos in the six scraped
+  profiles, ~1,200 are 720p (min side ≤ 720) and ~100 (2026 reels) are 1080p. A 2026 reel
+  (`DTpycSlDKHs`, archived 720×1280) offers **1080×1920 VP9 video-only DASH + separate AAC audio**.
+  A 2024 post (`C7RmlCcSWNR`) tops out at 1276×720 — same as the archive — so older videos
+  probably gain nothing. Expected beneficiaries: **2025–26 videos, roughly 700 posts**. Not
+  verified how far back 1080p exists.
+
+## Decisions already made (by the owner)
+
+1. **Videos in a separate pass** from anything else (images need none).
+2. **Keep the replaced files for review** in a `_superseded/` area — never overwrite/delete.
+3. **Output `.webm`, not `.mp4`** (video stream copied, audio re-encoded AAC → Opus 96 kbps; the
+   audio stream is ~43 kbps HE-AAC so the loss is negligible). **Tested on real hardware:**
+   iPhone, iOS 18.7 Safari plays the VP9+Opus `.webm` at 1080×1920 and **fails the VP9+AAC `.mp4`**
+   (`MediaError` code 4, no video track) — even though `canPlayType` reports "probably" for it, so
+   never trust `canPlayType` here. VP9-in-MP4 (the lossless remux) is therefore out.
+   Re-encoding VP9 → H.264 was rejected (bigger, softer, no gain over the 720p original).
+
+## How the sample was produced (reproducible, one post)
+
+```sh
+# portrait video: 1080 is the WIDTH — `height=1080` selects nothing and fails "format not available"
+yt-dlp --cookies-from-browser chrome:/home/matt/.config/google-chrome-devtools \
+  -f "bv*[width>=1080]+ba" --merge-output-format mp4 -o 'X.%(ext)s' https://www.instagram.com/p/<shortcode>/
+ffmpeg -i X.mp4 -c:v copy -c:a libopus -b:a 96k X.webm        # ffmpeg 7.1.5 on mattellite
+```
+Sizes for that post: 6.46 MB `.mp4` / 6.53 MB `.webm`. For a browser test you need HTTP **Range**
+support (Python's `http.server` has none; Safari requires it) — the throwaway server used is not kept.
+`gallery-dl` needs `$HOME/.local/bin` on PATH over plain ssh.
+
+## Open decisions / work before building
+
+| item | notes |
+|---|---|
+| **Viewer MIME type** | `src/lib/archive-cache.ts:127` and `src/hooks/useArchiveScanner.ts:272,344,391` mint every video blob URL as `video/mp4`. Derive the type from the extension (`.webm` → `video/webm`) or Safari/iOS may refuse a WebM labelled mp4. `MEDIA_RE` / `isVideo` already recognise `.webm`. |
+| **Filename / dedup** | The archive names `YYYY-MM-DD_<user> - <shortcode>[ - <n>].mp4`. Swapping to `.webm` must not leave **both** in `archives/` (the viewer would show duplicates) — the old `.mp4` has to be *moved out*, not just joined by a `.webm`. Check `parseArchiveFilename` (`src/lib/archive-patterns.ts`) for the extension handling and the JD2-vs-gdl same-stem duplicate logic. |
+| **Skip-archive** | `artms.db` keys on `instagram<media_id>`, so a normal run will never re-fetch these. The upgrade tool must fetch targeted items deliberately and must not touch/poison `artms.db` semantics. |
+| **`_superseded/` location** | Must be **outside** `/volume1/rslsync/sync/Instagram-archive/archives/` — that tree is scanned by the viewer **and** inside the Resilio-synced share (it would sync every replaced file to devices). Not decided. Also unchecked: whether `agentapi` can write outside that share. Use rsync `--backup --backup-dir=<dir>/<date>/` so the old copy is kept. |
+| **Carousel videos** | yt-dlp on a carousel post URL fails ("No video formats found") — see `DIa5NpWRnIP`. gdl's own metadata gives per-child `ytdl:` URLs for videos in carousels; the tool needs to handle child items, not post URLs. |
+| **Request budget** | One `yt-dlp -F`-style request per post to learn what 1080p exists (plus the download). ~700 candidates ⇒ suggested **≤ 40 posts/day**, resumable, behind `check-session.py`, hand-started or a slow timer — never a burst. Candidates can be pre-ranked from the archive alone (720p, 2025–26) at zero request cost. |
+| **Storage** | 1080p VP9 ≈ 1.6× the 720p file (6 MB vs ~4 MB for that sample, plus the kept original). Estimate before running at scale. |
+
+## Planned design
+
+1. **Candidate list, zero requests:** scan the NAS archive (read-only, ssh `agentapi@<nas>` from
+   mattellite; the scans in this doc used header parsing in plain Python over stdin because the
+   NAS has no ffprobe) for videos ≤ 720 min-side from 2025-01 onward, newest first.
+2. **Dry-run mode (default):** for each candidate, one request to learn the best offered stream,
+   record `{shortcode, archived WxH, offered WxH}` in a state file; print counts + estimated GB.
+   Downloads nothing.
+3. **Execute mode:** only where offered width > archived width; download video+audio, `ffmpeg
+   -c:v copy -c:a libopus -b:a 96k`, verify with `ffprobe` (dimensions, duration within 0.5 s of
+   the original, both streams present), publish the `.webm`, move the old file to `_superseded/`.
+   Cap per run (`--max-posts`), pace with the same 12–20 s gaps, stop on any 429 / login redirect.
+4. **First step when resuming:** do ONE reel end to end by hand (`DTpycSlDKHs`, ~3 requests) to prove
+   download → remux → verify → publish → supersede → viewer playback, *then* the dry-run, and only after
+   the weekly profiles timer has run cleanly on the new account (first fire ≈ 2026-10-23).
+
+## Things measured along the way (for reference)
+
+- Image header scan by year (six scraped profiles): ≥ 1440 px for the vast majority; ~75 images
+  < 1000 px, all checked samples genuinely small at the source. `official_artms` 2023–24 has a
+  cluster < 1440 px (~1080-class originals) — not probed.
+- Video min-side classes (six profiles): 2023 ≤720: 304, 2024 ≤720: 248, 2025 ≤720: 368 (+1 at
+  1080), 2026 ≤720: 302 and ≤1080: 100 (+1 at 1440×2560). A handful ≤540.
+- The scan tools (read-only, stdlib only, run on the NAS user over ssh stdin) were throwaway
+  and are not in the repo; recreate from this description if needed.