diff --git a/TOOLING.md b/TOOLING.md index 9b14cf0..e94b1c8 100644 --- a/TOOLING.md +++ b/TOOLING.md @@ -895,6 +895,12 @@ and three systemd timers now run the sync unattended. State as of 2026-08-20, after the sync run and the repo split. Nothing here is broken; these are decisions not yet made and cleanups not yet done. +> **Update 2026-10-14:** the "Fetching" notes below predate the timers being enabled — all three +> `gdl-sync@*` timers are installed and enabled on `mattellite` (see "Scheduling" and +> "2026-10-14"). Newer open work: **the video-upgrade pass** +> ([`docs/video-upgrade-plan.md`](docs/video-upgrade-plan.md) — measured and decided, not built), +> and watching the fresh scraping account through its first weekly profiles run (~2026-10-23). + ### Fetching - **The timers are DISABLED** after the 2026-08-21 scraping warning; see that diff --git a/docs/video-upgrade-plan.md b/docs/video-upgrade-plan.md new file mode 100644 index 0000000..ba73b69 --- /dev/null +++ b/docs/video-upgrade-plan.md @@ -0,0 +1,85 @@ +# Video upgrade pass — plan and findings (2026-10-14) + +**Status: designed and partly measured, NOT built. Nothing in the archive has been changed.** +Written so this can be picked up cold. Account context: the scraping account is the +fresh `erg05teur` (see `docs/gallery-dl.md`, top of "The account was suspended…") — every +request in this plan is spent on a new account, so the budget rules there apply in full. + +## Why this exists + +The question was whether a logged-out fetch had given us lower-resolution media and whether a +logged-in run could upgrade it. Measured result: + +- **Images: no upgrade needed.** Header-scan of 38.9k archived images, plus live metadata for 8 + posts across 2023–2026 (including the oddly small ones): archived sizes match exactly what a + logged-in fetch offers (1440×1800, 3277×4096, 2976×3720; 400×400 / 742×928 items are small at + the source, sitting next to 1440 px siblings in the same carousel). +- **Logged-out gdl does not download smaller files — it fails** (login redirect). No logged-out + media ever entered the archive. +- **Videos: this is where there is something to gain.** Of ~1,300 archived videos in the six scraped + profiles, ~1,200 are 720p (min side ≤ 720) and ~100 (2026 reels) are 1080p. A 2026 reel + (`DTpycSlDKHs`, archived 720×1280) offers **1080×1920 VP9 video-only DASH + separate AAC audio**. + A 2024 post (`C7RmlCcSWNR`) tops out at 1276×720 — same as the archive — so older videos + probably gain nothing. Expected beneficiaries: **2025–26 videos, roughly 700 posts**. Not + verified how far back 1080p exists. + +## Decisions already made (by the owner) + +1. **Videos in a separate pass** from anything else (images need none). +2. **Keep the replaced files for review** in a `_superseded/` area — never overwrite/delete. +3. **Output `.webm`, not `.mp4`** (video stream copied, audio re-encoded AAC → Opus 96 kbps; the + audio stream is ~43 kbps HE-AAC so the loss is negligible). **Tested on real hardware:** + iPhone, iOS 18.7 Safari plays the VP9+Opus `.webm` at 1080×1920 and **fails the VP9+AAC `.mp4`** + (`MediaError` code 4, no video track) — even though `canPlayType` reports "probably" for it, so + never trust `canPlayType` here. VP9-in-MP4 (the lossless remux) is therefore out. + Re-encoding VP9 → H.264 was rejected (bigger, softer, no gain over the 720p original). + +## How the sample was produced (reproducible, one post) + +```sh +# portrait video: 1080 is the WIDTH — `height=1080` selects nothing and fails "format not available" +yt-dlp --cookies-from-browser chrome:/home/matt/.config/google-chrome-devtools \ + -f "bv*[width>=1080]+ba" --merge-output-format mp4 -o 'X.%(ext)s' https://www.instagram.com/p// +ffmpeg -i X.mp4 -c:v copy -c:a libopus -b:a 96k X.webm # ffmpeg 7.1.5 on mattellite +``` +Sizes for that post: 6.46 MB `.mp4` / 6.53 MB `.webm`. For a browser test you need HTTP **Range** +support (Python's `http.server` has none; Safari requires it) — the throwaway server used is not kept. +`gallery-dl` needs `$HOME/.local/bin` on PATH over plain ssh. + +## Open decisions / work before building + +| item | notes | +|---|---| +| **Viewer MIME type** | `src/lib/archive-cache.ts:127` and `src/hooks/useArchiveScanner.ts:272,344,391` mint every video blob URL as `video/mp4`. Derive the type from the extension (`.webm` → `video/webm`) or Safari/iOS may refuse a WebM labelled mp4. `MEDIA_RE` / `isVideo` already recognise `.webm`. | +| **Filename / dedup** | The archive names `YYYY-MM-DD_ - [ - ].mp4`. Swapping to `.webm` must not leave **both** in `archives/` (the viewer would show duplicates) — the old `.mp4` has to be *moved out*, not just joined by a `.webm`. Check `parseArchiveFilename` (`src/lib/archive-patterns.ts`) for the extension handling and the JD2-vs-gdl same-stem duplicate logic. | +| **Skip-archive** | `artms.db` keys on `instagram`, so a normal run will never re-fetch these. The upgrade tool must fetch targeted items deliberately and must not touch/poison `artms.db` semantics. | +| **`_superseded/` location** | Must be **outside** `/volume1/rslsync/sync/Instagram-archive/archives/` — that tree is scanned by the viewer **and** inside the Resilio-synced share (it would sync every replaced file to devices). Not decided. Also unchecked: whether `agentapi` can write outside that share. Use rsync `--backup --backup-dir=//` so the old copy is kept. | +| **Carousel videos** | yt-dlp on a carousel post URL fails ("No video formats found") — see `DIa5NpWRnIP`. gdl's own metadata gives per-child `ytdl:` URLs for videos in carousels; the tool needs to handle child items, not post URLs. | +| **Request budget** | One `yt-dlp -F`-style request per post to learn what 1080p exists (plus the download). ~700 candidates ⇒ suggested **≤ 40 posts/day**, resumable, behind `check-session.py`, hand-started or a slow timer — never a burst. Candidates can be pre-ranked from the archive alone (720p, 2025–26) at zero request cost. | +| **Storage** | 1080p VP9 ≈ 1.6× the 720p file (6 MB vs ~4 MB for that sample, plus the kept original). Estimate before running at scale. | + +## Planned design + +1. **Candidate list, zero requests:** scan the NAS archive (read-only, ssh `agentapi@` from + mattellite; the scans in this doc used header parsing in plain Python over stdin because the + NAS has no ffprobe) for videos ≤ 720 min-side from 2025-01 onward, newest first. +2. **Dry-run mode (default):** for each candidate, one request to learn the best offered stream, + record `{shortcode, archived WxH, offered WxH}` in a state file; print counts + estimated GB. + Downloads nothing. +3. **Execute mode:** only where offered width > archived width; download video+audio, `ffmpeg + -c:v copy -c:a libopus -b:a 96k`, verify with `ffprobe` (dimensions, duration within 0.5 s of + the original, both streams present), publish the `.webm`, move the old file to `_superseded/`. + Cap per run (`--max-posts`), pace with the same 12–20 s gaps, stop on any 429 / login redirect. +4. **First step when resuming:** do ONE reel end to end by hand (`DTpycSlDKHs`, ~3 requests) to prove + download → remux → verify → publish → supersede → viewer playback, *then* the dry-run, and only after + the weekly profiles timer has run cleanly on the new account (first fire ≈ 2026-10-23). + +## Things measured along the way (for reference) + +- Image header scan by year (six scraped profiles): ≥ 1440 px for the vast majority; ~75 images + < 1000 px, all checked samples genuinely small at the source. `official_artms` 2023–24 has a + cluster < 1440 px (~1080-class originals) — not probed. +- Video min-side classes (six profiles): 2023 ≤720: 304, 2024 ≤720: 248, 2025 ≤720: 368 (+1 at + 1080), 2026 ≤720: 302 and ≤1080: 100 (+1 at 1440×2560). A handful ≤540. +- The scan tools (read-only, stdlib only, run on the NAS user over ssh stdin) were throwaway + and are not in the repo; recreate from this description if needed.