86 lines
6.9 KiB
Markdown
86 lines
6.9 KiB
Markdown
# Video upgrade pass — plan and findings (2026-10-14)
|
||
|
||
**Status: designed and partly measured, NOT built. Nothing in the archive has been changed.**
|
||
Written so this can be picked up cold. Account context: the scraping account is the
|
||
fresh `erg05teur` (see `docs/gallery-dl.md`, top of "The account was suspended…") — every
|
||
request in this plan is spent on a new account, so the budget rules there apply in full.
|
||
|
||
## Why this exists
|
||
|
||
The question was whether a logged-out fetch had given us lower-resolution media and whether a
|
||
logged-in run could upgrade it. Measured result:
|
||
|
||
- **Images: no upgrade needed.** Header-scan of 38.9k archived images, plus live metadata for 8
|
||
posts across 2023–2026 (including the oddly small ones): archived sizes match exactly what a
|
||
logged-in fetch offers (1440×1800, 3277×4096, 2976×3720; 400×400 / 742×928 items are small at
|
||
the source, sitting next to 1440 px siblings in the same carousel).
|
||
- **Logged-out gdl does not download smaller files — it fails** (login redirect). No logged-out
|
||
media ever entered the archive.
|
||
- **Videos: this is where there is something to gain.** Of ~1,300 archived videos in the six scraped
|
||
profiles, ~1,200 are 720p (min side ≤ 720) and ~100 (2026 reels) are 1080p. A 2026 reel
|
||
(`DTpycSlDKHs`, archived 720×1280) offers **1080×1920 VP9 video-only DASH + separate AAC audio**.
|
||
A 2024 post (`C7RmlCcSWNR`) tops out at 1276×720 — same as the archive — so older videos
|
||
probably gain nothing. Expected beneficiaries: **2025–26 videos, roughly 700 posts**. Not
|
||
verified how far back 1080p exists.
|
||
|
||
## Decisions already made (by the owner)
|
||
|
||
1. **Videos in a separate pass** from anything else (images need none).
|
||
2. **Keep the replaced files for review** in a `_superseded/` area — never overwrite/delete.
|
||
3. **Output `.webm`, not `.mp4`** (video stream copied, audio re-encoded AAC → Opus 96 kbps; the
|
||
audio stream is ~43 kbps HE-AAC so the loss is negligible). **Tested on real hardware:**
|
||
iPhone, iOS 18.7 Safari plays the VP9+Opus `.webm` at 1080×1920 and **fails the VP9+AAC `.mp4`**
|
||
(`MediaError` code 4, no video track) — even though `canPlayType` reports "probably" for it, so
|
||
never trust `canPlayType` here. VP9-in-MP4 (the lossless remux) is therefore out.
|
||
Re-encoding VP9 → H.264 was rejected (bigger, softer, no gain over the 720p original).
|
||
|
||
## How the sample was produced (reproducible, one post)
|
||
|
||
```sh
|
||
# portrait video: 1080 is the WIDTH — `height=1080` selects nothing and fails "format not available"
|
||
yt-dlp --cookies-from-browser chrome:/home/matt/.config/google-chrome-devtools \
|
||
-f "bv*[width>=1080]+ba" --merge-output-format mp4 -o 'X.%(ext)s' https://www.instagram.com/p/<shortcode>/
|
||
ffmpeg -i X.mp4 -c:v copy -c:a libopus -b:a 96k X.webm # ffmpeg 7.1.5 on mattellite
|
||
```
|
||
Sizes for that post: 6.46 MB `.mp4` / 6.53 MB `.webm`. For a browser test you need HTTP **Range**
|
||
support (Python's `http.server` has none; Safari requires it) — the throwaway server used is not kept.
|
||
`gallery-dl` needs `$HOME/.local/bin` on PATH over plain ssh.
|
||
|
||
## Open decisions / work before building
|
||
|
||
| item | notes |
|
||
|---|---|
|
||
| **Viewer MIME type** | `src/lib/archive-cache.ts:127` and `src/hooks/useArchiveScanner.ts:272,344,391` mint every video blob URL as `video/mp4`. Derive the type from the extension (`.webm` → `video/webm`) or Safari/iOS may refuse a WebM labelled mp4. `MEDIA_RE` / `isVideo` already recognise `.webm`. |
|
||
| **Filename / dedup** | The archive names `YYYY-MM-DD_<user> - <shortcode>[ - <n>].mp4`. Swapping to `.webm` must not leave **both** in `archives/` (the viewer would show duplicates) — the old `.mp4` has to be *moved out*, not just joined by a `.webm`. Check `parseArchiveFilename` (`src/lib/archive-patterns.ts`) for the extension handling and the JD2-vs-gdl same-stem duplicate logic. |
|
||
| **Skip-archive** | `artms.db` keys on `instagram<media_id>`, so a normal run will never re-fetch these. The upgrade tool must fetch targeted items deliberately and must not touch/poison `artms.db` semantics. |
|
||
| **`_superseded/` location** | Must be **outside** `/volume1/rslsync/sync/Instagram-archive/archives/` — that tree is scanned by the viewer **and** inside the Resilio-synced share (it would sync every replaced file to devices). Not decided. Also unchecked: whether `agentapi` can write outside that share. Use rsync `--backup --backup-dir=<dir>/<date>/` so the old copy is kept. |
|
||
| **Carousel videos** | yt-dlp on a carousel post URL fails ("No video formats found") — see `DIa5NpWRnIP`. gdl's own metadata gives per-child `ytdl:` URLs for videos in carousels; the tool needs to handle child items, not post URLs. |
|
||
| **Request budget** | One `yt-dlp -F`-style request per post to learn what 1080p exists (plus the download). ~700 candidates ⇒ suggested **≤ 40 posts/day**, resumable, behind `check-session.py`, hand-started or a slow timer — never a burst. Candidates can be pre-ranked from the archive alone (720p, 2025–26) at zero request cost. |
|
||
| **Storage** | 1080p VP9 ≈ 1.6× the 720p file (6 MB vs ~4 MB for that sample, plus the kept original). Estimate before running at scale. |
|
||
|
||
## Planned design
|
||
|
||
1. **Candidate list, zero requests:** scan the NAS archive (read-only, ssh `agentapi@<nas>` from
|
||
mattellite; the scans in this doc used header parsing in plain Python over stdin because the
|
||
NAS has no ffprobe) for videos ≤ 720 min-side from 2025-01 onward, newest first.
|
||
2. **Dry-run mode (default):** for each candidate, one request to learn the best offered stream,
|
||
record `{shortcode, archived WxH, offered WxH}` in a state file; print counts + estimated GB.
|
||
Downloads nothing.
|
||
3. **Execute mode:** only where offered width > archived width; download video+audio, `ffmpeg
|
||
-c:v copy -c:a libopus -b:a 96k`, verify with `ffprobe` (dimensions, duration within 0.5 s of
|
||
the original, both streams present), publish the `.webm`, move the old file to `_superseded/`.
|
||
Cap per run (`--max-posts`), pace with the same 12–20 s gaps, stop on any 429 / login redirect.
|
||
4. **First step when resuming:** do ONE reel end to end by hand (`DTpycSlDKHs`, ~3 requests) to prove
|
||
download → remux → verify → publish → supersede → viewer playback, *then* the dry-run, and only after
|
||
the weekly profiles timer has run cleanly on the new account (first fire ≈ 2026-10-23).
|
||
|
||
## Things measured along the way (for reference)
|
||
|
||
- Image header scan by year (six scraped profiles): ≥ 1440 px for the vast majority; ~75 images
|
||
< 1000 px, all checked samples genuinely small at the source. `official_artms` 2023–24 has a
|
||
cluster < 1440 px (~1080-class originals) — not probed.
|
||
- Video min-side classes (six profiles): 2023 ≤720: 304, 2024 ≤720: 248, 2025 ≤720: 368 (+1 at
|
||
1080), 2026 ≤720: 302 and ≤1080: 100 (+1 at 1440×2560). A handful ≤540.
|
||
- The scan tools (read-only, stdlib only, run on the NAS user over ssh stdin) were throwaway
|
||
and are not in the repo; recreate from this description if needed.
|