From ff0d0f9f787cbc3716d65ff0a0fd9006ddcec7ac Mon Sep 17 00:00:00 2001 From: ergosteur Date: Thu, 20 Aug 2026 16:48:17 -0400 Subject: [PATCH] docs: record what is still outstanding, and unstale the scanner section MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Nothing here is broken — these are decisions not made and cleanups not done, written down before the session's context is lost. The two that can actually cost something: no daily stories run is scheduled, and stories are the one surface that cannot be backfilled; and the `seeded` flags in the state file were reconstructed by hand from a log rather than derived from the archive DB, so losing artms.db without also clearing the state file would leave those sources permanently unseeded and re-download everything. Also corrects "Scanner work (not done yet)", which shipped in 53b1f80 — the three .json shapes are told apart structurally in gallery-dl-sidecar.ts and isReel comes from the sidecar's type. That file lives on main: it parses archives at display time and is viewer code, not tooling. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01UXfdJu7QhSJLr47K7koTDF --- TOOLING.md | 53 ++++++++++++++++++++++++++++++++++++++++++++++ docs/gallery-dl.md | 23 ++++++++++++-------- 2 files changed, 67 insertions(+), 9 deletions(-) diff --git a/TOOLING.md b/TOOLING.md index a1a1dd0..530669d 100644 --- a/TOOLING.md +++ b/TOOLING.md @@ -92,3 +92,56 @@ also catches edited carousels. attempt put tooling notes in `CLAUDE.md` and a `jd2` script in `package.json`; because `main` had *deleted* those lines, every merge re-applied the deletion. Keep branch-specific documentation in this file, which `main` does not have. + +## Outstanding + +State as of 2026-08-20, after the sync run and the repo split. Nothing here is +broken; these are decisions not yet made and cleanups not yet done. + +### Fetching + +- **5.3 GB of stale staging on `mattellite`** — `~/gdl/staging` and `~/gdl/out` + (2.2 GB each, from the 2026-08-17 run) and `~/gdl/staging-0820` / + `~/gdl/out-0820` (446 MB each, from 2026-08-20). Every file in all four was + verified present in the live archive, so they are safe to delete. 46 GB free, + so there is no urgency — but nothing will clean them up on its own. +- **No daily stories run is scheduled.** Stories expire in 24h and cannot be + backfilled, so this is the *only* surface where waiting loses content + permanently. `--only stories` never seeds and costs roughly six requests for + all six profiles. When scheduling it, randomise the minute and avoid the hour + boundary: a job firing at exactly 09:00 daily is obviously a machine. +- **`--abort 50` is opt-in and nothing uses it yet.** It is the right setting + for routine runs — it cut a 2151-post profile to 7 enumerated posts — but it + stops noticing **edited carousels** (test case 15), which only a full + enumeration finds. A full-sweep cadence has not been decided; quarterly was + suggested and never agreed. +- **The `seeded` flags in `.state.json` were hand-written**, reconstructed + from the 2026-08-17 log rather than derived from the archive DB. They assert + "the skip-archive already knows this source". If `artms.db` is ever rebuilt, + moved or lost, **clear the state file too** — otherwise those sources will + never re-seed and a fetch into empty staging re-downloads everything. +- **`~/gdl/gdl-sync.py` on the fetch host is a copy, not a checkout.** It + currently matches this branch (`96e5694e…`), but nothing keeps them in sync; + `scp` it after any change and re-check the hash. +- The 2026-08-20 run is split across two logs — `artms-run3.log` (12 sources, + no abort) and `artms-run4.log` (12 sources, `--abort 50`) — because it was + stopped midway to pick up the new flag. + +### Repo and infrastructure + +- **The `pre-rewrite-*` branches on gitea hold the unredacted history** — real + account names, the fetch host's IP, and the tooling, as it was before the + rewrite. They are deliberate backups. Decide whether they expire; the + `pre-push` hook does cover them (it allows only `main` and tags to GitHub). +- **The `pre-rewrite-full.bundle` backup is in a session scratchpad** and will + be deleted with it. If a durable backup outside gitea is wanted, move it now. +- **Only one container image exists.** 22 versions were pruned, so rolling back + to an older release means checking out its tag from gitea and pushing that + tag to GitHub to rebuild it — the old images are gone, not archived. +- **CI warns that the Node 20 actions are deprecated.** `actions/checkout@v4`, + `docker/login-action@v3`, `docker/metadata-action@v5` and + `docker/build-push-action@v5` are being forced onto Node 24. They work today; + bump when convenient. +- **`review-fixes`** on gitea is a stale v1.3.0-era branch, never merged, + published only because the whole local repo was pushed. Probably deletable. +- GitHub Actions run history was lost when the repo was recreated. Cosmetic. diff --git a/docs/gallery-dl.md b/docs/gallery-dl.md index d496d2e..65b3215 100644 --- a/docs/gallery-dl.md +++ b/docs/gallery-dl.md @@ -560,18 +560,23 @@ Verify by asking gallery-dl's own interpreter, not the shell: has a dated twin. The scanner dedupes by index so they render once; it is wasted disk, not a display bug. -## Scanner work (not done yet) +## Scanner work — done -`useArchiveScanner` currently treats any `.json` in the tree as a possible -manifest. Adding gallery-dl sidecars needs it to distinguish three things: +Shipped in `53b1f80` ("read gallery-dl sidecars for reel type and post dates"). +`useArchiveScanner` tells the three `.json` shapes apart **structurally**, not +by filename, in `src/lib/gallery-dl-sidecar.ts`: -1. Instagram export manifests (`posts_1.json`) — existing path. -2. Instaloader `.json.xz` — existing path, GraphQL node shape. -3. gallery-dl `.json` — new, flat shape, identified by having - `post_shortcode` + `type` at the top level. +1. Instagram export manifests (`posts_1.json`) — top-level `media` array. +2. Instaloader `.json.xz` — GraphQL node under `node` / `__typename`. +3. gallery-dl `.json` — flat, `post_shortcode` + `type`, none of the above. -Once (3) is read, `source`/`isStory` and the reel flag should come from `type` -rather than from the directory and the lone-video heuristic. +`post.isReel` now comes from the sidecar's `type`, which is Instagram's own +classification, and beats every fallback in `post-tabs.ts`. Post dates are +ranked rather than last-write-wins (`src/lib/post-dates.ts`): sidecar beats +filename beats mtime. + +Note those files live on **`main`** — they parse the archive at display time +and are viewer code, not fetching tooling. ## Test cases