docs: 2026-10-14 session-death incident, guards, resolution findings

Document the new exit codes (76/77), the wrapper/unit fix, recovery steps, the
verbose output, and the archive-resolution check (images match a logged-in
fetch; 1080p VP9 exists for newer videos; iOS plays VP9+Opus webm but not
VP9+AAC mp4). Correct the stale "timers are disabled" line, and flag that the
repo's weekly profiles timer was never installed on the host.
This commit is contained in:
2026-10-14 21:02:58 -04:00
parent 4a2bd7aa19
commit 832bbdb391
2 changed files with 106 additions and 5 deletions
+93 -5
View File
@@ -23,9 +23,10 @@
## Quick reference — getting new content onto the site
Timers are **disabled**, so this is manual today. Everything runs on
`mattellite`; it fetches, publishes to the NAS itself, and the site picks it up
with no deploy or restart.
The three systemd timers (see "Scheduling" below) run these unattended; run
them by hand only to get content sooner. Everything runs on `mattellite`; it
fetches, publishes to the NAS itself, and the site picks it up with no deploy
or restart.
```sh
ssh mattellite
@@ -67,6 +68,74 @@ Long runs: `setsid nohup ~/gdl/run-profiles.sh >/dev/null 2>&1 &` (or `run-full-
Nothing reaches the archive until a run finishes, so killing one midway is
safe.
## 2026-10-14 — the Instagram session died unnoticed; guards added
**What happened.** Between 2026-10-06 09:12 and 2026-10-07 04:36 the dedicated
Chrome's Instagram session died. Nothing alerted: the 10-07 full sweep logged
`HTTP redirect to login page` on 17 of 18 sources (and "failed"), but every
*stories* run after it fetched "No results" for all six profiles and still
exited 0 — about a week of stories, which cannot be backfilled, was lost.
Two compounding causes:
- **No scheduled run had ever pinged Healthchecks.** `gdl-sync@.service` ran
`gdl-cron.sh %i` directly (written 08-20), but `HC_URL` is exported by the
`run-*.sh` wrappers (added 09-14), which the unit bypassed. `gdl-cron.sh`
skips the ping silently when `HC_URL` is empty. Fixed: the unit now runs
`run-%i.sh`. (`run-full-sweep.sh` has no check URL, so that mode still
does not ping.)
- **An open Chrome tab keeps showing its logged-in UI after the session is
dead** — the SPA only notices on its next failed request. Looking at the
window proves nothing; **reload the tab, or read the cookie DB.** The cookie
jar had `csrftoken`, `ds_user_id`, `mid`… but no `sessionid`.
**Guards now in `gdl-cron.sh`** (tests: `scripts/test_session_guards.py`):
| exit | from | meaning |
|---|---|---|
| 76 | `check-session.py`, *before any request* | no unexpired `sessionid` in the Chrome profile gdl reads. The run aborts, logs why, pings `/76` (red check). Same DB, same profile as gdl, so it sees exactly what gdl would. Prints a `WARNING` (log only, no alert) when under 7 days of session remain |
| 77 | `stories-streak.py`, *after a stories run* | every fetched stories source returned "No results" on 3 consecutive days (`GDL_EMPTY_STREAK_FAIL` to change). Catches a session Instagram stopped honouring while the cookie is still present, which the preflight cannot see. One day per calendar date, so manual re-runs do not inflate it; state in `~/gdl/stories-empty-streak` |
| 75 | existing | every stories source skipped by `--min-interval` |
**To recover from exit 76** the user must log in to Instagram in the Chrome on
`mattellite` (VNC, display `:1`; neither password nor 2FA is something to
script). Then `~/gdl/check-session.py` should say `session ok ... expires
<date>`, and a manual `~/gdl/run-stories.sh` should find stories again.
**Verbose output.** A healthy run looked hung: pacing is 12–20 s, the config
sets `output: null` (no per-file lines), and the live-listing probe captured
output until it finished. `gdl-sync.py` now prints `[n/N] user / kind (time)`,
a `... fetching: 4m00s elapsed, 12 new file(s) so far (latest: …)` heartbeat
every 60 s, and `done in 6m12s, 14 new file(s)`. It only counts files already
written; no extra requests, pacing unchanged.
**Archive resolution check** (read-only scan of header dimensions for the six
scraped profiles, plus 8 single-post live metadata fetches, paced):
- *Logged-out gdl does not download smaller files — it fails* with a login
redirect, so no logged-out media ever entered the archive.
- Archived image sizes **match** what a logged-in fetch offers (1440×1800,
3277×4096, 2976×3720, and the genuinely small ones such as 400×400 or
742×928 sitting next to 1440 px siblings in one carousel). No image-upgrade
pass is needed.
- Videos: ~1,200 of ~1,300 archived videos are 720p. A 2026 reel offers a
1080×1920 VP9 stream (video-only DASH, plus separate AAC); a 2024 post tops
out at 1276×720, same as the archive. A video upgrade would mostly benefit
2025–26 posts (~700), at one request each.
- **iOS 18.7 Safari plays VP9 + Opus `.webm` but not VP9 + AAC `.mp4`** (error
code 4; `canPlayType` wrongly says "probably"). A video upgrade must
therefore output `.webm` — video stream copied, audio re-encoded to Opus
96k. **Not built yet.** Open points before building: the viewer mints every
video blob as `video/mp4` (`src/lib/archive-cache.ts`,
`src/hooks/useArchiveScanner.ts`) and must derive the MIME type from the
extension; superseded 720p files should be kept for review in a directory
*outside* the Resilio-synced, viewer-scanned `archives/` tree; and the
request budget (suggested: ≤40 posts/day, behind the preflight).
**Open: profiles cadence.** `scripts/systemd/gdl-sync@profiles.timer` says
weekly with a randomised day (commit 09-14) but the host is still on the
monthly (3rd, 04:00) unit installed 08-22. The repo version was never
installed; decide which one is wanted.
## 2026-09-14 — a tracked account renamed on Instagram: 0ct0ber19 → heejin
HeeJin's handle changed and the old one is now fully dead — `instagram.com/
@@ -173,6 +242,13 @@ above, so a fully-skipped stories run alerts, not silently passes.
Unset `HC_URL` (e.g. `HC_URL= ./run-stories.sh`) to run without pinging.
`run-full-sweep.sh` and the reels scripts have no check configured yet.
**The systemd unit must run the wrappers, not `gdl-cron.sh` directly** —
`HC_URL` lives in `run-<mode>.sh`, and `gdl-cron.sh` skips the ping silently
when it is empty. Until 2026-10-14 the unit bypassed the wrappers, so no
scheduled run had ever pinged. The preflight (exit 76) and the all-empty
streak (exit 77) are described in "2026-10-14" above; both reach the dashboard
as failures through the same trailing-status-code convention.
**This is also why `gdl-sync.py`'s `publish()` no longer treats the routine
chown-permission warning as a failure (2026-09-14).** The archive share is
never owned by the ssh user, so rsync's own `chown` step fails on *every*
@@ -230,6 +306,9 @@ every deletion because of it.
| `scripts/gdl-cron.sh` | unattended wrapper: `stories` \| `profiles` \| `full-sweep` |
| `scripts/systemd/` | the timers actually installed on the fetch host |
| `scripts/test_gdl_sync.py` | its tests |
| `scripts/check-session.py` | preflight: is there a usable Instagram `sessionid` in the Chrome cookie DB? (exit 1 = no) |
| `scripts/stories-streak.py` | postflight: all stories sources empty for several days running? |
| `scripts/test_session_guards.py` | tests for the two guards above (offline) |
| `scripts/jd2-sync.ts` | JDownloader `.crawljob` generator, still used elsewhere |
| `docs/gallery-dl.md` | the measurements behind every option in the fetcher — **read before changing pacing** |
| `docs/jdownloader.md` | the older JD2 flow |
@@ -372,7 +451,11 @@ systemctl --user list-timers 'gdl-sync@*'
Unit files are version-controlled in `scripts/systemd/` and installed to
`~/.config/systemd/user/`. One templated service, `gdl-sync@.service`, takes
the mode as its instance name and runs `gdl-cron.sh %i`.
the mode as its instance name and runs `run-%i.sh` (the fixed-mode wrappers
around `gdl-cron.sh` — they carry the Healthchecks URL; see "Healthchecks.io").
Before 2026-10-14 it ran `gdl-cron.sh %i` directly, which skipped every ping.
The profiles timer in the table is as originally installed; the repo's copy is
weekly (see "2026-10-14" → "Open: profiles cadence").
Three settings are load-bearing:
@@ -561,7 +644,12 @@ Two results are **not** failures, despite how they look:
### 2. Check the cookies before concluding you are blocked
Free, and it separates a local fault from a server-side one:
First, `~/gdl/check-session.py` — it reads the same cookie DB gdl does and says
whether a `sessionid` exists and when it expires (never prints values). Do not
trust the open browser tab to tell you: it keeps showing a logged-in page after
the session has died; reload it.
Then, free, and it separates a local fault from a server-side one:
```sh
PATH=$HOME/.local/bin:$PATH gallery-dl \
+13
View File
@@ -146,6 +146,19 @@ Anonymous access is **not** a viable fallback: it serves lower-resolution media,
caps profile pagination at 12 posts, and returns `AuthRequired` for stories and
highlights.
(Observed 2026-10-07, after the session died: with no `sessionid` gallery-dl did
not fall back to anonymous downloads at all — most sources failed with
`HTTP redirect to login page` / `HTTP redirect to home page`, and stories came
back as "No results". The lower-resolution claim above was **not** re-measured
then; what *was* measured is that archived image sizes match a logged-in fetch.)
**The session can die silently, and nothing about the browser window shows it** —
an open Instagram tab keeps rendering its logged-in UI until its next failed
request. The only reliable signal is the cookie DB: `scripts/check-session.py`
looks for an unexpired `sessionid` in this same profile, and `gdl-cron.sh` runs
it before every scheduled run (exit 76 = refuse to scrape). See `TOOLING.md`,
"2026-10-14".
## Output format
The viewer's parser is the contract, not JD2's exact bytes. `EXPORT_RE` in