Compare commits
8
Commits
| Author | SHA1 | Date | |
|---|---|---|---|
 ergosteur
|
fe6ebdd75b | ||
|
ergosteur
|
1b67765d8b | ||
|
ergosteur
|
30c9fbe611 | ||
|
ergosteur
|
556eab1169 | ||
|
ergosteur
|
99719d164b | ||
|
ergosteur
|
5c3307685a | ||
|
ergosteur
|
832bbdb391 | ||
|
ergosteur
|
4a2bd7aa19 |
+109
@@ -0,0 +1,109 @@
|
||||
# Manual run — scrape, archive, push to the NAS
|
||||
|
||||
**`tooling` branch only. Never push this to GitHub.** Like `TOOLING.md`, it names the fetch host,
|
||||
the NAS hosts and their paths.
|
||||
|
||||
How to do by hand everything the timers normally do, plus the second NAS hop that feeds
|
||||
`cosmo_web`. `TOOLING.md` has the details behind each step.
|
||||
|
||||
## Where things go
|
||||
|
||||
1. **mattellite → ana-nas.** Every scrape script ends by rsyncing its new files to
|
||||
`ana-nas:/volume1/rslsync/sync/Instagram-archive/archives/` (10.20.28.200).
|
||||
<https://instaarchive.ergosteur.com> reads straight from there. **There is no separate step
|
||||
for this hop.**
|
||||
2. **ana-nas → DH2300.** The `archiver-instagram` container on `DH2300-D7CA` pulls that archive,
|
||||
makes thumbnails, remuxes video, normalizes into `cosmo_web`'s data and translates captions.
|
||||
It runs weekly (Wednesday 04:00). Step 6 below runs it by hand.
|
||||
|
||||
## 1. Pre-flight (mattellite)
|
||||
|
||||
```sh
|
||||
ssh matt@mattellite
|
||||
~/gdl/check-session.py # want: "session ok ... expires <date>"
|
||||
curl -s http://localhost:9222/json/version | head -2 # Chrome DevTools port is up
|
||||
~/gdl/gdl-upgrade.sh # gallery-dl + yt-dlp to latest
|
||||
```
|
||||
|
||||
If the session check fails, log in to Instagram in the VNC Chrome (display `:1`), **reload the
|
||||
tab** (a dead session can still look logged in) and re-check.
|
||||
|
||||
## 2. Stories — first; they expire in 24 h
|
||||
|
||||
```sh
|
||||
~/gdl/run-stories.sh
|
||||
```
|
||||
|
||||
About 2 minutes. `rsync ... (code 23)` / `failed to set permissions` is expected: the script
|
||||
says "files landed" when that is the only problem. `No results` for a profile just means it has
|
||||
no active story.
|
||||
|
||||
## 3. Posts and reels
|
||||
|
||||
**While gallery-dl's listing is broken** ([gh#9829](https://github.com/mikf/gallery-dl/issues/9829),
|
||||
from 2026-10-22), use the browser scraper, one profile at a time:
|
||||
|
||||
```sh
|
||||
for p in heejin kimxxlip; do ~/gdl/posts-sync.sh $p; done
|
||||
for p in withaseul cher_ryppo zindoriyam; do ~/gdl/posts-sync.sh $p; done
|
||||
~/gdl/posts-sync.sh official_artms
|
||||
```
|
||||
|
||||
- `posts-sync.sh` scrolls `/<user>/` in the signed-in Chrome, stops once it reaches posts the
|
||||
archive already holds, and fetches only the new ones as single posts. Each profile takes a few
|
||||
minutes, mostly the 12–20 s gap between fetches.
|
||||
- It prints `no new posts for <user>` when a profile is caught up.
|
||||
- Logs: `~/gdl/logs/posts-<user>-*.log`.
|
||||
- **Then sweep the Reels tab too:** `~/gdl/run-reels-all.sh` (~15 min). Most reels never
|
||||
appear in the main grid. On 2026-10-23 only 3 of heejin's 114 reels did, so `posts-sync.sh`
|
||||
alone misses them. `reels-sync.sh <user>` does a single profile.
|
||||
|
||||
**Or the whole thing in one command**, the same run the weekly timer does: gallery-dl for
|
||||
stories and highlights, then the browser for every profile's posts grid and Reels tab, each
|
||||
stopping once it reaches archived content:
|
||||
|
||||
```sh
|
||||
~/gdl/run-profiles.sh
|
||||
```
|
||||
|
||||
Once a gallery-dl release fixes the listing, `GDL_LISTING=api ~/gdl/run-profiles.sh` does posts
|
||||
and reels through gallery-dl again (~5 min).
|
||||
|
||||
For long runs, detach:
|
||||
`setsid nohup ~/gdl/run-profiles.sh >/dev/null 2>&1 &`, then
|
||||
`tail -f ~/gdl/logs/$(ls -1t ~/gdl/logs | head -1)`.
|
||||
|
||||
**Stop at once** on a `429` from `scontent-*.cdninstagram.com` or a `400 Bad Request` on
|
||||
`/api/v1/feed/reels_media/`. See "When a run fails" in `TOOLING.md`.
|
||||
|
||||
## 4. Highlights
|
||||
|
||||
These only come through `run-profiles.sh`. Expect them to fail with the same
|
||||
`Requested user could not be found` until gh#9829 is fixed in a release, so skip them until then.
|
||||
|
||||
## 5. Check it landed (any machine)
|
||||
|
||||
```sh
|
||||
for u in heejin kimxxlip withaseul cher_ryppo zindoriyam official_artms; do
|
||||
n=$(curl -s "https://instaarchive.ergosteur.com/api/archives/$u/files" | python3 -c 'import json,sys; d=json.load(sys.stdin); print(len(d if isinstance(d,list) else d["files"]))')
|
||||
printf '%-16s %s\n' "$u" "$n"
|
||||
done
|
||||
```
|
||||
|
||||
Or browse <https://instaarchive.ergosteur.com>. New files appear without a redeploy.
|
||||
|
||||
## 6. Push to `cosmo_web` (the DH2300 hop)
|
||||
|
||||
This runs the same command the container's cron line runs (from `docker/archiver-instagram-entrypoint.sh`
|
||||
in Cosmo-Live-Downloads):
|
||||
|
||||
```sh
|
||||
ssh matt@DH2300-D7CA
|
||||
docker exec cosmo-web-archiver-instagram-1 sh -c \
|
||||
'cd /work && . /app/instagram-archiver.env && python3 scripts/instagram_archive_run.py'
|
||||
docker logs --tail 50 cosmo-web-archiver-instagram-1
|
||||
```
|
||||
|
||||
It pings that container's Healthchecks check, same as a scheduled run. The pipeline itself
|
||||
(mirror, thumbnail, remux, normalize, collaborators, translate) is described in the docstring of
|
||||
`scripts/instagram_archive_run.py` in Cosmo-Live-Downloads.
|
||||
+197
-16
@@ -23,9 +23,13 @@
|
||||
|
||||
## Quick reference — getting new content onto the site
|
||||
|
||||
Timers are **disabled**, so this is manual today. Everything runs on
|
||||
`mattellite`; it fetches, publishes to the NAS itself, and the site picks it up
|
||||
with no deploy or restart.
|
||||
For the whole thing by hand, step by step (including the DH2300 / `cosmo_web` hop), see
|
||||
[MANUAL-RUN.md](MANUAL-RUN.md).
|
||||
|
||||
The three systemd timers (see "Scheduling" below) run these unattended; run
|
||||
them by hand only to get content sooner. Everything runs on `mattellite`; it
|
||||
fetches, publishes to the NAS itself, and the site picks it up with no deploy
|
||||
or restart.
|
||||
|
||||
```sh
|
||||
ssh mattellite
|
||||
@@ -67,6 +71,86 @@ Long runs: `setsid nohup ~/gdl/run-profiles.sh >/dev/null 2>&1 &` (or `run-full-
|
||||
Nothing reaches the archive until a run finishes, so killing one midway is
|
||||
safe.
|
||||
|
||||
## 2026-10-14 — the Instagram session died unnoticed; guards added
|
||||
|
||||
**What happened.** Between 2026-10-06 09:12 and 2026-10-07 04:36 the dedicated
|
||||
Chrome's Instagram session died. Nothing alerted: the 10-07 full sweep logged
|
||||
`HTTP redirect to login page` on 17 of 18 sources (and "failed"), but every
|
||||
*stories* run after it fetched "No results" for all six profiles and still
|
||||
exited 0 — about a week of stories, which cannot be backfilled, was lost.
|
||||
Two compounding causes:
|
||||
|
||||
- **No scheduled run had ever pinged Healthchecks.** `gdl-sync@.service` ran
|
||||
`gdl-cron.sh %i` directly (written 08-20), but `HC_URL` is exported by the
|
||||
`run-*.sh` wrappers (added 09-14), which the unit bypassed. `gdl-cron.sh`
|
||||
skips the ping silently when `HC_URL` is empty. Fixed: the unit now runs
|
||||
`run-%i.sh`. (`run-full-sweep.sh` has no check URL, so that mode still
|
||||
does not ping.)
|
||||
- **An open Chrome tab keeps showing its logged-in UI after the session is
|
||||
dead** — the SPA only notices on its next failed request. Looking at the
|
||||
window proves nothing; **reload the tab, or read the cookie DB.** The cookie
|
||||
jar had `csrftoken`, `ds_user_id`, `mid`… but no `sessionid`.
|
||||
|
||||
**Guards now in `gdl-cron.sh`** (tests: `scripts/test_session_guards.py`):
|
||||
|
||||
| exit | from | meaning |
|
||||
|---|---|---|
|
||||
| 76 | `check-session.py`, *before any request* | no unexpired `sessionid` in the Chrome profile gdl reads. The run aborts, logs why, pings `/76` (red check). Same DB, same profile as gdl, so it sees exactly what gdl would. Prints a `WARNING` (log only, no alert) when under 7 days of session remain |
|
||||
| 77 | `stories-streak.py`, *after a stories run* | every fetched stories source returned "No results" on 3 consecutive days (`GDL_EMPTY_STREAK_FAIL` to change). Catches a session Instagram stopped honouring while the cookie is still present, which the preflight cannot see. One day per calendar date, so manual re-runs do not inflate it; state in `~/gdl/stories-empty-streak` |
|
||||
| 75 | existing | every stories source skipped by `--min-interval` |
|
||||
|
||||
**To recover from exit 76** the user must log in to Instagram in the Chrome on
|
||||
`mattellite` (VNC, display `:1`; neither password nor 2FA is something to
|
||||
script). Then `~/gdl/check-session.py` should say `session ok ... expires
|
||||
<date>`, and a manual `~/gdl/run-stories.sh` should find stories again.
|
||||
|
||||
**Verbose output.** A healthy run looked hung: pacing is 12–20 s, the config
|
||||
sets `output: null` (no per-file lines), and the live-listing probe captured
|
||||
output until it finished. `gdl-sync.py` now prints `[n/N] user / kind (time)`,
|
||||
a `... fetching: 4m00s elapsed, 12 new file(s) so far (latest: …)` heartbeat
|
||||
every 60 s, and `done in 6m12s, 14 new file(s)`. It only counts files already
|
||||
written; no extra requests, pacing unchanged.
|
||||
|
||||
**Archive resolution check** (read-only scan of header dimensions for the six
|
||||
scraped profiles, plus 8 single-post live metadata fetches, paced):
|
||||
|
||||
- *Logged-out gdl does not download smaller files — it fails* with a login
|
||||
redirect, so no logged-out media ever entered the archive.
|
||||
- Archived image sizes **match** what a logged-in fetch offers (1440×1800,
|
||||
3277×4096, 2976×3720, and the genuinely small ones such as 400×400 or
|
||||
742×928 sitting next to 1440 px siblings in one carousel). No image-upgrade
|
||||
pass is needed.
|
||||
- Videos: ~1,200 of ~1,300 archived videos are 720p. A 2026 reel offers a
|
||||
1080×1920 VP9 stream (video-only DASH, plus separate AAC); a 2024 post tops
|
||||
out at 1276×720, same as the archive. A video upgrade would mostly benefit
|
||||
2025–26 posts (~700), at one request each.
|
||||
- **iOS 18.7 Safari plays VP9 + Opus `.webm` but not VP9 + AAC `.mp4`** (error
|
||||
code 4; `canPlayType` wrongly says "probably"). A video upgrade must
|
||||
therefore output `.webm` — video stream copied, audio re-encoded to Opus
|
||||
96k. **Not built yet.** Open points before building: the viewer mints every
|
||||
video blob as `video/mp4` (`src/lib/archive-cache.ts`,
|
||||
`src/hooks/useArchiveScanner.ts`) and must derive the MIME type from the
|
||||
extension; superseded 720p files should be kept for review in a directory
|
||||
*outside* the Resilio-synced, viewer-scanned `archives/` tree; and the
|
||||
request budget (suggested: ≤40 posts/day, behind the preflight).
|
||||
|
||||
**Account swapped (same day).** The scraping account is now `erg05teur`
|
||||
(also signed in on the owner's phone). It is a fresh account, so the
|
||||
"replacement account" rules in `docs/gallery-dl.md` apply — and were not fully
|
||||
followed on day one: the resolution check and an unplanned catch-up run (below)
|
||||
spent roughly 90 requests within hours of the login. The owner chose to leave the
|
||||
scheduled timers as they are rather than pause them; the preflight (exit 76) and
|
||||
the all-empty-stories streak (exit 77) are the safety net. Worth watching: the
|
||||
weekly profiles timer's first fire (~2026-10-23) is a much larger spend than the
|
||||
daily stories runs.
|
||||
|
||||
**Host brought in line with the repo (same day).** Every tracked script and unit
|
||||
on `mattellite` is now byte-identical to `scripts/`, including the weekly
|
||||
profiles timer (it had been left monthly since 08-22). Installing it fired an
|
||||
unplanned catch-up profiles run (see "Scheduling"); that run honoured the
|
||||
20 h floor (20 of 24 sources skipped) and fetched the four `reels` sources,
|
||||
~15 min each at the usual pacing.
|
||||
|
||||
## 2026-09-14 — a tracked account renamed on Instagram: 0ct0ber19 → heejin
|
||||
|
||||
HeeJin's handle changed and the old one is now fully dead — `instagram.com/
|
||||
@@ -173,6 +257,13 @@ above, so a fully-skipped stories run alerts, not silently passes.
|
||||
Unset `HC_URL` (e.g. `HC_URL= ./run-stories.sh`) to run without pinging.
|
||||
`run-full-sweep.sh` and the reels scripts have no check configured yet.
|
||||
|
||||
**The systemd unit must run the wrappers, not `gdl-cron.sh` directly** —
|
||||
`HC_URL` lives in `run-<mode>.sh`, and `gdl-cron.sh` skips the ping silently
|
||||
when it is empty. Until 2026-10-14 the unit bypassed the wrappers, so no
|
||||
scheduled run had ever pinged. The preflight (exit 76) and the all-empty
|
||||
streak (exit 77) are described in "2026-10-14" above; both reach the dashboard
|
||||
as failures through the same trailing-status-code convention.
|
||||
|
||||
**This is also why `gdl-sync.py`'s `publish()` no longer treats the routine
|
||||
chown-permission warning as a failure (2026-09-14).** The archive share is
|
||||
never owned by the ssh user, so rsync's own `chown` step fails on *every*
|
||||
@@ -195,6 +286,68 @@ loudly, verified with both a real captured rsync failure and unit tests.
|
||||
The other 22 release tags stay on gitea. Pushing them all to GitHub triggers
|
||||
one container build per tag, because each tag carries its own workflow file.
|
||||
|
||||
## 2026-10-22 — gallery-dl auto-upgrade; posts listing went empty
|
||||
|
||||
Scheduled and manual runs were failing quietly from early October: after the session was
|
||||
restored, `posts` listings for `heejin`/`kimxxlip` returned `No results` in ~3 s (a caught-up
|
||||
profile skips; it does not say that), while stories worked and the posts opened fine in the
|
||||
browser. The host was on gallery-dl 1.32.14; 1.32.17 (2026-10-17) lists "instagram: update
|
||||
`user` & `posts` query parameters". After upgrading, the profile lookup failed instead
|
||||
(`KeyError: 'data'`, `Requested user could not be found`). That is upstream
|
||||
[gh#9829](https://github.com/mikf/gallery-dl/issues/9829) -- Instagram changed the query again
|
||||
on ~10-21; a fix is merged upstream but **not in a release** as of 10-22. Single-post fetches
|
||||
(`/p/<code>/`) still work, so the gap is the *listing*, which the browser covers -- see
|
||||
"Posts grid by browser" below.
|
||||
|
||||
`scripts/gdl-upgrade.sh` (`pipx upgrade --include-injected gallery-dl`, so yt-dlp too) now runs:
|
||||
|
||||
- at the start of every `gdl-cron.sh` run, after the session preflight and before any request;
|
||||
logs `upgrade: up to date` or `upgrade: CHANGED a -> b`, and never fails the run;
|
||||
- weekly from `gdl-upgrade.timer` (systemd user timer, installed on `mattellite` with the
|
||||
others -- systemd rather than cron, to match the rest of the host).
|
||||
|
||||
It costs no Instagram requests. The trade-off: a bad upstream release is picked up
|
||||
automatically; roll back with `pipx install --force gallery-dl==<old>` and re-inject yt-dlp.
|
||||
|
||||
### Posts grid by browser: `posts-sync.sh`
|
||||
|
||||
The reels workaround from August, generalised. `reels-scrape.py` is now
|
||||
**`instagram-chrome-scrape.py`** with `--tab reels|posts`, and `reels-sync.sh`'s body is now
|
||||
**`chrome-sync.sh`** (`GDL_TAB=reels|posts`); `reels-sync.sh` and the new `posts-sync.sh` are
|
||||
two-line wrappers that set the tab, so every older command still works.
|
||||
|
||||
```sh
|
||||
ssh mattellite
|
||||
~/gdl/posts-sync.sh heejin # scroll /heejin/, fetch new /p/ links as single posts
|
||||
```
|
||||
|
||||
`--tab posts` scrolls the main grid (`/<user>/`), collects `/p/` and `/reel/` links, and dedupes
|
||||
against the whole archive like the reels tab does. Unlike reels it **stops early**: the grid is
|
||||
newest-first, so after 2 scroll rounds in a row that turned up nothing unarchived
|
||||
(`--max-known-rounds`, round 1 exempt because pinned posts sit on top) there is nothing newer to
|
||||
find. First run, `heejin`: 3 scrolls, 36 posts seen, 30 already held, 6 new.
|
||||
|
||||
**Scheduled since 2026-10-23.** `gdl-cron.sh profiles` and `full-sweep` now list posts *and*
|
||||
reels this way: gallery-dl only does `--only stories,highlights`, then `chrome-sync.sh` runs per
|
||||
profile for the `posts` grid and the `reels` tab (both tabs needed: on 10-23 only 3 of heejin's
|
||||
114 reels were in the main grid). `profiles` uses the early stop (2 known rounds) on both tabs,
|
||||
and `full-sweep` scrolls every grid to the bottom. A failed browser step fails the run, which
|
||||
shows on Healthchecks. This reverses the earlier "by hand only" decision on purpose: the API
|
||||
listing is broken and the owner chose the browser. The trade-off is that unattended runs now
|
||||
need Chrome up on `:9222` with a live session, and they open and close their own tabs in it.
|
||||
`GDL_LISTING=api` restores the all-gallery-dl run once a release fixes gh#9829.
|
||||
|
||||
**Deploy scripts atomically, never in place.** bash reads a script as it runs, so overwriting
|
||||
`gdl-cron.sh` under a running sync makes bash resume at the same byte offset in the *new* text.
|
||||
On 10-22 an `scp` over it mid-run made the wrapper die after the sync with
|
||||
`line 163: status: unbound variable`. The fetch and publish had finished, but the exit line and
|
||||
the Healthchecks ping were skipped. Copy to a temp name and `mv` it into place (a new inode; the
|
||||
running bash keeps the old one), or check `pgrep -af gdl-` first.
|
||||
|
||||
**A `--min-interval` trap:** a run that returns `No results` still counts as a fetch, so the
|
||||
next run skips the source for 20 h. Re-run after a bad result with `--min-interval 0`
|
||||
(not `--force`, which also drops the probe cache).
|
||||
|
||||
## Guards — recreate these after a fresh clone
|
||||
|
||||
Neither guard is versioned, so a new clone has **no protection at all**:
|
||||
@@ -227,9 +380,15 @@ every deletion because of it.
|
||||
| path | what it is |
|
||||
|---|---|
|
||||
| `scripts/gdl-sync.py` | the gallery-dl fetcher; replaced JD2 for the ARTMS profiles |
|
||||
| `scripts/gdl-upgrade.sh` | `pipx upgrade` gallery-dl + yt-dlp; run by `gdl-cron.sh` and `gdl-upgrade.timer` |
|
||||
| `scripts/instagram-chrome-scrape.py` | list a profile's reels tab or posts grid by scrolling the signed-in Chrome over CDP |
|
||||
| `scripts/chrome-sync.sh` | scrape + fetch new posts for one profile; `reels-sync.sh` / `posts-sync.sh` set the tab |
|
||||
| `scripts/gdl-cron.sh` | unattended wrapper: `stories` \| `profiles` \| `full-sweep` |
|
||||
| `scripts/systemd/` | the timers actually installed on the fetch host |
|
||||
| `scripts/test_gdl_sync.py` | its tests |
|
||||
| `scripts/check-session.py` | preflight: is there a usable Instagram `sessionid` in the Chrome cookie DB? (exit 1 = no) |
|
||||
| `scripts/stories-streak.py` | postflight: all stories sources empty for several days running? |
|
||||
| `scripts/test_session_guards.py` | tests for the two guards above (offline) |
|
||||
| `scripts/jd2-sync.ts` | JDownloader `.crawljob` generator, still used elsewhere |
|
||||
| `docs/gallery-dl.md` | the measurements behind every option in the fetcher — **read before changing pacing** |
|
||||
| `docs/jdownloader.md` | the older JD2 flow |
|
||||
@@ -367,12 +526,22 @@ systemctl --user list-timers 'gdl-sync@*'
|
||||
| unit | schedule | next fire (as installed) |
|
||||
|---|---|---|
|
||||
| `gdl-sync@stories.timer` | daily 09:00 | 09:36:45 — the delay is the randomisation working |
|
||||
| `gdl-sync@profiles.timer` | 3rd of each month, 04:00 | 04:37:44 |
|
||||
| `gdl-sync@profiles.timer` | weekly (`OnCalendar=weekly`) + `RandomizedDelaySec=6d` — a random day each week, not pinned | 2026-10-24 06:44 (first weekly fire after the 10-14 install) |
|
||||
| `gdl-sync@full-sweep.timer` | 7th of Jan/Apr/Jul/Oct, 04:00 | 04:42:39 |
|
||||
|
||||
Unit files are version-controlled in `scripts/systemd/` and installed to
|
||||
`~/.config/systemd/user/`. One templated service, `gdl-sync@.service`, takes
|
||||
the mode as its instance name and runs `gdl-cron.sh %i`.
|
||||
the mode as its instance name and runs `run-%i.sh` (the fixed-mode wrappers
|
||||
around `gdl-cron.sh` — they carry the Healthchecks URL; see "Healthchecks.io").
|
||||
Before 2026-10-14 it ran `gdl-cron.sh %i` directly, which skipped every ping.
|
||||
The profiles timer was monthly (3rd, 04:00) until 2026-10-14, when the repo's
|
||||
weekly/random-day unit (committed 09-14 but never installed) was put on the host.
|
||||
Timer installs: copy the unit to `~/.config/systemd/user/`, `systemctl --user
|
||||
daemon-reload`, `restart` the timer. **`Persistent=true` makes that restart fire
|
||||
a catch-up run immediately if a scheduled elapse was missed** (it did, 2026-10-14:
|
||||
a real profiles run started the moment the timer was restarted). Touch
|
||||
`~/.local/share/systemd/timers/stamp-gdl-sync@<mode>.timer` first if a run has
|
||||
just happened by hand and you do not want one.
|
||||
|
||||
Three settings are load-bearing:
|
||||
|
||||
@@ -561,7 +730,12 @@ Two results are **not** failures, despite how they look:
|
||||
|
||||
### 2. Check the cookies before concluding you are blocked
|
||||
|
||||
Free, and it separates a local fault from a server-side one:
|
||||
First, `~/gdl/check-session.py` — it reads the same cookie DB gdl does and says
|
||||
whether a `sessionid` exists and when it expires (never prints values). Do not
|
||||
trust the open browser tab to tell you: it keeps showing a logged-in page after
|
||||
the session has died; reload it.
|
||||
|
||||
Then, free, and it separates a local fault from a server-side one:
|
||||
|
||||
```sh
|
||||
PATH=$HOME/.local/bin:$PATH gallery-dl \
|
||||
@@ -616,7 +790,7 @@ before something worse.
|
||||
changelog cites "posts & reels extractors... content retrieval failures", and
|
||||
upgrading (`pipx upgrade gallery-dl`) confirmed it -- a full `run-profiles.sh`
|
||||
across all six profiles came back with **zero** `HTTP redirect` errors on
|
||||
posts or reels, first time since 2026-08-26. `reels-scrape.py`/`reels-sync.sh`/
|
||||
posts or reels, first time since 2026-08-26. `reels-scrape.py` (now `instagram-chrome-scrape.py`)/`reels-sync.sh`/
|
||||
`run-reels-all.sh` are no longer needed for routine use; left in place as a
|
||||
fallback (and because the whole-archive dedup fix they picked up along the
|
||||
way, see below, is independently useful). If this regresses, check the
|
||||
@@ -638,7 +812,8 @@ Posts, highlights and stories were unaffected at the time; only reels broke
|
||||
(posts later broke too, some time before 2026-09-13, in the same way — same
|
||||
fix resolved both).
|
||||
|
||||
`scripts/reels-scrape.py` works around it by never calling that endpoint:
|
||||
`scripts/reels-scrape.py` (renamed `instagram-chrome-scrape.py` on 2026-10-22, when it gained
|
||||
`--tab posts`) works around it by never calling that endpoint:
|
||||
it drives the same signed-in Chrome via its loopback CDP port (`:9222`,
|
||||
already exposed for MCP automation), scrolls the reels tab like a person
|
||||
would, and scrapes `/reel/<code>/` links out of the rendered page. It only
|
||||
@@ -667,11 +842,11 @@ after — left behind for inspection, same as `gdl-cron.sh`'s `staging-*`.
|
||||
Verified end to end against `zindoriyam` on 2026-08-27: 26 reels found on the
|
||||
page, 10 new ones fetched and published cleanly on the first run.
|
||||
|
||||
**Not wired into `gdl-cron.sh` or the timers.** Both scripts drive your
|
||||
actual browser session rather than firing a background request, and are
|
||||
slower by design (real scrolling, not an API call) — both good reasons not
|
||||
to run this unattended without deciding that deliberately. Today it is a
|
||||
per-profile, by-hand tool only.
|
||||
**Not wired into `gdl-cron.sh` or the timers** (as of 2026-08; **superseded 2026-10-23**: the
|
||||
weekly and quarterly runs now use the browser listing, see "Posts grid by browser" above). Both
|
||||
scripts drive your actual browser session rather than firing a background request, and are
|
||||
slower by design (real scrolling, not an API call), both good reasons not to run this
|
||||
unattended without deciding that deliberately.
|
||||
|
||||
For all six ARTMS profiles in one go, `~/gdl/run-reels-all.sh` just calls
|
||||
`reels-sync.sh` once per profile in `artms_account_links.txt`, same
|
||||
@@ -711,7 +886,7 @@ PROFILE=someuser
|
||||
# 1. Scrape by scrolling; dedupe against the archive; write new URLs to a file.
|
||||
# Needs gallery-dl's own pipx venv python -- that's where websocket-client
|
||||
# (the one extra dependency this needs) got injected.
|
||||
~/.local/share/pipx/venvs/gallery-dl/bin/python3 ~/gdl/reels-scrape.py \
|
||||
~/.local/share/pipx/venvs/gallery-dl/bin/python3 ~/gdl/instagram-chrome-scrape.py \
|
||||
--profile "$PROFILE" \
|
||||
--index https://instaarchive.ergosteur.com \
|
||||
--out ~/gdl/"$PROFILE"-reels.txt
|
||||
@@ -727,7 +902,7 @@ cd ~/gdl && PATH=$HOME/.local/bin:$PATH ./gdl-sync.py \
|
||||
|
||||
Notes:
|
||||
|
||||
- `reels-scrape.py` fails fast if Chrome/CDP isn't up
|
||||
- `instagram-chrome-scrape.py` fails fast if Chrome/CDP isn't up
|
||||
(`curl -s http://localhost:9222/json/version` to check first).
|
||||
- If it finds nothing new, `gdl-sync.py --post-urls-file` refuses to run
|
||||
("no usable URLs") rather than doing nothing quietly — expected when a
|
||||
@@ -750,7 +925,7 @@ them, filed them correctly under `{username}` (the post's *true* owner, per
|
||||
its own metadata) — where they already existed from that profile's own
|
||||
regular sync — and `rsync --ignore-existing` silently skipped every one, so
|
||||
nothing was lost or duplicated. But 9 Instagram requests were spent finding
|
||||
that out. Since a shortcode is globally unique, `reels-scrape.py` now checks
|
||||
that out. Since a shortcode is globally unique, the scraper now checks
|
||||
every archived profile's listing, not just the one being scraped — all local
|
||||
requests to the viewer's own API, never to `instagram.com`, so checking all
|
||||
16 profiles costs nothing on the budget that actually matters. Confirmed
|
||||
@@ -789,6 +964,12 @@ and three systemd timers now run the sync unattended.
|
||||
State as of 2026-08-20, after the sync run and the repo split. Nothing here is
|
||||
broken; these are decisions not yet made and cleanups not yet done.
|
||||
|
||||
> **Update 2026-10-14:** the "Fetching" notes below predate the timers being enabled — all three
|
||||
> `gdl-sync@*` timers are installed and enabled on `mattellite` (see "Scheduling" and
|
||||
> "2026-10-14"). Newer open work: **the video-upgrade pass**
|
||||
> ([`docs/video-upgrade-plan.md`](docs/video-upgrade-plan.md) — measured and decided, not built),
|
||||
> and watching the fresh scraping account through its first weekly profiles run (~2026-10-23).
|
||||
|
||||
### Fetching
|
||||
|
||||
- **The timers are DISABLED** after the 2026-08-21 scraping warning; see that
|
||||
|
||||
@@ -16,6 +16,23 @@ by hand.
|
||||
|
||||
## The account was suspended on 2026-08-17 — read this first
|
||||
|
||||
> [!NOTE]
|
||||
> **The scraping account was replaced on 2026-10-14.** It is now **`erg05teur`**,
|
||||
> which is also signed in on the owner's phone (a separate session; the cookie
|
||||
> gdl uses lives only in the dedicated Chrome on `mattellite`). Everything below
|
||||
> about the suspension is the **previous** account's history and the reasons for
|
||||
> every rule; none of it has stopped applying, and the new account has no track
|
||||
> record, so treat it as a replacement account (see "For a replacement account"
|
||||
> below). The new session cookie was created 2026-10-14 ~10:07 EDT.
|
||||
>
|
||||
> What this account has been asked since then, honestly counted, because the
|
||||
> warm-up advice below was not followed to the letter: ~13 single-post / format
|
||||
> probes (resolution check, 2026-10-14), plus ~80 listing requests from an
|
||||
> **unplanned** reels catch-up run (installing the weekly profiles timer
|
||||
> triggered it via `Persistent=true`; stopped partway, `heejin / reels`
|
||||
> completed, `kimxxlip / reels` was abandoned ~7 min in). Scheduled runs were
|
||||
> deliberately left on their normal timers by the owner.
|
||||
|
||||
The account used for all of the below was suspended the same day this tooling
|
||||
was built, for "activity that doesn't follow our Community Standards on spam".
|
||||
The fetching was not the expensive part. **Verification was.**
|
||||
@@ -79,6 +96,9 @@ complaining, the rate-limited one has been unhappy for a while.
|
||||
|
||||
### For a replacement account
|
||||
|
||||
(Applies to `erg05teur`, 2026-10-14 onward — see the note at the top of this
|
||||
section for what it has already been asked.)
|
||||
|
||||
- Let it exist and be used normally for a while before pointing any tool at it.
|
||||
- Keep the cookie on one machine and one public IP, as before.
|
||||
- Start with a single small profile and stop for the day afterwards.
|
||||
@@ -146,6 +166,19 @@ Anonymous access is **not** a viable fallback: it serves lower-resolution media,
|
||||
caps profile pagination at 12 posts, and returns `AuthRequired` for stories and
|
||||
highlights.
|
||||
|
||||
(Observed 2026-10-07, after the session died: with no `sessionid` gallery-dl did
|
||||
not fall back to anonymous downloads at all — most sources failed with
|
||||
`HTTP redirect to login page` / `HTTP redirect to home page`, and stories came
|
||||
back as "No results". The lower-resolution claim above was **not** re-measured
|
||||
then; what *was* measured is that archived image sizes match a logged-in fetch.)
|
||||
|
||||
**The session can die silently, and nothing about the browser window shows it** —
|
||||
an open Instagram tab keeps rendering its logged-in UI until its next failed
|
||||
request. The only reliable signal is the cookie DB: `scripts/check-session.py`
|
||||
looks for an unexpired `sessionid` in this same profile, and `gdl-cron.sh` runs
|
||||
it before every scheduled run (exit 76 = refuse to scrape). See `TOOLING.md`,
|
||||
"2026-10-14".
|
||||
|
||||
## Output format
|
||||
|
||||
The viewer's parser is the contract, not JD2's exact bytes. `EXPORT_RE` in
|
||||
|
||||
@@ -0,0 +1,85 @@
|
||||
# Video upgrade pass — plan and findings (2026-10-14)
|
||||
|
||||
**Status: designed and partly measured, NOT built. Nothing in the archive has been changed.**
|
||||
Written so this can be picked up cold. Account context: the scraping account is the
|
||||
fresh `erg05teur` (see `docs/gallery-dl.md`, top of "The account was suspended…") — every
|
||||
request in this plan is spent on a new account, so the budget rules there apply in full.
|
||||
|
||||
## Why this exists
|
||||
|
||||
The question was whether a logged-out fetch had given us lower-resolution media and whether a
|
||||
logged-in run could upgrade it. Measured result:
|
||||
|
||||
- **Images: no upgrade needed.** Header-scan of 38.9k archived images, plus live metadata for 8
|
||||
posts across 2023–2026 (including the oddly small ones): archived sizes match exactly what a
|
||||
logged-in fetch offers (1440×1800, 3277×4096, 2976×3720; 400×400 / 742×928 items are small at
|
||||
the source, sitting next to 1440 px siblings in the same carousel).
|
||||
- **Logged-out gdl does not download smaller files — it fails** (login redirect). No logged-out
|
||||
media ever entered the archive.
|
||||
- **Videos: this is where there is something to gain.** Of ~1,300 archived videos in the six scraped
|
||||
profiles, ~1,200 are 720p (min side ≤ 720) and ~100 (2026 reels) are 1080p. A 2026 reel
|
||||
(`DTpycSlDKHs`, archived 720×1280) offers **1080×1920 VP9 video-only DASH + separate AAC audio**.
|
||||
A 2024 post (`C7RmlCcSWNR`) tops out at 1276×720 — same as the archive — so older videos
|
||||
probably gain nothing. Expected beneficiaries: **2025–26 videos, roughly 700 posts**. Not
|
||||
verified how far back 1080p exists.
|
||||
|
||||
## Decisions already made (by the owner)
|
||||
|
||||
1. **Videos in a separate pass** from anything else (images need none).
|
||||
2. **Keep the replaced files for review** in a `_superseded/` area — never overwrite/delete.
|
||||
3. **Output `.webm`, not `.mp4`** (video stream copied, audio re-encoded AAC → Opus 96 kbps; the
|
||||
audio stream is ~43 kbps HE-AAC so the loss is negligible). **Tested on real hardware:**
|
||||
iPhone, iOS 18.7 Safari plays the VP9+Opus `.webm` at 1080×1920 and **fails the VP9+AAC `.mp4`**
|
||||
(`MediaError` code 4, no video track) — even though `canPlayType` reports "probably" for it, so
|
||||
never trust `canPlayType` here. VP9-in-MP4 (the lossless remux) is therefore out.
|
||||
Re-encoding VP9 → H.264 was rejected (bigger, softer, no gain over the 720p original).
|
||||
|
||||
## How the sample was produced (reproducible, one post)
|
||||
|
||||
```sh
|
||||
# portrait video: 1080 is the WIDTH — `height=1080` selects nothing and fails "format not available"
|
||||
yt-dlp --cookies-from-browser chrome:/home/matt/.config/google-chrome-devtools \
|
||||
-f "bv*[width>=1080]+ba" --merge-output-format mp4 -o 'X.%(ext)s' https://www.instagram.com/p/<shortcode>/
|
||||
ffmpeg -i X.mp4 -c:v copy -c:a libopus -b:a 96k X.webm # ffmpeg 7.1.5 on mattellite
|
||||
```
|
||||
Sizes for that post: 6.46 MB `.mp4` / 6.53 MB `.webm`. For a browser test you need HTTP **Range**
|
||||
support (Python's `http.server` has none; Safari requires it) — the throwaway server used is not kept.
|
||||
`gallery-dl` needs `$HOME/.local/bin` on PATH over plain ssh.
|
||||
|
||||
## Open decisions / work before building
|
||||
|
||||
| item | notes |
|
||||
|---|---|
|
||||
| **Viewer MIME type** | `src/lib/archive-cache.ts:127` and `src/hooks/useArchiveScanner.ts:272,344,391` mint every video blob URL as `video/mp4`. Derive the type from the extension (`.webm` → `video/webm`) or Safari/iOS may refuse a WebM labelled mp4. `MEDIA_RE` / `isVideo` already recognise `.webm`. |
|
||||
| **Filename / dedup** | The archive names `YYYY-MM-DD_<user> - <shortcode>[ - <n>].mp4`. Swapping to `.webm` must not leave **both** in `archives/` (the viewer would show duplicates) — the old `.mp4` has to be *moved out*, not just joined by a `.webm`. Check `parseArchiveFilename` (`src/lib/archive-patterns.ts`) for the extension handling and the JD2-vs-gdl same-stem duplicate logic. |
|
||||
| **Skip-archive** | `artms.db` keys on `instagram<media_id>`, so a normal run will never re-fetch these. The upgrade tool must fetch targeted items deliberately and must not touch/poison `artms.db` semantics. |
|
||||
| **`_superseded/` location** | Must be **outside** `/volume1/rslsync/sync/Instagram-archive/archives/` — that tree is scanned by the viewer **and** inside the Resilio-synced share (it would sync every replaced file to devices). Not decided. Also unchecked: whether `agentapi` can write outside that share. Use rsync `--backup --backup-dir=<dir>/<date>/` so the old copy is kept. |
|
||||
| **Carousel videos** | yt-dlp on a carousel post URL fails ("No video formats found") — see `DIa5NpWRnIP`. gdl's own metadata gives per-child `ytdl:` URLs for videos in carousels; the tool needs to handle child items, not post URLs. |
|
||||
| **Request budget** | One `yt-dlp -F`-style request per post to learn what 1080p exists (plus the download). ~700 candidates ⇒ suggested **≤ 40 posts/day**, resumable, behind `check-session.py`, hand-started or a slow timer — never a burst. Candidates can be pre-ranked from the archive alone (720p, 2025–26) at zero request cost. |
|
||||
| **Storage** | 1080p VP9 ≈ 1.6× the 720p file (6 MB vs ~4 MB for that sample, plus the kept original). Estimate before running at scale. |
|
||||
|
||||
## Planned design
|
||||
|
||||
1. **Candidate list, zero requests:** scan the NAS archive (read-only, ssh `agentapi@<nas>` from
|
||||
mattellite; the scans in this doc used header parsing in plain Python over stdin because the
|
||||
NAS has no ffprobe) for videos ≤ 720 min-side from 2025-01 onward, newest first.
|
||||
2. **Dry-run mode (default):** for each candidate, one request to learn the best offered stream,
|
||||
record `{shortcode, archived WxH, offered WxH}` in a state file; print counts + estimated GB.
|
||||
Downloads nothing.
|
||||
3. **Execute mode:** only where offered width > archived width; download video+audio, `ffmpeg
|
||||
-c:v copy -c:a libopus -b:a 96k`, verify with `ffprobe` (dimensions, duration within 0.5 s of
|
||||
the original, both streams present), publish the `.webm`, move the old file to `_superseded/`.
|
||||
Cap per run (`--max-posts`), pace with the same 12–20 s gaps, stop on any 429 / login redirect.
|
||||
4. **First step when resuming:** do ONE reel end to end by hand (`DTpycSlDKHs`, ~3 requests) to prove
|
||||
download → remux → verify → publish → supersede → viewer playback, *then* the dry-run, and only after
|
||||
the weekly profiles timer has run cleanly on the new account (first fire ≈ 2026-10-23).
|
||||
|
||||
## Things measured along the way (for reference)
|
||||
|
||||
- Image header scan by year (six scraped profiles): ≥ 1440 px for the vast majority; ~75 images
|
||||
< 1000 px, all checked samples genuinely small at the source. `official_artms` 2023–24 has a
|
||||
cluster < 1440 px (~1080-class originals) — not probed.
|
||||
- Video min-side classes (six profiles): 2023 ≤720: 304, 2024 ≤720: 248, 2025 ≤720: 368 (+1 at
|
||||
1080), 2026 ≤720: 302 and ≤1080: 100 (+1 at 1440×2560). A handful ≤540.
|
||||
- The scan tools (read-only, stdlib only, run on the NAS user over ssh stdin) were throwaway
|
||||
and are not in the repo; recreate from this description if needed.
|
||||
Executable
+70
@@ -0,0 +1,70 @@
|
||||
#!/usr/bin/env python3
|
||||
"""
|
||||
Preflight for gdl-cron.sh: is there a usable Instagram login in the cookie jar gdl reads?
|
||||
|
||||
gdl-sync.py runs gallery-dl with `--cookies-from-browser chrome:<profile>`, so what matters
|
||||
is not what the interactive Chrome window LOOKS like (an open tab keeps showing its logged-in
|
||||
UI long after the session died -- found 2026-10-14, stories had been empty for a week) but
|
||||
whether the profile's cookie DB holds a `sessionid`. This reads the same DB, the same way.
|
||||
|
||||
Never prints cookie values -- only the name, expiry, and a verdict.
|
||||
|
||||
Exit 0 logged in (a non-expired sessionid exists)
|
||||
Exit 1 logged out / unreadable -- the caller must not scrape
|
||||
"""
|
||||
import datetime as dt
|
||||
import os
|
||||
import shutil
|
||||
import sqlite3
|
||||
import sys
|
||||
import tempfile
|
||||
import time
|
||||
from pathlib import Path
|
||||
|
||||
PROFILE = Path(os.environ.get("GDL_COOKIE_PROFILE",
|
||||
str(Path.home() / ".config/google-chrome-devtools")))
|
||||
WARN_DAYS = 7
|
||||
|
||||
|
||||
def main() -> int:
|
||||
src = PROFILE / "Default" / "Cookies"
|
||||
if not src.exists():
|
||||
print(f"no cookie DB at {src} -- Chrome profile missing?")
|
||||
return 1
|
||||
with tempfile.TemporaryDirectory() as tmp:
|
||||
# Copy first: Chrome holds the live DB open, and a copy keeps this read-only.
|
||||
for suffix in ("", "-journal", "-wal", "-shm"):
|
||||
f = Path(str(src) + suffix)
|
||||
if f.exists():
|
||||
shutil.copy2(f, Path(tmp) / f.name)
|
||||
try:
|
||||
con = sqlite3.connect(str(Path(tmp) / "Cookies"))
|
||||
rows = con.execute(
|
||||
"SELECT expires_utc, has_expires FROM cookies "
|
||||
"WHERE host_key LIKE '%instagram.com' AND name = 'sessionid'").fetchall()
|
||||
except sqlite3.Error as exc:
|
||||
print(f"cannot read cookie DB: {exc}")
|
||||
return 1
|
||||
if not rows:
|
||||
print("NOT LOGGED IN: no Instagram sessionid in the Chrome profile "
|
||||
f"{PROFILE}. Log in to Instagram in that Chrome window (the open tab can "
|
||||
"look logged in while stale -- reload it), then re-run.")
|
||||
return 1
|
||||
expires, has_expires = rows[0]
|
||||
if has_expires:
|
||||
exp = dt.datetime.fromtimestamp(expires / 1_000_000 - 11644473600, dt.timezone.utc)
|
||||
left = (exp.timestamp() - time.time()) / 86400
|
||||
if left <= 0:
|
||||
print(f"NOT LOGGED IN: the Instagram sessionid expired {exp:%Y-%m-%d}.")
|
||||
return 1
|
||||
note = f"expires {exp:%Y-%m-%d}"
|
||||
if left < WARN_DAYS:
|
||||
note += f" (WARNING: only {left:.1f} days left)"
|
||||
print(f"session ok: sessionid present, {note}")
|
||||
else:
|
||||
print("session ok: sessionid present (session cookie, no expiry)")
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.exit(main())
|
||||
Executable
+120
@@ -0,0 +1,120 @@
|
||||
#!/bin/bash
|
||||
# The full scrape-then-fetch pipeline for one profile: scroll the real page in
|
||||
# the signed-in Chrome (instagram-chrome-scrape.py), then fetch and publish
|
||||
# whatever's new as single posts. See TOOLING.md ("Reels: the API is blocked,
|
||||
# scrape by scrolling instead", and 2026-10-22 for the posts grid) for why this
|
||||
# exists at all -- it drives the actual signed-in browser session instead of
|
||||
# the listing API, and is slower by design.
|
||||
#
|
||||
# Usage: GDL_TAB=reels|posts ./chrome-sync.sh <username-or-profile-url>
|
||||
# Normally called through reels-sync.sh / posts-sync.sh, which set GDL_TAB.
|
||||
#
|
||||
# Also run by gdl-cron.sh (profiles, full-sweep) once per profile and tab since
|
||||
# 2026-10-23, with GDL_MAX_KNOWN_ROUNDS set -- see TOOLING.md "Posts grid by browser".
|
||||
# Exits non-zero if either step does. Everything is logged.
|
||||
set -eu
|
||||
|
||||
RAW="${1:?usage: GDL_TAB=reels|posts chrome-sync.sh <username-or-profile-url>}"
|
||||
TAB="${GDL_TAB:?set GDL_TAB=reels or posts (or use reels-sync.sh / posts-sync.sh)}"
|
||||
|
||||
GDL_HOME="${GDL_HOME:-$HOME/gdl}"
|
||||
GDL_PYTHON="${GDL_PYTHON:-$HOME/.local/share/pipx/venvs/gallery-dl/bin/python3}"
|
||||
INDEX="${GDL_INDEX:-https://instaarchive.ergosteur.com}"
|
||||
PUBLISH="${GDL_PUBLISH:-agentapi@10.20.28.200:/volume1/rslsync/sync/Instagram-archive/archives/}"
|
||||
|
||||
# Same hand-paced pacing gdl-cron.sh uses -- see its comment for why. Override
|
||||
# per-run with GDL_SLEEP_REQUEST etc. if you ever need to, but raise them
|
||||
# rather than lower them.
|
||||
SLEEP_REQUEST="${GDL_SLEEP_REQUEST:-12 20}"
|
||||
SLEEP="${GDL_SLEEP:-5 10}"
|
||||
RATE="${GDL_RATE:-500K}"
|
||||
SCROLL_PAUSE="${GDL_SCROLL_PAUSE:-2.0 3.5}"
|
||||
MAX_IDLE_ROUNDS="${GDL_MAX_IDLE_ROUNDS:-3}"
|
||||
# Empty = instagram-chrome-scrape.py's own default (2 for posts, 0 = scroll to the end for reels).
|
||||
MAX_KNOWN_ROUNDS="${GDL_MAX_KNOWN_ROUNDS:-}"
|
||||
|
||||
PATH="$HOME/.local/bin:$PATH"; export PATH
|
||||
|
||||
if [ ! -x "$GDL_PYTHON" ]; then
|
||||
echo "instagram-chrome-scrape.py needs gallery-dl's own pipx venv python (websocket-client" >&2
|
||||
echo "was injected there, not into the system python): $GDL_PYTHON not found" >&2
|
||||
exit 2
|
||||
fi
|
||||
|
||||
# Same username-from-URL parsing gdl-sync.py already does for --urls-file,
|
||||
# reused rather than re-implemented so the two never drift apart.
|
||||
PROFILE=$("$GDL_PYTHON" -c "
|
||||
import re, sys, importlib.util
|
||||
from pathlib import Path
|
||||
spec = importlib.util.spec_from_file_location('gdl_sync', Path('$GDL_HOME/gdl-sync.py'))
|
||||
gdl = importlib.util.module_from_spec(spec)
|
||||
sys.modules['gdl_sync'] = gdl
|
||||
spec.loader.exec_module(gdl)
|
||||
raw = '$RAW'
|
||||
m = gdl.RE_PROFILE_URL.match(raw)
|
||||
user = m.group('user') if m else raw.strip('/')
|
||||
if not user or '/' in user or ' ' in user:
|
||||
print(f'cannot read a username from {raw!r}', file=sys.stderr)
|
||||
sys.exit(1)
|
||||
print(user)
|
||||
")
|
||||
|
||||
mkdir -p "$GDL_HOME/logs"
|
||||
LOG="$GDL_HOME/logs/$TAB-$PROFILE-$(date +%Y%m%d-%H%M%S)-$$.log"
|
||||
URLS_FILE="$GDL_HOME/$PROFILE-$TAB.txt"
|
||||
STAGING="$GDL_HOME/staging-$TAB-$PROFILE"
|
||||
|
||||
echo "=== $TAB-sync $PROFILE $(date -Is) ===" | tee -a "$LOG"
|
||||
|
||||
# The exit status has to survive the pipe into tee -- see gdl-cron.sh's
|
||||
# comment on PIPESTATUS for why this needs to be bash, not sh.
|
||||
set +e
|
||||
"$GDL_PYTHON" "$GDL_HOME/instagram-chrome-scrape.py" \
|
||||
--profile "$PROFILE" \
|
||||
--tab "$TAB" \
|
||||
--index "$INDEX" \
|
||||
--scroll-pause $SCROLL_PAUSE \
|
||||
--max-idle-rounds "$MAX_IDLE_ROUNDS" \
|
||||
${MAX_KNOWN_ROUNDS:+--max-known-rounds "$MAX_KNOWN_ROUNDS"} \
|
||||
--out "$URLS_FILE" 2>&1 | tee -a "$LOG"
|
||||
scrape_status=${PIPESTATUS[0]}
|
||||
set -e
|
||||
|
||||
if [ "$scrape_status" -ne 0 ]; then
|
||||
echo "=== exit $scrape_status (scrape failed) at $(date -Is) ===" | tee -a "$LOG"
|
||||
exit "$scrape_status"
|
||||
fi
|
||||
|
||||
if [ ! -s "$URLS_FILE" ]; then
|
||||
echo "no new $TAB for $PROFILE; nothing to fetch" | tee -a "$LOG"
|
||||
echo "=== exit 0 at $(date -Is) ===" | tee -a "$LOG"
|
||||
exit 0
|
||||
fi
|
||||
|
||||
# Staging is wiped every run on purpose -- same reasoning as gdl-cron.sh: what
|
||||
# we already hold is decided by the archive dedupe in instagram-chrome-scrape.py, not by
|
||||
# what happens to be sitting in staging.
|
||||
rm -rf "$STAGING"
|
||||
|
||||
set +e
|
||||
# shellcheck disable=SC2086
|
||||
"$GDL_HOME/gdl-sync.py" \
|
||||
--publish "$PUBLISH" \
|
||||
--staging "$STAGING" \
|
||||
--post-urls-file "$URLS_FILE" \
|
||||
--sleep-request $SLEEP_REQUEST \
|
||||
--sleep $SLEEP \
|
||||
--rate "$RATE" \
|
||||
--execute 2>&1 | tee -a "$LOG"
|
||||
status=${PIPESTATUS[0]}
|
||||
set -e
|
||||
|
||||
echo "=== exit $status at $(date -Is) ===" | tee -a "$LOG"
|
||||
|
||||
# Keep the log directory from growing without bound -- scoped to this
|
||||
# script's own logs so it never touches gdl-cron.sh's rotation.
|
||||
ls -1t "$GDL_HOME/logs" | grep "^$TAB-[^0-9]" | tail -n +30 | while read -r old; do
|
||||
rm -f "$GDL_HOME/logs/$old"
|
||||
done
|
||||
|
||||
exit "$status"
|
||||
+99
-3
@@ -10,6 +10,11 @@
|
||||
# EDITED after we archived them, and by far the most
|
||||
# expensive thing here -- see TOOLING.md before running.
|
||||
#
|
||||
# Posts and reels are LISTED by scrolling the signed-in Chrome (chrome-sync.sh,
|
||||
# per profile and tab) rather than by gallery-dl's listing API, which broke on
|
||||
# 2026-10-22 (gh#9829) while single-post fetches kept working. gallery-dl still
|
||||
# does stories and highlights. GDL_LISTING=api restores the old all-API run.
|
||||
#
|
||||
# Exits non-zero if the sync does, so cron mails you. Everything is logged.
|
||||
set -eu
|
||||
|
||||
@@ -41,6 +46,15 @@ SLEEP_REQUEST="${GDL_SLEEP_REQUEST:-12 20}"
|
||||
SLEEP="${GDL_SLEEP:-5 10}"
|
||||
RATE="${GDL_RATE:-500K}"
|
||||
|
||||
LISTING="${GDL_LISTING:-browser}"
|
||||
case "$LISTING" in
|
||||
browser|api) ;;
|
||||
*) echo "GDL_LISTING must be browser or api, not '$LISTING'" >&2; exit 2 ;;
|
||||
esac
|
||||
GDL_PYTHON="${GDL_PYTHON:-$HOME/.local/share/pipx/venvs/gallery-dl/bin/python3}"
|
||||
# Set per mode below when posts/reels go through the browser: empty = no browser step.
|
||||
BROWSER_KNOWN_ROUNDS=""
|
||||
|
||||
case "$MODE" in
|
||||
# --min-interval 8, not the 20h default. The floor exists to stop an ABORTED
|
||||
# RESTART re-enumerating profiles -- a minutes-to-hours concern. At 20h the
|
||||
@@ -50,10 +64,23 @@ case "$MODE" in
|
||||
# request per profile, so the worst case 8h permits is roughly twelve
|
||||
# requests in a day instead of six.
|
||||
stories) ARGS="--only stories --min-interval 8" ;;
|
||||
profiles) ARGS="--only posts,reels,stories,highlights --abort 50" ;;
|
||||
# Browser listing: stop scrolling a grid after 2 rounds of nothing unarchived,
|
||||
# the browser's equivalent of --abort 50.
|
||||
profiles)
|
||||
if [ "$LISTING" = browser ]; then
|
||||
ARGS="--only stories,highlights --abort 50"; BROWSER_KNOWN_ROUNDS=2
|
||||
else
|
||||
ARGS="--only posts,reels,stories,highlights --abort 50"
|
||||
fi ;;
|
||||
# No --abort: the whole point of full-sweep is enumerating to the end, so it
|
||||
# is the only run that notices carousels edited after we archived them.
|
||||
full-sweep) ARGS="--only posts,reels,stories,highlights" ;;
|
||||
# is the only run that notices carousels edited after we archived them. The
|
||||
# browser equivalent is scrolling every grid to the bottom (0 = no early stop).
|
||||
full-sweep)
|
||||
if [ "$LISTING" = browser ]; then
|
||||
ARGS="--only stories,highlights"; BROWSER_KNOWN_ROUNDS=0
|
||||
else
|
||||
ARGS="--only posts,reels,stories,highlights"
|
||||
fi ;;
|
||||
# Renamed 2026-08-22: "full" was misleading (it is the abort-LIMITED run) and
|
||||
# "sweep" did not say it was the exhaustive one. Catch the old names rather
|
||||
# than failing with a bare error, in case something still passes them.
|
||||
@@ -66,6 +93,32 @@ mkdir -p "$GDL_HOME/logs"
|
||||
|
||||
[ -n "$HC_URL" ] && curl -fsS -m 10 --retry 3 -o /dev/null "$HC_URL/start" || true
|
||||
|
||||
# Preflight: refuse to scrape logged out. gdl reads the Chrome profile's cookie DB, and the
|
||||
# session in it can die silently -- from 2026-10-07 every stories run fetched "No results"
|
||||
# for all six profiles and still exited 0, so nothing alerted and a week of stories (which
|
||||
# cannot be backfilled) was lost. An open Chrome tab keeps showing its logged-in UI after
|
||||
# that, so only the cookie DB tells the truth. Exit 76 turns the Healthchecks check red.
|
||||
SESSION_CHECK="$GDL_HOME/check-session.py"
|
||||
if [ -x "$SESSION_CHECK" ]; then
|
||||
if ! SESSION_MSG=$("$SESSION_CHECK" 2>&1); then
|
||||
echo "=== $MODE run $(date -Is) ===" | tee -a "$LOG"
|
||||
echo "ABORT (no requests sent): $SESSION_MSG" | tee -a "$LOG" >&2
|
||||
echo "=== exit 76 at $(date -Is) ===" | tee -a "$LOG"
|
||||
[ -n "$HC_URL" ] && curl -fsS -m 10 --retry 3 --data-binary "@$LOG" \
|
||||
-o /dev/null "$HC_URL/76" || true
|
||||
exit 76
|
||||
fi
|
||||
echo "preflight: $SESSION_MSG" | tee -a "$LOG"
|
||||
fi
|
||||
|
||||
# Upgrade gallery-dl/yt-dlp first (PyPI only, no Instagram requests; never fatal). Instagram
|
||||
# endpoints change and gallery-dl follows within days -- on 1.32.14 the posts listing silently
|
||||
# returned "No results" after 1.32.17 had already fixed the query. See gdl-upgrade.sh.
|
||||
UPGRADE="$GDL_HOME/gdl-upgrade.sh"
|
||||
if [ -x "$UPGRADE" ]; then
|
||||
"$UPGRADE" 2>&1 | tee -a "$LOG" || true
|
||||
fi
|
||||
|
||||
# Staging is wiped every run ON PURPOSE. What we already hold is decided by the
|
||||
# skip-archive (--download-archive), never by which files happen to be sitting
|
||||
# in staging, so starting empty is correct -- and it keeps the publish rsync
|
||||
@@ -99,6 +152,39 @@ set +e
|
||||
status=${PIPESTATUS[0]}
|
||||
set -e
|
||||
|
||||
# Posts and reels by browser: scrape each profile's grid and Reels tab, fetch only
|
||||
# the new shortcodes as single posts. chrome-sync.sh logs and publishes on its own;
|
||||
# its output is copied into this run's log so the Healthchecks body shows it too.
|
||||
if [ -n "$BROWSER_KNOWN_ROUNDS" ]; then
|
||||
mapfile -t PROFILES < <("$GDL_PYTHON" -c "
|
||||
import sys, importlib.util
|
||||
from pathlib import Path
|
||||
spec = importlib.util.spec_from_file_location('gdl_sync', Path('$GDL_HOME/gdl-sync.py'))
|
||||
gdl = importlib.util.module_from_spec(spec)
|
||||
sys.modules['gdl_sync'] = gdl
|
||||
spec.loader.exec_module(gdl)
|
||||
for p in gdl.read_urls_file(Path('$GDL_HOME/artms_account_links.txt')):
|
||||
print(p)
|
||||
")
|
||||
browser_failed=0
|
||||
for p in "${PROFILES[@]}"; do
|
||||
for tab in posts reels; do
|
||||
echo "==> browser: $p / $tab" | tee -a "$LOG"
|
||||
set +e
|
||||
GDL_TAB=$tab GDL_MAX_KNOWN_ROUNDS=$BROWSER_KNOWN_ROUNDS \
|
||||
"$GDL_HOME/chrome-sync.sh" "$p" 2>&1 | tee -a "$LOG" >/dev/null
|
||||
s=${PIPESTATUS[0]}
|
||||
set -e
|
||||
if [ "$s" -ne 0 ]; then
|
||||
echo " browser $p / $tab FAILED (exit $s)" | tee -a "$LOG" >&2
|
||||
browser_failed=$((browser_failed + 1))
|
||||
fi
|
||||
done
|
||||
done
|
||||
echo "browser listing: ${#PROFILES[@]} profile(s), $browser_failed step(s) failed" | tee -a "$LOG"
|
||||
[ "$browser_failed" -gt 0 ] && [ "$status" -eq 0 ] && status=1
|
||||
fi
|
||||
|
||||
# A stories run that skipped every source is NOT a success. It means the
|
||||
# min-interval floor blocked the one surface that cannot be backfilled, and
|
||||
# without this it looks identical to a clean run: exit 0, "0 step(s) failed".
|
||||
@@ -111,6 +197,16 @@ if [ "$MODE" = "stories" ] && grep -q "sources : 0 to sync" "$RUNOUT"; then
|
||||
[ "$status" -eq 0 ] && status=75
|
||||
fi
|
||||
|
||||
# Every stories source empty for several days running = a session Instagram stopped
|
||||
# honouring while the cookie is still in the jar (the preflight cannot see that). Stories-only:
|
||||
# other modes legitimately return few items per source. See stories-streak.py.
|
||||
if [ "$MODE" = "stories" ] && [ "$status" -eq 0 ] && [ -x "$GDL_HOME/stories-streak.py" ]; then
|
||||
if ! STREAK_MSG=$("$GDL_HOME/stories-streak.py" "$RUNOUT" "$GDL_HOME/stories-empty-streak" 2>&1); then
|
||||
echo "WARNING: $STREAK_MSG" | tee -a "$LOG" >&2
|
||||
status=77
|
||||
fi
|
||||
fi
|
||||
|
||||
echo "=== exit $status at $(date -Is) ===" | tee -a "$LOG"
|
||||
|
||||
# The log becomes the ping body -- healthchecks.io keeps the last ping's body
|
||||
|
||||
+79
-5
@@ -94,6 +94,7 @@ import re
|
||||
import shutil
|
||||
import subprocess
|
||||
import sys
|
||||
import threading
|
||||
import time
|
||||
from dataclasses import dataclass, field
|
||||
from pathlib import Path
|
||||
@@ -224,7 +225,7 @@ def read_urls_file(path: Path) -> list[str]:
|
||||
def read_post_urls_file(path: Path) -> list[str]:
|
||||
"""
|
||||
Read individual post/reel URLs from a file, one per line -- the output
|
||||
format `reels-scrape.py` writes. Blank lines and `#` comments are skipped;
|
||||
format `instagram-chrome-scrape.py` writes. Blank lines and `#` comments are skipped;
|
||||
order is kept and duplicates dropped, same conventions as read_urls_file.
|
||||
"""
|
||||
urls: list[str] = []
|
||||
@@ -460,6 +461,71 @@ def build_config(rate: str, sleep_request: list[float],
|
||||
# Planning and execution
|
||||
# --------------------------------------------------------------------------
|
||||
|
||||
def _fmt_elapsed(sec: float) -> str:
|
||||
sec = int(sec)
|
||||
h, rest = divmod(sec, 3600)
|
||||
m, s = divmod(rest, 60)
|
||||
return f"{h}h{m:02d}m" if h else (f"{m}m{s:02d}s" if m else f"{s}s")
|
||||
|
||||
|
||||
def _count_new(root: Path, since: float) -> tuple[int, str | None]:
|
||||
"""Media files under `root` written since `since`, and the newest one's name."""
|
||||
n, newest, newest_t = 0, None, 0.0
|
||||
for d, _dirs, files in os.walk(root):
|
||||
for f in files:
|
||||
if f.endswith((".txt", ".json", ".part")):
|
||||
continue
|
||||
try:
|
||||
mt = os.stat(os.path.join(d, f)).st_mtime
|
||||
except OSError:
|
||||
continue
|
||||
if mt >= since:
|
||||
n += 1
|
||||
if mt > newest_t:
|
||||
newest, newest_t = f, mt
|
||||
return n, newest
|
||||
|
||||
|
||||
class Heartbeat:
|
||||
"""
|
||||
Prints a progress line every `every` seconds while a quiet step runs.
|
||||
|
||||
A healthy run looks hung otherwise: pacing is 12-20s between requests, the
|
||||
generated config sets `output: null` (no per-file lines), and the live
|
||||
listing probe captures gallery-dl output until it finishes. With `watch`
|
||||
set, the line also reports how many files have landed there so far.
|
||||
"""
|
||||
|
||||
def __init__(self, label: str, watch: Path | None = None, every: float = 60.0):
|
||||
self.label, self.watch, self.every = label, watch, every
|
||||
self._stop = threading.Event()
|
||||
self._thread = threading.Thread(target=self._run, daemon=True)
|
||||
self.t0 = time.time()
|
||||
|
||||
def _run(self) -> None:
|
||||
while not self._stop.wait(self.every):
|
||||
msg = f" ... {self.label}: {_fmt_elapsed(time.time() - self.t0)} elapsed"
|
||||
if self.watch is not None:
|
||||
n, newest = _count_new(self.watch, self.t0)
|
||||
msg += f", {n} new file(s) so far"
|
||||
if newest:
|
||||
msg += f" (latest: {newest})"
|
||||
print(msg, flush=True)
|
||||
|
||||
def __enter__(self) -> "Heartbeat":
|
||||
self.t0 = time.time()
|
||||
self._thread.start()
|
||||
return self
|
||||
|
||||
def __exit__(self, *exc) -> None:
|
||||
self._stop.set()
|
||||
self._thread.join()
|
||||
|
||||
@property
|
||||
def elapsed(self) -> float:
|
||||
return time.time() - self.t0
|
||||
|
||||
|
||||
def gdl_command(src: Source, staging: Path, config: Path, cookies: str,
|
||||
archive_db: Path | None) -> list[str]:
|
||||
cmd = [
|
||||
@@ -863,7 +929,7 @@ def main() -> int:
|
||||
help="file of post/reel URLs, one per line; # comments and "
|
||||
"blank lines allowed. Same handling as --post-url, "
|
||||
"paced with --sleep-request between items. This is "
|
||||
"the format reels-scrape.py writes")
|
||||
"the format instagram-chrome-scrape.py writes")
|
||||
ap.add_argument("--cookies", default="chrome:/home/matt/.config/google-chrome-devtools",
|
||||
help="gallery-dl --cookies-from-browser value")
|
||||
ap.add_argument("--archive-db", type=Path, default=None,
|
||||
@@ -1003,8 +1069,9 @@ def main() -> int:
|
||||
0.0 if args.force else args.probe_ttl)
|
||||
failures = 0
|
||||
|
||||
for prof, src, seed in plan:
|
||||
print(f"==> {prof.user} / {src.kind}")
|
||||
for n_src, (prof, src, seed) in enumerate(plan, 1):
|
||||
print(f"==> [{n_src}/{len(plan)}] {prof.user} / {src.kind} "
|
||||
f"({dt.datetime.now().strftime('%H:%M:%S')})")
|
||||
stage_dir = args.staging / (src.directory or ".")
|
||||
stage_dir.mkdir(parents=True, exist_ok=True)
|
||||
|
||||
@@ -1016,7 +1083,10 @@ def main() -> int:
|
||||
try:
|
||||
live = probes.get(src.url, now_ts)
|
||||
if live is None:
|
||||
print(" listing live items (metadata only; quiet until done)...")
|
||||
with Heartbeat("listing") as hb:
|
||||
live = probe_live(src, config_path, args.cookies)
|
||||
print(f" listed {len(live)} item(s) in {_fmt_elapsed(hb.elapsed)}")
|
||||
probes.put(src.url, live, stamp)
|
||||
probes.save()
|
||||
else:
|
||||
@@ -1034,12 +1104,16 @@ def main() -> int:
|
||||
|
||||
cmd = gdl_command(src, args.staging, config_path, args.cookies,
|
||||
args.archive_db)
|
||||
with Heartbeat("fetching", watch=stage_dir) as hb:
|
||||
result = subprocess.run(cmd)
|
||||
n_new, _ = _count_new(stage_dir, hb.t0)
|
||||
if result.returncode != 0:
|
||||
failures += 1
|
||||
# Keep going: one private or renamed profile must not abort the run.
|
||||
print(f" FAILED (exit {result.returncode})", file=sys.stderr)
|
||||
print(f" FAILED (exit {result.returncode}) after "
|
||||
f"{_fmt_elapsed(hb.elapsed)}", file=sys.stderr)
|
||||
else:
|
||||
print(f" done in {_fmt_elapsed(hb.elapsed)}, {n_new} new file(s)")
|
||||
# Recorded even for an empty fetch: the request was still spent.
|
||||
state.mark_fetched(src.url, stamp)
|
||||
state.save()
|
||||
|
||||
Executable
+34
@@ -0,0 +1,34 @@
|
||||
#!/bin/bash
|
||||
# Upgrade gallery-dl and its injected yt-dlp (both pipx). Instagram changes its
|
||||
# endpoints often and gallery-dl follows within days -- 1.32.17 (2026-10-17)
|
||||
# "update user & posts query parameters" landed while we were still on 1.32.14
|
||||
# and the posts listing silently returned "No results".
|
||||
#
|
||||
# Called by gdl-cron.sh before every scrape and weekly by gdl-upgrade.timer.
|
||||
# Never fatal: a failed upgrade (offline, PyPI down) must not stop a stories
|
||||
# run, which cannot be backfilled -- it just scrapes with what is installed.
|
||||
# Costs no Instagram requests (PyPI only).
|
||||
set -u
|
||||
|
||||
PATH="$HOME/.local/bin:$PATH"; export PATH
|
||||
|
||||
VENV_PY="${GDL_VENV_PY:-$HOME/.local/share/pipx/venvs/gallery-dl/bin/python}"
|
||||
|
||||
versions() {
|
||||
printf 'gallery-dl %s, yt-dlp %s' \
|
||||
"$(gallery-dl --version 2>/dev/null || echo '?')" \
|
||||
"$("$VENV_PY" -c 'import yt_dlp; print(yt_dlp.version.__version__)' 2>/dev/null || echo '?')"
|
||||
}
|
||||
|
||||
before=$(versions)
|
||||
if ! timeout 180 pipx upgrade --quiet --include-injected gallery-dl >/dev/null 2>&1; then
|
||||
echo "upgrade: WARNING pipx upgrade failed; continuing with $before" >&2
|
||||
exit 0
|
||||
fi
|
||||
after=$(versions)
|
||||
|
||||
if [ "$before" = "$after" ]; then
|
||||
echo "upgrade: up to date ($after)"
|
||||
else
|
||||
echo "upgrade: CHANGED $before -> $after"
|
||||
fi
|
||||
Regular → Executable
+91
-22
@@ -1,6 +1,6 @@
|
||||
#!/usr/bin/env python3
|
||||
"""
|
||||
Find a profile's reels by scrolling the real page, not calling the API.
|
||||
Find a profile's reels (or posts) by scrolling the real page, not calling the API.
|
||||
|
||||
gallery-dl's dedicated reels extractor POSTs to /api/v1/clips/user/, which
|
||||
Instagram now 302-redirects to the home page for this account -- confirmed on
|
||||
@@ -11,11 +11,19 @@ on loopback CDP for MCP automation -- see TOOLING.md) via the DevTools
|
||||
protocol, scrolls it like a person would, and scrapes `/reel/<code>/` links
|
||||
out of the rendered page instead.
|
||||
|
||||
`--tab posts` does the same for the main profile grid (`/<user>/`, both `/p/` and
|
||||
`/reel/` links), for when gallery-dl's posts listing breaks -- on 2026-10-22,
|
||||
1.32.14 returned "No results" and 1.32.17 "Requested user could not be found"
|
||||
(upstream gh#9829), while single-post fetches kept working. Since the grid
|
||||
runs newest-first, it stops scrolling once rounds turn up nothing but
|
||||
already-archived posts (`--max-known-rounds`) instead of walking a whole
|
||||
profile's history.
|
||||
|
||||
This only finds shortcodes; it never downloads anything itself. What comes
|
||||
out the other end is deduped against the archive (via the same --index used
|
||||
elsewhere) and printed as plain post URLs -- feed them to gdl-sync.py:
|
||||
|
||||
./scripts/reels-scrape.py --profile someuser \\
|
||||
./scripts/instagram-chrome-scrape.py --profile someuser \\
|
||||
--index https://instaarchive.ergosteur.com > /tmp/someuser-reels.txt
|
||||
|
||||
./scripts/gdl-sync.py --publish user@host:/path --staging /var/tmp/gdl \\
|
||||
@@ -38,7 +46,13 @@ import time
|
||||
from pathlib import Path
|
||||
from urllib.request import urlopen
|
||||
|
||||
RE_REEL_HREF = re.compile(r"/reel/([^/?#]+)")
|
||||
RE_POST_HREF = re.compile(r"/(p|reel)/([^/?#]+)")
|
||||
|
||||
# tab -> (page path under the profile, link kinds to collect)
|
||||
TABS = {
|
||||
"reels": ("reels/", ("reel",)),
|
||||
"posts": ("", ("p", "reel")),
|
||||
}
|
||||
|
||||
# The reels tab has finished loading once a scroll round finds no NEW
|
||||
# shortcodes this many times in a row -- lazy-loaded feeds sometimes stall
|
||||
@@ -46,21 +60,29 @@ RE_REEL_HREF = re.compile(r"/reel/([^/?#]+)")
|
||||
DEFAULT_MAX_IDLE_ROUNDS = 3
|
||||
DEFAULT_MAX_SCROLLS = 200
|
||||
DEFAULT_SCROLL_PAUSE = (2.0, 3.5)
|
||||
# Posts tab only: stop after this many consecutive scroll rounds whose new
|
||||
# shortcodes were all already archived. The first round is exempt -- it holds
|
||||
# the pinned posts, which are old by nature and sit above the new ones.
|
||||
DEFAULT_MAX_KNOWN_ROUNDS = 2
|
||||
|
||||
SCRAPE_JS = (
|
||||
|
||||
def scrape_js(kinds: tuple[str, ...]) -> str:
|
||||
selector = ", ".join(f'a[href*="/{k}/"]' for k in kinds)
|
||||
return (
|
||||
"(() => {"
|
||||
"window.scrollTo(0, document.body.scrollHeight);"
|
||||
"return Array.from(document.querySelectorAll('a[href*=\"/reel/\"]'))"
|
||||
f"return Array.from(document.querySelectorAll({json.dumps(selector)}))"
|
||||
".map(a => a.getAttribute('href'));"
|
||||
"})()"
|
||||
)
|
||||
)
|
||||
|
||||
|
||||
def extract_shortcodes(hrefs: list[str]) -> list[str]:
|
||||
def extract_shortcodes(hrefs: list[str],
|
||||
kinds: tuple[str, ...] = ("reel",)) -> list[str]:
|
||||
codes = []
|
||||
for href in hrefs:
|
||||
if m := RE_REEL_HREF.search(href):
|
||||
codes.append(m.group(1))
|
||||
if (m := RE_POST_HREF.search(href)) and m.group(1) in kinds:
|
||||
codes.append(m.group(2))
|
||||
return codes
|
||||
|
||||
|
||||
@@ -100,23 +122,45 @@ class CDP:
|
||||
self.ws.close()
|
||||
|
||||
|
||||
def scrape_reels(user: str, cdp_port: int = 9222,
|
||||
def should_stop_known(round_no: int, new: list[str], have: set[str],
|
||||
streak: int, max_known_rounds: int) -> tuple[bool, int]:
|
||||
"""
|
||||
Early-stop bookkeeping for a newest-first grid: returns (stop, streak).
|
||||
A round counts toward the streak when it found new shortcodes and every
|
||||
one is already archived; a round with anything unarchived resets it. An
|
||||
empty round leaves it alone (that is what max_idle_rounds is for), and
|
||||
round 1 never counts -- pinned posts come first.
|
||||
"""
|
||||
if max_known_rounds <= 0 or round_no == 1 or not new:
|
||||
return False, streak
|
||||
if all(c in have for c in new):
|
||||
streak += 1
|
||||
else:
|
||||
streak = 0
|
||||
return streak >= max_known_rounds, streak
|
||||
|
||||
|
||||
def scrape_grid(user: str, tab: str = "reels", cdp_port: int = 9222,
|
||||
scroll_pause: tuple[float, float] = DEFAULT_SCROLL_PAUSE,
|
||||
max_idle_rounds: int = DEFAULT_MAX_IDLE_ROUNDS,
|
||||
max_scrolls: int = DEFAULT_MAX_SCROLLS,
|
||||
have: set[str] | None = None,
|
||||
max_known_rounds: int = 0,
|
||||
log=lambda msg: None) -> list[str]:
|
||||
"""
|
||||
Open the profile's reels tab in a NEW tab of the already-signed-in Chrome,
|
||||
scroll it to the bottom repeatedly, and collect every unique `/reel/`
|
||||
shortcode that appears. Closes the tab when done either way.
|
||||
Open the profile's reels tab (or main grid) in a NEW tab of the
|
||||
already-signed-in Chrome, scroll it to the bottom repeatedly, and collect
|
||||
every unique shortcode that appears. Closes the tab when done either way.
|
||||
"""
|
||||
path, kinds = TABS[tab]
|
||||
js = scrape_js(kinds)
|
||||
version = json.loads(urlopen(f"http://localhost:{cdp_port}/json/version",
|
||||
timeout=10).read())
|
||||
browser = CDP(version["webSocketDebuggerUrl"])
|
||||
target_id = None
|
||||
try:
|
||||
target = browser.send("Target.createTarget", {
|
||||
"url": f"https://www.instagram.com/{user}/reels/"})
|
||||
"url": f"https://www.instagram.com/{user}/{path}"})
|
||||
target_id = target["targetId"]
|
||||
attach = browser.send("Target.attachToTarget", {
|
||||
"targetId": target_id, "flatten": True})
|
||||
@@ -128,16 +172,24 @@ def scrape_reels(user: str, cdp_port: int = 9222,
|
||||
|
||||
seen: set[str] = set()
|
||||
idle = 0
|
||||
known_streak = 0
|
||||
for i in range(max_scrolls):
|
||||
result = browser.send(
|
||||
"Runtime.evaluate",
|
||||
{"expression": SCRAPE_JS, "returnByValue": True},
|
||||
{"expression": js, "returnByValue": True},
|
||||
session_id=session_id)
|
||||
hrefs = result.get("result", {}).get("value") or []
|
||||
codes = extract_shortcodes(hrefs)
|
||||
new = [c for c in codes if c not in seen]
|
||||
codes = extract_shortcodes(hrefs, kinds)
|
||||
new = list(dict.fromkeys(c for c in codes if c not in seen))
|
||||
seen.update(new)
|
||||
log(f" scroll {i + 1}: {len(seen)} unique reels so far (+{len(new)})")
|
||||
log(f" scroll {i + 1}: {len(seen)} unique {tab} so far (+{len(new)})")
|
||||
|
||||
stop, known_streak = should_stop_known(
|
||||
i + 1, new, have or set(), known_streak, max_known_rounds)
|
||||
if stop:
|
||||
log(f" {known_streak} round(s) of already-archived {tab} in a row; "
|
||||
f"stopping")
|
||||
break
|
||||
|
||||
if new:
|
||||
idle = 0
|
||||
@@ -162,7 +214,9 @@ def main() -> int:
|
||||
ap = argparse.ArgumentParser(description=__doc__,
|
||||
formatter_class=argparse.RawDescriptionHelpFormatter)
|
||||
ap.add_argument("--profile", required=True,
|
||||
help="Instagram username to scrape reels for")
|
||||
help="Instagram username to scrape")
|
||||
ap.add_argument("--tab", choices=sorted(TABS), default="reels",
|
||||
help="reels tab, or the main posts grid (default: reels)")
|
||||
ap.add_argument("--index", required=True,
|
||||
help="existing archive listing, same as gdl-sync.py's "
|
||||
"--index: a local root, or the viewer's base URL. "
|
||||
@@ -179,6 +233,11 @@ def main() -> int:
|
||||
ap.add_argument("--max-scrolls", type=int, default=DEFAULT_MAX_SCROLLS,
|
||||
help="hard ceiling on scroll rounds, in case a page never "
|
||||
"goes idle")
|
||||
ap.add_argument("--max-known-rounds", type=int, default=None,
|
||||
help="stop after this many consecutive scroll rounds of "
|
||||
"only already-archived shortcodes; 0 disables "
|
||||
f"(default: {DEFAULT_MAX_KNOWN_ROUNDS} for posts, "
|
||||
"0 for reels)")
|
||||
ap.add_argument("--out", type=Path,
|
||||
help="write new reel URLs here, one per line (default: "
|
||||
"stdout)")
|
||||
@@ -208,18 +267,28 @@ def main() -> int:
|
||||
print(f"archive already holds {len(have)} shortcode(s) across "
|
||||
f"{len(profiles)} profile(s)", file=sys.stderr)
|
||||
|
||||
print(f"scraping https://www.instagram.com/{args.profile}/reels/ ...",
|
||||
max_known = args.max_known_rounds
|
||||
if max_known is None:
|
||||
max_known = DEFAULT_MAX_KNOWN_ROUNDS if args.tab == "posts" else 0
|
||||
|
||||
path, _ = TABS[args.tab]
|
||||
print(f"scraping https://www.instagram.com/{args.profile}/{path} ...",
|
||||
file=sys.stderr)
|
||||
codes = scrape_reels(args.profile, cdp_port=args.cdp_port,
|
||||
codes = scrape_grid(args.profile, tab=args.tab, cdp_port=args.cdp_port,
|
||||
scroll_pause=tuple(args.scroll_pause),
|
||||
max_idle_rounds=args.max_idle_rounds,
|
||||
max_scrolls=args.max_scrolls,
|
||||
have=have, max_known_rounds=max_known,
|
||||
log=lambda msg: print(msg, file=sys.stderr))
|
||||
|
||||
new_codes = [c for c in codes if c not in have]
|
||||
print(f"found {len(codes)} reel(s) on the page, {len(codes) - len(new_codes)} "
|
||||
print(f"found {len(codes)} {args.tab} on the page, {len(codes) - len(new_codes)} "
|
||||
f"already archived, {len(new_codes)} new", file=sys.stderr)
|
||||
|
||||
# /p/<code>/ resolves reels too, so the posts grid needs no per-kind URL.
|
||||
if args.tab == "posts":
|
||||
urls = [f"https://www.instagram.com/p/{c}/" for c in new_codes]
|
||||
else:
|
||||
urls = [f"https://www.instagram.com/{args.profile}/reel/{c}/"
|
||||
for c in new_codes]
|
||||
text = "\n".join(urls)
|
||||
Executable
+11
@@ -0,0 +1,11 @@
|
||||
#!/bin/bash
|
||||
# chrome-sync.sh for the main profile grid: scroll /<user>/ in the signed-in
|
||||
# Chrome, collect /p/ and /reel/ links, fetch only the new ones as single
|
||||
# posts. A by-hand fallback for when gallery-dl's posts listing is broken
|
||||
# (2026-10-22: upstream gh#9829) while single-post fetches still work -- see
|
||||
# TOOLING.md. Stops scrolling once it reaches already-archived posts.
|
||||
#
|
||||
# Usage: ./posts-sync.sh <username-or-profile-url>
|
||||
set -eu
|
||||
export GDL_TAB=posts
|
||||
exec "$(dirname "$0")/chrome-sync.sh" "$@"
|
||||
+4
-110
@@ -1,114 +1,8 @@
|
||||
#!/bin/bash
|
||||
# The full reels pipeline for one profile: scrape by scrolling the real page,
|
||||
# then fetch and publish whatever's new. See TOOLING.md ("Reels: the API is
|
||||
# blocked, scrape by scrolling instead") for why this exists at all -- the
|
||||
# dedicated reels API is blocked, this drives the actual signed-in browser
|
||||
# session instead, and is slower by design.
|
||||
# chrome-sync.sh for a profile's reels tab. See TOOLING.md ("Reels: the API is
|
||||
# blocked, scrape by scrolling instead").
|
||||
#
|
||||
# Usage: ./reels-sync.sh <username-or-profile-url>
|
||||
# ./reels-sync.sh zindoriyam
|
||||
# ./reels-sync.sh https://www.instagram.com/zindoriyam/
|
||||
#
|
||||
# Deliberately NOT wired into gdl-cron.sh or the timers -- see TOOLING.md.
|
||||
# Exits non-zero if either step does. Everything is logged.
|
||||
set -eu
|
||||
|
||||
RAW="${1:?usage: reels-sync.sh <username-or-profile-url>}"
|
||||
|
||||
GDL_HOME="${GDL_HOME:-$HOME/gdl}"
|
||||
GDL_PYTHON="${GDL_PYTHON:-$HOME/.local/share/pipx/venvs/gallery-dl/bin/python3}"
|
||||
INDEX="${GDL_INDEX:-https://instaarchive.ergosteur.com}"
|
||||
PUBLISH="${GDL_PUBLISH:-agentapi@10.20.28.200:/volume1/rslsync/sync/Instagram-archive/archives/}"
|
||||
|
||||
# Same hand-paced pacing gdl-cron.sh uses -- see its comment for why. Override
|
||||
# per-run with GDL_SLEEP_REQUEST etc. if you ever need to, but raise them
|
||||
# rather than lower them.
|
||||
SLEEP_REQUEST="${GDL_SLEEP_REQUEST:-12 20}"
|
||||
SLEEP="${GDL_SLEEP:-5 10}"
|
||||
RATE="${GDL_RATE:-500K}"
|
||||
SCROLL_PAUSE="${GDL_SCROLL_PAUSE:-2.0 3.5}"
|
||||
MAX_IDLE_ROUNDS="${GDL_MAX_IDLE_ROUNDS:-3}"
|
||||
|
||||
PATH="$HOME/.local/bin:$PATH"; export PATH
|
||||
|
||||
if [ ! -x "$GDL_PYTHON" ]; then
|
||||
echo "reels-scrape.py needs gallery-dl's own pipx venv python (websocket-client" >&2
|
||||
echo "was injected there, not into the system python): $GDL_PYTHON not found" >&2
|
||||
exit 2
|
||||
fi
|
||||
|
||||
# Same username-from-URL parsing gdl-sync.py already does for --urls-file,
|
||||
# reused rather than re-implemented so the two never drift apart.
|
||||
PROFILE=$("$GDL_PYTHON" -c "
|
||||
import re, sys, importlib.util
|
||||
from pathlib import Path
|
||||
spec = importlib.util.spec_from_file_location('gdl_sync', Path('$GDL_HOME/gdl-sync.py'))
|
||||
gdl = importlib.util.module_from_spec(spec)
|
||||
sys.modules['gdl_sync'] = gdl
|
||||
spec.loader.exec_module(gdl)
|
||||
raw = '$RAW'
|
||||
m = gdl.RE_PROFILE_URL.match(raw)
|
||||
user = m.group('user') if m else raw.strip('/')
|
||||
if not user or '/' in user or ' ' in user:
|
||||
print(f'cannot read a username from {raw!r}', file=sys.stderr)
|
||||
sys.exit(1)
|
||||
print(user)
|
||||
")
|
||||
|
||||
mkdir -p "$GDL_HOME/logs"
|
||||
LOG="$GDL_HOME/logs/reels-$PROFILE-$(date +%Y%m%d-%H%M%S)-$$.log"
|
||||
URLS_FILE="$GDL_HOME/$PROFILE-reels.txt"
|
||||
STAGING="$GDL_HOME/staging-reels-$PROFILE"
|
||||
|
||||
echo "=== reels-sync $PROFILE $(date -Is) ===" | tee -a "$LOG"
|
||||
|
||||
# The exit status has to survive the pipe into tee -- see gdl-cron.sh's
|
||||
# comment on PIPESTATUS for why this needs to be bash, not sh.
|
||||
set +e
|
||||
"$GDL_PYTHON" "$GDL_HOME/reels-scrape.py" \
|
||||
--profile "$PROFILE" \
|
||||
--index "$INDEX" \
|
||||
--scroll-pause $SCROLL_PAUSE \
|
||||
--max-idle-rounds "$MAX_IDLE_ROUNDS" \
|
||||
--out "$URLS_FILE" 2>&1 | tee -a "$LOG"
|
||||
scrape_status=${PIPESTATUS[0]}
|
||||
set -e
|
||||
|
||||
if [ "$scrape_status" -ne 0 ]; then
|
||||
echo "=== exit $scrape_status (scrape failed) at $(date -Is) ===" | tee -a "$LOG"
|
||||
exit "$scrape_status"
|
||||
fi
|
||||
|
||||
if [ ! -s "$URLS_FILE" ]; then
|
||||
echo "no new reels for $PROFILE; nothing to fetch" | tee -a "$LOG"
|
||||
echo "=== exit 0 at $(date -Is) ===" | tee -a "$LOG"
|
||||
exit 0
|
||||
fi
|
||||
|
||||
# Staging is wiped every run on purpose -- same reasoning as gdl-cron.sh: what
|
||||
# we already hold is decided by the archive dedupe in reels-scrape.py, not by
|
||||
# what happens to be sitting in staging.
|
||||
rm -rf "$STAGING"
|
||||
|
||||
set +e
|
||||
# shellcheck disable=SC2086
|
||||
"$GDL_HOME/gdl-sync.py" \
|
||||
--publish "$PUBLISH" \
|
||||
--staging "$STAGING" \
|
||||
--post-urls-file "$URLS_FILE" \
|
||||
--sleep-request $SLEEP_REQUEST \
|
||||
--sleep $SLEEP \
|
||||
--rate "$RATE" \
|
||||
--execute 2>&1 | tee -a "$LOG"
|
||||
status=${PIPESTATUS[0]}
|
||||
set -e
|
||||
|
||||
echo "=== exit $status at $(date -Is) ===" | tee -a "$LOG"
|
||||
|
||||
# Keep the log directory from growing without bound -- scoped to this
|
||||
# script's own logs so it never touches gdl-cron.sh's rotation.
|
||||
ls -1t "$GDL_HOME/logs" | grep '^reels-' | tail -n +30 | while read -r old; do
|
||||
rm -f "$GDL_HOME/logs/$old"
|
||||
done
|
||||
|
||||
exit "$status"
|
||||
export GDL_TAB=reels
|
||||
exec "$(dirname "$0")/chrome-sync.sh" "$@"
|
||||
|
||||
@@ -5,9 +5,9 @@
|
||||
# version of running reels-sync.sh six times by hand, which is what the
|
||||
# 2026-09-03 run was before this script existed.
|
||||
#
|
||||
# Deliberately not wired into gdl-cron.sh or a timer -- same reasoning as
|
||||
# reels-sync.sh itself: it drives your actual browser session and is slower
|
||||
# by design, both good reasons to run it deliberately, not unattended.
|
||||
# A by-hand full sweep of every Reels tab (no early stop). The weekly
|
||||
# gdl-cron.sh profiles run lists reels through chrome-sync.sh itself, with an
|
||||
# early stop, since 2026-10-23 -- see TOOLING.md "Posts grid by browser".
|
||||
#
|
||||
# Usage: ./reels-all.sh
|
||||
# GDL_ARTMS_LINKS=/path/to/list ./reels-all.sh # override the profile list
|
||||
|
||||
Executable
+57
@@ -0,0 +1,57 @@
|
||||
#!/usr/bin/env python3
|
||||
"""
|
||||
Detect a silently dead Instagram session from a stories run's output.
|
||||
|
||||
A dead session (killed server-side while the sessionid is still in the cookie jar, so
|
||||
check-session.py passes) shows up as every fetched stories source returning
|
||||
"No results" -- the daily run still exits 0. From 2026-10-07 all six profiles came back
|
||||
empty every day and nothing alerted. One all-empty day is normal (nobody posted a story);
|
||||
several in a row across ALL sources is not.
|
||||
|
||||
Usage: stories-streak.py <this run's output file> [state file]
|
||||
Counts at most one day per calendar date (manual re-runs don't inflate it). Prints a message
|
||||
and exits 1 once the streak reaches GDL_EMPTY_STREAK_FAIL days (default 3); exits 0 otherwise.
|
||||
Runs that fetched nothing (all skipped / no sources) leave the streak untouched.
|
||||
"""
|
||||
import datetime as dt
|
||||
import os
|
||||
import re
|
||||
import sys
|
||||
from pathlib import Path
|
||||
|
||||
THRESHOLD = int(os.environ.get("GDL_EMPTY_STREAK_FAIL", "3"))
|
||||
|
||||
|
||||
def main() -> int:
|
||||
out = Path(sys.argv[1]).read_text(errors="replace")
|
||||
state = Path(sys.argv[2] if len(sys.argv) > 2
|
||||
else Path.home() / "gdl" / "stories-empty-streak")
|
||||
m = re.search(r"sources\s*:\s*(\d+) to sync", out)
|
||||
to_sync = int(m.group(1)) if m else 0
|
||||
if to_sync == 0:
|
||||
return 0 # nothing fetched: no evidence either way
|
||||
empty = len(re.findall(r"No results for \S*/stories/", out))
|
||||
today = dt.date.today().isoformat()
|
||||
last_date, streak = "", 0
|
||||
try:
|
||||
last_date, s = state.read_text().split()
|
||||
streak = int(s)
|
||||
except (OSError, ValueError):
|
||||
pass
|
||||
if empty >= to_sync: # every fetched source came back empty
|
||||
if last_date != today:
|
||||
streak += 1
|
||||
else:
|
||||
streak = 0
|
||||
state.write_text(f"{today} {streak}\n")
|
||||
if streak >= THRESHOLD:
|
||||
print(f"SESSION LIKELY DEAD: stories came back empty for ALL {to_sync} profile(s) on "
|
||||
f"{streak} consecutive day(s). The cookie is present but Instagram is probably "
|
||||
"no longer honouring it -- log in again in the Chrome window (reload the tab to "
|
||||
"check) and re-run.")
|
||||
return 1
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.exit(main())
|
||||
@@ -8,7 +8,7 @@ Wants=network-online.target
|
||||
|
||||
[Service]
|
||||
Type=oneshot
|
||||
ExecStart=%h/gdl/gdl-cron.sh %i
|
||||
ExecStart=%h/gdl/run-%i.sh
|
||||
# A sync has no deadline and the pacing is deliberately slow; a full sweep can
|
||||
# run for hours. Never let systemd kill one midway -- a half-published run is
|
||||
# the one state the publish step is designed to avoid.
|
||||
|
||||
@@ -0,0 +1,9 @@
|
||||
[Unit]
|
||||
Description=Upgrade gallery-dl and yt-dlp (pipx)
|
||||
After=network-online.target
|
||||
Wants=network-online.target
|
||||
|
||||
[Service]
|
||||
Type=oneshot
|
||||
ExecStart=%h/gdl/gdl-upgrade.sh
|
||||
Nice=10
|
||||
@@ -0,0 +1,12 @@
|
||||
[Unit]
|
||||
Description=Weekly gallery-dl and yt-dlp upgrade
|
||||
# gdl-cron.sh also upgrades before every scrape; this keeps the tools current
|
||||
# on weeks when no run happens to call it.
|
||||
|
||||
[Timer]
|
||||
OnCalendar=weekly
|
||||
RandomizedDelaySec=1d
|
||||
Persistent=true
|
||||
|
||||
[Install]
|
||||
WantedBy=timers.target
|
||||
@@ -326,7 +326,7 @@ class UrlsFile(unittest.TestCase):
|
||||
|
||||
|
||||
class PostUrlsFile(unittest.TestCase):
|
||||
"""The format reels-scrape.py writes: whole URLs, not usernames."""
|
||||
"""The format instagram-chrome-scrape.py writes: whole URLs, not usernames."""
|
||||
|
||||
def test_skips_comments_blanks_and_duplicates(self):
|
||||
with tempfile.TemporaryDirectory() as d:
|
||||
@@ -343,5 +343,43 @@ class PostUrlsFile(unittest.TestCase):
|
||||
])
|
||||
|
||||
|
||||
class ProgressOutput(unittest.TestCase):
|
||||
"""The heartbeat that stops a healthy paced run (12-20s between requests, no per-file
|
||||
output) from looking hung."""
|
||||
|
||||
def test_elapsed_formatting(self):
|
||||
self.assertEqual(gdl._fmt_elapsed(5), "5s")
|
||||
self.assertEqual(gdl._fmt_elapsed(125), "2m05s")
|
||||
self.assertEqual(gdl._fmt_elapsed(7300), "2h01m")
|
||||
|
||||
def test_count_new_counts_media_written_since_and_ignores_sidecars(self):
|
||||
with tempfile.TemporaryDirectory() as d:
|
||||
root = Path(d)
|
||||
(root / "sub").mkdir()
|
||||
old = root / "old.jpg"
|
||||
old.write_text("x")
|
||||
import os
|
||||
os.utime(old, (1, 1)) # long before `since`
|
||||
(root / "sub" / "new.jpg").write_text("x")
|
||||
(root / "sub" / "new.mp4").write_text("x")
|
||||
for sidecar in ("new.txt", "new.json", "half.jpg.part"):
|
||||
(root / "sub" / sidecar).write_text("x")
|
||||
n, newest = gdl._count_new(root, since=1000)
|
||||
self.assertEqual(n, 2)
|
||||
self.assertIn(newest, ("new.jpg", "new.mp4"))
|
||||
|
||||
def test_heartbeat_prints_while_running_and_stops_cleanly(self):
|
||||
with tempfile.TemporaryDirectory() as d:
|
||||
with mock.patch("builtins.print") as pr:
|
||||
with gdl.Heartbeat("fetching", watch=Path(d), every=0.05) as hb:
|
||||
(Path(d) / "a.jpg").write_text("x")
|
||||
import time
|
||||
time.sleep(0.2)
|
||||
lines = [str(c.args[0]) for c in pr.call_args_list]
|
||||
self.assertTrue(any("fetching" in s and "new file(s) so far" in s for s in lines))
|
||||
self.assertFalse(hb._thread.is_alive())
|
||||
self.assertGreater(hb.elapsed, 0.1)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main(verbosity=2)
|
||||
|
||||
@@ -0,0 +1,71 @@
|
||||
#!/usr/bin/env python3
|
||||
"""
|
||||
Tests for the pure parts of instagram-chrome-scrape.py -- everything except the actual
|
||||
CDP session, which needs a live signed-in Chrome and is exercised by hand.
|
||||
|
||||
python3 -m unittest discover -s scripts -p 'test_*.py'
|
||||
"""
|
||||
|
||||
import importlib.util
|
||||
import sys
|
||||
import unittest
|
||||
from pathlib import Path
|
||||
|
||||
_spec = importlib.util.spec_from_file_location(
|
||||
"instagram_chrome_scrape", Path(__file__).with_name("instagram-chrome-scrape.py"))
|
||||
scrape = importlib.util.module_from_spec(_spec)
|
||||
sys.modules["instagram_chrome_scrape"] = scrape
|
||||
_spec.loader.exec_module(scrape)
|
||||
|
||||
|
||||
class ExtractShortcodes(unittest.TestCase):
|
||||
def test_reads_relative_and_absolute_hrefs(self):
|
||||
hrefs = [
|
||||
"/someuser/reel/ABC123/",
|
||||
"https://www.instagram.com/someuser/reel/DEF456/",
|
||||
"/reel/GHI789/?img_index=1",
|
||||
]
|
||||
self.assertEqual(scrape.extract_shortcodes(hrefs),
|
||||
["ABC123", "DEF456", "GHI789"])
|
||||
|
||||
def test_ignores_non_reel_links(self):
|
||||
hrefs = ["/someuser/", "/someuser/p/ABC123/", "/explore/tags/foo/"]
|
||||
self.assertEqual(scrape.extract_shortcodes(hrefs), [])
|
||||
|
||||
def test_empty_input(self):
|
||||
self.assertEqual(scrape.extract_shortcodes([]), [])
|
||||
|
||||
def test_posts_kinds_read_both_post_and_reel_links(self):
|
||||
hrefs = ["/someuser/p/ABC123/", "/p/DEF456/?img_index=2",
|
||||
"/someuser/reel/GHI789/", "/explore/tags/foo/"]
|
||||
self.assertEqual(scrape.extract_shortcodes(hrefs, ("p", "reel")),
|
||||
["ABC123", "DEF456", "GHI789"])
|
||||
|
||||
|
||||
class ShouldStopKnown(unittest.TestCase):
|
||||
have = {"OLD1", "OLD2", "OLD3", "OLD4"}
|
||||
|
||||
def step(self, round_no, new, streak=0, max_known=2):
|
||||
return scrape.should_stop_known(round_no, new, self.have, streak, max_known)
|
||||
|
||||
def test_first_round_never_counts(self):
|
||||
# pinned posts: old, but on top of the grid
|
||||
self.assertEqual(self.step(1, ["OLD1", "OLD2"]), (False, 0))
|
||||
|
||||
def test_stops_after_consecutive_known_rounds(self):
|
||||
stop, streak = self.step(2, ["OLD1"])
|
||||
self.assertEqual((stop, streak), (False, 1))
|
||||
self.assertEqual(self.step(3, ["OLD2"], streak), (True, 2))
|
||||
|
||||
def test_unarchived_resets_streak(self):
|
||||
self.assertEqual(self.step(3, ["OLD3", "NEW1"], streak=1), (False, 0))
|
||||
|
||||
def test_empty_round_leaves_streak(self):
|
||||
self.assertEqual(self.step(3, [], streak=1), (False, 1))
|
||||
|
||||
def test_disabled(self):
|
||||
self.assertEqual(self.step(5, ["OLD1"], streak=9, max_known=0), (False, 9))
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main(verbosity=2)
|
||||
@@ -1,40 +0,0 @@
|
||||
#!/usr/bin/env python3
|
||||
"""
|
||||
Tests for the pure parts of reels-scrape.py -- everything except the actual
|
||||
CDP session, which needs a live signed-in Chrome and is exercised by hand.
|
||||
|
||||
python3 -m unittest discover -s scripts -p 'test_*.py'
|
||||
"""
|
||||
|
||||
import importlib.util
|
||||
import sys
|
||||
import unittest
|
||||
from pathlib import Path
|
||||
|
||||
_spec = importlib.util.spec_from_file_location(
|
||||
"reels_scrape", Path(__file__).with_name("reels-scrape.py"))
|
||||
scrape = importlib.util.module_from_spec(_spec)
|
||||
sys.modules["reels_scrape"] = scrape
|
||||
_spec.loader.exec_module(scrape)
|
||||
|
||||
|
||||
class ExtractShortcodes(unittest.TestCase):
|
||||
def test_reads_relative_and_absolute_hrefs(self):
|
||||
hrefs = [
|
||||
"/someuser/reel/ABC123/",
|
||||
"https://www.instagram.com/someuser/reel/DEF456/",
|
||||
"/reel/GHI789/?img_index=1",
|
||||
]
|
||||
self.assertEqual(scrape.extract_shortcodes(hrefs),
|
||||
["ABC123", "DEF456", "GHI789"])
|
||||
|
||||
def test_ignores_non_reel_links(self):
|
||||
hrefs = ["/someuser/", "/someuser/p/ABC123/", "/explore/tags/foo/"]
|
||||
self.assertEqual(scrape.extract_shortcodes(hrefs), [])
|
||||
|
||||
def test_empty_input(self):
|
||||
self.assertEqual(scrape.extract_shortcodes([]), [])
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main(verbosity=2)
|
||||
@@ -0,0 +1,160 @@
|
||||
"""
|
||||
Tests for the two guards that stop a dead Instagram session going unnoticed:
|
||||
|
||||
check-session.py preflight: is there a usable `sessionid` in the Chrome cookie DB?
|
||||
stories-streak.py postflight: did every stories source come back empty for days running?
|
||||
|
||||
Both are run as subprocesses (the way gdl-cron.sh runs them); neither touches the network.
|
||||
|
||||
python3 -I -m unittest scripts/test_session_guards.py
|
||||
"""
|
||||
import datetime as dt
|
||||
import os
|
||||
import sqlite3
|
||||
import subprocess
|
||||
import sys
|
||||
import tempfile
|
||||
import time
|
||||
import unittest
|
||||
from pathlib import Path
|
||||
|
||||
HERE = Path(__file__).resolve().parent
|
||||
CHROME_EPOCH = 11644473600 # seconds between 1601-01-01 and 1970-01-01
|
||||
|
||||
|
||||
def run(script, *args, env=None):
|
||||
e = {**os.environ, **(env or {})}
|
||||
return subprocess.run([sys.executable, str(HERE / script), *args],
|
||||
capture_output=True, text=True, env=e)
|
||||
|
||||
|
||||
class CheckSession(unittest.TestCase):
|
||||
def profile(self, rows):
|
||||
"""A fake Chrome profile whose cookie DB holds `rows` = [(host, name, expires_unix|None)]."""
|
||||
tmp = tempfile.TemporaryDirectory()
|
||||
self.addCleanup(tmp.cleanup)
|
||||
d = Path(tmp.name) / "Default"
|
||||
d.mkdir()
|
||||
con = sqlite3.connect(d / "Cookies")
|
||||
con.execute("CREATE TABLE cookies (host_key TEXT, name TEXT, expires_utc INTEGER, has_expires INTEGER)")
|
||||
for host, name, exp in rows:
|
||||
con.execute("INSERT INTO cookies VALUES (?,?,?,?)",
|
||||
(host, name, 0 if exp is None else (exp + CHROME_EPOCH) * 1_000_000,
|
||||
0 if exp is None else 1))
|
||||
con.commit()
|
||||
con.close()
|
||||
return tmp.name
|
||||
|
||||
def check(self, rows):
|
||||
return run("check-session.py", env={"GDL_COOKIE_PROFILE": self.profile(rows)})
|
||||
|
||||
def test_valid_sessionid_passes(self):
|
||||
r = self.check([(".instagram.com", "sessionid", time.time() + 200 * 86400)])
|
||||
self.assertEqual(r.returncode, 0, r.stdout)
|
||||
self.assertIn("session ok", r.stdout)
|
||||
self.assertNotIn("WARNING", r.stdout)
|
||||
|
||||
def test_missing_sessionid_fails_even_with_other_instagram_cookies(self):
|
||||
# What the real failure looked like: csrftoken/ds_user_id/mid linger after logout.
|
||||
r = self.check([(".instagram.com", "csrftoken", time.time() + 86400),
|
||||
(".instagram.com", "ds_user_id", time.time() + 86400),
|
||||
(".instagram.com", "mid", time.time() + 86400)])
|
||||
self.assertEqual(r.returncode, 1)
|
||||
self.assertIn("NOT LOGGED IN", r.stdout)
|
||||
|
||||
def test_expired_sessionid_fails(self):
|
||||
r = self.check([(".instagram.com", "sessionid", time.time() - 86400)])
|
||||
self.assertEqual(r.returncode, 1)
|
||||
self.assertIn("expired", r.stdout)
|
||||
|
||||
def test_near_expiry_warns_but_passes(self):
|
||||
r = self.check([(".instagram.com", "sessionid", time.time() + 3 * 86400)])
|
||||
self.assertEqual(r.returncode, 0)
|
||||
self.assertIn("WARNING", r.stdout)
|
||||
|
||||
def test_session_cookie_without_expiry_passes(self):
|
||||
r = self.check([(".instagram.com", "sessionid", None)])
|
||||
self.assertEqual(r.returncode, 0)
|
||||
|
||||
def test_other_sites_sessionid_does_not_count(self):
|
||||
r = self.check([(".example.com", "sessionid", time.time() + 86400)])
|
||||
self.assertEqual(r.returncode, 1)
|
||||
|
||||
def test_missing_profile_fails_loudly(self):
|
||||
r = run("check-session.py", env={"GDL_COOKIE_PROFILE": "/nonexistent/profile"})
|
||||
self.assertEqual(r.returncode, 1)
|
||||
self.assertIn("no cookie DB", r.stdout)
|
||||
|
||||
def test_never_prints_cookie_values(self):
|
||||
# The fake DB has no value column at all; the script must not need one.
|
||||
r = self.check([(".instagram.com", "sessionid", time.time() + 90 * 86400)])
|
||||
self.assertNotIn("sessionid=", r.stdout + r.stderr)
|
||||
|
||||
|
||||
class StoriesStreak(unittest.TestCase):
|
||||
def setUp(self):
|
||||
tmp = tempfile.TemporaryDirectory()
|
||||
self.addCleanup(tmp.cleanup)
|
||||
self.dir = Path(tmp.name)
|
||||
self.state = self.dir / "streak"
|
||||
self.today = dt.date.today().isoformat()
|
||||
self.yesterday = (dt.date.today() - dt.timedelta(days=1)).isoformat()
|
||||
|
||||
def out(self, to_sync, empty, skipped=0):
|
||||
lines = [f"sources : {to_sync} to sync, {skipped} skipped"]
|
||||
lines += [f"[instagram][info] No results for https://www.instagram.com/stories/user{i}/"
|
||||
for i in range(empty)]
|
||||
p = self.dir / "run.out"
|
||||
p.write_text("\n".join(lines) + "\n")
|
||||
return p
|
||||
|
||||
def streak(self, to_sync, empty, skipped=0):
|
||||
return run("stories-streak.py", str(self.out(to_sync, empty, skipped)), str(self.state))
|
||||
|
||||
def stored(self):
|
||||
return self.state.read_text().split()
|
||||
|
||||
def test_all_empty_for_three_days_fails(self):
|
||||
self.state.write_text(f"{self.yesterday} 2\n")
|
||||
r = self.streak(6, 6)
|
||||
self.assertEqual(r.returncode, 1)
|
||||
self.assertIn("SESSION LIKELY DEAD", r.stdout)
|
||||
self.assertEqual(self.stored(), [self.today, "3"])
|
||||
|
||||
def test_first_empty_day_is_quiet(self):
|
||||
r = self.streak(6, 6)
|
||||
self.assertEqual(r.returncode, 0)
|
||||
self.assertEqual(self.stored(), [self.today, "1"])
|
||||
|
||||
def test_same_day_rerun_does_not_inflate_the_streak(self):
|
||||
self.state.write_text(f"{self.today} 2\n")
|
||||
r = self.streak(6, 6)
|
||||
self.assertEqual(r.returncode, 0)
|
||||
self.assertEqual(self.stored(), [self.today, "2"])
|
||||
|
||||
def test_any_story_resets_the_streak(self):
|
||||
self.state.write_text(f"{self.yesterday} 2\n")
|
||||
r = self.streak(6, 5) # one profile had a story
|
||||
self.assertEqual(r.returncode, 0)
|
||||
self.assertEqual(self.stored(), [self.today, "0"])
|
||||
|
||||
def test_run_that_fetched_nothing_leaves_state_alone(self):
|
||||
self.state.write_text(f"{self.yesterday} 2\n")
|
||||
r = self.streak(0, 0, skipped=6)
|
||||
self.assertEqual(r.returncode, 0)
|
||||
self.assertEqual(self.stored(), [self.yesterday, "2"])
|
||||
|
||||
def test_threshold_is_configurable(self):
|
||||
r = run("stories-streak.py", str(self.out(6, 6)), str(self.state),
|
||||
env={"GDL_EMPTY_STREAK_FAIL": "1"})
|
||||
self.assertEqual(r.returncode, 1)
|
||||
|
||||
def test_corrupt_state_file_is_treated_as_no_history(self):
|
||||
self.state.write_text("garbage\n")
|
||||
r = self.streak(6, 6)
|
||||
self.assertEqual(r.returncode, 0)
|
||||
self.assertEqual(self.stored(), [self.today, "1"])
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
Reference in New Issue
Block a user