diff --git a/TOOLING.md b/TOOLING.md index 7a5a402..de07ccc 100644 --- a/TOOLING.md +++ b/TOOLING.md @@ -30,14 +30,22 @@ with no deploy or restart. ```sh ssh mattellite -~/gdl/gdl-cron.sh stories # ~1 min. DO THIS OFTEN - stories die in 24h -~/gdl/gdl-cron.sh profiles # ~5 min. posts, reels, highlights, stories +~/gdl/stories.sh # ~1 min. DO THIS OFTEN - stories die in 24h +~/gdl/profiles.sh # ~5 min. posts, highlights, stories (not reels -- see below) +~/gdl/reels-all.sh # ~15-20 min. reels, all six profiles, by scrolling +~/gdl/full-sweep.sh # rarely, ~420 requests. Read its own comment first. ``` -`profiles` covers everything, so it is the one to run if you only run one — but -it is no substitute for `stories`, because a story posted and expired between -two `profiles` runs is simply gone. Both are safe to run back-to-back; the -`--min-interval` floor makes a repeat a no-op rather than a re-fetch. +Fixed-mode entry points around `gdl-cron.sh`/`reels-sync.sh`, so the mode +never has to be remembered or typed — each just runs its own thing with the +right flags already set. `profiles.sh` covers every surface gallery-dl's API +still reaches, so it is the one to run if you only run one, but it is no +substitute for `stories.sh` (a story posted and expired between two runs of +it is simply gone) or `reels-all.sh` (`/api/v1/clips/user/` is blocked -- +see "Reels: the API is blocked" below -- `profiles.sh` silently fails that +one surface every time, by design, until that changes). All four are safe to +run back-to-back; the `--min-interval` floor makes a repeat a no-op rather +than a re-fetch. Then check — new posts appear without a redeploy. The archive index re-reads a directory when its mtime changes. @@ -54,9 +62,10 @@ redeploy. The archive index re-reads a directory when its mtime changes. - `400 Bad Request` on `/api/v1/feed/reels_media/` — the account is behind a scraping-warning interstitial; check the browser before anything else. -Long runs: `setsid nohup ~/gdl/gdl-cron.sh profiles >/dev/null 2>&1 &` and -`tail -f ~/gdl/logs/$(ls -1t ~/gdl/logs | head -1)`. Nothing reaches the -archive until a run finishes, so killing one midway is safe. +Long runs: `setsid nohup ~/gdl/profiles.sh >/dev/null 2>&1 &` (or `full-sweep.sh` +/ `reels-all.sh`) and `tail -f ~/gdl/logs/$(ls -1t ~/gdl/logs | head -1)`. +Nothing reaches the archive until a run finishes, so killing one midway is +safe. ## Safety net: a global gallery-dl config @@ -575,6 +584,12 @@ slower by design (real scrolling, not an API call) — both good reasons not to run this unattended without deciding that deliberately. Today it is a per-profile, by-hand tool only. +For all six ARTMS profiles in one go, `~/gdl/reels-all.sh` just calls +`reels-sync.sh` once per profile in `artms_account_links.txt`, same +reasoning as above — real scrolling, so still by hand. Run 2026-09-03: +`official_artms` alone has 609 real reels and took ~14 minutes of scrolling; +the whole six-profile run took about 15 minutes total, 2 new reels found. + ### Where a shared reel lands A reel found on one profile's `/reels/` page is not necessarily *owned* by diff --git a/scripts/full-sweep.sh b/scripts/full-sweep.sh new file mode 100755 index 0000000..5d29322 --- /dev/null +++ b/scripts/full-sweep.sh @@ -0,0 +1,8 @@ +#!/bin/bash +# Rarely: every surface, no --abort -- walks each profile to the end, ~420 +# requests. The only run that notices posts EDITED after they were archived +# (e.g. a caption added later -- see TOOLING.md "Reels: the API is blocked" +# for how that showed up), and by far the most expensive thing here. Read +# TOOLING.md before running this; prefer running it by hand, not on a timer. +set -eu +exec "$(dirname "$0")/gdl-cron.sh" full-sweep "$@" diff --git a/scripts/profiles.sh b/scripts/profiles.sh new file mode 100755 index 0000000..cb567ba --- /dev/null +++ b/scripts/profiles.sh @@ -0,0 +1,6 @@ +#!/bin/bash +# Routine run: every surface, --abort 50 (only gdl-cron.sh's "profiles" mode +# does this), ~5 min. Covers everything but is no substitute for stories.sh -- +# a story posted and expired between two runs of this is simply gone. +set -eu +exec "$(dirname "$0")/gdl-cron.sh" profiles "$@" diff --git a/scripts/reels-all.sh b/scripts/reels-all.sh new file mode 100755 index 0000000..662540e --- /dev/null +++ b/scripts/reels-all.sh @@ -0,0 +1,56 @@ +#!/bin/bash +# Run reels-sync.sh for every tracked ARTMS profile, one after another. +# TOOLING.md ("Reels: the API is blocked, scrape by scrolling instead") +# covers what this actually does and why it exists; this is just the loop +# version of running reels-sync.sh six times by hand, which is what the +# 2026-09-03 run was before this script existed. +# +# Deliberately not wired into gdl-cron.sh or a timer -- same reasoning as +# reels-sync.sh itself: it drives your actual browser session and is slower +# by design, both good reasons to run it deliberately, not unattended. +# +# Usage: ./reels-all.sh +# GDL_ARTMS_LINKS=/path/to/list ./reels-all.sh # override the profile list +set -eu + +GDL_HOME="${GDL_HOME:-$HOME/gdl}" +LINKS="${GDL_ARTMS_LINKS:-$GDL_HOME/artms_account_links.txt}" +GDL_PYTHON="${GDL_PYTHON:-$HOME/.local/share/pipx/venvs/gallery-dl/bin/python3}" + +if [ ! -f "$LINKS" ]; then + echo "profile list not found: $LINKS" >&2 + exit 2 +fi + +# Same parser gdl-sync.py uses for --urls-file (URLs or bare usernames, # +# comments, blank lines, duplicates dropped), reused so this never drifts +# from what a routine profiles.sh run actually tracks. +mapfile -t PROFILES < <("$GDL_PYTHON" -c " +import sys, importlib.util +from pathlib import Path +spec = importlib.util.spec_from_file_location('gdl_sync', Path('$GDL_HOME/gdl-sync.py')) +gdl = importlib.util.module_from_spec(spec) +sys.modules['gdl_sync'] = gdl +spec.loader.exec_module(gdl) +for p in gdl.read_urls_file(Path('$LINKS')): + print(p) +") + +if [ "${#PROFILES[@]}" -eq 0 ]; then + echo "no usable profiles in $LINKS" >&2 + exit 2 +fi + +echo "=== reels-all: ${#PROFILES[@]} profile(s) $(date -Is) ===" +status=0 +for p in "${PROFILES[@]}"; do + echo "=== starting $p ===" + if ! "$(dirname "$0")/reels-sync.sh" "$p"; then + status=1 + echo "=== $p FAILED ===" >&2 + fi + echo "=== finished $p ===" +done + +echo "=== reels-all: exit $status at $(date -Is) ===" +exit "$status" diff --git a/scripts/stories.sh b/scripts/stories.sh new file mode 100755 index 0000000..236b7d0 --- /dev/null +++ b/scripts/stories.sh @@ -0,0 +1,9 @@ +#!/bin/bash +# Daily run: stories only, ~1 min. The one surface that cannot be backfilled +# -- do this often. See TOOLING.md "Quick reference" for the full picture. +# +# Just a fixed-mode entry point so you don't have to remember gdl-cron.sh's +# mode argument; override pacing/paths the same way gdl-cron.sh does, with +# GDL_* env vars (see gdl-cron.sh's own comments for what's available). +set -eu +exec "$(dirname "$0")/gdl-cron.sh" stories "$@"