tooling: scheduled profiles/full-sweep list posts+reels by browser (chrome-sync per profile/tab, early stop for weekly); GDL_LISTING=api fallback; reels-tab sweep note in MANUAL-RUN
This commit is contained in:
+10
-5
@@ -54,16 +54,21 @@ for p in withaseul cher_ryppo zindoriyam; do ~/gdl/posts-sync.sh $p; done
|
||||
minutes, mostly the 12–20 s gap between fetches.
|
||||
- It prints `no new posts for <user>` when a profile is caught up.
|
||||
- Logs: `~/gdl/logs/posts-<user>-*.log`.
|
||||
- Reels also show up in the main grid. `~/gdl/run-reels-all.sh` sweeps the separate Reels tab
|
||||
too (~15 min) and is rarely needed.
|
||||
- **Then sweep the Reels tab too:** `~/gdl/run-reels-all.sh` (~15 min). Most reels never
|
||||
appear in the main grid. On 2026-10-23 only 3 of heejin's 114 reels did, so `posts-sync.sh`
|
||||
alone misses them. `reels-sync.sh <user>` does a single profile.
|
||||
|
||||
**Once a gallery-dl release fixes the listing**, one command covers posts, reels, stories and
|
||||
highlights for all six profiles:
|
||||
**Or the whole thing in one command**, the same run the weekly timer does: gallery-dl for
|
||||
stories and highlights, then the browser for every profile's posts grid and Reels tab, each
|
||||
stopping once it reaches archived content:
|
||||
|
||||
```sh
|
||||
~/gdl/run-profiles.sh # ~5 min
|
||||
~/gdl/run-profiles.sh
|
||||
```
|
||||
|
||||
Once a gallery-dl release fixes the listing, `GDL_LISTING=api ~/gdl/run-profiles.sh` does posts
|
||||
and reels through gallery-dl again (~5 min).
|
||||
|
||||
For long runs, detach:
|
||||
`setsid nohup ~/gdl/run-profiles.sh >/dev/null 2>&1 &`, then
|
||||
`tail -f ~/gdl/logs/$(ls -1t ~/gdl/logs | head -1)`.
|
||||
|
||||
+14
-8
@@ -327,9 +327,15 @@ newest-first, so after 2 scroll rounds in a row that turned up nothing unarchive
|
||||
(`--max-known-rounds`, round 1 exempt because pinned posts sit on top) there is nothing newer to
|
||||
find. First run, `heejin`: 3 scrolls, 36 posts seen, 30 already held, 6 new.
|
||||
|
||||
Same caveats as reels: it drives the signed-in Chrome on `:9222`, so it is by hand, not on a
|
||||
timer. Once a gallery-dl release fixes the listing, `run-profiles.sh` covers posts again and
|
||||
this goes back to being a fallback.
|
||||
**Scheduled since 2026-10-23.** `gdl-cron.sh profiles` and `full-sweep` now list posts *and*
|
||||
reels this way: gallery-dl only does `--only stories,highlights`, then `chrome-sync.sh` runs per
|
||||
profile for the `posts` grid and the `reels` tab (both tabs needed: on 10-23 only 3 of heejin's
|
||||
114 reels were in the main grid). `profiles` uses the early stop (2 known rounds) on both tabs,
|
||||
and `full-sweep` scrolls every grid to the bottom. A failed browser step fails the run, which
|
||||
shows on Healthchecks. This reverses the earlier "by hand only" decision on purpose: the API
|
||||
listing is broken and the owner chose the browser. The trade-off is that unattended runs now
|
||||
need Chrome up on `:9222` with a live session, and they open and close their own tabs in it.
|
||||
`GDL_LISTING=api` restores the all-gallery-dl run once a release fixes gh#9829.
|
||||
|
||||
**Deploy scripts atomically, never in place.** bash reads a script as it runs, so overwriting
|
||||
`gdl-cron.sh` under a running sync makes bash resume at the same byte offset in the *new* text.
|
||||
@@ -836,11 +842,11 @@ after — left behind for inspection, same as `gdl-cron.sh`'s `staging-*`.
|
||||
Verified end to end against `zindoriyam` on 2026-08-27: 26 reels found on the
|
||||
page, 10 new ones fetched and published cleanly on the first run.
|
||||
|
||||
**Not wired into `gdl-cron.sh` or the timers.** Both scripts drive your
|
||||
actual browser session rather than firing a background request, and are
|
||||
slower by design (real scrolling, not an API call) — both good reasons not
|
||||
to run this unattended without deciding that deliberately. Today it is a
|
||||
per-profile, by-hand tool only.
|
||||
**Not wired into `gdl-cron.sh` or the timers** (as of 2026-08; **superseded 2026-10-23**: the
|
||||
weekly and quarterly runs now use the browser listing, see "Posts grid by browser" above). Both
|
||||
scripts drive your actual browser session rather than firing a background request, and are
|
||||
slower by design (real scrolling, not an API call), both good reasons not to run this
|
||||
unattended without deciding that deliberately.
|
||||
|
||||
For all six ARTMS profiles in one go, `~/gdl/run-reels-all.sh` just calls
|
||||
`reels-sync.sh` once per profile in `artms_account_links.txt`, same
|
||||
|
||||
@@ -9,7 +9,8 @@
|
||||
# Usage: GDL_TAB=reels|posts ./chrome-sync.sh <username-or-profile-url>
|
||||
# Normally called through reels-sync.sh / posts-sync.sh, which set GDL_TAB.
|
||||
#
|
||||
# Deliberately NOT wired into gdl-cron.sh or the timers -- see TOOLING.md.
|
||||
# Also run by gdl-cron.sh (profiles, full-sweep) once per profile and tab since
|
||||
# 2026-10-23, with GDL_MAX_KNOWN_ROUNDS set -- see TOOLING.md "Posts grid by browser".
|
||||
# Exits non-zero if either step does. Everything is logged.
|
||||
set -eu
|
||||
|
||||
@@ -29,6 +30,8 @@ SLEEP="${GDL_SLEEP:-5 10}"
|
||||
RATE="${GDL_RATE:-500K}"
|
||||
SCROLL_PAUSE="${GDL_SCROLL_PAUSE:-2.0 3.5}"
|
||||
MAX_IDLE_ROUNDS="${GDL_MAX_IDLE_ROUNDS:-3}"
|
||||
# Empty = instagram-chrome-scrape.py's own default (2 for posts, 0 = scroll to the end for reels).
|
||||
MAX_KNOWN_ROUNDS="${GDL_MAX_KNOWN_ROUNDS:-}"
|
||||
|
||||
PATH="$HOME/.local/bin:$PATH"; export PATH
|
||||
|
||||
@@ -72,6 +75,7 @@ set +e
|
||||
--index "$INDEX" \
|
||||
--scroll-pause $SCROLL_PAUSE \
|
||||
--max-idle-rounds "$MAX_IDLE_ROUNDS" \
|
||||
${MAX_KNOWN_ROUNDS:+--max-known-rounds "$MAX_KNOWN_ROUNDS"} \
|
||||
--out "$URLS_FILE" 2>&1 | tee -a "$LOG"
|
||||
scrape_status=${PIPESTATUS[0]}
|
||||
set -e
|
||||
|
||||
+63
-3
@@ -10,6 +10,11 @@
|
||||
# EDITED after we archived them, and by far the most
|
||||
# expensive thing here -- see TOOLING.md before running.
|
||||
#
|
||||
# Posts and reels are LISTED by scrolling the signed-in Chrome (chrome-sync.sh,
|
||||
# per profile and tab) rather than by gallery-dl's listing API, which broke on
|
||||
# 2026-10-22 (gh#9829) while single-post fetches kept working. gallery-dl still
|
||||
# does stories and highlights. GDL_LISTING=api restores the old all-API run.
|
||||
#
|
||||
# Exits non-zero if the sync does, so cron mails you. Everything is logged.
|
||||
set -eu
|
||||
|
||||
@@ -41,6 +46,15 @@ SLEEP_REQUEST="${GDL_SLEEP_REQUEST:-12 20}"
|
||||
SLEEP="${GDL_SLEEP:-5 10}"
|
||||
RATE="${GDL_RATE:-500K}"
|
||||
|
||||
LISTING="${GDL_LISTING:-browser}"
|
||||
case "$LISTING" in
|
||||
browser|api) ;;
|
||||
*) echo "GDL_LISTING must be browser or api, not '$LISTING'" >&2; exit 2 ;;
|
||||
esac
|
||||
GDL_PYTHON="${GDL_PYTHON:-$HOME/.local/share/pipx/venvs/gallery-dl/bin/python3}"
|
||||
# Set per mode below when posts/reels go through the browser: empty = no browser step.
|
||||
BROWSER_KNOWN_ROUNDS=""
|
||||
|
||||
case "$MODE" in
|
||||
# --min-interval 8, not the 20h default. The floor exists to stop an ABORTED
|
||||
# RESTART re-enumerating profiles -- a minutes-to-hours concern. At 20h the
|
||||
@@ -50,10 +64,23 @@ case "$MODE" in
|
||||
# request per profile, so the worst case 8h permits is roughly twelve
|
||||
# requests in a day instead of six.
|
||||
stories) ARGS="--only stories --min-interval 8" ;;
|
||||
profiles) ARGS="--only posts,reels,stories,highlights --abort 50" ;;
|
||||
# Browser listing: stop scrolling a grid after 2 rounds of nothing unarchived,
|
||||
# the browser's equivalent of --abort 50.
|
||||
profiles)
|
||||
if [ "$LISTING" = browser ]; then
|
||||
ARGS="--only stories,highlights --abort 50"; BROWSER_KNOWN_ROUNDS=2
|
||||
else
|
||||
ARGS="--only posts,reels,stories,highlights --abort 50"
|
||||
fi ;;
|
||||
# No --abort: the whole point of full-sweep is enumerating to the end, so it
|
||||
# is the only run that notices carousels edited after we archived them.
|
||||
full-sweep) ARGS="--only posts,reels,stories,highlights" ;;
|
||||
# is the only run that notices carousels edited after we archived them. The
|
||||
# browser equivalent is scrolling every grid to the bottom (0 = no early stop).
|
||||
full-sweep)
|
||||
if [ "$LISTING" = browser ]; then
|
||||
ARGS="--only stories,highlights"; BROWSER_KNOWN_ROUNDS=0
|
||||
else
|
||||
ARGS="--only posts,reels,stories,highlights"
|
||||
fi ;;
|
||||
# Renamed 2026-08-22: "full" was misleading (it is the abort-LIMITED run) and
|
||||
# "sweep" did not say it was the exhaustive one. Catch the old names rather
|
||||
# than failing with a bare error, in case something still passes them.
|
||||
@@ -125,6 +152,39 @@ set +e
|
||||
status=${PIPESTATUS[0]}
|
||||
set -e
|
||||
|
||||
# Posts and reels by browser: scrape each profile's grid and Reels tab, fetch only
|
||||
# the new shortcodes as single posts. chrome-sync.sh logs and publishes on its own;
|
||||
# its output is copied into this run's log so the Healthchecks body shows it too.
|
||||
if [ -n "$BROWSER_KNOWN_ROUNDS" ]; then
|
||||
mapfile -t PROFILES < <("$GDL_PYTHON" -c "
|
||||
import sys, importlib.util
|
||||
from pathlib import Path
|
||||
spec = importlib.util.spec_from_file_location('gdl_sync', Path('$GDL_HOME/gdl-sync.py'))
|
||||
gdl = importlib.util.module_from_spec(spec)
|
||||
sys.modules['gdl_sync'] = gdl
|
||||
spec.loader.exec_module(gdl)
|
||||
for p in gdl.read_urls_file(Path('$GDL_HOME/artms_account_links.txt')):
|
||||
print(p)
|
||||
")
|
||||
browser_failed=0
|
||||
for p in "${PROFILES[@]}"; do
|
||||
for tab in posts reels; do
|
||||
echo "==> browser: $p / $tab" | tee -a "$LOG"
|
||||
set +e
|
||||
GDL_TAB=$tab GDL_MAX_KNOWN_ROUNDS=$BROWSER_KNOWN_ROUNDS \
|
||||
"$GDL_HOME/chrome-sync.sh" "$p" 2>&1 | tee -a "$LOG" >/dev/null
|
||||
s=${PIPESTATUS[0]}
|
||||
set -e
|
||||
if [ "$s" -ne 0 ]; then
|
||||
echo " browser $p / $tab FAILED (exit $s)" | tee -a "$LOG" >&2
|
||||
browser_failed=$((browser_failed + 1))
|
||||
fi
|
||||
done
|
||||
done
|
||||
echo "browser listing: ${#PROFILES[@]} profile(s), $browser_failed step(s) failed" | tee -a "$LOG"
|
||||
[ "$browser_failed" -gt 0 ] && [ "$status" -eq 0 ] && status=1
|
||||
fi
|
||||
|
||||
# A stories run that skipped every source is NOT a success. It means the
|
||||
# min-interval floor blocked the one surface that cannot be backfilled, and
|
||||
# without this it looks identical to a clean run: exit 0, "0 step(s) failed".
|
||||
|
||||
@@ -5,9 +5,9 @@
|
||||
# version of running reels-sync.sh six times by hand, which is what the
|
||||
# 2026-09-03 run was before this script existed.
|
||||
#
|
||||
# Deliberately not wired into gdl-cron.sh or a timer -- same reasoning as
|
||||
# reels-sync.sh itself: it drives your actual browser session and is slower
|
||||
# by design, both good reasons to run it deliberately, not unattended.
|
||||
# A by-hand full sweep of every Reels tab (no early stop). The weekly
|
||||
# gdl-cron.sh profiles run lists reels through chrome-sync.sh itself, with an
|
||||
# early stop, since 2026-10-23 -- see TOOLING.md "Posts grid by browser".
|
||||
#
|
||||
# Usage: ./reels-all.sh
|
||||
# GDL_ARTMS_LINKS=/path/to/list ./reels-all.sh # override the profile list
|
||||
|
||||
Reference in New Issue
Block a user