Operator-flagged 2026-05-30: "the fail state of timeouts doesn't show
anything other than that the task timedout and was cleaned up. I can't
tell why it ran over or if it was stuck failed or there was just that
much to get."
The TimeoutExpired branch was returning a DownloadResult with no stdout,
no stderr, no files_downloaded, and a generic "Download timed out after
N seconds" message — even though subprocess.TimeoutExpired carries the
partial output gallery-dl emitted before being killed.
Now:
- Capture e.stdout / e.stderr (coerced str if bytes; "" if None).
- Count files_downloaded from partial stdout via _count_downloaded_files.
- Surface a tail-of-stderr hint in error_message so the UI summary tells
the operator at a glance whether it was "lots of content" (high count,
clean stderr), "stuck retrying" (any count, 429-spam stderr), or "hung
silent" (zero count, "no stderr output").
- Promote error_type to RATE_LIMITED when the partial stderr matches
RATE_LIMIT_PATTERNS — gallery-dl spinning on retries through the whole
900s window is the timeout-shaped tail of a real rate limit, and the
platform cooldown should kick in for the same reason.
Existing test_download_timeout strengthened to also assert empty-partial
case stays correctly TIMEOUT-classified with no preserved output.
New test_download_timeout_preserves_partial_output_and_classifies covers
the rich-partial-output → RATE_LIMITED promotion path.
DownloadEvent.metadata already flows stdout/stderr/run_stats from
DownloadResult via _phase3_persist — no UI change needed; the existing
DownloadDetailModal will surface the captured output automatically once
the build redeploys.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
select_due_sources returned rows in undefined order (Postgres-determined,
typically PK). At tick rates that outpace download-queue throughput, a
freshly-rerun source could keep getting re-queued ahead of one that's
still waiting for its first attempt this cycle. Operator-flagged
2026-05-30:
> if there are 8 hours before a source is due again and 40 full time
> downloads can happen in that period that means that there's a chance
> the first one to fire gets back into the download queue before item 41
> has a chance to get downloaded.
Added `ORDER BY last_checked_at ASC NULLS FIRST, id` to the due-source
SELECT. Never-checked sources go first, then longest-since-checked, then
ties broken by id. Combined with Celery's FIFO `download` queue, the
oldest-overdue source in each tick now reaches a worker before any
fresher one.
Test pins the ordering: a NULL-last_checked source, a 4-hour-overdue
source, and a 2-min-overdue source come back in that exact order from
select_due_sources.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The scan tick fired download_source.delay() for every due source without
grouping by platform; with multiple download workers, N due Patreon
sources could all hit Patreon's API in parallel and rate-limit each
other. Per-source consecutive_failures backoff REACTS to that (slows the
offender across cycles) but didn't PREVENT the first-tick burst.
When DownloadService._update_source_health sees a source error
classified as ErrorType.RATE_LIMITED, it now stamps an AppSetting row
`platform_cooldown:<platform>` with the cooldown expiry (now + 15 min,
PLATFORM_RATE_LIMIT_COOLDOWN_SECONDS). select_due_sources queries every
platform_cooldown:* key at the start of each tick and excludes every
source whose platform is in active cooldown. scheduler_status surfaces
active cooldowns as platform_cooldowns: {platform: expires_iso} so the
TopNav pipeline chip / activity summary can display them.
INSERT...ON CONFLICT DO UPDATE for the upsert so two workers racing
RATE_LIMITED responses on the same platform don't let one's
IntegrityError roll back the other's event-finalize transaction
(stranding the event for the recovery sweep). Atomic at the SQL level.
Tests cover: select_due_sources skips a platform in cooldown; other
platforms unaffected during single-platform cooldown; expired cooldown
rows don't filter; set_platform_cooldown is upsert-safe under repeated
calls.
Operator-flagged 2026-05-30 ("running multiple workers I don't know how
we'd keep the downloader from hitting a rate limit on a source").
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
SourceConfig.timeout defaulted to 3600s (1 hour), but download_source's
Celery task has soft_time_limit=900s and hard time_limit=1200s. So
gallery-dl never hit its own subprocess.run timeout — Celery always
killed it first. The hard SIGKILL leaves no terminal flip on the
DownloadEvent, which then sat pending/running until the recovery sweep
flipped it to error at 30 min from start. From the operator's seat that
was a ~30–40 min "hang" on every retry of a broken source.
Pinning the default to 900s (matching Celery's soft_time_limit) lets
subprocess.run raise TimeoutExpired cleanly inside Celery's window, and
the existing `except subprocess.TimeoutExpired` branch
(gallery_dl.py:635) captures it as a clean error_type='timeout' with a
real message. The DownloadEvent flips to error in ~15 min instead of
waiting on the recovery sweep at 30.
Per-source bumps still live in source.config_overrides for legitimately
long first-syncs. The new _DEFAULT_GDL_TIMEOUT_SECONDS constant carries
the rationale and the Celery-soft-limit dependency in its comment.
Operator-flagged 2026-05-30 on 59-source strand pile retries.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Provenance "View post" deep-links to /posts?post_id=X, which now opens the
feed centered on that post with infinite load in BOTH directions.
Backend: PostFeedService.scroll gains a direction (older|newer); new
around(post_id) returns a window of newer + the post + older with a cursor
for each end. /api/posts accepts ?around= and ?direction=. + API tests.
Frontend: posts store gains loadAround/loadOlder/loadNewer (older appends,
newer prepends) with per-end cursors; PostsView's anchored mode scrolls to
the post, observes top + bottom sentinels, and preserves scroll position on
upward prepend so the page doesn't jump. Normal feed mode unchanged.
Closes the remaining half of the post-navigation work.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
_find_or_create_source, _source_for_sidecar, and _find_or_create_post each
repeated the SELECT → savepoint-INSERT → on-IntegrityError rollback+re-SELECT
pattern. Extracted _get_or_create(stmt, factory): the statement is reused for
the scalar_one_or_none lookup and the scalar_one post-conflict re-fetch, so
all three are reproduced exactly. Centralizing the race-safe pattern in one
place also reduces the risk of the copies drifting (the bug class banked
2026-05-26).
Left _upsert_artist (no savepoint by design) and the ImageProvenance void
ensure-exists block (no return / no re-select) alone — they don't fit.
The rest of the ingest pipeline was already DRY: sidecar parsing lives in
utils/sidecar.py, per-platform quirks in the platforms package, and
_safe_ext/_categorize_error/_build_config are each single-instance.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The `select(ImportSettings).where(id == 1)).scalar_one()` singleton load was
repeated 15× across services, API, and 5 task modules. Added async load() +
sync load_sync() classmethods on the model and migrated all 15 full-row sites
(callers already imported ImportSettings, so no new imports; dropped download's
now-orphaned select import). Left maintenance.py's deliberate column-select
(import_scan_path only) as-is.
Rest of the service layer was already adequately DRY — the Record/to_dict
pattern is only 2 instances and the savepoint find-or-create recovery is
correctly per-entity, so neither was forced into a shared abstraction.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
D1 scheduler visibility: AppSetting last-tick stamp on every Beat tick +
GET /api/sources/schedule-status (last_tick_at/next_due_at/due_now/auto_sources)
+ SchedulerStatusBar on the Subscriptions tab (re-polled every 30s).
D2 failing-source rollup: ?failing=true on the sources list + FailingSourcesCard
on Downloads with per-source and bulk "retry" (re-runs the feed via /check).
D3 activity sparkline: GET /api/downloads/activity hourly buckets + CSS bar
chart by the stat chips (failures stacked in error color); refreshes on live poll.
D4 credential staleness: surface last_verified age + "re-verify recommended"
warning past 30d; also fixes the dead last_verified_at field-name mismatch so
the verification row renders at all.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Operator-flagged 2026-05-28, two asks.
**1. Credential Verify (was missing vs GS — and now actually verifies).**
GS's Verify was a stub (`TODO: implement actual verification` — just
stamped last_verified). FC does a real check, which matters given the
recent auth pain (subscribestar age cookie, HF host-only PHPSESSID):
- GalleryDLService.verify(url, platform, cookies_path, auth_token) runs
gallery-dl in `--simulate --range 1-1` mode (no download) against the
URL with the materialized credentials, then reuses _categorize_error:
returncode 0 / NO_NEW_CONTENT → valid; AUTH_ERROR → invalid; other →
inconclusive (reason surfaced). 45s timeout.
- POST /api/credentials/<platform>/verify picks an enabled Source for
the platform to probe, runs verify, and on success stamps
credential.last_verified (new CredentialService.mark_verified).
Returns {valid: bool|null, reason, last_verified?}. valid=null means
untestable (no credential, or no enabled source to point at).
- CredentialCard gains a Verify button (on credentialed cards) + a
result chip (Verified ✓ / Failed / Untestable) and a toast with the
reason. SettingsTab reloads on @verified so last_verified refreshes.
**2. Live download-activity feedback.** The Downloads tab was static —
no way to tell if downloads were succeeding without manually hitting
Refresh. It now auto-polls: stats every 4s, and the event list too
while anything is queued/running. Polling pauses when the tab is
backgrounded (document.hidden) and the list reload is skipped on idle
ticks to stay light. A pulsing "● live" indicator next to the stat
chips shows when auto-refresh is active (queued+running > 0); honors
prefers-reduced-motion.
Tests: verify endpoint — untestable with no credential, untestable with
no enabled source, valid+stamped on success (gallery-dl mocked), and
auth-failure reported without stamping.
Operator-confirmed 2026-05-28: the BlenderKnight:* tags are `archive`
kind (caught by the kind purge), but the `source:patreon`-style tags
are IR's old `source` kind that fell back to `general` during migration
(FC's enum has no `source` kind) — so they can't be matched by kind.
Broadened the purge to a two-rule match and renamed it for accuracy
(all dev-only, unreleased):
- cleanup_service.purge_tags_by_kind → purge_legacy_tags. Predicate is
now `kind IN (archive, post, artist) OR name LIKE 'source:%'`
(LEGACY_NAME_PREFIXES). Preview classifies each row into by_kind OR
by_prefix (source:* counts once under the prefix bucket regardless of
its general kind).
- endpoint /tags/purge-retired-kinds → /tags/purge-legacy. dry-run
returns by_kind + by_prefix + count + sample.
- store purgeRetiredKindTags → purgeLegacyTags.
- Tag Maintenance card copy + breakdown updated to show both buckets;
button reads "Preview/Delete legacy tags".
Tests updated + extended: dry-run reports by_kind {archive,post,artist}
AND by_prefix {source:*}, plain general/character tags survive; commit
deletes both the kind-matched and source:*-matched rows and leaves the
rest.
Operator-flagged 2026-05-28, two asks.
**1. Provenance posts ate the panel.** Each post card rendered its full
description (180px scroll box) inline, so a few posts pushed everything
else off-screen. ProvenancePanel now collapses the description by
default behind a per-post "Show description ▾ / Hide ▴" toggle (state
keyed by provenance_id, reset when the viewed image changes). Cards stay
compact — platform/date/title/meta/actions — and the operator expands
only the descriptions they want.
**2. Purge tags of retired/system kinds.** The IR migration left
`archive`/`post`/`artist`-kind tags (e.g. `BlenderKnight:Hannah_BJ_Loops`)
that FC no longer creates — the tag input only makes
character/fandom/series/general, and provenance + artists are their own
systems now. (meta/rating were already hard-deleted by alembic 0023.)
- cleanup_service.purge_tags_by_kind(kinds=PURGEABLE_TAG_KINDS) — counts
(dry_run) or deletes tags whose kind ∈ (archive, post, artist).
CASCADE clears the image_tag / alias / allowlist / etc. rows.
- POST /api/admin/tags/purge-retired-kinds (Tier-A, dry-run preview
returns per-kind counts + sample names — the preview IS the
verification of exactly what'll be deleted before committing).
- Tag Maintenance card gets a second section: "Preview retired-kind
tags" → per-kind breakdown + sample → "Delete N retired-kind tag(s)".
Tests: dry-run counts by kind (general survives), commit deletes only
the retired kinds (general + character survive, retired count → 0).
NOTE: a dry-run preview will show exactly which kinds/counts are
present. If the operator's noisy tags turn out to be `general` (e.g. an
IR `source:patreon` that fell back to general during migration), they
won't be caught by the kind purge — the preview makes that visible so
we can decide on a name-based pass separately.
Layer 2 — remediate a corrupt file by re-fetching a fresh copy from its
source, bounded to a single attempt. Operator-requested 2026-05-28.
New backend/app/services/refetch_service.py:
- resolve_refetch_source: parse the failed file's sidecar → platform,
derive the artist from the import path, find an ENABLED Source with a
real feed URL for (artist, platform). Returns None for filesystem-only
imports, missing sidecars, or `sidecar:<platform>:<slug>` synthetic
anchors (not pollable).
- attempt_refetch: if not already refetched AND a Source resolves,
delete the corrupt file (so gallery-dl's skip_existing re-fetches it),
set ImportTask.refetched=True, and trigger ONE download_source
re-check. Bounded by `refetched` so source-side corruption can't loop.
Wiring:
- Manual endpoint POST /api/import/tasks/<id>/refetch (only on 'failed'
tasks). Returns refetch_queued / no_source / already_refetched /
not_found / not_failed.
- Auto path in recover_interrupted_tasks: for each poison-pill row, if
env FC_AUTO_REFETCH_CORRUPT=1, attempt_refetch (default OFF — the
manual button is the primary path; auto is opt-in since re-fetch
deletes a file + re-runs the downloader).
- Frontend: a cloud-refresh icon button on failed rows in ImportTaskList
→ stores.import.refetchTask → toast keyed on the result status.
Filesystem imports with no upstream return no_source — the operator's
only remediation there is replacing the file on disk, surfaced clearly
in the toast.
Tests: 404 unknown task, 400 non-failed task, no_source when
unresolvable, and the full resolvable-source path (file deleted,
refetched flag set, one download_source dispatched, second call is a
no-op). The resolvable test repoints the migration-seeded
import_settings(id=1) scan path rather than inserting a conflicting row.
Layer 3 — prevent the hard worker crash rather than just recovering from
it. The realistic process-crash vectors (operator's observed slow/heavy
tasks) are video decode and archive extraction; images decode in-process
and Pillow raises-and-skips cleanly, and a subprocess per image would
wreck deep-scan throughput, so images are intentionally not probed.
New backend/app/utils/safe_probe.py (leaf module, lazy heavy imports so
the spawned child stays light):
- probe_video(path): validates the container + first video stream via
ffprobe (a separate binary — a decoder crash kills only ffprobe, not
the worker). Returns width/height, which the importer didn't capture
for videos before. crashed=True only on ffprobe timeout.
- probe_archive(path): an uncompressed-size bomb guard
(MAX_ARCHIVE_UNCOMPRESSED_BYTES = 4 GiB) plus the format integrity
test (zipfile.testzip / rarfile.testrar / py7zr.test) run in a
spawned child process. A decompression-bomb OOM or native-lib
segfault on a malformed archive shows up as a non-zero child exit
code → crashed=True, never a dead worker.
ProbeResult.crashed distinguishes a HARD failure (subprocess killed /
timed out — the poison-pill signature → caller returns terminal
'failed') from a CLEAN rejection (corrupt-but-handled, bomb cap,
integrity mismatch → caller's choice of skipped/attached).
Wired:
- importer._import_media video branch: probe_video before the pipeline;
crash → failed, clean reject → invalid_image skip, ok → capture dims.
- importer._import_archive: probe_archive before extract_archive; crash
→ failed, clean reject → still preserve the archive as a
PostAttachment (matches extract_archive's fail-soft contract).
- ml.tag_and_embed video branch: probe_video before sampling 10 frames,
so a corrupt video is rejected (status='bad_video') instead of
crashing the ml-worker on frame decode.
Tests (test_safe_probe.py): valid/corrupt zip via probe_archive, direct
_inspect_archive size+integrity, in-process _archive_probe_target bomb
guard (monkeypatch can't reach a spawned child, so the target is called
directly), and a non-video → ok=False that's robust to ffprobe presence
in CI.
Operator-requested 2026-05-27: centralize the per-platform quirks that
had been accumulating across credential_service, sidecar, and platforms
into a single per-platform module so adding/updating quirks becomes
"edit one file."
**Layout**
services/platforms/
base.py PlatformInfo dataclass + module-default key
chains + shared helpers (str_id_value, str_field)
__init__.py PLATFORMS dict + public API (auth_type_for,
known_platform_keys, to_dict,
external_post_id_keys_for, description_keys_for)
patreon.py metadata only — the reference platform, no quirks
subscribestar.py metadata + augment_cookies (18+ agreement) +
derive_post_url (synthetic /posts/<post_id>)
hentaifoundry.py metadata + augment_cookies (host-only PHPSESSID
duplicate) + derive_post_url (/pictures/user/...)
pixiv.py metadata + derive_post_url (/artworks/<id>)
discord.py metadata + derive_post_url
(channels/<server>/<channel>/<message>)
deviantart.py metadata only — un-audited; quirks to be added
when an operator first exercises DA
**PlatformInfo extensions**
Existing fields preserved. Four new optional fields:
external_post_id_keys: tuple[str, ...] | None
Override the sidecar external_post_id lookup chain. None falls
back to DEFAULT_EXTERNAL_POST_ID_KEYS in base.py
("post_id", "id", "index", "message_id") — covers every current
platform.
description_keys: tuple[str, ...] | None
Override the description body lookup chain. None falls back to
DEFAULT_DESCRIPTION_KEYS ("content", "description", "caption",
"message") — Discord's "message" body field is covered by the
default's trailing entry.
derive_post_url: Callable[[dict], str | None] | None
Synthesize the post permalink from sidecar metadata. None = trust
the bare `url` / `post_url` field (patreon, deviantart).
subscribestar/pixiv/hf/discord override this because their `url`
is the file CDN URL.
augment_cookies: Callable[[str], str] | None
Post-process the materialized cookies.txt before gallery-dl
consumes it. None = no-op. Used by subscribestar (age cookie) and
hentaifoundry (host-only PHPSESSID duplicate).
**Consumer changes**
- credential_service._augment_cookies(platform, netscape) shrunk from a
per-platform-conditional dispatcher (~80 lines of inlined helpers) to
a 5-line lookup: `info.augment_cookies(netscape) if info and
info.augment_cookies else netscape`. The platform-specific helper
bodies moved verbatim into the per-platform modules.
- sidecar.parse_sidecar similarly delegates: external_post_id chain via
external_post_id_keys_for(category), description chain via
description_keys_for(category), post_url via
PLATFORMS[category].derive_post_url. The _DERIVED_URL_PLATFORMS set
and inline _derive_post_url body both gone. Added a shared `_first_id`
helper for bool-safe id coercion.
**Public API preserved**
PLATFORMS, PlatformInfo, auth_type_for, known_platform_keys, to_dict
are all re-exported from the package's __init__.py. test_platforms_registry
test_credential_service, and test_sidecar_util pass without changes
because the behavior is identical; only the implementation moved.
**Adding a new platform**
1. Create services/platforms/<name>.py with `INFO = PlatformInfo(...)`
and any of the four optional hooks.
2. Import it in services/platforms/__init__.py + add to the PLATFORMS
tuple-comprehension.
3. Done. sidecar parsing, cookie materialization, /api/platforms all
pick it up automatically.
Operator-flagged 2026-05-27: HF source check 401'd on
`HEAD /?enterAgree=1` even with valid login cookies. Root cause is the
combination of (1) gallery-dl's HF extractor checking
`self.cookies.get("PHPSESSID", domain="www.hentai-foundry.com")` with
`requests`' EXACT domain matching, and (2) the extension's cookies.js
forcibly rewriting every captured cookie to a leading-dot subdomain-wide
form. HF's PHPSESSID is browser-stored as host-only on
`www.hentai-foundry.com`; the rewrite re-anchored it to
`.hentai-foundry.com`, which `cookies.get(...)` no longer matches even
though the cookie is still sent on actual HTTP requests (RFC 6265
subdomain rules). The extractor falls into its unauthenticated
`?enterAgree=1` fallback, which 401s (Cloudflare or HF's anti-bot HEAD
gating).
Two-part fix, no operator action required for existing stored cookies:
1. **Backend** (`credential_service._augment_cookies`) — refactored from
the subscribestar-only single function into a per-platform dispatcher.
New `_augment_hentaifoundry` parses the materialized netscape file
and, for each `.hentai-foundry.com` entry whose name is PHPSESSID or
YII_CSRF_TOKEN, appends a host-only duplicate
(`www.hentai-foundry.com\tFALSE\t...`). Originals preserved. Three
new tests pin: injection fires + originals preserved; idempotent
when host-only already exists; doesn't touch unrelated cookies
(e.g. `_ga`).
2. **Extension** (`cookies.js`) — `toNetscapeFormat` now respects
`c.hostOnly` from the browser instead of blindly forcing a
leading-dot subdomain-wide form. Host-only cookies are written with
the bare host + FALSE flag; non-host-only cookies retain the
leading-dot + TRUE form. Forward-compat — fresh captures from
v1.0.5+ no longer need the backend's host-only duplication.
Extension bumped 1.0.4 → 1.0.5; manifest + package.json in lockstep.
After deploy: the next HF source check on the operator's already-stored
cookies will succeed because the materialized cookies.txt now contains
host-only PHPSESSID. No browser re-export needed.
Operator-flagged 2026-05-27: subscribestar source check aborted with
`AbortExtraction: HTTP redirect to .../age_confirmation_warning`. The
captured `_personalization_id` cookie in the browser-stored file had
expired (annual rotation), and the user could not realistically refresh
it: SubscribeStar's frontend JS uses localStorage to suppress the
age-confirmation popup once dismissed, so a logged-in revisit doesn't
re-show the popup and the server-side cookie is never re-issued.
gallery-dl's own login flow (which FC doesn't exercise — cookies come
from the extension instead) sidesteps this by manually setting
`18_plus_agreement_generic=true` on `.subscribestar.adult`. The server
accepts that as the age-confirmation marker.
`credential_service._augment_cookies(platform, netscape)` mirrors that
behavior: when the materialized cookies file is for subscribestar and
the age cookie isn't already present, append a synthetic line for
`.subscribestar.adult` with name=`18_plus_agreement_generic` value=`true`
and a far-future expiry. No-op for other platforms; no-op if the cookie
is already present (idempotent for manual pastes / extension captures
that happen to include it).
Three new tests pin: (a) injection fires for subscribestar, preserves
existing cookies; (b) idempotent when already present (no double
injection); (c) does NOT fire for non-subscribestar platforms (Patreon
etc. don't get a foreign-domain cookie).
Not a curator handling bug per se — the extension faithfully captured
what the browser had. This is mirroring a documented gallery-dl
workaround so the cookies-via-extension auth path doesn't degrade as the
server-side cookie expires.
Operator-flagged 2026-05-26: Atole artist detail page showed 406 Sources where 1 was right.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
_apply_sidecar is additive: it find-or-creates Post/Source/ImageProvenance
and sets primary_post_id NULL-only, so any IR-migration provenance on the
existing row survives untouched and the new GS sidecar adds a second
ImageProvenance pointing at the freshly-created Post.
Wrapped in try/except so a malformed sidecar can't unwind the file-swap
commit — the file replacement is the critical operation.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Operator hit this 2026-05-25 after the IR tag_apply landed: ~57k images
all scanned into FC in the same week share image_record.created_at, so
the gallery timeline collapses them into a single month bucket and
scroll orders them all together at the top. Their actual publish dates
(spread over years) were already available in Post.post_date but the
gallery never read it.
Backend wire-up:
- tag_apply phase 4 now sets ImageRecord.primary_post_id when creating
ImageProvenance (only if currently NULL — preserves the canonical
download-time linkage set by the importer for new FC ingests).
- gallery_service.py introduces _effective_date_col() =
COALESCE(post.post_date, image_record.created_at), used in:
* scroll() ORDER BY + cursor WHERE clauses
* timeline() year/month group-by
* jump_cursor() year/month filter
* _neighbors() prev/next ordering
- Each method LEFT OUTER JOIN Post on primary_post_id so the COALESCE
works for images without a post (NULL on the Post side, fall back
to created_at).
- GalleryImage gains posted_at + effective_date fields; API /gallery
/scroll exposes both alongside the existing created_at so the UI
can render 'Posted on X (imported Y)' if desired.
- get_image_with_tags() returns posted_at for the modal.
Cursor format unchanged — the encoded datetime is now the effective_
date (whichever column won the COALESCE) and pagination remains
consistent.
To pick up new behavior for an already-migrated IR set: re-run
/api/migrate/tag_apply on the existing manifest (phase 4 is
idempotent; the new primary_post_id assignment backfills).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
_PLATFORM_PROFILE_URL had only patreon/subscribestar/hentaifoundry but
FC's extension_service.py recognizes 5 platforms. Any IR PostMetadata
with platform=deviantart or pixiv fell through _profile_url returning
None and the entry was silently skipped — explaining operator's
2026-05-25 finding that IR-migrated images had tags but no provenance
for the deviantart + pixiv subscriptions.
Pixiv caveat noted in comment: real profile URL takes numeric user_id
(https://www.pixiv.net/users/12345) but IR's PostMetadata.artist
stores display name. We slug the name and use it as if it were the id
so the artist->post->image linkage survives migration; the resulting
Source.url won't resolve in a browser and operator can fix via
Settings -> Subscriptions later if they want.
To recover existing IR-migrated state: re-run /api/migrate/tag_apply
on the existing manifest. Phase 4 is idempotent; new posts get
inserted only for the previously-skipped platforms.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Operator hit a corrupt JPEG in the IR set 2026-05-25: PIL.verify() only
validates header structure but doesn't catch truncated/broken pixel
data. The error surfaces later in _transparency_pct (via getchannel
'A' -> load) or compute_phash (load) — both blow up with OSError
'broken data stream when reading image file'. Celery's autoretry_for
then bounces the same file forever instead of marking it skipped.
Wrap both PIL.load-triggering call sites with try/except OSError ->
ImportResult(status=skipped, skip_reason=invalid_image).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The HF repo Camais03/camie-tagger-v2 has camie-tagger-v2.onnx (789 MB)
+ camie-tagger-v2-metadata.json (7.77 MB) at root, NOT model.onnx +
selected_tags.csv. Tags ship as nested JSON (dataset_info.tag_mapping)
not CSV. Per the published onnx_inference.py reference: input is NCHW
not NHWC, normalize with ImageNet mean/std, pad-square color (124,116,
104), sigmoid the second output (refined predictions) not the first.
Operator hit this during the IR migration ML backfill — download_models
silently fetched only 3 json files (allow_patterns matched nothing
useful), tagger.load() then raised RuntimeError. Fetched the actual
v2 layout via WebFetch, rewrote tagger to match published reference.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
bvandeusen