Operator-flagged 2026-05-27: subscribestar source check aborted with
`AbortExtraction: HTTP redirect to .../age_confirmation_warning`. The
captured `_personalization_id` cookie in the browser-stored file had
expired (annual rotation), and the user could not realistically refresh
it: SubscribeStar's frontend JS uses localStorage to suppress the
age-confirmation popup once dismissed, so a logged-in revisit doesn't
re-show the popup and the server-side cookie is never re-issued.
gallery-dl's own login flow (which FC doesn't exercise — cookies come
from the extension instead) sidesteps this by manually setting
`18_plus_agreement_generic=true` on `.subscribestar.adult`. The server
accepts that as the age-confirmation marker.
`credential_service._augment_cookies(platform, netscape)` mirrors that
behavior: when the materialized cookies file is for subscribestar and
the age cookie isn't already present, append a synthetic line for
`.subscribestar.adult` with name=`18_plus_agreement_generic` value=`true`
and a far-future expiry. No-op for other platforms; no-op if the cookie
is already present (idempotent for manual pastes / extension captures
that happen to include it).
Three new tests pin: (a) injection fires for subscribestar, preserves
existing cookies; (b) idempotent when already present (no double
injection); (c) does NOT fire for non-subscribestar platforms (Patreon
etc. don't get a foreign-domain cookie).
Not a curator handling bug per se — the extension faithfully captured
what the browser had. This is mirroring a documented gallery-dl
workaround so the cookies-via-extension auth path doesn't degrade as the
server-side cookie expires.
Operator-flagged: the claim that 'a future deep-scan via the new parser
will fix HF and Discord post_url' was conditional on the operator
actually running a deep-scan, which they might not do for ages. Until
then HF posts stay at post_url=NULL (HF sidecars have no `url` field)
and Discord posts stay pointing at cdn.discordapp.com/attachments/...
(the file URL, not the message permalink).
The migration was already opening sidecar files for SubscribeStar.
Generalizing the loop to also handle HF and Discord is a tiny addition
that closes the gap without operator intervention.
Per-platform Part 1 logic now:
subscribestar — read sidecar.post_id, overwrite external_post_id +
post_url with the derived /posts/<post_id> permalink.
hentaifoundry — read sidecar.user + .index, overwrite post_url with
/pictures/user/<u>/<i>. external_post_id (= index) unchanged.
discord — read sidecar.server_id + .channel_id + .message_id,
overwrite post_url with the discord.com/channels/.../<m> triple.
external_post_id (= message_id) unchanged.
Part 2 (SubscribeStar fragment merge) and Part 3 (pure-SQL Pixiv
post_url backfill) unchanged.
Posts whose related ImageRecord paths don't resolve on disk (orphan
filesystem state) are reported per-platform in the migration output —
those still need a future deep-scan, but the in-DB-with-on-disk-files
common case is now fully covered by the migration alone.
Audit of one sample sidecar per platform on the operator's
/mnt/Data/Patreon/ archive surfaced three parser bugs that have been
silently corrupting non-Patreon Posts since FC-3 shipped:
1. SubscribeStar `id` vs `post_id` confusion. gallery-dl puts the
per-attachment id in `id` (e.g. 711509) and the actual post id in
`post_id` (e.g. 360360). FC's external_post_id chain had `id`
winning, so every multi-image SubscribeStar post was fragmented into
N Post rows in the database. Reorder the chain to
`("post_id", "id", "index", "message_id")` — Patreon/Pixiv (no
`post_id`), HF (uses `index`), Discord (uses `message_id`) all
unaffected.
2. Discord `message` field not captured. Discord posts put the body in
`message`, not `content`. Append it to the description fallback chain
`("content", "description", "caption", "message")`.
3. post_url is the file URL on SubscribeStar/Pixiv/HF/Discord. New
`_derive_post_url(platform, data)` helper synthesizes proper
permalinks from per-platform fields:
subscribestar → https://www.subscribestar.com/posts/<post_id>
pixiv → https://www.pixiv.net/artworks/<id>
hentaifoundry → https://www.hentai-foundry.com/pictures/user/<user>/<index>
discord → https://discord.com/channels/<server>/<channel>/<message>
Patreon's bare `url` IS a real permalink and is used as-is. For the
four file-URL platforms, the bare `url` is NEVER trusted: derive or
return None rather than persist a CDN URL.
Tests:
- `test_parse_core_fields_and_id_priority` flipped to assert post_id
wins over id.
- New `test_parse_id_used_when_no_post_id` covers the Patreon real
shape.
- New `test_parse_message_used_as_description_fallback` covers Discord
bodies.
- Five new tests cover per-platform post_url derivation
(SubscribeStar/Pixiv/HF/Discord/Patreon-untouched + missing-fields →
None).
Cleanup migration alembic 0025_fix_subscribestar_post_ids:
- For each SubscribeStar Post: find a related ImageRecord.path, walk to
its sidecar JSON, read `post_id`, overwrite Post.external_post_id +
post_url with the corrected values.
- After all updates, every group of Posts under one source sharing the
same NEW external_post_id is a fragment-set — merge to a canonical
row using the same ImageProvenance pre-delete + repoint dance as
alembic 0022 (banked pattern).
- Pure-SQL backfill of Pixiv post_url: replace any `i.pximg.net`-shape
url with the derived `/artworks/<id>` permalink.
- HF and Discord post_url backfills skipped — HF would need the `user`
field (not stored on Post), Discord needs server/channel triple.
Both will be corrected by a deep-scan re-applying sidecars through
the new parser.
Idempotent: re-running on already-corrected data is a no-op.
SubscribeStar gallery-dl always writes `title: ""` and embeds the leading
sentence inside `content` HTML. Confirmed against the operator's
/mnt/Data/Patreon/Cheunart/subscribestar/ dump 2026-05-27: every
post's JSON has `title: ""` and a content like
`<div>Lets say hello to you guys with my Belle <br><br><br></div>`.
FC's sidecar parser, treating empty strings as missing, had been leaving
post_title NULL on every subscribestar post since FC-3 shipped.
Fix at two layers:
1. `backend/app/utils/sidecar.py` — new `_first_line_text(body, limit)`
helper strips HTML tags, collapses whitespace, returns the first
non-empty line truncated to 120 chars with ellipsis. `parse_sidecar`
now falls back to this when `title` resolves to None and a
`content`/`description`/`caption` value is present. Patreon's
non-empty titles short-circuit the fallback so existing behavior is
unchanged. Four new tests in test_sidecar_util.py pin: derivation
from content, truncation at 120 chars, explicit-title precedence,
no-content no-fallback.
2. `alembic 0024_backfill_post_title_from_description` — backfills the
same logic across existing Post rows where `post_title IS NULL OR
post_title = ''` AND description is present. Idempotent (re-running
is a no-op once titles are populated). Downgrade is a no-op since
there's no safe way to tell derived rows from genuine ones.
After deploy + migration: subscribestar posts will surface a meaningful
title in PostCard, post feed search, etc.
PostCard and PostModal competed for the same data and rendered redundant
chrome (header twice, image grid twice, attachment list twice). The wider
PostCard layout we shipped 2026-05-27 has enough real estate to be the
canonical post surface, so collapse the two into one.
Compact (default) state is unchanged: hero + 3-cell rail + truncated
title + 3/5-line description + attachment count badge. Whole-card click
expands in place. Expanded state shows: full title, mosaic of ALL post
images via PostImageGrid (uncapped, lazy-loaded via getPostFull), full
sanitized-HTML description with paragraph wrapping, attachments as
downloadable pill links. Click the chevron in the header to collapse;
mosaic image clicks open ImageViewer scoped to the post (modalStore's
postImageIds path is preserved — only the comment changed).
Per-card local state — no global modal store. Each PostCard owns its
own expanded ref and lazy-loaded detail; collapsing a card discards
neither (so re-expand is instant after the first fetch).
Deleted: PostModal.vue, postModal.js store. Removed the App.vue mount.
Cutting a release fires BOTH the push-to-main workflow AND the push-to-tag
workflow in parallel. main-push runs sign-extension (AMO round-trip 1-5min)
then publishes the ext-<version> Forgejo release; tag-push skips
sign-extension (gated to main) and races straight to build-web's Download
XPI step. Tag-push lost every time — got 404 from
releases/tags/ext-<version> before main-push had finished signing.
v26.05.27.0 hit this: tag-push build-web died on exit 22 because the
ext-1.0.4 release wasn't published yet (it arrived ~4min later).
Fix: wrap the release lookup in a 20-iteration sleep+retry loop, 30s
between attempts (10min total upper bound, generous for AMO). main-push's
signing eventually publishes the release; tag-push picks it up on a later
poll. No more manual rerun of the failed job after every release cut.
Banked the trap as reference_tag_push_main_push_race.md — same shape will
recur any time a tag-push workflow consumes a main-push-produced artifact.
Replaces the three top-level routes with a single `/subscriptions` parent
owning the whole download-pipeline domain. Internal tab state via `?tab=`
query param, mirroring ArtistView's pattern. TopNav auto-drops the two
removed entries (route-driven via meta.title). Bookmark-safe redirects
from `/credentials` and `/downloads` route into the appropriate subtab.
**Subtab 1 — Subscriptions (default).** Carries over the existing
artist-grouped expandable table; adds (a) status filter dropdown, (b)
bulk-select column with Enable/Disable/Delete-all actions, (c) GS-style
color-coded `PlatformChip` per distinct platform in the collapsed row.
Reuses SourceRow, SourceHealthDot, SourceFormDialog, ArtistCreateDialog.
**Subtab 2 — Downloads.** Full GS dashboard. Five colored stat chips up
top (Queued/Running/Completed/Failed/Skipped, sourced from new
`GET /api/downloads/stats?window_hours=`). Popover-style filter UI
(Status/Source/FromDate/ToDate) with active-filter pills below.
Maintenance menu wraps existing /api/import/retry-failed and
/api/import/clear-stuck endpoints; Export-failed-logs item disabled with
a "v2" tooltip. Per-row Retry preserved via existing DownloadEventRow.
**Subtab 3 — Settings.** Four sections: ExtensionKeyBar (top), GS-style
per-platform CredentialCard grid (md=6 v-row/v-col, dashed border if
unset / accent border if set, expandable how-to panel), Downloader card
(rate limit, validate_files), Schedule defaults card (default interval,
event retention, failure warning threshold). The Downloader and Schedule
sections were extracted out of components/settings/ImportFiltersForm.vue
— SettingsView's Import tab now owns only image-import filters.
**Backend:** new `GET /api/downloads/stats` returns
{pending, running, ok, error, skipped} count grouped by status over the
configurable window. Status keys stay raw from the ENUM; UI does the
display-label mapping. Two integration tests pin the response shape +
window_hours validation.
**Util:** `frontend/src/utils/platformColor.js` — single source of truth
for the six platforms' color + icon + label, mirroring GS's palette
(patreon=red mdi-patreon, subscribestar=amber mdi-star,
hentaifoundry=purple mdi-palette, discord=indigo mdi-discord,
pixiv=blue mdi-alpha-p-box, deviantart=green mdi-deviantart). Unknown
platform falls back to grey + mdi-web.
Deferred (explicit non-goals): subscription import/export, "Trigger Due
Now" scheduler-tick button (needs new backend endpoint), Export Failed
Logs CSV dump.
The two prefix-parsing tests were pinned to `artist:Eric`, but `artist`
was removed from KNOWN_KINDS in commit 4cad07a (provenance is a separate
axis from tags). The parser now keeps `artist:` literal, so the assertion
`body["name"] == "Eric"` failed.
Repointed to `character:Saber` (still in KNOWN_KINDS). Also updated the
stale `artist:` docstring example in parse_kind_prefix to `fandom:`.
Caught by [[reference-grep-pinned-tests-in-plans]] — should have grep'd
tests/ for `artist:` when shrinking KNOWN_KINDS. Banking the miss.
extension/manifest.json: add content_security_policy.extension_pages = "script-src 'self'; object-src 'self';" — explicitly omits the upgrade-insecure-requests directive that MV3 inherits by default. Without this, every fetch(http://curator.../...) silently upgrades to https:// at the browser layer (Sec-Fetch-Site=same-origin, NS_ERROR_GENERATE_FAILURE), regardless of about:config. Bump XPI version 1.0.3 → 1.0.4 so a fresh signed build replaces the cached one. Operator-troubleshot 2026-05-26 via Inspect-the-extension dev tools showing the silent scheme upgrade.
alembic 0023: drop ck_tag_fandom_requires_character before the tag_kind type swap and recreate after. Postgres can't resolve `kind = 'character'` across the rename (column on tag_kind_old, literal binds to new tag_kind → "operator does not exist"). Same dance on downgrade. Banked under reference_tag_kind_enum_swap_check_drop.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
ci.yml: drop pull_request: trigger — push: branches: [dev, main] already covers it; pull_request was duplicating ci.yml runs on every dev push with an open PR. (No fork PRs in this repo.)
build.yml: drop dev from push triggers — operator doesn't use the :dev image. Add tags: ['v*'] trigger + tag-push branch in the Determine-tag logic so cutting a release tag publishes an immutable :v26.05.26.X image (rollback story) without re-publishing :latest. Extend the XPI-download step to fire on tag pushes too so the versioned image carries the signed extension.
Net per hotfix cycle: 5 runs → 3 (no tag) / 4 (with tag).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Operator-flagged 2026-05-26: Atole artist detail page showed 406 Sources where 1 was right.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
_apply_sidecar is additive: it find-or-creates Post/Source/ImageProvenance
and sets primary_post_id NULL-only, so any IR-migration provenance on the
existing row survives untouched and the new GS sidecar adds a second
ImageProvenance pointing at the freshly-created Post.
Wrapped in try/except so a malformed sidecar can't unwind the file-swap
commit — the file replacement is the critical operation.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
uv falls back to pip install on runners without uv binary, so this
change is forward-compatible with the current ci-python image. When
the runner image gets uv pre-installed in a future bump, the warm
install path drops from ~2 min to ~10 seconds.
pytest-xdist parallelization is OUT OF SCOPE for this commit:
tests/conftest.py uses a TRUNCATE ALL TABLES RESTART IDENTITY CASCADE
fixture after every integration test against a single shared
database; xdist workers running in parallel would nuke each other's
mid-test state. A future refactor to per-worker databases or
per-worker schema isolation is the prerequisite.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Operator hit 3 large PNGs stuck in 'processing' for 2 days 2026-05-25:
the existing recover_interrupted_tasks flips processing > 5min back to
queued + .delay(), but if the underlying file is unfixably broken (e.g.,
PIL OSError, also patched in 68cffce), the loop never terminates and the
'Scanning...' banner sticks at 0/0 forever blocking new scans.
/api/import/clear-stuck:
- Flips every task in pending/queued/processing to 'failed' with a clear
marker error message
- Finalizes any 'running' ImportBatch that has no remaining active children
- Idempotent + non-destructive: rows survive, can be retried once the
underlying cause is resolved
UI button 'Clear stuck...' sits next to 'Retry failed' / 'Clear completed'
with a warning-tonal alert in the confirm dialog explaining what it does
and recommending Retry failed once the cause is fixed.
Tests: clears mixed non-terminal states, untouches complete rows,
finalizes orphan batch, no-op when nothing stuck.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Operator hit this 2026-05-25 after the IR tag_apply landed: ~57k images
all scanned into FC in the same week share image_record.created_at, so
the gallery timeline collapses them into a single month bucket and
scroll orders them all together at the top. Their actual publish dates
(spread over years) were already available in Post.post_date but the
gallery never read it.
Backend wire-up:
- tag_apply phase 4 now sets ImageRecord.primary_post_id when creating
ImageProvenance (only if currently NULL — preserves the canonical
download-time linkage set by the importer for new FC ingests).
- gallery_service.py introduces _effective_date_col() =
COALESCE(post.post_date, image_record.created_at), used in:
* scroll() ORDER BY + cursor WHERE clauses
* timeline() year/month group-by
* jump_cursor() year/month filter
* _neighbors() prev/next ordering
- Each method LEFT OUTER JOIN Post on primary_post_id so the COALESCE
works for images without a post (NULL on the Post side, fall back
to created_at).
- GalleryImage gains posted_at + effective_date fields; API /gallery
/scroll exposes both alongside the existing created_at so the UI
can render 'Posted on X (imported Y)' if desired.
- get_image_with_tags() returns posted_at for the modal.
Cursor format unchanged — the encoded datetime is now the effective_
date (whichever column won the COALESCE) and pagination remains
consistent.
To pick up new behavior for an already-migrated IR set: re-run
/api/migrate/tag_apply on the existing manifest (phase 4 is
idempotent; the new primary_post_id assignment backfills).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
_PLATFORM_PROFILE_URL had only patreon/subscribestar/hentaifoundry but
FC's extension_service.py recognizes 5 platforms. Any IR PostMetadata
with platform=deviantart or pixiv fell through _profile_url returning
None and the entry was silently skipped — explaining operator's
2026-05-25 finding that IR-migrated images had tags but no provenance
for the deviantart + pixiv subscriptions.
Pixiv caveat noted in comment: real profile URL takes numeric user_id
(https://www.pixiv.net/users/12345) but IR's PostMetadata.artist
stores display name. We slug the name and use it as if it were the id
so the artist->post->image linkage survives migration; the resulting
Source.url won't resolve in a browser and operator can fix via
Settings -> Subscriptions later if they want.
To recover existing IR-migrated state: re-run /api/migrate/tag_apply
on the existing manifest. Phase 4 is idempotent; new posts get
inserted only for the previously-skipped platforms.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Operator hit a corrupt JPEG in the IR set 2026-05-25: PIL.verify() only
validates header structure but doesn't catch truncated/broken pixel
data. The error surfaces later in _transparency_pct (via getchannel
'A' -> load) or compute_phash (load) — both blow up with OSError
'broken data stream when reading image file'. Celery's autoretry_for
then bounces the same file forever instead of marking it skipped.
Wrap both PIL.load-triggering call sites with try/except OSError ->
ImportResult(status=skipped, skip_reason=invalid_image).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
bvandeusen
