Phase 2 of #871: clean up the duplicate videos already in the library (the #859
"same video from multiple sources" clutter). Import-time dedup (Phase 1) only
prevents NEW dups; this is the operator-triggered cleanup of existing ones.
cleanup_service.dedup_videos(dry_run):
- backfill_video_durations: re-probe NULL-duration videos (pre-#871 rows) so the
existing library participates; idempotent (only NULL rows), writes a negative
sentinel for un-probeable files so they're neither re-probed forever nor matched.
- find_video_dup_groups: cluster same-artist videos by duration (±tol) + aspect,
anchored per cluster to bound the span (no chain drift); keeper = highest pixel
area then bytes. Reuses the importer's _VIDEO_DUP_* tolerances.
- apply: re-point each loser's post links to the keeper (so no post loses the
video) THEN delete the redundant records + files via delete_images (cascade).
dry_run shares the same discovery predicate and returns the projection only
(rule 93). Tags on a loser are NOT merged (noted; videos rarely hand-curated).
- dedup_videos_task (maintenance queue; summary → task_run.metadata).
- POST /maintenance/dedup-videos {dry_run} + GET /maintenance/task-result/<id> so
the card shows the dry-run projection before the destructive apply.
- VideoDedupCard: Preview → shows groups/redundant/reclaimable, then Apply behind
a confirm dialog. Mounted in the Maintenance panel.
Tests: dedup collapses + re-links the loser's post to the keeper + removes the
file; dry-run deletes nothing; distinct durations aren't grouped; task registered.
(Migration 0052 for duration_seconds already shipped with Phase 1.)
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Read cutover verified in prod (suggestions + allowlist read image_prediction;
backfill complete at 908k rows / 51k images). Removes the old JSON column and
everything that fed it:
- ImageRecord.tagger_predictions column removed; migration 0046 DROPs it.
tagger_model_version kept as the "tagged / current?" signal the backfill
sweep reads (needs-tagging check switched to tagger_model_version IS NULL).
- tag_and_embed no longer dual-writes the JSON — image_prediction is the only
write path.
- importer re-import reset drops the JSON line (image_prediction rows are
already deleted on re-import).
- Retired the one-time #768 backfill task + the #764 prune task, their admin
endpoints, and their Maintenance cards (Backfill/PrunePredictionsCard).
- Tests seed/assert via image_prediction; stale column refs removed.
Disk reclaim is NOT automatic: DROP COLUMN is a catalog change. Run
`VACUUM FULL image_record` off-hours afterward to return the ~100 GB to the OS
so DB backups go small (#739). image_prediction (~90 MB) stays in pg_dump — it's
the source of truth now.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The native Patreon backfill flooded the feed with bare 'Post <id>' shells
(1589 for Anduo). Root cause: PostAttachment.sha256 was GLOBALLY unique, so a
non-art file reused across posts only ever linked to the first one, and
_capture_attachment created the Post before that dedup check — leaving later
posts with no image and no attachment. Duplicate IMAGES had the mirror gap:
attach_in_place returned duplicate_hash/duplicate_phash before _apply_sidecar,
so the second post got no provenance row, and the feed only rendered via
primary_post_id (one post per image).
Operator requirement: a duplicate item must show on EVERY post it appears in.
Unify the fix as link-not-suppress:
- importer: on duplicate_hash / duplicate_phash(larger_exists), append an
image_provenance row for the new post (keep primary on the first). Both the
download path (attach_in_place) and the filesystem path (_import_media).
- post_feed_service: render thumbnails by image_provenance UNION primary_post_id,
so a cross-posted image shows on every post (and legacy primary-only images
still show).
- PostAttachment: per-post uniqueness — drop UNIQUE(sha256), add partial
UNIQUE(post_id, sha256) + partial UNIQUE(sha256) WHERE post_id IS NULL
(migration 0043); _capture_attachment dedups per-(post,sha) over the shared
sha-addressed blob, so no post is left bare.
- cleanup: new prune-bare-posts maintenance action (cleanup_service
_bare_post_conditions shared by preview/count/delete per preview/apply parity;
admin endpoint; PostMaintenanceCard). Deletes posts with zero image links
(primary or provenance) AND zero attachments. Run after the feed fix so a
hidden provenance link spares the post instead of deleting it.
Tests: dup image shows on both posts; dup attachment shows on both posts; feed
renders provenance-linked duplicates; prune-bare delete-path == preview.
Operator redeploys (migration 0043) then runs the prune to clear the shells.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The fandom/chapter exclusions added in fb05c5e only touched find_unused_tags
(the preview SAMPLE). prune_unused_tags re-implemented the predicate inline for
the dry-run COUNT and the live DELETE with only the image_tag + series_page
checks — so the preview showed a safe list of names while the delete removed
every fandom (and chaptered series). Operator-flagged 2026-06-08: real data loss
— assigned fandoms deleted, their characters SET-NULLed.
Extract _unused_tag_conditions() as the single source of truth and use it for
the preview, the count, AND the delete, so they can never diverge again. Added a
prune-commit test asserting the LIVE delete spares a character's fandom and a
chaptered series.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
find_unused_tags only excluded tags with image_tag or series_page references, so
it flagged every fandom as 'unused' — fandoms are NEVER applied to images (a
character carries its fandom via tag.fandom_id), and the FK is ondelete=SET NULL,
so deleting one silently strips the fandom off all its characters
(operator-flagged 2026-06-08: artist-OC fandoms showing as unused).
Exclude tags referenced as a character's fandom_id, and (same class of gap) tags
referenced by a series_chapter (an all-placeholder series has chapters but no
pages yet). A genuinely orphaned fandom with no characters is still swept.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
reextract_archive_attachments loaded ALL PostAttachments and ran in one pass up
to a 30-min soft limit, then died without re-enqueueing — a large archive
backlog would only ever partially process. And a naive re-run can't advance: an
already-extracted archive is still an archive on disk, so it'd re-extract the
same first batch forever.
Give it a real cursor + time-box + self-resume (mirrors normalize_tags_task,
operator-asked 2026-06-07: reasonable timeout, then re-queue so other work keeps
flowing):
- service scans attachments with id > after_id in ascending order, time-boxes
the chunk, and reports partial=True + resume_after_id (last scanned id).
- task passes a 600s budget and re-enqueues itself from the cursor until the
scan is exhausted. Routes on the maintenance_long lane.
- This is independent of the maintenance_long lane isolation (already shipped) —
that stops long tasks starving the quick maintenance queue; this stops the
re-extract itself dying on a big backlog.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The integration test caught it: _import_media re-derives the artist by path-walk
from the attribution path (ignoring the explicit artist) AND _copy_to_library
lands members relative to that path. Staging the archive in /tmp meant the
artist didn't resolve (provenance skipped) and members would land in the temp
dir. Stage under images_root/<slug>/<platform>/<post>/ instead so the artist
resolves and members land in the real library; remove only the staged archive +
sidecar afterward (members stay). Require a real artist+slug (skip + count
otherwise).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Existing PostAttachments that are actually archives (filed opaquely before the
magic-byte gate) need extracting retroactively. cleanup_service.
reextract_archive_attachments scans PostAttachments, magic-detects the archives,
and for each reconstructs the post's sidecar from the DB + re-runs attach_in_place
in a temp dir — so the members extract and re-link to the SAME post via
find_or_create_post (source_id + external_post_id). Idempotent (members dedupe by
sha256). Enqueues thumbnail+ML for new members.
Wired as a maintenance-queue Celery task (tasks/admin) + POST
/api/admin/maintenance/reextract-archives (202) + a "Re-extract archive
attachments" card in Settings → Maintenance.
Test: a zip stored under a mangled extension-less name extracts + links its
member to the post via ImageProvenance, and a second run is a no-op (idempotent).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Wipe every general + character tag so the operator can re-tag from scratch via
the Camie auto-suggest, while PRESERVING fandoms, series (+ series_page order),
and each image's stored tagger_predictions (so suggestions repopulate
immediately). One set-based DELETE FROM tag WHERE kind IN ('general','character')
— the five tag-referencing tables all cascade, so applications + aliases +
allowlist + rejections + centroids clear automatically; series tags aren't
deleted so series survive; Tag.fandom_id is SET NULL so fandoms are untouched.
Reuses the established dry-run-preview -> confirm pattern: cleanup_service.
reset_content_tagging() + POST /api/admin/tags/reset-content +
TagMaintenanceCard section with a backup-first warning and a red confirm
showing exact counts (tags by kind + image applications). Irreversible except
via DB backup restore; the wipe only fires when the operator confirms.
Tests: service dry-run counts + live delete preserves fandom/series/series_page
while content tags + their image_tag cascade away; API dry-run wiring.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Five small G5 findings from the 2026-06-02 audit. Each is local and
follows an established FC pattern.
- download_service: replace hardcoded ('discord','pixiv') tuple with
auth_type_for(platform) == 'token'. A 7th token-platform now picks
up the right credential path without touching this site.
- /api/tags/<source_id>/merge enqueues recompute_centroid.delay after
merge so the target's centroid reflects its new image set
immediately. Daily list_drifted catches it within 24h, but eager
recompute closes the suggestion-quality dip in the meantime.
- backfill_thumbnails added to beat_schedule (daily). The task
docstring claimed periodic Beat but the entry was never registered,
so the library got no self-healing thumbnail repair; only the
manual admin-UI button fired it.
- modal.createAndAdd pushes a kind='fandom' tag into
tagsStore.fandomCache so FandomPicker sees the new fandom on next
open. Was: cache-gated load (length===0) skipped refetch, new
fandom invisible until full page reload.
- cleanup cluster:
- Drop .webp from cleanup_service.unlink — thumbnailer only writes
.jpg/.png; the third tuple member was dead code.
- Drop effective_date from /api/gallery/scroll response — no FE
consumer reads it. Service still computes the attribute for
timeline ordering; this just trims the JSON.
- Rename store.recentMinute → store.recentRuns across the
systemActivity store + three consumers (SystemActivitySummary,
QueuesTable, SystemActivityTab). The data is the last 200 runs
(not actually "last minute"), so the name lied.
NOT in this bundle: the duplicate tag-merge endpoint
(/api/tags vs /api/admin/tags) is harder — has 1 FE caller and 3 tests
on the admin variant; consolidation is its own change.
Plugs the FC long-running-entity discipline gaps the 2026-06-02 audit
flagged: every entity that can get stuck now has recovery + retention +
timeout, and the long-runners no longer collide with the FC-3i sweep.
Recovery sweeps (every 5 min):
- recover_stalled_backup_runs — flips BackupRun stuck in
running/restoring past 7h (covers the 6.5h images-backup hard
limit) to error. prune_backups docstring corrected — the FC-3i
TaskRun sweep never touched BackupRun rows.
- recover_stalled_library_audit_runs — flips LibraryAuditRun stuck
past 135 min (10-min buffer above scan_library_for_rule's 2h5m
hard limit) to error. Previously a SIGKILL'd row blocked all
future audits until manual DB surgery.
- recover_stalled_import_batches — finalizes ImportBatch rows
stuck running >2h whose child tasks are all terminal (orphan case
where the orchestrator crashed before the closing UPDATE). Uses
the same EXISTS predicate /api/system/stats already had.
Retention (daily):
- prune_library_audit_runs — 30-day window. Audit rows carry
matched_ids JSONB blobs that can hold tens of thousands of ids.
- prune_import_batches — 30-day window. Cascades to ImportTask via
the model relationship.
time_limits on five long-runners that previously had none (the
audit's headline finding — every one of these collided with the
recover_stalled_task_runs 5-min default and could be marked
'error' mid-flight):
- scan_directory: 60m soft / 70m hard
- verify_integrity: 60m / 70m
- backfill_phash: 30m / 35m
- apply_allowlist_tags: 30m / 35m
- recompute_centroids: 30m / 35m
QUEUE_STUCK_THRESHOLD_MINUTES now covers maintenance (75) and scan
(75) — above the longest task on each — with per-task overrides
for the outliers (backup_images_task 420, restore_images_task 420,
scan_library_for_rule 130).
start_audit_run guard is now age-aware: a 'running' row older than
the audit hard limit doesn't block a new run (the sweep will catch
it within 5 min). Previously a SIGKILL'd row blocked forever.
/api/import/status now uses the same EXISTS predicate
/api/system/stats does, so the two endpoints no longer disagree on
the active-batch question.
DownloadEvent.started_at resets on pending→running so a freshly-
promoted event from a busy queue isn't measured against its
original enqueue time (was racing recover_stalled_download_events
on heavy-queue days).
- BACKFILL_TIMEOUT_SECONDS 1800→1170: keep the subprocess timeout
30s below Celery's hard time_limit=1200 so SIGKILL doesn't beat
TimeoutExpired (matched the tick 870s/900s rationale). Backfill
runs that hit the cap let the next tick continue via the archive.
- recover_interrupted_tasks orphan UPDATE now stamps finished_at;
without it cleanup_old_tasks' WHERE finished_at<cutoff never
reaped orphan-swept rows. recover_stalled_task_runs also now sets
duration_ms (matches celery_signals.finalize's millisecond math).
- ExtensionService.quick_add_source arms NEW_SOURCE_BACKFILL_RUNS=3
on Source creation, mirroring SourceService.create. Without it,
Firefox quick-add on a creator with >20 unsynced posts walked the
full feed until subprocess timeout. Renamed the constant from
_NEW_SOURCE_BACKFILL_RUNS so it can be imported cross-module.
- gallery_dl.verify() accepts TIER_LIMITED as auth-success alongside
NO_NEW_CONTENT — the download path (line 712) already does, and
TIER_LIMITED proves auth reached the post and was told it was
tier-gated. Verify endpoint previously showed red on this and
prompted operators to rotate working cookies.
- prune_unused_tags now runs a single DELETE with the NOT-IN
predicate find_unused_tags uses, instead of SELECT-ids →
DELETE-WHERE-IN. Removes the psycopg 65535-param cliff that
would have surfaced on a tag explosion (>65k unused tags).
- credentials.upload() reflects the returned record into the store
cache (`.set(platform, rec)`) instead of evicting it; previously
the card briefly rendered "no credential" between upload and
loadAll().
Operator-confirmed 2026-05-28: the BlenderKnight:* tags are `archive`
kind (caught by the kind purge), but the `source:patreon`-style tags
are IR's old `source` kind that fell back to `general` during migration
(FC's enum has no `source` kind) — so they can't be matched by kind.
Broadened the purge to a two-rule match and renamed it for accuracy
(all dev-only, unreleased):
- cleanup_service.purge_tags_by_kind → purge_legacy_tags. Predicate is
now `kind IN (archive, post, artist) OR name LIKE 'source:%'`
(LEGACY_NAME_PREFIXES). Preview classifies each row into by_kind OR
by_prefix (source:* counts once under the prefix bucket regardless of
its general kind).
- endpoint /tags/purge-retired-kinds → /tags/purge-legacy. dry-run
returns by_kind + by_prefix + count + sample.
- store purgeRetiredKindTags → purgeLegacyTags.
- Tag Maintenance card copy + breakdown updated to show both buckets;
button reads "Preview/Delete legacy tags".
Tests updated + extended: dry-run reports by_kind {archive,post,artist}
AND by_prefix {source:*}, plain general/character tags survive; commit
deletes both the kind-matched and source:*-matched rows and leaves the
rest.
Operator-flagged 2026-05-28, two asks.
**1. Provenance posts ate the panel.** Each post card rendered its full
description (180px scroll box) inline, so a few posts pushed everything
else off-screen. ProvenancePanel now collapses the description by
default behind a per-post "Show description ▾ / Hide ▴" toggle (state
keyed by provenance_id, reset when the viewed image changes). Cards stay
compact — platform/date/title/meta/actions — and the operator expands
only the descriptions they want.
**2. Purge tags of retired/system kinds.** The IR migration left
`archive`/`post`/`artist`-kind tags (e.g. `BlenderKnight:Hannah_BJ_Loops`)
that FC no longer creates — the tag input only makes
character/fandom/series/general, and provenance + artists are their own
systems now. (meta/rating were already hard-deleted by alembic 0023.)
- cleanup_service.purge_tags_by_kind(kinds=PURGEABLE_TAG_KINDS) — counts
(dry_run) or deletes tags whose kind ∈ (archive, post, artist).
CASCADE clears the image_tag / alias / allowlist / etc. rows.
- POST /api/admin/tags/purge-retired-kinds (Tier-A, dry-run preview
returns per-kind counts + sample names — the preview IS the
verification of exactly what'll be deleted before committing).
- Tag Maintenance card gets a second section: "Preview retired-kind
tags" → per-kind breakdown + sample → "Delete N retired-kind tag(s)".
Tests: dry-run counts by kind (general survives), commit deletes only
the retired kinds (general + character survive, retired count → 0).
NOTE: a dry-run preview will show exactly which kinds/counts are
present. If the operator's noisy tags turn out to be `general` (e.g. an
IR `source:patreon` that fell back to general during migration), they
won't be caught by the kind purge — the preview makes that visible so
we can decide on a name-based pass separately.
bvandeusen