Commit Graph

26 Commits

Author SHA1 Message Date
bvandeusen f77e75147d feat(tags): system-tag UI markers + full protection sweep (step 4 of #128)
CI / lint (push) Successful in 3s
CI / frontend-build (push) Successful in 22s
CI / backend-lint-and-test (push) Successful in 32s
CI / integration (push) Failing after 3m33s
UI: shield marker + tooltip on TagChip and TagCard; system tags hide
rename/merge/delete affordances (chip kebab entirely — set-fandom never
applies to their general kind; remove stays, un-tagging is normal use).
Aliases stay available: mapping model outputs ONTO a system tag is
useful. Directory cards carry is_system.

Every destructive path that could take out a system row is now guarded,
found by sweeping run 1891s off-by-three failures — each one was a
surface that would have eaten the seeded tags:
- prune-unused: predicate exempts is_system (they ship with zero
  applications and matched every unused condition)
- reset-content: predicate exempts is_system AND keeps their
  applications — hygiene flags describe the file, not content tagging
- admin tag DELETE: refused with system_tag error
- normalize_existing_tags: scan excludes is_system — canonicalization
  would recase wip -> Wip behind TagService.rename's guard, breaking
  the name-keyed presentation lookup

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CDgx8bQS5YrGRK76v8HUnM
2026-07-03 08:34:16 -04:00
bvandeusen aa12a57f97 feat(recovery): surgical re-fetch for deep posts via ExternalLink reset
CI / lint (push) Successful in 5s
CI / frontend-build (push) Successful in 47s
CI / backend-lint-and-test (push) Successful in 2m0s
CI / integration (push) Successful in 4m50s
Operator-flagged: the recovered defective files live DEEP in their artists'
back-catalogues — the normal download cadence (by design, via the seen-gates)
will never re-walk them, so recovery's source re-check alone can't bring them
back. The durable per-post handle is the ExternalLink row, which survives the
image delete:

- services/external_links.refetch_links_for_post: reset settled links to
  pending (fresh attempt budget, in-flight left alone) + dispatch their
  fetches; sha-dedupe at import discards payload files that still exist, so
  only the missing file lands.
- recover_defective_image now captures the image's post ids BEFORE the delete
  cascades provenance away and resets those posts' links — future recoveries
  are surgical automatically (response gains links_reset; source re-check
  stays for gallery-dl-native files within walk reach).
- POST /api/admin/posts/refetch-external {external_post_id, source_id?} — the
  manual tool for the three files recovered before this fix existed.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CDgx8bQS5YrGRK76v8HUnM
2026-07-02 21:07:21 -04:00
bvandeusen 7c19ad91ed feat: cap-aware autoscaler + token-gated whole-instance tag reset (operator feedback)
CI / lint (push) Successful in 2s
CI / frontend-build (push) Successful in 18s
CI / backend-lint-and-test (push) Successful in 37s
CI / integration (push) Successful in 3m28s
Autoscaler (agent 2026-07-02.5): the buffer-occupancy signal alone would peg
downloaders at DL_MAX while the bandwidth CAP — not concurrency — is the real
constraint (8 streams sharing 8 MB/s move no more data than 4). Growth is now
gated on the pipe having headroom (net < 85% of cap) and a pipe pinned at the
cap (>= 95%) sheds streams down to 3; dead band prevents flapping. The UI hint
says 'holding at the bandwidth cap' and /status reports bw_capped, so the
behavior is legible without tests that need the ML stack.

Reset content tagging: stays a FULL-instance reset (operator's call), but now
lives in a fenced 'Danger zone' section on Cleanup and the apply is gated by a
preview-derived confirm token (mirrors the Tier-C bulk-delete pattern — stale
counts are rejected server-side). Copy no longer claims suggestions repopulate:
it says plainly the heads' training examples are deleted and re-tagging starts
fresh. Moved out of TagMaintenanceCard into DangerZoneCard.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CDgx8bQS5YrGRK76v8HUnM
2026-07-02 16:14:48 -04:00
bvandeusen 1f27189b8f chore: retire ml-backfill-daily beat + the spent purge-legacy action (operator-approved)
- ml-backfill-daily: the CPU tag_and_embed backfill raced the GPU agent's
  daily embed backfill for the same NULL-embedding images at ~100x the cost
  (B1 audit verdict, milestone #124). The backfill TASK stays — the manual
  /api/ml/backfill button remains the deliberate CPU fallback pending B3.
- purge-legacy: one-time IR-migration cleanup, dry-run verified 0 targets on
  the live library before removal (A2 audit, milestone #123). Fully retired
  per rule 22: tile, store action, route, service fn, tests.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CDgx8bQS5YrGRK76v8HUnM
2026-07-02 11:24:08 -04:00
bvandeusen 7127714316 feat(tags): non-mutating merge preview + admin dry_run (#8a)
CI / lint (push) Successful in 3s
CI / frontend-build (push) Successful in 18s
CI / backend-lint-and-test (push) Successful in 27s
CI / integration (push) Failing after 3m17s
Cluster B, milestone #99. TagService.merge_preview(source, target) computes the
same counts the apply produces (rule 93 parity) without mutating: images_moving
(source links the apply UPDATEs), images_already_on_target (links it drops),
source_total, series_pages, will_alias (_keep_as_alias), a kind/fandom
compatible flag (surfaced, not raised, so the UI can warn), and up to 6
thumbnails of the moving images. The admin /tags/<dest>/merge route gains a
dry_run flag returning the preview JSON.

Tests: preview moving-count == apply merged_count (parity), incompatible flagged
without raising, self/missing raise, admin dry_run returns preview + no mutation.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XCUHUGQLrBrkgyk1t49kpX
2026-06-23 01:37:11 -04:00
bvandeusen 6599a07468 refactor(admin): consolidate maintenance-trigger 202 responses onto _queued() (#753 Finding B)
CI / lint (push) Successful in 2s
CI / frontend-build (push) Successful in 19s
CI / backend-lint-and-test (push) Successful in 27s
CI / integration (push) Successful in 3m21s
DRY pass follow-up (note #1026). Five handlers returned the identical
jsonify({task_id, status:queued}), 202 shape; extract _queued(async_result).

Consumers routed through it: tags_normalize (live branch), trigger_reextract_archives,
trigger_prune_missing_files, trigger_dedup_videos, trigger_purge_gated_previews.
trigger_vacuum stays bespoke (returns no task_id — the UI doesn't poll it).

Added route-level tests for all five consumers (these trigger endpoints had no
route coverage before): 202 + task_id via _queued, and the dry_run flag threading
through to dedup/purge-gated. Behavior unchanged.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-22 16:35:59 -04:00
bvandeusen 6281cb1e66 refactor(admin): consolidate Tier-A dry-run/apply handlers onto one helper (#753)
CI / lint (push) Successful in 3s
CI / frontend-build (push) Successful in 20s
CI / backend-lint-and-test (push) Successful in 27s
CI / integration (push) Successful in 3m18s
DRY pass on the cleanup/admin destructive-ops surface (task #753, hardened
process #594). Five Tier-A endpoints repeated the same get_json -> dry_run ->
run_sync(service_fn) -> jsonify block verbatim. Extract _run_dry_run_op(service_fn,
**kwargs); the five route handlers now delegate. reconcile keeps its source_id
validation and passes it through **kwargs.

The cleanup_service predicates were already shared between preview and apply
(find_*_conditions / find_duplicate_post_groups) — the post-data-loss fix — so no
backend-logic change; this is purely the HTTP-handler boilerplate.

Consumers (all routed through the helper, verified no copy left behind):
  prune_unused_tags, prune_bare_posts, reconcile_duplicate_posts (+source_id),
  purge_legacy_tags, reset_content_tagging.

Added route-level tests for prune-bare (apply) and reconcile (apply + source_id
passthrough + invalid-source_id 400) — the two helper consumers that previously
had only service-level coverage, so every consumer is exercised at the route.

Findings B (queued-response helper) and C (store dry-run POST helper) identified
but not applied this pass (operator scoped to A). The card preview->commit state
machine is deferred to a frontend pattern-consistency sweep.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-22 14:53:04 -04:00
bvandeusen eff64275fc feat(maintenance): reconcile duplicate posts (gallery-dl→native unify)
CI / lint (push) Successful in 3s
CI / frontend-build (push) Successful in 19s
CI / backend-lint-and-test (push) Successful in 38s
CI / integration (push) Successful in 3m17s
An artist first downloaded by gallery-dl gets Post rows keyed by the per-
attachment id; a later native walk keys the SAME real post by the post id. They
never dedup (uq_post_source_external_id is on external_post_id) → duplicate post
rows (cheunart: 943→1109). The real post id is recoverable in-DB from
raw_metadata['post_id'] (both eras store the sidecar there).

reconcile_duplicate_posts (cleanup_service): group posts by (source_id, canonical
post_id = raw_metadata.post_id else external_post_id); for each group >1, keep the
row already keyed by the post id (the format the CURRENT native downloader
produces, so future walks dedup and this can't recur), re-point
ImageRecord.primary_post_id / ImageProvenance / PostAttachment / ExternalLink onto
it conflict-safe (drop the loser's row where the keeper already has the equivalent,
per each table's uniqueness), backfill the keeper's empty date/title/body/raw_meta
from a loser, set external_post_id=post_id + derive post_url, delete losers.
IMAGES ARE NOT TOUCHED (content-addressed/deduped already; operator-confirmed).

Preview/apply share find_duplicate_post_groups (rule 93). API
/api/admin/posts/reconcile-duplicates (dry_run→{groups,posts_to_merge,sample};
apply→{groups,merged,sample}; optional source_id). UI: a second section on
PostMaintenanceCard (preview groups+sample → confirm merge). Tests: merge +
metadata backfill + image move, no-op when unique, provenance-collision dedup.

Design: milestone #73. Forensics: note #917. Out of scope (flagged): cheunart vs
Cheunart case-variant artist dirs/rows.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-17 21:57:21 -04:00
bvandeusen 540151290b feat(cleanup): purge misgrabbed gated-post blurred previews (#874 follow-up)
CI / lint (push) Successful in 2s
CI / frontend-build (push) Successful in 19s
CI / backend-lint-and-test (push) Successful in 31s
CI / integration (push) Successful in 3m18s
A one-shot Maintenance action to remove the blurred locked-preview images
the ingester downloaded from tier-gated Patreon posts before #874.

current_user_can_view was never persisted, so the cleanup re-walks each
enabled Patreon source (read-only) to re-derive which posts are gated now
and the blurred filehashes Patreon serves for them, then matches by
CONTENT HASH against stored source_filehash. Because the hash is
content-addressed, a real file downloaded when access existed has a
different hash and can never match — regained-then-lost-access content is
provably spared (operator's hard requirement). NULL source_filehash =>
unverifiable, kept + reported.

On apply: delete matched ImageRecords + files (provenance cascades),
clear seen/dead-letter ledger rows for those hashes so the real media
re-ingests if access returns, and delete gated posts left bare. Shares
one match predicate between preview and apply (rule 93).

- cleanup_service: collect_gated_previews + purge_gated_previews
- tasks.admin: purge_gated_previews_task (async re-walk bridge, timeboxed)
- api.admin: POST /maintenance/purge-gated-previews
- GatedPurgeCard.vue in Settings > Maintenance (preview -> confirm -> apply)
- tests: collect predicate, hash-match delete/spare/unverifiable, ledger
  clear, bare-post removal, no-op

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-16 15:20:35 -04:00
bvandeusen 41652db20f feat(maintenance): retroactive video-dedup action — preview + apply (#871)
CI / lint (push) Successful in 3s
CI / frontend-build (push) Successful in 22s
CI / backend-lint-and-test (push) Successful in 27s
CI / integration (push) Successful in 3m14s
Phase 2 of #871: clean up the duplicate videos already in the library (the #859
"same video from multiple sources" clutter). Import-time dedup (Phase 1) only
prevents NEW dups; this is the operator-triggered cleanup of existing ones.

cleanup_service.dedup_videos(dry_run):
- backfill_video_durations: re-probe NULL-duration videos (pre-#871 rows) so the
  existing library participates; idempotent (only NULL rows), writes a negative
  sentinel for un-probeable files so they're neither re-probed forever nor matched.
- find_video_dup_groups: cluster same-artist videos by duration (±tol) + aspect,
  anchored per cluster to bound the span (no chain drift); keeper = highest pixel
  area then bytes. Reuses the importer's _VIDEO_DUP_* tolerances.
- apply: re-point each loser's post links to the keeper (so no post loses the
  video) THEN delete the redundant records + files via delete_images (cascade).
  dry_run shares the same discovery predicate and returns the projection only
  (rule 93). Tags on a loser are NOT merged (noted; videos rarely hand-curated).

- dedup_videos_task (maintenance queue; summary → task_run.metadata).
- POST /maintenance/dedup-videos {dry_run} + GET /maintenance/task-result/<id> so
  the card shows the dry-run projection before the destructive apply.
- VideoDedupCard: Preview → shows groups/redundant/reclaimable, then Apply behind
  a confirm dialog. Mounted in the Maintenance panel.

Tests: dedup collapses + re-links the loser's post to the keeper + removes the
file; dry-run deletes nothing; distinct durations aren't grouped; task registered.
(Migration 0052 for duration_seconds already shipped with Phase 1.)

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-16 08:31:50 -04:00
bvandeusen 949c9abcc6 fix(external): path-safe unlink + per-link staging + orphan repair (#859)
CI / lint (push) Successful in 2s
CI / frontend-build (push) Successful in 21s
CI / backend-lint-and-test (push) Successful in 27s
CI / integration (push) Successful in 3m17s
External downloads import IN PLACE, so the post-attach dedup-skip unlink could
delete a file that IS an ImageRecord's backing file — orphaning the record and
404-ing on playback. Two sources of that:

- Two links on the same post (same film from mega + gdrive) emitted the same
  filename into one external/<post_id>/ dir; the second overwrote the first.
  Stage per-LINK now (external/<post_id>/<link_id>/) so each file keeps its path.
- The duplicate_hash/duplicate_phash branch unlinked `f` unconditionally. Make it
  path-safe: only unlink when `f` is NOT the existing record's canonical file.

Plus an operator-triggered orphan-repair maintenance task
(prune_missing_file_records_task) to clean up records already orphaned by the
bug: scans ImageRecords, deletes those whose file is gone (cascade), with an
NFS-stall guard that aborts without deleting if a large sample is mostly missing.
Wired through POST /api/admin/maintenance/prune-missing-files and a
MissingFileRepairCard in the Maintenance panel.

Tests: refetch-same-link keeps the canonical file; orphan repair deletes only
real orphans and aborts on the mostly-missing guard.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-15 01:48:38 -04:00
bvandeusen 3610ba495f feat(ml): drop image_record.tagger_predictions — image_prediction is sole store (#768 step 3)
CI / lint (push) Successful in 3s
CI / frontend-build (push) Successful in 23s
CI / backend-lint-and-test (push) Successful in 33s
CI / integration (push) Successful in 3m14s
Read cutover verified in prod (suggestions + allowlist read image_prediction;
backfill complete at 908k rows / 51k images). Removes the old JSON column and
everything that fed it:

- ImageRecord.tagger_predictions column removed; migration 0046 DROPs it.
  tagger_model_version kept as the "tagged / current?" signal the backfill
  sweep reads (needs-tagging check switched to tagger_model_version IS NULL).
- tag_and_embed no longer dual-writes the JSON — image_prediction is the only
  write path.
- importer re-import reset drops the JSON line (image_prediction rows are
  already deleted on re-import).
- Retired the one-time #768 backfill task + the #764 prune task, their admin
  endpoints, and their Maintenance cards (Backfill/PrunePredictionsCard).
- Tests seed/assert via image_prediction; stale column refs removed.

Disk reclaim is NOT automatic: DROP COLUMN is a catalog change. Run
`VACUUM FULL image_record` off-hours afterward to return the ~100 GB to the OS
so DB backups go small (#739). image_prediction (~90 MB) stays in pg_dump — it's
the source of truth now.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-11 18:52:33 -04:00
bvandeusen 65211a3f2f fix(migration): make 0045 DDL-only; backfill image_prediction via batched task (#768)
CI / lint (push) Successful in 3s
CI / backend-lint-and-test (push) Successful in 35s
CI / frontend-build (push) Successful in 42s
CI / integration (push) Successful in 3m16s
The inline INSERT…SELECT backfill in migration 0045 wrapped the table
creation and a ~100 GB pass over image_record.tagger_predictions in one
transaction: nothing committed until the end, it was unmonitorable, and an
earlier MATERIALIZED-CTE form spilled the full 100 GB to temp on NFS. A
deploy got stuck on it for ~2h with image_prediction never appearing.

Split the concerns:
- 0045 now creates ONLY the table + indexes (instant DDL → web boots).
- New backend.app.tasks.admin.backfill_image_predictions_task copies the
  >= store-floor predictions from the JSON into image_prediction, batched by
  id window and committed per chunk: live progress, resumable (re-enqueues
  from the last committed id), idempotent (ON CONFLICT DO NOTHING). json_each
  stays in the DB executor streaming each window — no Python-side 100 GB load,
  no materialization.
- POST /api/admin/maintenance/backfill-predictions + a Maintenance-tab card
  to trigger the one-time run after upgrading.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-11 09:18:25 -04:00
bvandeusen d55e52ae9b feat(admin): prune_low_confidence_predictions backfill task + UI (#764)
CI / lint (push) Successful in 3s
CI / frontend-build (push) Successful in 20s
CI / backend-lint-and-test (push) Successful in 32s
CI / integration (push) Successful in 3m13s
The one-time backfill that actually shrinks the DB: drops stored
tagger_predictions entries below ml_settings.tagger_store_floor from every
image_record row, and clamps any allowlist min_confidence below the floor up
to it. Keep predicate (confidence >= floor) mirrors Tagger.infer's store gate
so backfilled rows match new imports. Keyset by id ASC, idempotent,
self-resumes on the soft time limit; runs on the maintenance_long lane.

pg_dump copies live data only, so this alone fixes the #739 backup timeout —
the reclaim (VACUUM FULL / pg_repack on image_record) is a separate, optional
disk-return step, brief because post-prune the live data is tiny.

- admin.prune_low_confidence_predictions_task + POST /api/admin/maintenance/prune-predictions
- PrunePredictionsCard in the Maintenance panel (shows the current floor)
- tests: registration + prune-keeps->=floor/drops-<floor + allowlist clamp

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-10 13:57:39 -04:00
bvandeusen a8f624a0f1 fix(posts): link duplicate items to every post + prune bare shells
CI / lint (push) Successful in 2s
CI / frontend-build (push) Successful in 26s
CI / backend-lint-and-test (push) Successful in 30s
CI / integration (push) Successful in 3m20s
The native Patreon backfill flooded the feed with bare 'Post <id>' shells
(1589 for Anduo). Root cause: PostAttachment.sha256 was GLOBALLY unique, so a
non-art file reused across posts only ever linked to the first one, and
_capture_attachment created the Post before that dedup check — leaving later
posts with no image and no attachment. Duplicate IMAGES had the mirror gap:
attach_in_place returned duplicate_hash/duplicate_phash before _apply_sidecar,
so the second post got no provenance row, and the feed only rendered via
primary_post_id (one post per image).

Operator requirement: a duplicate item must show on EVERY post it appears in.
Unify the fix as link-not-suppress:

- importer: on duplicate_hash / duplicate_phash(larger_exists), append an
  image_provenance row for the new post (keep primary on the first). Both the
  download path (attach_in_place) and the filesystem path (_import_media).
- post_feed_service: render thumbnails by image_provenance UNION primary_post_id,
  so a cross-posted image shows on every post (and legacy primary-only images
  still show).
- PostAttachment: per-post uniqueness — drop UNIQUE(sha256), add partial
  UNIQUE(post_id, sha256) + partial UNIQUE(sha256) WHERE post_id IS NULL
  (migration 0043); _capture_attachment dedups per-(post,sha) over the shared
  sha-addressed blob, so no post is left bare.
- cleanup: new prune-bare-posts maintenance action (cleanup_service
  _bare_post_conditions shared by preview/count/delete per preview/apply parity;
  admin endpoint; PostMaintenanceCard). Deletes posts with zero image links
  (primary or provenance) AND zero attachments. Run after the feed fix so a
  hidden provenance link spares the post instead of deleting it.

Tests: dup image shows on both posts; dup attachment shows on both posts; feed
renders provenance-linked duplicates; prune-bare delete-path == preview.

Operator redeploys (migration 0043) then runs the prune to clear the shells.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-08 19:28:33 -04:00
bvandeusen 3c89223dcb feat(tags): retro-normalize existing tags to Title Case + merge case-collisions (plan #714)
CI / lint (push) Failing after 2s
CI / frontend-build (push) Successful in 26s
CI / backend-lint-and-test (push) Successful in 27s
CI / integration (push) Successful in 3m2s
Follow-up to #701: new tags are saved canonical, but the back-catalog keeps
whatever casing it was created with. This adds a maintenance action that
Title-Cases every existing tag (collapsing whitespace) and merges
case/whitespace-variant duplicates into one.

Backend:
- tag_service.normalize_existing_tags(session, *, dry_run): groups all tags by
  (kind, coalesce(fandom_id,-1), canonical_name). Per group it picks a survivor
  (prefer an already-canonical member → no rename/self-alias; else the
  best-connected tag → fewest FK repoints; else lowest id), merges the variants
  INTO it via the tested TagService._do_merge (image_tag/allowlist/embedding/
  aliases/series_page repoints + protective ML aliases), then renames the
  survivor to canonical. Losers are deleted before the rename so there's no
  transient unique-index clash; commits per group and isolates failures per
  group. Idempotent — an already-canonical lone tag is a no-op.
- normalize_tags_task (maintenance queue, asyncio.run + per-task NullPool async
  engine, soft 1800/hard 2400) — recovery/timeout/duration covered by FC-3i.
- POST /api/admin/tags/normalize: dry_run=true returns a projection inline
  (group/collision/rename counts + sample); dry_run=false enqueues the task.

Frontend: a "Standardize tag casing" section in TagMaintenanceCard (Cleanup
tab) — preview → apply (polls the activity dashboard to terminal status),
behind a back-up-first warning. admin store gains normalizeTags().

Tests: tests/test_tag_normalize.py — dry-run counts, live merge + image-tag
dedup/repoint, idempotency, same-name-different-fandom and -different-kind kept
separate, ML-known loser keeps a protective alias.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-06 16:28:34 -04:00
bvandeusen a497104661 feat(maintenance): re-extract archive attachments + link to post — #713 part 2
CI / lint (push) Successful in 2s
CI / frontend-build (push) Successful in 20s
CI / backend-lint-and-test (push) Successful in 27s
CI / integration (push) Failing after 3m3s
Existing PostAttachments that are actually archives (filed opaquely before the
magic-byte gate) need extracting retroactively. cleanup_service.
reextract_archive_attachments scans PostAttachments, magic-detects the archives,
and for each reconstructs the post's sidecar from the DB + re-runs attach_in_place
in a temp dir — so the members extract and re-link to the SAME post via
find_or_create_post (source_id + external_post_id). Idempotent (members dedupe by
sha256). Enqueues thumbnail+ML for new members.

Wired as a maintenance-queue Celery task (tasks/admin) + POST
/api/admin/maintenance/reextract-archives (202) + a "Re-extract archive
attachments" card in Settings → Maintenance.

Test: a zip stored under a mangled extension-less name extracts + links its
member to the post via ImageProvenance, and a second run is a no-op (idempotent).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-06 14:43:26 -04:00
bvandeusen 91b0145bc8 feat(tags): 'Reset content tagging' admin action
CI / lint (push) Successful in 3s
CI / backend-lint-and-test (push) Successful in 11s
CI / frontend-build (push) Successful in 30s
CI / integration (push) Successful in 2m57s
Wipe every general + character tag so the operator can re-tag from scratch via
the Camie auto-suggest, while PRESERVING fandoms, series (+ series_page order),
and each image's stored tagger_predictions (so suggestions repopulate
immediately). One set-based DELETE FROM tag WHERE kind IN ('general','character')
— the five tag-referencing tables all cascade, so applications + aliases +
allowlist + rejections + centroids clear automatically; series tags aren't
deleted so series survive; Tag.fandom_id is SET NULL so fandoms are untouched.

Reuses the established dry-run-preview -> confirm pattern: cleanup_service.
reset_content_tagging() + POST /api/admin/tags/reset-content +
TagMaintenanceCard section with a backup-first warning and a red confirm
showing exact counts (tags by kind + image applications). Irreversible except
via DB backup restore; the wipe only fires when the operator confirms.

Tests: service dry-run counts + live delete preserves fandom/series/series_page
while content tags + their image_tag cascade away; API dry-run wiring.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-04 16:47:36 -04:00
bvandeusen 914033db29 feat(maintenance): scheduled + manual DB VACUUM ANALYZE + bloat readout
CI / lint (push) Failing after 3s
CI / backend-lint-and-test (push) Successful in 22s
CI / frontend-build (push) Successful in 23s
CI / intimp (push) Successful in 3m35s
CI / intapi (push) Successful in 7m53s
CI / intcore (push) Successful in 9m15s
The TABLESAMPLE showcase reads physical blocks (bloat-sensitive), and the
periodic prune/backfill/recovery tasks churn dead tuples faster than
autovacuum always keeps up — so explicit maintenance earns its keep here.

- tasks.maintenance.vacuum_analyze: VACUUM (ANALYZE) over high-churn tables
  (VACUUM_TABLES) on an AUTOCOMMIT connection (VACUUM can't run in a txn).
  Scheduled weekly via Beat; also operator-triggerable.
- _sync_engine.get_sync_engine(): expose the process engine for the
  autocommit connection.
- GET  /api/admin/maintenance/db-stats: per-table n_live/n_dead/dead_pct +
  last (auto)vacuum/analyze from pg_stat_user_tables — visibility, not a
  black box.
- POST /api/admin/maintenance/vacuum: enqueue the task on demand.

Tests: vacuum task runs + reports tables; db-stats shape; trigger queues.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-04 00:49:39 -04:00
bvandeusen 597c6d48d3 refactor(dry-B1): consolidate duplicated _bad error helper into api/_responses.py
8 blueprints each defined an identical _bad() (two variants: with/without
detail). Extracted error_response() into api/_responses.py; each blueprint
now imports it `as _bad` so call sites are unchanged. The detail-aware
canonical subsumes both variants. Left settings.py's distinct _bad_int and
the inline jsonify error sites (not duplicated helpers).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-28 13:10:04 -04:00
bvandeusen bf8eb4468f feat(tags): legacy-tag purge also catches source:* general tags
Operator-confirmed 2026-05-28: the BlenderKnight:* tags are `archive`
kind (caught by the kind purge), but the `source:patreon`-style tags
are IR's old `source` kind that fell back to `general` during migration
(FC's enum has no `source` kind) — so they can't be matched by kind.

Broadened the purge to a two-rule match and renamed it for accuracy
(all dev-only, unreleased):
- cleanup_service.purge_tags_by_kind → purge_legacy_tags. Predicate is
  now `kind IN (archive, post, artist) OR name LIKE 'source:%'`
  (LEGACY_NAME_PREFIXES). Preview classifies each row into by_kind OR
  by_prefix (source:* counts once under the prefix bucket regardless of
  its general kind).
- endpoint /tags/purge-retired-kinds → /tags/purge-legacy. dry-run
  returns by_kind + by_prefix + count + sample.
- store purgeRetiredKindTags → purgeLegacyTags.
- Tag Maintenance card copy + breakdown updated to show both buckets;
  button reads "Preview/Delete legacy tags".

Tests updated + extended: dry-run reports by_kind {archive,post,artist}
AND by_prefix {source:*}, plain general/character tags survive; commit
deletes both the kind-matched and source:*-matched rows and leaves the
rest.
2026-05-28 01:20:41 -04:00
bvandeusen e1fc65bd1b feat(provenance+tags): collapse provenance descriptions; purge retired-kind tags
Operator-flagged 2026-05-28, two asks.

**1. Provenance posts ate the panel.** Each post card rendered its full
description (180px scroll box) inline, so a few posts pushed everything
else off-screen. ProvenancePanel now collapses the description by
default behind a per-post "Show description ▾ / Hide ▴" toggle (state
keyed by provenance_id, reset when the viewed image changes). Cards stay
compact — platform/date/title/meta/actions — and the operator expands
only the descriptions they want.

**2. Purge tags of retired/system kinds.** The IR migration left
`archive`/`post`/`artist`-kind tags (e.g. `BlenderKnight:Hannah_BJ_Loops`)
that FC no longer creates — the tag input only makes
character/fandom/series/general, and provenance + artists are their own
systems now. (meta/rating were already hard-deleted by alembic 0023.)

- cleanup_service.purge_tags_by_kind(kinds=PURGEABLE_TAG_KINDS) — counts
  (dry_run) or deletes tags whose kind ∈ (archive, post, artist).
  CASCADE clears the image_tag / alias / allowlist / etc. rows.
- POST /api/admin/tags/purge-retired-kinds (Tier-A, dry-run preview
  returns per-kind counts + sample names — the preview IS the
  verification of exactly what'll be deleted before committing).
- Tag Maintenance card gets a second section: "Preview retired-kind
  tags" → per-kind breakdown + sample → "Delete N retired-kind tag(s)".

Tests: dry-run counts by kind (general survives), commit deletes only
the retired kinds (general + character survive, retired count → 0).

NOTE: a dry-run preview will show exactly which kinds/counts are
present. If the operator's noisy tags turn out to be `general` (e.g. an
IR `source:patreon` that fell back to general during migration), they
won't be caught by the kind purge — the preview makes that visible so
we can decide on a name-based pass separately.
2026-05-28 01:12:20 -04:00
bvandeusen df6d89cb59 fix(secure-context): full audit — DestructiveConfirmModal.expectedTokenOverride + bulk-delete + min-dim use backend-computed tokens
Operator-flagged 2026-05-27: walk the whole project for the same shape
as the min-dim Delete-button silent failure (crypto.subtle TypeError
on plain HTTP). FC runs over plain HTTP per the homelab posture;
Secure-Context-gated browser APIs are undefined on the production
origin.

**Audit results across `frontend/src/`:**

  crypto.subtle.digest        — 2 sites:
    - MinDimensionCard (fixed 2026-05-27)
    - BulkEditorPanel (THIS FIX)
  navigator.clipboard         — 1 site, already guarded:
    - utils/clipboard.js writeText with execCommand fallback
  serviceWorker / mediaDevices / Push / Web USB|HID|Bluetooth|Serial /
  cookieStore / queryLocalFonts / WebAuthn / geolocation
                              — NOT USED, nothing to fix

  Extension scripts (background.js) use crypto.subtle but run from
  moz-extension:// which IS a Secure Context — left as-is.

**BulkEditorPanel double bug**

The bulk-delete UI on the gallery selection had been broken since
FC-3k shipped, in two ways:

1. `crypto.subtle.digest` swallowed TypeError on plain HTTP — modal
   never opened. Same symptom as min-dim.
2. Even on HTTPS, the modal's `kind="images-selection"` produced
   `delete-images-selection-<sha8>` while the backend expected
   `delete-images-<sha8>`. The two would never match.

Fix:

- Backend `/api/admin/images/bulk-delete` dry-run response now returns
  `confirm_token` (the canonical `delete-images-<sha8>` string).
  Integration test `test_bulk_delete_dry_run_returns_counts` pinned to
  assert the new field.
- DestructiveConfirmModal gains an `expectedTokenOverride` prop. When
  set, it bypasses the `${action}-${kind}-${runId}` formula and uses
  the explicit string. This decouples the UI label (`kind`) from the
  wire-format token (server-provided), so future endpoints can use a
  kind-specific label without their kind name leaking into the token.
- BulkEditorPanel passes `:expected-token-override="bulkProjected?.confirm_token"`
  — no client-side crypto, no kind-prefix mismatch.
- MinDimensionCard refactored to the same explicit pattern (was
  slicing the 8-char suffix off the backend's token and passing it
  through `runId`; now passes the full backend token via
  `expected-token-override` directly). Cleaner; one source of truth.

**Banked memory**

`feedback_no_secure_context_apis.md` documents the full table of
Secure-Context-gated APIs, which ones FC currently uses, and how each
is handled. Indexed in MEMORY.md. Sites for the audit also listed in
the memory for future drift-checking.

No other Secure-Context-gated APIs found in `frontend/src/`. The same
shape won't recur unless someone adds a new dependency on one — at
which point the banked memory should fire.
2026-05-27 21:17:40 -04:00
bvandeusen 676a86b514 fc3k: /api/admin tags prune-unused endpoint — Tier-A preview-then-commit flow
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-25 00:42:20 -04:00
bvandeusen 44cc625d4a fc3k: /api/admin tag endpoints — Tier-B delete + merge + usage-count helper
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-25 00:42:03 -04:00
bvandeusen f7ee122243 fc3k: /api/admin blueprint — Tier-C artist cascade + bulk image delete with sha8-keyed confirm tokens
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-25 00:40:53 -04:00