47 Commits

Include renames

Author	SHA1	Message	Date
bvandeusenandClaude Opus 5	235393c08b	feat: link the Patreon teaser to the Discord drop it announced (388 step E5) ... The point of the milestone rather than its tail. Two of the operator's artists post a deliberately cropped fragment on Patreon to signal that the real thing has landed in their Discord; this proposes those pairs. Confirm-only, following the FC-6.3 series matcher. A wrongly-asserted association tells the operator two different pieces are one, which is strictly worse than no link: no link leaves them where they already were, a wrong one actively misinforms and then propagates into whatever reads it. So the matcher's job is a SHORT list worth reading, not a long list worth trusting. **The threshold sits above every single signal weight, and that is the design.** Proximity is 0.55, declaration 0.45, the cut 0.60 — so neither signal can carry a pair alone. That makes "time proximity alone is never sufficient" an arithmetic property rather than an aspiration: on a busy day an artist posts several times, and a matcher that could pair on proximity alone would turn every one of those days into false pairs until the review queue got abandoned. A guard test asserts the relationship against WEIGHTS directly, so it survives any refactor of the scorer, and says in its own failure message not to fix it by lowering the assertion. **Crop-to-source matching is HELD, on the plan's instruction** — real work with real false-positive risk, worth building only once signals 1 and 2 are shown insufficient against the operator's actual artists. Worth stating: a naive whole-image SigLIP similarity is NOT that signal. A cropped teaser and its full version are precisely the pair a whole-image comparison handles worst, so adding one as a "bonus" would mostly add noise while looking like progress. Two premises in the plan corrected in the building: * **E4 is not actually a prerequisite.** A Patreon Source and a Discord Source the operator has added under one Artist already share `Post.artist_id`, and the synthetic grouping inherits it. E4 EXTENDS this to creators FC has to learn the association for; it is not needed to represent one FC was told. Same-artist is then a hard filter, not a scored signal — two different creators posting minutes apart is a coincidence, not evidence. * **`link_extract` cannot supply the declaration signal.** It exists, but `SUPPORTED_HOSTS` is file hosts only and `host_for()` returns None for a Discord URL, so no ExternalLink row is ever written for one. The signal reads the post body directly instead. And a bug my own test would have caught: `declared_signal` stripped the HTML before looking for an invite, but `html_to_plain` discards attributes and these creators put the invite in an anchor's `href` — so the strongest form of the signal was being thrown away, leaving only whatever the link text said. The invite now matches the raw body; the bare mention still matches stripped text, so `\bdiscord\b` is tested against prose rather than against markup. Dismissed rows are kept, not deleted: the row is what remembers the rejection, and re-proposing a rejected pair on every scan is the one behaviour that makes a review queue get ignored. Both FKs CASCADE, so E3's one-DELETE reversal cannot leave a proposal pointing at a post that no longer exists. Only ACCEPTED links reach the post payload. A pending proposal is a question for the review queue, not a claim to render beside the artwork. UI (rule 27) follows in the next commit. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LNXXULQDjVZmbuNa2G9mD9	2026-09-10 11:47:28 -04:00
bvandeusenandClaude Opus 5	1e45e2c56c	feat: an open grouping — a later drop joins its post (milestone 388 step E3) ... A synthetic post is no longer sealed at creation. A creator who adds two more variants the next day extends the existing post, its body grows with the new messages, and no rival post appears. That is what makes chat capture read as content trickling in rather than as a stream of separate arrivals. The sweep now runs two passes per source and the ORDER is load-bearing: offer new messages to still-open groups BEFORE founding new ones, because whichever runs first claims a message. E3's three named problems, each answered rather than discovered later: **Bridging.** A candidate near two groups joins NEITHER. Nearest-wins would silently make an arbitrary choice between two posts the operator may already have seen; merging them is worse still, because a merge rewrites history and anything pointing at the absorbed post dangles. Leaving it to found its own group is the recoverable failure. AMBIGUITY_MARGIN is a module constant and deliberately not a setting — it is not a quality dial anyone would tune toward a better feed, and exposing it would invite turning it to zero, which is exactly the silent arbitrary choice it prevents. **Re-surfacing without thrashing.** A grouping has two dates, and which one orders the feed is a real decision, so the feed orders by neither directly. Ordering by when the drop STARTED buries a group that grows a week later under a week of other posts — defeating the point of keeping it open. Ordering by every growth lets a group gaining one image a day live permanently at the top, so chat out-competes authored posts for the front page — the opposite of "post pacing stays front and centre". Instead `resurfaced_at` moves only when growth clears BOTH a minimum-images bar and a cooldown, so a drip-feed updates in place and a genuine second wave resurfaces exactly once. It is NULL on every ordinary post, so the sort key COALESCEs through it without moving anything that is not a grouping. **Reopening forever.** Groups close after a quiet period — artists reuse characters for years, and a group left open indefinitely will eventually absorb something it shouldn't. Openness is DERIVED, not stored: a group is open if it grew (or started) within the window. Lowering the setting closes old groups and raising it reopens them, with nothing to repair either way; a stored closed_at would have needed a sweep to set it and a repair path to ever change the policy. Rule 89 is satisfied structurally rather than by a parallel mechanism: celery_signals writes a TaskRun for every task, which already supplies duration, the 5-minute stalled-run recovery, and retention pruning. What this step owed on top of that was a wall-clock limit (present) and idempotence — re-running the joiner adds nothing, asserted directly rather than left to the unique (image, post) constraint to catch. Two bugs fixed in the writing, one of which my own test would have hit: * `assign_to_group` sorted bare (distance, Post) tuples, which falls through to comparing Posts when two distances tie — and a perfectly symmetric bridge, the exact case the function exists for, would have raised TypeError instead of declining to choose. Now keyed on the distance alone. * The cursor was still built from `post_date or downloaded_at` while the ORDER BY had gained `resurfaced_at`. Two expressions that disagree at a page boundary don't error, they silently skip or repeat rows; both sites now go through one `_post_sort_value`, and a test pages through one row at a time to prove the walk matches the whole list. Image linking is now one shared helper rather than written twice, because creation and joining would otherwise be free to drift on exactly the detail (which post owns the image) that makes a grouping reversible. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LNXXULQDjVZmbuNa2G9mD9	2026-09-10 11:30:17 -04:00
bvandeusenandClaude Opus 5	73eeb7a377	feat: FC authors the post that Discord never wrote (milestone 388 step E2) ... Discord is a delivery channel, not a publisher. One message is not one post, and today every message lands as its own `post` row, so chat lines compete with authored work for the same surface. Rather than demote them into a second-class feed, FC now writes the post itself: one row per DROP, its images the drop's images, its body the messages' text in arrival order. Synthesising a `Post` (rather than inventing a parallel entity) is the whole point — the result is post-shaped by construction, so feed, provenance, translation, attachments and series keep working on it unchanged. The predicate is three axes ANDed, and the time one does the real work: same source AND cosine distance <= threshold AND no gap > window Similarity alone over-groups, and that is the failure that would make this useless: any two pieces of the same character by the same artist sit close in SigLIP space, so a cosine-only rule collapses a month of one character into a single "post". Two details inside the predicate are load-bearing — * distance is measured to the group's SEED, never to the previous member, because chaining lets a group DRIFT: twenty small steps walk from one piece to a completely different one, each hop individually within threshold; * the window is measured between CONSECUTIVE messages, not from the first, so an artist trickling variants out over an evening stays one drop. Why a post-import sweep and not part of ingest. The obvious alternative was to migrate Discord to the native post-first ingester (#1266) and group at capture time. That cannot work: the grouping signal is `siglip_embedding`, which is produced asynchronously AFTER import (tasks/ml.py, the GPU backfill), so at capture time there is nothing to group on. Grouping is necessarily something that happens once the vectors catch up — hence a re-runnable sweep that skips what it cannot yet place, and an hourly (not daily) cadence. The honesty rule, enforced in the schema. `post.synthesized_by` names the grouper; `synthesis_details` records the members, the count, and the thresholds AS THEY WERE (they are operator-tunable, so without that "why did it group these" is unanswerable a month later). Member posts are absorbed, not destroyed — they remain the images' true origin and the audit trail — and `absorbed_by_post_id` is ON DELETE SET NULL, so deleting a synthetic post releases its members back into the feed in one DELETE with no repair step. `post_title` stays NULL deliberately: a synthesised title is the one place this could put words in a creator's mouth. Two guards the first draft would have failed: * the per-run cap took the lowest post IDs, not the oldest posts — DISTINCT ON forces its own ORDER BY, so the sort now happens outside the subquery; * a cap landing mid-drop would have published a truncated group claiming to be a whole drop, so the last group is left for the next run. And one vacuous test caught before it shipped: the support vector perturbed a single component of an all-ones vector, moving it ~1e-6, so every distance assertion passed regardless of what the predicate did. `_vec` now builds a unit vector at a stated angle, where distance is exactly 1 - cos(delta) — rule 167, a guard has to be able to fail. UI (rule 27) follows in the next commit. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LNXXULQDjVZmbuNa2G9mD9	2026-09-10 11:15:14 -04:00
bvandeusenandClaude Opus 5	2e0f8f8c61	feat(cleanup): reclaim orphaned attachments — rows and store blobs (#3068) ... PostAttachment's two FKs are both ON DELETE SET NULL, so a deleted post or artist left the row behind rather than taking it. Nothing ever pruned those rows, and nothing in the repo had ever unlinked a file under the attachment store — so both rows and bytes accumulated permanently, invisible to every existing diagnostic. Why a disk->DB reconciliation rather than a row sweep: the store is sha-addressed and idempotent, so ONE blob backs MANY rows. Deleting a row does not free its blob, and since the artist cascade (#3066) now deletes its attachment rows outright, a freed blob has no DB pointer left to find it by. Walking the store and asking "does any row still reference this sha?" catches orphans from every cause, including ones no future delete path will think to report. Preview and apply share `_orphan_attachment_conditions` (rule 93). The dry-run derives its surviving-sha set by NEGATING that same predicate, so it is honest about blobs the delete would free rather than counting them as still-referenced — the one place this was easy to get backwards, so it has its own parity test. Guards, each with a reason: - A blob is written before its row commits, so a just-stored file legitimately has no referencing row. Files under 6h are never judged — same guard and reasoning as ORPHAN_TEMP_MIN_AGE_HOURS. - `.partial` staging files belong to cleanup_orphaned_temp_files; skipped rather than raced. - The sha is parsed as the first 64 chars, not via Path.stem: store() takes the extension from the source filename, and a URL-encoded basename yields a multi-dot suffix that would make stem eat part of the sha. - A 900s walk budget reports partial=True instead of running to the task's hard limit (rule 89). - TASK_STUCK_THRESHOLD_MINUTES override at 30 (= time_limit 25 + 5). Without it a healthy 20-minute walk is phantom-flagged 'RecoverySweep' at the bare 5-min default — the #883 failure class; its invariant test is mirrored here. Defaults to the safe preview at both the task and the route, unlike the other maintenance triggers: this apply unlinks files. Operator-triggered only, never on a beat. Ships with its UI (rule 27): AttachmentReclaimCard in Cleanup → Duplicates & leftovers, built on the existing useMaintenanceTask/MaintenanceTile shapes, so a run survives navigating away. Surfaces files_failed and partial explicitly, since both change what the numbers mean. Also promotes humanBytes to utils/bytes.js — it was byte-identical in VideoDedupCard and GatedPurgeCard and this card would have been the third copy. The three divergent `formatBytes` helpers are deliberately left alone. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-08-26 22:37:24 -04:00
bvandeusenandClaude Opus 4.8	c87f8a1bb3	refactor(maintenance): DRY the recover_stalled head-run twins (#161) ... recover_stalled_head_training_runs and recover_stalled_head_auto_apply_runs were near-exact copies (coalesce-flip + keep-last-N prune, differing only in model + two constants). Extract _recover_stalled_runs(model, stall_minutes, keep_runs, label); the two tasks become thin wrappers. The other two recover tasks are deliberately NOT folded in (library-audit has no prune tail; backup uses a single started_at cutoff). test_recover_stalled_head_runs.py covers both wrappers (stalled→error, fresh survives) — previously untested. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NsmJSQxnNxGgtM5Yz4GAqi	2026-07-13 21:41:34 -04:00
bvandeusenandClaude Opus 4.8	af0d39ed52	feat(wip): soft title tier — sketch/doodle vocab + ring-loud audit (#1474) ... Extends WIP title-tagging to lower-precision cues (sketch/doodle/scribble) safely. - wip_title.py: soft matcher (word-anchored; sketchbook/kadoodle don't trip it); WIP_TITLE_SOFT_SOURCE + soft SQL prefilter; apply_wip_image_tags takes a source arg. - training_data._AUTO_SOURCES += 'wip_title_soft' → the soft tier is PROVISIONAL and never trains the wip head (a finished "sketch" can't pollute it). Only the hard tier (wip_title) + manual train. - ImportSettings.wip_soft_title_tagging_enabled (OFF by default, opt-in). Migration 0087. - importer: hard tier wins, soft is the fallback (source wip_title_soft). - backfill: refactored into a shared _backfill_wip_tier; hard always, soft when enabled. - heads.soft_wip_conflict_audit + daily beat: score soft-tagged images against content heads, flag ring-loud ones (PresentationReview mode=process) for the review strip — the operator's "measure if they got falsely tagged" safety. - api settings toggle; ImportFiltersForm soft toggle. - tests: soft matcher pos/neg; soft source not a training positive; audit flags ring-loud + spares quiet. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-07-13 10:14:38 -04:00
bvandeusenandClaude Opus 4.8	571938781a	feat(tagging): title-based WIP auto-tagging (#1458) ... Auto-apply the `wip` system tag to posts whose TITLE explicitly declares work-in-progress ("WIP" / "work in progress") — a deterministic, high-precision complement to the image-based ML `wip` head. WIP images are excluded from the Explore/gallery browse, so honouring the artist's own label keeps unfinished pieces out of the main browse. - services/wip_title.py: precision-first token-anchored matcher (swipe/wiped never trip it) + sync apply helpers (source='wip_title', ON CONFLICT DO NOTHING, chunked under the psycopg param ceiling). - importer: live hook on FRESH import only (never on deep-scan/supersede), so a manually-removed WIP tag is never re-applied by a routine re-scan. - maintenance.backfill_wip_title_tags: operator-triggered back-catalogue sweep (coarse SQL prefilter + regex confirm, keyset-paginated). Deliberately NOT a beat — a periodic re-run would silently undo manual removals. - ImportSettings.wip_title_tagging_enabled (default ON, migration 0085) gating the live hook; GET/PATCH + POST /settings/wip-title/scan. - Settings UI: toggle + "Scan existing posts" button. - Tests: pure matcher unit tests + integration (apply idempotency, backfill precision). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-07-12 12:12:19 -04:00
bvandeusenandClaude Opus 4.8	0b78264d62	feat(maintenance): daily janitor for orphaned .part/.partial staging files ... Downloads/imports stage into <name>.part / <name>.partial then os.replace() into place, so a kill mid-write leaves a discardable temp — never a corrupt final. cleanup_orphaned_temp_files sweeps ones left behind under the images root, only older than 6h so an in-flight download's staging file is never removed. Daily beat. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-07-07 22:15:33 -04:00
bvandeusenandClaude Fable 5	5b34c9221c	feat(ia): wave 1 — Import tab dissolves, Maintenance regroups by system, one extension home ... Settings IA per the approved A3 design (the old layout was the two-app merge fossilized): - Import tab retired: ImportTriggerPanel + ImportTaskList deleted (manual /import scans stay API-level; imports arrive via downloads/extension, heal via the Layer-2 auto-refetch sweep, and show in Activity). ImportFiltersForm moves to Maintenance → 'Ingestion & filters' and loads its own settings; the import store shrinks to settings-only (no remaining consumers of the scan/task-list machinery). Overview's pending banner now points at Activity. - Maintenance regrouped: Ingestion & filters / GPU agent & embeddings (GpuAgent, Failed processing, CPU embedding backfill) / Tagging (sliders, Heads, Aliases) / Library health (MissingFiles, Thumbnails, DB, Archive re-extract demoted last) / Storage. - One extension home: BrowserExtensionCard moves from Settings → Overview to Subscriptions → Settings, above the API key bar it authenticates. - Single-color import filter WIRED: skip_single_color/threshold existed since FC-2 but nothing read them (the audit module's docstring said as much) — now enforced on both import paths via the audit's canonical predicate (tolerance 30, matching the Cleanup card default; animated images exempt like the transparency check). Default stays off; test added. - Dead weight: PlaceholderView (zero refs) and the permanently-disabled 'Export failed logs (CSV — v2)' menu stub deleted; stale docs fixed (celery queue docstring, threshold comment citing retired tasks, ml package docstring, HeadsCard 'replaces Camie' blurb). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CDgx8bQS5YrGRK76v8HUnM	2026-07-02 17:37:21 -04:00
bvandeusenandClaude Fable 5	19b962f1a7	feat(b3): ml-worker becomes optional — embed-only role, decoupled GPU coordination, cpu-embed switch ... The ml-worker's ONLY processing role is now the CPU whole-image embed fallback (tag_and_embed renamed embed_image — Camie tagging was retired #1189 and the name kept implying otherwise; videos were already handled agent-style: frame sampling + mean-pool). Detection/cropping/CCIP stay GPU-agent-only, and their completion is judged per-pipeline: ccip by gpu_job rows, siglip by concept regions at the current model version — never by image_record.siglip_embedding. A CPU embed therefore can NEVER close crop work for the agent (regression test pins this; only the whole-image 'embed' job, the same artifact, is satisfied). Making removal actually safe (operator will drop the container): - GPU-queue coordination (enqueue_gpu_backfill, recover_orphaned_gpu_jobs, reprocess_gpu_jobs) moved verbatim to tasks/gpu_queue.py on the maintenance quick lane — it lived on the 'ml' queue only by module colocation, which made the ml-worker a hard dependency of the whole agent pipeline. - New ml_settings.cpu_embed_enabled (migration 0074, default ON so agent-less installs keep working): OFF stops the four import hooks queueing embed work nothing will consume and no-ops the manual backfill; switch lives on the renamed 'CPU embedding backfill' card. - NB heads training / auto-apply still run on the ml image (sklearn) — a stack that removes the container gives those up too. Deploy note: in-flight messages under the old task names are dropped by the new workers; the 60s orphan sweep + hourly backfill re-fire under the new names immediately. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CDgx8bQS5YrGRK76v8HUnM	2026-07-02 16:53:08 -04:00
bvandeusenandClaude Fable 5	eaea4308fc	chore: retire the tag-eval harness — it proved the heads system, job done (operator-approved) ... The head-vs-centroid eval (#1130) existed to prove the 'frozen embedding + trained head' spine; the operator accepted the tagging system and dropped the harness. Removed per rule 22: TagEvalCard + store, /api/tag_eval blueprint, tag_eval_run ml task, recover-stalled-tag-eval-runs sweep + beat entry, TagEvalRun model + table (migration 0073), and its tests. The eval's data loaders + metric helpers were NOT eval-specific — the nightly heads trainer runs on them — so they moved verbatim to services/ml/training_data.py (heads.py import updated; behavior unchanged). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CDgx8bQS5YrGRK76v8HUnM	2026-07-02 12:41:24 -04:00
bvandeusenandClaude Fable 5	a7abcc41ca	feat(triage): failed-processing triage — probe errored files, flag defects, recover (#125 C1-C3) ... An errored GPU job's stored reason is a suspicion; the file probe is the verdict. A 15-min beat sweep (triage_gpu_errors) runs verify_integrity's own probe (sha256 + decode) on each errored image ONCE and writes both verdicts: ImageRecord.integrity_status and the new GpuJob.triage_status ('defect' | 'file_ok', migration 0072). Every classification logs at WARNING so it surfaces in Logs/System Activity. - 'defect' rows are excluded from /retry_errors (re-running a known-bad file burns agent time re-minting the tombstone); response now reports defects_kept and the GpuAgentCard toast says so. - GET /api/gpu/errors: triage view — reason buckets (classify_reason), probe verdicts, per-job detail. POST /errors/triage runs the sweep now. - POST /api/gpu/errors/<id>/recover: reuses the Layer-2 refetch pattern — delete the defective copy + record (full cascade takes the tombstones too) and re-poll its subscription Source so a fresh copy re-imports and re-enters the pipeline; 'no_source' when nothing pollable resolves. - New 'Failed processing' card (GpuTriageCard) in Maintenance: verdict counts, reason summary, probe-now, defect list with thumbnails + per-image Recover. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CDgx8bQS5YrGRK76v8HUnM	2026-07-02 12:36:02 -04:00
bvandeusenandClaude Opus 4.8	9326a82b29	fix(heads): .all() before dict() in snapshot_head_metrics ... dict(session.execute(...)) on a bare Result invokes the mapping protocol (a Result has .keys() = column names) and subscripts it → "CursorResult is not subscriptable". Materialize with .all() so dict() consumes rows as key-value pairs. The API path already did this; the snapshot task missed it. Caught by test_snapshot_records_timeseries_point (run 1628). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Ttrj5P7upUTueSfoJcxEqa	2026-06-29 00:42:33 -04:00
bvandeusenandClaude Opus 4.8	48c8811d69	feat(heads): auto-apply observability + on by default (#114 auto-apply B) ... Auto-apply is now ON by default (operator-asked: opt-OUT, not opt-in) — migration 0059 + model default flipped. The support (>=30) + measured-precision gates keep it safe and every auto-tag is reversible. Observability so the operator can tune from real data: - MISFIRE = an auto-applied (source='head_auto') tag the operator later removes. UNDER-FIRE = a tag with a head the operator adds by hand (the head missed it). Both captured at correction time in TagService.add_to_image/remove_from_image (source is lost on delete) into durable per-tag counters (head_metric), keyed by tag so they survive head retrain/prune. - Daily snapshot_head_metrics writes a per-concept time-series point (head_metrics_snapshot): auto-applied volume + cumulative misfires/under-fires + head quality; 180-day retention; daily beat. - GET /api/heads/metrics: per-concept current counts + realized misfire rate + head quality, plus the snapshot time-series — the report to tune the precision target + support floor. Migration 0060. Tests: misfire/under-fire counting (and the negatives — manual removal isn't a misfire, headless manual add isn't an under-fire), snapshot time-series, metrics API. What's the autofire threshold? There's no single number — each graduated head derives its OWN probability cutoff from its PR curve: the operating point that holds precision >= head_auto_apply_precision (0.97) at max recall. The global knobs are that target + the >=30 support floor. NEXT (slice 3): UI — enable toggle, dry-run preview, per-concept trends. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Ttrj5P7upUTueSfoJcxEqa	2026-06-29 00:36:58 -04:00
bvandeusenandClaude Opus 4.8	74fef908d2	feat(heads): earned auto-apply — sweep mechanism, off by default (#114 auto-apply A) ... Graduated heads can now apply their tag without a human — gated so it's safe: - FIRING GATE: a head fires only when the master switch (head_auto_apply_enabled, default OFF) is on AND it has >= head_auto_apply_min_positives (default 30) clean labels. A precise-looking but under-supported low-N head can't spray tags. - auto_apply_sweep (heads.py): streams every embedded image in chunks, scores against the eligible heads (numpy, no sklearn), applies each head's tag where score >= its auto_apply_threshold and the tag isn't already applied/rejected, with source='head_auto' (distinguishable + reversible). dry_run counts only. - HeadAutoApplyRun (migration 0059) tracks each sweep / preview; apply_head_tags task (ml queue) + scheduled_apply_head_tags daily beat (no-op unless enabled) + recovery sweep + retention(20). - API: POST /api/heads/auto-apply {dry_run} (202 / 409 running / 400 disabled), GET /api/heads/auto-apply (recent runs + per-concept report). Settings head_auto_apply_enabled + min_positives via /api/ml/settings. Tests: sweep applies above threshold, dry-run writes nothing, skips under- supported + ungraduated heads; API disabled/dry-run/conflict guards. NEXT (slice 2): the observability the operator asked for — per-concept misfire (auto-applied-then-removed) + under-fire tracking, time-series snapshots, and a reporting API to tune. Slice 3: the UI (enable, preview, trends). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Ttrj5P7upUTueSfoJcxEqa	2026-06-29 00:22:54 -04:00
bvandeusenandClaude Opus 4.8	1ed0895e8d	style(heads): fix import ordering (ruff I001) ... Alphabetize HeadTrainingRun in models/__init__ + maintenance imports (H before I), and drop the inline comment that split heads.py's import block. Pure import ordering — no behavior change. (run 1601 lint) Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Ttrj5P7upUTueSfoJcxEqa	2026-06-28 10:41:12 -04:00
bvandeusenandClaude Opus 4.8	22c3b54746	feat(heads): production per-concept heads — train + score backend (#114 A) ... The eval (#1130) proved the frozen-embedding + trained-head spine; this lands its production form (the first of three slices that make heads the suggestion source, replacing Camie + centroid). - tag_head: one logistic-regression head per general/character concept with enough labelled positives. Weights (pgvector), honest CV-derived suggest threshold + earned-auto-apply point, and per-concept quality metrics. - head_training_run: persisted batch lifecycle (mirrors tag_eval_run) so the admin card shows live + historical status across navigation. - services/ml/heads.py: TRAIN (sync, ml worker, reuses tag_eval's proven data loaders + metric math so production heads match measured eval numbers) and SCORE (async, API worker — numpy via pgvector, no scikit-learn): score one image's embedding against all heads → the rail's suggestions, cached on (count, max trained_at) so a retrain invalidates without per-request loads. - tasks.ml.train_heads (ml queue, commits per head so a kill leaves progress) + recover_stalled_head_training_runs sweep + retention(20) + 5-min beat (rule 89). - api/heads.py: POST /api/heads/train (one run at a time, 409 guard) + GET /api/heads (count, graduated, last-trained, running, per-concept table, recent runs). - ml_settings: head_min_positives + head_auto_apply_precision, tunable via /api/ml/settings. Scoring isn't wired into the rail yet (slice C) and the admin UI is slice B — this slice makes training + scoring exist and CI-verifiable. 'precision' column stored as precision_cv (SQL reserved word). Migration 0058. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Ttrj5P7upUTueSfoJcxEqa	2026-06-28 10:36:25 -04:00
bvandeusenandClaude Opus 4.8	6e3c5f697f	feat(ml): tag-eval backend — head-vs-centroid learning-curve eval (persisted) ... Slice 1 of milestone #114 (tagging v2). Proves the frozen-embedding + trained- head spine on the operator's own data, reusing the SigLIP embeddings already stored on image_record — no re-embedding, no GPU. Per concept: train a logistic-regression HEAD (positives + negatives = explicit rejections + sampled unlabeled) vs the old single-CENTROID baseline; report cross-validated precision/recall/AP for both, a LEARNING CURVE (AP/F1 as tagged positives grow 10→30→100→300), and example image ids (head-would-suggest / head-doubts-positive) to eyeball. Persisted so the report SURVIVES navigation (operator-flagged): the run + full report live in a new tag_eval_run row (mirrors library_audit_run); the admin card will rehydrate from GET on mount, not transient state. - models.TagEvalRun + migration 0056; runs on the ml queue (only worker with numpy/sklearn) — numpy/sklearn lazy-imported so the API can still enqueue. - services/ml/tag_eval (compute + start helper, one-running guard), tasks.ml .tag_eval_run, api/tag-eval (POST create, GET history light / detail w/ report). - recover_stalled_tag_eval_runs sweep + retention (keep last 20) + 5-min beat (rule 89). scikit-learn added to requirements-ml. - tests: param normalization + the rehydrate read-path + create/conflict. Frontend admin card (trigger + render persisted report) follows next. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-06-27 22:49:10 -04:00
bvandeusenandClaude Opus 4.8	4272a19d40	fix(external): split fetch timeout into read (60s) + total (30m) budgets (#883) ... The single _FETCH_TIMEOUT=3000s meant different things per host: a TOTAL wall-clock for mega (subprocess), but only a per-read socket timeout for HTTP hosts (requests' timeout is the idle gap between bytes, never a total). So a stalled HTTP connection tied up a download-worker slot AND the per-host serialize lock for ~50 min before failing (operator-flagged 2026-06-17). Split into two limits in external_fetch: - read timeout (_READ_TIMEOUT=60s, with _CONNECT_TIMEOUT=30s) → requests gets (connect, read); a stalled socket now fails in ~60s. - total budget (_TOTAL_TIMEOUT=30min) → enforced as a wall-clock deadline across chunks in _stream_to_file (HTTP has no total-download timeout), and passed as the subprocess total for mega. fetch_external() signature: timeout= → read_timeout=/total_timeout=. gdrive (gdown) self-manages; the celery hard limit is the outer backstop. Also lowered the per-host lock TTL 3600→2400 so a worker that dies holding it can't wedge a host's links much past one fetch's budget. Each external link is already one Celery task (sweep enqueues one fetch_external_link.delay per link), so these budgets are per-link. Tests: total-budget-exceeded cleans the .part; HTTP gets (connect, read); mega gets the total. Worker fakes updated to **kwargs. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-06-16 21:15:49 -04:00
bvandeusenandClaude Opus 4.8	258c77dfcd	fix(maint): raise recovery-sweep threshold for fetch_external_link (#883) ... External file-host fetches run to a 60-min hard limit (time_limit=3600, per-fetch _FETCH_TIMEOUT=3000s), far longer than the recovery sweep's 5-min default. recover_stalled_task_runs was phantom-flagging healthy in-flight fetches as "RecoverySweep: no completion signal received within 5 min" before the task's own timeout/error handling could surface the real error (operator-flagged: target 414 swept at 6.6min). The sweep already has per-queue/per-task overrides for long tasks, but fetch_external_link was never added and its TaskRun records queue='default' (no queue override) despite external.* routing to download. Add a task-name override of 65 min (time_limit 60 + 5 buffer); task-name precedence makes it robust regardless of the recorded queue. No new internal timeout needed — the existing _FETCH_TIMEOUT + soft_time_limit + except-block log.exception already capture the real failure once the sweep stops preempting. Pinned tests: external-fetch override survives a 10-min row / flags a 70-min row on queue='default'; invariant guard asserts override >= hard time_limit. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-06-16 20:54:39 -04:00
bvandeusenandClaude Opus 4.8	9ba3db75fd	fix(maintenance): download queue needs a sweep threshold above its 25-min time_limit ... recover_stalled_task_runs used the 5-min default for the download queue, but download_source legitimately walks up to DOWNLOAD_HARD_TIME_LIMIT (1500s = 25m). Healthy in-flight Patreon/gallery-dl walks were flagged as phantom 'RecoverySweep' failures — visible in System Activity but absent from the Subscriptions view (the download finished ok, reset the source's consecutive_failures; only the orphaned task_run kept the stamp, since _finalize only updates rows still 'running'). Add download:30 to QUEUE_STUCK_THRESHOLD_MINUTES — clears the 25-min hard limit with buffer and matches DOWNLOAD_STALL_THRESHOLD_MINUTES so a real hard kill is swept by the task-run and event sweeps together. Restores the documented invariant (every override >= task time_limit). Regression test pins the threshold above the hard limit so a future limit bump can't silently re-break it. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-06-10 11:57:31 -04:00
bvandeusenandClaude Opus 4.8	daaa7543a8	fix(backup,tags): unwedge backups on NFS (#739) + tag-standardize "0 groups" (#740) ... #739 — DB backups hung on NFS in uninterruptible D-state, defeating the 12-min subprocess timeout AND Celery's hard limit, so a stuck pg_dump held the concurrency-1 maintenance_long lane for hours — starving normalize_tags, re-extract, audits, and the new series rescan (which is why #740 "never applied"). Three fixes: - _run_bounded: Popen + bounded post-kill reap; if the child is unkillable (D-state) we stop waiting and re-raise TimeoutExpired, freeing the slot. The orphan is reaped by the OS once its syscall clears. - backup_db dumps to a LOCAL temp file then moves the finished .sql to the (NFS) _backups dir — pg_dump's long phase is now a DB-socket wait + local writes (killable) instead of an NFS write that hangs. backup_images keeps bounded-kill (too big to stage locally). - recover_stalled_backup_runs: split the stall window — db 40 min (was sharing images' 7h), so a hung DB backup is flipped to error promptly. #740 — Standardize tag casing showed "0 groups to change" the instant it was clicked: onNormCommit overwrote the preview with zeros. Keep the real preview visible and disable the button while queued; backend apply was already correct. Tests: fake subprocess.Popen alongside run; bounded-kill fail-fast; local-temp target; per-kind stall sweep. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-06-07 19:20:16 -04:00
bvandeusenandClaude Opus 4.8	f2e9ae07dc	fix(audit): chunk + self-resume library scans (stop the 2h queue-hog timeouts) ... scan_library_for_rule ran one 2-hour pass that timed out on large libraries and held the concurrency-1 maintenance queue the whole time, starving vacuum/backup/ normalize (operator-flagged — it was the dominant entry in the 24h failures). It now runs ~10-min chunks and re-enqueues itself until the library is exhausted, matching the operator's preferred pattern (reasonable timeout → retry queued → other things process between). New columns (alembic 0039): resume_after_id persists the keyset cursor so a chunk continues where the last left off; last_progress_at lets the recovery sweep tell a progressing multi- chunk audit from a dead one (it now measures staleness from last_progress_at, not started_at). Matches accumulate across chunks. soft/hard limits dropped 2h→15/16.7 min so the in-chunk budget fires first; a soft-limit backstop re-enqueues to resume instead of erroring the whole run. Tests: time-box → re-enqueue (status stays running); resume carries prior matches and appends new ones. Existing full-scan tests unchanged (small sets finish in one chunk). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-06-07 00:08:19 -04:00
bvandeusenandClaude Opus 4.8	1f4ce8513b	style: ruff I001 — keep _sync_engine as-import on its own line (combine-as-imports=false) ... Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-06-04 00:59:26 -04:00
bvandeusenandClaude Opus 4.8	5a116ca9d0	style: ruff I001 — aliased _sync_session_factory sorts before get_sync_engine ... Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-06-04 00:55:18 -04:00
bvandeusenandClaude Opus 4.8	ef3ee5aceb	feat(maintenance): DB maintenance UI card + fix ruff I001 ... - Settings → Maintenance gains a "Database maintenance" card: a "Run VACUUM ANALYZE now" button (enqueues the maintenance task) plus a per-table bloat readout (live/dead/dead%/last vacuum) from /api/admin/maintenance/db-stats. - dbMaintenance store (loadStats / runVacuum) + test. - Fix ruff I001: combine the two _sync_engine imports onto one line. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-06-04 00:52:36 -04:00
bvandeusenandClaude Opus 4.8	914033db29	feat(maintenance): scheduled + manual DB VACUUM ANALYZE + bloat readout ... The TABLESAMPLE showcase reads physical blocks (bloat-sensitive), and the periodic prune/backfill/recovery tasks churn dead tuples faster than autovacuum always keeps up — so explicit maintenance earns its keep here. - tasks.maintenance.vacuum_analyze: VACUUM (ANALYZE) over high-churn tables (VACUUM_TABLES) on an AUTOCOMMIT connection (VACUUM can't run in a txn). Scheduled weekly via Beat; also operator-triggerable. - _sync_engine.get_sync_engine(): expose the process engine for the autocommit connection. - GET /api/admin/maintenance/db-stats: per-table n_live/n_dead/dead_pct + last (auto)vacuum/analyze from pg_stat_user_tables — visibility, not a black box. - POST /api/admin/maintenance/vacuum: enqueue the task on demand. Tests: vacuum task runs + reports tables; db-stats shape; trigger queues. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-06-04 00:49:39 -04:00
bvandeusenandClaude Opus 4.8	6590dcdb39	fix(download): salvage soft-time-limit kills + fix timeout ladder ... Backfill downloads stranded with empty logs + a generic "stranded by recovery sweep" error. Root cause: the backfill gallery-dl subprocess timeout (1170s) exceeded download_source's Celery soft_time_limit (900s), so SoftTimeLimitExceeded preempted subprocess.TimeoutExpired. The TimeoutExpired path (which captures partial stdout/stderr and finalizes the event) never ran, the event was left 'running', and phase 3 never decremented backfill_runs_remaining — so the source re-ran and re-stranded every tick (Anduo #39912). Two layers: 1. Raise download_source limits (soft 900→1350, hard 1200→1500) so both subprocess budgets (870 tick / 1170 backfill) sit below the soft limit with phase-3 persist headroom. Promote to module constants and guard the invariant with a test. 2. Catch SoftTimeLimitExceeded in download_source and finalize the in-flight event with a real reason, mirror phase-3 source-health, and decrement backfill so a chronically-slow source self-heals to tick mode. The existing celery_signals handler only covered TaskRun, not DownloadEvent — that was the gap. Updates stale 900/1200 references in gallery_dl.py + maintenance.py. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-06-03 16:56:13 -04:00
bvandeusen	e30f50e6fe	fix(audit-g3): lifecycle batch — recovery sweeps, retention, timeouts ... Plugs the FC long-running-entity discipline gaps the 2026-06-02 audit flagged: every entity that can get stuck now has recovery + retention + timeout, and the long-runners no longer collide with the FC-3i sweep. Recovery sweeps (every 5 min): - recover_stalled_backup_runs — flips BackupRun stuck in running/restoring past 7h (covers the 6.5h images-backup hard limit) to error. prune_backups docstring corrected — the FC-3i TaskRun sweep never touched BackupRun rows. - recover_stalled_library_audit_runs — flips LibraryAuditRun stuck past 135 min (10-min buffer above scan_library_for_rule's 2h5m hard limit) to error. Previously a SIGKILL'd row blocked all future audits until manual DB surgery. - recover_stalled_import_batches — finalizes ImportBatch rows stuck running >2h whose child tasks are all terminal (orphan case where the orchestrator crashed before the closing UPDATE). Uses the same EXISTS predicate /api/system/stats already had. Retention (daily): - prune_library_audit_runs — 30-day window. Audit rows carry matched_ids JSONB blobs that can hold tens of thousands of ids. - prune_import_batches — 30-day window. Cascades to ImportTask via the model relationship. time_limits on five long-runners that previously had none (the audit's headline finding — every one of these collided with the recover_stalled_task_runs 5-min default and could be marked 'error' mid-flight): - scan_directory: 60m soft / 70m hard - verify_integrity: 60m / 70m - backfill_phash: 30m / 35m - apply_allowlist_tags: 30m / 35m - recompute_centroids: 30m / 35m QUEUE_STUCK_THRESHOLD_MINUTES now covers maintenance (75) and scan (75) — above the longest task on each — with per-task overrides for the outliers (backup_images_task 420, restore_images_task 420, scan_library_for_rule 130). start_audit_run guard is now age-aware: a 'running' row older than the audit hard limit doesn't block a new run (the sweep will catch it within 5 min). Previously a SIGKILL'd row blocked forever. /api/import/status now uses the same EXISTS predicate /api/system/stats does, so the two endpoints no longer disagree on the active-batch question. DownloadEvent.started_at resets on pending→running so a freshly- promoted event from a busy queue isn't measured against its original enqueue time (was racing recover_stalled_download_events on heavy-queue days).	2026-06-02 14:30:46 -04:00
bvandeusen	3898ce7be4	fix(audit-g1): six one-liner drift fixes from 2026-06-02 audit ... - BACKFILL_TIMEOUT_SECONDS 1800→1170: keep the subprocess timeout 30s below Celery's hard time_limit=1200 so SIGKILL doesn't beat TimeoutExpired (matched the tick 870s/900s rationale). Backfill runs that hit the cap let the next tick continue via the archive. - recover_interrupted_tasks orphan UPDATE now stamps finished_at; without it cleanup_old_tasks' WHERE finished_at<cutoff never reaped orphan-swept rows. recover_stalled_task_runs also now sets duration_ms (matches celery_signals.finalize's millisecond math). - ExtensionService.quick_add_source arms NEW_SOURCE_BACKFILL_RUNS=3 on Source creation, mirroring SourceService.create. Without it, Firefox quick-add on a creator with >20 unsynced posts walked the full feed until subprocess timeout. Renamed the constant from _NEW_SOURCE_BACKFILL_RUNS so it can be imported cross-module. - gallery_dl.verify() accepts TIER_LIMITED as auth-success alongside NO_NEW_CONTENT — the download path (line 712) already does, and TIER_LIMITED proves auth reached the post and was told it was tier-gated. Verify endpoint previously showed red on this and prompted operators to rotate working cookies. - prune_unused_tags now runs a single DELETE with the NOT-IN predicate find_unused_tags uses, instead of SELECT-ids → DELETE-WHERE-IN. Removes the psycopg 65535-param cliff that would have surfaced on a tag explosion (>65k unused tags). - credentials.upload() reflects the returned record into the store cache (`.set(platform, rec)`) instead of evicting it; previously the card briefly rendered "no credential" between upload and loadAll().	2026-06-02 11:24:05 -04:00
bvandeusenandClaude Opus 4.7	e35fb1edf7	fix(scan): recovery sweep for stranded download events ... The scan tick (scan.py:_tick_due_sources_async) inserts DownloadEvent(status='pending') and fires download_source.delay(). If the task dies before finalizing the event — worker OOM/SIGKILL, lost task, or a gallery-dl that didn't unwind on the 1200s hard time_limit — the event stays in-flight forever. Every later tick then skips the source via the in-flight guard (scan.py:168), so Source.last_checked_at is never written and the operator sees "last check never" in the Subscriptions health column, permanently. cleanup_old_download_events only prunes terminal events (by design); no existing sweep covered the pending/running case. Operator confirmed 2026-05-29 with a diagnostic query: all 43 "never checked" sources were stranded behind stale in-flight events (eligible_stuck_inflight = 43, every other bucket zero). New recover_stalled_download_events task (Beat every 5 min): - Flips DownloadEvent rows pending/running > 30 min (10 min past the download_source 1200s hard kill, so legitimately-running tasks are never touched) to status='error' with a sentinel message. - Bumps each affected Source's consecutive_failures ONCE per source — backoff is 2^N on that counter so per-event bumps would needlessly inflate the next interval — sets last_error, stamps last_checked_at. UPDATE...RETURNING source_id avoids a SELECT-then-UPDATE-WHERE-IN that would hit the psycopg 65535-param ceiling on a large strand pile. Net: the 43 currently-stranded sources unstick on the first sweep after deploy, their health dots flip amber instead of unchecked, and the next scan tick re-queues them. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>	2026-05-29 21:40:52 -04:00
bvandeusenandClaude Opus 4.7	171c486939	refactor(dry-B2): ImportSettings.load()/load_sync() classmethods for the singleton row ... The `select(ImportSettings).where(id == 1)).scalar_one()` singleton load was repeated 15× across services, API, and 5 task modules. Added async load() + sync load_sync() classmethods on the model and migrated all 15 full-row sites (callers already imported ImportSettings, so no new imports; dropped download's now-orphaned select import). Left maintenance.py's deliberate column-select (import_scan_path only) as-is. Rest of the service layer was already adequately DRY — the Record/to_dict pattern is only 2 instances and the savepoint find-or-create recovery is correctly per-entity, so neither was forced into a shared abstraction. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>	2026-05-28 14:03:26 -04:00
bvandeusen	dcfe55d731	feat(import-resilience L2): one-shot re-download for corrupt downloaded files ... Layer 2 — remediate a corrupt file by re-fetching a fresh copy from its source, bounded to a single attempt. Operator-requested 2026-05-28. New backend/app/services/refetch_service.py: - resolve_refetch_source: parse the failed file's sidecar → platform, derive the artist from the import path, find an ENABLED Source with a real feed URL for (artist, platform). Returns None for filesystem-only imports, missing sidecars, or `sidecar:<platform>:<slug>` synthetic anchors (not pollable). - attempt_refetch: if not already refetched AND a Source resolves, delete the corrupt file (so gallery-dl's skip_existing re-fetches it), set ImportTask.refetched=True, and trigger ONE download_source re-check. Bounded by `refetched` so source-side corruption can't loop. Wiring: - Manual endpoint POST /api/import/tasks/<id>/refetch (only on 'failed' tasks). Returns refetch_queued / no_source / already_refetched / not_found / not_failed. - Auto path in recover_interrupted_tasks: for each poison-pill row, if env FC_AUTO_REFETCH_CORRUPT=1, attempt_refetch (default OFF — the manual button is the primary path; auto is opt-in since re-fetch deletes a file + re-runs the downloader). - Frontend: a cloud-refresh icon button on failed rows in ImportTaskList → stores.import.refetchTask → toast keyed on the result status. Filesystem imports with no upstream return no_source — the operator's only remediation there is replacing the file on disk, surfaced clearly in the toast. Tests: 404 unknown task, 400 non-failed task, no_source when unresolvable, and the full resolvable-source path (file deleted, refetched flag set, one download_source dispatched, second call is a no-op). The resolvable test repoints the migration-seeded import_settings(id=1) scan path rather than inserting a conflicting row.	2026-05-28 00:08:03 -04:00
bvandeusen	e77afe8295	feat(import-resilience L1): poison-pill circuit breaker — cap stuck-task re-queues ... Layer 1 of the import-task resilience work (operator-requested 2026-05-28). The recover_interrupted_tasks sweep re-queues rows stuck in 'processing' — correct for a worker crash, but without a cap a row that RELIABLY hard-crashes the worker (OOM/segfault/SIGKILL on a corrupt or oversized input) loops forever: re-queue → crash → re-queue, burning a worker slot every 5 min. A caught exception flips to terminal 'failed' and never enters this loop; only process-killing inputs do. - alembic 0026: import_task.recovery_count (int, default 0) + import_task.refetched (bool, default false — backs Layer 2). - recover_interrupted_tasks now runs a poison-pill UPDATE FIRST: stuck rows whose recovery_count has already reached MAX_RECOVERY_ATTEMPTS-1 are marked 'failed' with a diagnostic ("crashed or stalled the worker N times … likely a corrupt or oversized input … inspect/replace the file, then retry via /api/import/retry-failed") instead of re-queued. The re-queue pass then handles the remaining stuck rows and bumps recovery_count. Shared stuck_predicate (and_/or_) keeps the media-5min / archive-40min split. - MAX_RECOVERY_ATTEMPTS=3 (two recoveries then give up). The failed poison pill surfaces in the existing import-failures view with its file path, directly answering "help me identify them." Test test_recover_interrupted_poison_pill_caps_at_max pins both branches: a row at the cap is failed (not re-enqueued, diagnostic present), a row one short is re-queued + incremented.	2026-05-27 23:54:35 -04:00
bvandeusen	a85880f965	fix(import): split archive imports into their own task + budget; archive-aware recovery sweeps ... Operator-flagged 2026-05-28: import_media_file on target 1645019 hit SoftTimeLimitExceeded at exactly 5.0 min. Their diagnosis was correct — the timeout covered the WHOLE archive, not per object. Importer._import_archive (importer.py:409) runs the full per-member pipeline (sha256 + pHash + dedup query + copy + provenance) for EVERY media member inline, all under import_media_file's single 300s soft limit. A single media file is sub-second; a multi-hundred-member archive blows the budget. They shared one task name and one timeout. **Split archive into its own task** - New `import_archive_file` task: same body as import_media_file (dispatch is by file-kind inside Importer.import_one) but soft=30min / hard=35min. Shared `_run_import_task` helper holds the flip-to-processing + resilience-contract wrapper; both tasks call it. - New `enqueue_import(task_id, task_type)` router — single source of truth for media-vs-archive dispatch. Used by all three enqueue sites: scan_directory, /api/import/retry-failed, recover_interrupted_tasks. - scan_directory now sets ImportTask.task_type = "archive" when is_archive(entry) (the model field already existed, anticipating this; scan was hardcoding "media"). - import_archive_file routes to the existing 'import' queue via the task_routes `import_file.*` wildcard — no worker config change. **Archive-aware recovery sweeps** Both sweeps would otherwise preempt a legitimately-running archive: - recover_interrupted_tasks (ImportTask 'processing' sweep): now task-type-aware. Media stays at STUCK_THRESHOLD_MINUTES (5); archives get ARCHIVE_STUCK_THRESHOLD_MINUTES (40 = 5-min buffer past the 35-min hard limit). Single UPDATE with an OR predicate over the two (task_type, cutoff) pairs; requeue routes via enqueue_import. - recover_stalled_task_runs (TaskRun 'running' sweep): now supports per-task-name overrides (TASK_STUCK_THRESHOLD_MINUTES) layered above the per-queue overrides added for ml. import_archive_file gets 40 min while the 'import' queue stays at the 5-min default for single-file imports. Precedence: task_name → queue → default, each pass excluding rows claimed by a higher-precedence pass so every row is touched once. **Tests** - test_import_archive_file_registered - test_recover_stalled_task_runs_archive_task_uses_longer_threshold — pins that a 10-min archive task-run survives, a 50-min one is flagged, and a same-queue 10-min media import is flagged at the default. - _make_task_run gains queue= + task_name= params. After deploy: archive imports get a 30-min budget and aren't preempted by either sweep; single-file imports keep their tight 5-min detection.	2026-05-27 22:45:11 -04:00
bvandeusen	407de18ff6	fix(ml): video branch needs longer time limits; recovery sweep is now per-queue ... Operator-flagged 2026-05-28: tag_and_embed on image 6288 (an mp4) was marked failed by recover_stalled_task_runs at the 5-min sweep tick while still legitimately running. The error_type='RecoverySweep' / "no completion signal received within 5 min" message was misleading — the worker was busy, not stuck. Root cause is two interacting limits, both undersized for video work: tag_and_embed: soft_time_limit=300, time_limit=420 (sized for the image branch, ≈2 GPU ops) recovery sweep: STUCK_THRESHOLD_MINUTES = 5 across all queues The video branch samples 10 frames via ffmpeg, then runs tagger + embedder on EACH frame — ~20 GPU ops vs 2 for an image. A loaded ml-worker can take 5-10 min on a long video, which trips both limits well before the task naturally finishes. **Two-part fix** 1. `tag_and_embed` time limits bumped to soft=900 (15 min) / time=1200 (20 min). Sized for the video path's worst case; image runs return in seconds and don't care. 2. New `QUEUE_STUCK_THRESHOLD_MINUTES` override dict in maintenance.py. Queues with legitimately-long-running tasks (currently just `ml` at 25 min — 5-min buffer past the new hard kill) get their own threshold; queues not in the dict use the default 5 min. The sweep now issues one UPDATE per distinct threshold value, with `queue.notin_(override_queues)` on the default pass so each row is touched at most once. Tests: - _make_task_run helper accepts `queue=` (defaults to "default") so existing tests use the default-threshold path. - New test `test_recover_stalled_task_runs_ml_queue_uses_longer_threshold` pins both directions: a 10-min-old ml row survives (fresh by 25-min override), a 30-min-old ml row gets flagged. After deploy, operator's mp4 ML jobs run to completion without spurious RecoverySweep failures.	2026-05-27 22:23:35 -04:00
bvandeusenandClaude Opus 4.7	110c1c0e51	fix(maintenance): recover_interrupted_tasks — fold SELECT into UPDATE…WHERE…RETURNING so the IN-list no longer blows past psycopg's 65535-parameter ceiling (operator-hit 2026-05-26 after deep scan orphan pile) ... Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>	2026-05-26 11:46:06 -04:00
bvandeusenandClaude Opus 4.7	9d5abb09f6	fix(maintenance): recover_interrupted_tasks also sweeps pending/queued orphans (>30 min) to failed ... scan_directory creates ImportTask rows with status='pending' (commit) then in a second pass transitions to 'queued' + .delay() (commit). Crashes in that window leave rows orphaned with no recovery path. Operator hit 5490 such rows 2026-05-25; the existing sweep only handled 'processing'. Flipping to 'failed' (not re-enqueue) lets the operator drain via the existing /api/import/retry-failed endpoint at their own pace. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>	2026-05-25 02:00:29 -04:00
bvandeusenandClaude Opus 4.7	7782672a51	fc3i: recover_stalled_task_runs + prune_task_runs maintenance tasks + Beat entries ... Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>	2026-05-24 21:03:33 -04:00
bvandeusenandClaude Opus 4.7	be0f472894	fix(workers): worker-level recovery — autoretry on transient errors + tightened sweep (5min) + import_media_file body-wrap so no path leaves rows stuck in 'processing' ... Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>	2026-05-24 11:46:36 -04:00
bvandeusenandClaude Opus 4.7	7d8b9c3d90	fix(tasks): share one sync engine per worker process to stop connection-pool leak ... Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>	2026-05-23 10:44:16 -04:00
bvandeusenandClaude Opus 4.7	ec004933a5	fc3d: cleanup_old_download_events daily task (terminal-only, configurable retention) ... Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>	2026-05-21 16:02:36 -04:00
bvandeusenandClaude Opus 4.7	e28de547c7	feat(integrity): verify_integrity task (keyset, fail-soft) + weekly beat ... Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>	2026-05-19 17:51:33 -04:00
bvandeusenandClaude Opus 4.7	ae67e67299	feat(deep-scan): keyset-paginated backfill_phash task ... Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>	2026-05-19 14:59:04 -04:00
bvandeusenandClaude Opus 4.7	80a5690740	fix(fc2a): apply ruff autofix + skip vue-tsc check (no tsconfig) ... Ruff: The remaining I001 errors came from ruff treating `alembic` as a first- party module (because the alembic/ directory exists in the repo root) rather than third-party. Ran `ruff check --fix` locally — auto-sorted import groupings to put alembic/sqlalchemy alongside backend.* as first- party, and trimmed redundant blank lines after a few import blocks. Frontend: `npm run check` (vue-tsc --noEmit) was failing because vue-tsc has no tsconfig.json to read against, and the frontend is pure JS without JSDoc annotations — vue-tsc had nothing to do. Skipping the step until we add a tsconfig + convert to TS or add JSDoc annotations. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>	2026-05-14 14:24:25 -04:00
bvandeusenandClaude Opus 4.7	b4e0d680f1	fix(fc2a): satisfy ruff 0.15.13 lint — UP017, UP042, I001 ... Ruff lint surfaced 23 violations across three rules; all addressed: UP017 (Use datetime.UTC alias): Replaced 13 sites of datetime.now(timezone.utc) with datetime.now(UTC), also adjusted from-imports accordingly. UTC is a Python 3.11+ alias for timezone.utc that ruff's pyupgrade rules prefer. UP042 (StrEnum): Replaced `class TagKind(str, Enum)` and `class SkipReason(str, Enum)` with `class Foo(StrEnum)`. StrEnum was added in Python 3.11 stdlib and is the modern idiom. Behavior is equivalent for our usage (the .value attribute, str(member) semantics). I001 (Import sorting): Added `known-first-party = ["backend"]` to ruff.toml's [lint.isort] so ruff groups `backend.*` imports correctly. Without it, ruff treated them as third-party and demanded a different grouping. The existing import order is stdlib → third-party → first-party → local relative, which ruff now accepts. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>	2026-05-14 14:11:35 -04:00
bvandeusenandClaude Opus 4.7	509c19ce86	feat(fc2a): add maintenance tasks (recovery + cleanup) with beat schedule ... recover_interrupted_tasks runs every 5 minutes, finds ImportTask rows stuck in 'processing' for >30 minutes (well above any legitimate import duration), and re-queues them. cleanup_old_tasks runs daily and deletes finished tasks older than 7 days so the task table stays an operational view rather than an archive. Both thresholds match ImageRepo's precedent. The 30-min stuck threshold is documented inline so a future reader can adjust it intentionally rather than mistaking it for a 'magic number'. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>	2026-05-14 12:09:09 -04:00