28 Commits

Include renames

Author	SHA1	Message	Date
bvandeusenandClaude Opus 5	3313c3b10a	feat: a report that shows what the near-dup gates decide about real artwork (4223) ... The three pixel constants added with the #4223 fix were chosen without ever measuring real files — CI only has synthetic split/solid fixtures, and FC verifies nowhere else. This prints the measurements they should have been chosen from: per pair, the hash distance, the mean drift, the changed-pixel fraction, the verdict, and which gate produced it. It drives the real find_similar with the real confirm rather than restating the decision, so it cannot drift from what the importer does. Read-only: opens files, touches no database. Also splits fingerprint_diff out of fingerprints_match — same computation, now returning the numbers instead of only the boolean, so the report can show how far a pair sat from a limit rather than which side of it it fell on. Runs inside the published :dev image (PIL + imagehash already there, no local env needed) with the art folder mounted read-only — rule 147's channel, so nothing has to reach main to be tried. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LVjrnpQjRgHdvq95rASoiR	2026-09-21 06:30:54 -04:00
bvandeusenandClaude Opus 5	84f873eb1b	fix: ruff UP037 — unquote the fingerprint_path return annotation ... PIL's Image is imported at module scope, so the annotation never needed to be a string. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LVjrnpQjRgHdvq95rASoiR	2026-09-21 00:17:15 -04:00
bvandeusenandClaude Opus 5	a05cb66635	fix: variant artwork was dropped as a near-duplicate even at threshold 0 (4223) ... The operator reported a 15-image variant pack landing as 3 records, then reported variants STILL being dropped with phash_threshold at 0 — the floor of the dial. No setting could have fixed it: at hash_size=8 a pHash is 64 bits of coarse light/dark layout, so two variants sharing a composition produce the SAME bits. Distance 0 meant "identical hash", not "identical image", and the dial was simultaneously too coarse to keep variants and too tight to catch a re-encoded rescale. The hash no longer decides a merge on its own. find_similar now runs three gates, cheapest first: the threshold proposes candidates, aspect ratio (ASPECT_TOL, matching the tier-1 video path) rejects crops and re-canvases, and a pixel-level confirm on the two files accepts. Every gate fails closed — unknown dimensions, an unreadable candidate, a hash of the wrong width all mean "not a duplicate", because too strict keeps a redundant copy the operator can see while too loose deletes artwork only a source re-walk returns. - utils/phash.py: HASH_SIZE 8 -> 16 (256-bit, what ImageRepo always used); aspect_matches, fingerprint/fingerprint_path/fingerprints_match (PIL-only, mean drift + changed-pixel fraction), find_similar gains `confirm`. - importer: _pixel_confirmer supplies gate 3 on both dedup sites, lazily and cached, so a non-matching import costs no extra I/O. - 0098: widens image_record.phash to 64 chars and NULLs every value — a stored 64-bit hash cannot be compared to a 256-bit one, and backfill_phash is NULL-only, keyset-paginated and now on the daily beat, so the library re-hashes itself. Dedup degrades to sha256 until it finishes. - phash_threshold counts bits and the denominator went 64 -> 256, so the setting is reset to the new default of 24 (there is no honest carry-over) and the slider is rescaled to 0-64. - gallery_service dup_threshold 8 -> 32: the same fraction of the hash, so the Explore rail keeps the variance the operator tuned in on 2026-07-01. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LVjrnpQjRgHdvq95rASoiR	2026-09-21 00:16:14 -04:00
bvandeusenandClaude Opus 5	973db73221	db: collapse alembic 0001..0089 into one baseline (#3266) ... 89 files and 6,300 lines become one file of 807. Nothing about the resulting schema changes; what goes away is the requirement that a new installation replay our development history to arrive at it. revision = "0089", down_revision = None. That pairing IS the migration strategy for existing installs, not a detail of it: a deployed database already has alembic_version = '0089' from running the real 0089, so alembic reads the version table, sees head reached, and does nothing. No stamp is required — which matters, because `alembic stamp` writes a version string without validating anything about the schema it is writing it against, and a wrong stamp is indistinguishable from a right one until the next migration fails. An empty database runs the file and records 0089. Both paths converge. The next migration is 0090, as it would have been; the numbering is continuous across the collapse on purpose. Autogenerate produced nearly all of this unaided, which was NOT true of the first attempt — that one was reverted because the generator silently dropped eleven indexes and three uniqueness guarantees. #3275 put those on the models first, so the HNSW index with its opclass, the COALESCE expression index, the partial uniques, 107 server_defaults and the enum CHECKs are all emitted now. Doing the reconciliation before the squash, rather than after, is what made this work. Hand-added, because none of it can live in a model: * CREATE EXTENSION vector / tsm_system_rows (0001, 0004) — database objects, not table metadata. * The pgvector import. Autogenerate writes qualified pgvector.sqlalchemy.vector.VECTOR references without importing the package, so its own output cannot run (run 4988). * THE TWO SEED ROWS. 0002 and 0003 did not only build schema — each inserted a settings singleton, and nothing in the app ever creates them: ImportSettings.load() and MLSettings.load() are select(...).scalar_one(), which RAISES NoResultFound rather than returning None. A models-only baseline would leave both tables empty and crash a fresh install on first settings access, while baseline.yml reported a perfect schema match. Only running the app against a new database finds that. Not carried over: 0023's DELETE FROM tag and 0047's series deletes, which are historical cleanups operating on rows an empty database lacks. downgrade() raises. A baseline's downgrade is "drop every table", which is a data-loss event wearing a migration as a disguise; offering it as one invites someone to run it. Restore from a backup. Also removed, per the plan: the 10 test_migration_*.py files (they assert intermediate states and backfills that no longer exist — a test that a column exists is already the model tests' job) and backend/app/utils/artist_backfill.py, whose only importer was 0008. Verified no other consumer anywhere in backend/ or tests/. baseline.yml changes with it. chain_ref now DEFAULTS to 725bf15, since the tree no longer carries a chain to compare against — that pinned commit is the last one that does. And the CheckConstraint repair is removed, because it never fired. I added it claiming autogenerate re-doubles a constraint name on the round trip and asserted in b979062 that it was "still correct and still needed". It is not: autogenerate wraps names in op.f(), which marks them already-formatted and blocks the convention from re-applying. Tested against the real candidate line — the regex matches nothing. What actually fixed the mismatch was 0088's renames alone. The doubling is a real hazard, but of hand-writing a pre-prefixed name, not of the generator; the comment asserting otherwise was worse than the dead code under it. The header comment is rewritten for the same reason — it described 87 revisions, and claimed the HNSW index could not be expressed in a model, which #3275 disproved. It now also states plainly what this check CANNOT see: it compares schema, so a green run means the schema is right, not that the baseline is. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017QHszn9H8VBvx5Ke8x1hvw	2026-09-01 01:14:37 -04:00
bvandeusen	6959e1220c	Revert "db: collapse alembic 0001..0087 into one baseline" ... This reverts 2529b51. Not a retreat — a reordering, on the operator's call, and the better sequence. The squash's acceptance test (run 4971) found ~130 places where the ORM models do not describe the deployed schema (#3275), including a unique=True the database never had and two UNIQUE indexes that exist only in migrations. Collapsing now would have baked all of that into the one file a public installer starts from. So: fix the drift first as ordinary migrations on the intact chain, let the operator deploy so their database moves to the corrected head, and only then collapse. The baseline is then generated from reconciled models and reproduces a schema worth reproducing. Nothing is lost by reverting. The baseline was never deployed, and regenerating it after the fixes is strictly better than patching this copy — it will come out of autogenerate correct rather than needing the same hand-finishing twice.	2026-08-30 14:34:28 -04:00
bvandeusen	2529b516e6	db: collapse alembic 0001..0087 into one baseline (milestone 328 step 1) ... 87 revisions narrating this project's build-out become one file that creates the schema in a single step. They cost nothing at runtime — all 86 upgrade steps ran in 0.2s (note #3260) — so this is a presentation change, not a performance one: a new installer should not inherit our development history to stand up a database. Deleted: 87 revisions (6,052 lines), the 10 tests/test_migration_*.py files (483 lines) that asserted intermediate states and backfills which no longer exist, and backend/app/utils/artist_backfill.py — the only live module a migration imported, with no other consumer anywhere. That last one satisfies the operator's separate request to inline it into 0008 and delete the module; the squash removes both outright. THE REVISION ID IS "0087", NOT "0001", ON PURPOSE. It is the id of the last revision collapsed, so an existing database is already at head and `alembic upgrade head` does nothing. The alternative is `alembic stamp` against live data, and stamp validates NOTHING — it writes a version string whether or not the schema matches, so a wrong baseline surfaces later, via the next real migration, with no clean way back. This removes that operation rather than making it safe. Future revisions run from 0088. Four things are hand-written because SQLAlchemy metadata does not carry them, and none fail at generation time: 1. CREATE EXTENSION vector — the VECTOR columns cannot be created without it, so it is ordered first in upgrade(). 2. CREATE EXTENSION tsm_system_rows — surfaces only when the random sample query runs. 3. the HNSW index on image_record.siglip_embedding, raw SQL because create_index cannot express USING hnsw (... vector_cosine_ops). The quietest of the four: everything works, similarity search just stops using an index. 4. import pgvector.sqlalchemy.vector — autogenerate EMITS pgvector.sqlalchemy.vector.VECTOR references without importing it, so the generated file dies with NameError on first run. The candidate came out of CI (run 4967) as checksummed base64 rather than a plain cat, because run 4964's cat was truncated mid-line inside a column definition with the step still green — 29 tables instead of 42, and it looked entirely plausible. Verified here: 56,582 bytes, sha256 471acfca69c0…, 42 tables, 66 indexes, 42 drops. NOT YET PROVEN against the old chain. baseline.yml does that, and it is step 2's gate; this commit does not claim the schemas match.	2026-08-30 13:45:35 -04:00
bvandeusenandClaude Opus 4.8	f154603811	feat(import): Tier-1 video near-dup by duration+aspect (#871) ... Videos deduped on sha256 only (pHash is images-only), so a different encode/remux of the same clip imported as a distinct record — the "same video from multiple sources" clutter surfaced by #859. Tier-1 metadata fingerprint: identity = container duration (±1.0s) + matching aspect ratio, scoped to the same artist; quality axis = pixel dimensions (mirrors image pHash: larger_exists→skip+link, smaller_exists→supersede). Codec/bitrate are deliberately NOT part of identity (the point is matching across re-encodes). Tight tolerances because a wrong video merge is destructive. - image_record.duration_seconds (Float, nullable; migration 0052). NULL for images. - safe_probe.probe_video also reads format=duration (one extra ffprobe field on the call that already runs); ProbeResult.duration. - _find_similar_video(duration,w,h,artist) shared by both import pipelines. - _import_media (filesystem/archive path): captures duration, video near-dup branch, persists duration. - attach_in_place (download path — handles #859's videos, previously didn't probe video at all): best-effort probe for dims+duration (LENIENT — never newly rejects a downloaded video on probe failure), video near-dup branch, persists duration. - _supersede carries duration onto the kept row. Reuses SkipReason.duplicate_phash so the existing download/external dup-cleanup (path-safe unlink, #859) applies unchanged. Tests: skip-smaller, supersede-larger (+ duration adopted), and distinct-durations-not-merged (false-merge guard). Follow-up (Phase 2, #871): a backfill to re-probe NULL-duration existing videos so the current library participates in dedup; retroactive merge of existing dups is a separate destructive maintenance action. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-06-15 22:17:36 -04:00
bvandeusenandClaude Opus 4.8	976107bbe8	fix(patreon): read post body from content_json_string (ProseMirror), not the dead content field (#842) ... THE empty-body root cause. Patreon deprecated the flat `content` HTML field — it returns null on the feed AND the detail endpoint, for every post type (confirmed against the live API: all 135 StickySpoodge posts, text_only/ image_file/poll alike). The real body now lives in `content_json_string` (a ProseMirror/TipTap doc), returned only under the DEFAULT post fieldset — a sparse fields[post]=content request omits it. Not credential, not post_type: a request shape gone stale. - NEW utils/prosemirror.py: ProseMirror doc -> HTML (paragraphs, marks bold/italic/underline/strike/code/link, hardBreak, inline images, lists, headings; unknown nodes degrade to children). post_body_html(attrs) = the one resolver: legacy content HTML else convert content_json_string. - patreon_client: add content_json_string to the feed _FIELDS_POST; rewrite fetch_post_detail_content to use the DEFAULT fieldset (no sparse fields[post]) and resolve via post_body_html (replaces the wrong sparse req + full-fetch fallback). - patreon_downloader._write_sidecar_data: resolve body via post_body_html (feed content_json_string) before the detail-fetch; memoize resolved HTML. - tests: prosemirror converter unit tests; client legacy + content_json_string paths; contract pins content_json_string. Inline <img> nodes carry the CDN filehash → bodies now feed Phase-2 localization. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-06-15 00:14:52 -04:00
bvandeusenandClaude Opus 4.8	96c29c370b	feat(ingest): localize inline post-body images to local copies (Phase 2) ... Render a post body faithfully by serving our stored copies of inline images instead of hotlinking the public CDN. The join key is the CDN filehash (32-hex MD5) shared between a body <img src> and the media URL we downloaded (the same identity extract_media dedups by): - utils.paths.filehash_from_url — one source of truth for the extractor; patreon_client._filehash now delegates so capture- and render-time hashing cannot drift. - ImageRecord gains source_url (provenance) + source_filehash (indexed match key); migration 0051. - the per-media sidecar carries the file's source_url; the importer persists it (NULL-only) on the ImageRecord via _apply_sidecar. - post_feed_service.get_post remaps body <img src> -> /images/<path> for every inline image whose filehash maps to a stored image of THIS artist; unmatched / pre-Phase-2 images keep hotlinking. Pre-existing on-disk images have no filehash yet, so they fall back to hotlinking until re-downloaded; localization is forward-looking. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-06-14 16:39:58 -04:00
bvandeusenandClaude Opus 4.8	c342c73a25	feat(posts): faithful (semantic) HTML rendering of post bodies ... Phase 1 of milestone #64. The body is captured (Phase 0) but was shown as plain text. Now: - html_sanitize.py: widen the allowlist to a faithful-but-safe set — headings, inline images, lists, blockquote, hr, code/pre, figure, links (div/span stay stripped; their text is preserved). Benefits the existing ProvenancePanel too. - post_feed_service.get_post: add sanitized `description_html` to the DETAIL response (the feed list stays lightweight plain text by design). - PostCard.vue: render description_html via v-html once expanded (fetched with detail); collapsed + no-detail fallback stay plain text. Styled close to the source (headings, images max-width, accent links, lists, quotes, code). Tests: sanitizer (headings/img/lists survive, img javascript: src dropped); get_post returns sanitized description_html. Refs FC #830. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-06-14 13:02:21 -04:00
bvandeusenandClaude Opus 4.8	5bb25245a5	fix(archive): magic-byte archive detection so mis-named archives extract — #713 part 1 ... Patreon attachment downloads land with sanitized URL-blob filenames (01_https___www.patreon.com_media-u_v3_<id>) whose Path.suffix is junk, never .zip — so the extension-only is_archive() filed them as opaque PostAttachments and never extracted them ("No images attached to this post"). Add archive_extractor.detect_archive_format() — extension first, then magic-byte sniff (zipfile.is_zipfile + RAR/7z signatures). is_archive(), extract_archive(), and safe_probe._inspect_archive() (the bomb-guard) all route through it, so a mis-named/extension-less archive is now detected, bomb-guarded, integrity-tested, AND extracted regardless of filename. Stops new ones; part 2 re-extracts the already-imported backlog. Tests: mis-named zip detected + extracted; non-archive dotted name not misdetected; _inspect_archive on a mis-named zip; signature updated. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-06-06 14:33:24 -04:00
bvandeusenandClaude Opus 4.8	b211900390	refactor(downloads): DRY the ingester/gallery-dl seam — A1–A4 (plan #707) ... Consolidates duplication that owning the native ingester left against the still- live gallery-dl path, and fixes a parity gap the duplication hid. A1 — shared quarantine: extract file_validator.quarantine_file (move to _quarantine/<slug>/<platform> + write the .quarantine.json provenance sidecar). gallery_dl._validate_and_quarantine and patreon_downloader._validate_path both call it. PARITY FIX: the native path now writes the provenance sidecar it previously skipped — threads the media url through for source_url. A2 — make_run_stats(**counts) factory in gallery_dl for the canonical run_stats key set; gallery_dl._compute_run_stats and ingest_core both build through it so the shape can't drift (gallery-dl path gains a benign dead_lettered_count=0). A3 — one safe_ext in utils/paths.py; importer._safe_ext (thin wrapper, kept for the Path call sites + memory pointer) and patreon_client both use it. Closes the double-impl of the URL-encoded-basename ext gotcha. A4 — promote gallery_dl._truncate_log/_extract_errors_warnings to module-level truncate_log/extract_errors_warnings; download_service calls them directly instead of reaching through self.gdl for native-result log shaping. The staticmethods stay as thin delegators for existing callers/tests. Behavior-preserving except the A1 sidecar parity fix. Test: native quarantine writes a .quarantine.json (test_patreon_downloader). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-06-06 11:11:25 -04:00
bvandeusenandClaude Opus 4.7	810baf63ac	fix(import): archive probe → subprocess (was multiprocessing.Process) ... Every archive import was failing immediately with "AssertionError: daemonic processes are not allowed to have children" (operator-flagged 2026-05-30 — import_archive_file crashes in 49ms with the assertion). Celery's prefork pool runs tasks in daemon processes; Python's multiprocessing module refuses to let daemons spawn children, which is exactly what probe_archive was doing via mp.get_context("spawn") .Process. The Layer-3 crash-isolation feature added 2026-05-28 was effectively a hard-blocker on the very import path it was meant to protect. Switched probe_archive to subprocess.run — no daemon restriction, still isolates the probe (a probe segfault/OOM exits non-zero, doesn't kill the worker). The probe body lifted to a tiny runner module (_archive_probe_runner) that imports the unchanged _run_probe helper and prints a single JSON line; parent parses stdout, returns the ProbeResult exactly as before (timeout, signal, OOM, clean-rejection, ok — all preserved). cwd for the subprocess is the repo root derived from __file__ parents so `python -m backend.app.utils._archive_probe_runner` resolves both in the Celery container and pytest, regardless of where the worker was launched. Test refactor: test_archive_probe_target_bomb_guard → test_run_probe_bomb_guard. Same in-process call to the (renamed) probe body so the monkeypatched cap still takes effect; the real subprocess path is exercised by the existing test_probe_archive_valid_zip / test_probe_archive_corrupt_zip_clean_rejection tests. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>	2026-05-30 21:48:51 -04:00
bvandeusen	e3cdd0f92b	feat(import-resilience L3): subprocess-isolated probes for video + archive ... Layer 3 — prevent the hard worker crash rather than just recovering from it. The realistic process-crash vectors (operator's observed slow/heavy tasks) are video decode and archive extraction; images decode in-process and Pillow raises-and-skips cleanly, and a subprocess per image would wreck deep-scan throughput, so images are intentionally not probed. New backend/app/utils/safe_probe.py (leaf module, lazy heavy imports so the spawned child stays light): - probe_video(path): validates the container + first video stream via ffprobe (a separate binary — a decoder crash kills only ffprobe, not the worker). Returns width/height, which the importer didn't capture for videos before. crashed=True only on ffprobe timeout. - probe_archive(path): an uncompressed-size bomb guard (MAX_ARCHIVE_UNCOMPRESSED_BYTES = 4 GiB) plus the format integrity test (zipfile.testzip / rarfile.testrar / py7zr.test) run in a spawned child process. A decompression-bomb OOM or native-lib segfault on a malformed archive shows up as a non-zero child exit code → crashed=True, never a dead worker. ProbeResult.crashed distinguishes a HARD failure (subprocess killed / timed out — the poison-pill signature → caller returns terminal 'failed') from a CLEAN rejection (corrupt-but-handled, bomb cap, integrity mismatch → caller's choice of skipped/attached). Wired: - importer._import_media video branch: probe_video before the pipeline; crash → failed, clean reject → invalid_image skip, ok → capture dims. - importer._import_archive: probe_archive before extract_archive; crash → failed, clean reject → still preserve the archive as a PostAttachment (matches extract_archive's fail-soft contract). - ml.tag_and_embed video branch: probe_video before sampling 10 frames, so a corrupt video is rejected (status='bad_video') instead of crashing the ml-worker on frame decode. Tests (test_safe_probe.py): valid/corrupt zip via probe_archive, direct _inspect_archive size+integrity, in-process _archive_probe_target bomb guard (monkeypatch can't reach a spawned child, so the target is called directly), and a non-video → ok=False that's robust to ffprobe presence in CI.	2026-05-28 00:01:32 -04:00
bvandeusen	abafc3265e	refactor(platforms): promote services/platforms.py → services/platforms/ package with per-platform quirk colocation ... Operator-requested 2026-05-27: centralize the per-platform quirks that had been accumulating across credential_service, sidecar, and platforms into a single per-platform module so adding/updating quirks becomes "edit one file." **Layout** services/platforms/ base.py PlatformInfo dataclass + module-default key chains + shared helpers (str_id_value, str_field) __init__.py PLATFORMS dict + public API (auth_type_for, known_platform_keys, to_dict, external_post_id_keys_for, description_keys_for) patreon.py metadata only — the reference platform, no quirks subscribestar.py metadata + augment_cookies (18+ agreement) + derive_post_url (synthetic /posts/<post_id>) hentaifoundry.py metadata + augment_cookies (host-only PHPSESSID duplicate) + derive_post_url (/pictures/user/...) pixiv.py metadata + derive_post_url (/artworks/<id>) discord.py metadata + derive_post_url (channels/<server>/<channel>/<message>) deviantart.py metadata only — un-audited; quirks to be added when an operator first exercises DA **PlatformInfo extensions** Existing fields preserved. Four new optional fields: external_post_id_keys: tuple[str, ...] | None Override the sidecar external_post_id lookup chain. None falls back to DEFAULT_EXTERNAL_POST_ID_KEYS in base.py ("post_id", "id", "index", "message_id") — covers every current platform. description_keys: tuple[str, ...] | None Override the description body lookup chain. None falls back to DEFAULT_DESCRIPTION_KEYS ("content", "description", "caption", "message") — Discord's "message" body field is covered by the default's trailing entry. derive_post_url: Callable[[dict], str | None] | None Synthesize the post permalink from sidecar metadata. None = trust the bare `url` / `post_url` field (patreon, deviantart). subscribestar/pixiv/hf/discord override this because their `url` is the file CDN URL. augment_cookies: Callable[[str], str] | None Post-process the materialized cookies.txt before gallery-dl consumes it. None = no-op. Used by subscribestar (age cookie) and hentaifoundry (host-only PHPSESSID duplicate). **Consumer changes** - credential_service._augment_cookies(platform, netscape) shrunk from a per-platform-conditional dispatcher (~80 lines of inlined helpers) to a 5-line lookup: `info.augment_cookies(netscape) if info and info.augment_cookies else netscape`. The platform-specific helper bodies moved verbatim into the per-platform modules. - sidecar.parse_sidecar similarly delegates: external_post_id chain via external_post_id_keys_for(category), description chain via description_keys_for(category), post_url via PLATFORMS[category].derive_post_url. The _DERIVED_URL_PLATFORMS set and inline _derive_post_url body both gone. Added a shared `_first_id` helper for bool-safe id coercion. **Public API preserved** PLATFORMS, PlatformInfo, auth_type_for, known_platform_keys, to_dict are all re-exported from the package's __init__.py. test_platforms_registry test_credential_service, and test_sidecar_util pass without changes because the behavior is identical; only the implementation moved. **Adding a new platform** 1. Create services/platforms/<name>.py with `INFO = PlatformInfo(...)` and any of the four optional hooks. 2. Import it in services/platforms/__init__.py + add to the PLATFORMS tuple-comprehension. 3. Done. sidecar parsing, cookie materialization, /api/platforms all pick it up automatically.	2026-05-27 19:46:05 -04:00
bvandeusen	bd3f996582	fix(sidecar): correct external_post_id + post_url derivation for non-Patreon platforms ... Audit of one sample sidecar per platform on the operator's /mnt/Data/Patreon/ archive surfaced three parser bugs that have been silently corrupting non-Patreon Posts since FC-3 shipped: 1. SubscribeStar `id` vs `post_id` confusion. gallery-dl puts the per-attachment id in `id` (e.g. 711509) and the actual post id in `post_id` (e.g. 360360). FC's external_post_id chain had `id` winning, so every multi-image SubscribeStar post was fragmented into N Post rows in the database. Reorder the chain to `("post_id", "id", "index", "message_id")` — Patreon/Pixiv (no `post_id`), HF (uses `index`), Discord (uses `message_id`) all unaffected. 2. Discord `message` field not captured. Discord posts put the body in `message`, not `content`. Append it to the description fallback chain `("content", "description", "caption", "message")`. 3. post_url is the file URL on SubscribeStar/Pixiv/HF/Discord. New `_derive_post_url(platform, data)` helper synthesizes proper permalinks from per-platform fields: subscribestar → https://www.subscribestar.com/posts/<post_id> pixiv → https://www.pixiv.net/artworks/<id> hentaifoundry → https://www.hentai-foundry.com/pictures/user/<user>/<index> discord → https://discord.com/channels/<server>/<channel>/<message> Patreon's bare `url` IS a real permalink and is used as-is. For the four file-URL platforms, the bare `url` is NEVER trusted: derive or return None rather than persist a CDN URL. Tests: - `test_parse_core_fields_and_id_priority` flipped to assert post_id wins over id. - New `test_parse_id_used_when_no_post_id` covers the Patreon real shape. - New `test_parse_message_used_as_description_fallback` covers Discord bodies. - Five new tests cover per-platform post_url derivation (SubscribeStar/Pixiv/HF/Discord/Patreon-untouched + missing-fields → None). Cleanup migration alembic 0025_fix_subscribestar_post_ids: - For each SubscribeStar Post: find a related ImageRecord.path, walk to its sidecar JSON, read `post_id`, overwrite Post.external_post_id + post_url with the corrected values. - After all updates, every group of Posts under one source sharing the same NEW external_post_id is a fragment-set — merge to a canonical row using the same ImageProvenance pre-delete + repoint dance as alembic 0022 (banked pattern). - Pure-SQL backfill of Pixiv post_url: replace any `i.pximg.net`-shape url with the derived `/artworks/<id>` permalink. - HF and Discord post_url backfills skipped — HF would need the `user` field (not stored on Post), Discord needs server/channel triple. Both will be corrected by a deep-scan re-applying sidecars through the new parser. Idempotent: re-running on already-corrected data is a no-op.	2026-05-27 15:35:25 -04:00
bvandeusen	ae8c78ae09	fix(sidecar): synthesize post_title from content first-line when title is empty (subscribestar) ... SubscribeStar gallery-dl always writes `title: ""` and embeds the leading sentence inside `content` HTML. Confirmed against the operator's /mnt/Data/Patreon/Cheunart/subscribestar/ dump 2026-05-27: every post's JSON has `title: ""` and a content like `<div>Lets say hello to you guys with my Belle <br><br><br></div>`. FC's sidecar parser, treating empty strings as missing, had been leaving post_title NULL on every subscribestar post since FC-3 shipped. Fix at two layers: 1. `backend/app/utils/sidecar.py` — new `_first_line_text(body, limit)` helper strips HTML tags, collapses whitespace, returns the first non-empty line truncated to 120 chars with ellipsis. `parse_sidecar` now falls back to this when `title` resolves to None and a `content`/`description`/`caption` value is present. Patreon's non-empty titles short-circuit the fallback so existing behavior is unchanged. Four new tests in test_sidecar_util.py pin: derivation from content, truncation at 120 chars, explicit-title precedence, no-content no-fallback. 2. `alembic 0024_backfill_post_title_from_description` — backfills the same logic across existing Post rows where `post_title IS NULL OR post_title = ''` AND description is present. Idempotent (re-running is a no-op once titles are populated). Downgrade is a no-op since there's no safe way to tell derived rows from genuine ones. After deploy + migration: subscribestar posts will surface a meaningful title in PostCard, post feed search, etc.	2026-05-27 14:44:09 -04:00
bvandeusen	8675f105ad	fix(tests): test_api_tags prefix tests use character: not artist: (KNOWN_KINDS dropped artist) ... The two prefix-parsing tests were pinned to `artist:Eric`, but `artist` was removed from KNOWN_KINDS in commit 4cad07a (provenance is a separate axis from tags). The parser now keeps `artist:` literal, so the assertion `body["name"] == "Eric"` failed. Repointed to `character:Saber` (still in KNOWN_KINDS). Also updated the stale `artist:` docstring example in parse_kind_prefix to `fandom:`. Caught by [[reference-grep-pinned-tests-in-plans]] — should have grep'd tests/ for `artist:` when shrinking KNOWN_KINDS. Banking the miss.	2026-05-27 11:09:04 -04:00
bvandeusenandClaude Opus 4.7	42b1340324	fix(tag-prefix): drop artist/meta/rating from KNOWN_KINDS — artist tags retired in FC-2d-vii-c (provenance is its own axis), meta/rating retired by operator 2026-05-26. User-typeable prefixes now just character/fandom/series. Frontend placeholder + icon map + client-side mirror updated; new test confirms retired prefixes parse as literal text. ... Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>	2026-05-26 21:00:12 -04:00
bvandeusenandClaude Opus 4.7	ccee344099	feat(tag-prefix): parse_kind_prefix util — IR-style \kind:name\ parser at the input boundary; KNOWN_KINDS = artist/character/fandom/series/meta/rating (excludes default \general\ and system-managed archive/post) ... Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>	2026-05-26 20:51:49 -04:00
bvandeusen	c0c9e56fb9	fix(importer): skip transparency check on animated images (operator-flagged 2026-05-26: animated WebP triggered 5+ min PIL multi-frame decode → Celery hard-timeout SIGKILL); compute_phash seeks frame 0 defensively — Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>	2026-05-26 00:45:08 -04:00
bvandeusen	0978fbac66	fix(sidecar): strip gallery-dl 'NN_' numbering prefix when locating sidecars — fixes 'deep scan refresh count high but 0 Posts created' (operator-flagged 2026-05-26) — Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>	2026-05-26 00:27:43 -04:00
bvandeusenandClaude Opus 4.7	07edb73373	fc3e: html_to_plain + truncate_at_word utilities (stdlib html.parser, no new deps) ... Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>	2026-05-21 18:53:35 -04:00
bvandeusenandClaude Opus 4.7	63e7185811	feat(provenance): image_record.artist_id + 0008 backfill/drop-artist-tags ... Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>	2026-05-18 21:33:11 -04:00
bvandeusenandClaude Opus 4.7	3b5f894435	feat(provenance): nh3 HTML sanitizer for scraped post descriptions ... Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>	2026-05-18 18:05:45 -04:00
bvandeusenandClaude Opus 4.7	ef7e4d92a9	feat(sidecar): minimal gallery-dl sidecar find + parse util ... Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>	2026-05-18 14:44:37 -04:00
bvandeusenandClaude Opus 4.7	d2e8dd3a3b	feat(phash): compute_phash + find_similar util (hash_size=8, fits existing column) ... Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>	2026-05-17 22:02:42 -04:00
bvandeusenandClaude Opus 4.7	4cb319b393	feat(fc2a): add slug and paths utilities ... slugify() produces ASCII-only lowercase slugs from arbitrary text (used for artist slugs and tag slugs). paths.derive_subdir/derive_top_level_artist extract the destination layout and folder→artist convention from an import source path. hash_suffixed_name builds IR-style 'stem__hash10.ext' filenames. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>	2026-05-14 12:04:05 -04:00