feat: Settings can re-download the Discord images the None naming broke (3999)
CI / lint (push) Successful in 3s
Build images / sign-extension (push) Successful in 4s
CI / extension-version (push) Successful in 2s
Build images / build-agent (push) Successful in 6s
CI / frontend-build (push) Successful in 19s
CI / backend-lint-and-test (push) Successful in 33s
Build images / build-web (push) Successful in 1m9s
Build images / smoke-web (push) Skipped
Build images / build-ml (push) Successful in 2m3s
Build images / promote (push) Skipped
CI / integration (push) Successful in 2m35s
CI / lint (push) Successful in 3s
Build images / sign-extension (push) Successful in 4s
CI / extension-version (push) Successful in 2s
Build images / build-agent (push) Successful in 6s
CI / frontend-build (push) Successful in 19s
CI / backend-lint-and-test (push) Successful in 33s
Build images / build-web (push) Successful in 1m9s
Build images / smoke-web (push) Skipped
Build images / build-ml (push) Successful in 2m3s
Build images / promote (push) Skipped
CI / integration (push) Successful in 2m35s
The operator chose a clean re-download over relinking in place. The ~1,600 broken files can't be relinked reliably: their message ids are gone, and their sidecars collided.
Settings → Maintenance → "Repair Discord downloads" previews, then applies:
- Deletes every image whose path is `…/discord/None/<8 digits>_None_…`. Both the folder and the name are required, because that pair is only what the bug produced. It reuses cleanup_service.delete_images for the record and file deletes.
- Sweeps the leftover collided sidecars from those folders and removes the emptied folders.
- Only then clears gallery-dl's archive rows `discord%`, excluding `discordasset_%`. Upstream keys message attachments as `discord{message_id}_{num}`. Since the broken files lost their message ids, per-source forgetting is impossible. Every pre-fix Discord download is broken, and files fetched after the fix still exist on disk, so gallery-dl's `skip` won't re-fetch them.
- Arms a fresh backfill on every Discord source.
The apply defaults to preview at both the route and the task, runs on maintenance_long, and is never on a beat. The card uses the confirm-dialog pattern of AttachmentReclaimCard.
Supporting refactors, with no behaviour change:
- gallery_dl.archive_path() is the single definition of the archive location.
- source_service.arm_backfill() is the mutation start_backfill already did, now shared with the sync repair.
Tests (tests/test_discord_repair.py):
- The archive clear leaves other platforms and Discord assets alone, and counting mutates nothing.
- Case-twin artist folders are both found.
- The folder sweep works.
- An integration run shows only the broken image goes. A correctly named Discord file and a `None` folder under Patreon survive, and only Discord sources are re-armed.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SHQB1YukL3VyvMK8rcbmV9
This commit is contained in:
@@ -475,6 +475,20 @@ async def trigger_reclaim_attachments():
|
||||
return _queued(async_result)
|
||||
|
||||
|
||||
@admin_bp.route("/maintenance/repair-discord-downloads", methods=["POST"])
|
||||
async def trigger_repair_discord_downloads():
|
||||
"""Clean re-download of the Discord files broken by the `None` naming
|
||||
(#3999). Body {"dry_run": bool}; dry_run is the DEFAULT, because the apply
|
||||
deletes files and makes gallery-dl forget every Discord download. Returns the
|
||||
Celery task id — poll /maintenance/task-result/<id> for the summary."""
|
||||
from ..tasks.admin import repair_discord_downloads_task
|
||||
|
||||
body = await request.get_json(silent=True) or {}
|
||||
dry_run = bool(body.get("dry_run", True))
|
||||
async_result = repair_discord_downloads_task.delay(dry_run=dry_run)
|
||||
return _queued(async_result)
|
||||
|
||||
|
||||
@admin_bp.route("/maintenance/dedup-videos", methods=["POST"])
|
||||
async def trigger_dedup_videos():
|
||||
"""Tier-1 video dedup (#871). Body {"dry_run": bool}: dry_run=true previews
|
||||
|
||||
@@ -0,0 +1,183 @@
|
||||
"""Repair the Discord downloads made before the naming fix (issue #3999).
|
||||
|
||||
Until dc840fe, gallery-dl's Discord patterns asked for keys the extractor never
|
||||
emits, so every Discord download landed as
|
||||
`<artist>/discord/None/<date>_None_<original name>`, next to a sidecar named
|
||||
after the attachment's ORIGINAL name. That broke two things:
|
||||
|
||||
* **No Post, no date.** `find_sidecar` can never pair those names, so these
|
||||
files were imported as loose images with no Post, and a card shows the
|
||||
download time.
|
||||
* **No trustworthy metadata to relink from.** Every `image.png` in a channel
|
||||
wrote the same `image.json`, so the surviving sidecar describes whichever
|
||||
message was written last. The message id is gone from the filename too.
|
||||
|
||||
The operator chose a clean re-download (2026-09-13) over relinking in place:
|
||||
delete the broken files and their records, make gallery-dl forget it fetched
|
||||
them, and backfill every Discord source again under the fixed naming.
|
||||
|
||||
## Why the archive is cleared for ALL of Discord
|
||||
|
||||
gallery-dl records a download as `discord{message_id}_{num}` (upstream
|
||||
`DiscordExtractor.archive_fmt`, prefixed with the category). The broken files
|
||||
lost their message ids, so there is no way to forget one source's entries and
|
||||
not another's. Every Discord download made before the fix is broken, so
|
||||
forgetting all of them is exactly right. Anything downloaded AFTER the fix still
|
||||
exists on disk under its correct name, and gallery-dl's `skip` sees the file and
|
||||
does not fetch it again.
|
||||
|
||||
## What it does not touch
|
||||
|
||||
Discord posts FC grouped itself (#388 E2) are built from Posts, and these files
|
||||
never had one, so there is nothing grouped to unwind. Images outside a
|
||||
`discord/None/` directory are never selected: the path filter requires both the
|
||||
`None` directory and the `_None_` filename, the pair only the bug produced.
|
||||
|
||||
Operator-triggered only (Settings, preview first). Never on a beat.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import logging
|
||||
import sqlite3
|
||||
from pathlib import Path
|
||||
|
||||
from sqlalchemy import func, select
|
||||
from sqlalchemy.orm import Session
|
||||
|
||||
from ..models import ImageRecord, Source
|
||||
from .cleanup_service import delete_images
|
||||
from .gallery_dl import archive_path
|
||||
from .source_service import arm_backfill
|
||||
|
||||
log = logging.getLogger(__name__)
|
||||
|
||||
# `%` and `_` are LIKE wildcards, so the literal underscores around None are
|
||||
# escaped. `________` is the eight-digit date prefix the old pattern wrote.
|
||||
_BROKEN_PATH_LIKE = r"%/discord/None/________\_None\_%"
|
||||
|
||||
# Upstream keys asset downloads as `asset_{server_id}_{id}`. FC never fetches
|
||||
# server assets, but excluding them keeps this to exactly the message
|
||||
# attachments the bug mangled.
|
||||
_ARCHIVE_SQL_MATCH = r"entry LIKE 'discord%' AND entry NOT LIKE 'discordasset\_%' ESCAPE '\'"
|
||||
_COUNT_SQL = "SELECT COUNT(*) FROM archive WHERE " + _ARCHIVE_SQL_MATCH
|
||||
_DELETE_SQL = "DELETE FROM archive WHERE " + _ARCHIVE_SQL_MATCH
|
||||
|
||||
|
||||
def broken_directories(images_root: Path) -> list[Path]:
|
||||
"""Every `<artist>/discord/None` directory. Artist folders that differ only by
|
||||
case (`Conto` and `conto`) are separate directories and both are found."""
|
||||
return sorted(d for d in Path(images_root).glob("*/discord/None") if d.is_dir())
|
||||
|
||||
|
||||
def count_archive_entries(archive: Path) -> int:
|
||||
return _archive(archive, delete=False)
|
||||
|
||||
|
||||
def forget_archive_entries(archive: Path) -> int:
|
||||
return _archive(archive, delete=True)
|
||||
|
||||
|
||||
def _archive(archive: Path, *, delete: bool) -> int:
|
||||
if not archive.is_file():
|
||||
return 0
|
||||
# A download running at the same moment holds this file briefly. Waiting
|
||||
# 30s for its lock beats failing the repair over a transient contention.
|
||||
conn = sqlite3.connect(str(archive), timeout=30)
|
||||
try:
|
||||
has_table = conn.execute(
|
||||
"SELECT 1 FROM sqlite_master WHERE type='table' AND name='archive'"
|
||||
).fetchone()
|
||||
if not has_table:
|
||||
return 0
|
||||
if not delete:
|
||||
return conn.execute(_COUNT_SQL).fetchone()[0]
|
||||
cur = conn.execute(_DELETE_SQL)
|
||||
conn.commit()
|
||||
return cur.rowcount
|
||||
finally:
|
||||
conn.close()
|
||||
|
||||
|
||||
def _sweep_directory(directory: Path) -> tuple[int, bool]:
|
||||
"""Remove what the record deletes left behind: the collided sidecars, plus any
|
||||
file that never became a record (a quarantined or rejected download). Then
|
||||
the directory itself, if it is empty. Returns (files removed, dir removed)."""
|
||||
removed = 0
|
||||
for f in directory.iterdir():
|
||||
if f.is_file():
|
||||
try:
|
||||
f.unlink()
|
||||
removed += 1
|
||||
except OSError as exc:
|
||||
log.warning("discord repair: could not remove %s: %s", f, exc)
|
||||
try:
|
||||
directory.rmdir()
|
||||
return removed, True
|
||||
except OSError:
|
||||
return removed, False
|
||||
|
||||
|
||||
def repair_discord_downloads(
|
||||
session: Session, *, images_root: Path, dry_run: bool,
|
||||
) -> dict:
|
||||
images_root = Path(images_root)
|
||||
archive = archive_path(images_root)
|
||||
|
||||
broken = select(ImageRecord.id, ImageRecord.size_bytes).where(
|
||||
ImageRecord.path.like(_BROKEN_PATH_LIKE, escape="\\")
|
||||
)
|
||||
rows = session.execute(broken).all()
|
||||
image_ids = [r.id for r in rows]
|
||||
directories = broken_directories(images_root)
|
||||
sources = session.execute(
|
||||
select(Source).where(Source.platform == "discord")
|
||||
).scalars().all()
|
||||
|
||||
summary = {
|
||||
"images": len(image_ids),
|
||||
"bytes": sum(r.size_bytes or 0 for r in rows),
|
||||
"directories": len(directories),
|
||||
"sources": len(sources),
|
||||
"enabled_sources": sum(1 for s in sources if s.enabled),
|
||||
}
|
||||
|
||||
if dry_run:
|
||||
summary["archive_entries"] = count_archive_entries(archive)
|
||||
return summary
|
||||
|
||||
deleted = delete_images(session, image_ids=image_ids, images_root=images_root)
|
||||
|
||||
swept = 0
|
||||
directories_removed = 0
|
||||
for d in directories:
|
||||
n, gone = _sweep_directory(d)
|
||||
swept += n
|
||||
directories_removed += int(gone)
|
||||
|
||||
# Only after the files are gone. Forgetting first and failing half way would
|
||||
# leave gallery-dl free to re-fetch into a directory still full of the old
|
||||
# copies.
|
||||
forgotten = forget_archive_entries(archive)
|
||||
|
||||
for source in sources:
|
||||
arm_backfill(source)
|
||||
session.commit()
|
||||
|
||||
remaining = session.execute(
|
||||
select(func.count(ImageRecord.id)).where(
|
||||
ImageRecord.path.like(_BROKEN_PATH_LIKE, escape="\\")
|
||||
)
|
||||
).scalar_one()
|
||||
|
||||
summary.update(
|
||||
images_deleted=deleted["images_deleted"],
|
||||
files_failed=deleted["files_failed"],
|
||||
leftover_files_removed=swept,
|
||||
directories_removed=directories_removed,
|
||||
archive_entries=forgotten,
|
||||
backfills_started=len(sources),
|
||||
remaining=remaining,
|
||||
)
|
||||
log.info("discord repair applied: %s", summary)
|
||||
return summary
|
||||
@@ -136,6 +136,16 @@ def metadata_postprocessor(filename: str) -> dict:
|
||||
return {"name": "metadata", "mode": "json", "directory": ".", "filename": filename}
|
||||
|
||||
|
||||
def archive_path(images_root: Path) -> Path:
|
||||
"""gallery-dl's download archive: the record of what it has already fetched.
|
||||
|
||||
One definition, because the Discord repair (services/discord_repair.py) has
|
||||
to find the same file the downloader writes, without constructing a service
|
||||
whose __init__ creates directories.
|
||||
"""
|
||||
return Path(images_root) / ".gallery-dl" / "archive.sqlite3"
|
||||
|
||||
|
||||
@dataclass
|
||||
class SourceConfig:
|
||||
"""Per-source overrides loaded from Source.config_overrides JSON.
|
||||
@@ -447,7 +457,7 @@ class GalleryDLService:
|
||||
config = {
|
||||
"extractor": {
|
||||
"base-directory": str(self.images_root),
|
||||
"archive": str(self._config_dir / "archive.sqlite3"),
|
||||
"archive": str(archive_path(self.images_root)),
|
||||
"skip": True,
|
||||
"sleep": self._rate_limit,
|
||||
"sleep-request": max(0.5, self._rate_limit / 4),
|
||||
|
||||
@@ -158,6 +158,19 @@ def _is_app_managed(key: str) -> bool:
|
||||
BACKFILL_MAX_CHUNKS = 200
|
||||
|
||||
|
||||
def arm_backfill(source: Source) -> None:
|
||||
"""Arm a fresh run-until-done backfill on `source` (plan #693). Mutation
|
||||
only — the caller commits. Shared by `SourceService.start_backfill` and the
|
||||
sync Discord repair, so both clear exactly the same resume state."""
|
||||
co = dict(source.config_overrides or {})
|
||||
co["_backfill_state"] = "running"
|
||||
for k in ("_backfill_cursor", "_backfill_cursor_stalls", "_backfill_chunks",
|
||||
"_backfill_posts"):
|
||||
co.pop(k, None)
|
||||
source.config_overrides = co
|
||||
source.backfill_runs_remaining = BACKFILL_MAX_CHUNKS
|
||||
|
||||
|
||||
class SourceService:
|
||||
def __init__(self, session: AsyncSession):
|
||||
self.session = session
|
||||
@@ -473,13 +486,7 @@ class SourceService:
|
||||
)).scalar_one_or_none()
|
||||
if source is None:
|
||||
raise LookupError(f"source id={source_id} not found")
|
||||
co = dict(source.config_overrides or {})
|
||||
co["_backfill_state"] = "running"
|
||||
for k in ("_backfill_cursor", "_backfill_cursor_stalls", "_backfill_chunks",
|
||||
"_backfill_posts"):
|
||||
co.pop(k, None)
|
||||
source.config_overrides = co
|
||||
source.backfill_runs_remaining = BACKFILL_MAX_CHUNKS
|
||||
arm_backfill(source)
|
||||
await self.session.commit()
|
||||
return await self._row_to_record(source)
|
||||
|
||||
|
||||
@@ -122,6 +122,28 @@ def prune_missing_file_records_task(self) -> dict:
|
||||
return {"checked": checked, "missing": len(missing_ids), "deleted": deleted}
|
||||
|
||||
|
||||
@celery.task(
|
||||
name="backend.app.tasks.admin.repair_discord_downloads_task",
|
||||
bind=True,
|
||||
autoretry_for=(OperationalError, DBAPIError),
|
||||
retry_backoff=15, retry_backoff_max=180, max_retries=1,
|
||||
soft_time_limit=1800, time_limit=2400, # 30 min / 40 min
|
||||
)
|
||||
def repair_discord_downloads_task(self, dry_run: bool = True) -> dict:
|
||||
"""Clean re-download of the Discord files broken by the `None` naming
|
||||
(#3999). dry_run (the default) returns the projection; apply deletes the
|
||||
broken images and their files, makes gallery-dl forget every Discord
|
||||
download, and restarts every Discord source's backfill. Defaults to the SAFE
|
||||
preview because the apply deletes files. Operator-triggered only."""
|
||||
from ..services.discord_repair import repair_discord_downloads
|
||||
|
||||
SessionLocal = _sync_session_factory()
|
||||
with SessionLocal() as session:
|
||||
return repair_discord_downloads(
|
||||
session, images_root=IMAGES_ROOT, dry_run=dry_run,
|
||||
)
|
||||
|
||||
|
||||
@celery.task(
|
||||
name="backend.app.tasks.admin.dedup_videos_task",
|
||||
bind=True,
|
||||
|
||||
@@ -0,0 +1,118 @@
|
||||
<template>
|
||||
<!-- #3999: Discord downloads made before the naming fix landed in a `None`
|
||||
folder with no post and a download-time date. This re-downloads them
|
||||
cleanly. Preview first; the apply deletes files. -->
|
||||
<MaintenanceTile
|
||||
icon="mdi-download-off-outline"
|
||||
title="Repair Discord downloads"
|
||||
blurb="Re-download Discord images that were saved with no post and the wrong date."
|
||||
destructive
|
||||
:open="applying || previewing"
|
||||
>
|
||||
<p class="text-body-2 mb-3">
|
||||
Before the naming fix, every Discord download was saved into a folder
|
||||
called <code>None</code>. Those images never got a post, so they show the
|
||||
time Curator downloaded them rather than when they were posted.
|
||||
<strong>Apply</strong> deletes those images, clears Discord from the
|
||||
download history, and starts a fresh backfill of every Discord source so
|
||||
they come back with their posts and dates. Tags you added by hand to those
|
||||
images are lost. Nothing outside a Discord <code>None</code> folder is
|
||||
touched.
|
||||
</p>
|
||||
|
||||
<div class="d-flex align-center flex-wrap" style="gap: 12px;">
|
||||
<v-btn
|
||||
color="primary" variant="tonal" rounded="pill"
|
||||
:loading="previewing" :disabled="applying" @click="preview"
|
||||
>
|
||||
<v-icon start>mdi-magnify</v-icon> Preview
|
||||
</v-btn>
|
||||
<v-btn
|
||||
color="error" rounded="pill"
|
||||
:loading="applying"
|
||||
:disabled="previewing || !canApply"
|
||||
@click="confirmOpen = true"
|
||||
>
|
||||
<v-icon start>mdi-download-off-outline</v-icon> Apply
|
||||
</v-btn>
|
||||
</div>
|
||||
|
||||
<v-alert
|
||||
v-if="summary" :type="summaryType" variant="tonal" class="mt-4"
|
||||
density="comfortable"
|
||||
>
|
||||
<span v-if="applied">
|
||||
Deleted {{ summary.images_deleted }} image(s) and cleared
|
||||
{{ summary.archive_entries }} Discord download record(s). Backfills
|
||||
started on {{ summary.backfills_started }} Discord source(s).
|
||||
</span>
|
||||
<span v-else-if="hasWork">
|
||||
{{ summary.images }} broken image(s) ({{ humanBytes(summary.bytes) }})
|
||||
across {{ summary.directories }} folder(s), and
|
||||
{{ summary.archive_entries }} Discord download record(s) to clear.
|
||||
{{ summary.sources }} Discord source(s) will backfill again.
|
||||
</span>
|
||||
<span v-else>Nothing to repair — no broken Discord downloads found.</span>
|
||||
|
||||
<div v-if="applied && summary.files_failed" class="mt-1 text-caption">
|
||||
{{ summary.files_failed }} file(s) could not be removed — see the worker log.
|
||||
</div>
|
||||
<div v-if="applied && summary.remaining" class="mt-1 text-caption">
|
||||
{{ summary.remaining }} broken image(s) are still present. Run it again.
|
||||
</div>
|
||||
</v-alert>
|
||||
|
||||
<QueueStatusBar queue="maintenance_long" queue-label="Maintenance" />
|
||||
|
||||
<v-dialog v-model="confirmOpen" max-width="440">
|
||||
<v-card>
|
||||
<v-card-title>Repair Discord downloads?</v-card-title>
|
||||
<v-card-text class="text-body-2">
|
||||
This permanently deletes <strong>{{ summary?.images ?? 0 }}</strong>
|
||||
Discord image(s) ({{ humanBytes(summary?.bytes) }}) and re-downloads
|
||||
them from Discord with their posts and dates. Any tags you added to
|
||||
those images by hand will not come back.
|
||||
</v-card-text>
|
||||
<v-card-actions>
|
||||
<v-spacer />
|
||||
<v-btn variant="text" @click="confirmOpen = false">Cancel</v-btn>
|
||||
<v-btn color="error" @click="apply">Repair</v-btn>
|
||||
</v-card-actions>
|
||||
</v-card>
|
||||
</v-dialog>
|
||||
</MaintenanceTile>
|
||||
</template>
|
||||
|
||||
<script setup>
|
||||
import { computed, ref } from 'vue'
|
||||
|
||||
import { useMaintenanceTask } from '../../composables/useMaintenanceTask.js'
|
||||
import { humanBytes } from '../../utils/bytes.js'
|
||||
import MaintenanceTile from '../common/MaintenanceTile.vue'
|
||||
import QueueStatusBar from './QueueStatusBar.vue'
|
||||
|
||||
const confirmOpen = ref(false)
|
||||
|
||||
const { previewing, applying, summary, applied, preview, apply: applyTask } = useMaintenanceTask({
|
||||
endpoint: '/api/admin/maintenance/repair-discord-downloads',
|
||||
storageKey: 'fc.maint.repairDiscordDownloads',
|
||||
appliedToast: 'Discord downloads cleared — backfills started',
|
||||
})
|
||||
|
||||
// The archive count matters too: files could already be gone while gallery-dl
|
||||
// still believes it has them, which would stop the backfill re-fetching.
|
||||
const hasWork = computed(
|
||||
() => !!summary.value && (summary.value.images > 0 || summary.value.archive_entries > 0),
|
||||
)
|
||||
const canApply = computed(() => hasWork.value && !applied.value)
|
||||
const summaryType = computed(() => {
|
||||
if (applied.value) return 'success'
|
||||
return hasWork.value ? 'info' : 'success'
|
||||
})
|
||||
|
||||
// The confirm dialog gates the destructive apply; close it, then run.
|
||||
function apply () {
|
||||
confirmOpen.value = false
|
||||
applyTask()
|
||||
}
|
||||
</script>
|
||||
@@ -15,6 +15,7 @@
|
||||
<ImportFiltersForm />
|
||||
<TranslationCard />
|
||||
<DiscordGroupingCard />
|
||||
<DiscordRepairCard />
|
||||
<PostAssociationsCard />
|
||||
<MembershipRosterCard />
|
||||
<MembershipSuggestionsCard />
|
||||
@@ -85,6 +86,7 @@ import VideoEmbeddingCard from './VideoEmbeddingCard.vue'
|
||||
import CropProposersCard from './CropProposersCard.vue'
|
||||
import HeadsCard from './HeadsCard.vue'
|
||||
import DiscordGroupingCard from './DiscordGroupingCard.vue'
|
||||
import DiscordRepairCard from './DiscordRepairCard.vue'
|
||||
import MembershipRosterCard from './MembershipRosterCard.vue'
|
||||
import MembershipSuggestionsCard from './MembershipSuggestionsCard.vue'
|
||||
import PostAssociationsCard from './PostAssociationsCard.vue'
|
||||
|
||||
@@ -0,0 +1,170 @@
|
||||
"""The Discord repair (#3999): forget the broken downloads, delete them, backfill again.
|
||||
|
||||
The danger in a repair like this is reach. It deletes files and clears
|
||||
gallery-dl's memory of what it fetched, so most of these tests pin what it must
|
||||
NOT touch: other platforms' archive entries, Discord server assets, images
|
||||
outside a `discord/None/` folder, and a correctly named Discord file that
|
||||
happens to share a folder name.
|
||||
"""
|
||||
|
||||
import sqlite3
|
||||
|
||||
import pytest
|
||||
from sqlalchemy import func, select
|
||||
|
||||
from backend.app.models import Artist, ImageRecord, Source
|
||||
from backend.app.services import discord_repair
|
||||
from backend.app.services.gallery_dl import archive_path
|
||||
from backend.app.services.source_service import BACKFILL_MAX_CHUNKS
|
||||
|
||||
|
||||
def _archive(tmp_path, entries):
|
||||
path = archive_path(tmp_path)
|
||||
path.parent.mkdir(parents=True, exist_ok=True)
|
||||
conn = sqlite3.connect(path)
|
||||
# The table gallery-dl itself creates (gallery_dl/archive.py).
|
||||
conn.execute("CREATE TABLE archive (entry TEXT PRIMARY KEY) WITHOUT ROWID")
|
||||
conn.executemany("INSERT INTO archive (entry) VALUES (?)", [(e,) for e in entries])
|
||||
conn.commit()
|
||||
conn.close()
|
||||
return path
|
||||
|
||||
|
||||
def _entries(path):
|
||||
conn = sqlite3.connect(path)
|
||||
try:
|
||||
return sorted(r[0] for r in conn.execute("SELECT entry FROM archive"))
|
||||
finally:
|
||||
conn.close()
|
||||
|
||||
|
||||
# --- the archive ------------------------------------------------------------
|
||||
|
||||
|
||||
def test_only_discord_message_entries_are_forgotten(tmp_path):
|
||||
path = _archive(tmp_path, [
|
||||
"discord300000000000000003_1",
|
||||
"discord300000000000000004_2",
|
||||
"discordasset_400000000000000004_9",
|
||||
"hentaifoundry12345",
|
||||
])
|
||||
assert discord_repair.count_archive_entries(path) == 2
|
||||
assert discord_repair.forget_archive_entries(path) == 2
|
||||
assert _entries(path) == ["discordasset_400000000000000004_9", "hentaifoundry12345"]
|
||||
|
||||
|
||||
def test_counting_changes_nothing(tmp_path):
|
||||
path = _archive(tmp_path, ["discord1_1", "discord2_1"])
|
||||
discord_repair.count_archive_entries(path)
|
||||
assert _entries(path) == ["discord1_1", "discord2_1"]
|
||||
|
||||
|
||||
def test_a_missing_archive_or_table_is_zero_not_an_error(tmp_path):
|
||||
assert discord_repair.count_archive_entries(tmp_path / "nope.sqlite3") == 0
|
||||
empty = tmp_path / "empty.sqlite3"
|
||||
sqlite3.connect(empty).close()
|
||||
assert discord_repair.forget_archive_entries(empty) == 0
|
||||
|
||||
|
||||
# --- the folders ------------------------------------------------------------
|
||||
|
||||
|
||||
def test_broken_directories_finds_every_artist_folder_including_case_twins(tmp_path):
|
||||
for artist in ("Conto", "conto", "knuxy"):
|
||||
(tmp_path / artist / "discord" / "None").mkdir(parents=True)
|
||||
(tmp_path / "conto" / "discord" / "general").mkdir(parents=True)
|
||||
(tmp_path / "conto" / "patreon" / "None").mkdir(parents=True)
|
||||
found = [
|
||||
p.relative_to(tmp_path).as_posix()
|
||||
for p in discord_repair.broken_directories(tmp_path)
|
||||
]
|
||||
assert found == ["Conto/discord/None", "conto/discord/None", "knuxy/discord/None"]
|
||||
|
||||
|
||||
def test_sweep_removes_leftover_sidecars_and_the_empty_folder(tmp_path):
|
||||
d = tmp_path / "conto" / "discord" / "None"
|
||||
d.mkdir(parents=True)
|
||||
(d / "image.json").write_text("{}")
|
||||
(d / "20240716_None_rejected.png").write_bytes(b"x")
|
||||
removed, gone = discord_repair._sweep_directory(d)
|
||||
assert (removed, gone) == (2, True)
|
||||
assert not d.exists()
|
||||
|
||||
|
||||
# --- the whole repair, against the database -----------------------------------
|
||||
|
||||
|
||||
def _image(db_sync, artist, path, sha):
|
||||
img = ImageRecord(
|
||||
artist_id=artist.id, path=str(path), sha256=sha * 64, size_bytes=100,
|
||||
mime="image/png", origin="downloaded",
|
||||
)
|
||||
db_sync.add(img)
|
||||
return img
|
||||
|
||||
|
||||
@pytest.mark.integration
|
||||
def test_repair_deletes_only_broken_discord_images_and_rearms_discord_backfills(
|
||||
db_sync, tmp_path,
|
||||
):
|
||||
artist = Artist(name="Conto", slug="conto")
|
||||
db_sync.add(artist)
|
||||
db_sync.flush()
|
||||
discord = Source(
|
||||
artist_id=artist.id, platform="discord", url="https://discord.com/channels/1/2",
|
||||
enabled=True, config_overrides={"_backfill_state": "complete", "_backfill_cursor": "x"},
|
||||
)
|
||||
patreon = Source(
|
||||
artist_id=artist.id, platform="patreon", url="https://www.patreon.com/conto",
|
||||
enabled=True, config_overrides={},
|
||||
)
|
||||
db_sync.add_all([discord, patreon])
|
||||
|
||||
broken_dir = tmp_path / "conto" / "discord" / "None"
|
||||
good_dir = tmp_path / "conto" / "discord" / "general"
|
||||
patreon_dir = tmp_path / "conto" / "patreon" / "None"
|
||||
for d in (broken_dir, good_dir, patreon_dir):
|
||||
d.mkdir(parents=True)
|
||||
broken = broken_dir / "20240716_None_image.png"
|
||||
good = good_dir / "20240716_300000000000000003_01_image.png"
|
||||
# A `None` folder under another platform, and a `_None_` name outside one.
|
||||
# Neither is the bug's pair, so neither may be touched.
|
||||
elsewhere = patreon_dir / "20240716_None_image.png"
|
||||
for f in (broken, good, elsewhere):
|
||||
f.write_bytes(b"x")
|
||||
(broken_dir / "image.json").write_text("{}")
|
||||
_image(db_sync, artist, broken, "a")
|
||||
_image(db_sync, artist, good, "b")
|
||||
_image(db_sync, artist, elsewhere, "c")
|
||||
db_sync.commit()
|
||||
_archive(tmp_path, ["discord300000000000000003_1", "hentaifoundry1"])
|
||||
|
||||
preview = discord_repair.repair_discord_downloads(db_sync, images_root=tmp_path, dry_run=True)
|
||||
assert preview["images"] == 1
|
||||
assert preview["archive_entries"] == 1
|
||||
assert broken.exists()
|
||||
|
||||
result = discord_repair.repair_discord_downloads(db_sync, images_root=tmp_path, dry_run=False)
|
||||
assert result["images_deleted"] == 1
|
||||
assert result["remaining"] == 0
|
||||
assert result["backfills_started"] == 1
|
||||
|
||||
assert not broken.exists() and not broken_dir.exists()
|
||||
assert good.exists() and elsewhere.exists()
|
||||
assert _entries(archive_path(tmp_path)) == ["hentaifoundry1"]
|
||||
|
||||
kept = db_sync.execute(select(func.count(ImageRecord.id))).scalar_one()
|
||||
assert kept == 2
|
||||
|
||||
rows = {
|
||||
platform: (runs, overrides)
|
||||
for platform, runs, overrides in db_sync.execute(
|
||||
select(Source.platform, Source.backfill_runs_remaining, Source.config_overrides)
|
||||
).all()
|
||||
}
|
||||
runs, overrides = rows["discord"]
|
||||
assert runs == BACKFILL_MAX_CHUNKS
|
||||
assert overrides["_backfill_state"] == "running"
|
||||
assert "_backfill_cursor" not in overrides
|
||||
# Only Discord sources are re-armed.
|
||||
assert rows["patreon"][1] == {}
|
||||
Reference in New Issue
Block a user