bvandeusen/StashHandler
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Implement per-file import for in-progress torrents

Browse Source 
Files within a downloading torrent are now imported as soon as their
individual progress reaches 100%, without waiting for the full torrent
to complete. A new processed_files table (info_hash, file_name) tracks
which files have been handled to avoid redundant re-hashing each poll.

- db.py: add processed_files table, is_file_processed, record_processed_file
- qbit_client.py: replace get_completed_torrents with get_torrents(filter, category)
- stash_handler.py: extract process_torrent_files helper, daemon now polls
  both downloading and completed torrents each cycle

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
This commit is contained in:
Bryan Van Deusen
2026-02-18 21:51:52 -05:00
parent 6adfd77236
commit 5bcb35e737
4 changed files with 134 additions and 42 deletions
Download Patch File Download Diff File Expand all files Collapse all files
summary.md
+14 -5
View File
@@ -4,7 +4,7 @@
## Purpose
StashHandler sits between qBittorrent and Stash (both running in Docker). It polls qBit's API for completed torrents filtered by category, hashes media files, deduplicates them via a SQLite database, and copies new files into Stash's import directory (preserving originals for continued seeding).
StashHandler sits between qBittorrent and Stash (both running in Docker). It polls qBit's API for torrents filtered by category, hashes media files, deduplicates them via a SQLite database, and copies new files into Stash's import directory (preserving originals for continued seeding). Individual files within in-progress torrents are imported as soon as they complete — the daemon does not wait for the full torrent to finish.
**Problems solved:**
1. Deleted files in Stash don't get reimported (hash DB remembers them)
@@ -29,7 +29,7 @@ Designed for Docker Swarm — no host-local config files, everything via env var
| File | Purpose |
|---|---|
| `stash_handler.py` | Entry point — `daemon` and `scan` CLI commands, config loading from env vars |
| `db.py` | SQLite operations — `init_db`, `is_known`, `record_file`, `is_torrent_processed`, `record_torrent`, `increment_seen` |
| `db.py` | SQLite operations — `init_db`, `is_known`, `record_file`, `is_torrent_processed`, `record_torrent`, `increment_seen`, `is_file_processed`, `record_processed_file` |
| `hasher.py` | Streaming file hashing (sha256 default, blake3 optional) in 64KB chunks |
| `qbit_client.py` | qBittorrent API client — login, session cookies, auto re-auth on 403 |
| `Dockerfile` | python:3.12-slim, entrypoint is `stash_handler.py`, default command `daemon` |
@@ -45,9 +45,11 @@ docker run stash-handler
# or: python stash_handler.py daemon
```
- Polls qBit API at `POLL_INTERVAL` seconds
- Fetches completed torrents via `/api/v2/torrents/info?filter=completed&category=<QBIT_CATEGORY>`
- Skips already-processed torrents (by info_hash in `processed_torrents` table)
- For each new torrent: walks media files, hashes, deduplicates, transfers new files to import dir
- Each poll fetches both `downloading` and `completed` torrents filtered by `QBIT_CATEGORY`
- Uses `/api/v2/torrents/files` to get per-file progress (0.0–1.0) for each torrent
- Files with `progress < 1.0` are skipped — imported as soon as they reach 1.0
- Per-file state tracked in `processed_files` table to avoid re-hashing on subsequent polls
- Completed torrents are marked in `processed_torrents` and never checked again
- Clean shutdown on SIGINT/SIGTERM (1-second sleep granularity)
### Scan mode
@@ -95,6 +97,13 @@ CREATE TABLE processed_torrents (
torrent_name TEXT,
processed_at TEXT NOT NULL
);
CREATE TABLE processed_files (
info_hash TEXT NOT NULL,
file_name TEXT NOT NULL, -- relative to qBit save_path (= staging root)
processed_at TEXT NOT NULL,
PRIMARY KEY (info_hash, file_name)
);
```
## Volume Layout (Docker)