bvandeusen
5bcb35e737
Files within a downloading torrent are now imported as soon as their individual progress reaches 100%, without waiting for the full torrent to complete. A new processed_files table (info_hash, file_name) tracks which files have been handled to avoid redundant re-hashing each poll. - db.py: add processed_files table, is_file_processed, record_processed_file - qbit_client.py: replace get_completed_torrents with get_torrents(filter, category) - stash_handler.py: extract process_torrent_files helper, daemon now polls both downloading and completed torrents each cycle Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
117 lines
4.9 KiB
Markdown
117 lines
4.9 KiB
Markdown
# StashHandler — Project Summary
|
||
|
||
> **NOTE:** Update this file after any significant changes to the project (new features, architecture changes, new env vars, schema changes, etc.)
|
||
|
||
## Purpose
|
||
|
||
StashHandler sits between qBittorrent and Stash (both running in Docker). It polls qBit's API for torrents filtered by category, hashes media files, deduplicates them via a SQLite database, and copies new files into Stash's import directory (preserving originals for continued seeding). Individual files within in-progress torrents are imported as soon as they complete — the daemon does not wait for the full torrent to finish.
|
||
|
||
**Problems solved:**
|
||
1. Deleted files in Stash don't get reimported (hash DB remembers them)
|
||
2. Duplicate content is caught before reaching Stash
|
||
3. Existing library files can be recorded for dedup without re-processing
|
||
|
||
## Architecture
|
||
|
||
```
|
||
qBit (Docker) StashHandler (Docker) Stash (Docker)
|
||
/downloads/ ──bind──> /staging (read-only)
|
||
polls qBit API
|
||
hashes files, checks SQLite
|
||
/import (read-write) ──bind──> /data/import
|
||
/data/seen.db (persistent)
|
||
```
|
||
|
||
Designed for Docker Swarm — no host-local config files, everything via env vars.
|
||
|
||
## Files
|
||
|
||
| File | Purpose |
|
||
|---|---|
|
||
| `stash_handler.py` | Entry point — `daemon` and `scan` CLI commands, config loading from env vars |
|
||
| `db.py` | SQLite operations — `init_db`, `is_known`, `record_file`, `is_torrent_processed`, `record_torrent`, `increment_seen`, `is_file_processed`, `record_processed_file` |
|
||
| `hasher.py` | Streaming file hashing (sha256 default, blake3 optional) in 64KB chunks |
|
||
| `qbit_client.py` | qBittorrent API client — login, session cookies, auto re-auth on 403 |
|
||
| `Dockerfile` | python:3.12-slim, entrypoint is `stash_handler.py`, default command `daemon` |
|
||
| `docker-compose.yaml` | Dev compose — builds from source, includes qbittorrent + stash, DEBUG logging |
|
||
| `docker-compose.prod.example.yaml` | Prod compose template — pulls from `git.fabledsword.com/bvandeusen/stashhandler:latest`, stash-handler only |
|
||
| `requirements.txt` | `requests` (only dependency) |
|
||
|
||
## Modes of Operation
|
||
|
||
### Daemon mode (default)
|
||
```
|
||
docker run stash-handler
|
||
# or: python stash_handler.py daemon
|
||
```
|
||
- Polls qBit API at `POLL_INTERVAL` seconds
|
||
- Each poll fetches both `downloading` and `completed` torrents filtered by `QBIT_CATEGORY`
|
||
- Uses `/api/v2/torrents/files` to get per-file progress (0.0–1.0) for each torrent
|
||
- Files with `progress < 1.0` are skipped — imported as soon as they reach 1.0
|
||
- Per-file state tracked in `processed_files` table to avoid re-hashing on subsequent polls
|
||
- Completed torrents are marked in `processed_torrents` and never checked again
|
||
- Clean shutdown on SIGINT/SIGTERM (1-second sleep granularity)
|
||
|
||
### Scan mode
|
||
```
|
||
docker exec stash-handler python stash_handler.py scan [--import] [path]
|
||
```
|
||
- `scan /staging` — record hashes only (bootstrap dedup DB, no files moved)
|
||
- `scan --import /staging` — hash and move new files to import dir
|
||
- `scan /stash-library` — record existing Stash library hashes (mount read-only)
|
||
- Prints summary: total scanned, new, already known
|
||
|
||
## Environment Variables
|
||
|
||
| Variable | Default | Description |
|
||
|---|---|---|
|
||
| `QBIT_URL` | `http://qbittorrent:8080` | qBittorrent Web UI URL |
|
||
| `QBIT_USERNAME` | `admin` | qBit login username |
|
||
| `QBIT_PASSWORD` | `adminadmin` | qBit login password |
|
||
| `STAGING_DIR` | `/staging` | Mount point for qBit's download directory |
|
||
| `IMPORT_DIR` | `/import` | Mount point for Stash's import directory |
|
||
| `DB_PATH` | `/data/seen.db` | Path to SQLite database |
|
||
| `POLL_INTERVAL` | `60` | Seconds between qBit API polls |
|
||
| `MEDIA_EXTENSIONS` | `.mp4,.mkv,.avi,.wmv,.mov,.webm,.flv,.m4v` | Comma-separated list of media file extensions |
|
||
| `HASH_ALGORITHM` | `sha256` | Hash algorithm (`sha256` or `blake3`) |
|
||
| `MOVE_MODE` | `copy` | File transfer mode: `copy`, `hardlink`, or `move` |
|
||
| `QBIT_CATEGORY` | `Stash` | qBit category to filter torrents by (shared qBit instance support) |
|
||
| `LOG_LEVEL` | `INFO` | Python logging level |
|
||
|
||
## Database Schema
|
||
|
||
SQLite at `DB_PATH`, WAL mode enabled.
|
||
|
||
```sql
|
||
CREATE TABLE seen_files (
|
||
hash TEXT PRIMARY KEY,
|
||
filename TEXT NOT NULL,
|
||
source TEXT, -- "torrent", "scan", "bulk"
|
||
info_hash TEXT,
|
||
first_seen TEXT NOT NULL,
|
||
times_seen INTEGER DEFAULT 1
|
||
);
|
||
|
||
CREATE TABLE processed_torrents (
|
||
info_hash TEXT PRIMARY KEY,
|
||
torrent_name TEXT,
|
||
processed_at TEXT NOT NULL
|
||
);
|
||
|
||
CREATE TABLE processed_files (
|
||
info_hash TEXT NOT NULL,
|
||
file_name TEXT NOT NULL, -- relative to qBit save_path (= staging root)
|
||
processed_at TEXT NOT NULL,
|
||
PRIMARY KEY (info_hash, file_name)
|
||
);
|
||
```
|
||
|
||
## Volume Layout (Docker)
|
||
|
||
```yaml
|
||
volumes:
|
||
- /data/staging:/staging:ro # qBit's download dir (read-only)
|
||
- /data/stash-import:/import # Stash's import dir (read-write)
|
||
- /data/stash-handler:/data # persistent DB
|
||
```
|