BazaarGhost finds opponents in Twitch VODs of The Bazaar and makes those appearances searchable at bazaarghost.stream. This repository contains the cataloger, processing pipeline, database, and Discord bot integration.
A detection means a sampled frame contained a rank emblem and an opponent name that passed OCR validation. It includes the source VOD, absolute second, rank, OCR confidence, screenshot, truncation flags, and optional in-game day. It is not a complete match history or a claim that OCR is always correct.
- Twitch discovery catalogs new streamers with processing enabled and creates their EventSub subscriptions.
- The cataloger identifies Bazaar chapter ranges. PostgreSQL plans missing work in chunks of at most 1,800 seconds, including short tails. An empty Bazaar range list creates no work.
- EventSub or the scheduler asks
process-vodto dispatch pending chunks to GitHub Actions. Only available, ready VODs belonging to enabled streamers qualify. - Each SFDE container atomically claims one chunk. Streamlink resolves the rendition; FFmpeg seeks the HLS playlist, crops, and samples at 0.5 FPS. OpenCV detects the nameplate and PaddleOCR reads the opponent.
- The result worker uploads screenshots before inserting detections. A chunk completes only after all frames and the final result batch drain successfully. Search RPCs hide unavailable VODs.
| Path | Responsibility |
|---|---|
sfde/src/sfde.py, video.py |
Chunk ownership, decoder/OCR/result workers, timestamp pairing |
sfde/src/frame_processor.py |
Nameplate crop, OCR acceptance, duplicate suppression, optional day OCR |
sfde/src/*_detector.py |
Resolution-specific OpenCV template matching |
sfde/src/supabase_client.py |
Chunk state, screenshot persistence, retry-safe detection inserts |
supabase/functions/_shared/ |
Twitch API, authentication, cataloging, dispatch, telemetry |
supabase/functions/ |
Discovery, cataloging, EventSub, availability checks, Discord, maintenance endpoints |
supabase/migrations/ |
Versioned schema and processing RPCs |
supabase/tests/, sfde/tests/, scripts/tests/ |
Database, CV/OCR, pipeline, and workflow regressions |
scripts/vod_workflow.py |
Validated workflow inputs, rendition selection, failure reporting |
.github/workflows/ |
Tests, processing, deployment, Discord command registration |
Install Docker, Supabase CLI, and Deno. Work against local Supabase first:
supabase start
supabase db reset
supabase test db
deno check supabase/functions/*/index.ts
deno test --allow-env supabase/functions/_shared/*_test.ts
python -m unittest discover -s scripts/testsdb reset resets the local database and applies migrations plus any configured seed. Create migrations with supabase migration new <name>; do not create migration files manually.
For the complete offline SFDE suite, including cached OCR models and a real FFmpeg/HLS seek test:
docker build --target test -t sfde:test sfde/
docker run --rm --network none sfde:testModel downloads happen during the image build. Test execution needs no network or credentials. See SFDE usage for a real chunk run.
Use a root .env.dev file for local processing. Nothing implicitly loads .env or another credentials file:
SUPABASE_URL=http://127.0.0.1:54321
SUPABASE_SECRET_KEY=<local secret or service-role key>
TWITCH_CLIENT_ID=<development application ID>
TWITCH_CLIENT_SECRET=<development application secret>Optional OTLP settings are OTEL_EXPORTER_OTLP_ENDPOINT and OTEL_EXPORTER_OTLP_HEADERS. Disabled or failing telemetry must not determine a processing outcome.
Internal edge endpoints validate apikey or bearer secret credentials in application code. SECRET_KEY is also accepted for hosted functions where the SUPABASE_ prefix is reserved. EventSub and Discord requests use their provider signatures. Existing deployment uses --no-verify-jwt; local gateway settings are unchanged, so use a locally accepted JWT when invoking a gateway configured to verify JWTs.
seed-from-prod.shrequires an explicitPROD_DB_URL. It reads a bounded catalog snapshot intosupabase/seed.sql, uses current column names, and disables processing in the generated seed. Storage objects are not copied.sync-prod-to-dev.shrequires explicit source/destination connections and confirmation. It imports that catalog in one transaction, preserving migration-managed schemas, roles, and Vault. Apply the same migrations to dev first. This is a catalog refresh, not a full database clone.clear-detections-bucket.tslists files by default.--executedeletes them. It paginates and stops on errors. SupplySUPABASE_URLandSUPABASE_SECRET_KEYexplicitly.Clear Dev Storageempties the devdetectionsbucket every Sunday at 10:00 UTC. It uses dev credentials and refuses any other project URL. Detection records remain searchable, but their old screenshots are deleted.setup_test_fixtures.pycopies validated local annotations and images into the committed fixture corpus.
Production connections are unnecessary for local tests. Put experiments and generated output in .ignore/.
dev and main target development and production respectively. Existing workflows deploy changed functions and migrations when pushed. The cleanup migration must be applied before starting workers that call claim_sfde_chunk. Validate locally, deploy/test dev, then have the maintainer merge to main and perform production verification.
This project is not currently accepting external contributions and has no license grant; all rights reserved.