Project · ingestion pipeline

Inflow RSS Pipeline

A curated RSS/Atom ingestion pipeline that polls configured feeds, deduplicates globally by canonical URL in SQLite, and writes daily digest pages to the library.

Boundary: This is a read-only ingestion tool. It does not publish, syndicate, or republish feed content. Digest pages are internal library artifacts with source links and UTC timestamps.

How it works

Reads feeds.yaml (curated source list with per-feed settings).
Fetches each feed with feedparser; extracts article bodies via Trafilatura.
Canonicalises URLs (strips tracking params, normalises scheme/host) and deduplicates in state.db.
First run is a baseline — only new items after the baseline are appended.
New items are written to library/inflow/YYYY-MM-DD.md with title, URL, source, published date, and extracted text.
Rebuilds the library index and commits changed digest pages.

Sources and schedule

Feeds are defined in feeds.yaml. The pipeline runs every 2 hours via Hermes cron (inflow-rss-2h job). No external authentication is required; all feeds are public.

Verification

Run python3 -m unittest discover -s tests -v. Inspect state.db for deduplication state and library/inflow/ for dated digests. The pipeline is deterministic given the same feed responses.