Inside the GitPop sync pipeline: how thousands of repos stay fresh

Every refresh cycle, the GitPop sync pipeline touches thousands of repos and recomputes their PopScore, health, and maturity data. This is what that pipeline looks like, and the design decisions behind it.

The architecture (one diagram)

   ┌─────────────────┐
   │ Scheduler       │  fixed daily cadence
   └────────┬────────┘
            │  secret-protected internal trigger
            ▼
   ┌─────────────────┐
   │  discovery      │  GitHub Topics search → candidate list
   │                 │  noise pre-filter skips most junk early
   └────────┬────────┘
            ▼
   ┌─────────────────┐
   │  fetch + score  │  GraphQL bulk queries, rate-budget-aware backoff
   └────────┬────────┘
            │  upsert per repo
            ▼
   ┌─────────────────┐
   │  Postgres       │  single source of truth
   └─────────────────┘

The four pieces, in order

1. Discovery

GitHub Topics are the input — a curated, rotating seed list spanning AI, frontend, backend, devops, and beyond. Every topic gets regular coverage.

For each topic we pull top repos by stars. From that candidate list, a name-pattern pre-filter strips out obvious awesome-* and tutorial repos before any deep work starts. Skipping cheap noise early is what keeps the whole thing efficient.

2. Per-repo fetch

For each candidate we query the GitHub GraphQL API for:

• The repo metadata (owner, name, description, language, license, topics, counts)
• Recent stargazer history (for velocity)
• Weekly commits, merged PRs, issues opened/closed
• External mentions (HN, Reddit, Lobsters) from our own mention pipeline

Concurrency is adaptive: the fetcher watches rate-limit headers and backs off as it approaches them. Freshness comes from patience, not brute force.

3. Score recompute

After every successful fetch we recompute:

• PopScore (0–100) — see the methodology page
• Noise score (0–100) — name pattern, fork ratio, asset quality, activity
• Health score (0–100) — activity, responsiveness, recency
• Maturity tier — toy / early / mature / enterprise

High-noise repos are hidden from the radar entirely; they still appear in the directory with a flag badge rather than silently vanishing.

4. Persistence

One idempotent upsert per repo — each repo is its own transaction. Development runs against an embedded Postgres instance with an identical schema to production, so behavior is identical in both environments.

What fails when something fails

The pipeline is idempotent and resumable. If a run fails partway:

• The job record stays open so the next scheduled invocation picks up where it left off
• We never write partial data — a half-fetched repo simply rolls forward next time
• The mention scrapers are fully decoupled and fail independently of the main sync

The main failure modes we've handled in production:

• Rate-limit hits — back off, continue with what we have, finish later
• Database connection blips — bounded retries, then log-and-continue with the failure captured on the job record

What we deliberately didn't do

A few things we could do but chose not to:

• WebSocket-driven live ticker. A short polling interval is more than enough when data freshness is measured in hours.
• A separate popularity dataset beyond GitHub. We considered many extra sources, but GitHub's own signal is the dominant one for OSS.
• Per-project schedulers. One well-bucketed schedule scales across far more topics than we track today.
• AI summaries inside the sync loop. Snapshots are generated lazily — only when a project is viewed — keeping the sync loop lean.

The bottom line

Discover daily, fetch patiently, filter aggressively, store idempotently. The result: a radar that's always fresh without being fragile. Questions? Find us on the GitPop org.