Backtest: would the radar have called it?

Finished runs

Every finished run keeps its own page, with its results and hashes as they were published.

a4-2026-10-28

The frame

The frame is repositories created from 1 January 2023 to 30 June 2024 that reached 25 distinct stargazers within 60 days. The replay keeps one repository in 32, chosen by a fixed hash of its repository id (cityHash64(repo_id) % 32 = 0). The hash is of the id, not of stars, so the sample cannot select on the outcome: a repo's later popularity plays no part in whether it is in the sample.

What could be rebuilt from history

The replay has GH Archive and the stargazer counts, so it can rebuild references (other projects with a different owner linking to the repo in issues and pull requests), contributors (outside authors with merged pull requests) and momentum (28-day star and fork velocity). Stars and forks are the inputs to momentum.

It cannot rebuild adoption (direct dependents and weekly downloads), because GH Archive does not record them. Pedigree and discussion are not scored in the replay, so they are left out of its scores. The live radar gives adoption its weight once it has real results.

The hit definition in the backtest

The live definition of a hit needs any two of four conditions: stars (five times the count at the spot, and at least 2,000 within 180 days), dependents (at least 100), references (at least 25 referencing projects in 60 days, each with 100 or more stars) and weekly downloads (at least 10,000). Dependents and downloads cannot be rebuilt from GH Archive. The backtesttherefore asks for the two conditions it can rebuild, stars and references, and requires both. The references count here has no star qualifier, because the frame holds no star history for the referencing projects.

A backtest hit is not a live hit. A rate here answers a narrower question than the live one, and the two are never mixed. The lead time is the number of weeks from the spot to the later of the two conditions.

Frame composition

Many repositories in the frame have names that advertise game cheats, cracked software, key generators, activators or free downloads. This study cannot vouch for their star counts, so it reports their share and shows every result twice: for the whole frame and without those names.

922 of 2,774 frame repos (33.2%) have such names.

A name matches when the repo name (not the owner) contains one of these words once separators are removed and digits used as letters are read back (so C-R-A-C-K and Ch3at match): cheat, cheet, crack, keygen, activator, activation, aimbot, wallhack, triggerbot, norecoil, silentaim, spoof, hwid, modmenu, skinchanger, unlocker, synapsex, executor, drainer, grabber, freedownload, hackaim, hackesp, esphack, hck; or has one of these as a whole word: hack, hacks, hackk, hax, esp, aim. The list is fixed for the run (rule lure-names/1). It reads names only, so the share is a floor: repos named only after a game or a product are not caught.

Progress and method

The run has four stages: the frame (one band query per 28 creation days), the weekly series for each frame repo, the references counted from GH Archive hourly files in one-hour slices, and the replay that writes the results. The frame and the weekly series stay inside the ClickHouse budget. The full run is about 7 days.

The method and the measured costs are in docs/radar/BACKTEST.md. Its limits are listed below.

Results

2 hours of GH Archive were never published; links in those hours are not counted.

Frame repos: 2,774
Spotted repos (calls): 144
Resolved spotted repos: 144
Hit rate, spotted: 0.0% (95% interval 0.0% to 2.6%, n = 144)
Base sample: 200 never-spotted frame repos, same seeded mix; 200 resolved
Hit rate, base sample: 0.0% (95% interval 0.0% to 1.9%, n = 200)
Lift: Not measurable: the base sample has no hits, so there is no rate to divide by
Median lead time: None of the 0 arrivals has a lead time

Intervals are 95% Wilson intervals. Each rate's n is its resolved count. The base rate is the same kind of sample as the live base rate, drawn from never-spotted frame repos with a fixed seed. Lift is the spotted hit rate divided by the base-sample hit rate.

Without cheat and crack names

The same replay over the 1,852 frame repos whose names do not match the word list in Frame composition. Percentiles, calls and the base sample are all recomputed for this smaller frame. Read both runs: a result that holds only with those names included is not a result about ordinary projects.

Spotted repos (calls): 146, 146 resolved
Hit rate, spotted: 0.0% (95% interval 0.0% to 2.6%, n = 146)
Hit rate, base sample: 0.0% (95% interval 0.0% to 1.9%, n = 200) (200 drawn, 200 resolved)
Lift: Not measurable: the base sample has no hits, so there is no rate to divide by
Median lead time: No lead time

Manifest and hashes

Cohort label: a3-2026-10-20
Permanent page for this run: /-/study/radar-backtest/a3-2026-10-20
Method: replay/2026.10.1
Last complete week: 2025-05-11
Seed of the base-sample draw: 20230115
References: Measured: 21,792 of 21,792 GH Archive slices stored, 1,536 project-to-repo pairs, 4,130,855,311 link events read
GH Archive hours never published: 2
2025-06-12T17:00:00.000Z
2025-06-12T18:00:00.000Z

SHA-256 values

Input (the frame and weekly series the replay read)
Output (the replayed calls, base sample and rates)
Frame queries (the SQL of every frame page)
Merged reference pairs

To verify: the replay is seeded and reads only stored rows, so a rerun of the same cohort label must give the same hashes. A different hash means the inputs or the method changed. Compare the value above with the one a rerun reports for the same cohort label.

Limits

It is a replay of history. Nothing here is a live call, and nothing here predicts a live result.
Dependents and downloads are not rebuilt, so the hit definition is two of the four live conditions.
The references count has no star qualifier for the referencing projects.
Every frame repo shares one category, so percentiles are taken within the age band only.
The frame is a sample: one repo in 32, by a fixed hash of its id.
Anyone can buy stars for any repo. A hit here can be a repo whose stars were bought.
The data runs from 2023 to mid-2025, before GitHub stopped publishing repository-creation events (October 2025) and before GH Archive's hourly files lost most events (2026). The replay scores a complete sample; the live radar can score only the repos it finds, and since then it finds far fewer of them.

The live track record

The live record is separate from this study. Track record lists every live call and its outcome.