Demonblade Labs · Issue 003
I Built A Gym That Judges Its Own Best Strategies As Luck
My autonomous system searched thousands of trading strategies and found candidates that looked brilliant on the data they were built on. Then I built a judge whose only job is to treat them as luck until proven otherwise. On real market data, none of them passed. That is the system working, and the maths of why is one line you can run in a spreadsheet.
If you trade, or you have ever been shown a backtest, this one is for you. It sells nothing and it costs nothing. Everything here runs on paper, on a box that costs me about thirty dollars a month. No market data is published, because the licence on it says I cannot. What you get is the method, the maths and the engineering receipts, so you can check your own numbers against the same bar.
What you will get out of it
- The one number that explains why a steady 1 percent a day is a fraud band, with the maths.
- The expected best Sharpe of N strategies that have no edge at all, at four values of N.
- A five part judge, described plainly enough to build your own.
- Receipts on synthetic data, where the right answer is known.
- Three questions to ask of your own backtests today.
Chapter 0. The number is 37.78
A strategy that makes 1 percent a day sounds modest. Compounded for a year it is 1.01 to the power of 365, which is 37.78 times your money. Ten thousand dollars becomes about three hundred and seventy-eight thousand.
If you are ever shown a steady daily 1 percent, the right first question is where the audited track record is. That figure is not a strategy. It is the band that backtests land in when enough of them are tried.
Chapter 1. Why backtests land there
Take a strategy with no edge at all, only noise. Measure its Sharpe ratio over one year of data and the measurement wobbles with a standard error of about 1. Now try many of them and keep the best. The expected best of N noise strategies, in units of that wobble, grows slowly with N.
| Strategies tried (N) | Expected best Sharpe from pure noise |
|---|---|
| 100 | 2.53 |
| 10,000 | 3.86 |
| 1,000,000 | 4.87 |
| 47,500,000 | 5.58 |
That is the expected maximum of N standard normal draws, using the Bailey and Lopez de Prado approximation: (1 minus g) times the normal quantile at 1 minus 1/N, plus g times the quantile at 1 minus 1/(N times e), with g = 0.5772. Computed on 30 October 2026. Window assumed: one year.
Now the literature. A practitioner post (a blog, not peer reviewed, so weigh it that way) ran about 100 million bitcoin backtests, reported a raw Sharpe of 5.59 for its winner out of 47.5 million candidates, and showed that after deflating for the number of tries the winner came to about 0.10. I have not re-run their data. What I can offer is my own arithmetic: if their test window was about a year, which I do not know, then 5.58 is what pure noise hands you as the best of 47.5 million, and 5.59 sits right on that line. A beautiful winner and an expected winner can be the same number.
That is the trap. The more strategies you search, the better the best one looks, even when none of them works. Search is not the problem. Reporting the winner without the number of tries is.
Chapter 2. The judge, in five parts
I built the judge before I trusted the search. A candidate has to survive all five.
- Deflated Sharpe. The Sharpe ratio is corrected for the number of tries and for skew and fat tails in the returns, so a winner has to beat the luck bar for its own search size.
- Probability of backtest overfitting, by combinatorial cross-validation. The history is cut into 8 blocks, every way of training on 4 and testing on the other 4 is tried (70 combinations), and the question is how often the in-sample winner lands below the median out of sample. Over 0.5 and the candidate is marked OVERFIT.
- A thymus. In the immune system, cells that react to the body itself are killed before they leave the thymus. Mine scores every candidate three times on shuffled returns, where there is no edge by construction. A strategy that works on shuffled noise is reacting to itself, and it dies there.
- A cross-asset holdout. A champion is re-scored on every other market's training slice. An edge that exists in one instrument and nowhere else is a coincidence until shown otherwise.
- Forward paper days. Real days that happen after the search is sealed, so nothing can be tuned to them.
The verdict has two words: OVERFIT or PROVISIONAL. Never PASS. A system that is allowed to say PASS about itself will, eventually.
Chapter 3. Receipts on data where the answer is known
These are synthetic fixtures, so they prove the instruments and not a market.
| Instrument | Result |
|---|---|
| Immune-style search, first passer on a planted edge (median trials) | 15, against 45 for plain random search |
| Hidden-state model recovering planted regimes | 87.4 percent accuracy |
| Critical-slowing early warning on a planted crash | 19 of 20 runs, against 2 of 20 with the crash removed |
The immune-style search uses mutation, crossover, a memory of past regimes and a thymus that removes self-reactive candidates. The accuracy and detection figures were re-run on 30 October 2026 from the test suite (17 tests, all passing).
Chapter 4. The engineering, because speed makes a strict judge affordable
The search covers 100 pairs registered as cells. A cached indicator layer and batch evaluation took a real mix of strategy specs from 1.4 trials per second to 72.5 on the same loaded box, about 52 times, with 0 mismatches against the slow reference across 1,939 stress specs. 251 tests were green across the 14 markets modules at that commit, and 22 commits landed between 13:09 and 20:16 on 30 October. All of it is paper. No order has been placed.
The limits of all of this
The things I would attack if I were reading this.
- The 47.5 million figure and the 0.10 come from a blog post I have not re-run. The one year window in my arithmetic is an assumption.
- Synthetic fixtures prove the code does what I say, not that any market has an edge.
- The 15 versus 45 is a median over a small set of seeds.
- A judge this strict can also throw out a real edge. That is a cost I have chosen on purpose, and it is why forward days exist.
- On real market data, none of the candidates passed. That sentence states an outcome only. The market data is licensed for research, so no figure computed from it is published.
Your three questions
If you want to apply this to your own backtests today, three questions cost nothing. How many variants did you try before this one, including the ones you quietly dropped? What is the best Sharpe pure noise would have given you at that count? And does your winner still beat it on data it has never touched? If you cannot answer the first, the other two have no meaning.
What has to be true before a system is allowed to believe itself: it states how many things it tried, it is judged against that number, it is tested on data where the answer is known, it is tested on data it has never touched, and it is allowed to be wrong in public. That is the bar I hold mine to.
If you run backtests, I would genuinely like to hear how you count your tries. Reply on X with your method, or tell me where my judge is too soft or too hard. Would it be okay if I asked you to send me the one backtest you trust least?
Sources
- Bailey and Lopez de Prado, The Deflated Sharpe Ratio (SSRN 2460551).
- Bailey, Borwein, Lopez de Prado and Zhu, The Probability of Backtest Overfitting (combinatorial cross-validation), cited in the paper above.
- Practitioner post, a blog and not peer reviewed: We ran 100 million bitcoin backtests, only 3,130 beat the luck bar.
Next issue: I built a product assistant that is not allowed to guess.
Previous issues: I freed 334 gigabytes and the disk got smaller, Why more AI agents don't automatically mean more output. All issues: Demonblade Labs.