# AutoResearch — How the strategy research system was built

*Jake Choi · Aug 4 – Sep 17, 2026*

The decisions behind a system that uses AI to write and test Bitcoin trading strategies, in the order they were made.

- **6 weeks** — Aug 4 – Sep 17
- **30** — design documents, written before building
- **14,224** — strategies tested — 2% kept

## The idea

An AI writes a trading strategy, a fixed test scores it on past prices, and the change is kept only if the score improves. Most of the work went into making that test trustworthy.

- **Simulation first** (Scope) — Exchange trading was deferred until there was a strategy worth trading.
- **Fixed AI budget** (Constraint) — The AI runs on an existing subscription, so cost is flat and the limit is usage per period.
- **AI for logic, search for numbers** (Division of work) — The AI changes how a strategy decides. An automated search sets the exact values.

## Six stages

### 1. Foundation (Aug 4 – 5)

- **Question:** Can AI-written strategies be tested safely and compared fairly?
- **Finding:** AI-written code cannot be trusted by default, and trading tests are easy to distort by using prices that would not have been known yet.
- **Decision:** Every strategy is checked before it runs and tested for use of future information.

### 2. How results are judged (Aug 6 – 12)

- **Question:** Is one test period enough to show a strategy works?
- **Finding:** No. The margin of error was larger than the result, and the winner had been picked from many attempts on the same data.
- **Decision:** Eight fixed test periods, scored on the middle result, with a discount for the number of strategies tried.

### 3. What counts as better (Aug 11 – 17)

- **Question:** What should the system optimize?
- **Finding:** Scores kept rising while losses grew, and the search simply picked the largest trade size allowed.
- **Decision:** The score is return relative to losses. Trade size is set separately, from a fixed risk limit.

### 4. Costs and automation (Aug 20 – 28)

- **Question:** Are the simulated costs realistic, and can research run without supervision?
- **Finding:** A review found missing cost data and a stress test that did not require a profit. Every run was started by hand.
- **Decision:** Corrected the cost model after measuring its effect on past results. Research now runs on a nightly schedule.

### 5. From research to an exchange account (Aug 13 – Sep 8)

- **Question:** How does a strategy reach an exchange account without the system deciding what trades?
- **Finding:** The trading program and the simulator disagreed in places, and practice accounts that switched strategies produced records that described none of them.
- **Decision:** The trading program must match the simulator exactly. A person chooses what trades. Each practice account follows one fixed strategy. Trading stops when a safety check fails, and a separate machine raises an alert.

### 6. One strategy or several (Sep 7 – 10)

- **Question:** Is the top-scoring strategy the right one to use?
- **Finding:** Among the top strategies, a higher score did not predict better results on new data. A combination did about 3.6 times better than the typical single strategy.
- **Decision:** Several strategies are combined. New versions must improve the combination, and versions that fall behind an earlier one are flagged for rollback.

## Dashboard design

- **Aug 6 — First dashboard:** Showed how the practice account was doing, and flagged when data stopped arriving.
- **Aug 6 — Plain-language explainer:** Explained how a strategy is chosen, for readers without a finance background. Reviewed as a mockup first.
- **By Aug 20 — Visual design in Claude Design:** Drafted the look and the account page layout in Claude Design. The dashboard's styling was copied from it.
- **Aug 25 — Published:** Put online behind a password.
- **Sep 15 – 16 — Redesign:** Reviewed as a mockup over 14 versions, then built. Plain-language labels moved onto the account page.

## Working principles

- **Write the design first** — Each major change started as a design document reviewed before building.
- **Review the design as a mockup** — Both dashboard designs were drafted as mockups and reviewed.
- **Measure before changing a rule** — Scoring and cost changes were tested on past results, against a control, before being applied.
- **A person makes the final call** — The system ranks strategies. A person decides what trades.

## The system as of Sep 17

Top row: an AI writes a strategy from price history; it passes safety checks, is tested on eight past periods, and is compared with the previous best. Most attempts are rejected and the reason is given back to the AI. Bottom row: kept strategies move through practice, trial and combined accounts before the exchange account. A person decides what the exchange account trades.

## What the operator sees

One page shows how the account is doing. The others show why a strategy was chosen. Every screen says at the top whether the numbers are simulated.

- **Account page.** Balance, profit to date, today's profit, the open position, and a 7-day line against simply holding Bitcoin. Sample figures.
- **Current strategy.** One sentence on what it does, next to the numbers it was chosen on.
- **Strategy families.** Thirteen families run side by side. Each row shows its results line, score and how many of the eight periods it passed.
- **Attempt log.** Every attempt, kept or rejected, with the AI's reasoning and the score it had to beat.

- **Built with:** Python · pandas · SQLite · FastAPI · Optuna
- **Size:** 153 modules · 122 test files
- **Product demo:** https://portfolio.jakecx.com