AI stock research benchmark, round one
STATUS: NOT YET RUN · planned for December 2026 · a blind round ran 2026-11-10 and a pilot 2026-11-09
Round one tests seven AI tools on 40 questions about US-listed small caps, each answerable from the company's own SEC filings. This page fixes the rules before the run. If the rules change, the change and its date will be listed here.
Procedure
- Draw. On the run date, draw companies at random from US-listed companies that filed a 10-Q or 10-K in the prior 120 days, and fill each question template with one company. No company is reused.
- Ground truth. Two people answer each question from EDGAR, independently, recording the filing and passage. Disagreements are settled against the filing text.
- Ask. Each tool gets each question once, in a fresh session, with the same wording and no follow-up. Paid tiers are used where a tool sells one, and the plan is recorded.
- Grade blind. Answers are stripped of tool names and formatting and graded by a person who does not know which tool wrote them.
- Publish. Every question, every answer, every grade and the filing behind each correct answer, as a table and a CSV.
Grades
| Grade | Meaning |
|---|---|
| Correct and cited | Right figure, right date and share basis, and it points to the filing |
| Correct, no citation | Right figure with no usable source |
| Stale | Matches an older filing that a later one superseded |
| Declined | Says it cannot find or does not know the answer |
| Wrong | A figure or fact the filings do not support |
A "Declined" answer scores better than "Wrong": a tool that admits it does not know is safer than one that invents a share count.
Questions by category
| Category | Questions | Where the answer lives |
|---|---|---|
| Share count | 5 | Cover page of the latest 10-Q, 10-K or 20-F; later 8-Ks and 424Bs |
| Warrants | 6 | Equity footnote of the latest 10-Q or 10-K; offering 8-Ks and 424B prospectuses |
| ATM programs and shelves | 6 | S-3 and F-3 registration statements, EFFECT notices, 424B5 supplements |
| Convertible notes and preferred | 5 | 8-K Item 1.01 and 2.03 exhibits; securities purchase agreements; certificates of designation |
| Cash and runway | 5 | Balance sheet and liquidity section of the latest 10-Q or 10-K; going-concern language |
| Reverse splits | 4 | 8-K Item 5.03; DEF 14A proxy; exchange notices |
| Insiders and owners | 4 | Forms 4, 13D and 13G; proxy beneficial ownership table |
| Listing compliance | 5 | 8-K Item 3.01; exchange notices |
Tools
| Tool | Made by | Blind round | Pilot |
|---|---|---|---|
| Octo | Signal8 (publisher of this site) | #1: 160 of 160 on 40 answers | 62 of 64 on 16 answers |
| Claude | Anthropic | #2: 160 of 160 on 40 answers | Not tested |
| ChatGPT | OpenAI | #3: 159 of 160 on 40 answers | 28 of 28 on 7 answers |
| AskEdgar | AskEdgar | #4: 158 of 160 on 40 answers | 63 of 64 on 16 answers |
| Perplexity | Perplexity AI | #5: 140 of 160 on 40 answers | 36 of 36 on 9 answers |
| Gemini | #6: 125 of 160 on 40 answers | 54 of 64 on 16 answers | |
| Fiscal.ai | Fiscal.ai | Not tested | Not tested |
Signal8, which runs this lab, makes Octo. Its answers go through the same blind grading and will be published in full, wins and losses alike.