Owned by Signal8 Inc., maker of Octo and Signal8 Pro. Not investment advice. About this site
stockairesearch / lab

AI stock research benchmark, round one

STATUS: NOT YET RUN · planned for December 2026 · a blind round ran 2026-11-10 and a pilot 2026-11-09

Round one tests seven AI tools on 40 questions about US-listed small caps, each answerable from the company's own SEC filings. This page fixes the rules before the run. If the rules change, the change and its date will be listed here.

Procedure

  1. Draw. On the run date, draw companies at random from US-listed companies that filed a 10-Q or 10-K in the prior 120 days, and fill each question template with one company. No company is reused.
  2. Ground truth. Two people answer each question from EDGAR, independently, recording the filing and passage. Disagreements are settled against the filing text.
  3. Ask. Each tool gets each question once, in a fresh session, with the same wording and no follow-up. Paid tiers are used where a tool sells one, and the plan is recorded.
  4. Grade blind. Answers are stripped of tool names and formatting and graded by a person who does not know which tool wrote them.
  5. Publish. Every question, every answer, every grade and the filing behind each correct answer, as a table and a CSV.

Grades

One grade per answer
GradeMeaning
Correct and citedRight figure, right date and share basis, and it points to the filing
Correct, no citationRight figure with no usable source
StaleMatches an older filing that a later one superseded
DeclinedSays it cannot find or does not know the answer
WrongA figure or fact the filings do not support

A "Declined" answer scores better than "Wrong": a tool that admits it does not know is safer than one that invents a share count.

Questions by category

40 questions in 8 categories
CategoryQuestionsWhere the answer lives
Share count5Cover page of the latest 10-Q, 10-K or 20-F; later 8-Ks and 424Bs
Warrants6Equity footnote of the latest 10-Q or 10-K; offering 8-Ks and 424B prospectuses
ATM programs and shelves6S-3 and F-3 registration statements, EFFECT notices, 424B5 supplements
Convertible notes and preferred58-K Item 1.01 and 2.03 exhibits; securities purchase agreements; certificates of designation
Cash and runway5Balance sheet and liquidity section of the latest 10-Q or 10-K; going-concern language
Reverse splits48-K Item 5.03; DEF 14A proxy; exchange notices
Insiders and owners4Forms 4, 13D and 13G; proxy beneficial ownership table
Listing compliance58-K Item 3.01; exchange notices

Tools

Tools in the benchmark: blind round (2026-11-10) and pilot (2026-11-09); full round one not yet run
ToolMade byBlind roundPilot
OctoSignal8 (publisher of this site)#1: 160 of 160 on 40 answers62 of 64 on 16 answers
ClaudeAnthropic#2: 160 of 160 on 40 answersNot tested
ChatGPTOpenAI#3: 159 of 160 on 40 answers28 of 28 on 7 answers
AskEdgarAskEdgar#4: 158 of 160 on 40 answers63 of 64 on 16 answers
PerplexityPerplexity AI#5: 140 of 160 on 40 answers36 of 36 on 9 answers
GeminiGoogle#6: 125 of 160 on 40 answers54 of 64 on 16 answers
Fiscal.aiFiscal.aiNot testedNot tested

Signal8, which runs this lab, makes Octo. Its answers go through the same blind grading and will be published in full, wins and losses alike.