Methodology
The full procedure for round one is on the benchmark page. This page sets out the principles behind it.
Principles
- Answers must be checkable. Every question has one answer that a person can find in an SEC filing.
- Same conditions for every tool. Same wording, fresh session, no follow-up, no custom instructions, the tool's normal web or data access on.
- Random draw on the day. Companies are drawn when the round is run, so no tool, ours included, can be prepared for them.
- Blind grading. The grader does not know which tool wrote an answer.
- Everything published. Questions, answers, grades and the filing behind each correct answer.
What is not tested
Opinions, forecasts, stock picks and anything that is not written in a filing. Speed is recorded but not graded. Price is recorded on the day.
Pilot round rubric (2026-11-09)
Written before any answer was read. Each answer gets four yes or no marks:
- Figure correct. The key figure matches the filing exactly, or with correct rounding. Yes or no questions need the yes or no and the named detail. A wrong, stale or invented figure, or no figure at all, is a no.
- Cites a source. A URL, or a form type with a filing date or accession number. "SEC filings" with no document is a no.
- Source contains it. The cited document states the figure the answer gives, and that figure is correct. Checked by opening the cited URL or the named filing. If mark 1 or 2 is no, this is no. While grading the first answer with no figure, we wrote down that this mark needs a correct figure; it was then applied to every answer.
- As-of date right. The answer states the date the figure is measured at, and it matches the filing.
Run conditions: one fresh chat per question, the question text verbatim, no follow-up, the account and default model already set in the browser, no settings changed. The pilot departs from the round one protocol in four ways, listed on the results page.
Round one: scoring rubric and selection rule, fixed Nov 9, 2026
These rules are published before round one is run and are not changed after any answer is seen. If a rule must change, the change, its date and the reason are listed here, and the round is reported under both versions.
- Rubric. The four yes or no marks above, unchanged: figure correct, cites a source, source contains it, as-of date right. Maximum 4 per answer.
- Ranking criteria, in order. (1) Answered every question. (2) Total marks. (3) Fewest wrong figures (a declined answer is not a wrong figure and is listed separately). (4) Most questions answered. Totals within 2 marks of each other per 64 are reported as level, not ranked apart.
- Question selection. The category templates on the questions page are fixed now. Companies are drawn at random on the run date, and the full draw list is published before any tool is asked. A question whose answer cannot be found in the filing by two readers is dropped before any tool is run. No question or company is replaced after any tool's answer is seen.
- Grader. Grading is done by a reviewer assigned before the run who did not run the tools. The grader sees answers with tool names and formatting removed and is blind to which tool wrote each one. The reviewer's name is published with the results. Disagreements are settled against the filing text, not by Signal8's preference.
- Disclosure. Signal8 makes Octo and runs this lab. Octo is graded under the same rules, and every answer, grade and correction is published.
Status
A pilot round ran on 2026-11-09 and a blind round on 2026-11-10: 40 questions naming their filing, keys sealed first (SHA-256 published), graded by eight fresh Claude instances that saw only anonymised answers and the four-mark rubric above. Perplexity's free plan reached its search limit after 20 questions, so its other 20 were asked on a second free account and graded by four more fresh Claude instances, one answer per packet. Round one has not been run.
Blind round ranking rule (set November 11, 2026, after grading)
Tools are ranked by blind score. Equal scores are separated by (1) fewer wrong figures, then (2) the strict-reading score, which is Signal8's own re-score of every call the blind graders flagged as lenient, not a blind grade. Under this rule Octo ranks #1: Octo and Claude both scored 160 of 160 with no wrong figure, and Octo's strict reading is 159 against about 155 for Claude. We set this rule after the grades were known, so it is a disclosed publisher choice, not part of the pre-registered round one rules above, which would report totals within 2 marks as level.
Blind round 3 ranking rule (set November 12, 2026, after grading)
Round 3 (2026-11-11, 40 questions that do not name the filing) ranks only the tools that answered every question, by blind score, then fewer wrong figures. Octo and Claude tied on both (156 of 160, no wrong figure), and no strict re-score was made for round 3, so the tie is broken by the two blind rounds' combined record: wrong figures across all 80 questions (0 each), then answers that linked the original SEC filing (Octo 77 of 80, Claude 63 of 80). The same order ranks the combined two-round leaderboard. Signal8 chose this tie-break after grading; it is a disclosed publisher choice. Perplexity and Gemini were graded in a second blind pass and, with incomplete runs, are listed but not ranked. Details on the round 3 page.
Capability round (November 12, 2026): research jobs, scored done or not
A separate test of whether a tool can do a research job that needs real data or computation: 20 tasks (backtests, market screens, per-ticker lookups, event studies, multi-step research), each sent verbatim to a fresh chat with default settings, no uploaded data, no connectors added and no follow-up. A grading guide with sanity checks, reference values from public sources and tolerances was sealed before any tool ran. Five fresh Claude instances graded the anonymised answers: done 2, partial 1, not done 0, invented -1, so up to 40. Tools are ranked by points; no two tied. Gemini answered 11 of 20 on its free tier and is listed, not ranked. Claude's nine zero-point tasks in the first run were stalls caused by an approval setting and network access left off on our account; they were re-run on the same account and graded blind, and the result was corrected on November 13, 2026. Details on the capability round page.
Round 4: five marks, rules fixed in advance
The round 4 protocol, pre-registered November 12, 2026, adds a fifth mark for citing the original filing and fixes the ranking rule (total of five marks, then fewer wrong figures) before any question is drawn. It is not applied to earlier rounds.
Every number about tool accuracy on this site links the run it came from.