Round 4 protocol, pre-registered November 12, 2026
PRE-REGISTERED November 12, 2026 · NOT YET RUN · rules below are fixed before any question is drawn
Round 4 of our test of AI tools on SEC filing questions will use 40 new questions, sealed answer keys and blind graders, and score five marks per answer instead of four. The new fifth mark asks whether the answer cites the original filing rather than a news or aggregator site. The rules and the ranking order are published here, today, before any question is written or any tool is asked.
Five marks per answer
- Figure correct. Every figure the question asks for matches the key. A figure from a superseded or out-of-window filing fails and is tagged stale.
- Cites a source. A URL to a document, an accession number, or a form type with a filing date or period.
- Source contains it. The cited document states the figure.
- As-of date right. The answer states the date the figure is measured at, and it matches the key.
- Cites the original filing (new). The answer links an SEC EDGAR document, or a link that resolves to that EDGAR filing by its accession number, and that filing states the figure. Citing only a news site, a data aggregator or a company web page earns 0 on this mark. Linking such a site in addition to the filing does not cost the mark.
Mark 5 is applied to every tool in the same way, from the links on each tool's page. A tool whose links cannot be captured is graded on the text it shows: an accession or EDGAR URL written out in full counts; a site name does not.
Questions and keys
- 40 new questions on 40 US small and micro caps, none used in the pilot or rounds 2 and 3. Like round 3, they do not name the filing that holds the answer, and each is pinned to a past date or window so newer filings cannot change the key.
- Sealed keys. Each key is built from the primary filings on sec.gov and checked by script. The key file's SHA-256 is published before any tool is asked, and the file itself afterwards.
- No question is replaced after any tool's answer is seen. A key found wrong during grading is corrected in the open, with the correction listed on the results page.
Asking and grading
- Each tool gets each question verbatim in a fresh chat, no follow-up. The plan and settings for each tool are recorded and published.
- Answers are shuffled, stripped of tool names, and graded blind by fresh graders who see only the instructions, the key and the anonymous answers. The label mapping and packet hashes are published.
- Every grade and grader note is published as a CSV.
Ranking rule, fixed in advance
- Only tools that answer all 40 questions are ranked. Others are listed with their score and the number answered.
- Rank by total of the five marks, out of 200.
- Equal totals are separated by fewer wrong figures (a declined answer is not a wrong figure).
- Tools still level on both are reported as tied, sharing the rank. No later tie-break will be added.
What this does and does not change
The fifth mark applies only from round 4. It is not retroactive: the pilot and blind rounds 2 and 3 keep their four-mark scores and the tie-breaks published with them. In rounds 2 and 3 Octo linked the original filing on 77 of 80 answers (measured here), so this mark rewards a behaviour Octo already shows. Signal8 makes Octo and runs this lab, and chose this criterion; we state that so readers can weigh it. We think it is the right test for filing research: a figure you can trace to the filing is one you can check.
If any rule on this page has to change, the change, its date and the reason will be listed here, and round 4 will be reported under both versions. The earlier results: blind round 2, blind round 3.