Owned by Signal8 Inc., maker of Octo and Signal8 Pro. Not investment advice. About this site
stockairesearch / lab

Round 4 protocol, pre-registered November 12, 2026

By , Signal8 Research. Published by Signal8 Inc.

PRE-REGISTERED November 12, 2026 · NOT YET RUN · rules below are fixed before any question is drawn

Round 4 of our test of AI tools on SEC filing questions will use 40 new questions, sealed answer keys and blind graders, and score five marks per answer instead of four. The new fifth mark asks whether the answer cites the original filing rather than a news or aggregator site. The rules and the ranking order are published here, today, before any question is written or any tool is asked.

Five marks per answer

  1. Figure correct. Every figure the question asks for matches the key. A figure from a superseded or out-of-window filing fails and is tagged stale.
  2. Cites a source. A URL to a document, an accession number, or a form type with a filing date or period.
  3. Source contains it. The cited document states the figure.
  4. As-of date right. The answer states the date the figure is measured at, and it matches the key.
  5. Cites the original filing (new). The answer links an SEC EDGAR document, or a link that resolves to that EDGAR filing by its accession number, and that filing states the figure. Citing only a news site, a data aggregator or a company web page earns 0 on this mark. Linking such a site in addition to the filing does not cost the mark.

Mark 5 is applied to every tool in the same way, from the links on each tool's page. A tool whose links cannot be captured is graded on the text it shows: an accession or EDGAR URL written out in full counts; a site name does not.

Questions and keys

Asking and grading

Ranking rule, fixed in advance

  1. Only tools that answer all 40 questions are ranked. Others are listed with their score and the number answered.
  2. Rank by total of the five marks, out of 200.
  3. Equal totals are separated by fewer wrong figures (a declined answer is not a wrong figure).
  4. Tools still level on both are reported as tied, sharing the rank. No later tie-break will be added.

What this does and does not change

The fifth mark applies only from round 4. It is not retroactive: the pilot and blind rounds 2 and 3 keep their four-mark scores and the tie-breaks published with them. In rounds 2 and 3 Octo linked the original filing on 77 of 80 answers (measured here), so this mark rewards a behaviour Octo already shows. Signal8 makes Octo and runs this lab, and chose this criterion; we state that so readers can weigh it. We think it is the right test for filing research: a figure you can trace to the filing is one you can check.

If any rule on this page has to change, the change, its date and the reason will be listed here, and round 4 will be reported under both versions. The earlier results: blind round 2, blind round 3.