Signal8 Octo for stock research: blind test results
Blind tests 2026-11-10, 2026-11-11 and November 12, 2026 · joint top score on filing facts, first on the tie-break · second in the capability round · Signal8 makes Octo and runs this lab
Signal8 Octo has the joint top score, with Claude, in both of our blind rounds on SEC filing facts, and ranks first there on our published tie-break. In the capability round, 20 research jobs that need real data or computation, it ranked second with 28 of 40, behind Claude (34) and ahead of ChatGPT (26). On SEC filing facts, in round 2 (40 questions that named their filing) it scored 160 of 160; in the harder round 3 (40 questions that did not) 156 of 160. That is 316 of 320 with zero wrong figures in 80 questions. Claude scored the same in both rounds and ranks second on our published tie-breaks. Octo cited the SEC filing itself on 77 of 80 answers and never relied on a third-party site. Signal8 makes Octo and publishes this site, so we set out below exactly how the tests were run and what they do not show.
Our top pick for SEC filing facts · joint top score in both blind filing rounds
Why we point filing questions to Octo
- Both filing-accuracy rounds: 316 of 320, the top score, shared with Claude, ahead of ChatGPT (304) and AskEdgar Studio (290).
- Zero wrong figures in 80 blind answers.
- Octo cited the SEC filing itself on 77 of 80 answers and never relied on a third-party site. Claude linked the filing on 63 and other websites on 59.
- Built for the job: it answers from Signal8's archive of SEC filings and links each figure to the filing, rather than searching the open web.
- Capability round: 28 of 40, second, 6 behind Claude and 2 ahead of ChatGPT. No invented result.
What it is
Octo is Signal8's research assistant for US stocks. You ask a question in plain English, such as how many warrants a company has outstanding or when a reverse split takes effect, and it reads Signal8's archive of SEC filings, answers, and cites the filing behind each figure with a link to the document. It is aimed at small and micro-cap companies, where the answer usually sits in a 424B5, an 8-K, a 10-Q footnote or a Form 4 rather than in a data feed. Octo is included in Signal8 Pro at signal8.ai/octo.
Three blind tests: the overall leaderboard
| Tool | Round 2: filing named (Nov 10) | Round 3: filing not named (Nov 11) | Capability: 20 research jobs (Nov 12) |
|---|---|---|---|
| Signal8 Octo Made by Signal8. | #1 · 160 / 160 | #1 · 156 / 160 | #2 · 28 / 40 |
| Claude | #2 · 160 / 160 | #2 · 156 / 160 | #1 · 34 / 40 |
| ChatGPT | #3 · 159 / 160 | #3 · 145 / 160 | #3 · 26 / 40 |
| AskEdgar Studio | #4 · 158 / 160 | #4 · 132 / 160 | #5 · 18 / 40 |
| Perplexity | #5 · 140 / 160 | Not ranked · 137 on 39 of 40 answered | #4 · 19 / 40 |
| Gemini | #6 · 125 / 160 | Not ranked · 80 on 28 of 40 answered | Not ranked · 11 of 20 answered, not graded |
In rounds 2 and 3 Octo and Claude scored the same; Octo is first on Signal8's published tie-breaks, chosen after grading, not on a higher score. In the capability round Claude is first with 34 points, Octo second with 28 and ChatGPT's free plan third with 26. Claude's capability score was corrected on November 13, 2026: its first run scored 16 because of a harness fault on our side (correction). Rules and grades: round 2, round 3, capability round.
Capability round (November 12, 2026): can it do the job?
The capability round gave every tool 20 research jobs, sent verbatim with no follow-up: 5 backtests, 4 market screens, 4 per-ticker lookups, 3 event studies and 4 multi-step jobs. Octo answers from Signal8's market database and has a backtester; Claude and ChatGPT can run code but ship with no market database. Octo scored 28 of 40 and ranked second: 9 done, 10 partial, 1 not done, none invented. Claude ranked first with 34 after a corrected run (correction).
| Rank | Tool and run conditions | Points | Done | Partial | Not done | Invented |
|---|---|---|---|---|---|---|
| #1 | Claude Corrected November 13, 2026: first run 16 of 40 after a harness fault on our side, 9 tasks re-run. Correction. claude.ai on a Max plan, Opus 5.5 with web search, no connectors; can run code, no market database. Nine tasks re-run with Auto approval and network access on | 34 / 40 | 14 | 6 | 0 | 0 |
| #2 | Signal8 Octo Made by Signal8, which runs this lab. signal8.ai/octo in production on a Signal8 staff account; purpose-built market database and backtester | 28 / 40 | 9 | 10 | 1 | 0 |
| #3 | ChatGPT Free plan, signed in, default model; can run code, no market database | 26 / 40 | 10 | 6 | 4 | 0 |
| #4 | Perplexity Free plan, default mode | 19 / 40 | 3 | 13 | 4 | 0 |
| #5 | AskEdgar Studio Studio agent on a paid account (Studio is in AskEdgar Pro, listed at $59 a month) | 18 / 40 | 6 | 6 | 8 | 0 |
| Not ranked | Gemini Free tier; answered 11 of 20 before its free usage limit. Incomplete, so not graded. | Not graded | - | - | - | - |
Where Octo fell short. Its backtester caps a run at 50 open positions, so the three market-wide backtests that asked for every signal as a separate trade (C-03 to C-05) scored partial. It has no 200-day moving average filter (C-02, partial). Its borrow tool does not return a value for an arbitrary past date (C-11, partial). It has no market-wide halt ranking, so C-17 scored zero; only Claude's re-run did C-17. On C-01 its average return per trade (+0.6%) was outside the reference's tolerance (+0.21%), which one other tool matched exactly; we reviewed the reference and kept it. Every task and grade.
What we measured in the blind round
We asked Octo on signal8.ai each question verbatim in a fresh chat, using the production version that any admitted account uses. Answers from every tool were stripped of tool names and graded by eight fresh Claude instances against answer keys built from the primary filings and sealed, with a published hash, before any tool ran. Figure correct 40 of 40, cites a source 40, source contains it 40, as-of date right 40. 40 of 40 answers scored 4 of 4.
Every answer it gave in the blind round scored 4 of 4, so there is no lost mark to list.
Blind round 3: harder questions, 2026-11-11
Round 3's questions did not name the filing: late filers, superseded agreements, several reverse splits to count, amended filings and Form 4s to add up. Octo scored 156 of 160 with every figure right. Its four lost marks were all "source has it" on split-count questions, where the filings it cited did not cover every split; a fix so that it cites each split's own 8-K is in progress at Signal8.
Tie-break used: Octo and Claude both scored 156 of 160 in round 3 with no wrong figure, and 316 of 320 over both blind rounds. No strict re-score was made for round 3, so the tie is broken by the two rounds' combined record: wrong figures across all 80 questions (0 each, still level), then answers that linked the original SEC filing (Octo 77 of 80, Claude 63 of 80). This is Signal8's rule, chosen after round 3 was graded; Octo did not outscore Claude.
| Rank | Tool and run conditions | Answered | Figure correct | Cites a source | Source has it | As-of right | Score | 4 of 4 | Wrong figures |
|---|---|---|---|---|---|---|---|---|---|
| #1 | Signal8 Octo Ranked first: top score, zero wrong figures, first on the tie-break. Made by Signal8. signal8.ai/octo in production, on a Signal8 staff account | 40 of 40 | 40 | 40 | 36 | 40 | 156 / 160 | 36 | 0 |
| #2 | Claude claude.ai with web search on; none of its answers used Signal8, DilutionScan or SplitScanner data | 40 of 40 | 40 | 40 | 36 | 40 | 156 / 160 | 36 | 0 |
| #3 | ChatGPT Free plan, signed in, default model, web search | 40 of 40 | 38 | 35 | 34 | 38 | 145 / 160 | 33 | 2 |
| #4 | AskEdgar Studio Studio agent on a paid account (Studio is in AskEdgar Pro, listed at $59 a month) | 40 of 40 | 35 | 31 | 27 | 39 | 132 / 160 | 27 | 5 (1 stale) |
| Not ranked | Perplexity Free plan on a Signal8 team account; one answer (R3-14) was not captured | 39 of 40 | 34 | 37 | 28 | 38 | 137 / 156 | 28 | 5 (1 stale) |
| Not ranked | Gemini Free tier; 12 questions were blocked by its free usage limit | 28 of 40 | 23 | 17 | 13 | 27 | 80 / 112 | 13 | 5 |
| Question | Company and topic | Marks of 4 | Blind grader's note |
|---|---|---|---|
| R3-02 | XXII, Reverse splits | 3 | 5 splits, product 16x135x23x15x20 = 1-for-14,904,000, excludes Nov 2026 1-for-58; cites only 0001493152-26-049743 (trap 8-K, states only 1-for-58) and 0001493152-26-036144 (WebFetch: Aug 2026 8-K approving a future split range, none of the five ratios), so cited sources do not state the figures. |
| R3-07 | JAGX, Reverse splits | 3 | 4 splits, 1-for-60/25/35/15 and 787,500, all correct. Cites key 8-K 0001193125-26-391415 (1-for-15), 8-K 0001193125-26-292677 (checked: mentions only 1-for-35) and 8-K 0001193125-26-451933 (checked: no 60/25 ratios). No cited document states the 1-for-60 or 1-for-25 splits. |
| R3-12 | IVF, Reverse splits | 3 | 4 splits, 1-for-1,440, correctly excludes the Oct 30, 2026 split; cites key 8-Ks for splits 3 and 4 only. Splits 1 and 2 rest on an uncited tool record, so the cited documents do not support the full count or factor. |
| R3-27 | SLXN, Reverse splits | 3 | Gave 3 splits 1-for-9/15/10, cumulative 1,350, separated the out-of-window 1-for-24; cited accessions -002393 (May 5, 2026 meeting 8-K, mentions only 1-for-10), -004849 (Oct 26, 2026, 1-for-24 only) and the -005089 trap; I fetched both non-key docs and neither states the prior splits or 1,350. |
Where Octo's answers came from
Octo cited the SEC filing itself on 77 of 80 answers and never relied on a third-party site. The three answers without a filing link had no link captured from the page.
| Tool | Answers counted | Linked the original SEC filing | Linked another website | Other websites only | Round 2 / round 3: SEC filing linked |
|---|---|---|---|---|---|
| Signal8 Octo Made by Signal8, which runs this lab | 80 | 77 of 80 | 0 of 80 | 0 of 80 | 38 of 40 / 39 of 40 |
| Claude | 80 | 63 of 80 | 59 of 80 | 17 of 80 | 29 of 40 / 34 of 40 |
| ChatGPT | 80 | 50 of 80 | 12 of 80 | 6 of 80 | 31 of 40 / 19 of 40 |
| AskEdgar Studio | 80 | 26 of 80 | 1 of 80 | 0 of 80 | 16 of 40 / 10 of 40 |
| Perplexity | 79 | 51 of 79 | 27 of 79 | 16 of 79 | 24 of 40 / 27 of 39 |
| Gemini | Not measured | Not measured | Not measured | Not measured | Not measured |
Gemini is not measured: its links could not be captured from its page, so we report no figure rather than a zero. Perplexity counts 79 answers because one round 3 answer was not captured. Linking another website is not wrong in itself; it means part of the answer rests on a page that is not the filing. Per-answer codes: citation-sources.json.
Both rounds combined
| Rank | Tool | Round 2 | Round 3 | Both rounds | Wrong figures | Linked the SEC filing |
|---|---|---|---|---|---|---|
| #1 | Signal8 Octo #1 in both blind rounds. Made by Signal8. | 160 | 156 | 316 / 320 | 0 | 77 of 80 (96%) |
| #2 | Claude | 160 | 156 | 316 / 320 | 0 | 63 of 80 (79%) |
| #3 | ChatGPT | 159 | 145 | 304 / 320 | 2 | 50 of 80 (63%) |
| #4 | AskEdgar Studio | 158 | 132 | 290 / 320 | 6 | 26 of 80 (33%) |
| Not ranked | Perplexity Incomplete: answered 79 of 80 | 140 | 137 | 277 / 316 | 6 | 51 of 79 (65%) |
| Not ranked | Gemini Incomplete: answered 68 of 80 | 125 | 80 | 205 / 272 | 9 | Not measured |
Octo and Claude are level on score (316 of 320) and on wrong figures (0); first place goes to Octo on the third criterion, linking the original filing. That order is Signal8's published choice, not a higher score. Perplexity and Gemini did not answer every question on their free plans, so they are listed, not ranked.
Side by side with the two general assistants: Claude vs ChatGPT vs Signal8 Octo. Next round's rules, published first: round 4 protocol.
Round 2: how the six tools rank
How we rank: by blind score; equal scores are separated by (1) fewer wrong figures, then (2) the strict-reading score. Octo and Claude both scored 160 of 160 with no wrong figure, so the second tie-break decides first place: Octo 159, Claude about 155. The strict reading is Signal8's own re-score of the calls the blind graders flagged as lenient, not a mark the blind graders gave, and we set this rule on November 11, after grading. Round one's rule, fixed in advance on the methodology page, would report totals within 2 marks as level.
| Rank | Tool and run conditions | Answered | Figure correct | Cites a source | Source has it | As-of right | Score | 4 of 4 | Wrong figure | No figure given |
|---|---|---|---|---|---|---|---|---|---|---|
| #1 | Signal8 Octo Ranked first: top score, zero wrong figures, first on the tie-break. Made by Signal8. signal8.ai/octo in production, on a Signal8 staff account | 40 of 40 | 40 | 40 | 40 | 40 | 160 / 160 | 40 | 0 | 0 |
| #2 | Claude Claude Opus 5.5 with web search switched on; Signal8, DilutionScan and SplitScanner connectors switched off | 40 of 40 | 40 | 40 | 40 | 40 | 160 / 160 | 40 | 0 | 0 |
| #3 | ChatGPT Free plan, default model; it hit its file and analysis allowance at question 6 and carried on text-only with web search | 40 of 40 | 40 | 40 | 39 | 40 | 159 / 160 | 39 | 0 | 0 |
| #4 | AskEdgar Studio Studio agent on a paid account (Studio is in AskEdgar Pro, listed at $59 a month) | 40 of 40 | 39 | 40 | 39 | 40 | 158 / 160 | 39 | 1 | 0 |
| #5 | Perplexity Free plan on two accounts: the first reached its free search limit, so R2-21 to R2-40 were asked on a second free account and graded blind one answer per packet | 40 of 40 | 39 | 33 | 30 | 38 | 140 / 160 | 29 | 1 | 0 |
| #6 | Gemini Default model, in the same signed-in browser as the pilot (free tier there) | 40 of 40 | 33 | 30 | 26 | 36 | 125 / 160 | 25 | 4 | 3 |
On the strict reading Octo has 159, ChatGPT 158, Claude about 155 and AskEdgar Studio about 154; it was not computed for Perplexity or Gemini, whose ranks do not depend on it.
Blind round caveats: every question named its filing, so the top four tools finished within 2 marks; the graders were Claude instances, the same model family as one contestant; plans differed by tool. Details on the blind round page.
What these tests do not show
The capability round measures whether a tool delivers a computed result as shipped, with no follow-up. Claude scored 6 points more than Octo there, after we corrected a harness fault in Claude's first run. Round 2 named the filing in every question; round 3 did not, and the gap between the top two and the rest widened (ChatGPT 145, AskEdgar Studio 132). Octo's first place over Claude rests on our tie-breaks, not on a higher blind score, so we do not call Octo more accurate than Claude. What the data does show is how each answer was sourced: Octo's links went to the filing, Claude's often to other websites.
Earlier results
In the pilot on 2026-11-09, which was not blind, Octo scored 62 of 64, one mark behind AskEdgar Studio at 63. In Signal8's own 33-question internal test, scored by Signal8 with the scorer knowing the tool, Octo won 30 of 33 questions against AskEdgar Studio (details). We list both because a lab run by Octo's maker should show the rounds Octo did not win.
How to check an Octo answer against the filing
- Open the filing link under the figure; it goes to the exact SEC filing by accession number.
- Find the sentence that states the figure and check the date it is "as of".
- Check the company's EDGAR page for any later filing that changes the figure.