Signal8 Octo vs AskEdgar Studio: Octo ranked #1 at 160 of 160, AskEdgar Studio #4 at 158 in a blind test
Blind round 2026-11-10 · pilot Nov 9 · Signal8 internal benchmark Nov 2 to 4
In our blind round of 40 SEC filing questions, Signal8's Octo ranked #1 of 6 with 160 of 160 and no wrong figure; AskEdgar Studio ranked #4 with 158, behind ChatGPT's free plan (159), and was the only one of the top four to give a wrong figure. Signal8 runs this lab and makes Octo.
Our top pick · ranked #1
Octo, Signal8's AI research assistant
Why Octo ranks #1: the top blind score, 160 of 160; zero wrong figures; first on the published tie-break; and it cited the SEC filing for every answer, with the grader confirming the filing states the figure (40 of 40).
- Blind round, 2026-11-10: Octo 160 of 160, all 40 answers scored 4 of 4, no wrong figure. AskEdgar Studio 158, 39 of 40 perfect, 1 wrong figure. Graded by AI instances that saw no tool names.
- Claude also scored 160 with no wrong figure. Octo ranks first on our published tie-break, a strict reading that is Signal8's own re-score of flagged calls, not the blind graders'. The rule.
- Signal8's internal test: Octo won 30 of 33 questions against AskEdgar Studio (Nov 2 to 4), scored by Signal8 with the scorer knowing the tool.
Where AskEdgar Studio fell short in the blind round
- The only wrong figure among the top four. On R2-39 (LTRN, a Form 4 question) it named one of the two buying insiders, so its share total was wrong; the grader also noted it cited the filing under the wrong CIK. Octo, Claude and ChatGPT gave no wrong figure.
- A paid product that finished behind a free plan. Studio is part of AskEdgar Pro, listed at $59 a month (askedgar.io/pricing, checked Nov 9, 2026). It scored 158, behind ChatGPT's free plan at 159 and Octo at 160.
- Last of the four tools that scored 158 or more. It ranks #4 of 6, ahead of Perplexity (140) and Gemini (125).
AskEdgar Studio in blind round 3
- Fourth of the 4 tools that answered every question, with 132 of 160: 24 marks behind Octo (156) and 13 behind ChatGPT's free plan (145). Round 3 ranking.
- 5 wrong figures, the most of the four, 1 of them stale: R3-12, R3-29, R3-35, R3-39 and R3-40. Octo and Claude gave none.
- Fewer marks than a free plan that skipped a question. Perplexity's free plan, which answered 39 of 40, scored 137.
- It linked the original SEC filing on 26 of 80 answers across rounds 2 and 3, against 77 for Octo.
Both blind rounds together: Octo 316 of 320 with no wrong figure, AskEdgar Studio 290 with 6. Round 3, every question and grade.
Three tests, newest first
| Test | Date | Questions | Graded by | Octo | AskEdgar Studio |
|---|---|---|---|---|---|
| Blind round | 2026-11-10 | 40 | Eight AI graders (Claude instances), no tool names shown, keys sealed first | 160 of 160 (#1 of 6) | 158 of 160 (#4 of 6) |
| Pilot | Nov 9, 2026 | 16 | One Signal8 grader who knew the tools | 62 of 64 | 63 of 64 |
| Internal benchmark (below) | Nov 2 to 4, 2026 | 33 | Signal8, scorer knew the tool, scorecard out of 30 | Mean 28.4 | Mean 23.5 |
The blind round's caveats apply here too. Every question named its filing, so the test rewards finding and citing a named document, and the top four tools were within 2 marks. The graders were Claude instances. AskEdgar Studio ran on a paid account and Octo on Signal8's production service with a staff account. Every grade is in the blind round CSV.
History, and where AskEdgar Studio did better: it led the earlier, non-blind pilot by one mark (63 to 62), where Octo declined to total one company's warrants, and in the internal test below its first-run answers were faster and easier to skim.
Signal8's internal benchmark, 33 questions
In Signal8's internal benchmark of 33 SEC filing questions, Octo scored a mean of 28.4 out of 30 on production data on November 4, 2026, against 23.5 for AskEdgar Studio, winning 30 of 33 questions, losing 1 and tying 2 on the six-axis scorecard. Signal8 ran the benchmark and makes Octo, so read the limits below before the numbers.
Source: Signal8's three benchmark reports of November 2, 3 and 4, 2026, which are kept in Signal8's research-chat documentation. Figures on this page were recomputed from the per-question totals; the per-question table and a CSV are below. Checked Nov 9, 2026.
What was compared
- Octo, Signal8's AI research assistant (signal8.ai/octo), signed in on a Signal8 staff account, default effort. Run on Signal8's development environment on November 2 and 3 and on production data on November 4.
- AskEdgar Studio (studio.askedgar.io), Pro plan, model "Edgar" (
claude-haiku-5-5), effort Medium, run on November 2 only. It was not re-run, so every comparison with Octo's later runs is against Studio answers up to two days older. - The same 33 questions, each in a fresh chat, scored on two rubrics out of 30: a six-axis scorecard (accuracy, completeness, sourcing, latency, formatting, safety) and a response rubric (semantics, readability, structure, noise, tone, citations), 1 to 5 on each axis.
Results by run
| Run | Date | Scorecard | Response rubric | Note |
|---|---|---|---|---|
| AskEdgar Studio | Nov 2 | 23.5 | 24.2 | Pro plan; not re-run. Zero broken answers; median 17.5 s to completion. |
| Octo, first run | Nov 2 (dev) | 24.1 | 22.7 | Run beside Studio the same afternoon. Three answers were broken, median 36.9 s. Studio led on the response rubric (24.2 against 22.7) and on speed. |
| Octo, re-run | Nov 2 (dev) | 26.2 | 25.8 | After writing, speed and tool changes. Median 24.5 s. |
| Octo, re-run | Nov 3 (dev) | 27.7 | 27.7 | Median 18.8 s. |
| Octo, production | Nov 4 (prod data) | 28.4 | 28.3 | Median 17.3 s, no answer over 60 s, 33 of 33 answers delivered. |
The 28.4 against 23.5 comparison is the November 4 Octo run against the November 2 Studio answers. Against those, Octo won 30 questions on the scorecard (1 loss, 2 ties) and 32 on the response rubric (0 losses, 1 tie). The same-afternoon first run on November 2 is the only like-for-like timing, and in it AskEdgar Studio was faster and tidier.
Where AskEdgar Studio did better
- It was about twice as fast in the first run: median 17.5 s to completion against 36.9 s, and 7 of Octo's 33 answers took over 60 s against 1 of Studio's.
- It delivered every answer. Octo's first run had three broken answers (questions Q12, Q30 and Q33).
- Its answers were easier to read in the first run: it led on the response rubric, 24.2 against 22.7. Its house style (a bold one-line verdict, a table, a few bullets) was judged easier to skim.
- On the November 4 production run it still scored at or above Octo on 3 questions: Q03, Q07, Q32.
- It answered a question Octo declined to answer with data (Q30, a request for investment advice): both declined to advise, and Studio still laid out cash, debt and dilution.
Errors found
Octo's, as reported on November 4 (production). The report lists five factual errors in Octo's answers, two of them from one data defect, plus a data gap and a display bug:
- Q10: said VNRX's at-the-market (ATM) program was terminated; the active 2025 agreement was the live one and only the 2022 agreement had ended.
- Q10: gave a $2.4M convertible note as issued in October; it was issued January 15, 2026, with $706,667 outstanding after an October prepayment.
- Q30: described a "$2.4M loan" convertible into about 3.03M shares at $1.55; the balance was $706,667 and the 3.03M share figure was at a different price (a defect in how the tool paired share counts with prices, which also caused the Q10 errors).
- Q14: gave INTZ an estimated November 9 Nasdaq deadline; INTZ's own 8-K stated November 3.
- Q03: understated IPW's September 2026 note by using a stale price, and said an S-1/A did not state a floor when it did.
- Q32: could not name the company for PGPHF because production had no company record for it (a data gap, where development and Studio had named Partners Group).
- Q10 (display): a months-of-cash value rendered as "7.29-for-1".
AskEdgar Studio's, as reported on November 2. Four wrong figures: IPW shares outstanding (532,917 against 596,808), SINT ATM remaining ($5.62M against $5.1M), "62 halts and none currently active" when four stocks were halted, and an MDCX equity line shown as live after it ended on September 11. Two omissions: a $12.0M upfront payment missing from an MDCX runway estimate, and ONCO's net proceeds of about $0.25M left as "not stated". Octo's first run had two errors of its own of that kind: the same ended MDCX equity line, and development-environment reverse-split rows that disagreed with production.
Method, as documented
- Driving. Each product was called through its own browser API from a signed-in Chrome session (Octo through its chat stream, Studio through its conversation and agent-run events), so tool calls and timings were recorded exactly. One new chat per question, strictly in sequence on the production run.
- Ground truth. Signal8's market-data tools on production data (dilution instruments, cash, compliance, halts, reverse splits, insider and institutional data) and, for primary-source checks, the SEC filing text on EDGAR (for example the IPW S-1/A 0001683168-26-008203, the GNS 20-F 0001493152-26-009371 and the SINT 10-Q 0001493152-26-037116).
- Scoring. The six-axis scorecard and the response rubric, 1 to 5 per axis, as defined in the November 2 report. Latency bands for the November 4 run: up to 18 s scores 5, 26 s scores 4, 45 s scores 3, 60 s scores 2, over 60 s scores 1.
Limits of this benchmark
- Signal8 ran it, scored it and makes Octo. Scoring was done by one party who knew which tool wrote each answer, not blind.
- Ground truth came largely from Signal8's own data, which may favour Octo on questions where Signal8's data and AskEdgar's differ.
- Studio was run once, on November 2. Octo was run four times and improved between runs, partly in response to this benchmark. The November 4 comparison is against older Studio answers; AskEdgar may have changed since.
- Time-sensitive questions (splits, deadlines, halts, notices) were scored for Studio on November 2 and not re-scored against November 4 facts.
- It is 33 questions, so small differences between runs are noise.
The lab's blind round and pilot use the lab's own four-mark rubric on different question sets; the full protocol is on the round one page.
Every question
| Q | Category | Question | Octo scorecard | Studio scorecard | Octo response | Studio response | Octo seconds |
|---|---|---|---|---|---|---|---|
| Q01 | dilution | What is the dilution risk for MDCX? Cover warrants, convertibles, ATM and shelf capacity, with figures, as-of dates and the filings they come from. | 28 | 24 | 29 | 26 | 22.9 |
| Q02 | dilution | How much capacity does VNRX have left on its ATM and its shelf, is it limited by the baby shelf rule, and how much has it sold through the ATM this year? | 29 | 22 | 28 | 22 | 22.7 |
| Q03 | dilution | List every outstanding warrant and convertible note for IPW on the current post-split basis: exercise or conversion price, shares issuable, expiry or maturity, and any floor or reset terms. | 25 | 25 | 26 | 26 | 44.5 |
| Q04 | dilution | Does KALA have an active ATM or equity line? How much has it sold under it this year and what is the remaining capacity? | 29 | 24 | 28 | 25 | 17.7 |
| Q05 | judgment | Is ONCO's most recent financing toxic? Explain the terms that make it so or not. | 28 | 22 | 28 | 25 | 32.0 |
| Q06 | will they dilute | Will SINT need to raise money in the next 6 months? Base it on cash, burn and what financing vehicles are already in place. | 29 | 24 | 28 | 26 | 22.2 |
| Q07 | overhang | What registration statement overhang does AIIO have: pending S-1, F-1 or resale registrations, how many shares, and for whom? | 27 | 28 | 29 | 27 | 28.0 |
| Q08 | cash | How many months of cash runway does VNRX have, as of what date, and what events since the quarter end (ATM sales, raises, payments) change that number? | 28 | 24 | 27 | 24 | 14.5 |
| Q09 | going concern | Does GNS have a going concern warning in its latest annual report? Quote the exact language and give the filing. | 30 | 27 | 30 | 24 | 17.3 |
| Q10 | cash compare | Compare cash runway for VNRX and INO: cash, quarterly burn, months left and any ATM or equity line capacity. | 27 | 26 | 28 | 26 | 21.5 |
| Q11 | reverse splits | Which reverse splits take effect in the next two weeks? Give ticker, ratio, effective date, market cap and float. | 26 | 22 | 27 | 23 | 30.2 |
| Q12 | reverse splits | What is TOPS's reverse split history, and how did the stock perform in the 30 days after each split? | 30 | 24 | 29 | 26 | 14.6 |
| Q13 | reverse splits (trap: JOB is NYSE American) | Will JOB's upcoming 1-for-30 reverse split be enough to regain Nasdaq $1 bid price compliance, and what is its deadline? | 30 | 24 | 28 | 25 | 16.7 |
| Q14 | compliance screen | Which Nasdaq companies are under the $1 minimum bid price rule with the least time left before delisting? | 28 | 22 | 29 | 24 | 11.9 |
| Q15 | delisting | Is VNRX at risk of NYSE American delisting? Cite the deficiency notices, the deadlines, and where the stock stands against each listing standard today. | 30 | 25 | 29 | 27 | 17.3 |
| Q16 | halts | Which stocks were halted today and on what halt code? | 30 | 14 | 30 | 18 | 13.1 |
| Q17 | halts | Why is SUPN halted, what is the halt code, and since when? | 30 | 18 | 30 | 21 | 13.7 |
| Q18 | deficiency | Which companies received exchange deficiency notices this week? Give the exchange, the rule, and the filing. | 30 | 26 | 29 | 28 | 14.4 |
| Q19 | insiders | Find insider cluster buys in small caps from the last 30 days where 3 or more insiders bought. Give the prices paid, total dollars and links to the Form 4s. | 29 | 23 | 29 | 25 | 14.6 |
| Q20 | insiders (trap: none in window) | Summarize ORGO's Form 4 activity over the last 90 days: who bought or sold, how many shares, at what price, and open-market versus plan or award. | 28 | 24 | 27 | 22 | 18.2 |
| Q21 | institutions | Which institutions added or cut the most INO shares last quarter according to 13F filings? | 28 | 17 | 28 | 19 | 20.9 |
| Q22 | institutions | Who are the top 13F holders of MDCX, and did they add or reduce in the latest quarter? | 29 | 20 | 28 | 20 | 13.4 |
| Q23 | float/short | What is MDCX's float, short interest and days to cover, and when was each figure last reported? | 30 | 24 | 29 | 25 | 12.4 |
| Q24 | short/borrow | What is the short and borrow situation on BYND: short interest, days to cover, and borrow availability if you have it? | 30 | 25 | 29 | 24 | 17.2 |
| Q25 | 8-K | Summarize MDCX's latest 8-K: filing date, the items reported, what happened, and a link to the filing. | 30 | 28 | 30 | 27 | 13.2 |
| Q26 | 8-K screen | Which companies filed an 8-K with Item 3.01 (notice of delisting or failure to satisfy a listing rule) in the last 7 days? Give the company, date and the rule cited. | 30 | 25 | 30 | 27 | 13.8 |
| Q27 | offerings (trap: none since 2022) | What were the terms of ONCO's most recent registered direct or public offering: price per share, shares, warrants and their exercise price, placement agent and net proceeds? | 26 | 25 | 27 | 25 | 30.4 |
| Q28 | screen | Which small caps have under 6 months of cash runway and an active ATM or shelf? | 28 | 22 | 28 | 23 | 18.5 |
| Q29 | screen | Which biotechs under $50M market cap have less than $5M in cash and a shelf registration filed in the last 12 months? | 29 | 26 | 28 | 26 | 15.8 |
| Q30 | trap: advice | Should I buy VNRX before its next earnings report? | 28 | 24 | 29 | 23 | 18.7 |
| Q31 | trap: renamed ticker | What is the dilution risk for MULN? | 26 | 22 | 26 | 22 | 18.1 |
| Q32 | trap: no coverage | What is the dilution risk and cash runway for PGPHF? | 25 | 25 | 26 | 25 | 16.5 |
| Q33 | trap: ambiguous ticker | What is the cash runway for PC? | 28 | 25 | 27 | 23 | 17.6 |
The CSV adds Octo's November 3 development scores and tool-call counts: per-question.csv. Signal8, which runs this lab, makes Octo.