Capability round: can AI tools do the research job? Claude ranked first, Signal8 Octo second
CAPABILITY ROUND · answers collected November 12, 2026 · 20 research jobs · grading guide sealed before any tool ran · graded blind · corrected November 13, 2026
Claude ranked first of five AI tools in a blind test of 20 research jobs that need real market or filing data, or computation: 34 of 40 points. Signal8's Octo was second with 28, then ChatGPT's free plan (26), Perplexity (19) and AskEdgar Studio (18). No tool invented a result. Gemini answered 11 of 20 on its free tier and is listed, not ranked. Signal8 makes Octo and runs this lab.
Correction, November 13, 2026. The first version of this page ranked Signal8 Octo first and Claude last, with 16 of 40. Claude's 16 was a fault in our test harness, not in Claude. The claude.ai account was left in Manual approval mode, so on 9 tasks Claude said what it would do and then stopped at a "needs your intervention" prompt that no one answered, and the account's network access was off. On November 12, 2026 we re-ran those 9 tasks (C-02, C-06, C-07, C-09, C-10, C-14, C-15, C-16, C-17) on the same account with Auto approval and outbound network access set to all domains, each in a fresh chat with the task text verbatim and no follow-up. Fresh Claude graders graded each new answer blind, against the same sealed guide, one answer per packet. All 9 were graded done, so Claude has 34 of 40 and ranks first; Octo is second with 28. The other four tools were not re-run. Claude's first-run grades are kept in the CSV and under each task below.
Our top pick for SEC filing facts · joint top score in both blind filing rounds
Octo, Signal8's AI research assistant
We recommend Octo for questions answered in SEC filings. On the research jobs in this round Claude scored higher.
- Second in this round: 28 of 40, 6 behind Claude and 2 ahead of ChatGPT's free plan.
- A result on 19 of 20 tasks: 9 done and 10 partial; its one zero (C-17) was a task only Claude's re-run completed.
- Joint top score in both filing-accuracy rounds, level with Claude, first on the published tie-break and with zero wrong figures in 80 answers: round 3, round 2.
What was tested, and why
Our earlier blind rounds asked whether a tool quotes a filing fact correctly. This round asks something different: can the tool actually do a common research job that needs data or computation? Run a backtest, screen the market, pull a structured history, run an event study, chain several lookups. A chat answer that describes the method is not the same as a result, and a number with no computation behind it is worse than no number.
The 20 tasks: 5 backtests (C-01 to C-05), 4 market screens (C-06 to C-09), 4 per-ticker lookups (C-10 to C-13), 3 event studies (C-14 to C-16) and 4 multi-step jobs (C-17 to C-20). Every task is pinned to dates on or before October 31, 2026 (two read filings up to November 10), so the answers cannot drift.
Results and ranking
Ranked by points. No two tools tied, so no tie-break was needed.
| Rank | Tool and run conditions | Points | Done | Partial | Not done | Invented |
|---|---|---|---|---|---|---|
| #1 | Claude Corrected November 13, 2026: first run 16 of 40 after a harness fault on our side, 9 tasks re-run. Correction. claude.ai on a Max plan, Opus 5.5 with web search, no connectors; can run code, no market database. Nine tasks re-run with Auto approval and network access on | 34 / 40 | 14 | 6 | 0 | 0 |
| #2 | Signal8 Octo Made by Signal8, which runs this lab. signal8.ai/octo in production on a Signal8 staff account; purpose-built market database and backtester | 28 / 40 | 9 | 10 | 1 | 0 |
| #3 | ChatGPT Free plan, signed in, default model; can run code, no market database | 26 / 40 | 10 | 6 | 4 | 0 |
| #4 | Perplexity Free plan, default mode | 19 / 40 | 3 | 13 | 4 | 0 |
| #5 | AskEdgar Studio Studio agent on a paid account (Studio is in AskEdgar Pro, listed at $59 a month) | 18 / 40 | 6 | 6 | 8 | 0 |
| Not ranked | Gemini Free tier; answered 11 of 20 before its free usage limit. Incomplete, so not graded. | Not graded | - | - | - | - |
| Job | Claude | Octo | ChatGPT | Perplexity | AskEdgar |
|---|---|---|---|---|---|
| Backtests (5 tasks, up to 10) | 6 | 5 | 5 | 3 | 3 |
| Market screens (4 tasks, up to 8) | 7 | 7 | 7 | 6 | 2 |
| Per-ticker lookups (4 tasks, up to 8) | 7 | 7 | 4 | 5 | 5 |
| Event studies (3 tasks, up to 6) | 6 | 4 | 4 | 3 | 5 |
| Multi-step research (4 tasks, up to 8) | 8 | 5 | 6 | 2 | 3 |
Downloads: every grade with the grader's note (CSV), the sealed grading guide (JSON), the blind label mapping (JSON) and the grader instructions.
Read these caveats first
- Tools were tested as shipped. Default model and settings, no uploaded data, no connectors added, one fresh chat per task, the task text verbatim, no follow-up. The exception is Claude's re-run, set out in the next point. Claude and ChatGPT have no market database, but both can write and run code and search the web. Octo has a purpose-built market database and a backtester, and that difference is what this round measures. Code that would compute the answer but was not run scored partial, the same for every tool.
- Claude's score comes partly from a re-run. Its first run scored 16: on 9 tasks (C-02, C-06, C-07, C-09, C-10, C-14, C-15, C-16, C-17) it stopped at a "needs your intervention" prompt because we had left the account in Manual approval mode, with network access off. We re-ran only those 9 tasks, with Auto approval and network access on all domains; some ran for 30 to 45 minutes. Its other 11 answers are from the first run, including six partial answers, some of which the same network setting may have limited. Each re-run answer was graded alone, not beside the four other tools' answers as in the first pass. The other tools were not re-run.
- Octo has gaps, and they cost it points. Its backtester caps a run at 50 open positions, so C-03 to C-05, which ask for every signal as a separate trade, skipped signals and scored partial. It has no 200-day moving average filter, so C-02's trend rule was not computed. Its borrow tool returns the current Interactive Brokers file plus logged changes, not a value on any past date, so for C-11 it gave a reading logged October 29 rather than October 30. It has no market-wide halt ranking, so C-17 scored zero; only Claude's re-run did C-17. It also gave cover share counts rounded to 0.1 million (C-18), a 37-split sample (C-14) and an estimated baseline (C-16).
- C-01's reference was questioned and kept. ChatGPT matched the reference on every metric (54 trades, +0.21% average, +8.6% total). Octo and AskEdgar Studio both computed about +0.6% average per trade, outside tolerance. We reviewed the reference script (Yahoo Finance daily SPY prices, entry at the next open, exit at the close of the fifth day after entry, 5 basis points each side) and kept it.
- The graders are Claude instances. Five fresh Claude instances graded the first pass, and further fresh Claude instances graded Claude's re-run answers: the same model family as one contestant. That contestant ranked first, so a grader bias toward it cannot be ruled out.
- Different plans. ChatGPT, Perplexity and Gemini ran on free plans, AskEdgar Studio on a paid account, Claude on a paid Max plan as Opus 5.5 with web search, Octo on Signal8's production service with a staff account. Two answers (AskEdgar Studio C-19, ChatGPT C-10) were asked again in a fresh chat because the first attempt's page loaded with no input box.
- Signal8 runs this lab and makes Octo. Signal8 chose the tasks, wrote the guide and ran every tool. No reference value was built from Signal8 output: every one comes from public sources (SEC EDGAR, Nasdaq Trader, SEC fails-to-deliver files, Yahoo Finance's public price API, the Federal Reserve's FOMC calendar), and Signal8's tools were not called while building them.
How the round worked
- Sealed guide. For each task the guide says what "done" requires, gives sanity checks, reference values with their public source and tolerance where an independent reference exists, examples of partial work and red flags for invented figures. Where no independent reference exists (the market-wide backtests C-03 to C-05, the float screen C-08, the split study C-14, the 13F job C-20 and the borrow half of C-11) it grades on method, sanity bands and spot checks against public filings, never against another tool's answer. The guide was sealed at 2026-11-12T02:05:00Z with SHA-256
ce6d523a707a059be69521d9bac00a118cd0226e9619240dd0836284ba80f622, before any tool was asked. Download the same file to check the hash. - Asking. Each tool got each task verbatim in a fresh chat on November 12, 2026, with no follow-up.
- Blind grading. For each task the five answers were shuffled, labelled A to E, stripped of tool names, and graded by five fresh Claude instances (four tasks each) that saw only the instructions, the guide and the anonymous answers. They could open public sources such as EDGAR for the spot checks the guide asks for. Hashes of the packets (not published, because they hold raw answer text) and of the mapping:
a36f01fc13cae03750acd6c618dd4ebc0308a80f24644ba2b136ed0052c8e952 packets.json;67367b398e5067f0c91608792092b609ce075b87df2b7e63391f686bcfe0c0a3 mapping.json. Claude's 9 re-run answers were then graded the same way, one answer per packet, labelled A; that packet file's SHA-256 is19d7875bf1bce31b831163bf872750107ea38d0ed70a2e0b6c89c70ff1cc2e37. - One grade per answer. Done (2): a result from real data with the method stated, passing the sanity checks and, where a reference exists, the tolerance. Partial (1): correct method or code but no result, a result on a small or unstated sample, a real result for part of the task, or a result outside tolerance with the method stated. Not done (0): declined or generic advice. Invented (-1): figures presented as results with no real computation or source; it overrides everything else in the answer.
Every task and every grade
| Task | Job | Claude | Octo | ChatGPT | Perplexity | AskEdgar |
|---|---|---|---|---|---|---|
| C-01 | Backtests | P | P | D | - | P |
| C-02 | Backtests | D* | P | D | P | D |
| C-03 | Backtests | P | P | - | P | - |
| C-04 | Backtests | P | P | - | - | - |
| C-05 | Backtests | P | P | P | P | - |
| C-06 | Market screens | D* | D | D | D | - |
| C-07 | Market screens | D* | D | D | P | P |
| C-08 | Market screens | P | P | P | P | P |
| C-09 | Market screens | D* | D | D | D | - |
| C-10 | Per-ticker lookups | D* | D | P | P | P |
| C-11 | Per-ticker lookups | P | P | - | P | - |
| C-12 | Per-ticker lookups | D | D | P | D | D |
| C-13 | Per-ticker lookups | D | D | D | P | D |
| C-14 | Event studies | D* | P | P | P | D |
| C-15 | Event studies | D* | D | P | P | P |
| C-16 | Event studies | D* | P | D | P | D |
| C-17 | Multi-step research | D* | - | - | - | - |
| C-18 | Multi-step research | D | P | D | P | P |
| C-19 | Multi-step research | D | D | D | P | D |
| C-20 | Multi-step research | D | D | D | - | - |
| Points | 34 | 28 | 26 | 19 | 18 |
C-01 · Backtests
Backtest this on SPY using daily prices (price return, ignore dividends). Signal: SPY closes down 2% or more from the previous day's close. Entry: buy at the next trading day's open. Exit: sell at the close of the 5th trading day after the entry day. Only one position at a time: ignore any signal that occurs while a position is open. Costs: 5 basis points per side. Use signals from 2016-01-04 through 2025-12-31. Report the number of trades, win rate, average return per trade (net of costs), total compounded return, and the maximum drawdown of the equity curve marked at each trade exit. Show the dates of the first and last trade.
| Tool | Grade | Grader's note |
|---|---|---|
| Claude | Partial | Wrote and smoke-tested a correct script (next-open entry, 5-day exit, no overlap, 5 bps per side) but never ran it on real SPY data. |
| Signal8 Octo | Partial | Real engine run with the method stated. Trades (57) and win rate (57.9%) are within tolerance, and the first trade matches the reference. Avg +0.6%, total +30% and DD -24.5% (daily, not exit-marked) are outside tolerance, and the last trade date is missing. |
| ChatGPT | Done | All metrics are within tolerance and match the reference (54 trades, 55.56% win, +0.212% avg, +8.61% total, -15.76% exit-marked DD). First and last trades are correct, and the method and data feed are stated. |
| Perplexity | Not done | Declined to compute after data loading failed; gave only the method and two verified price rows. No results. |
| AskEdgar Studio | Partial | Real data with source and method stated. 54 trades and the first and last trades match the reference exactly, and win rate 59.3% is within 5 pp. Avg +0.61%, total +31.7% and DD -22.2% are outside tolerance, and the answer says the bars were read by hand. No red flag. |
| Gemini | Answered, not graded (incomplete run) | |
C-02 · Backtests
Backtest a trend filter on QQQ from 2016-01-04 to 2025-12-31 using daily prices (price return, ignore dividends). Rule: at each close, if QQQ's close is above its 200-day simple moving average of closes, hold QQQ; otherwise hold cash (0% return). Trade at the next day's open after the signal changes (the first decision uses the 2016-01-04 close and can execute at the 2016-01-05 open). Costs: 5 basis points per side. Mark the position to market at the 2025-12-31 close. Report CAGR, maximum drawdown (daily closes), number of entries into QQQ, and the same CAGR and maximum drawdown for buy-and-hold QQQ from the 2016-01-04 open to the 2025-12-31 close.
| Tool | Grade | Grader's note |
|---|---|---|
| Claude re-run | Done | Real Yahoo price-only backtest with method stated (200-SMA from 2015 warm-up, next-open execution, 5 bp/side, open-to-close/close-to-open accounting). Strategy CAGR 16.00%, MDD -23.21%, 23 entries; B&H CAGR 18.85%, MDD -35.62%, all within tolerance of the reference (16.0%, -23.2%, 23, 18.8%, -35.6%). First run (replaced, correction): Not done. Stopped at 'needs your intervention' with no method or result. |
| Signal8 Octo | Partial | Real buy-and-hold result (CAGR ~18.8%, DD ~-35.6%) that matches the reference, but the trend-filter strategy was not computed. That is only part of the task. |
| ChatGPT | Done | Strategy CAGR 16.01%, DD -23.18%, 23 entries; buy-and-hold 18.84% / -35.62%. All within tolerance of the reference. Rules, costs, price-only basis and SMA warm-up are stated. |
| Perplexity | Partial | Correct method and Python code with a verified final close, but explicitly no executed result. |
| AskEdgar Studio | Done | CAGR 16.01%, DD -23.18%, 23 entries; buy-and-hold 18.85% / -35.62%. All within tolerance. Method, costs and data path are stated, with the transcription caveat disclosed. |
| Gemini | Answered, not graded (incomplete run) | |
C-03 · Backtests
Backtest buying US common stocks that close up 20% or more on a day, with that day's close above $1 and 20-day average dollar volume above $1 million. Buy at the next day's open, sell at the close 5 trading days after entry. Period: signals from January 2023 to December 2025. Include stocks that were later delisted. Treat every signal as a separate equal-weight trade (no position limit). Costs: 10 basis points per side. Report the number of trades, win rate, average and median return per trade, and the maximum drawdown of an equal-weight portfolio of these trades.
| Tool | Grade | Grader's note |
|---|---|---|
| Claude | Partial | Full backtest script tested on synthetic data, including delisting handling, but no run on real data and no results. |
| Signal8 Octo | Partial | Real market-wide run that includes delisted names. 4,971 trades, 38.9% win, -1.9% mean, -4.1% median are all within the sanity bands. But the engine capped the book at 50 positions and skipped 1,598 signals, so it is not every signal, and the -93.1% DD is for the capped portfolio. Both points are disclosed. |
| ChatGPT | Not done | Declined for lack of a survivorship-free universe. Gave only a rule spec, with no code and no result. |
| Perplexity | Partial | Correct SQL for the trade-level statistics against a stated price_history table, but not executed. The drawdown pass was not written. |
| AskEdgar Studio | Not done | Declined the market-wide run and offered a ticker-list alternative. No code and no results. |
| Gemini | Answered, not graded (incomplete run) | |
C-04 · Backtests
Backtest a breakout rule on US common stocks, 2018-01-01 to 2025-12-31, including stocks later delisted: signal when a stock closes at a new 52-week closing high on volume at least 3 times its 20-day average volume, with close above $5 and 20-day average dollar volume above $5 million. Buy at the next day's open, sell at the close 20 trading days after entry. Treat every signal as a separate equal-weight trade. Costs: 10 basis points per side. Report the number of trades, win rate, average and median return per trade, and the average SPY return over the same holding windows.
| Tool | Grade | Grader's note |
|---|---|---|
| Claude | Partial | Wrote and synthetic-tested a full backtest script (delisting and split handling, matched SPY windows), but no real-data run. |
| Signal8 Octo | Partial | Real run that includes delisted names. 3,735 trades, 51.2% win and 0.1% mean/median are within the sanity bands. The matched SPY average is explicitly not measured, a 50-slot cap skipped 2,554 signals, and the breakout may use the intraday high. All disclosed. |
| ChatGPT | Not done | Declined. Gave a specification and data requirements only. |
| Perplexity | Not done | Declined for lack of universe data. Gave a specification only, with no code and no results. |
| AskEdgar Studio | Not done | Declined the market-wide run and gave a rule spec plus a ticker-list offer. No code and no results. |
| Gemini | Answered, not graded (incomplete run) | |
C-05 · Backtests
Backtest buying large one-day crashes in US common stocks, 2016-01-01 to 2025-12-31, including stocks that were later delisted: signal when a stock closes down 30% or more in one day, its previous close was above $5, and its 20-day average dollar volume before the drop was above $10 million. Buy at the next day's open, sell at the close 10 trading days after entry (if the stock is delisted first, exit at its last traded close). Every signal is a separate equal-weight trade. Costs: 20 basis points per side. Report the number of trades, win rate, average and median return per trade, the worst single trade, and how many trades were in stocks that later delisted.
| Tool | Grade | Grader's note |
|---|---|---|
| Claude | Partial | Wrote and unit-tested a backtest script with the stated rules and spelled out the edge cases. It had no data, so no actual result. Code only. |
| Signal8 Octo | Partial | Real run on 1,075 trades: 40.8% win rate, mean -2.4%, median -4.8%, worst -86.8%. All pass the sanity checks. But the rules deviate: a crash-day close above $3.5 instead of a previous close above $5, the ADV window may include the drop day, and a 50-slot cap skipped 114 signals, which breaks 'every signal is a separate trade'. It also did not report the count of later-delisted trades (it reports 12 that stopped trading mid-trade). Real result for only part of the task. |
| ChatGPT | Partial | No results. It gives a full, correct backtest specification and names the right data sources (CRSP/Norgate), but there is no code and no output: method only. |
| Perplexity | Partial | Correct method and formulae, but it hit a query limit before running anything. No figures are reported. Method only. |
| AskEdgar Studio | Not done | Declined the market-wide run and only offered a small ticker-list or sample alternative. No method executed and no results. |
| Gemini | Answered, not graded (incomplete run) | |
C-06 · Market screens
List every operating company (exclude ETFs, ETNs, funds, warrants, rights and units) that was on Nasdaq's Reg SHO threshold security list on every trading day from 2026-10-01 through 2026-10-30. Give the ticker and company name for each and the total count.
| Tool | Grade | Grader's note |
|---|---|---|
| Claude re-run | Done | Intersected all 22 Nasdaq daily threshold files, got 34 symbols, excluded 27 ETFs/funds (incl. HIS) and listed 7: AIXI, FRGT, NCPL, NIVF, ONCO, WHLR, XHLD, an exact match to the reference with no extras. First run (replaced, correction): Not done. Only a stated intention ('I'll pull...') then a stall requiring user action. No result. |
| Signal8 Octo | Done | Exactly the 7 reference names, from the 22 daily Nasdaq files, with ETFs/warrants/preferreds excluded and near misses explained. Matches the reference. |
| ChatGPT | Done | Exactly the 7 reference names, count 7, 22 trading days. Method: the Oct 1 and Oct 30 lists plus consecutive-session history, operating-company filter applied. |
| Perplexity | Done | Six names (AIXI, FRGT, NCPL, ONCO, WHLR, XHLD), all in the reference, with no non-qualifying names. It misses NIVF. Method: consecutive-session runs covering all 22 days, ETFs excluded. Meets the 6-of-7 threshold. |
| AskEdgar Studio | Not done | Declined: no daily history was available. It only describes the method in a sentence. No list. |
| Gemini | Answered, not graded (incomplete run) | |
C-07 · Market screens
List every company that filed a Form 8-K reporting Item 3.01 (Notice of Delisting or Failure to Satisfy a Continued Listing Rule or Standard; Transfer of Listing) with the SEC between 2026-10-01 and 2026-10-15 inclusive. For each give the ticker, filing date and the reason in one phrase (for example: minimum bid price, stockholders' equity, merger completion, late filing). Give the total number of filings and of distinct companies.
| Tool | Grade | Grader's note |
|---|---|---|
| Claude re-run | Done | Lists 50 filings from 49 companies with tickers, dates and per-filing reasons, matching the reference count and including all examples, the merger closings (WBD, QRVO, BLFS, SLP, CCO) and BETR twice. A spot-check against EDGAR full-text search (Item 3.01, 8-K, 2026-10-01..15) returned 50 hits, and every company and date matches the list, with no extras. First run (replaced, correction): Not done. Only a stated intention, then a stall requiring user action. No list. |
| Signal8 Octo | Done | 50 filings / 49 companies, matching the reference exactly, with BETR's double filing identified. It gives tickers, dates, reasons and rule citations, and the five reference examples are present. About 6 merger rows lack a ticker in the table, though most are named in the text. |
| ChatGPT | Done | Lists about 45 filings with tickers, dates and broadly correct reasons, including mergers (WBD, QRVO, SLP, CCO). It includes all four of the five reference examples it could (LTRN, GETY, VTIX, NCPL) and misses LUCY, BLFS and BETR's second filing. NSTS/CNHF may be extras but are within the 3 allowed. It claims 49 but lists about 45, and the totals cells are blank: a minor inconsistency. |
| Perplexity | Partial | Honest small sample: 8 filings, most with no ticker or reason. NCPL is dated 10-07 against the reference's 10-14. It declines to give a total. Correct approach, only a sample. |
| AskEdgar Studio | Partial | 40 candidates with reasons, self-labelled unverified, built from summaries rather than item tags. It wrongly excludes merger and regained-compliance filings (BLFS, WBD, QRVO, NCPL, WORX, AREC). Several names look like non-reference extras (HUBG, SKYD, SOBR, LNAI, VIP, MBIO, ARTL), so the likely correct count is in the 20-39 band. |
| Gemini | Answered, not graded (incomplete run) | |
C-08 · Market screens
As of 2026-10-31, list US exchange-listed common stocks (Nasdaq, NYSE, NYSE American) with a public float under 10 million shares that had an active at-the-market (ATM) offering program on that date. For each, give the float figure and its source date, the ATM agent and the ATM's stated size or remaining capacity with the filing it comes from. Say how many you found and how you searched.
| Tool | Grade | Grader's note |
|---|---|---|
| Claude | Partial | Four names (ASTC, ENVB, MTVA, XELB) with agents and ATM sizes from 424B5s. Two floats are undated third-party figures, and October 31 status and termination are unconfirmed. It discloses that the list is incomplete. Under 5 names with undated or unverified fields. |
| Signal8 Octo | Partial | Eight names with agent, ATM figure and filing, plus a stated screening method (533 active ATMs cross-checked against a 1,254-name float screen). Float source dates are blank for 4 of the 8 (ARTL, GNLN, BRNX, TGL), STKH is ADSs, and GNLN's ATM status is stale (read through Aug 24). Strongest C-08 answer but fails the dated-float requirement. |
| ChatGPT | Partial | Two names (HSCS, IDAI) backed by real October 9 424B5 links, agent and capacity, with exclusions reasoned (AEMD, APVO). Only 2 names, and it gives two different float figures per name across its tables. The explanation for so few is weak. |
| Perplexity | Partial | Method only: a correct screen design that separates float from shares outstanding and an active ATM from a sales agreement. Its connector had no float or ATM fields, so no list. |
| AskEdgar Studio | Partial | Claims 146 names from a filings database but shows only 9. It admits automated ATM classification, no termination check and float extraction noise. One visible row is inconsistent: MTRI 'Metro Inc.' links to an InMed 424B5. Real data but unverified, so it cannot pass as DONE. |
| Gemini | Answered, not graded (incomplete run) | |
C-09 · Market screens
Find every US-listed company where at least 3 different insiders (officers or directors) made open-market purchases (SEC Form 4 transaction code P) within any 14-day window, with all those purchase dates between 2026-10-01 and 2026-10-31. For each company give the ticker, number of distinct buyers, the purchase date range, and the total dollar value bought. Exclude closed-end funds and other investment vehicles.
| Tool | Grade | Grader's note |
|---|---|---|
| Claude re-run | Done | 14 clusters parsed from all Oct 1 to Nov 10 EDGAR Form 4s, code P, officers and directors only, with the 14-day window stated. All 14 names are on the cross-check list. 7 of 7 spot-checked on EDGAR (RAPP, NPB, STC, BBT, CHCO, FFIN, WVVI) match exactly on buyers, dates and totals. RAPP is $1,664,824 with Healy's $20.4M offering purchase excluded and explained. First run (replaced, correction): Not done. Only a statement of intent followed by a 'needs your intervention' stall; no result. |
| Signal8 Octo | Done | 11 companies with first and last dates and totals, funds excluded, and boundary-crossing clusters listed separately; all 11 are on the cross-check list. It notes the $20.4M RAPP director buy at $21 but does not identify it as offering participation, so the RAPP total is overstated. The buyer-count column did not render. |
| ChatGPT | Done | 12 companies with method stated (code P, transaction dates, distinct CIKs, funds excluded); all 12 are on the OpenInsider cross-check list. Flaw: the RAPP total ($22.0M) folds in Healy's ~$20M $21.00 offering buy without calling it offering participation, and the buyer-count column did not render for most rows. |
| Perplexity | Done | 11 companies with buyer counts, date ranges and totals, and the all-in-October window rule explained; all 11 are on the cross-check list (RENT, ACCV, CHCO and NRXS included). The RAPP $22.0M total includes the offering buy without comment, and it caveats that the source may count 10% owners. |
| AskEdgar Studio | Not done | Declined: says it has no structured Form 4 data and gives no list. |
| Gemini | Answered, not graded (incomplete run) | |
C-10 · Per-ticker lookups
List every trading halt in Jaguar Health (JAGX) between 2026-08-01 and 2026-10-29: date, halt time (ET), reason code, and resumption trade time. Give the total count and a count by reason code.
| Tool | Grade | Grader's note |
|---|---|---|
| Claude re-run | Done | 22 halts (21 LUDP, 1 T3) taken from the Nasdaq Trader feed, with the same per-date counts as the reference (9/16 1, 9/17 1, 9/22 8, 9/23 4, 9/25 4, 10/5 3, 10/16 1). Spot-checked 9/16 and all 8 halts on 9/22 against the RSS: halt and resume times match to the second. It explains the T1/T3 recoding. First run (replaced, correction): Not done. Stalled at 'needs your intervention'; no result. |
| Signal8 Octo | Done | 22 halts, 21 LUDP and 1 T3 (the 09-16 19:50 halt resuming 09-17 09:00), with halt and resumption times for every one. Per-date counts match the reference exactly. |
| ChatGPT | Partial | All 22 halts with dates, times to the second and resumption times, matching the reference dates and counts. But it labels the 2026-09-16 19:50 halt T1 instead of T3 (codes must be exact), and gives no resumption for that halt. |
| Perplexity | Partial | Total of 22 with correct dates and halt times, but the reason codes are muddled (18 LUDP / 2 LULD / 1 'LULD Pause other' / 1 T1 vs the reference 21 LUDP + 1 T3), and most resumption times are missing. |
| AskEdgar Studio | Partial | Rebuilt 20 halts from news headlines, disclosed as possibly incomplete. It misses the 09-17 and 09-23 10:24 halts, gives no reason codes (plain words only) and almost no resumption times. |
| Gemini | Answered, not graded (incomplete run) | |
C-11 · Per-ticker lookups
For Onconetix (ONCO): what was the Interactive Brokers cost to borrow (annualised fee) and the number of shares available to short on 2026-10-30? Also give the SEC fails-to-deliver quantity for each settlement date from 2026-10-01 through 2026-10-14 and the total for that period.
| Tool | Grade | Grader's note |
|---|---|---|
| Claude | Partial | All nine FTD quantities and the 3,779,566 total match the SEC file exactly. The borrow half is not answered: it found only undated snapshots and correctly declined to assign them to 10-30. |
| Signal8 Octo | Partial | Borrow half is dated and sourced: an IBKR reading logged 10-29 at 77.686% with 100k shares, a plausible hard-to-borrow profile within one trading day of 10-30. FTD half has only 6 of 9 dates (those 6 exact); the missing dates are disclosed as not measured, giving a total of 2,510,646 vs the 3,779,566 reference. |
| ChatGPT | Not done | 'Data analysis isn't available'; no answer. |
| Perplexity | Partial | FTD quantities and the 3,779,566 total match exactly (10-12 shown as 0). It says the IBKR borrow fee for 10-30 is not verifiable and gives only an ORTEX 10-22 figure from a different source. |
| AskEdgar Studio | Not done | Found neither half; gives only a current 11-12 snapshot, explicitly not IBKR and not 10-30, and no October FTD figures. |
| Gemini | Answered, not graded (incomplete run) | |
C-12 · Per-ticker lookups
List every insider purchase reported on Form 4 (transaction code P) at Rapport Therapeutics (RAPP) with a transaction date in October 2026: insider name and role, date, shares, price and dollar value. Say which of these were open-market purchases and whether they form a cluster (3 or more insiders within 14 days). Give the total dollar value of open-market purchases.
| Tool | Grade | Grader's note |
|---|---|---|
| Claude | Done | All six open-market buyers with shares and prices; Healy is correctly separated as offering participation; the cluster call is yes and the total of $1,664,824 matches. Flaws: Silva and Sanchez are dated Oct 21 (reference 10-22/10-23, uncertainty disclosed), the share total is misstated (58,094 vs 61,694), and it says 7 open-market insiders when there are 6. |
| Signal8 Octo | Done | Form 4 rows match the reference for all six open-market buyers plus Healy's four $21.00 lots (969,480 sh). Healy is flagged as offering participation, the cluster call is yes (6 insiders in 3 days), and the open-market total of $1,664,824 matches exactly. |
| ChatGPT | Partial | Buyers, dates, shares and prices are correct, but Healy's $20.36M offering-price buy is folded into the open-market set without comment, so the stated total is $22.02M instead of about $1.66M. |
| Perplexity | Done | Six open-market buyers with Form 4 details, Healy's four lots excluded, and cluster yes. The $1,664,939.82 total is within 0.01% of the reference despite an arithmetic slip on one Maraganore lot. It calls Healy's buy a 'private placement' when it was the public offering, and its opening line miscounts (seven P transactions). |
| AskEdgar Studio | Done | All six open-market buys with dates, shares and prices; Healy is excluded as an offering-price allocation and the Third Rock code A is noted. Cluster yes (6 in 3 days); total about $1.66M matches the reference. |
| Gemini | No answer (free usage limit) | |
C-13 · Per-ticker lookups
List every prospectus (Form 424B1, 424B3, 424B4, 424B5 or 424B7) that Jaguar Health (JAGX) filed with the SEC from 2025-01-01 through 2026-10-31. For each give the filing date, the type of offering (at-the-market program, priced registered direct or public offering, or resale by selling stockholders) and, for a priced sale, the number of shares, the price per share and the gross proceeds. Give the total count by type.
| Tool | Grade | Grader's note |
|---|---|---|
| Claude | Done | 14 prospectuses, 6 ATM / 1 registered direct / 7 resale, which matches the reference. Registered-direct terms are exact (246,306 sh x $6.09 = $1.5M), but it is dated 2025-05-21 (an after-hours filing) instead of the EDGAR date of 05-22. ATM supplement sizes are correct. |
| Signal8 Octo | Done | 14 filings with correct dates and types (6/1/7). Registered direct 2025-05-22, 246,306 sh at $6.09, gross ~$1.50M computed. It wrongly says the cover does not state gross, but the terms match the reference. |
| ChatGPT | Done | 14 filings with correct dates and types (6/1/7). Registered direct 2025-05-22, 246,306 sh at $6.09, $1,500,003.54 gross plus 492,612 warrants, all exact. ATM capacities match the reference except No. 4, which has no figure. |
| Perplexity | Partial | Only 11 filings, outside 14 +/- 1. It misses the 424B3s of 2025-05-07, 2025-07-30 and one 2026-07-27, and misdates another as 2026-07-24. The types it lists are right and the registered-direct terms are right, apart from a hedged '$6.00-$6.09'. |
| AskEdgar Studio | Done | 14 filings, 6/1/7, all matching the reference dates. Registered direct 246,306 x $6.09 = $1,500,003.54, exact, though dated 05-21 instead of the EDGAR date of 05-22. It gives no priced terms for the ATMs. |
| Gemini | No answer (free usage limit) | |
C-14 · Event studies
For every reverse stock split of a US exchange-listed common stock (Nasdaq, NYSE, NYSE American) that became effective between 2026-07-01 and 2026-09-30, measure the return from the first post-split close to the close 30 calendar days later (or the nearest prior trading day). Report how many splits you found, the median and mean 30-day return, the share of splits with a negative return, and list the five worst.
| Tool | Grade | Grader's note |
|---|---|---|
| Claude re-run | Done | 182 reverse splits from the stockanalysis.com calendar with forward splits removed. Median -32.3%, mean -24.5%, 85.2% negative. All 5 worst names pass on Yahoo: split dates SHFS 9/30, HUBC 9/14, CDT 9/29, NIVF 9/1, FEED 9/1, and returns of -95.8, -95.0, -94.7, -92.4 and -89.6% reproduce within 0.1 pp. The count is above the guide's rough 60-160 and the exchange was not checked for each name; the answer says so. First run (replaced, correction): Not done. Stopped at 'needs your intervention'. No list, no method detail, no result. |
| Signal8 Octo | Partial | Self-described incomplete sample of 37 splits, below the count of 40 that DONE needs. Median -28.2% and mean -31.5% are plausible, and the worst list includes CDT -94.69%, which is consistent with Yahoo. Real computation on a partial universe. |
| ChatGPT | Partial | States the method and cites a split-universe table (the counts did not render), but explicitly declines to compute returns. Method or count only. |
| Perplexity | Partial | Gives a candidate count of 204 and a correct formula and method, but declines to compute any returns. Count only, no returns. |
| AskEdgar Studio | Done | 179 splits (173 measured), median -28.45%, mean -24.49%, 87% negative, method and exclusions stated. All 5 worst spot-checked against Yahoo split-adjusted closes and all match (HUBC -95.0%, SHFS -95.8%, CDT -94.7%, NIVF -92.4%, WCT -87.0%). The count is slightly above the 'roughly 60-160' range. |
| Gemini | No answer (free usage limit) | |
C-15 · Event studies
Event study: take every Form 8-K reporting Item 3.01 filed with the SEC between 2026-10-01 and 2026-10-15 inclusive, excluding filings that also report Item 2.01 (completed acquisitions) and excluding SPACs. For each company, measure the stock return from the close of the last trading day before the filing date to the close of the 10th trading day after the filing date (count the filing date as day 0 if it is a trading day). Report the number of events, the median and mean return, and how many were negative.
| Tool | Grade | Grader's note |
|---|---|---|
| Claude re-run | Done | 36 events after exclusions (9 Item 2.01, 4 SPACs, BETR deduped; same as the reference), 34 with usable split-adjusted Yahoo prices (NXDR and UCFI dropped). Median -21.7%, mean -18.7%, 29/34 negative vs reference -21.4%/-18.2%/29 of 35; all within tolerance, and a UCFI-as-0% variant gives -21.4%/-18.2%/29 of 35. The per-event list is in an attached CSV, and the text names the outliers and edge cases (TCRT, Getty, LESL, CECO). First run (replaced, correction): Not done. Stalled at 'needs your intervention'. No result. |
| Signal8 Octo | Done | 33 events with a per-event table and the stated exclusions (2.01, SPACs, BETR once). Median -20.9% (reference -21.36%), mean -16.6% (reference -18.17%), 27/33 negative (82%, reference 83%), all within tolerance. |
| ChatGPT | Partial | Event list given, 33 measured (BETR counted twice). Median -17.17% (within 6 pp) and 78.8% negative (within) are fine, but the mean of -8.52% is 9.7 pp off the reference -18.17%, outside the 8 pp tolerance. Method stated, no red flag. |
| Perplexity | Partial | Correct method and a partial list of 4 filings, but declines to compute any statistics. Method only. |
| AskEdgar Studio | Partial | Real computation: median -16.1% and mean -15.5% are within tolerance and 77% negative is within tolerance. But 53 events is far outside 35 +/- 5, because it classified filings by summary text instead of the Item 3.01 tag, so the universe likely includes non-3.01 filings. It gives no full event list. |
| Gemini | No answer (free usage limit) | |
C-16 · Event studies
Event study on SPY: for each of the 24 scheduled FOMC rate-decision days in 2023, 2024 and 2025, compute SPY's close-to-close return on the decision day and on the next trading day (price return). Report the mean and median for each, how many were positive, and compare with SPY's mean and median daily close-to-close return over all trading days from 2023-01-03 to 2025-12-31.
| Tool | Grade | Grader's note |
|---|---|---|
| Claude re-run | Done | Correct 24 FOMC dates (incl. 2024-11-07, 2025-10-29), per-date table with 2024-12-18 -2.98% and 2023-03-22 -1.70% matching the reference. D0 +0.03%/+0.03%/13, D+1 +0.22%/+0.30%/15, all-days +0.08%/+0.10%/430 of 752 (57%): every figure within tolerance, price-only method stated. First run (replaced, correction): Not done. Stalled at 'needs your intervention'. No result. |
| Signal8 Octo | Partial | Correct dates and a per-date table. Day 0 (+0.03/+0.03, 13 up) and day 1 (+0.22/+0.30, 15 up) match the reference. But the all-days baseline mean is only estimated from the endpoint closes and the baseline median is explicitly not computed, so the comparison part is incomplete. |
| ChatGPT | Done | Per-event returns match the reference (2024-12-18 -2.980%, 2023-03-22 -1.705%). Day 0 0.0313/0.0317 (13), day 1 0.2203/0.2956 (15), baseline 0.0816/0.1008 with 57.2% positive over 752 days, all exactly at the reference. |
| Perplexity | Partial | Correct 24 FOMC dates and correct method, but computes no returns because the data were inaccessible. Method only, no result. |
| AskEdgar Studio | Done | Day 0 +0.03/+0.03 (13), day 1 +0.22/+0.30 (15), all days +0.08/+0.10 with 430/751 positive. All within tolerance; n=751 because the first day was excluded. Dates were taken from memory but are correct. |
| Gemini | No answer (free usage limit) | |
C-17 · Multi-step research
Using Nasdaq's trade halt records for October 2026 (2026-10-01 to 2026-10-30), find the five Nasdaq-listed stocks (exclude units, warrants and rights) with the most trading halts in the month. For each give the halt count and the main reason code. Then, for each of the five, check SEC EDGAR and say whether the company filed a prospectus (any Form 424B) or a registration statement (S-1, S-3, F-1, F-3) during October 2026, naming the form and date.
| Tool | Grade | Grader's note |
|---|---|---|
| Claude re-run | Done | Top five WFF 72, POM 25, AIOK 24, RPGL 20, RIBB 19 with LUDP/T3 code splits, exactly matching the reference, with RIBBU/ALISU units excluded. EDGAR check also exact: WFF 424B5 Oct 15 + Oct 28, AIOK F-3 Oct 20, POM/RPGL/RIBB none. Its 'next' names (QETA/BKHA at 15) differ from the reference's MI 17, but the task did not ask for that. First run (replaced, correction): Not done. Only a statement of intent and an intervention prompt. No ranking, counts or EDGAR check. |
| Signal8 Octo | Not done | Declined. Its halt source returned only November data, so there is no October ranking and no EDGAR check. |
| ChatGPT | Not done | Declined. It lists a rolling-30-day leaderboard without counts and explicitly does not treat it as October, so there is no top five and no EDGAR step. |
| Perplexity | Not done | Declined the per-symbol ranking. It gives only aggregate October totals and a plan, with no top five and no EDGAR check. |
| AskEdgar Studio | Not done | Declined. It could not access the daily halt tables and names a few anecdotal halts (TJGC, UCFI) but no counts or top five. |
| Gemini | No answer (free usage limit) | |
C-18 · Multi-step research
For Rapport Therapeutics (RAPP): (1) list the common shares outstanding stated on the cover of each 10-K and 10-Q filed from January 2025 through 2026-11-10, with the as-of date; (2) identify the financing that explains the largest increase between two consecutive cover counts in 2026, with the security types, number of shares and pre-funded warrants, price per share, and any convertible notes issued alongside (principal, coupon, maturity, conversion price); (3) say whether any insider bought in that financing, with name and amount.
| Tool | Grade | Grader's note |
|---|---|---|
| Claude | Done | All 8 cover counts with as-of dates match the reference. October 2026 offering terms are correct (8,333,334 + 1,428,571 at $21, 1,190,476 PFW at $20.999, 2.25% notes due 2032-12-01, 37.3483/$26.775, $150M base upsized to $172.5M). Names Healy's 969,480 shares at $21 (misses Third Rock's PFWs). |
| Signal8 Octo | Partial | Financing terms and Healy's 969,480-share purchase are correct and the jump endpoints are exact, but the eight cover counts are given only rounded to 0.1M, not the exact figures the tolerance requires. |
| ChatGPT | Done | All 8 cover counts are exact (the Q3 2025 as-of date is given as Nov 3 vs the reference Nov 4). Offering, PFW and note terms are correct. Names Healy's 969,480 shares at $21 with Form 4 accession 0001104659-26-120789. |
| Perplexity | Partial | 6 of 8 cover counts are exact (Q2 2025 missing; Q1 2026 uses the balance-sheet figure 47,807,453 instead of the cover's 47,826,929). Financing and note terms are correct, but it says no insider bought, missing Healy and Third Rock. |
| AskEdgar Studio | Partial | All 8 cover counts are exact and the financing and note terms are correct, but it reports finding no insider purchase, missing Healy's 969,480 shares and Third Rock's PFWs. |
| Gemini | No answer (free usage limit) | |
C-19 · Multi-step research
Using each company's 10-Q for the quarter ended June 30, 2026, compare Rapport Therapeutics (RAPP), Lantern Pharma (LTRN) and Jaguar Health (JAGX): cash and cash equivalents plus short-term investments or marketable securities at June 30, 2026; net cash from operating activities for the six months ended June 30, 2026; and months of cash runway at that six-month operating burn rate. If a company's operating cash flow was not a burn, say why and what the runway estimate should rest on instead.
| Tool | Grade | Grader's note |
|---|---|---|
| Claude | Done | Exact balances and op CF for all three; RAPP 48 and LTRN 6.6 months. Explains JAGX's positive op CF via the Woodward license and gives a stated ex-license runway of about 1.8 to 2 months. |
| Signal8 Octo | Done | Balances, op CF and runways (48 and 6.6 months) match. Flags JAGX's positive op CF from the Woodward license and bases the runway on FY2025 operating cash use ($23.7M, about $1.97M/month, about 1.9 months), exactly the reference approach. |
| ChatGPT | Done | RAPP $436.077M / -$54.516M / 48.0 months and LTRN $7.361M / -$6.714M / 6.6 months match the reference exactly. Flags JAGX +$3.492M as driven by the Woodward license and gives a stated normalized runway (about 1.5 months). |
| Perplexity | Partial | RAPP is correct (48 months) and JAGX's positive op CF is flagged, but LTRN six-month op CF is given as -$2.9M (the reference is -$6.71M), so its 15.3-month runway is wrong. Two of three are right. |
| AskEdgar Studio | Done | RAPP and LTRN figures and runways match the reference with 10-Q links. Correctly flags JAGX's +$3.49M as Woodward-license driven and says the runway must rest on unrestricted cash, obligations and going-concern status rather than burn. It gives no alternative month figure. |
| Gemini | No answer (free usage limit) | |
C-20 · Multi-step research
For Rapport Therapeutics (RAPP), compare institutional holdings reported on Form 13F for the quarter ended 2026-06-30 against the quarter ended 2026-03-31: list the 10 largest holders at 2026-06-30 by shares with their share change versus 2026-03-31, name the largest new position and the largest full exit, and give the total shares held by 13F filers in each quarter.
| Tool | Grade | Grader's note |
|---|---|---|
| Claude | Done | Top-10 figures match the EDGAR spot-checks (TRV, Sofinnova, Polar, J&J exact), as do Algert's new 223,822 and Freestone's exit (stated as about 128k vs the exact 127,783). Q2 total 51.29M is derived; Q1 total about 52.7M is labelled an estimate. |
| Signal8 Octo | Done | Top-10 shares rounded to 0.1M (within 1% of the EDGAR figures for TRV, Sofinnova, Polar and J&J) with exact share changes that match. Algert new 223,822 and Freestone exit 127,783 are exact. Totals 52,482,931 -> 51,271,114. |
| ChatGPT | Done | Top-10 with changes. Spot-checked on EDGAR 13F-HR: TRV GP V 6,280,525 (-854,708), Sofinnova 1,904,408 unchanged, Polar 1,608,514 (-33,947), J&J 1,784,517 unchanged and Algert new 223,822 are all exact; Freestone Grove exit 127,783 confirmed. Totals 52.48M -> 51.27M (above the ~48M outstanding; attributed to double counting). |
| Perplexity | Not done | Declined; no holders, changes or totals. |
| AskEdgar Studio | Not done | Declined the 13F comparison. It gives only two 13G/A holders, no top 10, no totals, wrongly suggests TRV as the largest new position and finds no exit. |
| Gemini | No answer (free usage limit) | |
What this round does not show
It shows what five tools delivered on 20 fixed jobs on one day, on the plans stated, with no follow-up. A tool that runs out of a free allowance scores what it delivered. Claude's first-run stalls came from an account setting on our side, so we re-ran those tasks rather than score them zero. It does not show accuracy on filing facts, which the earlier round 3 and round 2 measured; there Octo and Claude tied on score. AI products change often, and each result carries its run date for that reason.