Owned by Signal8 Inc., maker of Octo and Signal8 Pro. Not investment advice. About this site
stockairesearch / lab

ChatGPT and SEC filings: what to check

By , Signal8 Research. Published by Signal8 Inc.

A general AI chatbot can read an SEC filing well when you give it the filing. The trouble starts when it has to find the right filing itself: small-cap facts change with every 8-K, and an answer that was right last quarter can be wrong today. Our pilot round is too small to say how often any tool gets it wrong in general; these are the failure modes the benchmark is built to catch, and the pilot caught examples of several.

Where answers go wrong

Five checks for any AI answer about a filing

  1. Ask which filing the answer came from, and open it on EDGAR.
  2. Check the filing date, then look for any later 8-K, 424B or 10-Q that could change the answer.
  3. Check for a reverse split since that filing.
  4. Check that the company name on the filing matches the company you mean.
  5. Prefer an answer that says "I could not find it" to one that gives a number with no source.

In our blind round, where every question named its filing, ChatGPT's free plan scored 159 of 160: its one lost mark was a source page that could not be opened. Gemini scored 125, with 4 wrong figures. Tools built on a filings archive remove the search step: Octo from Signal8 (which runs this site) scored 160 with no wrong figure and ranked #1 on our published tie-break; Claude also scored 160. In the harder round 3, where questions did not name the filing, ChatGPT dropped to 145 of 160 with 2 wrong figures, while Octo and Claude scored 156 with none.