ChatGPT and SEC filings: what to check
A general AI chatbot can read an SEC filing well when you give it the filing. The trouble starts when it has to find the right filing itself: small-cap facts change with every 8-K, and an answer that was right last quarter can be wrong today. Our pilot round is too small to say how often any tool gets it wrong in general; these are the failure modes the benchmark is built to catch, and the pilot caught examples of several.
Where answers go wrong
- Stale numbers. The share count on the last 10-Q cover is out of date after any offering, conversion or warrant exercise filed since.
- Reverse splits. A warrant count or exercise price from before a split is wrong by the split ratio until it is restated.
- The wrong company. Tickers get reused and companies rename; a web search can return an older company that used the same symbol.
- Summaries of summaries. An answer built from news articles or forum posts inherits their errors and their dates.
- Confident gaps. Asked for a figure it cannot find, a model may produce a plausible number instead of saying so.
Five checks for any AI answer about a filing
- Ask which filing the answer came from, and open it on EDGAR.
- Check the filing date, then look for any later 8-K, 424B or 10-Q that could change the answer.
- Check for a reverse split since that filing.
- Check that the company name on the filing matches the company you mean.
- Prefer an answer that says "I could not find it" to one that gives a number with no source.
In our blind round, where every question named its filing, ChatGPT's free plan scored 159 of 160: its one lost mark was a source page that could not be opened. Gemini scored 125, with 4 wrong figures. Tools built on a filings archive remove the search step: Octo from Signal8 (which runs this site) scored 160 with no wrong figure and ranked #1 on our published tie-break; Claude also scored 160. In the harder round 3, where questions did not name the filing, ChatGPT dropped to 145 of 160 with 2 wrong figures, while Octo and Claude scored 156 with none.