Joshua Williams
What Joshua reviews here
Joshua is the named author of the benchmark pages, the question set, and the guide on checking ChatGPT answers against SEC filings. For each benchmark question the lab records the correct answer from the company's own filing, with its accession number, and then grades every tool's answer against it using the rubric published on the methodology page.
Joshua's concern on this site is that the grading stays the same for every tool, including Octo, which Signal8 makes. The rubric is written down before answers are graded, every answer is published verbatim in the raw CSV, and the losses of Signal8's own tool are reported in the same table as its wins.
What the results do and do not show
A benchmark round shows how a set of tools answered a fixed set of questions on one date, on the plans stated. It does not show how those tools will answer different questions, and AI products change often. Each result carries its run date for that reason.
Who publishes this
Signal8 Inc. publishes this lab and makes Octo. The methodology page sets out the rubric and the deviations from the full protocol.