Nylon scores highest in new benchmark of AI tax research

How good are AI tools at answering real tax questions? 

For accountants, tax advisers and bookkeepers, a useful answer needs to be more than plausible. It needs to be correct, complete, and supported by sources you can actually check, especially when you’re using the answer to inform client advice. 

So, we put six AI research systems to the test on 91 current-law Australian tax questions. Nylon scored a composite 4.68 out of 5, ahead of Claude (Fable 5.1) at 4.32 and ChatGPT (GPT-6 Astra) at 4.28. 

Nylon ranked highest across all five dimensions we tested, with a statistically significant lead on the overall score the individual measures, including correctness and citation grounding. 

The full benchmark is available for independent review, including every question fingerprint, every answer and verdict, the complete grading instructions and a script that reproduces the calculations. 

As far as we know, this is the first benchmark created for Australian tax research, and part of Nylon’s commitment to transparency and supporting trusted advisers.  

Read the full report: Tax Research Tested

What the benchmark found 

Figure 1: Composite score (Nylon Practitioner Rubric v1.0, judged by Kimi K3).

Nylon scored highest across all five dimensions we tested: correctness, completeness, helpfulness, faithfulness and citation grounding. Its lead over the other systems was statistically significant at the 95% confidence level. When thinking about which tools to use, advisers can look to this benchmark to understand the difference in models and help evaluate systems.  

Figure 2: Nylon against the two nearest systems, by criterion. Scores are the mean from 91 questions on a 1 to 5 scale. 

The difference becomes particularly relevant when you look at correctness and how easy an answer is to verify. 62% of Nylon’s answers received a perfect correctness score of 5 out of 5, compared with 40% for Claude Fable 5.1 and between 2% and 18% for the other systems. This means that Nylon gives more correct answers much more often than other providers. 

Nylon’s answers were also more likely to anchor claims to the relevant authority, with statutory rules supported by the legislation itself where it was available, rather than relying on guidance pages or secondary summaries. For a practitioner reviewing an AI-generated answer, that distinction matters because finding the right answer is only part of the job. You also need to be able to verify the answer, establish where it came from and whether the cited source actually supports the conclusion. 

How the benchmark was run 

The questions came from an established, independent source: the 120 end-of-module review questions in CPA Program Australia Taxation (6th edition), each with a suggested expert answer. Because tax law moves on, we reviewed the questions against current primary sources before running the benchmark and excluded 29 questions where the original reference could no longer be used reliably. 

That left 91 current-law questions, including on income and deductions, CGT, GST, FBT and entities. 

The six systems answered every question between 28 September and 1 October 2026: Nylon’s standard research service, using the October Snapshot, and five frontier models, each run through its provider’s API with web search and page reading. The systems tested were Claude Fable 5.1, Claude Opus 5.5, GPT-6 Astra, GPT-6 Sol and Gemini 3.1 Pro. 

Every answer was then graded blind by Kimi K3 from Moonshot AI, a judge model developed by a company independent to any of the systems being tested. The judge was given the question, the expert reference answer, the answer being graded and the text each system had actually retrieved from every source it cited, allowing it to assess not only whether an answer was correct but whether the claims made were supported by the sources behind them. 

The judge was not told which system had produced which answer. 

The practitioner rubric 

Rubric and weighting:

• Citation grounding: 35%  Do the cited sources actually support the claims being made? 

• Faithfulness: 30%  Does the answer accurately represent what its sources say? 

• Correctness: 15%  Is the answer correct against the expert reference answer and current primary sources? 

• Completeness: 10%  Does the answer cover the important elements of the question? 

• Helpfulness: 10%  Would the answer actually help a practitioner research the issue? 

Citation grounding and faithfulness carry the greatest weight because a tax answer that cannot be traced back to supporting authority is much harder for a practitioner to verify and rely on, even when the conclusion itself happens to be correct. 

Nylon is publishing the full benchmark for independent review, including the question fingerprints, exclusions, rubric, judge’s exact instructions, every answer, every verdict and the judge’s reasons, together with a script that reproduces every number in the report. 

Why the difference? 

We have long known that Nylon’s unique reasoning and research agents, linked directly to primary law and commentary, produce better answers than generic AI tools. This is because Nylon is able to navigate the law in a specialized way, using our database of case law, legislation and government guidance. Because we don’t just store the sources, but also additional context and connections, our system understands the meaning of the documents, paragraph by paragraph; leading to more reasoned answers. This benchmark proves that our investment in this data layer and a focus on the research lawyers and accountants need, makes an meaningiful difference to answer quality and user ability to rely on outputs.  

Why test one thing? 

There are already plenty of AI benchmarks that measure how well a system performs across a wide range of tasks. We deliberately took a narrower approach because we know accountants and tax advisers want to know which tools are best for their work. Researching and writing an answer to a tax question is something every general-purpose AI model can do, making it a useful way to compare the quality of the underlying research process. 

The question we wanted to answer was straightforward: when an AI system is asked to research an Australian tax issue, does it apply the right law, use appropriate authority and give the practitioner an answer that can be checked? 

That distinction matters because a tax practitioner is not simply looking for an answer that sounds convincing. They need to understand the reasoning behind it, identify the authority supporting the conclusion, and be able to verify the relevant legislation, ruling or other source before relying on it in their work. 

It is also worth being clear about what did not contribute to the result. Much of the value firms get from Nylon sits outside a single researched answer, including non-public, government-origin data, an internal precedent system, a certainty score on every answer, single-interface source verification and siloed engagement context. None of those features were used in this benchmark, but all provide additional value to our users beyond the great answers. 

The benchmark therefore measures the research engine itself rather than attempting to measure every difference between Nylon and a general-purpose AI assistant. 

A benchmark is only a snapshot 

AI models are changing quickly, and the systems tested here will change too. A result that is accurate today should not be treated as a permanent leaderboard, particularly as model capabilities; search systems and retrieval methods continue to evolve. 

We see it as our responsibility to keep testing this as the market changes, so Nylon will continue to re-run the benchmark and publish updated results as new models become available. 

We also plan a follow-up that combines leading models from the major AI labs with Nylon’s system through Nylon’s MCP, allowing us to test how those models perform when they have access to Nylon’s tax research capabilities. 

FAQ 

What is the best AI for tax research? 

In our October 2026 benchmark of 91 current-law Australian tax questions, Nylon scored 4.68 out of 5, ahead of Claude Fable 5.1 at 4.32, GPT-6 Astra at 4.28, Claude Opus 5.5 at 4.07, GPT-6 Sol at 3.96 and Gemini 3.1 Pro at 2.97. 

The result applies specifically to the questions, methodology, and systems tested in this benchmark, rather than being a permanent ranking of AI models. 

Can I trust a benchmark Nylon ran itself? 

That is the right question to ask of any vendor benchmark, including this one. We have tried to make the results as easy as possible to inspect rather than asking readers to take the conclusion on trust. 

The questions came from an established external source with suggested expert answers. Grading was performed blindly by a judge model from a developer unconnected to every system being tested, and the judge had access to the source text retrieved by each system so that citation claims could be checked directly. 

Most importantly, the full benchmark materials are being published, including the questions, answers, verdicts, grading instructions and a script that reproduces the calculations, so anyone who wants to inspect or re-run the analysis can do so. 

How accurate is ChatGPT for tax research? 

In this benchmark, OpenAI’s GPT-6 Astra scored 4.08 out of 5 on correctness, compared with 4.46 for Nylon. 

The benchmark also showed differences in how systems supported their answers. The general-purpose systems were more likely to rely on guidance pages or secondary material where the legislation itself was available, or to make claims that were not fully supported by the sources they cited. 

That is why the benchmark measures citation grounding and faithfulness separately from correctness. An answer can arrive at the right conclusion while still being difficult to verify, and that distinction is important when AI is being used as part of professional tax research. 

Does the benchmark cover New Zealand or UK tax? 

Not yet. This edition tests Australian tax because the question set with expert reference answers exists for Australia. The same principles of research apply to all jurisdictions and these results can be extrapolated. 

Nylon also provides tax research for Australia, New Zealand and the UK through the same underlying research system tested in this benchmark. 

What was the benchmark judged on? 

The benchmark used five dimensions, weighted as follows: 

• Citation grounding: 35%

• Faithfulness: 30%

• Correctness: 15%

• Completeness: 10%

• Helpfulness: 10%

The weighting reflects the importance of being able to verify AI-generated tax research against the sources supporting it, rather than relying solely on whether the final answer appears correct. 

See the full benchmark 

The headline number matters, but the more important question is whether you can see how it was produced. 

The full report sets out the methodology, rubric, exclusions and judge’s instructions, while the complete benchmark materials make it possible to inspect the answers and verdicts for yourself. 

Read the full report: Tax Research Tested

If you’d like to see how Nylon handles your own tax research, book a demo or start researching with Nylon today.