Why AI Review That Passes the Demo Can Miss on a Real SWMS
AI SWMS review is usually shown on clean, typed documents. Real contractor SWMS arrive as scans, photographs and forms where the answer is a tick, a shaded cell or a handwritten date. A reader that only extracts text can miss those, so test any AI review on your own messiest documents, and keep a person signing the result.
An AI review demo is a controlled test. The documents are clean, the inputs are predictable and the result looks excellent, so everyone in the room is impressed. Then the AI meets the first week of real contractor paperwork, and the question changes from whether it works to whether it works on what your contractors actually send.
I have spent more than 25 years in safety, compliance and risk, and most of that time reading documents other people wrote.
This post is for the safety manager, WHS manager or facilities manager who has seen an AI review demo and now has to decide whether to trust it on their own contractors' safe work method statements (SWMS). It explains why review that tests well can miss on real documents, what closes the gap, and the test to run on any vendor.
The examples are a facilities provider that approves contractor work in the plant rooms and on the roofs of a commercial property portfolio, and a distribution centre that lets outside trades work beside live forklift aisles. They are illustrations, not customers.
Why Does AI Review Look Better in a Demo Than on Your Own Documents?
Because a demo is built from documents that suit the reader. A typed SWMS with clear headings, one answer per field and text the software can extract cleanly is the easiest document an AI will ever see, and it is the one most demos use.
Your contractors do not send that document. They send what their own template, their own software or their own office produces.
Some are typed, and many are exported from another system with a layout nobody chose. Others are printed, signed, scanned and emailed, or arrive as a phone photo of a page on a ute bonnet. The words in the criteria are the same as in the demo, but the evidence on the page is not.
Australia's own guidance does not treat AI as something you switch on and leave.
The National AI Centre's guidance tells buyers to ask the supplier for proof the system has been properly tested, to test it before they deploy it, and to monitor it afterwards, because an AI system can behave less predictably than conventional software.1
The Office of the Australian Information Commissioner (OAIC) puts it more bluntly: AI systems are known to produce inaccurate or false results, and due diligence should consider whether a product has been tested for the use you have in mind.2 A demo on someone else's clean documents is not that test.
A SWMS Is Short, and Its Layout Is the Problem.
It is tempting to think a long document is the hard one for an AI. A project safety plan is often far longer, but it is usually predictable: paragraphs, headings and tables, written by people who write safety plans for a living. An AI reading it is reading prose.
A SWMS is often a few pages, and it is harder. It answers a short list of questions about one job: the work, the hazards, the controls, who checks them and what the crew wears.
But each contractor answers those questions in its own layout, and a lot of the answers are not sentences at all. They are a tick in a box, a cross in a column, a shaded cell in a risk matrix, a signature, or a date written sideways in a margin.
So the difficulty is not the size of the document; it is the variance between documents. A reader that copes with one layout has learned one layout.
The question for any AI review is whether it understands what the page means, whatever shape the page takes, and that is a much harder test than reading clean text. For what a reviewer should be checking in the first place, see our SWMS review checklist.
One Question, Six Ways to Answer It.
Take one criterion every reviewer cares about: does the SWMS say what personal protective equipment the crew must wear for this task? Across a month of real submissions, the same question can come back in at least six forms:
- a cross in a column header beside a printed list of equipment
- a hand-drawn cross on a printed form
- a row of tick boxes, some ticked, some blank
- table cells shaded green to mean required
- a scanned page, slightly rotated, where the ticks are faint
- a plain bulleted list in typed text
A person reads all six in seconds. A system that only pulls the text out of the file sees a list of equipment in every case, and may not see which items were ticked, which were crossed out and which were left blank. That is how a review can report that the equipment is addressed when the page says the harness was never selected.
At the facilities provider, the risk shows up on plant-room work. A mechanical contractor's SWMS for a chiller replacement lists isolation, lifting and hearing protection, and the contractor ticked only two of the three.
At the distribution centre, a racking repair SWMS arrives as a photographed form, and the exclusion zone around the bay is a hand-drawn diagram, not a sentence. In both cases the answer the reviewer needs is on the page, but not in the text.
Can AI Read a Scanned or Handwritten SWMS?
It can, if it is built to look at the page and not only read the text. The approach that closes the gap is to read the document twice.
The first pass extracts the text and grades it against the criteria. Where the page is a scan or a photograph, or where the text alone cannot settle a criterion, a second pass looks at the page as an image: the ticks, the shading, the diagram and the handwriting.
Where the two passes disagree, the image should win, because the image is what the contractor actually submitted and what an inspector would look at.
That is a design choice, not a feature every product has, so ask about it directly. Four questions settle it:
- Does your AI look at each page as an image, or only at the text it extracts?
- When the text and the image disagree, which one decides the grade?
- What happens to a page it cannot read, such as a blurred photo? Does it say so, or does it grade it anyway?
- Does the result show which page and which part of the page each grade relied on?
The third question matters most. A review that says it could not read page four is useful. A review that quietly grades page four from whatever text it could find is the miss you will not see until an audit.
Say What Was Identified, Never That It Is Compliant.
An AI reader is not deterministic: run it twice on a borderline document and the wording, and sometimes the grade, can differ. The OAIC describes generative AI as probabilistic in nature, and notes that it does not understand the data it handles in the way a person does.2 The language of the result has to reflect that.
So the result should never say a SWMS is compliant, and a single verdict on the whole document is never enough on its own. It should grade each criterion by what it found, what it partly found and what it did not find, with the reason and the place in the document it relied on.
"Exclusion zone not identified on the method page" tells the reviewer where to look. A grade with no reason tells them nothing, and a compliance verdict claims a judgement the software is in no position to make.
This is also how to read any vendor's demo. If every result on screen is a confident green label with no reasons, the product is asking you to trust the label. If every grade comes with its evidence, it is asking you to check it, which is the right request.
Who Signs Off an AI-Reviewed SWMS?
A person does, every time. The National AI Centre's guidance asks organisations to match human oversight to the stakes, and gives mandatory human review for high-stakes decisions as the example.1 Approving a SWMS decides whether a crew works at height on a roof, inside a plant room or beside a live forklift aisle, which is about as high-stakes as an approval gets.
What the AI removes is the heavy reading, not the decision. Before, a reviewer reads every page, checks every box and cross-references every criterion.
After, the AI does the read-through and lays out what it found, and the reviewer looks at its output and the edge cases, opens the document where a criterion is not identified, and signs off on the result. The reviewer's name goes on the decision, so the reviewer needs to be able to see why each grade was given.
There is a second benefit worth more than the time saved. If the contractor can run the review before submitting, they see the gaps first and fix them, so the document that reaches the reviewer is better. Why review backs up in the first place is covered in why SWMS review becomes a bottleneck.
Test It on Your Own Worst Documents Before You Trust It.
What we recommend: do not judge AI review on the vendor's documents. Judge it on yours, and pick the ones that would trip it up. Five steps work for any vendor:
- Pick 20 real SWMS from your own contractors, including the messiest: scans, phone photos, handwritten amendments, forms full of tick boxes, and one built on another company's template. Include ones you approved and ones you rejected. The number is our rule of thumb, not a published figure.
- Run them blind beside a reviewer. The reviewer grades each one against the same criteria without seeing the AI's result.
- Compare criterion by criterion, not document by document. Count the misses, where the AI said a criterion was met and the reviewer found it was not, separately from the false alarms.
- Look at what it could not read. Every unreadable page should be reported as unreadable. A page graded without being read is a fail of the test, whatever the grade.
- Decide what a person must always see. Some criteria, such as a rescue plan for work at height, are worth a person's eyes on every document whatever the AI says.
The facilities provider might run its 20 from last year's plant-room and roof permits. The distribution centre might use its dock and racking repairs. For the longer version, including shadow running and what to measure, see how AI should be used in contractor and permit software.
How ComplyFlow's AI SWMS Review Handles This.
ComplyFlow is contractor management software, and its AI SWMS Review reads a contractor's SWMS against criteria you set, on a permit, a SWMS form or a prequalification. It is part of the wider AI in ComplyFlow, and its Help Centre documents three of the principles above.
- You can test it on your own documents first. When you build a review agent, you can run it against a real sample document and see the result before it goes live.3
- Each criterion is graded with feedback. The result marks each criterion Pass, Partial or Fail, with specific comments, so the reviewer can see why.4
- A person still decides. The AI review does not replace human review and does not block submission.5
Contractors can see the criteria before they run a review, so they know what the document will be read against.3 For how it handles the four reading questions above, ask us in a demo and test it on your own documents. For the permit workflow around it, see digital permits, and for the record each review sits in, contractor management.
Start With the SWMS Your Reviewers Dread.
Every safety team has one: the contractor whose SWMS arrives as a blurry scan of a form filled in by hand. Put that one first in your test.
If an AI review can tell you what that page says, show you where it found it, and admit when it cannot read it, it is ready to sit beside your reviewers. If it cannot, you have learned that before a crew went on the roof.
To see AI review on your own paperwork, book a demo and bring the SWMS your reviewers dread, along with one you approved and one you rejected.
Sources
See It Against Your Own Contractors, Sites & Rules.
Book a 30-minute demo. We will show ComplyFlow working with your kind of sites, your kind of contractors, and your requirements. No slides, no hard sell.
Or Speak to Us
- ISO 27001
- Hosted on AWS
- Microsoft & Okta SSO
- API & MCP
- Data in Australia
Written by
Sam BourneCo-Founder & Director of Risk & Compliance, ComplyFlow
Sam co-founded ComplyFlow and brings over 25 years in safety, compliance, and risk management to it. He writes about what a safety document has to prove, and what AI can and cannot be trusted to read.
Writes about: Safety documents and SWMS, AI document review, Risk and compliance
Questions
Questions People Ask About This.
How accurate is AI document review on contractor paperwork?
It depends on the documents, not only the AI. Review that looks accurate on tidy typed files can miss answers given as ticks, shaded cells, scans or handwriting. The only accuracy figure worth trusting is one you measure on your own contractors' documents, beside a reviewer, before anyone relies on it.
What documents should I use to test an AI SWMS review?
Real ones from your own contractors, chosen to be difficult: a scanned SWMS, a phone photo of a page, one with handwritten amendments, one built on an old template with another company's name in it, and one you rejected. Include ones you approved too, so you can count false alarms as well as misses.
Should AI review approve a SWMS automatically?
No. Let the AI read the document against your criteria and say what it found and what it could not find. A named person decides whether the work goes ahead, because that decision puts people on a roof, in a plant room or beside a loading dock.
Should an AI read a SWMS as text or as an image?
Both, where the page needs it. Text extraction handles typed pages well but can miss ticks, shaded cells, diagrams and handwriting. A reader that also looks at the page as an image, and lets the image decide when the two disagree, sees what the contractor actually submitted. Ask any vendor which it does, and what it reports when it cannot read a page.
What should an AI SWMS review result look like?
Each criterion graded, with the reason and the place in the document it relied on, so the reviewer knows exactly where to look. A score or a pass on its own, with no reasons, gives the reviewer nothing to check.