Zimmerman: AI is brilliant at summarisation
Lawyers need to avoid being “so seduced” by what artificial intelligence (AI) can do “that they don’t appreciate what it cannot”, a professional negligence specialist warned last week.
Felix Zimmerman told the Law Society risk and compliance annual conference in London: “The danger is that you get too bewitched by what it can do and delegate to the machine tasks that aren’t appropriate.
“It’s important to bear in mind that it’s ‘just’ a large language model – it’s just a text predictor. It doesn’t really understand in the way a human does.”
Mr Zimmerman, a partner at City firm Simmons & Simmons who has defended many solicitors, explained how “brilliant” AI was at summarising documents.
He said he fed the firm’s AI tool – called Percy, after one of its founding brothers – the famous case of Donoghue v Stevenson and asked it for a five-sentence summary in style of Shakespeare.
“What came back was truly astonishing, in the sense that it was not just a brilliant parody of William Shakespeare in rhyming couplets but it also suggested that the machine totally understood the key points of the case…
“You would think this is a sentient being that completely understands both Shakespeare and law…
“It is brilliant at summarisation… and produces them in about three seconds… That is undeniably life-changing for practitioners.”
He also recounted how he used Microsoft Copilot to take “coherent, structured, real-time” notes of meetings.
These were examples of how AI for a litigator “is a brilliant innovation and anybody who doesn’t get on top of those sorts of things is going to get left behind”.
But Mr Zimmerman went on to explain AI’s current limitations around legal research. For example, he asked Percy a question around valuer negligence and the summary it provided was “specious, superficial and in some key respects wrong”.
In professional negligence claims, he continued, AI could not currently cope with the hypotheticals and counterfactuals that form part of considering causation.
“To me the danger for lawyers is that they are so seduced by what it can do that they don’t appreciate what it cannot – and there’s a danger that lawyers will seek to pass off as their own work what is in fact the product of a large language model.”
Another example involved a matter where the claimant was resident in Arizona. He asked an associate to check if they could apply for security for costs, one test for which is whether the jurisdiction is bound by the Hague Convention.
Percy said they could not make an application because the US was a signatory to the convention; however, the US did not actually ratify it and so they could. Fortunately, the associate double-checked.
“That’s the key point – the lawyer always has to test the output of the AI because it will get it wrong on occasion, especially if you’re asking it legal questions.”
Mr Zimmerman concluded: “We all have to use this fantastic technology because life is too fast-paced, there’s too much to do, we can’t keep up with all the burdens of what we have to do – and the AI can help us.
“But on the other hand, it’s as important as ever for the lawyers to appreciate that they are the ones who have to do the analysis and research and ultimately give the legal advice.”
Asked if there were circumstances where it could be negligent for a solicitor not to use AI, Mr Zimmerman suggested disclosure was an area where the machine was better than humans at finding, for example, what was privileged among millions of documents.
I suspect this view won’t age very well. The AI he is speaking of is ‘narrow AI’ in the form of LLMs, not AGI which, depending on who you ask, is anywhere between 3 and 10 years’ away from being realised. The Great Wave is coming, and fixating on the current limitations of AI for legal practice is a mistake – instead, lawyers need to prepare themselves for a future where 80-90% of what they did before is fully automated.