Waters: Huge potential gains
Three-quarters of lawyers agree that using generative artificial intelligence (GenAI) helps them “get to a first draft quicker”, according to a report on trials run by a leading City law firm.
The research also found that lawyers at Ashurst struggled to identify legally correct case studies created by GenAI as being generated by technology rather than humans.
Researchers said “the greatest initial value for GenAI in a law firm context is in helping lawyers to create first drafts quicker and more efficiently”.
Some 77% of lawyers in a post-trial survey agreed or strongly agreed that usage of GenAI had helped them get to a first draft quicker.
Ashurst recorded time savings of 80% in drafting UK corporate filings based on review and extraction of information from company articles of association, 59% in drafting industry/sector-specific reports that required reviewing and extracting key information from public company filings, and 45% in creating first draft legal briefings.
GenAI was viewed by lawyers as “very helpful” in enabling them to overcome so-called blank page syndrome and could save “two hours per participant per task” when it came to preparing first drafts of non-legal content, such as job descriptions, policy documents and social media posts.
“It is Ashurst’s view that using GenAI in this way has the potential to impact the drafter’s workflow by freeing them up to conduct higher-value work or other development activities,” said the report, Vox PopulAI: Lessons from a global law firm’s exploration of generative AI.
However, time savings were always subject to the content being correct. “If the initial steer provided by GenAI is incorrect, there is a risk that the drafter is sent along an incorrect path, which may result in more time being taken to correct the work once checked”.
Ashurst carried out three GenAI trials involving 411 partners, lawyers and staff representing all of its practice areas and business services functions across 23 offices in 14 countries.
The data used was publicly available, not client data, with “rigorous guardrails” adopted throughout.
In a blind study, common assessment criteria were designed to allow Ashurst to compare the GenAI output of different tools against the outputs of lawyers working without them to produce case summaries.
An expert panel, asked to assess whether the summaries were legally correct, found that, when judged on accuracy alone, human lawyers scored higher on average.
The panel correctly identified all the case studies written by lawyers as human-generated.
When it came to the case studies written by GenAI, they were less successful. Exactly half the GenAI output was misidentified as human-produced or the experts were not sure.
In two-thirds of the GenAI case studies that were not correctly categorised, the level of accuracy was as good or better than those produced by lawyers.
Aside from accuracy, “a mix of language choice, tone and structure all provide indications of source – our experts specifically referenced language and writing techniques as clear indicators of human work”.
The report continued: “Writing techniques referenced as human included well-structured executive summaries and analysis of issues in addition to pitching vocabulary correctly.
“Commentary also focused on how the author applied the judgment and whether broader legal implications were explored. Interestingly, inconsistencies in grammar and spelling were also called out as human traits too.”
With users prepared to accept certain errors on the part of the AI, the early evidence was that “quality is a multi-dimensional concept and one that needs further exploration”.
Ashurst said the value uncovered during the trials was “far broader than we originally anticipated” with some lawyers regarding it as a ‘second pair of eyes’ or a ‘spare pair of hands’ to check work; 61% of respondents felt that using GenAI would help them feel more supported in managing their workload.
The vast majority of respondents (88%) at the end of the biggest trial said that using the technology helped them to feel more prepared for the future.
However, lawyers “reported frequent hallucinations across all tools trialled, which strongly supports the need to keep ‘the human in the loop’”.
Tara Waters, Ashurst’s chief digital officer, said: “Generative AI offers huge potential gains, not only in terms of efficiency but also new value-creation opportunities, for both law firms and our clients…
“There are lots of possibilities this new technology offers and much more testing to be done. It is now vital that moving forward we embrace sharing and collaborating in this way, as we all navigate the noise around GenAI and promote meaningful, beneficial change in the legal industry.”
Hilary Goodier, global head of Ashurst Advance, the firm’s innovative arm, added: “We now have a better understanding of pain-points, user-needs and our lawyers’ daily experience, which has allowed us to make investment decisions based on measurable data and to move forward with designing our own GenAI policies and implementing its use in our firm.”
Leave a Comment