ChatGPT: All lawyers should be investigating AI, academics say
A groundbreaking study has found that using generative AI to support, rather than replace, humans results in “large and consistent increases in speed” in completing legal tasks.
The US academics who conducted it said the findings suggested that the practice of law “is on the precipice of significant – and potentially foundational – change and transformation”.
This change will, however, “occur unevenly across legal domains and practice areas”.
The study, Lawyering in the age of artificial intelligence, was released by three law professors: Daniel Schwarcz and Amy Monahan from the University of Minnesota, and Jon Choi from the University of Southern California.
To date, research has focused on AI’s ability to provide legal analysis itself, but the trio said its ability to assist humans was “the more plausible use case for the foreseeable future”.
So they conducted what they said was the first randomised controlled trial of AI assistance’s effect on human legal analysis.
They randomly assigned 60 law students to complete four separate legal tasks each – drafting a complaint, a contract, a section of an employee handbook and a client memo – either with or without the assistance of GPT-4, after receiving training on how to use it.
They then blind-graded the results and tracked how long the students took on each task.
“We found that access to GPT-4 slightly and inconsistently improved the quality of participants’ legal analysis but induced large and consistent increases in speed [of between 12% and 32%, depending on the task].
“The benefits of AI assistance were not evenly distributed: in the tasks on which AI was the most useful, it was significantly more useful to lower-skilled participants.
“On the other hand, AI assistance reduced the amount of time that participants took to complete the tasks roughly uniformly regardless of their baseline speed.”
This showed, they said, that AI was likely to serve “as an equalising force in a notoriously unequal profession”.
In follow-up surveys, the students reported increased satisfaction from using AI to complete legal tasks and that they correctly predicted the tasks for which GPT-4 would be most helpful.
“Standing alone, these results suggest that generative AI will almost certainly become a vital tool for many lawyers in the near future, comparable to more familiar legal-tech tools like Westlaw, Lexis and ediscovery software.”
The researchers said the results likely understated – substantially – AI’s capacity to improve the efficiency of legal services, especially as the technology improves.
“For lawyers and judges, they suggest that the time to embrace AI is now, though the contours of what that will mean can and should vary significantly by practice area, task, and the stakes of the underlying matters.
“And for purchasers of legal services, our results suggest that it is time to reconsider what types of legal matters should be sent to outside counsel rather than handled in-house, and how matters that are handled externally are managed and billed.”
They recommended that first-year law students be banned from, or substantially limited in, using AI to ensure they developed necessary legal reasoning skills, but thereafter trained to use it.
They added: “Our results strongly suggest that virtually all lawyers and law firms should be proactively exploring how best to incorporate generative AI tools into their practice. Of course, many law firms are doing just that.”
The academics referenced the now-infamous case of Steven Schwartz, the New York lawyer who used ChatGPT to draft a court brief, only for it to make up cases that supported the argument.
Lawyers who interpreted this incident to suggest the need to avoid generative AI “reach precisely the wrong conclusion”, they cautioned.
“Like any other tool, generative AI can be misused. The lesson to draw from this case, when considered in concert with the results of this study and prior evidence, is that lawyers and law firms that use generative AI tools must develop systems and procedures for doing so effectively.”
At the very least, they said, these systems should include confirming the veracity of any factual statements or source materials made by AIs, “experimenting with different prompting strategies when using general-purpose Ais”, assessing legal issues and tasks independently of AI, and avoiding entering any confidential information into general-purpose AIs that do not include a trustworthy assurance of confidentiality.
The professors concluded that the higher-level impact of AI was much harder to predict: “Will demand for legal services increase or decrease? Will firms alter the range of legal services that they send to outside counsel relative to the tasks that they perform in house? Will lawyer pay become higher, lower, or more uneven? And what impact will all of the above have on the demand and supply of lawyers and law students?
“Our empirical results offer limited guidance on these questions, other than to suggest that the assumption that the future will resemble the past is likely tenuous, at best.”
Leave a Comment