ChatGPT: Just don’t ask it about prescriptive easements
AI chatbot ChatGPT has struggled to cope with a dozen legal scenarios, all based on the kind of questions faced by high street lawyers, put to it by a Northern Irish solicitor.
Shane McCann said that, as well as producing one factually incorrect answer, three-quarters of the chatbot’s responses were below the standard of a “real lawyer”, but it was better at drafting simple legal agreements.
Mr McCann, managing director of SG Murphy Solicitors in Belfast, said he carried out the research with a paralegal to “try and gauge the impact of ChatGPT on the legal industry” in terms of the kind of questions actually faced by small to medium-sized practices.
The types of law reflected the services provided by SG Murphy – family, personal injury, property, employment and wills.
The question the chatbot answered incorrectly was about prescriptive easements in the UK.
Mr McCann said the answer stated that claimants needed to prove necessity, which was not the case, and that prescriptive easements were rare, which “does not align with our experience”.
In the majority of answers, the information was “at a basic and general level”, with 73% being “below what we would expect from a real lawyer”.
Mr McCann said lack of advice tailored to the client was another challenge.
“A large part of what a lawyer does is the asking of questions. By doing this, the lawyer can give advice tailored to the needs, circumstances, and goals of that particular client.
“As ChatGPT does not do this, it means the information it gives in response to the questions posed is, in the majority of cases we found, very general and quite basic.”
However, the chatbot was able to distinguish the jurisdiction in most cases. “In those questions where we highlighted that the questioner was asking a Northern Ireland-specific query, the chatbot did seem to recognise this, and tailor its response accordingly.”
Mr McCann did not believe that in the short term, clients would use chatbots in a way that would compete with the services provided by lawyers.
“At present, they are more of a starting position, that a client would perhaps use to carry out some general research before going to meet their own lawyer.”
In his study, ChatGPT answered a detailed question from an imaginary client with mesothelioma, producing an answer that was correct, but gave “limited detail”.
In the case of a domestic violence victim living in Northern Ireland, the answer failed to deliver “full advice”, such as how to apply for a non-molestation order.
The chatbot performed better when asked about personal injury limitation periods, giving “factually correct information” and a “reasonable level of response”.
ChatGPT did particularly well at drafting a simple will while also pointing out that it was “not qualified to draft a legal will for you”.
Mr McCann added the chatbot gave a lot more information than he expected. “I can definitely see the value of it. Clients will use it for their initial research. They’re already doing research on Google and reading a lot about the law before coming in.
“ChatGPT does say that it makes mistakes. As the amount of data going into it increases, the number of mistakes will decrease.
“The problem may be that clients think that all of its answers are 100% correct. That would be a concern.”
When asked by Mr McCann whether AI chatbots like itself would disrupt the legal industry and compete with lawyers, ChatGPT replied that the legal industry was “being impacted by AI and automation technologies”, but it was unlikely that chatbots would replace lawyers entirely.
“However, AI chatbots can help streamline certain tasks, such as answering frequently asked questions, generating documents, and providing legal advice to clients.
“This can free up lawyers’ time to focus on more complex tasks that require legal expertise and judgement.”
Elsewhere in a lengthy answer, ChatGPT observed that “AI chatbots cannot provide the same level of empathy, emotional support, and personalised attention that human lawyers can provide”.
Leave a Comment