A guest post from Paul Philip, chief executive of the Solicitors Regulation Authority
Philip: Introduction of SQE as in-person test is timely
Every result on the Solicitors Qualifying Examination (SQE) matters. Yet one recent result caught my eye – a candidate who scored a respectable if unspectacular 50%. The candidate was ChatGPT – the artificial intelligence (AI) chatbot.
This is impressive. SQE1 is not about regurgitating legal knowledge – you must be able to understand and practically apply the law in fresh scenarios.
It probably won’t be long until AI achieves near-perfect scores. Launched last month, GPT-4 took the US Uniform Bar exam, excelling with a score that would put it in the top 10% of candidates. It did similarly well in medicine, maths and biology, not to mention passing the advanced sommelier course.
I won’t speculate on whether laptops will replace waiters, but what does this mean for professional assessments?
A key point is that even if a computer can do something, it doesn’t necessarily follow that students don’t need to learn about it. The calculator has not made understanding multiplication redundant. Spell checks do not remove the need to learn spelling and grammar.
Just because a lawyer can look something up doesn’t invalidate the need for them to be able to understand the law.
Even if a lawyer can generate a decent legal answer from AI, they still need to be able to assess its validity. They also need the skills to be able to apply that knowledge usefully – from problem solving to being able to build a rapport with a client and understand their needs.
Challenges for academic integrity
In education, the biggest concern about AI has been its threat to academic integrity. If students can generate strong essay answers with a few clicks, how we will know if learning is genuine?
Some are looking to tackle this risk head-on through detection tools, but their effectiveness is questionable. And for every patch to detect robot wrongdoing, there will be a counter-measure to cover it up.
Others are embracing AI. The International Baccalaureate has already confirmed students can use AI, but must clearly reference it as a source. It predicts that AI will see a shift away from essay writing, with a greater focus on testing skills, such as assessing the quality of an argument or detecting bias.
It’s likely that skills such as critical thinking, creatively working through problems and interpersonal skills will become increasingly important.
The plagiarism challenge will also likely see more emphasis on in-person exams. In this respect, the introduction of the SQE is timely. Reassurance that all candidates have taken the same assessment, under the same controlled conditions, is now more important than ever.
Valid assessment
The content of the SQE also holds up well. As well as testing the practical application of legal knowledge, SQE2 also focuses on legal skills, including advocacy and client interviewing.
Effective oral communications, being able to impart difficult messages sensitively, or understanding how to deal with a client in vulnerable circumstances, are all skills which AI will not be replacing anytime soon.
All the analysis shows the SQE is a fair, rigorous and valid assessment. The competences it tests are the right ones for today’s qualifying solicitors.
Yet I have no doubt that AI will have huge implications for the legal sector. Some tasks that have traditionally been done by lawyers or paralegals will likely be replaced by the instantaneous efficiency of tech.
Even if, in many instances, qualified lawyers will need to provide oversight, they will also be focusing more on those nuanced tasks where human input adds the most value.
Will the use of AI become so ubiquitous in legal services that we will have to incorporate it into the SQE? Or conversely will AI become so embedded in society and education that there won’t be a need to test it specifically for qualifying solicitors? Will sophisticated use of AI remain a specialist skill needed by some but not all lawyers?
There are plenty of valuable skills and areas of law we don’t test to qualify. The SQE focuses on the core fundamentals, while giving firms the flexibility to tailor additional staff training to their business needs.
There is a risk that if, in the future, we try to incorporate practical tech skills into the SQE mix, by the time those changes have washed through the assessment system, the world will have moved on.
Revolution in training
There is also the question of marking. Research shows that, if AI is trained with thousands of sample marked answers, it can then mark specific questions well.
It’s not perfect – for instance if that dataset reflects any societal prejudice, the AI will replicate that bias. It also raises tricky questions around accountability and trust in the validity of the results.
The exam regulator Ofqual has explored its use. The focus was not on replacing human markers, but testing how AI could assist with marking, for example by helping spot errors or identifying an inconsistent marker.
Given the risks, solely using AI to mark high stakes professional exams like the SQE would not be appropriate. However, the potential for such technology to provide immediate feedback to candidates on their performance in lower-stakes scenarios, such as mock tests, could be invaluable.
AI can also use your data to understand your interests and learning styles. It can then tailor training content to keep you engaged, while also focusing on areas of weakness. China has already rolled out this technology to aid the learning of millions of its pupils.
We could be on the cusp of a revolution in the training market. This could make the profession more accessible if it leads to more effective, affordable training.
No easy answers
The pace of change is mind boggling. None of us can predict with certainty where this will take us.
I – like most others – have more questions than answers, so I thought I’d finish by seeing if AI could help. I asked ChatGPT whether it thought the SQE would change to reflect AI.
After making several sensible points, it concluded that it depends on how much the sector adopts AI, while any decision to change would require consultation and approval from the SRA.
So it looks like the ball is back in our court, a helpful reminder that currently AI is only an assistant not a replacement for important decision making. So I’ll confirm that we have no plans to change any of the fundamentals of the SQE.
Yet all of us – from training providers to firms – need to focus on AI’s impacts; we won’t be able to ignore them.
Leave a Comment