Bradley: Assessments were robust, fair and reliable
The first solicitors to qualify via the Solicitors Qualifying Examination (SQE) will enter practice next month after 77% of candidates passed the second set of assessments.
However, the results of the first sitting of the SQE2 have continued to show stark difference in attainment between white and certain minority ethnic students.
The SQE is split into two: SQE1 tests functioning legal knowledge, consisting of just over 10 hours of assessment taken over two days. SQE2 is made up of 16 stations – 12 written and four oral – that test legal skills and application of legal knowledge.
To qualify, students also need a degree or equivalent level 6 qualification – such as an apprenticeship – two years of qualifying work experience and to have passed a character and suitability test. The Solicitors Regulation Authority (SRA) said some would now qualify next month.
Of the 726 candidates who took SQE2 in 24 countries, 390 did not need to sit SQE1 because they were either lawyers from other parts of the profession or jurisdictions who had passed the qualified lawyers transfer scheme test or candidates who had passed the legal practice course.
In the first sitting of SQE1 last November, 53% of students passed, enabling them to take SQE2, and a review published yesterday by the SRA on April’s sitting of SQE2 said it was not surprising that the pass rate was higher among.
“Candidates who sit SQE2 will have already demonstrated that they have sufficient functioning legal knowledge to have passed SQE1. We would expect the pass rate for SQE2 to be higher than for SQE1, therefore.”
Those with a higher degree classification achieved a higher pass rate: 92% of those with a first-class degree had a pass rate of 92% compared to 82% of those with a 2:1 and 57% of those with a 2:2. Candidates who did well on SQE2 had typically done well on SQE1.
The pass rate among the 22 apprentices who took it was “well above” the overall pass rate, as it was for SQE1. Those who had completed some qualifying work experience recorded an 80% pass rate.
Women were a little more successful than men – 79% compared to 75% – while 92% of candidates who declared a disability passed.
There was some, but not large, differences in results based on socio-economic background.
Some 85% of white candidates passed, compared to 72% of Asian/Asian British candidates and 53% of Black/Black British candidates, although 92% of candidates in mixed/multi-ethnic groups were successful.
The attainment gap has long been a feature of legal and other professional education, and the SRA has committed to exploring the reasons for this, recently appointing Exeter University to research it.
The review said: “Additional (multivariate) analyses, which looked at multiple demographic categories including education and training alongside protected characteristics and socio-economic background, did not provide any evidence of ethnic bias in the assessment and, as with SQE1, with candidates with better prior educational attainment doing better on SQE2.”
There are a range of measures in place to ensure the fairness of the SQE and a report from Geoff Coombes, the SQE’s independent reviewer, said that while the outcomes “appear to be fair, reliable, valid and defensible, there is a troubling disparity in achievement levels across different ethnic groups”.
The disparity was slightly wider than what was seen in SQE1 and had been “thoroughly investigated” by psychometric experts employed by Kaplan, which runs the SQE for the SRA, and, independently, by the SRA.
Mr Coombes, a former director of qualifications and markets at exam board AQA, said: “These investigations cannot find fault in how the assessments have performed; they have met the assessment objectives and have been demonstrated to be impressively reliable.”
He recounted observing equality, diversity and inclusion training being provided to assessment task writers and assessors about the risks of unconscious bias. “There was no evidence of bias in the administration or conduct of the exam, including in feedback from candidates.”
Mr Coombes continued: “Many of the significant differences in performance by candidate groups suggest the underlying influence of educational factors, notably prior educational achievement. This reinforces the importance of the work that SRA have commissioned with the University of Exeter…
“This issue requires Kaplan’s and SRA’s ongoing close attention, so that any, and every, risk of (unconscious) bias in the assessment process continues to be addressed as far as possible, ahead of future SQE2 deliveries.”
SRA chair Anna Bradley pledged to consider what more the regulator could do after the Exeter research was completed. “We are pleased the first SQE2 sitting went well and that analyses show the assessments were robust, fair and reliable,” she added.
Mr Coombe described the SQE2 exams as “appropriate for a high-stakes, competency-based exam used for professional qualification”.
He explained: “I was reassured and satisfied that the tasks set were valid and that the pass/fail grades awarded were fitting of the standards and competency expected of a newly qualified solicitor.
“I observed good levels of planning and preparation and a great deal of care and attention in setting valid and reliable assessment tasks.”
The review found that, overall, candidates performed better in the advocacy, interview and attendance note and legal research tasks, with the exception of the legal research task in business.
Candidates did not perform as well in the legal drafting, legal writing and case and matter analysis stations.
Leave a Comment