National Assembly for Wales

I'd like to welcome Members to the meeting of the Children, Young People and Education Committee. The public items of this meeting are being broadcast live on Senedd.tv, with all participants joining via video-conference. A Record of Proceedings will be published as usual. Aside from the procedural adaptations related to conducting proceedings remotely, all other Standing Order requirements for committees remain in place. The meeting is bilingual and simultaneous translation from Welsh to English is available. We have Heledd Fychan, who is substituting for Sioned Williams for items 1, 6 and 7. Are there any declarations of interest from Members? I see no declarations of interest. 

First of all, I'd like to say to all those children and young people, hopefully, watching today, 'Congratulations and well done' on putting yourselves forward for those exams. It's been a really difficult time, but a big 'Well done' from all of the committee here to all those children and young people and for the results that they achieved.

In August, we asked stakeholders and the public to submit questions that they wanted Members to put to a panel on their behalf. So, I'd also like to thank all those people who took the time to submit questions to the committee ahead of our session today. Although we won't be able to ask every question that was submitted—I'm sure the panel are pleased with that as well—they have been shared with Members and have all been fed into our briefing material as well. We published them, anonymised, on Twitter and on our webpages yesterday.

So, I'd like to welcome our panel today for our evidence session on the summer exam session 2022. We have Ian Morgan, who is the chief executive of the Welsh Joint Education Committee; Elaine Carlile, director of qualifications, assessment and responsible officer, WJEC; David Jones, chair, Qualifications Wales; Philip Blaker, who is on screen, who is chief executive of Qualifications Wales; and Jo Richards, executive director, regulation, Qualifications Wales. Thank you so much, again, to all of you for joining us for perhaps a longer session than we'd originally planned. So, we'll make sure that we get through all the questions that Members want to ask and those, as I said, that were submitted by members of the public and stakeholders. We'll make a start on those questions; the first set of questions will come from Ken Skates. Ken.

Thanks, Chair. I was just hoping to get from both bodies an overall assessment of how they think the 2022 exam series went and whether there are any striking observations or thoughts from either, perhaps beginning with WJEC.

Morning, everybody, and thanks very much for your time and allowing us to come to meet with you today. I think, overall, the reflection is a positive one. If we look at the last academic year, going back to probably July 2021 and the adaptations that we put in place for this summer's activity, from then, what we've seen is a generally positive experience throughout the academic year. We do need to congratulate the learners in terms of their outcomes, but also the professionalism of the teaching staff, as well, in terms of what they've needed to do to respond to the ever-changing situation.

So, I think in terms of when we've gone through those processes and we've gone through the setting of the exam papers and the assessment materials, again, that's been challenging for us as an organisation in terms of being reflective of the adaptations that have been put in place, because, obviously, we changed the construct, the style and the nature of some of those papers to make sure that we reflected right throughout the process those changes that have been made.

I think, then, as we start to move into that non-examined assessment or coursework activity, schools have responded particularly well to that, and I think learners have responded to the approaches that were taken there. Into the exam series, obviously, we saw, like you, that nervousness in the system, that anxiety and anxiousness for learners, perhaps, because it was the first time for lots of them that they would have sat a formal exam. But, certainly, based on some of the feedback that we've had, which is anecdotal, it's very much that they've responded to it in a really positive way. And early into the exam series, what we were seeing in feedback from headteachers was positive—that it's gone really well, learners were engaged in the process, and, actually, when they came out of the exam, felt a lot better than perhaps they did when they went in, because they'd gone through a cycle, they'd gone through the process.

I think what we've seen through the marking, then, is learners able to respond to the questions that they've been asked, in the main. As you've gone through those processes, you've gone through the awarding processes, the setting of grade boundaries and the application, then, of the guidance from Qualifications Wales in terms of midway, midpoint grading, we've applied those techniques and those approaches, and I think, overall, the outcomes are good. The sense that we've seen and, again, the feedback that we've seen from both learners and from school leaders and teachers is a positive one in terms of the system has treated those learners fairly, and they've got outcomes that are reflective of the hard work that they've done in the last 12 months. And, even in the run up, we need to remember that this is a return to an exam series, but based on two years of significant disruption in terms of the centre-assessed grades and centre-determined grades. So, from our perspective, I think I'm comfortable with what's been done in the last 12 months, I'm comfortable with the outcomes, and I think we've got a good opportunity to move forward now under the new academic year.

David.

Qualifications Wales. 

Good morning, everyone. We're pleased with the invitation to be here this morning. I think I would first of all want to add my congratulations, Chair, to yours to all of the learners and, indeed, the teachers and everybody who's been involved in the education system this summer for the fantastic work they've put in and how hard the young people have had to operate, under really difficult times. I think the backdrop for all of this is really recognising that, since early 2020, these learners, largely young people, have had to deal with really dreadful circumstances. A really definitive time of their life has been taken away from them in their learning. It's affected them all, hasn't it, but, for some of them, who hit this maybe when they were doing their GCSEs, it's gone through the whole of their examination process, whether they did a general qualification or did a vocational qualification.

So, everything that we've done as an organisation at Qualifications Wales has been with the learner in mind and recognising that we can't fix the fact that they've lost their learning. It's just a really unfortunate situation. But, as a regulator, we are able to do things to try and support, with justification, with adaptation, with flexibility, ways of being able to ensure that learners can go on to the next step. I think that's the big driver in the whole thing, whether it's a step within a school, into a college or university. But, in doing that, I think we have to recognise that they have lost their learning and adaptations have had to be made.

I think the whole system has come together with the support of Government and others to hopefully make sure that we don't just say, 'Fine, you've got your results and it's business as usual in terms of the next steps'. I think those next steps are going to continue to have to be really well supported, whether it's additional support in sixth form or in an apprenticeship setting or in a college setting or indeed at university, because there are basically three years of hugely impacted learning loss for these learners, so that's the backdrop for it all. 

But then, specifically responding to your question, and I suppose concurring with Ian largely, we felt that they went really well on the whole. I think we've achieved what we wanted to do. Has it been perfect? No, it hasn't. It's never perfect. There are always issues in any given examination series, and there were a few things that happened along the way—no doubt you'll probably ask us about them shortly—and we've had to deal with them. But I would add that things always happen in any given series, and I think the key thing is to respond in a reasonable and a fair way, and also to use those issues to hopefully inform future developments to hopefully avoid things from happening again in future. I think, as a board, and on behalf of the organisation Qualifications Wales, we're happy with where we're at in terms of what we've achieved for learners, and our focus now is very much on next year and beyond.

Ken.

Thank you. How fair do you think the 2022 series was compared to previous years?

So, I think, from our perspective, we have to take account of the fact that assessment arrangements have changed over the last two years, and we have had in 2022 another unique set of events, and moving back on the journey to the pre-pandemic processes has been part of it. I think, in terms of fairness, what learners have experienced this summer as part of the exam series is a consistent experience. They've all gone in and sat standardised tests with standardised marking and have been graded externally and have been able to progress with, as has been noted, the adaptations and grading policy that were put in place. But, certainly, in terms of fairness, learners will have experienced lots of different things over the past two years, but the direction of travel has been such that it's to promote fairness for them as they move forward.

Might I add as well? There's been support that's been provided through various resources to support that fairness as well, so that students and teachers can access resources to support the adaptations, to understand what they meant, communications regarding what the grading policy would be, and learning resources as well to help the students to do their best, and that would be promoting fairness as well for all of the learners taking qualifications this year.

Lovely. Thanks. And just finally from me, Chair, a direct question to specifically Qualifications Wales. You've got in legislation two principal aims set out—one to meet the needs of learners, and then the second to ensure that there's public confidence in the qualifications system. How do you think you've balanced these two aims over the past year?

I think it's probably fairly important that it has been a balancing act that we've had this year. We feel, and I think certainly reaction and stakeholder feedback to where we've ended up would suggest, that the general view is that people are confident. They had confidence going into the exam series, they knew what was ahead of them. The preparation, as we've talked about, in terms of the adaptations and the support for learners, and quite a significant communications campaign for transparency and understanding of the system, for not only learners, schools, colleges, but also the wider public in terms of higher education, parents, we've done quite a bit of focusing around that. We feel that quite a bit of that has supported the confidence moving into the summer series. And I think, probably, reaction to the results days and the reaction to where results generally, at a national level, have landed, would suggest that there is confidence in returning on the journey back to the pre-pandemic processes and examinations.

Philip, I think, wants to come in as well. Philip.

Thank you. First of all, good morning, everybody, and forgive me for not attending. I've got COVID, so I'm sat in my garden, not feeling terribly well this morning, but still here. I just wanted to note one other thing in addition to the points that Jo raised, which is, sitting below the two principal aims we've got we've got eight matters to which we have to pay due regard, one of which is around comparability in qualifications—the same qualifications—across jurisdictions, and that plays into the public confidence principal aim as well.

One of the things we've done quite carefully is work with our fellow regulators across other UK jurisdictions to make sure that the broad approaches that we're taking are all similar. So, everybody has taken similar approaches broadly in terms of some form of adaptations to the testing arrangements this year, whether it's changes in course coverage, or whether it's in advance information, and everybody has taken a more generous position with grading, albeit taking slightly different approaches in how that's taken place. So, I think it's just worth noting the work that's been done to make sure that there's consistency across the UK, because we want to make sure that Welsh learners aren't disadvantaged relative to their peers elsewhere, especially when competing on the UK market for higher education.

Thank you.

We'll move on to some questions now from Buffy Williams.

Thank you, Chair. Knowing that the position in Wales changed following the decisions taken in England, what was the initial rationale for not giving learners advance information of all qualifications? And are you satisfied that the approach taken to preparing learners for exams was sufficient, including adaptations to content to be assessed? I'll ask that to Qualifications Wales, please.

So, it's probably useful just to reflect on the journey that we took here in Wales. So, in summer 2021, we took the decision, following concern around the loss of learning that had happened in the first year of A-level and GCSE courses, and the likely disruption of their second year of those courses, that there would need to be some adaptations put in place. So, we set the parameters for those adaptations, and WJEC did the detailed work and confirmed those adaptations in the summer of 2021. So, learners in Wales and schools in Wales had that information from summer 2021. The direction of travel, as you noted, in England was that there would be advance information, and that advance information was put in place in February 2022, just before the exam series, those final two terms. So, as Philip has mentioned, there were different varieties of adaptations put in place both here and in England, and I think we felt that early sight for schools and colleges in Wales was important in terms of understanding what the focus of the assessments would be and, in effect, that was what the adaptations were. England, of course, didn’t do that until February 2022, and they came out with, as you know, the advance information. At that point we asked WJEC to review the adaptations and to ensure that there was parity across both England and Wales in terms of the support that had been put in place for those learners, and, as a result of this, there were a small number of subjects where some further information was provided to schools and colleges to support learners.

The adaptations were bespoke to individual subjects, so there were different adaptations depending upon what subject they were looking at. So, some had removal of units, some had streamlined content; there were adaptations to non-exam assessment or coursework et cetera. England in comparison did a blanket advance information approach. I think you could probably look at pros and cons of both of those methods that we took forward. I think it’s probably right to say that, at the time that the England information came out, there was some noise here in Wales about concern that similar wasn’t being done here, but the reason it wasn’t was because, six months prior to that, information had been provided in terms of how the exam series would run.

So, from our perspective, I think we felt that something quite significant had to be done last year, and that the adaptations that were put in place did support learners. As you may well be aware, we have moved from those adaptations into some advance information. So, we announced at the end of the summer term that those adaptations won’t continue forward into 2023, and what will happen in 2023 is that there will be advance information. Part of this is linked to this journey back to pre-pandemic processes. There was significant support put in that year; we need to move forward. It was put in particularly because exams were being sat for the first time—a lot of anxiety, heightened anxiety and concern over that, both from schools, colleges and learners themselves. The exam series, as we know, has passed, and we hope that the exam series runs again next year, but that there will still be some support put in for learners next year. But we need to move on that journey.

Thank you. In making the adaptations, how were reductions in course content decided, and what involvement did teachers and other education practitioners have in informing such decisions?

Shall I take that question? Thank you. So, we obviously looked at the level of disruption for learning and were aware of that. Qualifications Wales provided some principles for us to work towards, so, as Jo mentioned earlier, a bespoke approach was looked at to consider the specification and assessment design as it was and how the adaptations could fit into that, ensuring that key knowledge and skills that were needed for progression were maintained, obviously, because they were really important to go forward, and then looking to see where adaptations could be made. So, we originally, in our approach to this, would have had some focus groups based on some of our senior examiners, perhaps, who would be working as teachers as well, to tease out some ideas as to how they would progress, looking at the design of the qualification and the assessments as they standard were, and then looking to see how we could make those adaptations. So, that formulated our first set of approaches. 

We then provided a document to outline what the approach was for each qualification, based on adaptations to the non-examination assessment, to adaptations to the specification for examined assessments. And we had a survey then; we surveyed all centres in Wales. We provided them with an opportunity with the teachers for each of the qualifications to come back and feed back on the approaches that we were taking. We did all of that in the summer term, and we had some really good feedback from those surveys. Some were very happy with what we had proposed, and therefore our adaptations and proposals progressed.

In other areas, we had some really interesting feedback providing some alternative proposals, which we considered. It was very good feedback that we had, so we incorporated some of that feedback into our final proposals. We then published that information for teachers only to begin with, and provided a rationale for our decisions, providing the feedback that we'd had and the decisions that we'd taken and where we had taken on board some of the suggestions from the teachers.

We also undertook a survey with students and parents as well later on, and we published then our adaptations on our public website in the autumn term, but the teachers had obviously access to those in the summer term to give them an opportunity to consider how their approach to teaching and learning might be affected by the adaptations. 

Thank you. Can you explain briefly how the COVID tariff worked in practice, and why, on balance, you decided against it? Given some schools and some learners were impacted more than others, what, if any, consideration was given to lost teaching and learning time?

Jo.

Okay. To set the context of this, autumn 2021, some schools saw significant absence both of learners and of staff, which clearly impacted on the learning. And there was a difficult Christmas period, and then other schools whose autumn term had maybe gone a little better then started to experience similar issues early in January. So, concerns were raised with us and expressed about this differential loss of learning, and we took the decision that we would consider whether there was anything more that could be done. 

One of the processes that we considered and thought about was what we have talked about in our documentation about the COVID tariff. The COVID tariff, in effect, was a way to give extra marks to students in recognition of time that they had missed. Clearly, when we started considering how we could implement such a COVID tariff, we needed to think about what sort of criteria would need to be in place, and we were clear from the experience, really, of 2020 and 2021 that it needed to be what we referred to as factual criteria, rather than any subjective decisions that would need to be made on a school-by-school basis, because we were very clear that we needed consistency and fairness across the cohort for any sort of process to be put in place. 

And really, after considerable thought about it, the only criteria that could be operationally implemented and would be as fair as it possibly could be would be to look at an individual's attendance and put some sort of criteria around a period missed. If you missed, for example, x weeks, then you would fulfil the criteria and be able to have this COVID tariff potentially applied to you. The issue with that that we considered was that, clearly, you then have, in effect, a line drawn, and some will fall above that line and some will fall below that line. So, there are some fairness issues in drawing any sort of line or criteria in that sense.

Wider than that, there were a number of risks around implementing this sort of process. The fairness aspect was critical, and certainly our priority had been the progression of learners and fairness and consistency for them this year. We were also concerned about the lack of parity across qualifications. This would be something that we would be able to implement for GCSEs, AS and A-levels, but what about wider qualifications that learners would also be taking? We were concerned that, unless there were incredibly detailed criteria, it would be difficult for centres to ensure consistency across Wales. Clearly, it would put more of a burden on centres, and we were mindful of that. Schools and colleges had been under significant pressure in the run-up to this, and it would be, in effect, another administrative process to add to the burden on them.

We were particularly minded that there was something about rewarding absence and what message that potentially sent out there, because we certainly knew, from the stakeholder engagement that we did around this, that there were learners who weren't well, had very difficult circumstances, yet attended their school or college, versus learners where they had dropped out, in effect, of school and college, and there was a balance to be had there as well. We were also minded about comparability across nations. We knew that there was no process akin to this that was going to be used across England or Northern Ireland, and we had to consider, therefore, whether we would disadvantage Welsh learners by putting in place any such process.

When we weighed all of this up, it was talked about that it was some sort of special consideration process. That's what it was commonly referred to by stakeholders. But actually, the special consideration process that runs, and is an established process, is very different to this; it gives extra support and extra marks to students who have an event that affects their performance at the time of exams. What we would be doing here is looking at an event that had happened some time during the lead-up to the exams for them.

Critically, as well, one of the big risks was that we would be rewarding learners, in a sense, for things that they didn't know, by adding extra marks, whereas qualifications are a measure of what you do know. So, we took this to the Qualifications Wales board, we talked about the benefits of this, that people were concerned about this differential loss of learning, and we then talked as well about the risks. And, on balance, the board took the decision that, actually, there were greater risks to doing something like this and that it actually isn't going to necessarily impact on all of those learners that had had significant things happen to them in the two-year course of study that they had done.

Thank you. That's really helpful. Thank you, Jo. Are you happy, Buffy? 

Yes. Thank you, Chair. 

No problem. Now questions from Laura Jones.

Thank you, Chair. I'd like to first join everybody present today in reiterating how wonderful teachers, children and parents have been over the past few years, particularly with these exam results. I just want to ask you about entries. What conclusions do you draw from the number of entries in the 2022 exam series? They are slightly lower than last year but remain higher than 2019 and 2020. Thank you.

The entry patterns do vary from year to year, and what qualifications students wish to study will change. I think what is also important is to understand the age profile of entries. So, there are some changes that happen. We also will see, usually in a standard series—. Let's take A-level as an example, where you have a staged approach. So, you'll have an AS that you take in year 12, and then you go on to do your A2, your A-level units, in year 13. Some students may wish to go back and revisit their AS qualifications to improve on their outcomes, for example. So you may have students who would go back and resit their ASs. Obviously, this year, because we didn't have that AS/A-level structure, and there hasn't been an opportunity for students to take their exams over the last two years, you won't be in a position where students will be taking that unitised approach to their qualifications. 

Additionally, again, centres take different approaches to GCSE and how they organise their teaching and learning. So, for example, English language and English literature GCSE are often taught as a programme over the two or three-year period that people will take to study GCSEs. And in that respect then, it's how they enter their students for the qualifications—whether they choose to do English literature, perhaps, when students are in year 10, or they may choose to do English language at a particular time when the students are ready for it. So, all of those contribute to the changing entry patterns that we could see, and is probably reflective of some of the situations we've seen this year in the entries.

Can I just add to that as well? We've talked about the fact that learners haven't been able to sit formal exams for the last two or three years. The year 10 entry is an opportunity for learners to engage in that formal process before you get to year 11. So, I think you probably see some of those patterns being reflective of an opportunity for somebody to sit a formalised exam before they move on to that kind of concluding activity in year 11. 

As you've obviously both just touched on, data on entries published by Qualifications Wales shows that there has been a large increase in early entries for GCSEs among year 10 learners—a 79 per cent increase between 2020 and 2022—which is mainly accounted for by a sharp rise between 2020 and 2021. This compares to a 3 per cent increase in entries from all learners between 2020 and 2022.

You've already offered some explanations, but does this suggest that schools and learners have taken advantage of the more favourable conditions—no exams in 2021, and generous grade boundaries in 2022—to enter more year 10 learners for exams than they normally would have? How much of a concern is this, given the previous policy intentions to discourage excessive early entries? Would you expect year 10 entries to revert back to pre-pandemic levels in 2023?

There's a lot there. Who wants to start? 

It's difficult to determine what's happening in centres. Obviously, as I say, it is the patterns of the teaching and learning and what their entry policies are. So, it's difficult to speculate on some aspects of that. But I think it's considering what the students have been taught and what they know. I think it's really important that, when the students are ready at a particular point in time, depending on how their teaching has been organised—. I'm aware that some students perhaps begin their GCSE courses a little bit earlier than in year 10 itself, so they could have that two years ready to actually be prepared to do their learning. We've seen some changes, I think, in the approaches to teaching and learning in that respect, so that can account for some of it. We've talked about English language and lit; we talked about GCSE maths and mathematics numeracy as well. And, of course, we'll also have the November series in those core subjects also, which can account for different types of entries. So, some students may take them in the November series, and some may take their qualifications in the summer series. I'll pass over to Jo in case she wants to add any more to that. 

I think what we discouraged is inappropriate early entry. I think the critical thing from our perspective is that learners have covered the course of study and have been prepared fully for the assessments. Clearly, those best placed to take that decision are schools and colleges as the deliverers of qualifications and courses of study. 

Lovely. Thank you. Philip, did you want to come in?

Thank you. It was just to note one other point, really, which is we see early entry largely in the core subjects around English, English literature and maths. I think that reflects the focus that schools have got on those core subjects. But I think it's worth noting that the patterns of early entry have changed significantly over a number of years anyway, and there is a relationship here with accountability measures that Welsh Government puts in place. If we cast our mind back a few years, there were concerns in the system that there were fewer and fewer learners doing English literature. Now there are quite a lot of learners who are doing English literature, and they're doing it in year 10, and there is a correlation with the timing of when accountability measures were changed to allow English literature or English language to count in accountability for schools. Accountability measures have been suspended for the last couple of years because of COVID, but I think some of those cultures can be quite ingrained, so I think it is worth noting that there is a relationship there.

The other thing just to note is that, at the moment, there are two maths; there is maths numeracy and maths. There are two Englishes; there is English language and English literature. And there's Welsh language and Welsh literature, although we see less early entry in Welsh language and literature. In our reforms that we're looking at for the future—for 'Qualified for the future' and the new curriculum—the proposals that we consulted on earlier look to bring that maths back down to one maths and actually to combine English language and literature so that they are assessed as a single subject in the future, and Welsh language and literature as well. So, we have got some changes that we're looking to introduce over the coming years, which again will have some impact in this space.

Thank you. We've seen, in AS-level physics, a drop of 542 entries, from 2,488 in 2021 to 1,946 in 2022. That's 21 per cent, which is the largest decrease of any science subject in Wales. Have you looked at, or will you be looking at, why certain subjects saw such a decrease of entries in 2021 and 2022? Is there any evidence that the pandemic GCSE arrangements put learners off AS physics, or did reintroducing exams have an impact? I'd just like your thoughts on it. Thank you.

Philip, did you want to come in on that one?

Yes. Thank you. I was interested to see this point in the questions that were published last night, so I did a bit of digging yesterday evening on AS physics. If we think about AS as being principally a year 12 qualification, on that stepping stone through to the A-level in year 13, the numbers are actually remarkably stable, year on year, except for one anomalous year, which was last year. So, it's around 1,900 learners. In 2019 there were 1,845, in 2020 there were 1,910, and in 2022 there were 1,880. So, it's all pretty stable around that 1,900 learner mark. What's different this year is that there are fewer retakes in year 13. Obviously, we made the decision that the AS results wouldn't contribute towards A-level results this year, and they would be based purely on the A2 units. So, I think that explains the drop principally—that it's a lack of retakes. That's somewhat masked; there were fewer retakes last year, but there was an anomalously larger number of entries last year. There were 2,365 year 12 entries, so that masked it and made it look like the drop had occurred this year rather than, in effect, that it was a pretty stable position year on year.

Thanks for explaining that. What assessment has been made, or will be made, on the impact of the 2021 arrangements from an equality, diversity and inclusion point of view? Thank you.

As part of our decision-making process, we have completed what we term as regulatory impact assessments, and as part of that there is an equalities assessment done—so, that's pre the decision being made—to support the decision making. Following that, we then will be publishing, at the end of October, equalities analysis of the results. We have a statistical release that will be looked at and published, I think on 20 October. 

That's very helpful. Thank you. Anybody else? Laura. 

Okay, thank you, Chair. 

Thank you. Now questions from James Evans. James. 

Thank you. I seem to get all the nice questions to ask in this format. So, as you said, there has been a three-year gap since the last examination system, so we could say the system may be a little bit rusty, shall we say. So, how confident are you that papers were robustly marked and about how consistently they were marked as well? How confident are you that all that was done?

Does Elaine want to start?

I think we will have confidence that papers are reliably marked. The reason we have that confidence is first of all in the way that we would recruit our examiners and how we go out. We have a very steady group of mainly teachers or those who have been previously teachers who examine, mark our papers. And we've been fortunate that many of those have examined for us over many years and continue to do so. We have a high retention rate of the examiners that mark the papers. So, first of all, they're very experienced. But, you're right, have they been marking over the last two years?

This year, we had a different method of training. So, we moved a lot, as many people have done, to more online training as well, and that gave us opportunities. We had various sessions with our examiners. So, getting them back in and doing some sort of work with them and providing some materials for them in advance of when they would normally go through their training to prepare them and looking at some of the materials that would have been marked from previous series so that they had a feeling for what they had done previously. And then we would have gone through our rigorous training processes, where we go through the mark scheme, explanations from the principal examiner of how to interpret a mark scheme, and then work through marking examples and discuss how marks are awarded for some of the sample scripts based on the live scripts that students would have produced. And they have opportunities to discuss how they've achieved their marks and how they've awarded marks. 

Following that, we would do what we would do in a standard series, then, we continue to quality assure the marking. So, the marks are sampled at various stages. We do an awful lot of our marking now, again, online, and therefore that marking is randomly sampled at various stages to ensure that the markers are marking to the standard that was set at that training conference. They begin by being monitored quite a lot at the outset to make sure they can qualify for marking. So, their marking is looked at and then they are checked to say, 'Yes, that's fine, you can continue on.' But always making sure that they're monitored throughout the process to ensure that they're retaining that standard right to the very end of the marking. So, in that respect, we feel confident that those marks that the students get are reliable and are robust. 

Okay. Thank you. Does anybody else want to come in?

I think, just as the regulator, we would monitor those standardisation meetings and provide feedback reports to WJEC on those meetings as well. 

Okay, thank you. From some of the questions we've received from the public as well, it was perceived that the examination papers were more difficult this year. That could potentially be because people haven't sat exams for a long time and didn't quite know what to expect. That could be the reason why some of the marks were lower, especially in certain areas. It was A-level maths that people thought was a lot harder. What sort of preparation did you do with schools and teachers to make sure that pupils were prepared to take their exams, that they knew what to expect? Because there were some people who had never taken exams before; it was all done by marking throughout the year. I think it's very important that people who've asked these questions get some assurance from you that teachers were prepared properly and pupils were prepared to take these and knew exactly what to expect when they got in the examination halls. [Interruption.] Yes, from the WJEC.

That's fine, yes. So, again, through our professional learning, we made sure that people were aware of what the adaptations were. So, we produced some supporting materials to ensure it was clear for each qualification what those adaptations were. In some areas, where these assessments will have changed a bit more significantly, we produced some additional sample assessment materials, so, again, that the students could have an opportunity to look at what the new layout of the paper might be, particularly if they had an option on questions that they wouldn't have had previously. So, it gave them an opportunity to consider what the layout of that paper would be. So, that helped, I think, to prepare the teachers to prepare the students on what their experience would be like in the exam hall.

We have lots of guidance on how to respond to our questions and our question papers et cetera—that's generic guidance we produce all the time. So, that was available as well to students and to teachers. I think what was really important is that we were clearly outlining the specification and what those assessments would look like and gave the students—. They had a vast amount of past papers to look at if possible. They're published on our website and they're there as well for students. And we have our teachers' guides that were produced as well to guide that teaching throughout. So, we feel that gave the students the preparation that they needed, ensuring that we signposted those. And there was a group of resources that was available also for students through our knowledge 'walk through'—if you want to go through some of those areas, Ian.

Yes, I think what's important is the holistic response to this situation as well. As an exam board, we can provide a whole range of materials and support materials, and we did—we worked closely with Welsh Government on blended learning resources for those areas of subject knowledge and catch-up. We did knowledge organisers for each of those subjects as well. So, it was almost a one- or two-paged document that says to learners, 'Look, overall, these are the key themes and topics you need to be aware of'. So, there's a significant amount of effort in terms of putting support materials and support resources in place. Elaine's already alluded to professional learning that we did and options for teachers to engage in that dialogue and in that discussion. But I think it is right and proper that it's a holistic response. So, schools rightly had undertaken mock exams or test exams or demo exams in order just to give the learners that environment and that opportunity to operate in that way. It may not be in a formal setting as per the exam, but I think, collectively, as I said at the start, how everybody has responded to this has allowed us to have the confidence in terms of where we are as we progress.

So, I think, from our perspective, we did as much as we could. We looked at learner and learner well-being. We've got resources in terms of learner well-being—again, back to that anxiousness and anxiety that was in the system—just to provide as much support as we can. We are one step removed from the direct learner because our interaction is with the school and the college, and it's the school and college that interact with the learner. So, we've tried to do as much as we can in the round for that kind of holistic pastoral support as well.

Yes, because, as I said, a lot of the questions that have come in thought the examination papers were more difficult this year. So, you're confident that you don't think they were and they were quite fair.

If I could come in to say that I think we set out always to make sure our assessments are consistent year on year. We have a whole quality assurance process. So, the first step of that is to make sure that the assessments are matching the specification. And then we also look to see how we look year on year at how the challenges, the difficulty and the demand of the assessment are. So, we have key roles within that process that look at those and we compare questions from previous exams to see if these look to be comparable. I think it's fair to say that we have to assess the specification over a period of time, so we won't be assessing the same aspects of the specification every year. So, we would look at that. So, it may be that some areas of the specification are more challenging than others. That can happen—it depends on how that learning occurs.

So, we have a quality assurance process that looks at making sure our systems are fair and consistent across years. We don't pre-pilot our questions. We don't go out there and test them with a group of learners and then give those assessments to others. You can understand the reason why we wouldn't do that. So, you can only understand if that all works out, from all the people who have looked at the quality assurance processes, when the learners engage with that. And that's the process then where you see that some questions may have become a bit more challenging than were anticipated in the design process. And that comes through when we look, as I said, in the training of our examiners, at how the students have responded to certain questions and say, 'Well, okay, this looks to be a bit more challenging than anticipated. What can we do in the marking scheme—the mark schemes here—in order to ensure that students are given every opportunity and to give credit for what they show they know and what they can do within those assessments?' So, that's our measure that we look at, after that, in our marking scheme to ensure we give as much credit as possible to students that have responded to certain questions in that way.

Okay, that's fine. I'm not picking on WJEC, but I've got another question for you here, and this is about the Shakespearean text on the English A-level paper where four pages were omitted. I know you've addressed that in your letter to us, but you did state in there that processes and procedures have been put in place to make sure that doesn't happen again. Can you outline what they are, please?

We have several quality checks that go through in the production of our printed question papers, and in this instance it is very unfortunate, which we apologise for to the learners and to everybody involved, the quality checks failed to pick up on this collation error on that particular paper. So, immediately when that incident occurred, we went through all the quality checks, and where the quality checks had failed in that instance. So, we walked through the whole end-to-end process to identify how that quality check didn't materialise in spotting the error before it was sent out.

We do have other areas, just to say, where we will have had papers that will have had errors in collation or whatever that would have been picked up through the quality assurance process before they get to the centre. In this case, that didn't happen, so that process has been looked at and has been strengthened. Also, for the individuals that were involved, there's been additional training that's gone in place to make sure that they're completely aware of what needs to happen.

How has it been strengthened?

It's making sure that where you have a process that moves from end to end, you'll have a quality check that happens. It's about the check upon the quality check that happens, so therefore it's that 'second pair of eyes' type of check, et cetera, where you have an additional quality assurance process that's put in place to double-check and treble-check that it has worked out. So, those stages have been put in as additional safety nets to be put in there, and, obviously, the training then as well for those that are involved in that process.

Okay, thank you. I've got another issue I want to raise about the GCSE science paper as well, where you had to save the file under the same name as the existing document that was there. I know Qualifications Wales have said in their review that this probably didn't have any impact on the overall result, but are you all completely satisfied with that—that it didn't have a negative effect on results?

I think we have sought assurance from WJEC around that, and Elaine will probably provide a bit more detail, but, certainly, part of the work that WJEC did was to look at individual candidates to ensure that they weren't disadvantaged by that error. I don't know whether Elaine wants to give the details.

So, when any issue will come up within an assessment, we will look to every student's exam paper and analyse their responses to all the questions in the paper, particularly those that would be assessing similar skills and knowledge, and analyse their performance on each question to see how they may or may not have been impacted by the issue. I think, in that particular case, there would be one mark or two marks that would have been affected by students, so therefore we identified questions assessing similar skills and looked to see how the students performed and also performed in the rest of that question, because it was a build-up type of question. In that, we found most students, because of their instinctive nature, would save files in certain ways as part of the process anyway, so many of them responded very well to that question.

We looked, then, at a small number where we think perhaps maybe they didn't perform as well, based on their performance in other questions, and therefore we looked to adjust their marks in that way. And there were very few in that bracket, but we did look at those as well and made those adjustments. We communicated that to the centres, and, so far, we haven't had any feedback from students on that one  at this stage, or from the teachers.

Okay, thank you. And my final question is to Qualifications Wales. Was the number of incidents that awarding bodies reported to you—I think it's around 90 or so—what you were expecting?

It's difficult to know what we were expecting. I think one of the risks that we identified before the exam series started was that we could see a greater number of incidents this year, so not just WJEC, but across all awarding bodies, in the sense that there could be security risks where people were unfamiliar with the processes that they needed to go through in terms of delivering the exam series. But, in comparison to previous years, there was no significant difference in terms of incidents reported. So, we didn't have any particular concerns.

I think, in fact, we were very pleased with the delivery within schools and colleges, so there are often errors that are made, where the wrong packet of exam papers can accidentally be opened, but we didn't see significant numbers of those this year. That, of course, is to the credit of the examinations officers in schools and colleges.

Do WJEC want to comment on this at all?

Similarly, I think, as you say, you will have incidents that occur during a big exam series of the magnitude that we deliver every single time. I think what's important is that we'll all have our risk registers and have our plans, mitigations and controls in the event that a risk becomes an issue, so we implement those quite quickly, and, again, work with the schools and colleges, the exam officers, the teachers and headteachers, and engage with those, as well, throughout the process, and they've been marvellous.

Thank you very much. Diolch.

Thank you, James. Just quickly, if I can go back to the year 10 entries and what Philip was saying earlier about school performance measures, can you clarify, Philip, if that was likely to be a combination of the Welsh Government's suspension of performance measures and the favourable conditions—no exams in 2021 and generous grade behaviour in 2022—that has resulted in a vastly higher number of early entries?

I think my point is it's shifted over time anyway, so it's difficult to attribute anything to any particular factor. Schools have been adapting to have more year 10 entries over a period of time. Indeed, we've seen more of a shift towards schools delivering key stage 4, or the qualification phase, so to speak, over three years rather than two years, as a shift in the way that teaching and learning are happening. So, I think there are lots of forces at play here, and I wouldn't attribute it to gaming of the system for a year that has got more generous arrangements; there are lots of things at play.

Thank you, Philip. That's really helpful, I appreciate you clarifying that. Just a few questions on the broad midway point between 2019 and 2021. We've talked about and we've mentioned confidence in the system, promoting fairness and consistent experiences, how satisfied are you that the examination system in 2022 delivered that policy aim of results being broadly midway between 2019 and 2021, because we know it's not really a perfect midway system? Perhaps thinking about next year, really, and concerns that you might have about a potential cliff edge next year, rather than what we were thinking of this year.

I think, from our perspective, overall, the broadly midway policy was delivered this year. We did, in our requirements, when we set them out, want to err on the side of generosity. We deliberately didn't talk about a midpoint, because actually it's quite difficult to achieve that, if for nothing else. But, actually, particularly if you look at A-level, there was quite a jump in outcomes between 2019 and 2021, and this really was about the start of the journey back to the pre-pandemic levels and processes.

Last year, when we announced the grading policy for summer 2022, we talked about the fact that this wasn't something that could be done in one year, the whole cliff edge, and that we expected that we would move: in 2022, we would go to this midway point, and to look forward then for 2023. We committed at that point that, after the summer series of 2022, we would consider where we'd ended up and, therefore, where we went in 2023. We have currently looked at a bit of an options analysis for the grading for 2023, and I don't know whether David wants to say something. We have a board meeting tomorrow to discuss that.

Yes. It's clearly a really important decision, as Jo says. In the same way, I was just reflecting, to go back briefly, we talked about the issue of the COVID tariff, and Jo's answered that, but, just to reassure this committee that, at Qualifications Wales, there was a huge amount of work that went into that, looking at the options. It wasn't just dismissed in any way, and it was something that the board were involved in. 

But, getting back to this point, yes, we have a paper at tomorrow's board meeting, which looks at the various options for 2023 with an eye beyond that as well. So, clearly, that decision hasn't been made, it'll be a board decision. In making that decision, we're really going to have to still consider the situation now and what might happen. We haven't got a crystal ball, but, equally, let's think about it; COVID is still around, Philip's away at the moment because of COVID, and I know it's an issue in schools and colleges at the moment. There are potential issues around industrial action across the public sector, including schools, and potential issues linked to the cost of fuel, energy and other challenges. I think there's a challenging year ahead for all of us, but not least those involved in schools and colleges. So, that's something we've got to bear in mind.

And also, very importantly, those learners in the system who are, particularly, say, a year 13 learner, which I was referring to right at the start—this year, the ones who are finishing their A-levels—they've had a pretty difficult experience, haven't they? They've lost a lot and they haven't got back to the point where it's a steady state for them. So, I think we have to really take on board, 'Are there things we need to do there?' I know that we've made some adaptations already. But also, I think, even if you look behind them—the learners who are now in year 12, 11, 10 and so on— they've all had challenges. So, I think, in looking at it, we'd need to take all of those issues on board. It's an interesting concept, fairness, but that is very much what we're about. We've got to try and find the best option for learners in Wales. I think that's it: we will make a decision based on learners in Wales and the best for them, taking on board all of those issues.

Thank you. Philip.

Thank you. I just wanted to add a couple of points, really. One is around comparability, as we've already touched on. In England, a decision was made to weight this broadly midway position towards 2021 outcomes to make sure that it was generous to that position and couldn't seem to be a harsh award. That was quite a late decision to shift that weighting, and we followed that decision because, as I said previously, we didn't want learners in Wales to be disadvantaged relative to their peers elsewhere. 

What that means in effect is that, especially at A-level, it has probably only made about a third of the correction back to the pre-pandemic standard, which means there are two thirds of the distance still to make, which does seem like a lot to make in one year, which is one of the considerations that we're putting to the board tomorrow for its decision. So, just really to give that sort of real position—that we see there's still quite a way to go to recover that pre-pandemic performance standard.

Okay, thank you. You've touched on the A-levels in particular, but was the process of seeking a broadly midway point between 2019 and 2021 also applied to the skills challenge certificate grades?

It was. The skills challenge certificate, as you're aware, is a unitised qualification, skills-based, which doesn't have any external exams as part of it. Some of those units are taken earlier on in the year, so there's a January series as well. So, we require WJEC to have a broadly midway position for those two qualifications at key stage 4 and at advanced level. I think if you look at the outcomes of them, they are—well, broadly midway would probably be a slight exaggeration, they're probably the upper end of broadly midway. But, we were content, given the parameters of the design of the qualification, around where those results finally ended up.

Okay. That's really helpful, thank you. Just thinking about the future, really, or the generation that's just been through all this COVID. We've heard all the things that they have been through and the challenges they've had. Do you think that they're at an advantage or disadvantage in terms of the qualifications they've obtained, and how do you feel, in terms of their future prospects, really, and any further study that they tend to do?

I think there are probably two schools of thought around this. Learners have had special arrangements put in place over the last three years to support them and to enable progression, and some may claim that that has advantaged them. But I think the other side of that is that there are some gaps in the knowledge, understanding and skills of those learners, because of the impact of the pandemic on teaching and learning. And, as David touched on right at the beginning, there needs to be the whole-system support in order that they are supported as they move through. They have been able to progress, because of the various different assessment arrangements that have been in place, but that still doesn't mean that there is no impact. I don't think we would necessarily talk about them being either advantaged or disadvantaged. The pandemic has clearly disadvantaged them in life, but I think, in terms of the education system and the exam system and qualifications, there have been these exceptional arrangements put in place in order to support them, to allow them to progress, moving forward. 

Thank you. Philip.

Just to add to what Jo said, I think we've got to remember that the pandemic has affected everybody across the whole of the education system—so, right the way through to higher education as well, and university courses—which means there's an understanding that COVID will have had an impact. But everybody has been focused on allowing progression for those learners so that their learning journey, or journey on to employment, isn't interrupted by COVID. What that essentially means for me is that it's that progression step that they're making at that particular point—so, at 16, when they're progressing on to sixth form to do A-levels, colleges to do A-levels, or vocational qualifications, or at 18, progressing on to university. And whilst there's always a mix, particularly at that progression point on to university, of people of different ages, people are principally competing with people of their own age who have taken exams, or been assessed, or had centre-assessed grades, at the same time. So, everybody's in the same boat.

I think the thing for me is it's that first progression step after they've received their results, and making sure that there's some consistency in approaches there. We've seen that progression hasn't been adversely affected, learners have been able to make that progression step on. We've seen that there have been higher numbers of applicants to universities. I think the thing that we'll probably want to see is just what does that mean in reality for those learners who have progressed on. How do they cope with the courses that they've taken? And to what extent is there dropout from university? Do people fail to complete their FE courses? I think the system needs to look at some of that data, as it starts to become available.

Thank you, Philip. Just to move on to some final questions now from Sioned Williams.

So, as we've touched on, the board are taking the decision about grading for 2022-23, which would include November 2022, January 2023 and summer 2023. That decision is being taken tomorrow and will be communicated shortly, in terms of the grading position. We have reflected on summer 2022 and reflected on the ongoing experiences as well, as part of the options analysis that we put forward to our board. So, I don't think that there is much more that I can probably say about that, but, certainly, considerations are being taken, and we very much are viewing this about our journey back, and being cognisant of the context of the year that we are moving into as well.  

So, we have taken the decision at the end of summer 2022, when we announced that the adaptations wouldn't move forward into winter 2022 and summer 2023. I think that the context of the November series—. Here in Wales, the November series is used both as an early entry and a resitting series. Clearly, whatever decision is made about the way that those assessments are for November has to then be reflected throughout the rest of the year—otherwise, there is an unfairness on that year's cohort.

When you look at entry numbers for the November series, here in Wales, predominantly it is early entry candidates. Therefore, they are taking the opportunity, if they have studied the course and they are ready to sit it in November. But they do have another opportunity to sit it in the summer, at the end of their year 11, year 12, year 13. So, our focus was on the predominant age group taking it. Therefore, the decision that was taken about November reflected where we would be next summer.

Could I add to that? I think we spoke earlier about advance information or adaptations, and what are the benefits or disbenefits of the approaches. So, I think that it's fair to say, for the adaptations, mainly, it was about saying that these things would not be subject to assessment in the series, whereas with the advance information, it's saying, 'This is more the focus of the assessment.' Therefore, for the November series, the advance information is already published—it has been published on our website. So, it gives people the opportunity to consider what the focus of some of those assessments and those questions will be. Therefore, the students shouldn't be that disadvantaged. So, hopefully they'll get some assurances from that advance information that's there. It's just how those two balance out against each other, but they are given that support for the November series.

Thank you. That's the end of our evidence session today. Thank you very much for joining us. It's been a really informative session, I think, and we'll obviously be watching for some of the decisions that come out of your board meeting tomorrow. Obviously, the committee keeps a very keen interest in all of this, so we’ll be following the work that you're doing on it. I'd also like to thank Philip in particular for joining us from home, particularly when he hasn’t been feeling very well. So, a speedy recovery, Philip, and thank you very much for joining us. Diolch yn fawr. You'll be sent a transcript, as well, of today, so diolch yn fawr.

Okay, we'll move on item 4 on our agenda, which is papers to note. As you'll see, Members, we've got a number of papers to note. There are 17 papers to note. Full details are set out on the agenda and in the paper pack. Perhaps we can take them all together because there are so many. Is everybody content to note those letters? Yes, I see everybody's content.

So, as agreed at our meeting on 14 July, the committee agreed a motion under Standing Order 17.42(ix) to exclude the public from the rest of today's meeting.