National Assembly for Wales

Morning, everyone. Welcome to the Children, Young People and Education Committee. We've received apologies for absence from Dawn Bowden, and there is no substitute, and we've been advised that Michelle Brown is running late. Can I ask Members whether there are any declarations of interest, please? No. Okay. Thank you.

We'll go on then to item 2, which is our scrutiny of Qualifications Wales annual report 2017-18. I'm really pleased to welcome Ann Evans, chair of Qualifications Wales, and Philip Blaker, who is the chief executive of Qualifications Wales. Thank you for attending, and we're looking forward to hearing what you've got to say. So, if you're happy, we'll go straight into questions. If I can just ask a general question about what you feel the biggest challenges and the biggest strengths have been in this reporting period.

Okay, I'll kick off with that because I think that, over the year, we've continued with our innovative approach to looking at qualifications within sectors. So, in health and social care, we're now getting to the implementation phase; in construction of the built environment, we're just looking at the qualifications; and we're about to publish information and communications technology. I think where we're coming from, and one of our biggest challenges through the year that we discovered through the work with health and social care and childcare, is building that consensus around what the qualifications should contain. And that has taken time, it's taken a lot of hard work, and it's bringing people together to really thrash out what was needed. So, we would identify that as one of our biggest challenges that has happened this year.

In terms of successes and the way in which we think the year has gone, at the very start, we really were determined to be an evidence-led organisation, and we've become more experienced at gathering evidence and using that evidence. So, for example, the work we published on early and multiple entry GCSEs, which was back at the beginning of the year and which was welcomed by Welsh Government, led to a change in performance measures. Our research programme has also grown, and we've also taken on the capacity to publish statistical data. Prior to that date, that was done by England on our behalf. So, we can now do this in Wales ourselves. So, we think that's one of our biggest successes because we've been able, if you like, to gather the evidence. We've got better at doing it. We're three years in. We've learned a great deal over the last three years, and so we feel that's one of our biggest successes over 2017-18.

Okay. Thank you. We've got some questions now from Julie Morgan.

Yes. Thank you. In your paper, you highlight four key challenges. Could you tell us what action you've taken to address those challenges?

Yes, certainly. Particularly in relation to general qualifications, so GCSEs and A-levels, we identified some challenges, the first of which was moving to more of a devolved system, so where qualifications are developed in Wales for Welsh learners and are slightly different to those that are taken in other jurisdictions. When you do that, one of the most important things is to try and make sure that those qualifications are trusted and that they provide the currency that young people need to progress into further study and into employment. And whilst we've stuck with the same brands of GCSE and A-level, it's important that the differences are understood and that there isn't any devaluation of the currency that goes with those brands as a result of some difference.

I think at the last committee, the last scrutiny session, we talked about the fact that we were planning to employ an HE engagement officer—so, somebody who could go out and talk to universities. We touched on this when we gave evidence around the Welsh baccalaureate as well. So, we do have somebody now who's going out and talking to universities, talking to sixth-form colleges, talking to schools, understanding the issues that they might face in terms of any perceptions that universities might have. And also, bringing intelligence back to us so that we understand what the issues might be, if there are issues that universities have.

We've done less with employers; we're actually moving on to employers more in the next year. We engage with organisations like the Confederation of British Industry, the Institute of Directors, the Federation of Small Businesses; we're looking to bring some of those bodies together more next year to try and find more ways of engaging with people. But, of course, one of the things is where the grades have remained the same with GCSE, there is one less difference. So, A* to G being the grades rather than 9 to 1, as they are in England. Employers are more familiar with that grade scale.

Other areas we've been looking at are things like the availability of resources for teachers, and the committee has had a scrutiny session around that. We're playing a part in Welsh Government's strategic stakeholder group there. Of course, the last round of reforms have pretty much gone through now; we've got the last tranche of first assessments next year, in June. So, really, we're looking at resources now more for the next tranche of changes that will be ready for the new curriculum.

The other area with difference is around just the range of qualifications that are available. We know that the range of qualifications that have been developed in a Wales-only specification by WJEC don't cover the full range of qualifications that young people might want to take. So, there are low-uptake subjects that are simply not viable—either economically or technically—for WJEC to develop. In those cases, we've allowed the England specification to be used in Wales. One of the limitations there is that they're not always available—in fact, they're most frequently not available—through the medium of Welsh. So, what we want to do is to try and make grants available and encourage those awarding bodies to make them available through the medium of Welsh, if we possibly can.

Managing the introduction of the new qualifications has been a big thing as well. We've done quite a lot of work looking at what lessons we might learn from the introduction of these new qualifications. It's one of the things—. It's about thinking about the fact that the new curriculum's going to lead to more change, so there's a real opportunity to learn lessons now that can be employed almost immediately, really, as we start to think ahead. So, we've done quite a bit of work reviewing new qualifications, thinking about how they've been implemented.

I'm just going to run through some of the key features that we've identified. There's clearly a need for early engagement. The last round of reforms were driven by an agenda in England; they were driven to a timescale that was responding to changes in England, and where Wales had made the choice to do something different, there was little opportunity but to go along with the same timelines. Moving forward, now that we've made that step away and there is a more devolved system, it gives us more agency over the future. So, we've been looking at early engagement and the need to communicate with schools so that schools really understand what's coming, and, clearly, the new curriculum provides a platform for that.

Thinking about what the design features will be and making sure there's a clear understanding in schools about what the design features will be for the new qualifications. Making sure there's plenty of lead-in time, and we've done a fair amount of consultation on this. Clearly, schools need enough time to prepare for new teaching, and there needs to be enough time for resources to be developed—especially if those are textbooks—and, again, this is something we've talked to the committee about before. So, we're looking at at least 12 months advance notice of a finalisation of specifications and the like, and there's a clear need for that.

Changing the structure of the general qualifications market, so moving to a position where WJEC are the sole providers of the vast majority of GCSEs and A-levels in Wales is a big change. That can mean a change in the perception of people. WJEC had about 89 per cent of the market before these changes came through anyway. We know one of the main reasons why people will move from one awarding body to another—and this is looking at evidence that Ofqual looked at in England where it's looking at a competitive market. The most frequent reason is where a head of department moves from one school to another school and they take the awarding body that they're used to delivering with them. Now, in Wales, where 89 per cent of the market was WJEC's anyway, it provides a more limited pool, because if a head of department moves from one school to another, they will be more likely to be taking WJEC with them. So, there was less churn, so to speak, within that environment.

Now, the move away from that being a matter of choice to it being something that is there by default—it's not by design, it's there by default—changes people's perceptions. So, what we need to do is make sure that we are holding WJEC to account and having close scrutiny of their activities to make sure that they are doing the right thing.

Lastly, thinking about the new curriculum, clearly, the new curriculum is going to lead to some changes to GCSEs. We're working very closely with the areas of learning and experience. We strongly believe that the curriculum needs to clear the way for qualifications, so we don't want qualifications to be defining what the curriculum is. So, we've given space for the curriculum development, we've been watching very closely what's been going on and we're looking, really, to follow in the spirit of working with pioneer schools in the spirit of co-construction, so taking that same ethos forward as we start to look at qualifications, and we think that we're going to be in a position to start doing that in a more concerted way from the new year, as the schools that have been involved in developing the curriculum are starting to think about what's next.

Thank you for those comprehensive answers to those four points. A theme throughout what you say is the relationship with English qualifications, and I just wondered if you could say what is your relationship with equivalent people in England. I think you mentioned earlier on the issues about 1, 2, 3 and A, B, C. How do you discuss those issues?

We've got a very good working relationship with Ofqual, our counterpart in England. There are three-country regulatory bodies, as in groups, where we come together. And three-country there—. There's a closer relationship between Northern Ireland, England and Wales, because of using the same qualifications, using GCSEs and A-levels. It naturally leads to a much closer relationship. There are four-country meetings including Scotland, but where Scotland has got a very different system, I guess the working is much closer around those brands of GCSE and A-level. And you'd expect that, because the regulators are all the owners of the brand, so we have an interest in making sure that the qualifications are functioning properly, and similar.

Our working relationship with Ofqual has got better and better, I would say. It's strengthened considerably. I think, as a new organisation, you have to credentialise yourself; you have to demonstrate that you properly appear to bodies like Ofqual that have been established longer. We've moved to a position where we're sharing much more information. We're trying to work in the spirit of having some efficiency from each other's work as well. So, where we might be investigating one thing, like we've looked at first aid, we're currently looking at food safety qualifications, we'll share the outcomes of those with Ofqual and we're sharing plans for audits of awarding bodies and the like so that we don't place too much burden on awarding bodies by all following the same topic.

Thanks. I wanted to ask about—. I think you referred to public confidence, you know, part of your role is to get public confidence and you've referred to the fact that you have an engagement officer now. How long has that person been in post?

Since September, so relatively new.

Very recently, right. So, you haven't been able to judge what progress has been made yet, then, it's too early.

No. We get useful intelligence coming back. Inevitably, when there's change, there's a bit of uncertainty around the period of change. As changes have bedded in, we're getting much more reassurance coming back to us that things have settled. So, I'll mention something—we talked about the Welsh bac a couple of weeks ago. There was much more of a sense coming back from our liaison officer that there was more confidence that universities were accepting—. Schools were feeling more confident about acceptance of the Welsh bac. So, I think there's a really useful feedback loop that we get that suggests things are quite positive.

Right, and how do you feel about the fact that 45 per cent of the pupils and teachers felt that exams were too difficult, or that 40 per cent felt there was not sufficient time for the exams?

I thought that was about right. We would always be a bit concerned if people felt it was too easy and if there was too much time. I think, from that, that's looking at our survey which is self-elected as well, so it's not a representative sample, i.e. it's not a designed sample.

People offer themselves?

They offer themselves forward, so it's a self-selecting sample. And in that self-selecting sample, it's roughly where it was last year in terms of about half thinking there was enough time, half thinking there wasn't quite enough time—

And, actually, you would expect that to be—

We would expect that.

Right, so what about—? Fewer than half of the respondents felt that standards in GCSEs and A-levels were being maintained year on year.

That was from a different survey. We do three exercises at the moment that are deliberately around looking at public confidence, and we use external bodies to do that. So, we do two surveys and we've done those for the first time this year, and those are using pretty much the same questions as Ofqual uses in its public confidence survey, the main reason being we can then draw direct parallels with England around confidence in GCSEs and A-levels in England.

So, we do two surveys along those lines. Those are of about 1,000 people. It is a representative sample of the general population in Wales, so it's people who will either be close or not close to the qualification. So, it's not designed to be parents of learners or learners or teachers—it's a general population survey.

The other one we do is with York Consulting which is more of a focus group-type activity, which will tend to have people who are closer to the system. It's interesting, there are two questions that seem to get relatively low levels of agreement. One is around whether standards are being maintained and the other is around whether marking is accurate. England gets very similar responses, so there's no real difference in the percentages that England gets.

We think it gets a low response rate because they're quite technical issues. So, things like Maintenance of standards—there are lots of different conceptualisations of what a standard is, and we know that teachers get confused about standards in relation to qualifications as well. So, we think the low level of agreement is in large part because it's quite a technical area.

I think it would be fair to say as well, if you looked at the survey results, it's only a small percentage that actually disagree. So, there's a lot of 'don't know's in those survey results.

So, you don't feel concerned about that at all.

Not particularly. I think also, it's that point of—England, who's been doing the same survey for much longer, gets similar results. 

Right, and what about the fact that only 52 per cent of the public who took part in the face-to-face interviews feel that GCSEs are a good preparation for work, or that only 44 per cent feel the same for A/AS-levels? Does that indicate that they should be more work orientated?

Do you want me to kick off? I'll kick off with this one. I think it's quite interesting when you start to talk to employers about what they expect from young people coming out of the education system. And when you ask them what do they want, they're very clear; they say they want literacy, numeracy, communication skills, team working—all of those soft skills. Within Wales, we've already changed our GCSEs in English, maths and Welsh to make them a better guarantee of the skills employers say they want. We've got the skills challenge certificate within the Welsh bac, which directly assesses those broader, softer skills that I was just talking about.

There's also space within our system for GCSEs that are a better reflection of the world of work, and that's why, within health and social care, we're looking at a GCSE and an A-level that will support those developments. We're looking at similar things for construction and the built environment, and for ICT. And with the new curriculum, there may well be more opportunities to look at this, so it's something we're very alive to and alert to, and looking carefully at how we can make our young people best prepared to move into the work of work.

Okay, thank you—

Yes, thank you very much.

—we've got some questions now about marking. If I can just kick off and let you know that the committee's been made aware of the case of three pupils of Y Pant comprehensive who had their grades changed after they'd received their marks, as a result of WJEC errors, and that this caused a lot of distress to those pupils. Are you aware of that incident, and what action have you taken to address it?

Can I just kick off and say we want everyone to get the right results from the outset? And we absolutely recognise the distress that an incorrect result causes pupils, their parents, and their teachers. It is a big and complex system, and, fortunately, errors are few. But I think, Philip, you've got detail on that.

Yes. So, we were made aware of this incident, through being copied into correspondence to WJEC. So, our normal way of handling complaints of this nature is to let things run their natural course through WJEC, and then allow a complaint to come in to us if the complainant is dissatisfied with the service they've received from WJEC. So, we were made aware of this. There were some errors by the original marker, I believe. I'm aware of the detail of one particular part of this, which relates to two candidates, where two candidates with the same surname had their marks transposed, so they both got the wrong mark, each other's mark, when the mark was copied from the script onto the system that captures marks.

Yes. I mean, I think what the committee's interested in, really, is how you handle the issue of—I think they're called 'adverse incidents', aren't they?

Absolutely. So, I think in this particular case, we've got to remember that WJEC handles something like 4 million individual assessments for learners in Wales. There are mishaps. Any system that involves human judgment—. And that's where reviews of marking come in, and there are relatively few changes of grade associated with that element of marker judgment. There are even fewer changes as a result of this sort of error where there's essentially an administrative error of either marks not being included or marks being transposed in mark sheets. Normally, any errors like that will be captured by the quality assurance systems at WJEC before results are issued. I believe, in this case, the scripts weren't returned to WJEC before results were issued. So, those additional checks didn't take place. When WJEC were made aware of the issues, they responded to them appropriately, I think. They looked at the impact and they actually checked with us their proposed actions, to make sure that we were comfortable with their proposed actions, and they corrected the grades for the candidates, which is what we would have expected them to do. So, I think there's—. We have to look at it in terms of: was there a mishap in the system, or was there a systemic failure? We wouldn't see what happened in this particular instance, however unfortunate it was, as a systemic failure; we would see it as a mishap on the part of the examiner.

Okay, thank you. The next questions are from Siân Gwenllian.

Okay, bore da. We've touched on the lack of public confidence in qualifications in Wales, and it is a mixed picture that we're seeing coming through. To what extent do you think incidents of mismarking contribute to this, with only 32 per cent of the survey respondents agreeing that marking for AS and A-levels was accurate? So, 68 per cent think that the marking isn't accurate.

So, I think, as I've touched on, it's the same with Ofqual's surveys; these are the two areas. Part of the survey results, we think—it's got a large number of people who don't know or neither agree nor disagree about the accuracy of marking. So, there's a large number of people who don't say, who don't state an opinion. What we've got to look at is the facts of things like reviews of marking. So, there's just about a million GCSEs sat in Wales and just under 200,000 A-levels sat in Wales. Of those, I think it's 2.5 per cent—we don't have data for this year yet; data for this year will be published as official statistics before Christmas, so I'll relate back to last summer's figures—I think it's 2.5 per cent of GCSEs get put forward for reviews of marking, where the marking is checked. About 2.8 per cent of A-levels are put forward for reviews of marking, and the number, or the percentage, of grades that are changed is 0.5 per cent. So, if we're looking at a system that is based on human judgment, which is the application of the mark schemes, with an error rate of 0.5 per cent, I think that's pretty good, and I think that's actually something that the system should be proud of. And I'm cautious in saying that, because you'd want it to be 100 per cent accurate—you'd always want it to be 100 per cent accurate, and you would want WJEC and all of the awarding bodies to be doing everything that they possibly can to minimise the possibility of somebody getting the wrong grade.

But there seems to be a perception—. I mean, you're explaining away—that it may not actually be a particular problem, but there is an obvious perception that is leading to a lack of public confidence. 

Indeed.

So, that is a challenge. 

And we think that's something that we can play a role in in communicating about the system. The difficulty always, with these omnibus surveys of people—I think the surveys were conducted in September—is that they'll go out and they'll reflect, often, what somebody saw in the press quite recently, and at around that time, they're more likely to see problems that are being raised. So, public perception is quite a fragile thing, but we think there is something to be done in trying to give people confidence through communication. 

Can I turn now to the—?

If you're moving on to something else—Janet, did you have a supplementary? Is it on this? 

Do you know what, I've been visiting schools in my own constituency and this has been raised with me by headteachers? So, it's not perception. 

The accuracy of marking? And are they raising reviews of marking? Are they querying the grades with the WJEC?

Very much so, yes. 

Okay. We'll look at that with interest to see what this summer's figures look like. 

Okay, thank you.

You may be referring, in part, to what I'm going to talk about next, which is the perceived injustice that 700 pupils have around the English language GCSE results of August last year. You'll be very aware of all this, I know. Can you, first of all, explain the rationale for changing the C-grade boundary for GCSE English language three times between summer 2017 and summer 2018?

Okay. I think, just to start off, I want to make very clear what our role is in all of this, which is to ensure that performance standards required to gain a particular grade are maintained year on year. That is to prevent grade inflation, but it's also a matter of fundamental fairness for pupils—past pupils, present pupils and future pupils—so that nobody is being advantaged or disadvantaged in this system. And so, I think Philip was going to go on and talk about the specifics of the GCSE—English language—but we have to talk about the system as well, and the way in which we operate.

Yes. So, the principle reason why there have been three changes in grade boundaries is there have been three exams. Question papers are different each time they're sat. Each time they're sat, despite great efforts to make sure that the demand is equal from year-to-year so that there's parity, the actual difficulty of the papers will vary slightly each time. For something like English language, that can be related to the nature of the reading texts—so the texts and the reading comprehension questions. It can be related to the topics that young people are asked to write about. So, each paper will perform slightly differently. 

Now, if you want to maintain a common performance standard when the difficulty of the paper changes each time, the way of calibrating for that is by moving the grade boundaries. So, if a paper is slightly easier than the last paper, then you would expect to see the grade boundary go up slightly to maintain the same performance standard. So, what we do as the regulator is set the method by which WJEC conducts its awarding meetings. So, WJEC established the grade boundaries, not us. We prescribe the method by which they should do that. And the method that we ask them to use is commonly known as 'comparable outcomes', and that has an assumption that, all things being equal, you would expect broadly the same number of pupils to get the same grade profile, so the same number of grade Cs and above from one year to the next.

Now, all things are never equal. The thing that's been most volatile over the last few years in Wales has been the nature of the cohort—so, whether there have been lots of year 10s for early entry. And, this summer, there's been a big impact through the washback of that early entry. So, you're seeing lots of candidates who maybe gained a grade C in English language early not come back in the summer. So, you've got a very different cohort that's gone through this summer.

I think the problem stems from the fact that some schools have been following Government advice and not putting in their pupils for the early entrance exams and actually keeping to what Government is asking them to do, but they have been caught out in that because that has led then to that cohort of children being disadvantaged. That is what I'm being told. Now, it has obviously created a huge problem. You can try and explain it all, yes, and I'm not in a position to actually judge the nitty-gritty of it, but I do know that it has created a huge uproar in north Wales in particular, and I have here a note from the education leaders in north Wales, saying that there was a meeting that you convened last night between officers from Qualifications Wales and the directors of education across the six authorities and representative headteachers across north Wales who wanted to further debate the 'injustice' that over 700 learners had received in English language GCSE results in August 2018.

They say that it was extremely disappointing that you didn't choose to attend that meeting, and they go on to say that confidence has been lost in your organisation to effectively regulate the examination system in Wales. Now, that is a huge thing to be saying, and that is across north Wales. So, there is obviously a problem of confidence, and we need to be assured that you will be working to regain that confidence, and maybe going to the meeting last night would have helped slightly, but, anyway, that's up to you. And they also want assurances that what happened this year won't be repeated again—you know, that early entry and following guidance from Government around that will be respected and that it won't result in some pupils losing out.

Can I respond to that?

Of course.

First of all, in terms of my attendance, when we established the date—. We wanted to see colleagues in north Wales as soon we could. Yesterday was the first day that was available, but then we had strong feedback that they wanted a twilight session at the end of the day rather than a session during the day. I then had to make a choice of either go to north Wales or come and see you this morning, and given the fact that this is a scrutiny session of the committee, it was safest to come here this morning. My colleagues who went yesterday I have absolute confidence in. They know the issue inside out—my director of regulation, director for general qualifications and head of research. So, I did send a senior team to go and meet colleagues in north Wales.

In terms of confidence in us, I think it's really important—. You know, we are a body for you for you, as the National Assembly, and you've established us as an independent regulator for qualifications and with a particular focus on standards, which we necessarily need to have. Not everybody will welcome the message that we can find no evidence that the award was made incorrectly. It is an inconsistent pattern across Wales. North Wales, as you say, seems to have been hit hardest in results this summer. That's not a consistent pattern, and our report shows that. There has been some difference in relation to those schools that have entered early and those that haven't entered early. Equally, there have been schools that have entered early that have done much better, and schools that have done worse. Equally, among those that haven't entered early, there have been schools that have done much better and schools that have done worse. So, there's no consistency in terms of performance. We don't see that when we look at national data. I'll be frank, if there were any evidence that we had seen through any of our investigations that the award had been incorrect, then we would have taken action—and we still will, if we uncover anything that suggests that the award was inappropriate.

There are some interesting things. You'll know that when we look at standards, using this comparable outcomes approach, you're looking to expect that there will be variations from one year to the next. So, it's not a quota system—it's not the idea that there were 66 per cent last year, there will be 66 per cent this year. There's a tolerance set around that, and for a subject like GCSE English language, it's a 1 per cent tolerance either way. Now, that's basically a reporting tolerance that says that, if it's outside of that tolerance, you need to make a case to us to justify why there's a bigger difference than one would anticipate. The system uses a combination—the awarding process uses a combination of statistical evidence and judgment from experts in establishing where the grade boundary should be. The statistics suggested—. Let me go through the grade boundaries—it's useful context. Last summer, as in 2017, the grade C boundary was at 200 out of 400, so 50 per cent. In November, it was at 204, so 51 per cent, and this summer, it was at 220, so 55 per cent.

Which is quite a big jump.

It is quite a big jump, but it's not unheard of for there to be changes.

Not unheard of but quite unusual.

Quite unusual. You've got to remember it's a new paper as well.

So, you can see why people are concerned, because there's a big difference there, and some children aren't getting over that boundary, are they?

But the thing is—

And they feel that their life chances—. They feel that they are disadvantaged.

There's a common misconception there. Because the papers vary in difficulty, what the grade boundaries are trying to do is keep the performance standard in the right place. Now, there's no guarantee that a candidate who would have got 201 last year, and therefore got a grade C, would get 221 this time. So, there's always variation in candidates' performance.

Okay. But the teachers think that they would have. That's the point.

They do.

That is the point—the teachers think that, had there been a level playing field, their children would have got a C or above.

The one thing is—it is a level playing field in terms of the qualification and the functioning of the qualification. What the qualification can't do is factor in different entry patterns by different schools. So, it can't do that. All it can try and do is provide a fair assessment and maintain a standard. That's all one can expect the qualification to do.

So, what we're looking at is the functioning of the qualification and we're confident that it has functioned properly. What we do want to do is that we want to try and move the conversation forward. So, hence the offer to engage with schools and local authorities yesterday. We have asked the WJEC to do some additional work, where we've asked them to—using this summer's answers—exemplify what that performance standard looks like at grade C. The reason we want to do that is that we want to move away from a focus on a particular mark, because the mark will change from session to session, and if schools are targeting 201 marks because the grade boundary was 200 last time, they're doing the wrong thing; they need to be targeting the performance standard, rather than a particular mark.

So, we want to do that. We're also going to do some additional analysis looking at individual item responses, so we're going to request all the data from WJEC so that we can do what's called 'DIF' analysis—so, terribly technical, but it's 'differential item functioning'. So, it's looking to see how the questions have performed for different groups of pupils. So, we might be able to look at GwE, as a region, versus other regions to see whether there's any particular performance pattern that suggests that there was something different. Now, we're not assuming that there is something different, but it shows our commitment to continue looking at this issue to see if there is anything that can be helpful.

Suzy, you've got a supplementary on this.

Just very quickly—because I'm not sure that I've quite caught it with you—I accept the point that you say that the threshold changes if the paper is marginally easier, but, as Siân said, there's quite a difference between 200 and 220 marks; it's an additional 10 per cent. So, how come the paper was allowed to be that much easier in order to allow the threshold to be that much higher?

I think the difficulty is that, however well-designed question papers might be, their actual received difficulty can change quite considerably once they're taken. So, there are ways around that; you can do a thing called pre-testing. So, that is a much more complex process for setting question papers, where you trial them for two years before you actually run with them, and, when you do that, you end up with much more consistency from one paper to the next. So, that's the way that national curriculum tests are developed in England, which is my experience of using that pre-test method. There are all sorts of issues associated with it and it's very different to the way that GCSEs have developed.

We would hope that the grade boundaries settle. This is only the third time out for a new qualification, because it was reformed, the first assessment was in summer 2017, so it has only been assessed in summer 2017, November of 2017 and this summer. We would hope to see that the grade boundaries start to settle. But that actually can start to create problems in itself, because a very settled grade boundary becomes an absolute expectation, and then, for one reason or another, if the grade boundary changes because of the difficulty of the papers, it's seen to be a shock to the system that the grade boundary has changed. So, there is some comfort in having very stable grade boundaries, but, clearly, that can be false comfort from time to time.

Okay. Something for us to watch in years to come, then.

I think so, and I just want to reiterate: we are engaging with people from north Wales. We understand the concerns that they've got and, ultimately, we care about the system and it doesn't help anybody—. It doesn't help pupils if you've got teachers who are feeling unconfident about what they're delivering. So, we want to continue and we want to move the debate forward because we think that's where we can really begin to make a difference.

There's one interesting statistic that you might be interested in. 'An interesting statistic that you're interested in'; there you go—doubly interesting. We did some modelling and we looked at what the outcomes would have looked like if the grade boundary had been at 200, the same as it was in 2017, and that would have led to a 7 per cent increase in grade Cs and above and there was no evidence to—. I've already said that we look at 1 per cent being roughly what one would expect to see; there was no evidence to suggest that there would be a 7 per cent increase in performance. I think that's one of those—you know, it's one of those tests around was the grade—. If the grade boundary had been where it was last year, there would have been a lack of public confidence in the maintenance of standards, quite justifiably, and we couldn't see any evidence to support that big a change.

Okay, thank you. The next questions are from Suzy, anyway, so—.

Oh, right. Thank you, Chair. I'm just going to move on to the new vocational qualifications and the sector reviews that you've done. One of the purposes of having Qualifications Wales, of course, was to reduce the number of vocational qualifications, because there is a massive plethora of them out there. You're creating 19 new ones at the same time as getting rid of some. What are you getting rid of that was—? Well, I suppose my question is: what's in your new qualifications that wasn't already out there in some form or another?

Ah, right, yes.

Shall I—?

Yes.

Why do we need new ones—

'Why do we need new ones?', yes.

—when you're getting rid of old ones?

So, that goes back to—. Maybe I can touch on health and social care, but then I'll touch on some more generalised issues. 

Could we start with health and social care? I think that would be great.

So, if we're looking at health and social care, we're replacing 200-odd qualifications that are out there with 19. If we go back to some of the principal findings from the health and social care review, there were unclear progression routes between qualifications, so rather than—. It was a sort of naturally developed qualification landscape rather than a designed qualification landscape, so, as a consequence of that, there weren't logical progression routes. There was a lack of qualifications at level 4 specialisms, so people were jumping from level 3 qualifications to level 5 management qualifications just to develop their career, without necessarily going down the right routes. There were inconsistencies between qualifications. There were inconsistencies in the way that they were assessed. One of the principal things was that the qualifications that were being used were qualifications that had been developed in England, and therefore quite frequently related to England legislation rather than to Welsh legislation, and, of course, this is a devolved area of policy, so there is different legislation. So, frankly, we didn't think that the current range of health and social care qualifications and childcare qualifications were serving Welsh candidates correctly, and they certainly weren't meeting the needs of the sector bodies, who we've been working with very closely in the development of the qualifications. 

As we've gone on and we've looked at other qualifications, or other sectors, there are some similar patterns there in terms of complex progression pathways that people don't understand—that was something that construction found—duplication and inefficiency, where a learner might be doing one qualification, in construction, say, then doing another qualification and going back over learning they'd already done, because they weren't articulating properly. So, there are a number of common issues that have come out.

I was just looking here; I've got some statistics, which—

Can we—? I don't want to close down what you want to say, but can we stick with health and social care first, because that's a sector of great interest to this Assembly at the moment? So, basically, you've stripped out the inadequate, the confusing and the duplication and created these new qualifications, which progress through the different levels, I guess. Are these qualifications still available in England? Sorry—your new qualifications, will they be available in England, I should say, if people mysteriously wanted to do them. 

There is no reason—. We wouldn't have any prohibit of it. So, we wouldn't have a problem with that. It's whether they would be available on publicly funded courses in England, and, of course, that would be a decision for England. We would place no restriction on the awarding bodies.

Okay. And, in putting together these new qualifications, obviously you want them to be portable. Can you give us a little bit of indication about what conversations you've had with other parts of the UK about the portability? Because this looks great, doesn't it? Simply 19—

Portability is really important, and it's something, in fact, the board talks about a great deal, because we don't want to disadvantage our learners. In terms of portability, the key players in this are actually Social Care Wales and their counterpart in England, because they are responsible in Wales for registering individuals within the sector, and we're very clear that that relationship between Social Care Wales and their counterparts in England is critical to this, but we've not heard anything that would mean that, in England, our candidates would be disadvantaged. They don't register the workforce, I believe, in the same way as we do in Wales in England, and so it is slightly different from that point of view.

In terms of candidates coming the other way, because that matters as well—somebody might have picked up their qualifications in England and now moved into Wales and want to do things. Again, that's Social Care Wales's responsibility, because they register the workforce, but what we understand they're going to do is very pragmatic. In other words, they will accept the England-based qualifications, but will ask candidates to do a module that gives them the Welsh context, because our legislation is different.

Well, you've spotted my question there. If you're training to do different legislative frameworks, which is absolutely in order, at what point do they become incompatible? 

Well, that's what they're going to do: they want to put in an additional unit that brings the Welsh context in, because it's not just legislation; there's a difference in emphasis in Wales to the emphasis in England as well. 

Have you had any indication that that might work the other way round as well? So, someone from Wales may need to do—let's make it simple—an English-legislation-based module. 

Because there isn't registration of the workforce in the same way—

But good care homes—let's just use those, for example—will be looking at what qualifications their staff have. 

Yes, and—. I think it's a difficult one in something like health and social care, in particular, where it is a devolved area of policy. So, there's a simple choice: do we, as a Welsh body, promote the development of a set of qualifications that are designed for Wales, and meet the needs of the sector body in Wales and Welsh legislation? And, if you do that, you are necessarily potentially going to have some impact on portability. But the learning that people will be doing, the competencies they're building through the workplace, are all things that become highly portable. The job of work for the sector bodies—so, Social Care Wales, in particular—is to work with its counterpart bodies in the UK so that there is a good understanding of what the qualifications are delivering. But there will necessarily be differences because it's being focused on the needs of Wales, which we think is the right thing to do. 

And that was the purpose of Qualifications Wales, I accept that. So, there's always a core of portable skills, provided that you're able to communicate those to your potential employer—. Can we move on, then, to construction and building? A similar sort of situation's arisen there. What are the key differences between the two sectors in terms of how you've had to look at what goes into the new Welsh qualification? Is the legislative framework the key driver of that, or emphasis within the sectors? It strikes me as quite different. I'm wondering if you can—. 

It's very different. There are much more diverse sector bodies in construction. We worked very closely with the Construction Industry Training Board on the proposals. We've worked closely with the British Association of Construction Heads network, which is the network of construction heads in further education. We've worked with lots of employers as well, both in the review and as we're progressing our ideas. 

Construction's a different one, so what we've found is that it's less about legislation and it's more about the broader needs of Wales. So, what we found was that construction qualifications weren't necessarily focusing on things like the housing stock within Wales, where there's—. There was limited skills development with historic buildings. So, a lot of the building qualifications have been dominated by the needs of big employers about putting up housing estates, rather than necessarily working in different contexts. Similarly, they weren't looking at things like renewable energy, and how renewables work in houses. So, it'd had become very focused on one particular area of the construction industry, which is an important area, but it's not the whole thing. 

There were also real concerns about the duplication that was going on, where a learner might be doing a qualification, then going and doing the next qualification and going back over stuff they'd already learnt, going through FE college, coming out with qualifications at the end of FE, getting a job on site and then having to go through their card—so, getting their construction card—where they'd have to go through the same things again, but on a competency basis. So, there was lots of duplication, which was making the system very inefficient. And, working with CITB, the sector bodies, the BACH network and employers, what we think we've found is a way of being much more efficient and doing things differently in Wales that will be to the benefit of everybody. We've got a very wide consensus of support from all of the people that we're working with. It's still very early days with construction, and we're still working on shaping those qualifications and on what they might look like.  

You mentioned consensus, but there was also a bit of pushback as well, wasn't there, from the construction sector. You mentioned in your evidence—sorry, in your report—that there were 25 concerns raised that you needed to be able to respond to. 

We've had—. I mean, you never get an absolute consensus on anything. 

No. What were the main areas of challenge to you, then? Let's put it that way. 

Well, it's probably worth thinking about the main areas of challenge we've had since we've gone ahead with the proposals. We went out with our proposals to restrict construction qualifications, and we had two representations, because we consult on our intention to restrict with the full range of proposals, and we see what comes back in from those, and then measure whether we're still doing the right thing once we've done that. We've gone out. We had two representations come back to us. One was from an awarding body that wanted to protect its range of qualifications, which is not an unnatural thing for an awarding body to do. And there was one representation from an FE college that had concerns about some qualifications it was delivering through a learning programme. 

Now, actually, when we had a look at those, they wouldn't have been affected by the restriction that we were looking to put in place. I think there were one or two qualifications that we thought might creep into the scope of the restriction. But if there's a good case for the restriction being amended in some way, then we're always open to that. So, we haven't had pushback since the proposals have bedded in. The next acid test will be when we go out and consult on our more detailed proposals. So, the review is looking at what is the general shape of things, how might we move forward. Now that that has become more bedded in, we'll be going out to consult on detailed proposals, we think about April, May time, so it's going to be late spring. And that's before we bring an awarding body on board in the summer. So, we're in the middle of a procurement process at the moment to bring an awarding body on board to develop and deliver these qualifications. So, there will be another opportunity. 

One of the concerns will always be difference in Wales to what's going on in England. So, there's already concern about apprenticeship frameworks as an example, where, in Wales, the apprenticeship framework requires qualifications, and in England, the apprenticeship standards require an end-point assessment, which can look quite different. The area where you will always get concerns, if you're proposing to do something different in Wales, will be those employers that cut across both countries. And where that is the case, you need to do more work to reassure them that what we're doing in Wales is every bit as good as what's going on in England, if not better. 

Or the other way around, I was going to say. 

Yes. 

Just before I go on to my final question, I just wanted to ask: in both these sectors now where you will have new qualifications, good employers will always offer continuous professional development, but CPD that doesn't necessarily come with a bit of paper attached to it; it's just that you've had two hours sitting in front of a computer doing something. You don't get involved in that area of work, at all, do you? No. That's fine.

Okay. And, then, just finally, Ann Evans, you mentioned that you're about to publish an update on the ICT. 

Yes. Next week.

Do you want to fill us in? Do you want to give us a little snippet of what might be in there?

Give you a bit of pre-notice.

Yes. Are you allowed to do that?

I think we can do a little bit of it, yes. I think it's been quite interesting the ICT work, because one of the reasons we went for sector by sector is, we thought that you couldn't have a one-size-fits-all. We thought the different sectors would bring different challenges. And we're quite right because ICT has been quite different to the sectors that we've looked at prior to that—health and social care and construction and the built environment. So, we're pleased that our assumptions were right, and that's meant we've come up with a smaller list of recommendations, which are part of that report. Do you want to go through them?

Shall I go through the highlights?

Yes. Just the highlights. 

You'll have to be quick. 

Very quick?

Yes, very briefly. 

So, highlights are: IT employers pay a lot of regard to degrees, A-levels and GCSEs, and to what we call 'vendor qualifications', so things like Microsoft and Cisco certifications. Those sit outside of the regulated market. So, they don't pay much regard to other qualifications. So, there's a particular focus that IT companies have. Those learners that are taking level 3 qualifications in IT are generally looking to progress to higher education; they're doing that. 

Employers have a strong focus on degrees, but, equally, they have a strong focus on softer skills. So, things like teamworking, problem-solving, and communication skills are some of the things they're really looking for. We found that content of qualifications, the existing range of qualifications, can be quite outdated. That can often be because, you know, still referring to floppy disks, if you can remember floppy disks. Certainly, most young people will never have come across a floppy disk. So, you know, there was outdated content in there. That’s not to be unexpected in a fast-paced, dynamic environment, things like cloud computing and the like.

Okay, can I cut across, then? Because, obviously, we need to bring this session to an end. Does this mean—? I hear what you say about these things being out of date, but there’s actually no great need to produce a whole new range of qualifications in ICT.

No, and what we’re proposing to do is to have a new GCSE and A-level. So, there’s a GCSE and an A-level in ICT at the moment, which is essentially an unreformed GCSE and A-level, which we let continue while we did our review. So, as a consequence of the review, we're going to do that.

There are some qualifications that have been sponsored by the Tech Partnership for the apprenticeships. When those come online, we'll monitor those to see what they’re like.

Okay. We're moving on to the next sector soon, so I won't ask that question. Thank you, Chair, for being so patient.

Okay. Thank you.

You referred earlier to the ongoing reforms that we're grappling with in Wales. We've got some questions on that now, and the first questions are from Hefin David.

Yes, with regard to the new curriculum. You said in your annual accounts that the implementation of the new curriculum is a systematic risk for your organisation. Can you expand on that, please?

Sure. I'm going to cut it into two parts, because I think that calling it a 'systematic risk' for our organisation is probably conflating two things. So, what I'll talk about is a qualifications systems risk, which you would expect us to be identifying, and then some specific risk to us as an organisation. So, that will answer your question.

In terms of the qualifications systems risk, one of the things that we're concerned about is what I call change fatigue. We are part way through a big reform programme that hasn't finished yet, and now we're already talking about more reforms. So, the question that we've been bouncing around is: is this a good or bad thing? Should we not talk about this yet and let the current system bed down a bit? But on balance, we've decided that it’s a good thing, because we have time before these qualifications are needed, and if we are engaging now and over the next few years, then the whole process of getting the qualifications in place and making sure you've got the training and the resources in place for the new curriculum, it gives us a better chance to do that. So, that’s the qualifications systems risk that we've identified. We also think that we've got a lot of lessons learned from the last round, and that’s fresh in everybody’s mind at the moment, and that might help us in moving forward into new qualifications that will support the new curriculum.

In terms of Qualifications Wales and the systems risk, for us, it is our resources getting stretched very thinly. Because, clearly, we already have a full programme of work without the additional reform work that is going to be needed to undertake the development required for qualifications to support the new system. We've already been in discussions with Welsh Government about it. We know we can absorb some of these costs, but not all of them. And so, that’s where you get the systems risk for us, in that if we had to move everything into looking at that, other things would have to be stopped.

So, what specific things have you done to alleviate that risk? I know you've set up a working group with Estyn and the Welsh Government. Is that part of what you've been doing to date to reduce the risk?

Yes, I think producing the risk is something that—. This risk isn't realised at the moment, because we're six years away from those new qualifications needing to be in place. What we're highlighting is a timeline that says the pressure points for us as an organisation are going to be coming later down the line. And so, I think some of those discussions that Philip's already had with Welsh Government and with others are helpful.

The working group is very useful for us to develop our thinking. What we're planning to do is to communicate our approach more widely, probably at the beginning of next year. What we want to do is we want to be working in parallel to the curriculum being finalised, to develop the proposals for GCSEs, and to make sure there's really strong engagement with schools as we do that.

You said earlier that you didn't want the qualifications to define the curriculum, but you do need, by 2020, to confirm a future qualifications offer with the Welsh Government. So, although you say it's six years, by 2020 you've got to define your future qualifications offer.

Yes.

How are you going to make progress to that if there's this change fatigue, as you mentioned, but also the need to make progress quickly?

We're planning to run a consultation in September of this year, so we're going to use the time period when the area of learning and experience working groups and the pioneer school network is moving from developing the curriculum into the next stage. We're going to be looking at forming a sub-group of those people who are particularly interested in secondary education and qualifications to start looking at the shape of that. We're going to engage a panel of experts to help us as well to develop proposals and then consult on those proposals in September, so that we are ready for January 2020. Our intention is to be able to get awarding bodies on board. We've written to awarding bodies, the GQ awarding bodies, to see who's interested in developing these qualifications. So, we've started that piece of work. What we want to be in is a position from 2020 to really be working on the development of these qualifications. What we would like there to be is, if not final specifications, then at least a good shape of what these qualifications are going to look like for September 2022 when schools start teaching the new curriculum. That will be well ahead of first teaching in September 2025, assuming that the first GCSE cohort will be summer 2027.

And in many ways you're reliant on the pace of the development of the curriculum. Do you feel that it's developing at a pace that's appropriate?

I guess it's developing at the pace that we expected. We were expecting it to be published in April of next year. We were thinking, ideally, we probably would have liked to consult on our proposals a little bit sooner than September. Actually, if it's April, it only gives one term for schools to really get their heads around what the new curriculum is—those that haven't been deeply engaged with it. So, it would be disingenuous to consult on our proposals before they've had a chance to get their heads around the new curriculum. So, inevitably, we are a bit impatient, we'd like to get on with things, but we have to go at the pace—

I was just thinking that 2027 seems a long time.

It won't be.

It gives us time to think about doing it properly. Of course, the curriculum is being implemented with year 7s in September 2022, so if you follow that cohort through, it means that they will reach year 10 in 2025 and they will reach year 11 in summer 2027 for assessment. 

Can I just come back to that statement that you don't want the qualifications to influence the curriculum? Is it realistic to think that's going to happen? Can we expect teachers to cover the qualification and the curriculum—the curriculum to develop with the qualification? Isn't it more likely to be kind of cyclical?

It'll be a mix. It'll inevitably be a mix. What we wouldn't want to see—and when I say 'we' I guess I'm speaking for education bodies more widely. This might be a question for your next session. We wouldn't want to see a reductive model of education where it's just about teaching to the test. There's plenty of evidence from all sorts of spokespeople on education at the moment that that's a real danger that I think is being realised as well. So, if the new curriculum is going to be transformative, it needs to have a different culture around it, and the different culture has to be more expansive in its model of education. 

One last question. Can you just give me one example of how you've engaged with a pioneer school?

We go to all of the AoLE working groups where they're developing the curriculum. We go to those as observers—that's our official status—because we don't want to be overly influencing the development of the curriculum; that's not for us to do. So, we engage with pioneer schools through those groups. What we will be doing more actively is engaging directly with them and forming our own groups from January onwards. 

Right. Okay.

Okay. Thank you very much. Janet. 

Thank you, Chair. How do you envisage the 10 or so individual qualifications that pupils currently take at the end of compulsory schooling being aligned with the six areas of learning and experience that the new curriculum will centre on?

We've done some mapping. We've mapped the existing offer of GCSEs into the six areas of learning and experience. Of course, that's quite interesting because one subject domain might cut across more than one area of learning and experience. So, writing—is that creative and is it expressive art or is it literacy? There are things that cut across—elements of food science come into health and well-being and also come into science. So, there are areas that cut across more than one AoLE.

We've also done an interesting piece of work—we've tried mapping them to the four proposes, so if the four purposes are the core of what the curriculum is trying to achieve, how do qualifications map to them? And that looks quite interesting as well.

There are inevitably some areas of these that don't have as much coverage—so health and well-being don't have as many qualifications that would naturally fit within them. What we're planning to do is to do more work around that, but it's also trying to work out what Welsh Government's policies are going to be. So, at the moment, if you look at the 10 GCSEs that a learner might be taking, many of those will be taken up with the core offering of English, two maths or Welsh, Welsh second language, two sciences, Welsh bac. By the time you take that core as it is at the moment, it only leaves two or three subject options, which is the common concern about a narrowing of the curriculum in key stage 4.

We will be working with Welsh Government to understand what their policy objectives are going to be. Are they going to be looking that a learner takes a qualification from each of the AoLEs, or are they just looking for curriculum coverage across each of the AoLEs? Those things have yet to be defined. Now, some of those things will have an impact on the viability of some subject areas. So, suddenly, if there's a requirement to take something from the expressive arts area of learning and experience, that will increase the number of learners that are taking GCSEs in those sorts of subjects.

Can you share your views on what the impact of the introduction of T-levels will be in Wales?

And relatively brief views, because we're running short on time, please.

Okay. Shall I kick off? Just to put this in its context, the T-level programme in England was a response to the Sainsbury review. It is a programme that is made up of a technical qualification, literacy and numeracy, some other trade-specific qualification and work experience. Now, our understanding is Welsh Government has no wish so take the whole programme. Where we are very interested is in those technical qualifications. And I think it would be fair to say, Philip, that you've done a huge amount of work in this area, working with the Department for Education in England and now, increasingly, with the Institute for Apprenticeships who are going to be taking this over in England, to try to make sure that, in Wales, we have access to that technical qualification bit of the T-level programme. Do you want to add anything? Do you want anything more, because I'll turn to Philip?

Well, just a quick one, really, and a quick response, please. How have you sought to influence the development of T-levels in England, and how are you preparing for the impact that they may have in Wales?

We've been working quite closely with the DfE, as Ann said. One thing is: what they're trying to do with T-levels is very similar to what we've done with health and social care, in terms of restricting and commissioning. So, we've been working with the DfE to share our learning in what we've done in that area. We've engaged with consultations and we've worked with them on their procurement documents to try and open the door to those technical qualifications being available in Wales.

Okay, thank you.

Thank you. Hefin David on vocational qualifications.

Do you think it's acceptable that five awarding bodies were so non-compliant with your regulations that they posed a significant risk to the validity of the first aid qualifications they were delivering?

The simple answer to that is 'no'.

So, how do you deal with it?

Okay. When we found the problem, what we have done is ask those awarding bodies to produce an action plan. This is not some kind of sterile action plan that sits on a shelf somewhere and nobody looks at again. We monitor how they are going to improve their systems. We've monitored all the actions that they have signed up to, and part of that is not just about improving the assessment that they were doing, but improving their own quality-assurance processes, so that you can guarantee the ongoing validity of those qualifications. So, there's a huge amount of work being done with them. All of those actions have now been signed off, but, clearly, that's an ongoing issue for us, because we don't want to see slippage of them falling back in any way. And having looked at first aid, I think as Philip has already mentioned, we're on to look at food safety qualifications and then, ultimately, health and safety qualifications. These are small qualifications with very large numbers of learners in Wales taking them, and we think it's absolutely right that we're looking at them, to make sure they're as good as they can possibly be. But, no, we weren't happy.

How are you sustaining the monitoring and ensuring that the improvements continue?

So, we don't release them from the undertaking until they've done everything that they were supposed to do.

Okay, thank you. We've got some other issues to ask about now. Before I bring Suzy in, can I just ask about modern foreign languages? The figures are becoming increasingly alarming, particularly at A-level, and are surely reaching the point where it's not going to even become sustainable to offer those qualifications in Wales. What is your view on the situation, and how concerned should we be as a committee?

Certainly, we've had conversations with WJEC over the viability of the qualifications. We're working quite hard to encourage WJEC to keep offering them, especially German, where we're down to about 70 learners. That's clearly not economically viable for them—it's on the threshold of technically viable—because it's difficult to award subjects with very, very low numbers. There's clearly—. Welsh Government has got Global Futures as an initiative that it's trying to do. In terms of qualifications, what we want to do is to keep the qualification offer there, because without a qualification offer, it really is the death knell, certainly for a subject like German, as an example, because if German A-level were to go, it would reduce the number of people doing GCSE, because it wouldn't lead on to an A-level for those that wanted to do it.

'Where's the solution to this?' I think is the interesting question. Maybe the new curriculum has a part to play in that. If it's more about learning language skills, whether that be English, Welsh and a modern foreign language—a much more blended approach to learning languages—that might be something that improves the viability of those areas. I think, for us, it's keeping life support on these qualifications as far as we possibly can, until there is a change.

Okay. Suzy.

I think we could have another session on modern foreign language altogether. Thank you.

I just had a couple of quick questions about your budgeting and audit. You had an underspend last year, a smaller underspend this year—does that mean you'll have no underspend next year?

We anticipate the underspend will be lower this year, and we hope to minimise it as much as possible, but we can't guarantee there won't be a small underspend at this point, and you need the rationale for some of that as well, which I think is important. So, we want to make best use of the funds. We are very careful in the way in which we use funds, and we're also totally transparent with Welsh Government about the use of our funds, because we think that's a fundamental way a public body should behave.

We have a few challenges that are unique to a particular year. So, this year, when we talked about health and social care, we didn't actually talk about the fact that we have delayed the implementation of some of those qualifications. That means the spend on their development is next year, not this year, so things like that will happen. But we have two perennial problems that we are getting better at handling, I suppose. One of them is our Welsh-medium assessment grants. So, our Welsh-medium assessment grant pot is there at the start of the year, and awarding bodies bid for money to support them to provide assessment through the medium of Welsh. What we've discovered over the last two years is that those bids don't actually turn into a full claim, so there's a drop. But you can't take the money out of the pot until you know how much money is going to be taken from it, and so that's definitely one of the issues. Now, as to why that happens is a long and complex answer, which I'm not going to go into now. The other thing that we've got better at understanding is the implication of our very small staff turnover. We're a small organisation, but, inevitably, staff will leave, and that does mean that sometimes, on our staffing budget, by the time you've replaced that person, you've had a slight underspend during the year. So, it's an area that the board focuses on, virtually every meeting. We talk about money a great deal, and we're getting better at it. Three years in, we now have historical information about how our budgets are being spent. 

Looking forward, we've got—it's called 'incremental drift'. As all our staff climb their scales, it puts more pressure on the budget. There are potentially big pension changes coming, which will put more pressures on our budget, and we've also talked about the new curriculum and the pressures that that would put on our budget. So, we're forecasting a very different scenario in the future.

Okay. Where does this appear on your risk registers at the moment? One of the concerns we had a committee when Qualifications Wales was in the process of being created—we had questions about the finance at the time, which is not about you as you are now but in the process of setting you up. So, we still have that question about judging how much money you need, I guess. So, as I say, where does this appear on your risk register at the moment in terms of governance? 

It's on our corporate risk register.

It is. We manage a medium-term finance plan, so we're always looking three years out on what the pressures might be. 

What colour is it?

At the moment—. Actually, do you know, that will depend on what happens with pensions? 

Okay. It's a perfectly reasonable response. 

So, we've looked at what the potential increase in employer contributions will be for the civil service pension because we're part of the civil service pension scheme. That could be quite crippling for us on a relatively small budget. 

Okay. Well, a similar question, on governance: the grants that you give out—you mentioned quite a lot goes to Welsh language assessment. What's your process for monitoring value for money on that? Without a twenty-minute explanation, obviously. 

Quite simple: we look at claims very closely. Most of the grants that we give out are for Welsh-medium translation. We pay a standard rate, which is between £70 and £90 per 1,000 words, which is the going rate and the rate that Welsh Government will pay. So, we pay the going rate and we make sure that everything is done. We've got full visibility and transparency of what's being claimed for. 

Okay, thank you very much. And just very, very quickly to finish: obviously some of this underspend—you may still have an underspend. What do you expect it to be going on next year? You mentioned the delayed introduction of the qualifications for health and social care. Do you think that'll eat up any other money, as well as the incremental drift that you mentioned? 

Yes. I mean, clearly, when you move work from this year into the next financial year, that's part of our budget planning for next year. 

You don't think it's likely to be an indication to Welsh Government to reduce funding, then? 

We would hope not.

You hope not, yes. 

We're very open with them.

We're working hard with them to let them understand that we're going to be under pressure for our budget in the future. 

Yes, okay. Lovely, thank you. 

Okay, thank you. If there are no other questions from Members, then we have come to the end of our session. Can I thank you for answering all our questions? It's been a very informative session. Thank you for attending. As usual, you'll get a transcript to check for accuracy. Thank you very much.

The committee will break until 11.15 a.m.

Welcome back, everyone, for our next item, which is our session on the Estyn annual report. I'm very pleased to welcome Meilyr Rowlands, Her Majesty's chief inspector, Claire Morgan, who is a strategic director of Estyn, and Jassa Scott, who is also a strategic director of Estyn. Thank you very much for your attendance today, and we're looking forward to hearing what you've got to say.

If I can just kick off with some questions about the new common inspection framework—whether you feel it has significantly changed the way schools are inspected and whether there have been any identifiable changes in outcomes.

We introduced a new way of inspecting, as you know, in September 2017, and that was at the end of the previous cycle of inspections—the seven-year cycle. We obviously refresh the framework every so often. We do it in a way that we hope improves the education system, and we do it in a way that we think improves the inspection process. In particular, this time, we wanted to make sure that inspection is as good a professional learning experience as it possibly can be, under the circumstances, for the schools and the other providers we inspect. But we do it in a way, obviously, that still meets the requirements of legislation, and we work within that legislative structure. So, we don't expect it to change what we can report to you, for example, at a high level or that the way we inspect makes a big difference to the actual inspection outcomes themselves. So, for example, what we did this time was we simplified and streamlined the inspection framework—we reduced the number of aspects by amalgamating them together from 29 to 15 aspects. We simplified the grading system. We had 15 different grades at three different levels in the previous cycle, and we've simplified that to concentrating on the five main areas, which are set out in legislation: standards, well-being—those sorts of things—quality of education and leadership.

So, I think we can be very confident that we can make sensible comparisons from year to year. I mean, obviously, we change our inspection guidance from year to year as well, to make sure that it's totally up-to-date with the changes and the expectations out there in the education system, which constantly changes. But the big change happens when we change the framework at the end of the cycle.

So, to answer your question briefly: we didn't expect it to, in itself, have any major implications for the outcomes of inspection.

Okay, thank you. And you've made some changes as well that make it more difficult for schools to predict when they're going to be inspected. Have you noticed a significant change in schools' levels of preparedness for inspection as a result?

I think the changes, the recent changes, have gone down very well with schools, and I think generally people have thought that making inspections less predictable—when they are going to occur—has also been warmly accepted. That goes back to, I think, 2014. So, we've had a few years now where people can't really predict when they're going to be inspected. I don't think either that that has made a huge difference, and we wouldn't expect it to have a real big difference to the inspection outcomes in the sense that the big things we are looking at: standards, quality of education, leadership—those aren't things that schools can easily change in preparation just for an inspection.

But I do think it gives reassurance to parents that schools aren't over-preparing for inspection, and also that the schools themselves don't—some schools might or other schools don't—but it's possible that some schools might have, in the past, over-prepared for inspection, and having this unpredictability does help that as well.

Okay, thank you. And the format of the report is slightly different this year. Can you just briefly explain what you've done with the changes and also what we can expect next year to coincide with the Programme for International Student Assessment results?

Yes. So, what we've done this year is that the inspection report that you have in front of you consists of sections on each of the sectors that we inspect, looking at those big areas of standards, well-being and so forth. So, those are the requirements for us to report to you in legislation, and I think that will be a constant each year. What we've done in previous years is, in addition to that, we've added a thematic section focusing on a particular topic—the curriculum or professional learning, or leadership or something like that—and that's the area that we didn't do this year. We've done that in order to address one of the recommendations in 'A Learning Inspectorate', which suggested that we have a lighter-touch annual report. So, every year, other than the PISA year—so, every three years—we have a bigger annual report that is a kind of 'state of the nation'. Next year—so, next December—the PISA results will come out and so we will publish an annual report. It'll probably go back, then, to January or February to give us time to process the PISA results. But the idea is to put the PISA results and our inspection findings in a wider context and bring everything together in a more substantial report every three years.

Okay, thank you very much. We've got some specific questions now on early years from Suzy Davies.

Thank you very much. I'm just asking questions about the non-maintained sector, but for the record, could you explain what the non-maintained sector is? It's more than just non-state nurseries, isn't it? Can you explain what the sector includes?

Yes. They're all kinds of nurseries and day nurseries, those sorts of—cylch meithrin. They are non-maintained, which means they're independent. They are not supported by the local authority, although they are financially. But the local authority does have some responsibility for supporting them in terms of guidance and that sort of thing. So, these are children before the age of statutory education.

But it doesn't include home-schooled children.

No, sorry—no, it doesn't.

That's what I wanted to—. That's fine. I just wonder, then, if—. Obviously, we were talking about childcare fees yesterday, so can you give us an indication of the main challenges you see in developing a joint inspection framework for education and, of course, care?

Yes. I'm sure my colleagues will go into all the challenges involved in that, but I want to emphasise, really, how immensely rewarding that process actually was. We worked with Care Inspectorate Wales for a long period to develop that, and we learnt an awful lot from one another. It is a challenge—we are two separate inspectorates with very different legal roles and responsibilities. They're a regulator and we are not a regulator—we are an inspectorate. So, bringing those two did have its challenges, but it was hugely rewarding. We've learnt a lot and the outcome has been an inspection framework for this non-maintained setting sector, which I think is very, very positive. We've obviously been piloting it, and it's gone down really well. I think it was introduced in order to reduce duplication. I think that's a bit of a red herring, to be perfectly honest. There wasn't a huge overlap in what CIW were doing and what we were doing.

I think the big benefit that we found from working together, other than institutional learning, has been that for these providers—these early-years settings—what they now will get is an integrated inspection. They will get a view from CIW and Estyn simultaneously, and I think that is a much richer kind of information that they will get and will help them move forward, rather than two separate inspections.

Okay, thank you. I must be honest—personally, I was less exercised about having a common inspectorate, if you like. But, as a result of—. I was aware of nurseries in particular saying that they had two separate inspections and that, occasionally, the recommendations of each would contradict each other. So, I can see that this is a move on. Have you, as a result of this process, been able to introduce anything new into the inspection framework, or has it literally been trying to weld the two together in a meaningful way?

I don’t think there’s anything particularly new, no.

No, well, that's great. I just wanted to ask you, as well, about the results that they’ve had from Estyn. I notice in one case that, in actual fact: 

‘in most of the cases requiring Estyn monitoring, children do not make enough progress in developing their skills...because practitioners do not use assessments well to identify and plan for children’s next steps in learning.’

In the process of coming to that conclusion, did you decide that this was something that’s missing in the initial training for these practitioners, or just some bad habits that develop over time?

It's a good question. I think one of the challenges—. This is a sector that is very, very high performing, by the way. I should explain that before we maybe nit-pick a little bit. And they're doing that in quite difficult circumstances. Because one of the advantages of being in the maintained sector is that there's no competition, and schools come together and share good practice. And that can be a little bit more difficult in this sector, depending on—. There are umbrella organisations for some of these settings, but not for all of them. So, sharing good practice is a challenge.

I can’t say that we’ve identified that as a particular challenge in initial training of these, but what I’m saying is that the continual professional development is a bit more of a challenge in this particular sector. And we’ve been working quite a lot with the advisory—. As I said earlier, the local authority does have some responsibilities, and it does have an advisory teacher who works—it’s a relatively small element of resource, but they do work with this sector. And we’d be working with them as well. So, I think the sector is getting better at sharing good practice in that way together.

Just a final question, have you noticed—? Or are you able to say, I should say, any specific bits of best practice that you’ve seen in Flying Start settings that would be valuable in others that aren’t?

I'm not sure if there are any, colleagues, on that. I'll look into that, and I'll—

It would be quite useful for us, to help identify how Flying Start's going, really.

Yes. We don't inspect that, of course.

No, no, but the settings in which there are some Flying Start children.

Yes, sure. Yes, I'll have to come back to you on that.

Okay, thank you.

Okay? Thank you. The next questions are from Siân Gwenllian.

Recently, we've started to see more good practice and, really, a better understanding of what good foundation phase pedagogy looks like. So, it's certainly an improvement on the picture from last year. We're also seeing signs that there's better use of outdoor spaces as well, which was one of the areas we mentioned last year that there was a weakness in.

Now, in our interaction with the regional consortia, local authorities and schools, many of them tell us that some of the recent improvements have been triggered by one of our publications, which was on active and experiential learning, which we published around this time last year, in December—no, yes, last year, December 2017. And we know that the materials, the report and also the film, have helped to raise the profile, I suppose, of foundation phase. We know that the actual report was downloaded about 3,000 times, and there's quite a useful film that goes with it, and we know that that's been viewed 3,500 times. So, I think raising the concern last year about the somewhat lacking progress on foundation phase and the report and the professional learning have had an impact. So, it's a more positive picture than we mentioned last year—

Absolutely. I think there's some way to go, and I think where there's some more work that needs to be done is where there's too much of a focus on formal assessment, still, at the expense of developing the foundation phase approach.

Yes, certainly.

Suzy, did you have a supplementary?

Claire, I don't know if you want to add there.

Certainly. We're talking about a fifth of primary schools, but it is a very important issue. It's partly a leadership issue, that school leaders are not ensuring that the staff have the skills to teach reading properly. As a result then, pupils' reading doesn't progress as well as it should. Often, pupils are asked to answer questions that only require them to give information rather than give opinions or make evaluations. So, it's at a much lower level than perhaps they are capable of.

In these schools, we're also concerned about the range of vocabulary they're using. So, they're not having enough opportunities to develop that and to gain even more confidence in their reading. So, it certainly highlights the importance of widening pupils' opportunities so that they hear and use quite a wide range of language.

We're also going to be doing some work, similar—. I mentioned earlier that we had done some work to support the foundation phase. We've done a thematic review, and then conferences, and we will do the same for reading next year. So there will be a thematic review of reading, and conferences in all the four regions then towards the end of 2019. So, we're hoping to work closely with practitioners to identify some of the best practice in developing reading skills actually to try and address this issue.   

Yes, absolutely. I think, talking about leadership, it is actually leaders' responsibility to ensure that their staff have access to the professional learning to make sure that they can make progress in this. You're absolutely right. 

But can new people coming through the system, newly—. Is it still a problem? 

Do you want to go on to secondary schools, please? 

On the national academy, we're pleased that they've prioritised serving headteacher development, because obviously they've got quite an ambitious vision for developing leadership in Wales, but there's an acute problem where we need to make sure that our weaker headteachers have the support they need to bring about improvements. We're also leading a project with the OECD to develop a national approach to self-evaluation for improvement, and we've also identified leadership as one of the priority work streams on that, so we hope that that will go some way to support the system to improve as well. But there's certainly a lot of challenges in leadership, as Meilyr said, in secondary at all levels, and it is complex. The new curriculum will give us a great opportunity to engage learners in secondaries far more, because we've seen a narrowing of the curriculum. There's very little choice for learners in secondary in some schools. So, this is an opportunity, I think, to engage the learners but to also address the issues in leadership in secondary.