National Assembly for Wales

Morning, everyone. Welcome to the Children, Young People and Education Committee. We've received apologies for absence from Dawn Bowden, and there is no substitute, and we've been advised that Michelle Brown is running late. Can I ask Members whether there are any declarations of interest, please? No. Okay. Thank you.

We'll go on then to item 2, which is our scrutiny of Qualifications Wales annual report 2017-18. I'm really pleased to welcome Ann Evans, chair of Qualifications Wales, and Philip Blaker, who is the chief executive of Qualifications Wales. Thank you for attending, and we're looking forward to hearing what you've got to say. So, if you're happy, we'll go straight into questions. If I can just ask a general question about what you feel the biggest challenges and the biggest strengths have been in this reporting period.

Okay, I'll kick off with that because I think that, over the year, we've continued with our innovative approach to looking at qualifications within sectors. So, in health and social care, we're now getting to the implementation phase; in construction of the built environment, we're just looking at the qualifications; and we're about to publish information and communications technology. I think where we're coming from, and one of our biggest challenges through the year that we discovered through the work with health and social care and childcare, is building that consensus around what the qualifications should contain. And that has taken time, it's taken a lot of hard work, and it's bringing people together to really thrash out what was needed. So, we would identify that as one of our biggest challenges that has happened this year.

In terms of successes and the way in which we think the year has gone, at the very start, we really were determined to be an evidence-led organisation, and we've become more experienced at gathering evidence and using that evidence. So, for example, the work we published on early and multiple entry GCSEs, which was back at the beginning of the year and which was welcomed by Welsh Government, led to a change in performance measures. Our research programme has also grown, and we've also taken on the capacity to publish statistical data. Prior to that date, that was done by England on our behalf. So, we can now do this in Wales ourselves. So, we think that's one of our biggest successes because we've been able, if you like, to gather the evidence. We've got better at doing it. We're three years in. We've learned a great deal over the last three years, and so we feel that's one of our biggest successes over 2017-18.

Okay. Thank you. We've got some questions now from Julie Morgan.

Yes. Thank you. In your paper, you highlight four key challenges. Could you tell us what action you've taken to address those challenges?

Yes, certainly. Particularly in relation to general qualifications, so GCSEs and A-levels, we identified some challenges, the first of which was moving to more of a devolved system, so where qualifications are developed in Wales for Welsh learners and are slightly different to those that are taken in other jurisdictions. When you do that, one of the most important things is to try and make sure that those qualifications are trusted and that they provide the currency that young people need to progress into further study and into employment. And whilst we've stuck with the same brands of GCSE and A-level, it's important that the differences are understood and that there isn't any devaluation of the currency that goes with those brands as a result of some difference.

I think at the last committee, the last scrutiny session, we talked about the fact that we were planning to employ an HE engagement officer—so, somebody who could go out and talk to universities. We touched on this when we gave evidence around the Welsh baccalaureate as well. So, we do have somebody now who's going out and talking to universities, talking to sixth-form colleges, talking to schools, understanding the issues that they might face in terms of any perceptions that universities might have. And also, bringing intelligence back to us so that we understand what the issues might be, if there are issues that universities have.

We've done less with employers; we're actually moving on to employers more in the next year. We engage with organisations like the Confederation of British Industry, the Institute of Directors, the Federation of Small Businesses; we're looking to bring some of those bodies together more next year to try and find more ways of engaging with people. But, of course, one of the things is where the grades have remained the same with GCSE, there is one less difference. So, A* to G being the grades rather than 9 to 1, as they are in England. Employers are more familiar with that grade scale.

Other areas we've been looking at are things like the availability of resources for teachers, and the committee has had a scrutiny session around that. We're playing a part in Welsh Government's strategic stakeholder group there. Of course, the last round of reforms have pretty much gone through now; we've got the last tranche of first assessments next year, in June. So, really, we're looking at resources now more for the next tranche of changes that will be ready for the new curriculum.

The other area with difference is around just the range of qualifications that are available. We know that the range of qualifications that have been developed in a Wales-only specification by WJEC don't cover the full range of qualifications that young people might want to take. So, there are low-uptake subjects that are simply not viable—either economically or technically—for WJEC to develop. In those cases, we've allowed the England specification to be used in Wales. One of the limitations there is that they're not always available—in fact, they're most frequently not available—through the medium of Welsh. So, what we want to do is to try and make grants available and encourage those awarding bodies to make them available through the medium of Welsh, if we possibly can.

Managing the introduction of the new qualifications has been a big thing as well. We've done quite a lot of work looking at what lessons we might learn from the introduction of these new qualifications. It's one of the things—. It's about thinking about the fact that the new curriculum's going to lead to more change, so there's a real opportunity to learn lessons now that can be employed almost immediately, really, as we start to think ahead. So, we've done quite a bit of work reviewing new qualifications, thinking about how they've been implemented.

I'm just going to run through some of the key features that we've identified. There's clearly a need for early engagement. The last round of reforms were driven by an agenda in England; they were driven to a timescale that was responding to changes in England, and where Wales had made the choice to do something different, there was little opportunity but to go along with the same timelines. Moving forward, now that we've made that step away and there is a more devolved system, it gives us more agency over the future. So, we've been looking at early engagement and the need to communicate with schools so that schools really understand what's coming, and, clearly, the new curriculum provides a platform for that.

Thinking about what the design features will be and making sure there's a clear understanding in schools about what the design features will be for the new qualifications. Making sure there's plenty of lead-in time, and we've done a fair amount of consultation on this. Clearly, schools need enough time to prepare for new teaching, and there needs to be enough time for resources to be developed—especially if those are textbooks—and, again, this is something we've talked to the committee about before. So, we're looking at at least 12 months advance notice of a finalisation of specifications and the like, and there's a clear need for that.

Changing the structure of the general qualifications market, so moving to a position where WJEC are the sole providers of the vast majority of GCSEs and A-levels in Wales is a big change. That can mean a change in the perception of people. WJEC had about 89 per cent of the market before these changes came through anyway. We know one of the main reasons why people will move from one awarding body to another—and this is looking at evidence that Ofqual looked at in England where it's looking at a competitive market. The most frequent reason is where a head of department moves from one school to another school and they take the awarding body that they're used to delivering with them. Now, in Wales, where 89 per cent of the market was WJEC's anyway, it provides a more limited pool, because if a head of department moves from one school to another, they will be more likely to be taking WJEC with them. So, there was less churn, so to speak, within that environment.

Now, the move away from that being a matter of choice to it being something that is there by default—it's not by design, it's there by default—changes people's perceptions. So, what we need to do is make sure that we are holding WJEC to account and having close scrutiny of their activities to make sure that they are doing the right thing.

Lastly, thinking about the new curriculum, clearly, the new curriculum is going to lead to some changes to GCSEs. We're working very closely with the areas of learning and experience. We strongly believe that the curriculum needs to clear the way for qualifications, so we don't want qualifications to be defining what the curriculum is. So, we've given space for the curriculum development, we've been watching very closely what's been going on and we're looking, really, to follow in the spirit of working with pioneer schools in the spirit of co-construction, so taking that same ethos forward as we start to look at qualifications, and we think that we're going to be in a position to start doing that in a more concerted way from the new year, as the schools that have been involved in developing the curriculum are starting to think about what's next.

Thank you for those comprehensive answers to those four points. A theme throughout what you say is the relationship with English qualifications, and I just wondered if you could say what is your relationship with equivalent people in England. I think you mentioned earlier on the issues about 1, 2, 3 and A, B, C. How do you discuss those issues?

We've got a very good working relationship with Ofqual, our counterpart in England. There are three-country regulatory bodies, as in groups, where we come together. And three-country there—. There's a closer relationship between Northern Ireland, England and Wales, because of using the same qualifications, using GCSEs and A-levels. It naturally leads to a much closer relationship. There are four-country meetings including Scotland, but where Scotland has got a very different system, I guess the working is much closer around those brands of GCSE and A-level. And you'd expect that, because the regulators are all the owners of the brand, so we have an interest in making sure that the qualifications are functioning properly, and similar.

Our working relationship with Ofqual has got better and better, I would say. It's strengthened considerably. I think, as a new organisation, you have to credentialise yourself; you have to demonstrate that you properly appear to bodies like Ofqual that have been established longer. We've moved to a position where we're sharing much more information. We're trying to work in the spirit of having some efficiency from each other's work as well. So, where we might be investigating one thing, like we've looked at first aid, we're currently looking at food safety qualifications, we'll share the outcomes of those with Ofqual and we're sharing plans for audits of awarding bodies and the like so that we don't place too much burden on awarding bodies by all following the same topic.

Thanks. I wanted to ask about—. I think you referred to public confidence, you know, part of your role is to get public confidence and you've referred to the fact that you have an engagement officer now. How long has that person been in post?

Since September, so relatively new.

Very recently, right. So, you haven't been able to judge what progress has been made yet, then, it's too early.

No. We get useful intelligence coming back. Inevitably, when there's change, there's a bit of uncertainty around the period of change. As changes have bedded in, we're getting much more reassurance coming back to us that things have settled. So, I'll mention something—we talked about the Welsh bac a couple of weeks ago. There was much more of a sense coming back from our liaison officer that there was more confidence that universities were accepting—. Schools were feeling more confident about acceptance of the Welsh bac. So, I think there's a really useful feedback loop that we get that suggests things are quite positive.

Right, and how do you feel about the fact that 45 per cent of the pupils and teachers felt that exams were too difficult, or that 40 per cent felt there was not sufficient time for the exams?

I thought that was about right. We would always be a bit concerned if people felt it was too easy and if there was too much time. I think, from that, that's looking at our survey which is self-elected as well, so it's not a representative sample, i.e. it's not a designed sample.

People offer themselves?

They offer themselves forward, so it's a self-selecting sample. And in that self-selecting sample, it's roughly where it was last year in terms of about half thinking there was enough time, half thinking there wasn't quite enough time—

And, actually, you would expect that to be—

We would expect that.

Right, so what about—? Fewer than half of the respondents felt that standards in GCSEs and A-levels were being maintained year on year.

That was from a different survey. We do three exercises at the moment that are deliberately around looking at public confidence, and we use external bodies to do that. So, we do two surveys and we've done those for the first time this year, and those are using pretty much the same questions as Ofqual uses in its public confidence survey, the main reason being we can then draw direct parallels with England around confidence in GCSEs and A-levels in England.

So, we do two surveys along those lines. Those are of about 1,000 people. It is a representative sample of the general population in Wales, so it's people who will either be close or not close to the qualification. So, it's not designed to be parents of learners or learners or teachers—it's a general population survey.

The other one we do is with York Consulting which is more of a focus group-type activity, which will tend to have people who are closer to the system. It's interesting, there are two questions that seem to get relatively low levels of agreement. One is around whether standards are being maintained and the other is around whether marking is accurate. England gets very similar responses, so there's no real difference in the percentages that England gets.

We think it gets a low response rate because they're quite technical issues. So, things like Maintenance of standards—there are lots of different conceptualisations of what a standard is, and we know that teachers get confused about standards in relation to qualifications as well. So, we think the low level of agreement is in large part because it's quite a technical area.

I think it would be fair to say as well, if you looked at the survey results, it's only a small percentage that actually disagree. So, there's a lot of 'don't know's in those survey results.

So, you don't feel concerned about that at all.

Not particularly. I think also, it's that point of—England, who's been doing the same survey for much longer, gets similar results. 

Right, and what about the fact that only 52 per cent of the public who took part in the face-to-face interviews feel that GCSEs are a good preparation for work, or that only 44 per cent feel the same for A/AS-levels? Does that indicate that they should be more work orientated?

Do you want me to kick off? I'll kick off with this one. I think it's quite interesting when you start to talk to employers about what they expect from young people coming out of the education system. And when you ask them what do they want, they're very clear; they say they want literacy, numeracy, communication skills, team working—all of those soft skills. Within Wales, we've already changed our GCSEs in English, maths and Welsh to make them a better guarantee of the skills employers say they want. We've got the skills challenge certificate within the Welsh bac, which directly assesses those broader, softer skills that I was just talking about.

There's also space within our system for GCSEs that are a better reflection of the world of work, and that's why, within health and social care, we're looking at a GCSE and an A-level that will support those developments. We're looking at similar things for construction and the built environment, and for ICT. And with the new curriculum, there may well be more opportunities to look at this, so it's something we're very alive to and alert to, and looking carefully at how we can make our young people best prepared to move into the work of work.

Okay, thank you—

Yes, thank you very much.

—we've got some questions now about marking. If I can just kick off and let you know that the committee's been made aware of the case of three pupils of Y Pant comprehensive who had their grades changed after they'd received their marks, as a result of WJEC errors, and that this caused a lot of distress to those pupils. Are you aware of that incident, and what action have you taken to address it?

Can I just kick off and say we want everyone to get the right results from the outset? And we absolutely recognise the distress that an incorrect result causes pupils, their parents, and their teachers. It is a big and complex system, and, fortunately, errors are few. But I think, Philip, you've got detail on that.

Yes. So, we were made aware of this incident, through being copied into correspondence to WJEC. So, our normal way of handling complaints of this nature is to let things run their natural course through WJEC, and then allow a complaint to come in to us if the complainant is dissatisfied with the service they've received from WJEC. So, we were made aware of this. There were some errors by the original marker, I believe. I'm aware of the detail of one particular part of this, which relates to two candidates, where two candidates with the same surname had their marks transposed, so they both got the wrong mark, each other's mark, when the mark was copied from the script onto the system that captures marks.

Yes. I mean, I think what the committee's interested in, really, is how you handle the issue of—I think they're called 'adverse incidents', aren't they?

Absolutely. So, I think in this particular case, we've got to remember that WJEC handles something like 4 million individual assessments for learners in Wales. There are mishaps. Any system that involves human judgment—. And that's where reviews of marking come in, and there are relatively few changes of grade associated with that element of marker judgment. There are even fewer changes as a result of this sort of error where there's essentially an administrative error of either marks not being included or marks being transposed in mark sheets. Normally, any errors like that will be captured by the quality assurance systems at WJEC before results are issued. I believe, in this case, the scripts weren't returned to WJEC before results were issued. So, those additional checks didn't take place. When WJEC were made aware of the issues, they responded to them appropriately, I think. They looked at the impact and they actually checked with us their proposed actions, to make sure that we were comfortable with their proposed actions, and they corrected the grades for the candidates, which is what we would have expected them to do. So, I think there's—. We have to look at it in terms of: was there a mishap in the system, or was there a systemic failure? We wouldn't see what happened in this particular instance, however unfortunate it was, as a systemic failure; we would see it as a mishap on the part of the examiner.

Okay, thank you. The next questions are from Siân Gwenllian.

Okay, bore da. We've touched on the lack of public confidence in qualifications in Wales, and it is a mixed picture that we're seeing coming through. To what extent do you think incidents of mismarking contribute to this, with only 32 per cent of the survey respondents agreeing that marking for AS and A-levels was accurate? So, 68 per cent think that the marking isn't accurate.

So, I think, as I've touched on, it's the same with Ofqual's surveys; these are the two areas. Part of the survey results, we think—it's got a large number of people who don't know or neither agree nor disagree about the accuracy of marking. So, there's a large number of people who don't say, who don't state an opinion. What we've got to look at is the facts of things like reviews of marking. So, there's just about a million GCSEs sat in Wales and just under 200,000 A-levels sat in Wales. Of those, I think it's 2.5 per cent—we don't have data for this year yet; data for this year will be published as official statistics before Christmas, so I'll relate back to last summer's figures—I think it's 2.5 per cent of GCSEs get put forward for reviews of marking, where the marking is checked. About 2.8 per cent of A-levels are put forward for reviews of marking, and the number, or the percentage, of grades that are changed is 0.5 per cent. So, if we're looking at a system that is based on human judgment, which is the application of the mark schemes, with an error rate of 0.5 per cent, I think that's pretty good, and I think that's actually something that the system should be proud of. And I'm cautious in saying that, because you'd want it to be 100 per cent accurate—you'd always want it to be 100 per cent accurate, and you would want WJEC and all of the awarding bodies to be doing everything that they possibly can to minimise the possibility of somebody getting the wrong grade.

But there seems to be a perception—. I mean, you're explaining away—that it may not actually be a particular problem, but there is an obvious perception that is leading to a lack of public confidence. 

Indeed.

So, that is a challenge. 

And we think that's something that we can play a role in in communicating about the system. The difficulty always, with these omnibus surveys of people—I think the surveys were conducted in September—is that they'll go out and they'll reflect, often, what somebody saw in the press quite recently, and at around that time, they're more likely to see problems that are being raised. So, public perception is quite a fragile thing, but we think there is something to be done in trying to give people confidence through communication. 

Can I turn now to the—?

If you're moving on to something else—Janet, did you have a supplementary? Is it on this? 

Do you know what, I've been visiting schools in my own constituency and this has been raised with me by headteachers? So, it's not perception. 

The accuracy of marking? And are they raising reviews of marking? Are they querying the grades with the WJEC?

Very much so, yes. 

Okay. We'll look at that with interest to see what this summer's figures look like. 

Okay, thank you.

You may be referring, in part, to what I'm going to talk about next, which is the perceived injustice that 700 pupils have around the English language GCSE results of August last year. You'll be very aware of all this, I know. Can you, first of all, explain the rationale for changing the C-grade boundary for GCSE English language three times between summer 2017 and summer 2018?

Okay. I think, just to start off, I want to make very clear what our role is in all of this, which is to ensure that performance standards required to gain a particular grade are maintained year on year. That is to prevent grade inflation, but it's also a matter of fundamental fairness for pupils—past pupils, present pupils and future pupils—so that nobody is being advantaged or disadvantaged in this system. And so, I think Philip was going to go on and talk about the specifics of the GCSE—English language—but we have to talk about the system as well, and the way in which we operate.

Yes. So, the principle reason why there have been three changes in grade boundaries is there have been three exams. Question papers are different each time they're sat. Each time they're sat, despite great efforts to make sure that the demand is equal from year-to-year so that there's parity, the actual difficulty of the papers will vary slightly each time. For something like English language, that can be related to the nature of the reading texts—so the texts and the reading comprehension questions. It can be related to the topics that young people are asked to write about. So, each paper will perform slightly differently. 

Now, if you want to maintain a common performance standard when the difficulty of the paper changes each time, the way of calibrating for that is by moving the grade boundaries. So, if a paper is slightly easier than the last paper, then you would expect to see the grade boundary go up slightly to maintain the same performance standard. So, what we do as the regulator is set the method by which WJEC conducts its awarding meetings. So, WJEC established the grade boundaries, not us. We prescribe the method by which they should do that. And the method that we ask them to use is commonly known as 'comparable outcomes', and that has an assumption that, all things being equal, you would expect broadly the same number of pupils to get the same grade profile, so the same number of grade Cs and above from one year to the next.

Now, all things are never equal. The thing that's been most volatile over the last few years in Wales has been the nature of the cohort—so, whether there have been lots of year 10s for early entry. And, this summer, there's been a big impact through the washback of that early entry. So, you're seeing lots of candidates who maybe gained a grade C in English language early not come back in the summer. So, you've got a very different cohort that's gone through this summer.

I think the problem stems from the fact that some schools have been following Government advice and not putting in their pupils for the early entrance exams and actually keeping to what Government is asking them to do, but they have been caught out in that because that has led then to that cohort of children being disadvantaged. That is what I'm being told. Now, it has obviously created a huge problem. You can try and explain it all, yes, and I'm not in a position to actually judge the nitty-gritty of it, but I do know that it has created a huge uproar in north Wales in particular, and I have here a note from the education leaders in north Wales, saying that there was a meeting that you convened last night between officers from Qualifications Wales and the directors of education across the six authorities and representative headteachers across north Wales who wanted to further debate the 'injustice' that over 700 learners had received in English language GCSE results in August 2018.

They say that it was extremely disappointing that you didn't choose to attend that meeting, and they go on to say that confidence has been lost in your organisation to effectively regulate the examination system in Wales. Now, that is a huge thing to be saying, and that is across north Wales. So, there is obviously a problem of confidence, and we need to be assured that you will be working to regain that confidence, and maybe going to the meeting last night would have helped slightly, but, anyway, that's up to you. And they also want assurances that what happened this year won't be repeated again—you know, that early entry and following guidance from Government around that will be respected and that it won't result in some pupils losing out.

Can I respond to that?

Of course.

First of all, in terms of my attendance, when we established the date—. We wanted to see colleagues in north Wales as soon we could. Yesterday was the first day that was available, but then we had strong feedback that they wanted a twilight session at the end of the day rather than a session during the day. I then had to make a choice of either go to north Wales or come and see you this morning, and given the fact that this is a scrutiny session of the committee, it was safest to come here this morning. My colleagues who went yesterday I have absolute confidence in. They know the issue inside out—my director of regulation, director for general qualifications and head of research. So, I did send a senior team to go and meet colleagues in north Wales.

In terms of confidence in us, I think it's really important—. You know, we are a body for you for you, as the National Assembly, and you've established us as an independent regulator for qualifications and with a particular focus on standards, which we necessarily need to have. Not everybody will welcome the message that we can find no evidence that the award was made incorrectly. It is an inconsistent pattern across Wales. North Wales, as you say, seems to have been hit hardest in results this summer. That's not a consistent pattern, and our report shows that. There has been some difference in relation to those schools that have entered early and those that haven't entered early. Equally, there have been schools that have entered early that have done much better, and schools that have done worse. Equally, among those that haven't entered early, there have been schools that have done much better and schools that have done worse. So, there's no consistency in terms of performance. We don't see that when we look at national data. I'll be frank, if there were any evidence that we had seen through any of our investigations that the award had been incorrect, then we would have taken action—and we still will, if we uncover anything that suggests that the award was inappropriate.

There are some interesting things. You'll know that when we look at standards, using this comparable outcomes approach, you're looking to expect that there will be variations from one year to the next. So, it's not a quota system—it's not the idea that there were 66 per cent last year, there will be 66 per cent this year. There's a tolerance set around that, and for a subject like GCSE English language, it's a 1 per cent tolerance either way. Now, that's basically a reporting tolerance that says that, if it's outside of that tolerance, you need to make a case to us to justify why there's a bigger difference than one would anticipate. The system uses a combination—the awarding process uses a combination of statistical evidence and judgment from experts in establishing where the grade boundary should be. The statistics suggested—. Let me go through the grade boundaries—it's useful context. Last summer, as in 2017, the grade C boundary was at 200 out of 400, so 50 per cent. In November, it was at 204, so 51 per cent, and this summer, it was at 220, so 55 per cent.

Which is quite a big jump.

It is quite a big jump, but it's not unheard of for there to be changes.

Not unheard of but quite unusual.

Quite unusual. You've got to remember it's a new paper as well.

So, you can see why people are concerned, because there's a big difference there, and some children aren't getting over that boundary, are they?

But the thing is—

And they feel that their life chances—. They feel that they are disadvantaged.

There's a common misconception there. Because the papers vary in difficulty, what the grade boundaries are trying to do is keep the performance standard in the right place. Now, there's no guarantee that a candidate who would have got 201 last year, and therefore got a grade C, would get 221 this time. So, there's always variation in candidates' performance.

Okay. But the teachers think that they would have. That's the point.

They do.

That is the point—the teachers think that, had there been a level playing field, their children would have got a C or above.

The one thing is—it is a level playing field in terms of the qualification and the functioning of the qualification. What the qualification can't do is factor in different entry patterns by different schools. So, it can't do that. All it can try and do is provide a fair assessment and maintain a standard. That's all one can expect the qualification to do.

So, what we're looking at is the functioning of the qualification and we're confident that it has functioned properly. What we do want to do is that we want to try and move the conversation forward. So, hence the offer to engage with schools and local authorities yesterday. We have asked the WJEC to do some additional work, where we've asked them to—using this summer's answers—exemplify what that performance standard looks like at grade C. The reason we want to do that is that we want to move away from a focus on a particular mark, because the mark will change from session to session, and if schools are targeting 201 marks because the grade boundary was 200 last time, they're doing the wrong thing; they need to be targeting the performance standard, rather than a particular mark.

So, we want to do that. We're also going to do some additional analysis looking at individual item responses, so we're going to request all the data from WJEC so that we can do what's called 'DIF' analysis—so, terribly technical, but it's 'differential item functioning'. So, it's looking to see how the questions have performed for different groups of pupils. So, we might be able to look at GwE, as a region, versus other regions to see whether there's any particular performance pattern that suggests that there was something different. Now, we're not assuming that there is something different, but it shows our commitment to continue looking at this issue to see if there is anything that can be helpful.

Suzy, you've got a supplementary on this.

Just very quickly—because I'm not sure that I've quite caught it with you—I accept the point that you say that the threshold changes if the paper is marginally easier, but, as Siân said, there's quite a difference between 200 and 220 marks; it's an additional 10 per cent. So, how come the paper was allowed to be that much easier in order to allow the threshold to be that much higher?

I think the difficulty is that, however well-designed question papers might be, their actual received difficulty can change quite considerably once they're taken. So, there are ways around that; you can do a thing called pre-testing. So, that is a much more complex process for setting question papers, where you trial them for two years before you actually run with them, and, when you do that, you end up with much more consistency from one paper to the next. So, that's the way that national curriculum tests are developed in England, which is my experience of using that pre-test method. There are all sorts of issues associated with it and it's very different to the way that GCSEs have developed.

We would hope that the grade boundaries settle. This is only the third time out for a new qualification, because it was reformed, the first assessment was in summer 2017, so it has only been assessed in summer 2017, November of 2017 and this summer. We would hope to see that the grade boundaries start to settle. But that actually can start to create problems in itself, because a very settled grade boundary becomes an absolute expectation, and then, for one reason or another, if the grade boundary changes because of the difficulty of the papers, it's seen to be a shock to the system that the grade boundary has changed. So, there is some comfort in having very stable grade boundaries, but, clearly, that can be false comfort from time to time.

Okay. Something for us to watch in years to come, then.

I think so, and I just want to reiterate: we are engaging with people from north Wales. We understand the concerns that they've got and, ultimately, we care about the system and it doesn't help anybody—. It doesn't help pupils if you've got teachers who are feeling unconfident about what they're delivering. So, we want to continue and we want to move the debate forward because we think that's where we can really begin to make a difference.

There's one interesting statistic that you might be interested in. 'An interesting statistic that you're interested in'; there you go—doubly interesting. We did some modelling and we looked at what the outcomes would have looked like if the grade boundary had been at 200, the same as it was in 2017, and that would have led to a 7 per cent increase in grade Cs and above and there was no evidence to—. I've already said that we look at 1 per cent being roughly what one would expect to see; there was no evidence to suggest that there would be a 7 per cent increase in performance. I think that's one of those—you know, it's one of those tests around was the grade—. If the grade boundary had been where it was last year, there would have been a lack of public confidence in the maintenance of standards, quite justifiably, and we couldn't see any evidence to support that big a change.

Okay, thank you. The next questions are from Suzy, anyway, so—.

Oh, right. Thank you, Chair. I'm just going to move on to the new vocational qualifications and the sector reviews that you've done. One of the purposes of having Qualifications Wales, of course, was to reduce the number of vocational qualifications, because there is a massive plethora of them out there. You're creating 19 new ones at the same time as getting rid of some. What are you getting rid of that was—? Well, I suppose my question is: what's in your new qualifications that wasn't already out there in some form or another?

Ah, right, yes.

Shall I—?

Yes.

Why do we need new ones—

'Why do we need new ones?', yes.

—when you're getting rid of old ones?

So, that goes back to—. Maybe I can touch on health and social care, but then I'll touch on some more generalised issues. 

Could we start with health and social care? I think that would be great.

So, if we're looking at health and social care, we're replacing 200-odd qualifications that are out there with 19. If we go back to some of the principal findings from the health and social care review, there were unclear progression routes between qualifications, so rather than—. It was a sort of naturally developed qualification landscape rather than a designed qualification landscape, so, as a consequence of that, there weren't logical progression routes. There was a lack of qualifications at level 4 specialisms, so people were jumping from level 3 qualifications to level 5 management qualifications just to develop their career, without necessarily going down the right routes. There were inconsistencies between qualifications. There were inconsistencies in the way that they were assessed. One of the principal things was that the qualifications that were being used were qualifications that had been developed in England, and therefore quite frequently related to England legislation rather than to Welsh legislation, and, of course, this is a devolved area of policy, so there is different legislation. So, frankly, we didn't think that the current range of health and social care qualifications and childcare qualifications were serving Welsh candidates correctly, and they certainly weren't meeting the needs of the sector bodies, who we've been working with very closely in the development of the qualifications. 

As we've gone on and we've looked at other qualifications, or other sectors, there are some similar patterns there in terms of complex progression pathways that people don't understand—that was something that construction found—duplication and inefficiency, where a learner might be doing one qualification, in construction, say, then doing another qualification and going back over learning they'd already done, because they weren't articulating properly. So, there are a number of common issues that have come out.

I was just looking here; I've got some statistics, which—

Can we—? I don't want to close down what you want to say, but can we stick with health and social care first, because that's a sector of great interest to this Assembly at the moment? So, basically, you've stripped out the inadequate, the confusing and the duplication and created these new qualifications, which progress through the different levels, I guess. Are these qualifications still available in England? Sorry—your new qualifications, will they be available in England, I should say, if people mysteriously wanted to do them. 

There is no reason—. We wouldn't have any prohibit of it. So, we wouldn't have a problem with that. It's whether they would be available on publicly funded courses in England, and, of course, that would be a decision for England. We would place no restriction on the awarding bodies.

Okay. And, in putting together these new qualifications, obviously you want them to be portable. Can you give us a little bit of indication about what conversations you've had with other parts of the UK about the portability? Because this looks great, doesn't it? Simply 19—

Portability is really important, and it's something, in fact, the board talks about a great deal, because we don't want to disadvantage our learners. In terms of portability, the key players in this are actually Social Care Wales and their counterpart in England, because they are responsible in Wales for registering individuals within the sector, and we're very clear that that relationship between Social Care Wales and their counterparts in England is critical to this, but we've not heard anything that would mean that, in England, our candidates would be disadvantaged. They don't register the workforce, I believe, in the same way as we do in Wales in England, and so it is slightly different from that point of view.

In terms of candidates coming the other way, because that matters as well—somebody might have picked up their qualifications in England and now moved into Wales and want to do things. Again, that's Social Care Wales's responsibility, because they register the workforce, but what we understand they're going to do is very pragmatic. In other words, they will accept the England-based qualifications, but will ask candidates to do a module that gives them the Welsh context, because our legislation is different.

Well, you've spotted my question there. If you're training to do different legislative frameworks, which is absolutely in order, at what point do they become incompatible? 

Well, that's what they're going to do: they want to put in an additional unit that brings the Welsh context in, because it's not just legislation; there's a difference in emphasis in Wales to the emphasis in England as well. 

Have you had any indication that that might work the other way round as well? So, someone from Wales may need to do—let's make it simple—an English-legislation-based module. 

Because there isn't registration of the workforce in the same way—

But good care homes—let's just use those, for example—will be looking at what qualifications their staff have. 

Yes, and—. I think it's a difficult one in something like health and social care, in particular, where it is a devolved area of policy. So, there's a simple choice: do we, as a Welsh body, promote the development of a set of qualifications that are designed for Wales, and meet the needs of the sector body in Wales and Welsh legislation? And, if you do that, you are necessarily potentially going to have some impact on portability. But the learning that people will be doing, the competencies they're building through the workplace, are all things that become highly portable. The job of work for the sector bodies—so, Social Care Wales, in particular—is to work with its counterpart bodies in the UK so that there is a good understanding of what the qualifications are delivering. But there will necessarily be differences because it's being focused on the needs of Wales, which we think is the right thing to do. 

And that was the purpose of Qualifications Wales, I accept that. So, there's always a core of portable skills, provided that you're able to communicate those to your potential employer—. Can we move on, then, to construction and building? A similar sort of situation's arisen there. What are the key differences between the two sectors in terms of how you've had to look at what goes into the new Welsh qualification? Is the legislative framework the key driver of that, or emphasis within the sectors? It strikes me as quite different. I'm wondering if you can—. 

It's very different. There are much more diverse sector bodies in construction. We worked very closely with the Construction Industry Training Board on the proposals. We've worked closely with the British Association of Construction Heads network, which is the network of construction heads in further education. We've worked with lots of employers as well, both in the review and as we're progressing our ideas. 

Construction's a different one, so what we've found is that it's less about legislation and it's more about the broader needs of Wales. So, what we found was that construction qualifications weren't necessarily focusing on things like the housing stock within Wales, where there's—. There was limited skills development with historic buildings. So, a lot of the building qualifications have been dominated by the needs of big employers about putting up housing estates, rather than necessarily working in different contexts. Similarly, they weren't looking at things like renewable energy, and how renewables work in houses. So, it'd had become very focused on one particular area of the construction industry, which is an important area, but it's not the whole thing. 

There were also real concerns about the duplication that was going on, where a learner might be doing a qualification, then going and doing the next qualification and going back over stuff they'd already learnt, going through FE college, coming out with qualifications at the end of FE, getting a job on site and then having to go through their card—so, getting their construction card—where they'd have to go through the same things again, but on a competency basis. So, there was lots of duplication, which was making the system very inefficient. And, working with CITB, the sector bodies, the BACH network and employers, what we think we've found is a way of being much more efficient and doing things differently in Wales that will be to the benefit of everybody. We've got a very wide consensus of support from all of the people that we're working with. It's still very early days with construction, and we're still working on shaping those qualifications and on what they might look like.  

You mentioned consensus, but there was also a bit of pushback as well, wasn't there, from the construction sector. You mentioned in your evidence—sorry, in your report—that there were 25 concerns raised that you needed to be able to respond to. 

We've had—. I mean, you never get an absolute consensus on anything. 

No. What were the main areas of challenge to you, then? Let's put it that way. 

Well, it's probably worth thinking about the main areas of challenge we've had since we've gone ahead with the proposals. We went out with our proposals to restrict construction qualifications, and we had two representations, because we consult on our intention to restrict with the full range of proposals, and we see what comes back in from those, and then measure whether we're still doing the right thing once we've done that. We've gone out. We had two representations come back to us. One was from an awarding body that wanted to protect its range of qualifications, which is not an unnatural thing for an awarding body to do. And there was one representation from an FE college that had concerns about some qualifications it was delivering through a learning programme. 

Now, actually, when we had a look at those, they wouldn't have been affected by the restriction that we were looking to put in place. I think there were one or two qualifications that we thought might creep into the scope of the restriction. But if there's a good case for the restriction being amended in some way, then we're always open to that. So, we haven't had pushback since the proposals have bedded in. The next acid test will be when we go out and consult on our more detailed proposals. So, the review is looking at what is the general shape of things, how might we move forward. Now that that has become more bedded in, we'll be going out to consult on detailed proposals, we think about April, May time, so it's going to be late spring. And that's before we bring an awarding body on board in the summer. So, we're in the middle of a procurement process at the moment to bring an awarding body on board to develop and deliver these qualifications. So, there will be another opportunity. 

One of the concerns will always be difference in Wales to what's going on in England. So, there's already concern about apprenticeship frameworks as an example, where, in Wales, the apprenticeship framework requires qualifications, and in England, the apprenticeship standards require an end-point assessment, which can look quite different. The area where you will always get concerns, if you're proposing to do something different in Wales, will be those employers that cut across both countries. And where that is the case, you need to do more work to reassure them that what we're doing in Wales is every bit as good as what's going on in England, if not better. 

Or the other way around, I was going to say. 

Yes. 

Just before I go on to my final question, I just wanted to ask: in both these sectors now where you will have new qualifications, good employers will always offer continuous professional development, but CPD that doesn't necessarily come with a bit of paper attached to it; it's just that you've had two hours sitting in front of a computer doing something. You don't get involved in that area of work, at all, do you? No. That's fine.

Okay. And, then, just finally, Ann Evans, you mentioned that you're about to publish an update on the ICT. 

Yes. Next week.

Do you want to fill us in? Do you want to give us a little snippet of what might be in there?

Give you a bit of pre-notice.

Yes. Are you allowed to do that?

I think we can do a little bit of it, yes. I think it's been quite interesting the ICT work, because one of the reasons we went for sector by sector is, we thought that you couldn't have a one-size-fits-all. We thought the different sectors would bring different challenges. And we're quite right because ICT has been quite different to the sectors that we've looked at prior to that—health and social care and construction and the built environment. So, we're pleased that our assumptions were right, and that's meant we've come up with a smaller list of recommendations, which are part of that report. Do you want to go through them?

Shall I go through the highlights?

Yes. Just the highlights. 

You'll have to be quick. 

Very quick?

Yes, very briefly. 

So, highlights are: IT employers pay a lot of regard to degrees, A-levels and GCSEs, and to what we call 'vendor qualifications', so things like Microsoft and Cisco certifications. Those sit outside of the regulated market. So, they don't pay much regard to other qualifications. So, there's a particular focus that IT companies have. Those learners that are taking level 3 qualifications in IT are generally looking to progress to higher education; they're doing that. 

Employers have a strong focus on degrees, but, equally, they have a strong focus on softer skills. So, things like teamworking, problem-solving, and communication skills are some of the things they're really looking for. We found that content of qualifications, the existing range of qualifications, can be quite outdated. That can often be because, you know, still referring to floppy disks, if you can remember floppy disks. Certainly, most young people will never have come across a floppy disk. So, you know, there was outdated content in there. That’s not to be unexpected in a fast-paced, dynamic environment, things like cloud computing and the like.

Okay, can I cut across, then? Because, obviously, we need to bring this session to an end. Does this mean—? I hear what you say about these things being out of date, but there’s actually no great need to produce a whole new range of qualifications in ICT.

No, and what we’re proposing to do is to have a new GCSE and A-level. So, there’s a GCSE and an A-level in ICT at the moment, which is essentially an unreformed GCSE and A-level, which we let continue while we did our review. So, as a consequence of the review, we're going to do that.

There are some qualifications that have been sponsored by the Tech Partnership for the apprenticeships. When those come online, we'll monitor those to see what they’re like.

Okay. We're moving on to the next sector soon, so I won't ask that question. Thank you, Chair, for being so patient.

Okay. Thank you.

You referred earlier to the ongoing reforms that we're grappling with in Wales. We've got some questions on that now, and the first questions are from Hefin David.

Yes, with regard to the new curriculum. You said in your annual accounts that the implementation of the new curriculum is a systematic risk for your organisation. Can you expand on that, please?

Sure. I'm going to cut it into two parts, because I think that calling it a 'systematic risk' for our organisation is probably conflating two things. So, what I'll talk about is a qualifications systems risk, which you would expect us to be identifying, and then some specific risk to us as an organisation. So, that will answer your question.

In terms of the qualifications systems risk, one of the things that we're concerned about is what I call change fatigue. We are part way through a big reform programme that hasn't finished yet, and now we're already talking about more reforms. So, the question that we've been bouncing around is: is this a good or bad thing? Should we not talk about this yet and let the current system bed down a bit? But on balance, we've decided that it’s a good thing, because we have time before these qualifications are needed, and if we are engaging now and over the next few years, then the whole process of getting the qualifications in place and making sure you've got the training and the resources in place for the new curriculum, it gives us a better chance to do that. So, that’s the qualifications systems risk that we've identified. We also think that we've got a lot of lessons learned from the last round, and that’s fresh in everybody’s mind at the moment, and that might help us in moving forward into new qualifications that will support the new curriculum.

In terms of Qualifications Wales and the systems risk, for us, it is our resources getting stretched very thinly. Because, clearly, we already have a full programme of work without the additional reform work that is going to be needed to undertake the development required for qualifications to support the new system. We've already been in discussions with Welsh Government about it. We know we can absorb some of these costs, but not all of them. And so, that’s where you get the systems risk for us, in that if we had to move everything into looking at that, other things would have to be stopped.

So, what specific things have you done to alleviate that risk? I know you've set up a working group with Estyn and the Welsh Government. Is that part of what you've been doing to date to reduce the risk?

Yes, I think producing the risk is something that—. This risk isn't realised at the moment, because we're six years away from those new qualifications needing to be in place. What we're highlighting is a timeline that says the pressure points for us as an organisation are going to be coming later down the line. And so, I think some of those discussions that Philip's already had with Welsh Government and with others are helpful.

The working group is very useful for us to develop our thinking. What we're planning to do is to communicate our approach more widely, probably at the beginning of next year. What we want to do is we want to be working in parallel to the curriculum being finalised, to develop the proposals for GCSEs, and to make sure there's really strong engagement with schools as we do that.

You said earlier that you didn't want the qualifications to define the curriculum, but you do need, by 2020, to confirm a future qualifications offer with the Welsh Government. So, although you say it's six years, by 2020 you've got to define your future qualifications offer.

Yes.

How are you going to make progress to that if there's this change fatigue, as you mentioned, but also the need to make progress quickly?

We're planning to run a consultation in September of this year, so we're going to use the time period when the area of learning and experience working groups and the pioneer school network is moving from developing the curriculum into the next stage. We're going to be looking at forming a sub-group of those people who are particularly interested in secondary education and qualifications to start looking at the shape of that. We're going to engage a panel of experts to help us as well to develop proposals and then consult on those proposals in September, so that we are ready for January 2020. Our intention is to be able to get awarding bodies on board. We've written to awarding bodies, the GQ awarding bodies, to see who's interested in developing these qualifications. So, we've started that piece of work. What we want to be in is a position from 2020 to really be working on the development of these qualifications. What we would like there to be is, if not final specifications, then at least a good shape of what these qualifications are going to look like for September 2022 when schools start teaching the new curriculum. That will be well ahead of first teaching in September 2025, assuming that the first GCSE cohort will be summer 2027.

And in many ways you're reliant on the pace of the development of the curriculum. Do you feel that it's developing at a pace that's appropriate?

I guess it's developing at the pace that we expected. We were expecting it to be published in April of next year. We were thinking, ideally, we probably would have liked to consult on our proposals a little bit sooner than September. Actually, if it's April, it only gives one term for schools to really get their heads around what the new curriculum is—those that haven't been deeply engaged with it. So, it would be disingenuous to consult on our proposals before they've had a chance to get their heads around the new curriculum. So, inevitably, we are a bit impatient, we'd like to get on with things, but we have to go at the pace—

I was just thinking that 2027 seems a long time.

It won't be.

It gives us time to think about doing it properly. Of course, the curriculum is being implemented with year 7s in September 2022, so if you follow that cohort through, it means that they will reach year 10 in 2025 and they will reach year 11 in summer 2027 for assessment. 

Can I just come back to that statement that you don't want the qualifications to influence the curriculum? Is it realistic to think that's going to happen? Can we expect teachers to cover the qualification and the curriculum—the curriculum to develop with the qualification? Isn't it more likely to be kind of cyclical?

It'll be a mix. It'll inevitably be a mix. What we wouldn't want to see—and when I say 'we' I guess I'm speaking for education bodies more widely. This might be a question for your next session. We wouldn't want to see a reductive model of education where it's just about teaching to the test. There's plenty of evidence from all sorts of spokespeople on education at the moment that that's a real danger that I think is being realised as well. So, if the new curriculum is going to be transformative, it needs to have a different culture around it, and the different culture has to be more expansive in its model of education. 

One last question. Can you just give me one example of how you've engaged with a pioneer school?

We go to all of the AoLE working groups where they're developing the curriculum. We go to those as observers—that's our official status—because we don't want to be overly influencing the development of the curriculum; that's not for us to do. So, we engage with pioneer schools through those groups. What we will be doing more actively is engaging directly with them and forming our own groups from January onwards. 

Right. Okay.

Okay. Thank you very much. Janet. 

Thank you, Chair. How do you envisage the 10 or so individual qualifications that pupils currently take at the end of compulsory schooling being aligned with the six areas of learning and experience that the new curriculum will centre on?

We've done some mapping. We've mapped the existing offer of GCSEs into the six areas of learning and experience. Of course, that's quite interesting because one subject domain might cut across more than one area of learning and experience. So, writing—is that creative and is it expressive art or is it literacy? There are things that cut across—elements of food science come into health and well-being and also come into science. So, there are areas that cut across more than one AoLE.

We've also done an interesting piece of work—we've tried mapping them to the four proposes, so if the four purposes are the core of what the curriculum is trying to achieve, how do qualifications map to them? And that looks quite interesting as well.

There are inevitably some areas of these that don't have as much coverage—so health and well-being don't have as many qualifications that would naturally fit within them. What we're planning to do is to do more work around that, but it's also trying to work out what Welsh Government's policies are going to be. So, at the moment, if you look at the 10 GCSEs that a learner might be taking, many of those will be taken up with the core offering of English, two maths or Welsh, Welsh second language, two sciences, Welsh bac. By the time you take that core as it is at the moment, it only leaves two or three subject options, which is the common concern about a narrowing of the curriculum in key stage 4.

We will be working with Welsh Government to understand what their policy objectives are going to be. Are they going to be looking that a learner takes a qualification from each of the AoLEs, or are they just looking for curriculum coverage across each of the AoLEs? Those things have yet to be defined. Now, some of those things will have an impact on the viability of some subject areas. So, suddenly, if there's a requirement to take something from the expressive arts area of learning and experience, that will increase the number of learners that are taking GCSEs in those sorts of subjects.

Can you share your views on what the impact of the introduction of T-levels will be in Wales?

And relatively brief views, because we're running short on time, please.

Okay. Shall I kick off? Just to put this in its context, the T-level programme in England was a response to the Sainsbury review. It is a programme that is made up of a technical qualification, literacy and numeracy, some other trade-specific qualification and work experience. Now, our understanding is Welsh Government has no wish so take the whole programme. Where we are very interested is in those technical qualifications. And I think it would be fair to say, Philip, that you've done a huge amount of work in this area, working with the Department for Education in England and now, increasingly, with the Institute for Apprenticeships who are going to be taking this over in England, to try to make sure that, in Wales, we have access to that technical qualification bit of the T-level programme. Do you want to add anything? Do you want anything more, because I'll turn to Philip?

Well, just a quick one, really, and a quick response, please. How have you sought to influence the development of T-levels in England, and how are you preparing for the impact that they may have in Wales?

We've been working quite closely with the DfE, as Ann said. One thing is: what they're trying to do with T-levels is very similar to what we've done with health and social care, in terms of restricting and commissioning. So, we've been working with the DfE to share our learning in what we've done in that area. We've engaged with consultations and we've worked with them on their procurement documents to try and open the door to those technical qualifications being available in Wales.

Okay, thank you.

Thank you. Hefin David on vocational qualifications.

Do you think it's acceptable that five awarding bodies were so non-compliant with your regulations that they posed a significant risk to the validity of the first aid qualifications they were delivering?

The simple answer to that is 'no'.

So, how do you deal with it?

Okay. When we found the problem, what we have done is ask those awarding bodies to produce an action plan. This is not some kind of sterile action plan that sits on a shelf somewhere and nobody looks at again. We monitor how they are going to improve their systems. We've monitored all the actions that they have signed up to, and part of that is not just about improving the assessment that they were doing, but improving their own quality-assurance processes, so that you can guarantee the ongoing validity of those qualifications. So, there's a huge amount of work being done with them. All of those actions have now been signed off, but, clearly, that's an ongoing issue for us, because we don't want to see slippage of them falling back in any way. And having looked at first aid, I think as Philip has already mentioned, we're on to look at food safety qualifications and then, ultimately, health and safety qualifications. These are small qualifications with very large numbers of learners in Wales taking them, and we think it's absolutely right that we're looking at them, to make sure they're as good as they can possibly be. But, no, we weren't happy.

How are you sustaining the monitoring and ensuring that the improvements continue?

So, we don't release them from the undertaking until they've done everything that they were supposed to do.

Okay, thank you. We've got some other issues to ask about now. Before I bring Suzy in, can I just ask about modern foreign languages? The figures are becoming increasingly alarming, particularly at A-level, and are surely reaching the point where it's not going to even become sustainable to offer those qualifications in Wales. What is your view on the situation, and how concerned should we be as a committee?

Certainly, we've had conversations with WJEC over the viability of the qualifications. We're working quite hard to encourage WJEC to keep offering them, especially German, where we're down to about 70 learners. That's clearly not economically viable for them—it's on the threshold of technically viable—because it's difficult to award subjects with very, very low numbers. There's clearly—. Welsh Government has got Global Futures as an initiative that it's trying to do. In terms of qualifications, what we want to do is to keep the qualification offer there, because without a qualification offer, it really is the death knell, certainly for a subject like German, as an example, because if German A-level were to go, it would reduce the number of people doing GCSE, because it wouldn't lead on to an A-level for those that wanted to do it.

'Where's the solution to this?' I think is the interesting question. Maybe the new curriculum has a part to play in that. If it's more about learning language skills, whether that be English, Welsh and a modern foreign language—a much more blended approach to learning languages—that might be something that improves the viability of those areas. I think, for us, it's keeping life support on these qualifications as far as we possibly can, until there is a change.

Okay. Suzy.

I think we could have another session on modern foreign language altogether. Thank you.

I just had a couple of quick questions about your budgeting and audit. You had an underspend last year, a smaller underspend this year—does that mean you'll have no underspend next year?

We anticipate the underspend will be lower this year, and we hope to minimise it as much as possible, but we can't guarantee there won't be a small underspend at this point, and you need the rationale for some of that as well, which I think is important. So, we want to make best use of the funds. We are very careful in the way in which we use funds, and we're also totally transparent with Welsh Government about the use of our funds, because we think that's a fundamental way a public body should behave.

We have a few challenges that are unique to a particular year. So, this year, when we talked about health and social care, we didn't actually talk about the fact that we have delayed the implementation of some of those qualifications. That means the spend on their development is next year, not this year, so things like that will happen. But we have two perennial problems that we are getting better at handling, I suppose. One of them is our Welsh-medium assessment grants. So, our Welsh-medium assessment grant pot is there at the start of the year, and awarding bodies bid for money to support them to provide assessment through the medium of Welsh. What we've discovered over the last two years is that those bids don't actually turn into a full claim, so there's a drop. But you can't take the money out of the pot until you know how much money is going to be taken from it, and so that's definitely one of the issues. Now, as to why that happens is a long and complex answer, which I'm not going to go into now. The other thing that we've got better at understanding is the implication of our very small staff turnover. We're a small organisation, but, inevitably, staff will leave, and that does mean that sometimes, on our staffing budget, by the time you've replaced that person, you've had a slight underspend during the year. So, it's an area that the board focuses on, virtually every meeting. We talk about money a great deal, and we're getting better at it. Three years in, we now have historical information about how our budgets are being spent. 

Looking forward, we've got—it's called 'incremental drift'. As all our staff climb their scales, it puts more pressure on the budget. There are potentially big pension changes coming, which will put more pressures on our budget, and we've also talked about the new curriculum and the pressures that that would put on our budget. So, we're forecasting a very different scenario in the future.

Okay. Where does this appear on your risk registers at the moment? One of the concerns we had a committee when Qualifications Wales was in the process of being created—we had questions about the finance at the time, which is not about you as you are now but in the process of setting you up. So, we still have that question about judging how much money you need, I guess. So, as I say, where does this appear on your risk register at the moment in terms of governance? 

It's on our corporate risk register.

It is. We manage a medium-term finance plan, so we're always looking three years out on what the pressures might be. 

What colour is it?

At the moment—. Actually, do you know, that will depend on what happens with pensions? 

Okay. It's a perfectly reasonable response. 

So, we've looked at what the potential increase in employer contributions will be for the civil service pension because we're part of the civil service pension scheme. That could be quite crippling for us on a relatively small budget. 

Okay. Well, a similar question, on governance: the grants that you give out—you mentioned quite a lot goes to Welsh language assessment. What's your process for monitoring value for money on that? Without a twenty-minute explanation, obviously. 

Quite simple: we look at claims very closely. Most of the grants that we give out are for Welsh-medium translation. We pay a standard rate, which is between £70 and £90 per 1,000 words, which is the going rate and the rate that Welsh Government will pay. So, we pay the going rate and we make sure that everything is done. We've got full visibility and transparency of what's being claimed for. 

Okay, thank you very much. And just very, very quickly to finish: obviously some of this underspend—you may still have an underspend. What do you expect it to be going on next year? You mentioned the delayed introduction of the qualifications for health and social care. Do you think that'll eat up any other money, as well as the incremental drift that you mentioned? 

Yes. I mean, clearly, when you move work from this year into the next financial year, that's part of our budget planning for next year. 

You don't think it's likely to be an indication to Welsh Government to reduce funding, then? 

We would hope not.

You hope not, yes. 

We're very open with them.

We're working hard with them to let them understand that we're going to be under pressure for our budget in the future. 

Yes, okay. Lovely, thank you. 

Okay, thank you. If there are no other questions from Members, then we have come to the end of our session. Can I thank you for answering all our questions? It's been a very informative session. Thank you for attending. As usual, you'll get a transcript to check for accuracy. Thank you very much.

The committee will break until 11.15 a.m.

Welcome back, everyone, for our next item, which is our session on the Estyn annual report. I'm very pleased to welcome Meilyr Rowlands, Her Majesty's chief inspector, Claire Morgan, who is a strategic director of Estyn, and Jassa Scott, who is also a strategic director of Estyn. Thank you very much for your attendance today, and we're looking forward to hearing what you've got to say.

If I can just kick off with some questions about the new common inspection framework—whether you feel it has significantly changed the way schools are inspected and whether there have been any identifiable changes in outcomes.

We introduced a new way of inspecting, as you know, in September 2017, and that was at the end of the previous cycle of inspections—the seven-year cycle. We obviously refresh the framework every so often. We do it in a way that we hope improves the education system, and we do it in a way that we think improves the inspection process. In particular, this time, we wanted to make sure that inspection is as good a professional learning experience as it possibly can be, under the circumstances, for the schools and the other providers we inspect. But we do it in a way, obviously, that still meets the requirements of legislation, and we work within that legislative structure. So, we don't expect it to change what we can report to you, for example, at a high level or that the way we inspect makes a big difference to the actual inspection outcomes themselves. So, for example, what we did this time was we simplified and streamlined the inspection framework—we reduced the number of aspects by amalgamating them together from 29 to 15 aspects. We simplified the grading system. We had 15 different grades at three different levels in the previous cycle, and we've simplified that to concentrating on the five main areas, which are set out in legislation: standards, well-being—those sorts of things—quality of education and leadership.

So, I think we can be very confident that we can make sensible comparisons from year to year. I mean, obviously, we change our inspection guidance from year to year as well, to make sure that it's totally up-to-date with the changes and the expectations out there in the education system, which constantly changes. But the big change happens when we change the framework at the end of the cycle.

So, to answer your question briefly: we didn't expect it to, in itself, have any major implications for the outcomes of inspection.

Okay, thank you. And you've made some changes as well that make it more difficult for schools to predict when they're going to be inspected. Have you noticed a significant change in schools' levels of preparedness for inspection as a result?

I think the changes, the recent changes, have gone down very well with schools, and I think generally people have thought that making inspections less predictable—when they are going to occur—has also been warmly accepted. That goes back to, I think, 2014. So, we've had a few years now where people can't really predict when they're going to be inspected. I don't think either that that has made a huge difference, and we wouldn't expect it to have a real big difference to the inspection outcomes in the sense that the big things we are looking at: standards, quality of education, leadership—those aren't things that schools can easily change in preparation just for an inspection.

But I do think it gives reassurance to parents that schools aren't over-preparing for inspection, and also that the schools themselves don't—some schools might or other schools don't—but it's possible that some schools might have, in the past, over-prepared for inspection, and having this unpredictability does help that as well.

Okay, thank you. And the format of the report is slightly different this year. Can you just briefly explain what you've done with the changes and also what we can expect next year to coincide with the Programme for International Student Assessment results?

Yes. So, what we've done this year is that the inspection report that you have in front of you consists of sections on each of the sectors that we inspect, looking at those big areas of standards, well-being and so forth. So, those are the requirements for us to report to you in legislation, and I think that will be a constant each year. What we've done in previous years is, in addition to that, we've added a thematic section focusing on a particular topic—the curriculum or professional learning, or leadership or something like that—and that's the area that we didn't do this year. We've done that in order to address one of the recommendations in 'A Learning Inspectorate', which suggested that we have a lighter-touch annual report. So, every year, other than the PISA year—so, every three years—we have a bigger annual report that is a kind of 'state of the nation'. Next year—so, next December—the PISA results will come out and so we will publish an annual report. It'll probably go back, then, to January or February to give us time to process the PISA results. But the idea is to put the PISA results and our inspection findings in a wider context and bring everything together in a more substantial report every three years.

Okay, thank you very much. We've got some specific questions now on early years from Suzy Davies.

Thank you very much. I'm just asking questions about the non-maintained sector, but for the record, could you explain what the non-maintained sector is? It's more than just non-state nurseries, isn't it? Can you explain what the sector includes?

Yes. They're all kinds of nurseries and day nurseries, those sorts of—cylch meithrin. They are non-maintained, which means they're independent. They are not supported by the local authority, although they are financially. But the local authority does have some responsibility for supporting them in terms of guidance and that sort of thing. So, these are children before the age of statutory education.

But it doesn't include home-schooled children.

No, sorry—no, it doesn't.

That's what I wanted to—. That's fine. I just wonder, then, if—. Obviously, we were talking about childcare fees yesterday, so can you give us an indication of the main challenges you see in developing a joint inspection framework for education and, of course, care?

Yes. I'm sure my colleagues will go into all the challenges involved in that, but I want to emphasise, really, how immensely rewarding that process actually was. We worked with Care Inspectorate Wales for a long period to develop that, and we learnt an awful lot from one another. It is a challenge—we are two separate inspectorates with very different legal roles and responsibilities. They're a regulator and we are not a regulator—we are an inspectorate. So, bringing those two did have its challenges, but it was hugely rewarding. We've learnt a lot and the outcome has been an inspection framework for this non-maintained setting sector, which I think is very, very positive. We've obviously been piloting it, and it's gone down really well. I think it was introduced in order to reduce duplication. I think that's a bit of a red herring, to be perfectly honest. There wasn't a huge overlap in what CIW were doing and what we were doing.

I think the big benefit that we found from working together, other than institutional learning, has been that for these providers—these early-years settings—what they now will get is an integrated inspection. They will get a view from CIW and Estyn simultaneously, and I think that is a much richer kind of information that they will get and will help them move forward, rather than two separate inspections.

Okay, thank you. I must be honest—personally, I was less exercised about having a common inspectorate, if you like. But, as a result of—. I was aware of nurseries in particular saying that they had two separate inspections and that, occasionally, the recommendations of each would contradict each other. So, I can see that this is a move on. Have you, as a result of this process, been able to introduce anything new into the inspection framework, or has it literally been trying to weld the two together in a meaningful way?

I don’t think there’s anything particularly new, no.

No, well, that's great. I just wanted to ask you, as well, about the results that they’ve had from Estyn. I notice in one case that, in actual fact: 

‘in most of the cases requiring Estyn monitoring, children do not make enough progress in developing their skills...because practitioners do not use assessments well to identify and plan for children’s next steps in learning.’

In the process of coming to that conclusion, did you decide that this was something that’s missing in the initial training for these practitioners, or just some bad habits that develop over time?

It's a good question. I think one of the challenges—. This is a sector that is very, very high performing, by the way. I should explain that before we maybe nit-pick a little bit. And they're doing that in quite difficult circumstances. Because one of the advantages of being in the maintained sector is that there's no competition, and schools come together and share good practice. And that can be a little bit more difficult in this sector, depending on—. There are umbrella organisations for some of these settings, but not for all of them. So, sharing good practice is a challenge.

I can’t say that we’ve identified that as a particular challenge in initial training of these, but what I’m saying is that the continual professional development is a bit more of a challenge in this particular sector. And we’ve been working quite a lot with the advisory—. As I said earlier, the local authority does have some responsibilities, and it does have an advisory teacher who works—it’s a relatively small element of resource, but they do work with this sector. And we’d be working with them as well. So, I think the sector is getting better at sharing good practice in that way together.

Just a final question, have you noticed—? Or are you able to say, I should say, any specific bits of best practice that you’ve seen in Flying Start settings that would be valuable in others that aren’t?

I'm not sure if there are any, colleagues, on that. I'll look into that, and I'll—

It would be quite useful for us, to help identify how Flying Start's going, really.

Yes. We don't inspect that, of course.

No, no, but the settings in which there are some Flying Start children.

Yes, sure. Yes, I'll have to come back to you on that.

Okay, thank you.

Okay? Thank you. The next questions are from Siân Gwenllian.

Recently, we've started to see more good practice and, really, a better understanding of what good foundation phase pedagogy looks like. So, it's certainly an improvement on the picture from last year. We're also seeing signs that there's better use of outdoor spaces as well, which was one of the areas we mentioned last year that there was a weakness in.

Now, in our interaction with the regional consortia, local authorities and schools, many of them tell us that some of the recent improvements have been triggered by one of our publications, which was on active and experiential learning, which we published around this time last year, in December—no, yes, last year, December 2017. And we know that the materials, the report and also the film, have helped to raise the profile, I suppose, of foundation phase. We know that the actual report was downloaded about 3,000 times, and there's quite a useful film that goes with it, and we know that that's been viewed 3,500 times. So, I think raising the concern last year about the somewhat lacking progress on foundation phase and the report and the professional learning have had an impact. So, it's a more positive picture than we mentioned last year—

Absolutely. I think there's some way to go, and I think where there's some more work that needs to be done is where there's too much of a focus on formal assessment, still, at the expense of developing the foundation phase approach.

Yes, certainly.

Suzy, did you have a supplementary?

Claire, I don't know if you want to add there.

Certainly. We're talking about a fifth of primary schools, but it is a very important issue. It's partly a leadership issue, that school leaders are not ensuring that the staff have the skills to teach reading properly. As a result then, pupils' reading doesn't progress as well as it should. Often, pupils are asked to answer questions that only require them to give information rather than give opinions or make evaluations. So, it's at a much lower level than perhaps they are capable of.

In these schools, we're also concerned about the range of vocabulary they're using. So, they're not having enough opportunities to develop that and to gain even more confidence in their reading. So, it certainly highlights the importance of widening pupils' opportunities so that they hear and use quite a wide range of language.

We're also going to be doing some work, similar—. I mentioned earlier that we had done some work to support the foundation phase. We've done a thematic review, and then conferences, and we will do the same for reading next year. So there will be a thematic review of reading, and conferences in all the four regions then towards the end of 2019. So, we're hoping to work closely with practitioners to identify some of the best practice in developing reading skills actually to try and address this issue.   

Yes, absolutely. I think, talking about leadership, it is actually leaders' responsibility to ensure that their staff have access to the professional learning to make sure that they can make progress in this. You're absolutely right. 

But can new people coming through the system, newly—. Is it still a problem? 

Do you want to go on to secondary schools, please? 

On the national academy, we're pleased that they've prioritised serving headteacher development, because obviously they've got quite an ambitious vision for developing leadership in Wales, but there's an acute problem where we need to make sure that our weaker headteachers have the support they need to bring about improvements. We're also leading a project with the OECD to develop a national approach to self-evaluation for improvement, and we've also identified leadership as one of the priority work streams on that, so we hope that that will go some way to support the system to improve as well. But there's certainly a lot of challenges in leadership, as Meilyr said, in secondary at all levels, and it is complex. The new curriculum will give us a great opportunity to engage learners in secondaries far more, because we've seen a narrowing of the curriculum. There's very little choice for learners in secondary in some schools. So, this is an opportunity, I think, to engage the learners but to also address the issues in leadership in secondary. 

I think where we see that assessment needs to improve, it tends to be quite superficial, or it's a routine that is done without enough thought about how that particular feedback is helping a pupil to make the next step. And I'd agree with Meilyr—often, the most straightforward, immediate feedback is what helps children to just move on. So, we're not saying that there needs to be more assessment, but we need to go for quality rather than quantity and to ensure that students are clear on where they are now and what exactly they need to do to make progress. So, it's actually focusing on that, really.

Yes, you're right that, for the third of schools where we're saying that that's not good enough, that's certainly too much, and I know that the committee itself has done lots of work recently around this. What we do find is that, in the two thirds of schools where this is happening really well, staff across the whole school take ownership for young people's well-being and emotional and mental health, that they have a good understanding of some of the issues that they may be facing, and that they and the pupils in the school have positive relationships with each other that can be supportive of their well-being. But what we do find is that it seems that it's becoming increasingly challenging for schools to provide the support that is needed for young people, especially when a lot of the issues that find their way into the school—they are actually happening sometimes outside of that school day, but having an impact on the learning that is able to happen in school.

So, generally, we find that there are good relationships with specialist services, that young people do have access to school-based counselling and so on, but we do find that there are frustrations sometimes in gaining access to perhaps more specialist services, as I know that the work that you've done on child and adolescent mental health services has found. And I think it's important that staff continue in schools to develop their understanding of young people's emotional well-being and how that can impact on their learning. And I think we're seeing, in a few places, some really positive pilots happening that are supporting that. We're certainly continuing to look at this, so we're doing a thematic piece of work at the moment around well-being more broadly, and the hypotheses that we're testing out there are those things around whole-school approaches. So, where it does happen well, it's about those whole-school approaches, about strong leadership, it's about those relationships, it's about good staff well-being so that they're in a good place to support pupil well-being. It's about the curriculum and helping young people to understand what well-being is. And then it's about that care, support—that whole environment, really, and how that is supportive and is a positive learning environment.

So, there's certainly more to do, but I think, with the ministerial task and finish group, the new curriculum has a potential, I think, to embed some of that in the core learning that's happening, and then those wider projects that start to try and make as strong as possible the links between what's happening in school and those wider services. So, there's certainly—there is more to do, but I think we're optimistic that's been recognised and that there are some things in place to start addressing it.

Thank you. You referred in a few of your answers to the new curriculum. Can I just ask you whether you are satisfied that we are on track to deliver the curriculum according to the timescales that have been set out by Welsh Government and whether you have any areas of concern at all around the development of the curriculum?

Colleagues can maybe help and say about some specific areas, but, generally, my understanding is that they're on track to deliver the curriculum to the timescale that has been agreed in terms of, first of all, it coming out for consultation in April. It's not so much, 'Are there going to be milestones missed?'; it's the quality of what's happening, isn't it, in schools. So, as I said earlier, there are schools that have been very intensely involved in all of these developments. The whole pioneer school idea is that the curriculum is co-constructed by those practitioners, and I think that's an excellent idea, and it's one that is getting quite a lot of attention internationally. People ask us what's happening in Wales, from other countries, and that's very positive. 

The point I was making in the annual report is that all schools now need to realise how much work is involved in preparing for the new curriculum. It's not a question of just picking up a folder; it's a new way of working. I think there's a lot of preparation, a lot of staff development, that needs to be done. It's a new way of working; it's a new culture, really. It's more of a framework, as you all know, and it's putting more responsibility on the school and on the teachers themselves to develop what they're teaching. So, we shouldn't, any of us, underestimate how much work is needed to prepare for that. That's why we've got some resources in the annual report for those schools to self-evaluate where they are on all those elements that need to be in place for them to be able to deliver the new curriculum.

Okay, thank you. We've got some questions now on provision for vulnerable learners, from Julie Morgan.

Thank you, Chair. Good morning. I wanted to ask you first of all about special schools, and, from the data that you give us, it does look as though they are relatively performing well. Would you confirm that, or—?

Yes, they are a sector that work very well together. They share good practice very effectively amongst themselves. The only slight caveat, I would say, is that those schools for pupils with social, emotional and behavioural difficulties tend not to do quite as well as those schools for children with severe learning difficulties, and I think, when you go into special schools for children with learning difficulties, nearly always it's a wonderful experience—it's a really, really positive atmosphere. They're very engaged with parents and the community. There's a lot of support that comes in from all kinds of organisations. It's very, very positive. Maybe society doesn't help those children with emotional and behavioural difficulties in quite the same way. There isn't that kind of buzz for those children with mental health problems, as we were talking about earlier. I don't know if you want to add anything. 

I'd agree, and I think Meilyr touched upon the special schools working well together, but I think some of the elements that we've talked about as we've been going through the discussion today that make effective schools—around strong leadership, around a really strong focus on professional learning—they've been obviously there in some of our maintained special schools for quite a while. And, so, over a number of years, they've managed to build really strong staff teams with really strong knowledge and skills, which has created a really strong learning environment, and I think what we see in some of the schools for pupils with social, emotional and behavioural difficulties—and we may touch upon this again with pupil referral units as well—is that, sometimes, the investment in staff to be able to create that kind of learning environment, where their knowledge and skills are valued and they feel part of a really strong team, is not always there, and, sometimes, the firefighting that they're doing means that they're not able to access some of the opportunities that are available to them through consortia and so on to network and to access training, and I think we see that with—. You asked about the maintained special. We've seen an increase in the number of pupils who are accessing independent special schools—so, publicly-funded placements into independent special schools—and I think they face some of the challenges that we talked about with the schools that are predominantly for social, emotional and behavioural difficulties. Quite often, they don't necessarily attract strong leaders, keep staff for long enough, to build up that strong staff team and invest in their skills. 

You're talking about independent—

What I'm saying is that some of the things that we see in a few of the maintained special schools— the lack of some of those positive factors that we're talking about with maintained special schools, we're also seeing that in the independent special school sector as well. 

Right. And did you say that there is a growth in the use of independent special schools? 

There's a growth in the number of pupils who are accessing education, and it's been steady over the last few years. 

And why is that? 

I think because, within the maintained sector, there perhaps aren't always the number of places. I think we see, when we're out inspecting in maintained special schools, that there is a pressure on some of the places in there, and we have actually seen some new schools open in the last couple of years, which we haven't done for a while. So, I think there is growth in the maintained special school sector, but, in the interim, I think local authorities have been forced to commission places within the independent provision instead.

Right. And do you feel that—? I think you report that pupils with special educational needs make suitable progress against their targets, but the targets may not be challenging enough. Could you expand on that?

Yes. I think that's particularly in mainstream schools. I think what we find is that, sometimes, although there may be longer-term targets, the milestones that they need to get there aren't always in place, and I think that we do find that sometimes those targets aren't challenging enough. Whether it's a lack of aspiration, I don't know. I think it's about understanding the possibility and coming back down to really good assessment and understanding of needs. But we've seen some positive progress over the last year in things like person-centred practice and how that's having an impact on planning for young people with special educational needs. We published a thematic report about readiness for the ALN reforms, and that, in particular, picked up that we're seeing pockets of good practice in taking that more child-centred approach to assessment and planning and target setting, which means that, hopefully, then those targets are more suitable for that young person and take account of their needs and interests to be able to help them to achieve as much as they can.

And you mentioned pupil referral units. So, I don't know if you'd like to comment on how they're progressing, because I think there have been some worrying reports on that, I think.

Yes. I think this, again, is a sector that's kind of related to those schools for children with social, emotional and behavioural difficulties. So, they have the same challenges as those special schools, but they've got additional challenges as well, in that they're not schools. I think, despite the work that the education otherwise than at school task and finish group did, which started to improve the status of pupil referral units, I think there's still more to be done. They don't have a headteacher, they have a teacher in charge, and they don't have a governing body, as such, and so forth. I think we do need to re-examine why these are not schools at all. The danger of them not being a school is that they sort of pop up and they disappear. And as you know, and as Jassa was saying earlier, you need that continuity to build on to attract good staff, to have a common vision from the leadership to build that. If it suddenly opens and then suddenly closes, that can't happen. And even if that doesn't happen, I'm sure the staff are worried that it might happen. So, I think you need that continuity. I don't know if we want to add anything.

I think that, traditionally, PRUs haven't performed well. We've seen some positive signs in the last couple of years; we've had some PRUs that we visited and inspected five or six years ago, and we've really seen that they've gone from strength to strength. I think that has been supported by having some stable leadership, having some stable structures in place, and therefore, being able to keep and retain staff and build their way of working there. But, unfortunately, we have seen recently, during the last year as well, PRUs that were in a statutory category on the last inspection that made some good, early progress, they came out of a category, but are now back in a statutory category. So, I think there are still some concerns about how that support is sustained over time to help them grow as a functioning provision, which can then do the best by the young people who are there.

Do you think they're a system that ought to continue?

I think Meilyr mentioned that some of PRUs perhaps would benefit from being more, perhaps, a type of school, but I do think also, what we've looked at is whether, actually, some of these young people could be supported better in secondary schools. We've talked about secondary schools a little bit already today, but perhaps if some of those secondary schools were strengthened, and that support was there at an earlier stage, there may not be a need for the level of provision that we have in PRUs at the moment. Where they work well, they're really effective provisions that provide really good support, quite often for a short period of time, at a time when it's needed, so those pupils can go back into mainstream school. But I think there are too many pupils who stay for too long in pupil referral units, who perhaps could be better supported in a different way.

Thank you.

Thank you. We've got some questions now from Hefin David.

Your report holds up Denbighshire as an exemplar for leadership and driving school improvements, and it mentions, of course, the issues in Pembrokeshire, Powys and Wrexham. Can you give us an overall picture of the 22 authorities in Wales and the extent to which they are driving improvement as a whole?

Okay, well, as you know, we recently started re-inspecting local authorities, so that's starting to give us a fuller picture of the detail of how they're working with schools. You mentioned Denbighshire, which was one of the pilots that we did last year, along with Neath Port Talbot, and what we found there, in terms of leadership, was that there were some really positive working together across the services in the council to co-ordinate some of that support. Meilyr talked earlier about, actually, in terms of supporting secondary schools in particular to move forward, there is that need to co-ordinate the support of various agencies, and particularly some of the statutory services provided by the local authority around inclusion and special educational needs with the school improvement support. So, we've got a good picture there.

Generally, from a consortia level, those consortia are developing. We revisited them during the last academic year to follow up on the thematic work we'd done the previous year, and we found that, in most of them, there was good progress. We did have some concerns about the progress in ERW, in particular, and we remain concerned that they aren't providing perhaps the co-ordinated approach and delivering on the plans that they did have in place at the time of our visit, and we've written to them to ask for updates. In terms of the other local authorities, we don't have an in-depth picture, because we haven't done those inspections, but overall they are generally working with those consortia to provide some of the improvement support that we would expect to see.

Just before I get on to the consortia—I wasn't going to get on to the consortia quite yet—the case study you want from—. You want Denbighshire to produce a case study, you say, outlining good practice. So, I imagine that case study you intend, then, to share with other authorities and you expect them to take up very similar practices.

Well, we wouldn't normally say that you have to organise your services in a certain way within a local authority, because we see that they can be organised differently and be effective, but the way that Denbighshire have organised their services has certainly helped them to create links between different services, which has brought a more joined-up approach, more seamless services and better understanding between them, which has helped them to support schools. But we wouldn't say that, by doing that model, you would necessarily achieve that, but they have—

And when would that case study be available? And would it be publicly available?

It will be publicly available. It may be on our website now—I should have checked before we came—but we were certainly following it up.

Okay. I'll look out for that one, then.

And then—you've mentioned ERW—you said,

'in the 18 months following the initial inspection of ERW, the executive board and joint committee too often accepted reports from management without robust challenge.'

I think that was an issue originally as well.

'This means that there are limited actions or decisions arising from reports presented and progress in key areas of work in the region has been slow'.

That's quite strong criticism of an organisation where you've already identified a need for improvement. So, what further steps are going to be taken there?

Well, I've very recently written to the lead chief executive asking for reassurance that steps are being taken, and I've received an answer. As you know, there's a new acting managing director in place, and I've been told that a new agreement between the different authorities and the consortia is due and we'll be kept up to date on that.

So, the role of the consortia is to identify early what school improvements are needed and action them. One of the things that you've said is that the failure, which you've just reflected, is—some of the failures are due to 'disagreements across the local authorities'. Is that suggesting that the consortia as a whole are not as effective as they could be and that part of the reason is that there are so many different bodies they serve that can't agree on practice?

Are you talking specifically about supporting schools causing concern?

Yes—identifying early those schools that need support. So, those schools that may in future identify issues, the consortia are there to be a proactive body that supports and develops improvements. But you've identified that disagreements among local authorities can hamper the success of that.

Yes, it clearly can, but even when there are no disagreements—. And that obviously makes things more difficult. I think that, clearly, the relationship between local authorities and regional consortia is a new one and a developing one. So, where, in the past, you had just the one organisation, which was the local authority, helping a school causing concern, now you've got two, but they're not the only organisations involved in all of this. So, it is quite complex.

It's been a few years now.

Absolutely. For example, we suggested a few years ago that there should just be one plan for helping the school to develop, but regulations still require a school to have their own plan and the local authority to have their own plan. I think it makes much more sense, actually, just to have one plan.

So, reflecting on that, are consortia working? Or do you think they're not? Is the system and the structure effective?

I think the outcome of our revisits was that three of them were working but that one needed what we've been talking about: a fresh look at the relationship.

So, it can be done.

It can be done. And the other thing we did during the last year in respect of ERW was that we did improvement conferences in two of the local authorities that are part of that consortium. So, there are concerns both at a consortium level but also in terms of individual local authorities. We talked about it in relation to schools earlier, but in terms of trying to bring those various agencies round the table, and that was particularly with a focus on secondary schools and the performance within Powys and Pembrokeshire of secondary schools—the number that were in statutory categories. And we still have concerns about those authorities as well.

I think that gives us reassurance.

Yes. Suzy, you had a supplementary on this, briefly.

Yes. It will be brief. In the report, particularly when you're talking about schools that are performing well and schools that are underperforming, you suspect that the gap between those may grow. That's one of the quotes from there. In terms of consortia, are you able to say, where the schools that have been underperforming have improved the most and where those schools that are performing well might have started to slide back—are they related in any way at all, those observations, to the consortia? Are the underperforming ones in a particular consortium area and—

Generally, no. The consortia are so large that there's variation, which you do see at local authority level. There is a variation at local authority level and, as we discussed earlier, three particular local authorities have been subject to particular scrutiny from Estyn.

But the lower performing ones that are starting to improve, are they improving more slowly in one particular consortium region than any others or is that—

I don't think there's any pattern on a regional basis for that.

Lovely. Thank you.

Thank you. Hefin on work-based learning.

Just the one question on work-based learning: in last year's session with the committee on the annual report, you expressed concerns about the provision of in-work-based learning, and we'd just like an updated assessment.

Yes. You're right that last year's annual report, which looked across the seven years, showed that this sector wasn't a particularly strongly performing sector. In particular, I think what worries us is the lack of excellence within the sector. This year's position is only based on three inspections, so it's a small sample. Overall, it was, I guess, relatively positive compared to the last seven years, but, as I say, it was only three inspections. But there was still no excellence. So, that's a continuous worry for us about how we support generally to identify and share excellence in that sector.

We did do a remit report around higher apprenticeships this year as well, and that's fed in to some of what we said in the annual report. And generally we found that the quality of provision—there are some things that need to be strengthened around the consistency of learner outcomes, some of the aspects of the rates at which they complete their qualifications, and in particular some of the skills that they develop. Their digital skills, for example, is a particular area that we've identified. We've also identified that too few trainers and assessors identify the usefulness as well of Welsh as a language skill, and that continues to be a worry. So, people are not promoting (a) Welsh-medium opportunities, but (b) how useful as an employment skill Welsh can be as well.

Generally, the other areas that could be improved are around some of the off-the-job training—the balance between on-the-job and off-the-job training isn't always quite right. And sometimes the care, support and guidance that learners receive to enable them to understand where they're going and make the progress they need to can be improved. So, I think there's some positive progress. There's still a concern about that excellence overall, but it's a slightly better picture than across the last seven years.

Okay. Thank you very much. Our final set of questions is on 'A Learning Inspectorate', from Janet Finch-Saunders.

Yes. Thank you. I'm going to need some fairly—. I'm sorry, I'm just looking at the time, and I want to get these answers from you as quickly as possible. When will there be a response to Professor Donaldson's review of Estyn's role in the context of the considerable education reforms taking place, and do you consider this to be a matter for Estyn or the Welsh Government?

Well, to some extent, I have responded. I've welcomed the report and I've accepted the report. I have spoken in six national conferences, to every headteacher and some council leaders as well, saying as much. I welcomed the report. It was very complimentary to Estyn, and it certainly builds on ideas we've already developed internally in Estyn and on the new inspection arrangements that we have, and it helped crystallise, really, our long-term vision for the inspectorate. Most of those recommendations are in our gift. We can make progress with them. We've made progress with about a third of the recommendations already. I guess the one that's attracted most media attention is the one about a partial suspension of inspection, and that is one that isn't in our gift. That is something for Welsh Government, because there's detailed legislation and regulation surrounding what and how many schools and so forth we inspect. And there would have to be a change in regulation for that.

Okay. On that point, then, have you made your wishes clear to the Welsh Government that you'd like that pause?

Yes, we've had very detailed discussions with officials in Welsh Government about the whole—

So, you agree with it.

I can certainly see the advantages of it. The new curriculum is certainly the biggest change in Welsh education for decades, and one can certainly see how an input from the inspectorate could help that process. We'll be able to help schools, and we would also develop ourselves a better and clearer understanding of where schools are at on that journey so that we can give advice to Welsh Government and understand what the barriers are that are facing schools that are not maybe making a lot of progress with this and what's really working well in other schools. So, we need to develop that understanding.

And we would continue to support and, indeed, inspect those schools causing concern. So, we would have that continuing, and that's why we're calling it a partial suspension.

Okay. Thank you.

So, if you are going to have this pause, the Government is going to need to make a decision on that fairly quickly, aren't they? When would you expect them to do that?

Well, my understanding is that there will be some developments in the new year. There will be a policy on evaluation and improvement arrangements generally, of which accountability and inspection is a part. So, that's my understanding—there will be announcements in the new year.

Okay. Janet, are you—?

No, that's it now.

Okay. Well, we've come to the end of our time, so can I thank you all for attending and for answering such a wide range of questions? It's been a really useful and interesting discussion this morning. As usual, you'll be sent a transcript to check for accuracy following the meeting, but thank you again for your attendance.

Item 4 is papers to note. Papers to note 1 is a letter from me to the Children's Commissioner for Wales on elective home education. That's the only paper to note. Are Members happy to note that? Thank you.

Item 5, then. Can I propose, in accordance with Standing Order 17.42, that the committee resolves to meet in private for the remainder of the meeting? Are Members content? Thank you.