Management Information in Education.

Comptroller and Auditor General Management Information in Education 22 September 2016

R.95/2016

Management Information in Education

Introduction

1. Access to high quality and relevant management information is essential to enable organisations to make effective and efficient strategic and operational decisions at corporate, departmental and team level for both business as usual' and change initiatives. A commitment at all levels to effective specification, preparation and use of management information is a key component of a culture that drives improvement.
2. In 2014 I reported my findings on the availability and use of management information in the Health and Social Services Department, focussing on operating theatre utilisation.
3. This second review considers education services provided by the Education Department. High quality, secure information accessible within the Education Department and beyond is needed, not only to drive educational attainment but also to facilitate the wider provision of services to vulnerable children and families.

Background

4. The Education Department currently provides services to or supports more than 16,600 pupils, from nursery to university. The Department is organised into six teams to support service delivery which includes 24 primary schools, seven secondary schools, two special schools and alternative curriculum provision where appropriate.
5. The current organisation of the Education Department, which is evolving in response to changing need, is illustrated in Exhibit 1.

Exhibit 1: Organisation of the Education Department

Six teams:

Standards and Achievement

Inclusion and Family Support

Policy and Planning

Resources and School Support

Youth Support Human Resources

10 front line services: Nurseries

 Early years

 Primary schools

 Secondary schools

 Higher education

 Adult education

 Youth service

 Post 16

 Careers

 Libraries

6. In 2016 the total budget for the Department is £105 million, £61.3 million of which funds nursery, primary, secondary and special education provision.
7. Historically, attainment of GCSE grades A* to C by Jersey's 16 year-old pupils had been better than for pupils in England. But since 2009/10 England as a whole and most English local authority areas have outperformed Jersey. This only became apparent when management information was used and reported effectively.

8. The Education Department recognised the need for better management information to allow it to make evidence based decisions to evaluate the impact of initiatives to tackle the deterioration in relative performance. In early 2012 a report commissioned by the Department examined how data was used within the Department and by schools (see Exhibit 2).

Exhibit 2: Findings from 2012 review of use of data by the Education Department and schools

Area  Findings

Data availability  no single source of data

• often manual processes

Systems and data  schools had a Central Management Information System

(CMIS) but difficulties in its use meant alternative local systems were established

• data entered numerous times into different systems
• accuracy and completeness of data often poor

Consistency,  schools not always clear on how data will be used transparency and

guidance

Evaluation  only high level indicators reported

• need for consistent measure of pupil progress across primary and secondary education

9. Following this initial review, three further workstreams looked at:

• best practice in analysis, reporting and use of information in education services;
• developing a strategic approach; and
• the experience and skill sets of those people involved in the provision of data analysis and reporting within the Education Department.

10. In response, in 2013 the Department established its Insight' team to drive better data and statistical analysis for education services. In 2015 the team grew to three people – a head of statistics (with a Masters in Official Statistics), one data manager and one data analyst. The Department recognises the need for strong leadership within the Insight team so that it remains independent and able to provide robust management information.

Objectives and scope of the review

11. This review focuses on the extent to which the Education Department:

• has access to, and actively uses, high quality and relevant information for day to day performance management; and
• has a robust information base from which to make decisions for the longer-term.

12. Collecting and using good management information is important for education services in all its areas of business.
13. The review seeks to answer four inter-related questions (see Exhibit 3). Exhibit 3: Questions asked

Question 1: How

management  arrQuesangemgood tion entsare 2: Howfor

well have

information  securing data requirements  quality?

been specified?

Question 4: How effectively is

Queseffection tivel3: Howy is  management management  working to

secure

information being  improvements in used?  management

information?

14. To answer each of the four questions, I have concentrated my work on three specific workstreams that I have used as examples throughout this report (see Exhibit 4).

Exhibit 4: Workstreams reviewed Workstream  Scope

Pupil and School  Monitoring of performance against expected performance at attainment  Key Stages of the National Curriculum set by the UK

Department for Education (DfE) e.g. proportion of pupils achieving A* to C at GCSE.

Jersey Premium  Targeted funding to support evidence based interventions for

pupils from more disadvantaged backgrounds, similar to the UK Pupil Premium and based on the clear link between social deprivation and literacy levels. £0.8m is available for pilot schemes in 2016 with £1.9m for full implementation in 2017.

Education Business  Drawing together of four existing initiatives and replacing the Partnerships  Skills Board, an employer body, to support the education,

training and development of young people by linking businesses and their employees with schools and colleges. The target date for commencement has slipped from April to October 2016.

How well have management information requirements been specified?

1. Useful information needs to relate to the objectives of the organisation. Failure to collect and report information that relates to organisational objectives impedes the ability to determine whether those objectives are being achieved and increases the risk of poor value for money.
2. High performing organisations:

• link management information requirements to strategic objectives;
• link information about activity and outcomes to that about resources used; and
• plan to prepare and present management information in a way that facilitates decision making.

3. I consider in turn the Department's identification of relevant management information requirements:

• generally, in its business planning process; and
• specifically, in the context of the individual workstreams selected as tracers.

Specification of management information requirements across the Department

4. The 2015 Education Department's Business Plan sets out high level departmental objectives aligned to the States' strategic goals, and the accompanying signs of success' indicating how progress will be gauged. An example of this approach is given in Exhibit 5.

Exhibit 5: Education Department Business Plan – Extract

5. As I reported in my 2016 Review of Financial Management – Part 2, some of the Business Plan's objectives and aligned signs of success' are not sufficiently SMART (Specific, Measurable, Achievable, Relevant and Time-bound). Taking the example in Exhibit 5 above:

• many of the signs of success reflect a direction of travel, such as reduce' or increase' without a quantified outcome;
• some measure inputs (such as hours of observation) rather than outputs or outcomes;
• some (such as an increase in nursery places) do not directly link to the objective (of improving standards and outcomes);
• in some cases (such as saving time and resources), the basis of measurement is not specified; and
• some relate to future development of measures rather than agreed measures that will be used for internal monitoring and external accountability.

6. The Department's Business Plan does not include financial information – that is, how the Department's budget will be used in achieving objectives. Similarly, most of the signs of success' do not relate activities or outcomes to the financial resources used. This hinders the ability of the Department to assess the value for money secured from agreed actions or target improvement initiatives.
7. The Department intends to refresh and re-publish its Business Plan in the Autumn of 2016 and as part of this is working to improve the specification of Key Performance Indicators (KPIs) including:

• the methodology used to calculate outcomes;
• how often performance against the KPIs will be evaluated;
• targets and tolerances against each defined outcome measure; and
• how and where each KPI will be reported.

8. Historically, arrangements for collation and communication of management information have been underdeveloped. The Department is currently piloting a template to show strategic progress against Business Plan objectives, bringing together an evaluation of progress against objectives and an assessment of risk to delivery. Using the templates for the first time, in June 2016 the Departmental Senior Management Team considered information about projects, including Pupil and School Attainment and the Jersey Premium but not the Education Business Partnership initiative.
9. The template is evolving but not does not yet include:

• hard data against KPIs;
• financial information, for example budget position; or
• critical achievements or activities for the next time period, which would help in monitoring direction of travel.

10. The Department has yet to:

• establish criteria for routine and by exception' reporting;
• identify and formalise the mechanisms for reporting more widely, including to the Corporate Management Board; or
• demonstrate that information on current performance is the basis not only for remedial action but also for future planning and prioritising.

Specification of management information requirements for the three tracer workstreams

11. The extent to which the Department has specified relevant management information requirements varies across the three workstreams examined (see Exhibit 6).

Exhibit 6: Specification of management information requirements for workstreams Workstream  Management information  Areas not yet fully developed

requirements identified

Pupil and  The Department has adopted KPIs  Developing indicators to link pupil school  developed by the UK DfE and set  and school attainment to attainment  specific targets for levels of  investment has been more

attainment by specified dates e.g.  challenging.

All key performance indicators will  The island's four non-funded

be in line with the average  independent schools are not performance by the island's  required routinely to provide the statistical neighbours by 2020'.  same level of information on

attainment as States and part- funded non-States schools. The four schools provide varying levels of information. The Department recognises that work is required to ensure it identifies and collects sufficient, high quality performance information from these schools.

Since 2015 the Special Educational Needs (SEN) team has been working with schools to record SEN interventions and their cost in a Provision Map'. The team is now developing a way of evaluating the success of SEN interventions against expenditure.

Success criteria have not been consistently established to enable the Department to determine whether changes to policy or practice have achieved the desired outcome. For example, on introduction of the new School Attendance and Absence policy.

Workstream  Management information  Areas not yet fully developed

requirements identified

Jersey  Amongst the criteria for  Indicators are being developed to Premium  acceptance of pilot scheme  measure the impact of

proposals was that the impact of  interventions e.g. through the interventions could be measured in  impact on DfE KPIs.

a timely manner.

Education  KPIs have recently been adopted  KPIs have not yet been identified Business  for the Trackers (apprenticeship)  for the new Partnership, covering Partnership  scheme but are not yet sufficiently  areas such as:

comprehensive:  current and anticipated skills

• some relate to achievements  shortages;

and qualifications gained but  employer investment in others are predominantly about  capacity; and

process; for example, for  the effectiveness of work- Retention rate achievement' the  related interventions such as measure is: A six monthly  Trident (work experience), review by management' and not  Work Related Learning

a target performance rating;   (learning in a work-based

• KPIs do not clearly align to  setting) and careers advice. departmental objectives; for  The Department is anticipating example, there is no measure to

that management information meet the Business Plan

objective: New areas, such as  needs will be established by financial services and the digital  October 2016 when the

sector, will be included in the  Partnership is due to be Trackers apprenticeship  launched.

programme'; and  

• there is no measurement of the

relationship between resources

expended and outcomes.

Recommendations

R1  Include within the Education Department's Business Plan, for each Departmental

objective:

• KPIs linked to the objective (including strategic objectives set by the Council of Ministers);
• KPIs linking outputs and, where feasible, outcomes to resources used; and
• quantified targets/tolerances.

R2 Develop reporting arrangements for management information to include:

• hard data against KPIs;
• financial information, for example budget position; and
• critical achievement of activities for the next time period.

R3 Establish criteria for routine and exception reporting, including the mechanism for

reporting to the Corporate Management Board as appropriate.

R4 Take steps to demonstrate that information on current performance is the basis not

only for remedial action but also for future planning and prioritising.

R5 For changes in policy and practice together with individual initiatives or workstreams,

ensure that prior to roll out:

• KPIs are developed, linked to objectives;
• KPIs are developed, linking outputs or outcomes to resources used; and
• quantified targets/tolerances for KPIs are set.

How good are arrangements for securing data quality?

1. Information for decision making is most useful when derived from high quality data. Where data is of a low quality there is:

• an increased risk that decisions are made which do not promote organisational objectives;
• a risk that information derived from the data is ignored in decision making.

2. Data quality has a number of attributes (see Exhibit 7). Exhibit 7: Attributes of data quality

Source: Developed from Improving information to support decision making: standards for better data quality Audit Commission (2007)

3. I have evaluated arrangements for securing data quality by reviewing:

• arrangements in place across the Department; and
• arrangements for the three tracer workstreams.

Department-wide arrangements

4. Data quality definition and control is a responsibility of individual States Departments. However, the Education Department's Insight team has not issued formal guidance to support data quality for business as usual' or project-level management information.
5. A States-wide approach is being developed as part of the eGovernment initiative: the Data Governance Council (DGC) is focused on the design and use of data sets and data elements. The Insight team plans to use the formal template and defined process for data collection. However as the DGC's focus is on new data sets within eGovernment projects, this might not meet all of the Department's needs

Arrangements in the three tracer workstreams

6. Arrangements for securing the quality of data supporting management information vary between the three workstreams (see Exhibit 8).

Exhibit 8: Data quality in the three tracer workstreams

Workstream  Strengths  Areas for development

- Identifying  Substantial work undertaken  Reliance on School identification eligible  including:  and self-certification by parents to pupils  identify pupils from families who

• mSoacial Stchinge ocuf ridtyat recoa to rds;  would be eligible but for the five
• cleansing of data;  year residence requirement.
• checking with schools; and
• consultation with parents.

- Measuring  Heavy reliance on existing,  the impact of  standardised data already  interventions  routinely collected and where  

data quality has improved.  

Guidance from the  Department to schools  developing pilots.  

Strong emphasis on  establishing data sets at pilot  approval stage.  

Use of control groups so that  management information  relates to comparative  performance.  

Recognition of need for both  hard data and softer  intelligence.  

Education Business  Process for checking and  Use of three separate databases Partnership  cleansing data for Trackers  to store employer and student

(apprenticeship) scheme.  information, affecting accuracy and completeness.

Heavy reliance on manual input of data at several stages, affecting accuracy and completeness.

Multiple uncoordinated contacts with employers and schools with different levels of information available from each.

Recommendations

R6 Consider extending corporate standards on data quality to all data rather than just

that covered by eGovernment projects.

R7 In the absence of corporate standards for data quality, provide guidance and monitor

its implementation within the Department.

R8 Undertake an assessment of data quality for individual workstreams as a benchmark

to drive improvement.

How effectively is management information being used?

1. Good quality data is most valuable when it is compiled, reported and used appropriately to provide management information to support evidence based decision making.
2. Department-wide review of management information is only being piloted. I have therefore focussed on the use of management information within the three tracer workstreams.
3. The maturity of approach to reporting and using management information in each of the three areas reviewed varies (see Exhibit 9).

Exhibit 9: Use of management information for the tracer workstreams

Strengths  Areas for development

Pupil and  Management information is  The inclusion of performance school  used to report to the public on:  characteristics of pupils and attainment  performance against Key  schools:

Stages 2 to 4 and at A'  receiving the Jersey Premium levels Island-wide and by  funding; and

school; and  benefitting from services within

• progress between age 11  the EBP initiative.

and 16, including by gender,

those with English as an  Howto un tdheis drstaantad w thilel b impae rouct tinofe wlyourk sed additional language and

those with Special  aanpdprotapkeria cote rris ebctiveinge aactciontive alys

Educational Needs.  considered as these workstreams Intelligence used within the  progress.

Department includes:

• this same information but at a named pupil level;  
• comparative information

which enables the value

add' to be assessed; and

• contextual information such

as attendance rates.

Jersey  Management information on  The number of eligible pupils is Premium  eligible pupils has been used to  higher than anticipated but no

direct funding.  decision on how best to use the funding – target fewer children or

Self-evaluation templates for the  spend less per child – has been outcome of interventions are

aPnremalyseiumd bpyrojec the Jet terseyam and  taActiokenn. if pilot funding has not been

applied in line with plans has yet to Professional Partners, with  be determined.

advice from UK Pupil Premium

experts, with the aim of  Evaluation mechanisms following

the inclusion of the Jersey Premium

Strengths  Areas for development

Education  Important research has been  As the objectives, management Business  undertaken to provide  information requirements and data Partnership  management information to  quality arrangements for the

inform the Business Case for  Partnership have yet to be fully the creation of the EBP.  developed, the Partnership is not yet at the stage where it is

identifying and using management information routinely.

In particular, there is as yet insufficient information as a basis for development of a skills agenda for Jersey.

Recommendations

R9  Promote targeted improvement in the routine use of management information to

inform decision making by relevant staff, including through the use of appropriate objectives in individual performance appraisals.

R10  Develop action plans to address the weaknesses in use of management information

identified for the tracer workstreams.

How effectively is management working to secure improvements in management information?

1. High-performing organisations recognise that information needs, information technology and the capacity to use information change – and that they must respond to make the best use of information. Such organisations have a culture which not only values information but also values learning and uses it to drive improvement.
2. Many ofmy reports have focussed on corporate challenges in using information effectively (see Exhibit 10).

Exhibit 10: Comptroller and Auditor General reports focussing on information

Financial management (April 2014)

Information security (June 2014)

Financial management - Part 2 (February 2016)

Freedom of Information (March 2016)

eGovernment (May 2016)

3. Although many of the messages in these reports are corporate, many of the findings and recommendations are applicable in whole or part to individual departments.
4. The Department's awareness of data and information issues has improved and it has recognised the need to develop the capacity and capability of its existing information systems, to improve:

• flexibility and usability;
• data integration and sharing;
• data quality; and
• information presentation to support decision making.

5. Key initiatives taken by the Department are set out in Exhibit 11.

Exhibit 11: Initiatives to improve use of information

Issue  Steps taken  Challenges

Replacement  CMIS does not  Jointly-managed  Original timetable has

of the Central  integrate with the  Education/HSSD  slipped by one year to Management  case management  procurement of  September 2017. Information  system used by  replacement systems  Difficulties in finding a System  Children's  which are intended to  system that meets (CMIS)  Services within the  meet departmental

Health and Social  and cross- fufonr ctiobothn athl spe Deecipficatartmioenn ts Services  departmental

Department  requirements.  aaffndordfoar bschle. oUndols,ear thnde is (HSSD). As a  Close involvement of  most recent plans two

result key  eGovernment project  systems would be contextual  manager to future- procured but Education information about

a child or young  proof the systems  astandff wChoiludld renbe's Sabele rvticeo s person may not be  chosen.  access relevant shared

shared.  information. However, Information on  whereas Education staff

Special  would be able to see this Educational Needs  information in one place, is held on a  system limitations mean separate system.  Children's Services staff

would need to sign into the Education reporting system to do so.

Concluding how best to share information between the Department and HSSD.

Risks associated with data migration.

Issue  Steps taken  Challenges

Operational  Routine reporting  Tableau business  Value depends on the data  of operational data  intelligence software  relevance and quality of reporting  was limited by the  now in use enabling  underlying data and

functionality of  creation of live and  information.

CMIS. There was  bespoke dashboards  Linking effectively with a reliance on  and storyboards,  corporate approach and associated  making it easier, for  the eGovernment spreadsheets.  example, to look at  agenda.

KPIs against

resources.

Data and  Recognised need  The Insight team is  Cleansing data.

informasharing tion  information with  approach to collecting  Idwithen tiwfyhingich tdhaetafreq in uency

for sharing  developing an

other States  information on  different systems is departments and  destinations of school  updated.

beyond. For  leavers.

numbers of school  Datandao soutsidurces e thwe ithin  Eanndsu srihnagre tdh aat t indadtaiv is iduhael ld example, reducing

Not in  States have been  pEustapilb lelishveingl. follow-up leavers that are

Employment,  identified and  mechanisms where Education or  engaged.  destinations of school

Training' (NEET) is  Work is now  leavers cannot be

a States objective.   underway to access  established from data But getting robust  datasets.  sources.

information on

destinations of

school leavers is

challenging.

Lean  A States-wide  Development of the  Using these principles in

initiative to drive  EBP is following Lean  business as usual' as efficiency using  principles in bringing  part of continuous

Lean principles.  together and  improvement rather than

streamlining  as a project' when allocation and use of  structural change resources.  provides an opportunity.

Developing use of the principles in schools and colleges, including in rationalising course delivery.

Issue  Steps taken  Challenges

eGovernment  A States-wide  The Insight team  Establishing expectations

initiative to  leader is closely  and mechanisms to transform the  involved with the  routinely work with the relationship  eGovernment  eGovernment initiative to: between the  workstreams Tell Us

States and  Once' so changes in  share learning - for citizens, facilitated  family and pupil  example from the

by the use of  information within the  issues associated with technology.  education system is  data matching for the

updated across other  Jersey Premium; States' systems.  secure the benefits of thinking differently'

The procurement  about business,

panel for the new  management as Management  eGovernment is Information System  mainstreamed; and (MIS) includes the  engage schools and eGovernment

colleges.

Programme Director.

Recommendations

R11  Consider the relevance of findings and recommendations of Comptroller and Auditor

General reports relating to information to the Education Department and identify appropriate action.

R12  Foster a culture of continuous improvement in management information:

• driven throughout the Department, schools and colleges;
• working with other States departments, to secure benefits across the States;
• by promoting adoption of good practice through the mechanism of the Corporate Management Board; and
• by reporting back to teachers the impact that the data they have input has had on decision making.

Conclusion and way forward

1. The Department has taken important and positive steps in developing and using management information.
2. Expertise has been secured and capacity increased through the establishment of the Insight team. The need for integrated systems has been recognised and procurement of a key system is in hand. Steps have been taken to make it easier to access information. There is experience of working with others to share data for mutual benefit and further plans are in hand. Arrangements for Department-wide use of management information are improving. A culture where data and information are valued is being fostered.
3. But there is more to do. The Department's business plan does not yet drive its management information requirements. Information does not routinely link outputs and outcomes to the resources used to secure them. Key Performance Indicators are not embedded as a way of managing the Department. Targets for improvement are often not specified or quantified. Arrangements for securing data quality are inconsistent. There is more to do to specify and use information effectively, across the Department, within schools and in individual workstreams.
4. Improved information is needed to:

• support the ambitious plans the Department has for improvement, including initiatives such as the development of a programme of whole school reviews; and
• to facilitate working with other departments (including to facilitate the wider provision of services to vulnerable children and families).

5. The Department needs to maintain the momentum to specify the information it needs, secure high quality data to provide the information and then routinely use the information to drive decision making. This will help secure a culture of learning and improvement across the whole Department and in schools which in turn drives improvement in public services.
6. In addition embracing information-led management within the Department will contribute to the cultural change that is needed to improve service delivery across the States.

Appendix 1: Summary of Recommendations

How well have management information requirements been specified?

R1 Include within the Education Department's Business Plan, for each Departmental

objective:

• KPIs linked to the objective (including strategic objectives set by the Council of Ministers);
• KPIs linking outputs and, where feasible, outcomes to resources used; and
• quantified targets/tolerances.

R2 Develop reporting arrangements for management information to include:

• hard data against KPIs;
• financial information, for example budget position; and
• critical achievement of activities for the next time period.

R3 Establish criteria for routine and exception reporting, including the mechanism for

reporting to the Corporate Management Board as appropriate.

R4 Take steps to demonstrate that information on current performance is the basis not

only for remedial action but also for future planning and prioritising.

R5 For changes in policy and practice together with individual initiatives or workstreams,

ensure that prior to roll out:

• KPIs are developed, linked to objectives;
• KPIs are developed, linking outputs or outcomes to resources used; and
• quantified targets/tolerances for KPIs are set.

How good are arrangements for securing data quality?

R6 Consider extending corporate standards on data quality to all data rather than just

that covered by eGovernment projects.

R7 In the absence of corporate standards for data quality, provide guidance and monitor

its implementation within the Department.

R8 Undertake an assessment of data quality for individual workstreams as a benchmark

to drive improvement.

How effectively is management information being used?

R9 Promote targeted improvement in the routine use of management information to

inform decision making by relevant staff, including through the use of appropriate objectives in individual performance appraisals.

R10 Develop action plans to address the weaknesses in use of management information

identified for the tracer workstreams.

How effectively is management working to secure improvements in management information?

R11 Consider the relevance of findings and recommendations of Comptroller and Auditor

General reports relating to information to the Education Department and identify appropriate action.

R12 Foster a culture of continuous improvement in management information:

• driven throughout the Department, schools and colleges;
• working with other States departments, to secure benefits across the States;
• by promoting adoption of good practice through the mechanism of the Corporate Management Board; and
• by reporting back to teachers the impact that the data they have input has had on decision making.

KAREN McCONNELL COMPTROLLER and AUDITOR GENERAL

JERSEY AUDIT OFFICE, LINCOLN CHAMBERS (1ST FLOOR), 31 BROAD STREET, ST HELIER, JE2 3RR T: 00 44 1534 716800   E: enquiries@jerseyauditoffice.je  W: www.jerseyauditoffice.je