The official version of this document can be found via the PDF button.
The below content has been automatically generated from the original PDF and some formatting may have been lost, therefore it should not be relied upon to extract citations or propose amendments.
Comptroller and Auditor General
Use of Management Information in the Health and Social Services Department – Operating Theatres: Follow-up
27 August 2015
R.97/2015
Use of Management Information in the Health and Social Services Department - Operating Theatres: Follow-up
Introduction
Background
- Access to relevant and high quality management information allows organisations to make strategic and operational decisions, efficiently and effectively.
- The Health and Social Services Department (HSSD) is planning and implementing significant changes in the way it provides services. The White Paper Caring for each other, caring for ourselves' sets out an ambitious programme of change, designed to address current and future challenges. It can be found on the States of Jersey Website[1] .The aim is to ensure that future services are not only safe, sustainable and affordable but that they meet the needs of Islanders, supporting their health and wellbeing'. As part of this, changes to, and significant investment in, acute hospital facilities are planned. HSSD has recognised the need for management information to underpin health reforms.
- In 2014 I reviewed how well HSSD identified, collected and used management information for planning and decision-making. I focused on this in the context of utilisation of operating theatre capacity.
- I issued a report detailing my findings and recommendations in July 2014. The review found that the Theatres and Anaesthesia Division that runs the operating theatres had taken some steps towards identifying the management information it needed and developing data capture systems. However there were weaknesses in:
- arrangements for collecting, testing, analysing and reporting theatre utilisation data and information;
- how the information is used; and
- the coverage and quality of current data and other information on theatre utilisation.
Objectives and scope
- This review evaluates:
- the adequacy of the arrangements HSSD has put in place to manage and monitor the implementation of the recommendations made in my report and to evaluate the impact of implementation;
- the progress HSSD has made in implementing agreed recommendations; and
- how HSSD has used what it learned from the Use of Management Information in the Health and Social Services Department – Operating Theatres review in other areas of its business.
- This follow-up work does not extend to areas excluded from the previous review. It does not:
- consider utilisation information for the theatre dedicated to maternity services;
- assess actual performance of operating theatres and whether that compares well with good practice.
- In addition to recommendations this report contains areas for continuing management action relating to more detailed aspects of implementation.
Arrangements to manage and monitor the implementation of recommendations
- The Theatres and Anaesthesia (T&A) Division which runs HSSD's operating theatres, has established good arrangements to implement the recommendations:
- In September 2014 it developed an action plan in response to my report's recommendations and established a Theatre Information Group (TIG), a task and finish group comprising senior clinicians and managers. Chaired by the General Hospital's Director of Operations, TIG included the Associate Medical Director, who is the clinical lead for HSSD on Informatics. In early 2015, the newly appointed Head of Informatics joined the group, which met monthly until May 2015. Attendance and participation in the group throughout its lifetime, including from senior nurses, surgeons and anesthetists, was good.
- The TIG prioritised the actions needed to establish a strong basis for high quality management information. The initial focus was on improving the quality of the baseline data collected. The TIG looked at existing systems and processes, identifying where changes were required. The completeness of data was an early priority: exception reports highlighting missing data were routinely reviewed at the TIG meetings, to identify and resolve issues iteratively.
- The TIG developed Key Performance Indicators (KPIs) for economy, effectiveness and efficiency and set targets and
tolerances for these.
- In November 2014 the T&A Division underwent a management restructure to better align the roles of Divisional Lead and Clinical Lead. Revised job descriptions set out how these roles should work in partnership, including in taking responsibility for aspects of data quality and management information.
- In May 2015 the TIG was replaced by a standing Theatre Utilisation and Planning Group that is focusing on the meaningful use of data as management information.
- T&A's arrangements for implementing the recommendations have key strengths:
- a phased approach recognising early priorities and next steps;
- good and sustained senior clinical and managerial involvement;
- wide ranging relevant input from across HSSD;
- active involvement from the Head of Informatics, offering knowledgeable, practical advice and a strategic view;
- development from a focus on securing high quality management information to effective use of the information; and
- better alignment of management roles including in supporting data quality.
- HSSD has established arrangements to monitor the implementation of recommendations: the T&A action plan was a standing item at TIG meetings.
However:
- documentation to demonstrate monitoring of the plan, primarily notes of TIG meetings and ad hoc updates of the Action Plan, is not comprehensive, making it somewhat difficult to establish the current position;
- there are no embedded arrangements for evaluating whether action taken has had the desired impact. That is not to say that such evaluation has not taken place. Indeed, in one instance an agreed action was changed as, on reflection, T&A identified that it would not deliver the required improvement (see Exhibit 5 below).
Recommendation
R1 Design and implement robust monitoring and record-keeping arrangements, to enhance monitoring of the implementation of
recommendations, including arrangements to evaluate the changes made to ensure that the actions have achieved the desired outcome.
Progress against recommendations
Identifying relevant management information needs
- Useful information needs to relate to the objectives of the organisation. Failure to collect information that relates to organisational objectives impedes the ability to determine whether those objectives are being achieved and increases the risk of poor value for money.
- The information needed to support delivery of objectives should be aligned with the outcomes those objectives are designed to deliver. Relevant KPIs, with appropriate targets and tolerances, should encompass:
- What the Division is seeking to achieve (its objectives); and
- The things it wants to avoid happening (driven by risk logs).
- In my previous report I identified scope for improvement in these areas and made three recommendations. In Exhibit 1 I evaluate the steps taken and planned in response to these recommendations.
Exhibit 1: Identifying relevant management information needs: action in response to recommendations
Rec | Recommendation - 2014 | Update – 2015 | Status | Evaluation |
1 | Update the Theatres and Anaesthesia Division Business Plan to:
| HSSD decided not to develop its 2015 Business Plan until final decisions on budgets had been made. This delay led T&A to not update its own objectives for 2015 but instead to wait until 2016, in line with the timescales for the next Medium Term Financial Plan. However, T&A has edited its Business Plan to:
| Partially implemented | Non-alignment of T&A's Business Plan with HSSD's increases the risk that the KPIs being developed (see R2 and R3) will not be adequate to support strategic objectives. Development of a core HSSD Business Plan before commencement of 2015 would have reduced this risk, even if subsequent changes had been required. |
2 | Identify appropriate KPIs measuring economy, efficiency and effectiveness. | TIG has focused on developing new KPIs to monitor the 3 Es'. However: | Implementation in progress | The T&A Division has made progress in developing KPIs to |
Rec | Recommendation - 2014 | Update – 2015 | Status | Evaluation |
|
| there is confusion in the definitions of some of the KPIs between the Business Plan; weekly / monthly performance reports; the monthly dashboard; and ad-hoc analyses in other reports. For example, a late start is defined as >10 minutes in the Business Plan; and as >30 minutes in the dashboard;
some KPIs fail to capture relevant dimensions of performance. For example, under-utilised lists' defines an early finish as > 1 hour so that early finishes of up to 59 minutes would not be monitored or managed. T&A recognises that KPIs are not yet entirely fit for purpose'. |
| measure economy, effectiveness and efficiency. In doing so it has achieved better levels of clinical engagement in data and information quality. However, the KPIs are not yet entirely fit for purpose'. |
3 | For each KPI set appropriate targets and tolerances to support delivery of objectives | Targets and tolerances have been set for the 3 Es' KPIs. However, there are inconsistencies in the targets and tolerances set out across various documents. For example, for the vacancies' KPI, the Business Plan states T&A will have a zero tolerance' of the use of Agency and Bank staff; but a monitoring report states there is no target or tolerance level and this KPI is for information only. | Implemented | Further development of consistent, relevant and clearly communicated targets and tolerances is necessary to facilitate effective management. |
Recommendations
R2 Plan to produce the annual HSSD Business Plan before the commencement of the financial year, even if subsequent
changes may be required.
R3 Prioritise updating of the Theatres and Anaesthesia Division Business Plan to:
- Align objectives clearly to HSSD Business Plan objectives; and
- Include appropriate outcome-focused objectives.
Areas for management action
A1 Review definitions of KPIs, targets and tolerances in different documents and update as necessary to ensure consistency. A2 Review KPIs to ensure that they capture all relevant dimensions of performance to effectively manage the service and
update them as necessary.
Ensuring data quality
- Information for decision-making is most useful when derived from high quality data. Where data is of a low quality there is an increased risk that decisions are made which do not promote organisational objectives. There is also a risk that information derived from the data is ignored in decision-making.
- My previous report compared T&A's theatre utilisation information against seven key attributes of data quality:
- Accuracy – data should be a clear representation of an event or activity.
- Validity – data should be recorded and used in accordance with agreed requirements, rules and definitions.
- Reliability - data collection processes should be clearly defined and stable to ensure consistency over time.
- Timeliness - data should be collected and recorded as quickly as possible after the event or activity.
- Relevance – data requirements should be clearly specified and regularly reviewed to reflect any changes in needs.
- Completeness - should be complete and not contain redundant records.
- Compliance - data should comply with statutory requirements on data protection and data security.
- From this comparison I made five recommendations for improvement (see Exhibit 2).
Exhibit 2: Ensuring data quality: action in response to recommendations
Rec | Recommendation - 2014 | Update – 2015 | Status | Evaluation |
4 | Develop and utilise management information reporting capability for the Endobase Olympus system used in the endoscopy suite. | The Endoscopy Suite is not yet reporting using the KPIs for the 3 Es' which have been adopted in Main Theatres and Day Surgery. However, the Endobase system's minimum data set' includes the necessary data requirements to report on the 3Es' in the same way as Main Theatres and Day Surgery. HSSD has identified areas where endoscopy management information could be improved and has a plan to achieve this by September 2015. | Implementation in progress | Exploiting the reporting capability of Endobase should enhance the information available to manage the utilisation of the Endoscopy Suite. |
5 | Establish and reinforce clear accountabilities for the completeness and accuracy of data entry. | The T&A Divisional Lead presents exception reports, highlighting missing - and more recently, inaccurate - data, at Theatre Utilisation and Planning Group meetings and at weekly staff meetings. These reports are used to identify any issues relating to accountability for data quality. Changes to TRAKcare (the data entry system) facilitate completeness and accuracy by:
Roles and responsibilities have been more clearly set out and circulated. However, proper accountability for data completeness and accuracy has yet to be embedded. | Implementation in progress | Steps are being taken to establish and promote clear accountabilities for the completeness and accuracy of data entry. However, reporting on data issues is not yet adequate to provide trends on data quality and therefore to allow evaluation of the effectiveness of the steps taken to improve data quality. |
Rec | Recommendation - 2014 | Update – 2015 | Status | Evaluation |
6 | Further develop and utilise TRAKcare's management information reporting by:
| T&A has:
developed new system guidance using annotated screen shots to help staff understand data requirements; and mandated some key data and information fields. Whilst HSSD has identified future system requirements, these are taking some time and are now planned for 2016 subject to funding. Moreover:
| Implementation in progress | Improvements in TRAKcare's functionality should enhance management information reporting but plans for further improvements are not yet concrete. |
7 | Implement arrangements for promoting and testing the quality of data | T&A runs a series of exception reports that are provided to T&A staff. The Data Quality Team within HSSD's Informatics function routinely reviews T&A data and provides weekly reports. Review is identifying completeness and accuracy issues including:
| Implementation in progress | Arrangements for monitoring and managing the completeness of theatre data are now established. The development of arrangements for testing data accuracy is highlighting other issues. |
Rec | Recommendation - 2014 | Update – 2015 | Status | Evaluation |
8 | Establish and implement arrangements actively to collect softer information from all relevant staff groups about operating theatre utilisation in order to feed management information. | T&A is taking two key approaches to enabling softer' information from a wide range of theatre users to be shared. Firstly:
Notes of these meetings are available to staff not on duty. In addition, a twice-yearly User Group meeting has been established. | Implemented | There are good arrangements in place to enable all theatre users to contribute softer' information to help improve theatre utilisation. The challenge now is demonstrating how softer information is used. |
Areas for management action
A3 Closely monitor the delivery of the plan to develop and utilise the management information reporting capability for the
Endobase Olympus system used in the endoscopy suite.
A4 Establish a way of reporting on issues with data quality that allows those with the accountability for data quality to take
appropriate action.
A5 Set a clear timetable for delivery of improvements to the TRAKcare system and monitor delivery against the timetable.
A6 Establish mechanisms for demonstrating how softer information about theatre utilisation captured from staff is collated and
used.
Using management information about operating theatre utilisation effectively: compiling information
- Good quality data is most valuable when it is compiled, reported and used appropriately to provide management information to support evidence-based decision-making.
- In the report issued in July 2014 I set out an evaluation of the way T&A calculated the theatre utilisation rate. This identified weaknesses in key elements of the data items involved. I made three recommendations (see Exhibit 3).
Exhibit 3: Compiling information: action in response to recommendations
Rec | Recommendation - 2014 | Update – 2015 | Status | Evaluation |
9 | Collect information on gaps between patients during lists and amend the calculation of the utilisation rate performance indicator to account for this lost' time. | The Head of Informatics, who joined HSSD in February 2015, has been working with the TIG to develop utilisation reports. The data to enable gaps' between patients to be calculated is now routinely collected. One analysis of utilisation now undertaken attributes negative utilisation' to any over-runs, so that the use of planned time (rather than actual time) is clear. Another utilisation calculation compares individual patient level' timings with planned list' timings. This takes into account gaps' – and in some cases overlaps' - between patients (although this nets off gaps and overlaps within a single session masking potential issues). T&A has prioritised monitoring later starts and early finishes – gaps' at the start and end of theatre lists – as these have been identified as areas in need of improvement. | Implementation in progress | Although HSSD is now using a sophisticated approach to analyse theatre utilisation and calculate utilisation rates, it still does not have a report to monitor and assess the impact of gaps between patients. |
Rec | Recommendation - 2014 | Update – 2015 | Status | Evaluation |
10 | Amend the calculation of the utilisation rate performance indicator to reflect actual available time in each theatre for each calendar month | By March 2015 the calculation of theatre utilisation was changed. The revised approach addresses some of the concerns I raised. However, as the new approach focuses on utilisation against planned lists rather than against potential available time, a key dimension of theatre utilisation is not captured. In addition, the indicator used does not reflect last minute list cancellations even though these lead to potential capacity not being used | Implemented | T&A understands the need to make improvements in its calculation of theatre utilisation and took steps to deliver the recommendation. However, the new indicator does not capture all dimensions of theatre utilisation rates. |
11 | Agree performance information required for Theatre 4 and other emergency activity, implement systems to collect it and routinely prepare an appropriate performance indicator | T&A has agreed KPIs for emergency activity which focus on patient safety, based on the CEPOD (Confidential Enquiry into Peri-Operative Deaths) standards. The use of the KPIs has already reduced the activity T&A is undertaking after 8pm, reducing patient risk and enabling better staff planning. | Implemented | Appropriate KPIs are being collated and there is evidence of a demonstrable impact on performance. |
Areas for management action
A7 Actively consider the options for calculating theatre utilisation rates and set out the key reasons for choosing a particular
approach, noting how any risks in the chosen approach will be mitigated.
Using management information about operating theatre utilisation effectively: Reporting management information
- The way in which management information is reported and presented, including how, when and to whom, has a direct impact on its value in driving changes in practice.
- At the time of my original review, T&A was in the early stages of considering options for reporting theatre utilisation information. I included two recommendations in my report (see Exhibit 4).
Exhibit 4: Reporting management information: action in response to recommendations
Rec | Recommendation - 2014 | Update – 2015 | Status | Evaluation |
12 | Disseminate relevant management information more widely and provide training on its use | TIG helped clinicians and managers understand and engage with theatre management information. The regular staff meetings within Main Theatres, the Day Surgery Unit and the Endoscopy Suite increasingly include consideration of performance reports. | Implemented | Although the T&A Division has improved the availability of management information to different groups of staff, there is scope for structured training on understanding and using management information in future. |
13 | Review KPIs to align with business needs, including considering:
disaggregating some KPIs to specialty level. | KPIs need to flow from business objectives: this recommendation cannot be fully implemented until HSSD and T&A business plans are updated to align to HSSD objectives (see Exhibit 1, R1 above) T&A has however:
The T&A Business Plan includes an intention to report performance by specialty but the approach has not yet been agreed. The Head of Informatics is considering opportunities to report KPIs at specialty and clinician level. The Theatre Utilisation and Planning Group aims to increase the focus on ensuring the right KPIs are in place to provide management information for short, medium and longer term business planning and delivery. | Implementation in progress | Finalisation of KPIs aligned to business needs is dependent on agreement of business objectives. |
Recommendation
R4 As soon as business objectives are agreed:
- review and as necessary update KPIs; and
- take steps to agree and implement KPIs at specialty and clinician level.
Areas for management action
A8 Establish a programme for rolling out training to clinicians and other relevant staff on understanding and using management
information on operating theatres.
A9 Monitor the effectiveness of the Theatre Utilisation and Planning Group and take action as necessary to support it in aligning
KPIs with business needs.
Using management information about operating theatre utilisation effectively: Using management information on a day- to-day basis
- In my previous report I noted examples of where management information from theatres is already being used effectively. I made one recommendation for improvement. T&A identified difficulties in implementation but has identified appropriate alternative steps that it is implementing (see Exhibit 5).
Exhibit 5: Using management information on a day to day basis: action in response to recommendations
Rec | Recommendation - 2014 | Update – 2015 | Status | Evaluation |
14 | Use the full potential of data from TRAKcare, including to calculate consultants' typical procedure times, to inform scheduling of patients for theatre | T&A has identified that due to relatively high levels of pooled' lists (where patient procedures might be undertaken by any of two or more surgeons), it is not possible to pinpoint a surgeon's average time for a procedure and allocate that within a scheduled list. Instead, it has initiated a Lean review that, among other things, seeks to make patient scheduling more efficient by improving information flows. The primary focus is trauma and orthopaedic surgery but it intends to roll out lessons learned across all theatre work. T&A plans to consider surgeon operating times by procedure, to identify outliers and to understand variance whilst at the same time ensuring safe practice. | Appropriate alternative action in progress | T&A has set a way forward to achieve the intended outcome of this recommendation. |
Using management information to inform longer-term planning
- Ensuring robust management information underpins longer-term planning decisions is very pertinent to HSSD, given the planned changes I noted above. In 2014 I made three recommendations (see Exhibit 6).
Exhibit 6: Using management information to inform longer-term planning: action in response to recommendations
Rec | Recommendation - 2014 | Update – 2015 | Status | Evaluation |
15 | As data quality improves, expand the use of data and management information to inform longer-term decision-making | There is as yet no evidence that HSSD has used theatre performance information to exploit new opportunities or manage identified risks. | Not yet implemented | The improved clinical interest in theatre management information that T&A has achieved is a crucially important step, but processes to meaningfully use the information in decision making have yet to be established. HSSD's use of data as management information and business intelligence is not yet developed. |
16 | Prioritise development of the capacity to use cost information alongside activity information to inform decision-making. | HSSD's submission as part of the process to develop the next Medium Term Financial Plan includes provision for the development of costing across its services. HSSD has not considered an alternative approach if funding is not available. | Not yet implemented | While HSSD hopes to be able to develop patient level costing, funding has not yet been identified. Alternative approaches to integrating cost and activity information may be available. |
17 | Ensure decisions on the future requirements for, and use of, operating theatre capacity are made - and where necessary revisited - using high quality theatre utilisation information. | The Future Hospital Board has commissioned an external company to assess previous assumptions made regarding theatre requirements. I have been advised that the detailed modelling exercise undertaken has relied not only on current and projected HSSD data but also on information from professional advisors for the Future Hospital Project. | Not yet implemented | It is premature to evaluate the extent to which theatre utilisation information has been used to inform decisions on future requirements for and use of operating theatre capacity. Indeed, until HSSD management is satisfied as to the quality of the data, it would not be appropriate to rely solely on information generated from HSSD's systems. |
R5 Set out in one place the expectations of how and when operating theatre information will be used to inform longer-term
decision-making.
R6 Evaluate alternatives to a full patient level costing system to enable cost information to be used alongside activity information
to inform decision-making.
R7 Develop and communicate a confidence' rating for the use of operating theatre information to inform longer-term decision-
making.
Using the learning
- Learning organisations translate individual experiences and findings and apply them more broadly to drive improvement. Whilst my review related to one activity in HSSD, the lessons about use of management information are of wider relevance across the whole department.
- No specific evidence has been provided to show how HSSD has taken the learning from my previous report into other areas of its business.
- The Informatics Strategy, that HSSD developed before my previous report was published, takes forward many of the recommendations into different parts of HSSD. The Informatics Lead took up post in February 2015 and has a key role in driving forward this strategy. She is working to:
- support Divisions in developing relevant and meaningful KPIs, linked to strategic objectives and business intelligence needs;
- assess the integrity of systems and processes for producing management information;
- identify sources of data and information;
- improve the quality of underpinning data; and
- develop ways of analysing and reporting data, including KPIs, to support performance and delivery of services.
- The Informatics Lead is now responsible for the production of HSSD's Integrated Report. This has the potential to be a very important focus of management information to help HSSD to achieve its business objectives.
R8 Review the Informatics Strategy for continued relevance of both objectives and proposed means of delivery.
R9 Develop and roll out a programme for sharing the valuable learning about management information within operating theatres
to clinicians and other staff within HSSD.
Conclusion
- T&A has actively embraced the recommendations contained in my previous report, where appropriate adapting them to achieve their underlying objective. It has made good progress. Implementation is underway for the majority of the recommendations but as expected at this stage, the full benefits are yet to be seen.
- However, my recommendations were directed to HSSD more widely and there are key areas where HSSD needs to take action so that the full benefits of my recommendations can be secured.
- Firstly, HSSD deferred updating its own Business Plan. The absence of a Business Plan, even if subject to potential future change, means that T&A is working in a vacuum: without clear business objectives it cannot align its own objectives with those of HSSD or be confident that the KPIs it develops are relevant and comprehensive.
- Secondly, progress on using management information on operating theatres to inform longer-term strategic planning has been slower. Effective use of management information is vital for both operational and strategic decision-making.
- Thirdly, there has been valuable learning and demonstrable progress on the use of management information within T&A. Whilst some of the issues are specific to T&A, many maybe relevant more widely to HSSD. HSSD has yet to adopt a structured approach to ensure that learning has been shared and embraced more widely.
Appendix 1: Summary of Recommendations
R1 Design and implement robust monitoring and record-keeping arrangements, to enhance monitoring of the implementation of
recommendations, including arrangements to evaluate the changes made to ensure that the actions have achieved the desired outcome.
R2 Plan to produce the annual HSSD Business Plan before the commencement of the financial year, even if subsequent
changes may be required.
R3 Prioritise updating of the Theatres and Anaesthesia Division Business Plan to:
- Align objectives clearly to HSSD Business Plan objectives; and
- Include appropriate outcome-focused objectives.
R4 As soon as business objectives are agreed:
- review and as necessary update KPIs; and
- take steps to agree and implement KPIs at specialty and clinician level
R5 Set out in one place the expectations of how and when operating theatre information will be used to inform longer-term
decision-making.
R6 Evaluate alternatives to a full patient level costing system to enable cost information to be used alongside activity information
to inform decision-making.
R7 Develop and communicate a confidence' rating for the use of operating theatre information to inform longer-term decision-
making.
R8 Review the Informatics Strategy for continued relevance of both objectives and proposed means of delivery.
R9 Develop and roll out a programme for sharing the valuable learning about management information within operating theatres
to clinicians and other staff within HSSD.
Appendix 2: Summary: Areas for management action
A1 Review definitions of KPIs, targets and tolerances in different documents and update as necessary to ensure consistency. A2 Review KPIs to ensure that they capture all relevant dimensions of performance to effectively manage the service and
update them as necessary.
A3 Closely monitor the delivery of the plan to develop and utilise the management information reporting capability for the
Endobase Olympus system used in the endoscopy suite.
A4 Establish a way of reporting on issues with data quality that allows those with the accountability for data quality to take
appropriate action.
A5 Set a clear timetable for delivery of improvements to the TRAKcare system and monitor delivery against the timetable.
A6 Establish mechanisms for demonstrating how softer information about theatre utilisation captured from staff is collated and
used.
A7 Actively consider the options for calculating theatre utilisation rates and set out the key reasons for choosing a particular
approach, noting how any risks in the chosen approach will be mitigated.
A8 Establish a programme for rolling out training to clinicians and other relevant staff on understanding and using management
information on operating theatres.
A9 Monitor the effectiveness of the Theatre Utilisation and Planning Group and take action as necessary to support it in aligning
KPIs with business needs.
KAREN McCONNELL COMPTROLLER and AUDITOR GENERAL
JERSEY AUDIT OFFICE, LINCOLN CHAMBERS (1ST FLOOR), 31 BROAD STREET, ST HELIER, JE2 3RR
T: 00 44 1534 716800 E: enquiries@jerseyauditoffice.je W: www.jerseyauditoffice.je