Article Text
PDF
Abstract
Background/objective Cincinnati Children's Hospital Medical Center created the Intermediate Improvement Science Series (I2S2) training course to develop organisational leaders to do improvement, lead improvement and get results on specific projects.
Design methods Each multidisciplinary class consists of 25–30 participants and 12 in-class training days over 6 months. Instructional methods include lectures, case studies, interactive application exercises and dialogue, participant reports and assigned readings. Participants demonstrate competence in improvement science by completing a project with improvement in outcome and/or process measures. They present on their projects and receive feedback during each session and one-on-one coaching between sessions.
Results Since 2006, 279 participants in 11 classes have completed the I2S2 course. Participant evaluations have consistently rated satisfaction, learning, application, impact and value very high. Large and statistically significant changes were observed in pre-course to post-course self-assessment of knowledge of five quality improvement topics. Approximately 85% of the projects demonstrated measurable improvement. At follow-up, 72% of improvement projects were completed and made a part of everyday operations in the participant's unit or were the focus of continuing improvement work. Many changes were spread to other units or programmes. Most (88%) responding graduates continued to participate in formal quality improvement efforts and many led other improvement projects. Nearly half of the respondents presented their results at one or more professional conference.
Conclusions Through the I2S2 course, the authors are developing improvement leaders, accelerating the shift in the culture from a traditional academic medical centre to an improvement-focused culture, and building cross-silo relationships by developing leaders who understand the organisation as a large system of interdependent subsystems focused on improving health.
- Clinical leader development
- quality improvement
- implementation science
- physician education
- nursing education
- health professions education
- healthcare quality improvement
- implementation science
- leadership
- collaborative
- breakthrough groups
- comparative effectiveness research
- continuous quality improvement
- control charts
- run charts
- quality improvement methodologies
- patient safety
- paediatrics
Statistics from Altmetric.com
- Clinical leader development
- quality improvement
- implementation science
- physician education
- nursing education
- health professions education
- healthcare quality improvement
- implementation science
- leadership
- collaborative
- breakthrough groups
- comparative effectiveness research
- continuous quality improvement
- control charts
- run charts
- quality improvement methodologies
- patient safety
- paediatrics
Introduction
A fundamental and transformational change in the healthcare delivery system and how providers perceive, think, and behave is required to close the gap between care as usual and best practice. Such transformation also requires leaders who are trained to direct the effort. Yet, a review of the available literature yields few examples of programmes to develop quality improvement (QI) leaders.
A systematic review of the effectiveness of teaching QI to clinicians showed that providing learners with a package of QI tools and individualised coaching in QI methods can improve clinical outcomes compared with audit and feedback and limited QI training.1 In addition, encouraging learners to implement and test interventions through several small tests of change appeared to be more successful than undertaking a single comprehensive intervention. While most studies showed improvement in clinicians' self-reported knowledge or confidence to perform QI, the effects on patient outcomes were less clear.
Some organisations have implemented different QI training programmes for leaders and frontline staff. For example, Emory Healthcare has a 2-day leadership programme that provides a basic understanding of QI concepts and vocabulary and a 12-day practical methods programme to develop frontline staff and middle managers capable of designing and implementing QI initiatives.2
The Medical Center of Central Massachusetts conducts a 20 h QI educational course for managers and a similar, but abbreviated, 12 h course for all other employees.3 Staff physicians are invited to attend three 90 min QI awareness overviews.
The Baylor Health Care System's Accelerating Best Care at Baylor (ABC Baylor) QI education programme includes a 2-day strategic overview of QI for senior management, a 6-day core course aimed at physician and nursing/administrative change leaders, with didactic training and experiential learning through the design, implementation and evaluation of a QI project, and a 1-day fast track course for frontline caregivers participating in quality initiatives.4 ,5
The Institute for Health Care Delivery Research at Intermountain Healthcare in Utah conducts a 20-day Advanced Training Program in Health Care Delivery Improvement (ATP). Participants receive training in the theory and application of cost and quality control and the health services academic infrastructure, and they select, complete and report on an improvement project.6 The Institute also conducts an abbreviated 9-day mini-ATP course in which participants also complete an improvement project. Intermountain Healthcare has not described how they evaluate their participants and courses or provided data on the success of participants' improvement projects in any journal article.
We report here on an intermediate-level training course created at Cincinnati Children's Hospital Medical Center (CCHMC) to develop organisational leaders to do improvement, lead improvement and get results on specific projects. The Intermediate Improvement Science Series (I2S2) combines experiential learning7 ,8 and leadership development experiences based on Deming's System of Profound Knowledge9 to develop improvement leaders, shift the culture from a traditional academic medical centre to an improvement focus, build cross-silo relationships to create a web of leaders who understand the organisation as a large system consisting of interdependent subsystems focused on improving child health, and get improvement in clinical and non-clinical measures.
The I2S2 course instructional design is based on David Kolb's theory of experiential learning,7 ,8 which has formed the basis for many adult education efforts. It defines learning as the process through which knowledge is created, where knowledge is the combination of grasping and transforming experience. Kolb describes two approaches to grasping experience (perception): concrete experience (eg, involvement in an incomplete and unsafe handoff) and abstract conceptualisation (eg, thinking about the concepts involved in a high-reliability process), and two approaches to transforming experience (processing): reflective observation (eg, asking why an event may have occurred) and active experimentation (eg, completing a small test of change). He describes a four-stage learning cycle including concrete experience, reflective observation, abstract conceptualisation and active experimentation. Based on these constructs, Kolb identified four basic learning styles, each of which combines one approach to grasping experience with one approach to transforming experience: diverging—concrete experience and reflective observation, with the learner preferring brainstorming, working in groups and listening with an open mind; assimilating—conceptualisation and reflective observation, with the learner preferring readings, lectures, exploring analytical models and having time to think things through; converging—abstract conceptualisation and active experimentation, with the learner preferring to experiment with new ideas, simulations, practice assignments and practical applications; and accommodating—concrete experience and active experimentation, with the learner preferring to work with others to get work done, set goals, do field work and test different approaches. Our training programme was designed to accommodate all four learning styles through the inclusion of a variety of learning experiences, from readings and lectures to concrete project work, with feedback to interactive application exercises to small group discussions.
Development, implementation and evaluation of the programme
History and context
CCHMC is a large, urban paediatric academic medical centre that has maintained a significant focus on improving evidence-based and family-centred care since the late 1990s.10 After receiving a grant from the Robert Wood Johnson Foundation as part of The Pursuing Perfection: Raising the Bar for Health Care Performance initiative in April 2002,11 we needed to build improvement capability in our leaders and staff. We chose the Model for Improvement as our core improvement methodology.12 From 2002 to 2005, 18 key leaders and early adopters (physician faculty and patient services leaders) attended the Intermountain Healthcare ATP course.13 It provided a good foundation for our quality and transformation efforts. However, as improvement efforts increased and spread throughout the hospital, we needed to expand beyond this small critical mass of trained leaders. In addition, new strategic goals at CCHMC called for learning and transformation of care delivery and continuously improving clinical and functional outcomes. I2S2 was created to address these goals by preparing QI leaders to practise and teach improvement science and to lead improvement.
Course aims and strategy
I2S2 was designed to fill gaps in leadership skills and improvement science knowledge and application in physicians, nurses, allied health professionals, clinical and non-clinical support services staff, and formal and informal executive leaders. This diverse cohort required an instructional design that appealed to all learning styles and a variety of needs for improvement. Because we are an academic medical centre, the course had to be academically rigorous, but also practically oriented, with support for all learners. We assumed that participants would already have a basic understanding of improvement science and the Model for Improvement12 from working on QI projects. The course was first designed in consultation with Brent James, MD, MStat, and with feedback from our staff who completed the ATP training course.13
The specific aims of I2S2 are to develop an intermediate level of knowledge and skill to conduct and lead improvement; get results on a specific project; and develop a common language and culture/behaviours. The course is focused on developing improvement leaders, not just teaching technical QI skills. There is an expectation that graduates will go on to lead additional QI projects and develop QI skills in their staff.
Course structure and design
Training sessions occur over 2 full days a month, for 6 months. All sessions are conducted off site to minimise the distraction caused by daily work; 100% attendance is required. Each class consists of a multidisciplinary cohort of 25–30 participants. The first class targeted early adopters. Classes 2 and 3 included key hospital leaders who were expected to lead improvement. Key safety and patient flow leaders were given priority for classes 4–6 to accelerate progress on these strategic initiatives. More recently, participants have been chosen based on nominations from senior leaders, who are expected to develop a multi-year plan to build improvement leaders in their areas. Priority is given to leaders actively involved in improvement efforts.
Instructional methods include lectures, case studies, interactive application exercises and dialogue, participant book reports, and assigned readings of textbooks and journal articles. Participants demonstrate competence in improvement science by completing a project with improvement in outcome and/or process measures.14 They give project presentations and receive feedback during each session and one-on-one coaching between sessions. To emphasise systems thinking and interdependencies, multidisciplinary participant groups are rotated at tables throughout the course so that the participants have a continuing opportunity to form a network of relationships and understanding outside of their own specific professional discipline. The design includes careful attention to the flow of the content (eg, we introduce principles of reliability15 very early to emphasise the importance of testing higher reliability interventions), significant emphasis on statistics in three sessions to allow time for clarification of this content, and repetitive emphasis of core concepts through project feedback. Course faculty include physicians, nursing staff, administrative leaders and QI experts at CCHMC who have been actively involved in successful and unsuccessful projects.
Course content
Selected course content is summarised in the online supplement (table A). Deming's system of profound knowledge serves as the conceptual framework for the course.9 It includes four key areas: understanding variation, theory of knowledge or action learning, appreciation of the system, and psychology or change management. The critical leadership emphasis is not on general leadership skills, but rather on leadership of large systems to achieve breakthrough changes in clinical outcomes, patient experience and value through designing and leading a portfolio of work. Hence, our topics are chosen to focus the participants on viewing their work differently, seeing the complexity of large system change in a healthcare organisation.
During the first 2 h of every I2S2 session, five to seven participants present updates on their projects to the whole class using a structured template. The template includes the project title and team members, a key driver diagram (which depicts the underlying hypothesis to achieve improvement), a SMART aim (specific, measurable, actionable, relevant and time bound),16 outcomes, key drivers and interventions, an annotated run chart to show change over time, plan–do–study–act (PDSA) cycles and ramps,12 learnings and challenges, and next steps. Participants receive feedback from the class and faculty.
At course completion, each project is evaluated by five criteria: rating on an assessment scale developed by the course director based on the Assessment Scale for Projects in Collaboratives created by the Institute for Healthcare Improvement17 (online supplement table B); whether the aim is SMART; whether the key drivers are clear, stated in a positive manner, indicate what must be in place to drive the aim, and are at the correct complexity level (not so detailed that they are actually interventions, but not so global that they provide no direction); whether statistical process run and control charts18 ,19 are properly labelled and annotated; and whether small tests of change were used effectively: linked to a key driver, documented with a prediction and thoughtful study phase, and executed with PDSA ramps. Participants typically run at least 12–15 PDSA cycles during the course.
Programme evaluation and analysis
The course is assessed using Kirkpatrick's four-level model for evaluating training programmes: reactions of the participants—what they thought and felt about the training; learning—the resulting increase in knowledge or capacity; behaviour—the extent of behaviour and capability improvement and implementation/application; and results—the effects of a trainee's performance on the business or environment.20 ,21 Participant reactions are obtained through an online evaluation survey and verbal debriefings conducted after each monthly 2-day session. Participant learning is measured via QI knowledge self-assessments administered prior to the first day and on the last day of the programme, peer feedback from project reviews, and the course director's rating of the application of QI principles and tools in each participant's final project report. Behaviour change and results are measured through a follow-up survey sent to graduates at least 6 months after course completion that asks about sustained results for the original project, leadership on subsequent projects and use of QI techniques. Specific project results are analysed at the end of the class to determine if they meet statistical process control special cause rules indicating a statistically significant improvement.22
Descriptive statistics were calculated using data from the first 11 classes to assess results of post-event surveys, follow-up surveys, and pre-programme and post-programme self-assessments. Representative quotes from verbal debriefings were used to further elucidate participants' experiences. Univariate analyses and a repeated measures multivariate analysis of variance (MANOVA)23 were performed to assess the statistical significance and degree of change in self-assessments from pre to post programme.
The CCHMC institutional review board reviewed the plans for the proposed programme and declared it exempt.
Results
Course participants
Since I2S2 was initiated in 2006, 279 participants in 11 classes have completed the course; 106 faculty members, 94 management and staff members from the Division of Patient Services Administration (eg, nursing, respiratory care, social services, speech pathology and nutrition therapy), and 79 administrative and support management and staff. Five other participants left the organisation prior to completing the course.
Nursing leaders who completed the course include the senior vice president of patient services and all of her direct reports. Non-clinical administrative leaders who have completed the course include the chief executive officer, senior vice president for planning and business development, the vice president for access services, and the senior vice president for care delivery, infrastructure and operations. Faculty leaders who have completed the course include the director of the paediatric residency programme and 13 other clinical division directors. Through class 11, physicians and staff from 19 of the 25 paediatric clinical divisions and 9 of the 11 surgical divisions have attended.
Reactions of the participants (Kirkpatrick level 1)
On a scale from one (strongly disagree) to seven (strongly agree), participant evaluations averaged 6.5 on overall satisfaction, 6.6 on applying knowledge and skills, and 6.4 on impact on quality of life, patient and provider experiences and value (table 1). Responses were very similar for each class.
Participant reactions
Representative participant comments from the surveys and debriefings include ‘This education opportunity is the best and most practical that I've encountered in 18 years in healthcare,’ ‘I really feel as though I have taken the first step on a path that will really redefine me professionally,’ and ‘This will become a way of life rather than just an isolated way to manage a project or two.’
Participant learning (Kirkpatrick level 2)
On a scale from 1 (no knowledge) to 6 (expert), the participants assessed their awareness of and expertise with five QI topics before and after completing the programme (table 2). Prior to beginning the programme, participants, on average, indicated through self-assessment that they had basic knowledge of various QI skills, tools and methods (level 2), with their knowledge of the theory of leadership slightly higher than other topics (table 2). After completing the course, participant's self-assessed knowledge improved to level 4. On average, participants reported the ability to independently and accurately apply QI skills, methods and tools.
Results of participant self-assessments
Because the multivariate effect of time was statistically significant in the MANOVA analysis (Wilks' λ-associated F (5, 232)=289.76, p<0.0001, η2=0.86), demonstrating improvement, univariate analyses of the individual components of QI were also analysed. Large and statistically significant changes were observed in the pre-course to post-course self-assessments for each topic, suggesting greater knowledge of, ability to independently and accurately apply, and experience with QI skills, tools and methods.
Change in behaviour (Kirkpatrick level 3)
Eighty-two per cent of graduates from classes 1–10 completed the follow-up survey about the current status and impact of their I2S2 course project (table 3). At follow-up, most respondents had made additional updates to their run charts since completing the programme, suggesting that participants continued to collect and display data relative to their original course project to monitor progress and/or study the effects of new tests of change. Half of the I2S2 projects were completed and made a part of everyday operations in the participant's unit and 22% were the focus of continuing improvement work. In addition, the changes from many projects were spread to other units or programmes within the organisation or to other organisations.
Change in behaviour
Most of the responding graduates continued to participate in formal QI efforts and many actually led other improvement projects. Nearly half of the respondents had presented their results at one or more professional conferences, but only 13% had yet published an article based on their QI work.
Project results (Kirkpatrick level 4)
Table 4 provides a sample of I2S2 projects. Of the 274 completed projects, 42% addressed a clinical issue (eg, timely administration of prophylactic antibiotics), 37% addressed an indirect clinical issue (eg, timely patient discharge) and 21% addressed a non-clinical issue (eg, tracking inventory). Five pairs of participants completed shared, rather than individual, projects.
Sample of Intermediate Improvement Science Series (I2S2) 6-month projects
Approximately 85% of all projects demonstrated measurable improvement. Figure 1 shows the percentage of participants with projects rated 5 (modest improvement) to 8 (sustainable improvement) by the course director. Eighty-seven per cent of the projects addressing clinical issues, 82% that addressed indirect clinical issues, and 78% that addressed non-clinical issues were rated five or above.
Participants with a rating of 5–8 on course director's assessment scale.
Discussion
Our I2S2 internal training course was designed using Kolb's theory of experiential learning2 ,7 ,8 and Deming's System of Profound Knowledge9 to develop improvement leaders and accelerate CCHMC's transformation to an improvement-focused culture. Using evaluation methods based on Kirkpatrick's four-level model,20 ,21 we demonstrated improvements in self-assessed QI knowledge and behaviours through 11 I2S2 classes and 279 participants. Most graduates continue to participate in formal QI efforts and many have gone on to lead new improvement projects, confirming CCHMC's commitment to QI as an institutional priority. Approximately 85% of participant projects demonstrated measurable improvement, 72% were completed and made a part of everyday operations in the participant's unit or were the focus of continuing improvement work, and many have been spread to other CCHMC units or to other organisations.
In describing the two-pronged QI training programme at Emory Healthcare, Rask et al reported that QI knowledge increased from 47% to 82% for the 545 leaders and supervisors who participated in the Leadership for Healthcare Improvement course.2 The 44 improvement projects implemented by the 85 frontline employees and department managers who participated in the Practical Methods for Healthcare Improvement course were rated an average of 10.8 on a scale of 0 to 14 (median 11.0, range 7.5–14.0). Sixty-six per cent of projects demonstrated significant improvements and 23% achieved their aim statement target. At course completion, 41% of projects were judged to be mature enough to be sustainable. At follow-up, 6–12 months after graduation, 86% of the 22 participants interviewed reported that their project has been fully implemented as originally planned and 68% of projects remained active.
Preliminary results from demonstration projects in Pennsylvania based on the ABC Baylor model suggested improvements in clinical and financial measures.24 ,25 Filardo et al examined the effectiveness of a QI training session programme based on the Baylor programme in rural hospitals in Texas.26 At 12 months, hospitals randomised to receive the education and coaching support to complete an improvement project performed no better on core measures for community-acquired pneumonia and congestive heart failure than hospitals that did not receive the formal education programme. The authors noted that continuous QI training in the absence of organisational support, such as involvement of leadership in QI, physician champions to lead and promote improvement efforts and financial incentives linked to performance on quality indicators, may be insufficient to effect meaningful improvements in quality of care.
CCHMC considered the more traditional approach of having separate and more limited QI training programmes for leaders versus longer and more detailed training for frontline staff,2–5 as described in the above examples, but rejected it. We believed that, in an academic medical centre, it was critical for leaders to have in-depth knowledge of improvement and that we could not accomplish the type of dramatic shift in thinking and acting required for transformation with just a traditional short-term programme for leaders. We observed that participants who entered with scepticism or no real QI knowledge often did not really understand the shift in mental models from an intense research approach, with pre and post data analysis, to a small test of change, active intervention in a system that drives learning and is rooted in understanding the system and variation in the system approach until the third session. We believe our success in shifting mental models is because we allow time for reflection and internalisation, with repeated emphasis of Deming's core concepts and specific feedback on their projects.
Participants have told us that the format of six monthly meetings, conducted at an off-site location, gives them the time and distance required for internalisation and reflection. We believe it is an appropriate vehicle to help leaders shift their mental model of leadership to an improvement-based model. Conducting the course with an intact, heterogeneous cohort stimulates interaction and learning, reinforces cultural change, and allows participants to see CCHMC as an interdependent system. Achieving the right balance between learning experiences that are intellectually challenging and others that are more basic is critical.
We believe that participant QI projects accelerate learning and build improvement capability through concrete experience. They provide a teaching vehicle to reinforce key concepts, drive home the ‘all teach, all learn’ non-hierarchical philosophy (everyone, without regard to title, has something to contribute), provide practice in critiquing and coaching others on their project work, and accelerate improvement with fewer resources.
Project selection is critical. Generally, projects that were considered not successful at the end of the course have fallen into two categories. First, the participant simply ran out of time because he or she had difficulty at the start getting baseline data; many are able to successfully complete their project several months after the course is completed. Second, the participant, who in spite of coaching, chose and stuck with an inappropriate topic or one for which there was no real leadership support or alignment with important priorities. We believe that the best projects have a significant but narrow scope, with a line of sight to a meaningful outcome, available baseline data and frequent testing cycles. It is important that the project has the support of the department or division head and that the participant understands the resources available to complete the project on schedule. Getting meaningful results in 6 months can sometimes be a challenge. Initially, participants were asked to come to the first session with ideas for their project. We now provide specific project selection criteria and assign a coach 6–8 weeks before the first session to insure participants start the course, ideally, with a draft aim and baseline data. Regular project review is a key learning opportunity. It reinforces important content, allows participants to practise making logical and succinct presentations, and helps them develop the ability to critique and coach. Project feedback has evolved as a major instructional method to reinforce learning in the context of real-time projects. Private coaching between sessions has been shown to be very helpful. We now have a standardised structure for coaching and project feedback.
Other changes have been made to the I2S2 course over time in response to comments and suggestions from participants and lecturers. QI tools are now introduced in the first session and we have added a simulation to teach the application of the tools. We have developed easy-to-use templates for key driver diagrams, pareto charts, run charts and commonly used control charts. We have also developed and implemented two online modules that participants can complete prior to starting the course (Introduction to quality) and early in the course (Introduction to measurement). A continuing challenge has been our inability to meet the demand for the I2S2 course. We normally select each class of 25–30 from a list of 60–70 interested faculty and non-faculty staff members.
In conclusion, the I2S2 course has helped us to shift our culture from a traditional academic medical centre to a measurement and results focus by developing the capability of participants to lead improvement. Since the implementation of I2S2, we have observed an increasing number of ongoing improvement projects at CCHMC and the adoption of a common improvement language in presentations to employees and the Patient Care Committee of the Board of Trustees. Through the I2S2 course, we are developing improvement leaders, accelerating the shift in the culture from a traditional academic medical centre to an improvement-focused culture, and building cross-silo relationships by developing leaders who understand the organisation as a large system of interdependent subsystems focused on improving health.
Acknowledgments
The authors would like to thank Brent James, MD, MStat (Institute for Health Care Delivery Research at Intermountain Healthcare in Utah) for his helpful consultation during the design of the Intermediate Improvement Science Series (I2S2) training course and John Murphy, MS (Education Research and Measurement, Cincinnati Children's Hospital Medical Center) for providing data management and analysis assistance that aided the writing of this report.
References
Supplementary materials
Supplementary Data
This web only file has been produced by the BMJ Publishing Group from an electronic file supplied by the author(s) and has not been edited for content.
Files in this Data Supplement:
- Data supplement 1 - Online Table A
- Data supplement 2 - Online Table B
Footnotes
-
Funding Data used for this report were collected in part with the support of a grant from the Robert Wood Johnson Foundation, Evaluating Quality Improvement Training Programs 65 499. SF had financial support from the Robert Wood Johnson Foundation for analysis of the submitted work.
-
Competing interests There have been no financial relationships with any organisations that might have an interest in the submitted work in the previous 3 years, and no other relationships or activities that could appear to have influenced the submitted work.
-
Ethics approval Cincinnati Children's Hospital Medical Center Institutional Review Board.
-
Provenance and peer review Not commissioned; externally peer reviewed.
-
Data sharing statement Data are available on request from Uma R Kotagal at uma.kotagal{at}cchmc.org.