NetworkZ Programme of Evaluation
What do we know about team-based simulation from existing research?
Delivery of safe and effective care to surgical patients depends on effective teamwork and communication between the whole operating room team. Unintended patient harm is a substantial burden on global health. On many occasions failures in teamwork and communication are an important contributing factor.
Team training interventions can improve teamwork and communication, and potentially clinical processes and patient outcomes. Limitations in the quality of existing research, and the differences between NetworkZ and team training programmes tested in previous research means that an evaluation of this programme is warranted.
Why is the evaluation important?
NetworkZ represents a major quality initiative to improve outcomes for our patients. The stepped roll-out of the initiative across all 20 DHBs in NZ affords a limited time opportunity to produce evidence of the effectiveness of this innovative team training initiative, and develop knowledge that may inform the implementation of future safety interventions. This nationwide programme is an international first that could lead the world in changing culture and practices in operating rooms.
NetworkZ is comprised of realistic simulated surgical cases presenting challenges or uncertainty to surgical teams. Each session is followed by a debrief to enable participants to reflect on the events, expose assumptions and communication issues, and identify good practice and opportunities to improve processes and systems.
What are we hoping to find out from the evaluation?
1) Does NetworkZ lead to improved patient outcomes?
2) Does NetworkZ lead to improved teamwork and communication in the operating room?
3) Does NetworkZ lead to improvements in the safety culture of the operating room?
4) What organisational and cultural factors determine a successful (or unsuccessful) implementation?
What is the timeline for the evaluation?
NetworkZ will be rolled out consecutively over 5 years across the 20 DHBs in four temporal cohorts with 12-month intervals between cohorts. The progressive roll out of NetworkZ supports a stepped wedge cluster research design, which balances the practicality of the rollout, with a more robust methodology than a typical pre-post study.
What data will be collected?
Our research approach is mixed methods. It includes a suite of outcomes measures, with hard outcomes from existing databases, and softer outcomes on processes relevant to the intervention. Baseline data and ongoing data will be collected at regular intervals over the course of the project for all DHBs in NZ.
Summary of NetworkZ measures
Patient Outcome measures
Days Alive and Out of Hospital at 90 Days (primary outcome measure)
ACC treatment injury claims
Staff surveys on teamwork and safety culture
In-theatre observation using WHOBARS
Staff turnover and sick leave records
Participant end of course evaluations
Uptake of NetworkZ within DHBs
Interviews with DHB managers and OR staff on implementation process
Quantitative measures will be compared between pre- and post-rollout phases for this stepped wedge design. Qualitative data from interviews will undergo generic thematic analysis.
We are also undertaking a review of the post-course reports to assess whether there are patterns in the types of staff training and systems issues that have been identified, and in turn implications for national training and operating theatre systems.
The data will be analysed at a national and cohort level – no results will be presented at the level of the DHB.
What data collection has been completed so far?
Pre-data collection observations and surveys have been completed for Cohort 1-3 and is underway in Cohort 4. We have completed ‘post’ data collection for Cohort 1-2. Interviews about the implementation of NetworkZ have been conducted in Cohort 1 and 2, and used to update the process for implementation in subsequent cohorts.
The final analysis of the programmes impact on patient safety will happen in May 2022, one year after the final cohort completes their 15-month transition or embedding period.
Figure 1: The stepped-wedge cluster cohort design
What have we learnt about implementation?
We have analysed two sets of qualitative information about the process of implementation, including factors that impede or support this type of training.
Findings from these interviews provide an in-depth understanding of unique organisational change processes. We use this learning use it to collaboratively inform implementation in new DHBs joining the programme. For Study 1, interviews were conducted with 31 instructors or managers in Cohort 1. Local structural support, including support from senior and middle management, was described as key to overcoming implementation challenges, such as recruitment. Observed changes, such as improvements in communication and speaking up, and staff a desire to improve both patient safety and teamwork further motivated implementation.
For Study 2, interviews were conducted with 31 instructors or managers in Cohort 2. The realistic, in-situ, multidisciplinary nature of the training was valued, seen as unique, relevant and generalisable other areas. However NetworkZ was described as complex, due to multidisciplinary participation and the multiple roles and skillsets of instructors needed to run simulations smoothly, making the programme resource intensive to deliver. Interviewees also identified improvements in teamwork and crisis management.
Read the full paper here https://bmjopen.bmj.com/content/9/10/e027122
You can view the published research protocol at the Australian and New Zealand Clinical Trials Network website.
Your participation in NetworkZ research
Key research activities
In order to investigate the effect that NetworkZ has on processes within your DHB, we will:
- Administer staff surveys prior to the start of NetworkZ training, and 15-18 months later when training has been embedded
- Conduct in-theatre observations prior to the start of NetworkZ training, and 15-18 months later when training has been embedded
- Conduct interviews during NetworkZ roll-out
- Pre-NetworkZ data collected within a three month period before training starts
- Post-NetworkZ data collected approximately fifteen months after the start of training
- Interviews conducted after the start of NetworkZ training
We anticipate administering two surveys to all operating theatre staff in each DHB:
1. The Teamwork Perceptions Survey has been designed by the NetworkZ team in an attempt to investigate the aspects of teamwork and communication that are taught on the NetworkZ course.
2. The Surgical Culture Safety Survey was developed by the Harvard University School of Public Health, and adapted by the Health Quality and Safety Commission to evaluate the recent Safe Surgery NZ programme. This is administered by HQSC every two years and we intend to include this in our evaluation work.
By observing actual behaviour in the operating theatre we can gain a first-hand measure of teamwork. To achieve this we will utilise a tool that our group have previously developed. The World Health Organisation (WHO) Behaviourally Anchored Rating Scale (BARS) as an observational tool for measuring the quality of administration of the WHO Surgical Safety Checklist.
A trained WHOBARS rater will observe 5 -10 surgical cases with a variety of teams in your DHB.
We conducted two sets of semi-structured interviews with managers, who direct and oversee the implementation of NetworkZ, and project teams and instructors, who are responsible for implementation of processes and training OR staff.
The findings from the Cohort 1 interviews are published in BMJ open https://bmjopen.bmj.com/content/9/10/e027122
Lingard L, Espin S, Whyte S, Regehr G, Baker GR, Reznick R, et al. Communication failures in the operating room: an observational classification of recurrent types and effects. Qual Saf Health Care. 2004;13(5):330-4.
Gawande AA, Zinner MJ, Studdert DM, Brennan TA. Analysis of errors reported by surgeons at three teaching hospitals. Surgery. 2003;133(6):614-21.
Buljac-Samardzic M, Dekker-van Doorn CM, van Wijngaarden JDH, van Wijk KP. Interventions to improve team effectiveness: A systematic review. Health Policy. 2010;94(3):183-95.
Hughes AM, Gregory ME, Joseph DL, Sonesh SC, Marlow SL, Lacerenza CN, et al. Saving lives: a meta-analysis of team training in healthcare. J Appl Psychol. 2016; 101(9):1266-1304