Evaluating Nursing and Midwifery Metrics: Providing evidence to support change and renewal in healthcare services in Ireland.

: This study was commissioned by senior nursing and midwifery management to evaluate the impact of ‘medication management metrics’ (Medication Storage and Custody and Medication Administration) upon the delivery of nursing and midwifery care in the Health Service Executive (HSE) North West Area. The study employed a mixed-methodology using both quantitative and qualitative approaches to data collection and analysis. Methodological and source triangulation were incorporated to crosscheck and affirm the reliability and validity of the findings. Quantitative and qualitative data were gathered across two time points from 3 locations (38 settings) across the HSE North West Area. The evaluation design incorporated three research strands. The first strand evaluated the performance of each ward/unit using standardized ‘metrics’ criteria’. The second strand evaluated the experiences of both patients and staff within the context of local implementation. Strand three examined the impact of the initiative upon the context within which care is delivered-the workplace setting. The study findings have important policy implications that should inform any future corporate approach towards system wide implementation of nursing and midwifery metrics. Outcomes from this should promote improvements in care delivery and enhanced corporate understanding of the central role of nurses and midwives in the delivery of safe, effective and person-centred care.


Introduction
Hospitals and healthcare settings in Ireland have commenced re-evaluation of policies and practices in order to maintain and improve the quality of care delivered to patients.Central priorities in national health care policy have identified the requirement for safe, effective and person-centred care services (Ireland Department of Health, 2012).This has been accompanied by a focus on the importance of data and information in order to monitor and strive for high quality, safe and effective care.An emphasis upon data and information is vital, especially when making decisions and planning aligned to health and social care settings.
There exists an emerging amount of research that examines the merits of measuring the quality of nursing and midwifery care, at both strategic and practice levels (Griffiths et al., 2008).Measuring quality within health and social care settings provides evidence in order to assess performance, improve the quality of care delivery and support process change.Additionally, the measurement of quality allows the generation of evidence so that standards can be assessed and for accountability (Mooney, 2009).This, accordingly, provides a more detailed account of the quality of care that is being measured (HIQA, 2010).

Context
This paper examines the findings and recommendations from an evaluation research study that was conducted during 2012/2013 in the North West of Ireland.The study was commissioned by senior nursing and midwifery management to specifically evaluate the impact of medication management metrics (Medication Storage and Custody and Medication Administration) upon the delivery of nursing and midwifery care in the Health Service Executive (HSE) North West Area.These metrics were introduced in 2012 using the 'Test Your Care' (TYC) system (Heart of England NHS Foundation Trust).
Nurses and midwives are the single largest provider of care at home, in the community or in a hospital setting within Ireland and comprise almost 40% of the health care workforce (NCNM, 2009).They are pivotal to influencing patient outcomes, meeting organizational needs and are seen as paramount (Riehle et al., 2007) in the provision of safe and efficacious patient care in all sectors of the health care system.By virtue of the potential that nurses and midwives have to influence the care outcomes for individuals and their families, it is appropriate that this care is qualified as effective.
To achieve this, from the outset there is a requirement to provide generic evidence around nursing and midwifery quality.One method for attaining this is through the use of nursing and midwifery metrics to improve outcomes and experiences for patients.Aligned to this, the conclusive test is located in whether the wellbeing of patients is maintained or rather improved and whether this well-being is adversely affected by the presence or absence of nursing and midwifery interventions (NHS Scotland, 2005).
Despite this acknowledgement, there has been limited evidence to date, within both the Health Service Executive and National Health Service (NHS), of the benefits to patient care from system wide measurement of nursing and midwifery metrics.Indeed a number of limitations have been recognized that merit consideration; firstly it has been identified that nurses and midwives can spend more time inputting data rather than spending time on care (Stevens, 2010); secondly some processes have focused solely on the quality and safety around nursing and midwifery outcomes and not on the quality of care (Foulkes, 2011).Lastly, it should also be mentioned that a metrics programme should not replace appropriately structured primary nursing and midwifery research and should link with the patient experience (Negus and Howat, 2010).Further limitations focus upon a lack of clarity regarding the indicators used in the process and also difficulty arising from measuring factors such as dignity, respect, communication and privacy, which are difficult constructs to measure or connect with (McCance et al., 2012).
Nonetheless, Griffiths et al. (2008) have previously argued strongly for the development of nursing and midwifery metrics.It is contended that evidence gained from the process allows the general public, service users, managers and government to be informed of quality and safety standards within hospital units and wards.Meaningful benchmarking, between matched comparators or specific specialties (Maben et al., 2012), can also be applied through the use of metrics between local, regional and national health care settings.The utilization of metrics can provide the evidence to support change and renewal within healthcare, particularly supporting nurses and midwives in the delivery of evidence based and compassionate care towards service users.
Griffiths and Colleagues have formerly identified three key interrelated elements of the impact of nursing and midwifery, namely safety, effectiveness and compassion.The RCN (2009), in explicating the role of nursing and midwifery in improving care outcomes for patients and clients, have further expounded these elements (Fig. 1).
Although data solely associated with metrics criteria will not portray the entire picture regarding the contribution of nursing and midwifery to the delivery of high quality care, the use of appropriate and relevant nursing and midwifery metrics may provide an opportunity to demonstrate the unique contribution of both nurses and midwives in delivering safe, effective and person-centred outcomes for patients and clients (Maben et al., 2012).Furthermore, this may allow decision-makers at practice level to concentrate work on areas that are seen as priorities, from a patient care, policy and organizational perspective.As a consequence this type of approach 'should' compel improvements in the quality of nursing and midwifery care experienced by patients and the development of a positive work place culture.

Aim
The purpose of this study is to evaluate the impact of medication management metrics upon the delivery of nursing and midwifery care.

Objectives:
• To investigate the impact of nursing and midwifery medication management metrics as regards safe and effective are • To examine the experiences of patients with a specific focus on how their nursing and midwifery care needs are met • To examine the experiences of staff with regard to the implementation of this programme within the workplace • To create a wider understanding of how the nursing and midwifery metrics programme impacts upon the contexts within which nursing and midwifery care is delivered • To clarify the links between this programme and other strategic and organizational priorities

Methods
The evaluation employed a mixed-methodology using both quantitative and qualitative approaches to data collection and analysis.Methodological and source triangulation were incorporated to cross check and affirm the reliability and validity of the findings.This also provided for a richer understanding of the data.Methodological triangulation involved utilizing a range of approaches to examine the study aim.Survey instruments, focus groups and observations of practice/audits (Table 1) were employed for this purpose.The evaluation incorporated the opinions of patients, who are the recipients of care within the study sites and registered nurses and midwives who deliver nursing and midwifery care within these settings.
Data were collected from across three locations (38 settings) throughout the HSE North West.At the time of the study, these settings were actively engaged with the implementation of medication management metrics.Quantitative and qualitative data were gathered over two time points, providing a repeated application of standardized measures to examine the overall impact of change within the settings.
The evaluation design incorporated three strands.The first research strand evaluated the performance of each ward/unit using standardized metrics criteria.Each of the criteria reflects national and/or regional policy/guidance.Additionally, certain criteria were informed by previous work undertaken within Heart of England NHS Foundation Trust (Sunderland, 2009).Observations of practice/audits were utilized to corroborate data emerging from measurement of the metrics criteria.This data assisted in measuring impact associated with safe and effective care.
The second research strand evaluated the experiences of both patients and staff within the context of local implementation.Patient experiences of care were examined using the Person-centred Climate Questionnaire (PCQ-P) (Edvardsson et al., 2009).Additionally, staff experiences regarding implementation were examined by means of focus group methodology.Both of these approaches enabled an integrated understanding of not only the performance of the ward/unit, but also the experiences of patients and staff involved in the programme.Patient experiences are important to ensure quality around healthcare delivery.Likewise gaining staff perceptions of their work context and the systems within are important to identify areas for improvement.
Strand three examined the impact of the programme upon the context within which care is delivered-the workplace setting.Data were collected via the Context Assessment Index (CAI) (McCormack et al., 2009) and the Nursing Context Index (NCI) (Slater et al., 2009).It was anticipated that these standardized instruments would facilitate a greater understanding of the current systems of care within the a workplace context and the subsequent impact upon the delivery of person-centred, It is important to note that the research team applied a 'precautionary principle' in undertaking this evaluation study, whereby issues of concern were immediately brought to the attention of the senior nurse or midwife on duty and the Director of Nursing and Midwifery/Service Manager.This action was taken to protect patients from harm where instances of unsafe practice had been identified.

Data Analysis
The quantitative data were analysed using the soft ware package SPSS (IBM, 2012).Pre and post intervention measures were compared to identify the effects of the intervention.Descriptive statistics provided a measure of the central tendency and scope of differences within the settings and total sample.Measures of reliability and validity were calculated where appropriate.Independentt-tests were used to compare scores across time points.
Qualitative data were transcribed and analysed using content analysis to identify the major themes of satisfaction, involvement in care anddecision-makingand areas of improvement.The 10-step approach to data analysis promoted by Ely (1991) was adopted so that recurring themes could be identified.

Sample
It was proposed that a probability sampling technique of stratified random sampling be used to identify the patient sample.In relation to staff, a prospective opportunistic stratified sampling frame was used.Calculating a minimum sample size was important for statistical analyses and inference of findings within these populations.
Patients: In regards to sampling patients they were randomly chosen from participating wards/units for stratified variations in regional area, gender and age.Thus, the final suggested sample size across participant units was two hundred and sixty-six (N = 266) in total across all 38 participating sites.This sample size is adequate for facilitating statistical analysis (>200) as suggested by Barrett (2007).
Staff: Similarly with the nurse/midwife population an opportunistic stratified sampling approach was used to examine variations in nurse/midwife backgrounds, gender and ward experience.The sample size for staff was one hundred and ninety (N = 190) in total across all 38 sites.
Full ethical clearance was sought prior to project commencement and a favorable opinion was gained via the respective Regional Research Ethics Committees.

Discussion
Three sources of evidence were employed to address the study objectives: Observations of practice/audits (both self-reported and research team reported) • Standardized questionnaires (patients and nurses/midwives) • Focus groups with nurses and midwives The findings from each of the data sources were triangulated and blended to address each of the research objectives respectively.

Study Objective 1
The evidence explored to address this objective included: observations of practice/audit data on the' test your care' system (conducted by internal auditors) (n = 38 sites); observations of practice/audits conducted by the research team as part of the evaluation research (n = 10 sites) and findings from the focus groups (n = 2).
The evaluation research study commencedin October 2012, therefore, data were examined from the TYC system during a nine month period (October 2012-June 2013).Audit completion rates on the TYC system across the three locations (n = 38 sites) for both of the metrics (Medication Storage and Custody and Medication administration) were reviewed.This indicated a varied completion rate ranging from 57 to 79% across the three locations (Table 2).This also suggests that a significant number of data collection points were not completed.
A potential explanation for the moderate completion rate sacross the settings emerged viathe focus groups.Findings from the focus groups suggested that the audit schedule was too frequent.In these circumstances both auditors and staff may not have accorded the process and findings the high level of importance that was necessary.

Comparative Analysis of Research Team and Self-Report Observations of Practice/Audit Data
The research team undertook planned observations of practice/audits across ten research settings during the nine month period (Oct 12 to June 2013).Data were collected across each of the three locations so that a representative sample was compiled.The research team randomly selected patient medication records from each of the ten settings and these were examined in respect of the metrics criteria.The scores were compared with the relevant scores recorded on the TYC system; this provided a comparative analysis of the observations of practice/audit process.
Findings indicate there were fewer discrepancies in scoring (within a 10% range) when the percentage agreement was at its maximum (100%).When scores were below the maximum level of agreement, observed scores and self-reported scores deviated considerably (e.g., drug prescription 23% observed and 77% selfreported).A trend emerged particularly in criteria relating to the drug prescription, medication chart and medication administration (Table 3).There were differences in scoring, both overestimated and underestimated, on self-reported scores when compared to the research team observations.These differences in scoring may be indicative of confusion in the interpretation of guidelines and scoring.

Observations of Practice/Audit Data-June 2013
At the final point of data collection (June 2013) data were again examined across the 38 settings as determined by the metrics criteria (Table 4).Total DOI: 10.3844/ijrnsp.2015.■■.■■ ■■ percentage scores were calculated and examined for each of the criteria according to location and overall total.

Study Objective 2
The impact of the programme on patient care was measured using the Person-centred Climate Questionnaire-Patients (PCQ-P) across two time points.A total of 147 patients (55%) responded at time 1 and 98 patients (37%) at time2.
Patients perceived that the level of care provided was person-centred, a positive and encouraging score (Table 5).All 17 statements were positively scored on both occasions by patients.Highest scores were reported on statements relating to being in safe hands and feeling welcomed in the care environment.Lowest scores were reported on the aesthetics of the care environment and the potential of getting unpleasant thoughts out of your head.
The 17 items of the PCQ-P were summated to produce a total score that is indicative of person-centred care (Table 5).Measures of homogeneity (internal consistency) provide psychometric evidence to support the summation of items (Cronbach's alpha 0.89).Internal consistency describes the extent to which all the items in a test measure the same concept or construct.Scores range from 0 to 1 with higher scores indicating higher levels of internal consistency.Scores above 0.7indicate acceptable statistical levels (Brace et al., 2006).

Study Objectives 3 and 4
The impact of the programme upon staff experiences and the workplace context were measured using constructs derived from the Nursing Context Index (Slater et al., 2009) and the Context Assessment Index (McCormack et al., 2009).Data was collected from a sample of 97 registered nurses and midwives at time point 1 and 76 registered nurses and midwives at time point 2. Respondents were predominantly female and aged 36-45 years of age.The largest sample across settings was drawn from location 2 at time 1andlocation 3 at time 2.

Measures of Job Satisfaction
Overall, nurses and midwives reported being somewhat satisfied with their job and this sense of satisfaction was consistent across all three locations (Table 6).The nursing and midwifery staff reported being neither satisfied nor dissatisfied with their pay and prospects.There were no statistical differences in scoring across the 3 locations and across time points.
There was a slight sense of dissatisfaction among nursing and midwifery staff in relation to continuous professional development and opportunites to attend training.This was somewhat more pronounced in locations 2 and 3 but not at a statistically significant level.There was no statistical difference in satisfaction with training scores across time points.
The subscale 'Personal Satisfaction' was positively scored indicating nurses and midwives are somewhat personally satisfied with the care they provide and this was evident across all three locations.The total score on 'Professional Satisfaction' indicates that nurses and midwives felt positive, at a professional level, regarding their contribution to patient care.There were no statistical differences in scoring across locations and time points.

Work Stress Scale
The Work Stress Scale examines four areas of nursing and midwifery work life which are indicative of stress.It can be summated to provide a total score and four construct scores.
The Work Stress Scale was measured on a7pointscale ranging from 1-no stress to 7-extremestress.
Overall, nurses and midwives reported having 'Some Stress' and this sense of stress was relatively consistent across the three locations.There was a statistical difference in scores between settings over time (f = 4.54; df = 2; p = 0.012).Work Stress Scale scores at location1 increased in comparison to locations 2 and 3 where a decrease was recorded.These changes were small but statistically significant.

Context Assessment Index
The Context Assessment Index comprises five constructs that assess the context within which care is delivered.These are 'Collaborative Practice',' Evidenceinformed Practice', 'Respect for Persons',' Practice Boundaries' and 'Evaluation'.It is a psychometrically proven instrument that has shown evidence of acceptable validity and reliability and it has also been previously tested (Parlour and McCormack, 2012) within the culture of the Irish Health Service.The Context Assessment Index was measured on a 4-pointLikertscale ranging from 1-Strongly Agree to4-StronglyDisagree.A score of 2.5 indicates a mid-point.
The overall results for the total sample indicate agreement of nurses and midwives on the presence of a positive context for care (Table 7).This was more evident in location 1 when compared with locations 2 and 3. Location 2 had the least positive context of care.This pattern of responding was generally consistent across the five constructs that comprise the measures of context.
The quantitative evidence gathered highlights that the metrics programme had a minimal impact on culture change.Findings from the focus groups provided some explanation of these scoring patterns.The participants in focus group 2 reported that the implementation and impact of the medication management metrics programme with staff was piecemeal.A 'cascade method' of diffusion of knowledge was used to communicate the programme aims, objectives and processes inherent to programme implementation.The evidence from the two focus groups highlights that the transference of relevant information to all staff members did not materialize in a comprehensive and systematic manner.

Table 4. Adherence to metrics criteria
Location ------------------------------- There was an acknowledgement that, if the programme is to be effectively implemented, nursing and midwifery staff require ownership of the programme.Staff reported suspicion regarding the collection of metrics data on such a regular basis and felt that they were being 'checked up on'.Nursing and midwifery management felt that if staff were more involved in the auditing process, this would clarify the objectives of the programme for them; highlight the fact that the programme is designed to safeguard both patients and nursing and midwifery staff; and increase adherence to the requisite changes in practice.

Study Objective 5
The triangulation of source and method data provided a deeper insight in to the overall impact of the metrics programme and helped identify areas for potential improvement.Overall three key themes were identified from the qualitative and quantitative data sources: Theme 1: Engagement with Multi-Disciplinary Partners A major theme to emerge from the focus groups was the impact of the role of medical clinicians.Focus group participants indicated that engagement of medical clinicians was critical to the overall success of the programme.Participants stated that attempts had been made to involve medical clinicians in addressing the deficits identified within the audit process.Medical clinicians were requested to address issues regarding the use of generic drug names, identifying the correct dosage and signing all prescribed medications appropriately.The findings from the focus groups indicate that this process was unsuccessful and engagement with medical clinicians was difficult to achieve.This presents a difficult juxtaposition whereby medical clinicians persist in unsafe practice in medication prescription and administration and nurses and midwives continue to administer medication when not prescribed in a clear and concise manner.Lack of appropriate action around these issues had a significant impact on the quantitative outcomes of the programme.
The way forward for measuring outcomes of care undoubtedly lies in every member of the multidisciplinary team being responsible for the part they play in delivering patient care.They also need to understand the impact their actions or inactions will have on resultant patient outcomes.Crucially, they must appreciate that what they do has the potential to support or negate the actions of colleagues.This will ensure that every member takes responsibility for examining their performance within the multidisciplinary team and that each individual profession recognizes and addresses their own specific issues.

Theme 2: Staff Experiences
A second theme to emerge related to the experiences of staff during programme implementation.Staff experience is an important area for drawing evidence about nursing and midwifery quality (Maben, 2013).Recent research on staff wellbeing has established a link between staff motivation, affect and wellbeing and patient experiences of care (Maben et al., 2012).
Focus group participants felt there were issues regarding the involvement of staff and the effective use of information.They expressed a sense of frustration in that they did not have timely and comprehensive access to the findings from audits and that this restricted the completion of the 'loop' in the audit process.In order for feedback to be effective it must be presented in a clear, consistent and standardized format.The absence of realtime feed back to teams about performance and areas for improvement under mined staff experience around the metrics programme.Ultimately this has the potential to result in adverse outcomes.Focus group participants suggested this could be improved by development and implementation of a more inclusive and comprehensive communication strategy.McCance et al. (2012) highlight the importance of promoting ownership at all levels during the metrics implementation process.In addition, Maben and Colleagues reinforce that usability of any measurement system, the usefulness to end-users (e.g., practice teams) and staff understanding of the purpose and benefits of measurement is central to the efficacy of implementation.To achieve this there is a requirement for the organizational system to embed effective implementation infrastructure that incorporates the metrics programme as an integral part of its strategic direction and governance frame work.
Theme 3: Person-Centred Care; Effective Care; Systems of Carea Nursing and midwifery care services are provided to patients in an environment with complex interactions that can generate harm, errors and unintended outcomes.As a result, patient safety is largely considered an indicator of high performing nursing and midwifery care.A further indicator of effective and personcentred care is the patient experience.This is considered (Dubois et al., 2013) the result of clinical and organizational processes that should optimally ensure patients receive the right care at the right time and in the right way.Such a measure is essential to assess the acceptability and appropriateness of nursing and midwifery care from the perspectives of patients.
In recent times there has been a considerable increase in interest in evaluating the experiences of patients' within the Irish healthcare system and the services they receive (ISQSH, 2010).To date, this intention has remained unfulfilled due primarily to a lack of standardized instruments and problems associated with capturing the 'real' person-centred experiences of patients.
Overall the metrics programme had, at best, a slight improvement, on patients' perceptions of person-centred care across both time points.Healthcare teams, healthcare provider organizations and governments of ten articulate an intention to deliver person-centred care (McCance et al., 2011).It is a central tenet in key national strategy documents such as 'Crossing the Quality Chasm' (Institute of Medicine, 2001); the 'National Service Framework for Older People' (England Department of Health, 2001); National Standards for Safer Better Healthcare (HIQA, 2012); Future Health: A Strategic Framework for Reform of the Health Service (Ireland Department of Health, 2012); and within professional organizations such as An Bord Altranais (2000;2007;2009) and the' Royal College of Nursing' Principles of Nursing Practice (PrincipleD) which identifies the tenets of person-centred care as key indicators of quality care (Manley et al., 2011).
Analysis of data relating to both medication management metrics indicates an inconsistent picture.Further analysis of data from the PCQ (P) indicates that patients felt that they had experienced acceptable levels of person-centred care across all three locations.As previously stated this is a positive outcome but must be situated within the context of the wider research outcomes.

Conclusion
There are valuable lessons to be gained from the experiences of implementing and evaluating systems and processes for nursing and midwifery metrics.It must be recognised that this has encouraged increased transparency as regards the performance of individual wards, units and services.As a consequence, this should also encourage individuals and teams to regain control of the quality of nursing and midwifery care and thus lead to increased accountability.
The study findings have important policy implications that should inform any future corporate approach towards system wide implementation of nursing and midwifery metrics.Outcomes from this should promote improvements in care delivery and enhanced corporate understanding of the central role of nurses and midwives in the delivery of safe, effective and person-centred care.

Table 1 .
Data collection and associated analysis

Table 3 .
Percentage adherence to medication management metrics guidelines

Table 7 .
Context Assessment Index total scores across time points