Evaluation of an Emerging Research Center: Lessons Learned

Problem statement: Rigorous evaluation assures that research endeavors meet their purpose and achieve stated goals. This is especially true for federally funded exploratory research centers, which tend to be more complex due to the involvement of multiple, interdisciplinary investigators. This study provides an overview of the approach used to develop an evaluation strategy and reports the lessons learned during the initial development of the Center for Ohana SelfManagement of Chronic Illness (COSMCI) at the University of Hawai’i at Mānoa School of Nursing and Dental Hygiene. The COSMCI is composed of an interdisciplinary team of researchers and practitioners and aims to advance knowledge in the field of self management of chronic disease in the community setting. Approach: A systematic approach was utilized that included formative and summative strategies for ongoing evaluation. The problem was solved by addressing five key concerns: (1) development of research structure, (2) observing the process of the research pilot projects, (3) scholarly activity of COSMCI faculty, (4) dissemination and translation and (5) sustainability prospects. The method of research included formulating process strategies and determine if the plans for developing the Center were followed and whether these plans were effective. Interviews were also conducted at year one and at mid-point though the project. Results: Themes that emerged from our evaluation included inclusion, timelines, realistic expectations, ongoing evaluation and preparing for changes in the team. This provided timely recognition of successes and challenges and facilitated a rapid response for interventions especially during the early development stage of the center. Conclusion/Recommendations: Effective development of a successful Center is highly dependent upon having a strong evaluation process in place that can inform ongoing development. An exploratory research center requires ongoing evaluation that allows for celebration of successes, as well as early identification of problems and rapid response.


INTRODUCTION
The United States spends approximately $2.25 trillion per year on healthcare (1) and $122 billion on health care related research (2). This represents a significant portion of the national gross domestic product and exceeds that of any other nation (2). The US Federal Government funds a large portion of this health research through grant programs administered by a multitude of agencies, the most notable being the National Institutes of Health (NIH).
The NIH utilizes several types of grants to support research related programs. These grants are categorized as: (1) research grants, which are the most common and usually support discrete, circumscribed research projects; (2) program project or center grants, which usually support integrated, multiple project endeavors that have a wide array of research activities focused on a common purpose or theme; (3) resource grants, which are designed to provide access to specialized research support or resources and (4) trans NIH Programs, which support broad reaching programs across multiple agencies involved in health research endeavors (3). These grant types are all directed toward meeting at least one of the two overarching goals of Healthy People 2010: namely, to increase quality and years of healthy life and eliminate health disparities (4).
A key method to assure that any research endeavor is meeting its purpose and achieving its stated goals is through rigorous evaluation. This is especially true for research centers, which tend to be more complex due to the involvement of multiple investigators from different backgrounds or disciplines who are required to collaborate on a series of different but related research projects. To coordinate the efforts of a diverse team requires planning, a degree of skill and management expertise that is easy to underestimate at project inception. This study describes the evaluation strategy and reports the lessons learned during the initial development of the University of Hawai'i at Mānoa School of Nursing and Dental Hygiene (UH SONDH) Exploratory Center for `Ohana Self-Management of Chronic Illness (COSMCI).

COSMCI background:
Research center grants provide a mechanism to inform different research foci established by NIH. One such center is the COSCMI at the UH SONDH. The COSMCI is a National Institute of Nursing Research (NINR), NIH funded P-20 grant (5P20NR010671), which has a mission to advance knowledge in the field of `ohana (family) and self management of chronic illness in order to improve health related quality of life and reduce health disparities among ethnically diverse populations in the state of Hawai'i with a focus on building interdisciplinary biobehavioral research capacity.
The COSMCI, funded in 2008 for 5 years, has been developed to solidify the research structures within the SONDH. It is comprised of an Executive Committee; an Administrative Core, which includes research and evaluation sub-cores; the Center research projects; an External Advisory Committee (EAC); and center affiliates. The COSMCI began with three pilot research projects that each focused on 'ohana (family) and self management of a chronic illness. These chronic illnesses include: chronic obstructive pulmonary disease, diabetes mellitus and HIV infection. At the beginning of each of three years, one of the three projects was initiated. The COSMCI resources are used to support the three new investigatorsthe project directors for each research project. Ongoing evaluation processes serve to provide information that is used to refine the COSMCI operation.

MATERIALS AND METHODS
Evaluation strategy: The overall evaluation strategy for the COSMCI includes both formative and summative (process and outcome) evaluation strategies addressing the Center's five key aims: (1) development of the organizational research structure; (2) progress of each of the three research pilot projects; (3) scholarly productivity of COSMCI faculty; (4) dissemination and translation; and (5) sustainability prospects. The University of Hawai'i Institutional Review Board (IRB) approved the evaluation protocol.
Formative strategies monitor processes and determine if the plans for developing the Center are being followed; and if so, whether the plans are effective. These strategies include: (1) identification of issues, analysis of successes and opportunities for improvement reviewed at monthly investigator and executive committee meetings; (2) periodic key informant interviews of the COSMCI stakeholders; (3) ongoing monitoring of the project budget and (4) periodic external advisory committee review.
Summative evaluation strategies are used to examine the actual outputs of the COSMCI. Table 1 presents the formative and summative evaluation strategies as they relate to the COSMCI five focus areas with key indicators.

Key informant interviews: Key informant interviews
were conducted at year one and mid-point through the project.
Year 1 interviews: At the end of the first year of the COSMCI, an undergraduate research fellow, who was not associated with the COSMCI, was tasked with conducting telephone interviews of ten (10) COSMCI participants (internal and external). The participants were selected by the evaluation sub-core leader, the project manager and the center Principal Investigator (PI) ensuring representation of the project PIs, the COSMCI cores and the EAC members. The potential participants were called without prior notification and the interviews began with a brief explanation of the purpose of the evaluation and consent for participation. If the participant could not do the interview at that time, an attempt to schedule the interview at a later time was made. The interviews used Strengths, Weaknesses, Opportunities and Threats approach (SWOT) with a question developed for each domain. An extra question was added regarding recommendations for continued development of the COSMCI. Six (6) participants (60%) completed the 10-15 min. interview. Notes were taken during the interview, followed by a content analysis using an inductive approach. Themes were coded as they emerged (if mentioned more than once) from the data and then primary themes and subthemes were identified.

Mid-point interviews:
At the mid-point of COSMCI's development, an evaluator met with the Evaluation Sub-core Leader and the Center PI and developed methods and identification of interview questions. COSMCI participants (Executive Committee members, pilot project investigators and staff and EAC members) were included in the sample. Questions were formulated to evaluate their perceptions and recommendations regarding the activities of COSMCI.
A letter outlining the objectives, as well as an invitation to participate and the interview guide was sent to potential participants under the signature of the PI. For those who agreed to participate, a time was scheduled for an in-person or telephone interview. Implementation and institutionalization sustainability of activities that will support continuation of research related to chronic disease self-management after the NINR P-20 Center funding ends: The interviews were semi-structured and included six open-ended questions that aimed to obtain feedback and recommendations for the Center. Each interview began with a brief explanation of the purpose of the external evaluation and review of the consent form. Interviews averaged about 15 min.
All interviews were audio recorded on a digital recorder and saved with a numeric ID number. The deidentified recording was transcribed and provided to the external evaluator. Transcriptions were imported and coded using NVivo 8 TM . Content analysis used an inductive approach and the identification of codes and analysis for primary and sub themes was conducted across all transcripts. For example, in coding responses for the question, "How well do you think COSMCI is facilitating and supporting sustainability of interdisciplinary, research capacity relevant to 'ohana self management of chronic illness?", participants identified disciplines that collaborate with the Center. Codes for "disciplines" or specific academic departments, e.g., "School of Medicine," "Cancer Research Center," were created as coding themes.

RESULTS
Formative evaluation findings: Formative evaluation included ongoing monthly Center participant meetings, weekly and monthly budget report reviews and key informant interviews that were undertaken after one year and then at the halfway point (approximately after two and a half years) of the Center's five year development schedule. Issues that negatively impacted the formation of the Center were identified and addressed as they arose.

Building leader capacity:
Four key challenge areas were identified to be addressed. First, there were noted delays in the initiation of one project and progress of another due to a change in Principal Investigator (PI) and procurement of supplies and equipment that were required by the projects. The processes were examined and procurement logistics were addressed, which then eliminated future delays in ordering and obtaining supplies and equipment. Second, there was a lack of clarity regarding the role of the research Core Coleaders. The pilot project investigators were not sure who to go to for what and at times this provided mixed messages and an unclear chain of command. It was agreed that the research core Co-leaders would serve as the initial contact for communication, issues and celebrations with the project investigators. As a result, clarity regarding each member's mission served to enhance communication and the flow of the work. Third, many Center participants came to realize that being a member of such a Center entails a large amount of energy and effort. Initially, there was not a full understanding of the professional gain that could be achieved as a Center member and the participation expectations. Discussions were held and consensus was achieved. All involved recognized that the center is a long-term investment and opportunity to build a professional research career in the area of chronic disease self management. Fourth, some of the center participants were not well versed in the processes and procedures for writing collaboratively. This lengthened the process for completing scholarly papers. Once this issue was recognized, smaller writing groups were formed and a master schedule was developed, which identified the papers to be written, authorship order and timelines.

Fiscal management:
The average funding per pilot project was $22,000 per year for two years and did not include PIs' salary support. This in-kind salary support was provided by the SONDH. At times there were misunderstandings regarding the amount of release time provided on an in-kind basis. For one of the projects, the budget barely covered operating expenses required for the project.

Key informant interviews:
The key informant interviews at year one and mid-point through the project are detailed below.
Year 1 interviews: The interviews were conducted relatively early (end of year one) in the building of the center and many of the respondents' answers were similar but appeared to be tentative. In retrospect, this may have occurred because most were still learning how the center would work and participants were not aware of the evaluation until the interviewer contacted them. The strengths and opportunities identified by respondents were that the design of the center was very good and had good intentions, a strong infrastructure existed and there was strong center leadership. Weaknesses and threats included researchers' reports about a lack of support for their projects and fear of termination of the center before it could fully get off the ground. Recommendations to improve the center were to continue the Center's development, advertise center activities and accomplishments and gain more center support.

Mid-point interviews:
At the mid-point of the Center's 5-year development, a second round of key informant interviews based on the EAC's recommendation for qualitative evaluation was conducted to support continued formative evaluation.
The COSMCI Evaluation Sub-Core leader identified an independent external program evaluator who was not affiliated with the center.
A total of 21 out of 25 participants agreed to be interviewed (response rate 84.0%). Of this sample, 14 were COSMCI researchers and staff comprised of coinvestigators, research pilot PIs and co-investigators, sub-core leaders, research assistants, administrative staff, while seven (7) participants were external members comprised of affiliates and external advisors.
COSMCI researchers, staff and external members were satisfied with the Center's activities to support nurse investigator research. The journal club, guest speakers and training in research topics were identified as being helpful. Additionally, many participants and especially the external members discussed how this Center would begin to build and support a research culture in the school of nursing. Participants were also satisfied with evaluation and monitoring activities and external members particularly noted their participation in the second round evaluation to obtain their feedback on the center.
Recommendations to improve three areas were identified based on both responses to the questions on how well the Center had been conducting its activities and the two questions that asked for specific COSMCI suggestions. The three areas noted were: (1) publications; (2) interdisciplinary research and (3) research support. The recommendations were to: (1) develop an actual structure to support dissemination through developing and submitting scientific publications, such as writing groups; (2) develop a structure to foster interdisciplinary research that would include interdisciplinary publishing as some participants believe that interdisciplinary collaboration is not easily assembled and (3) provide direct support of faculty researchers, such as dedicated time and research mentoring of faculty who are new to research, that serves as concrete opportunities and resources to implement research related activities. Low utilization of resources due to proximity of location statistical analysis, were initiated and used by investigators (high volume) Fiscal office support provided System for annual mentoring of faculty relative to research trajectory and planning initiated Ongoing evaluation system conducted via the Evaluation Sub-Core Resource library initiated (contains: 70+ scales; research articles and books) Use of common demographic and chronic illness data elements across projects allowed for \ cross project analysis Newsletter and web site developed and disseminate information on regular basis PI Resource Manual developed Progress of each of the Pilot 1 (COPD study): Initiated on time; Pilot 1. Behind schedule due to change in PI and three pilot research projects feasibility trial initiated difficulty with recruitment of subjects Pilot 2 (DM prevention study): Initiated on time; Stresses related to faculty time for research data collected; analysis in progress; on schedule Pilot 3: Not scheduled to begin until 2011 Lessons learned regarding difficulty and possible solutions to recruitment efforts Scholarly productivity of Manuscript writing group formed Change in research sleader in year 2 COSMCI faculty Articles: One (1) published, One (1) in Need to develop a structure to facilitate for press and two (2) articles submitted publication ongoing manuscript development Initial slow start, but volume now improving and submission revision Presentations: One (1) conference presentation Dissemination and Journal Club initiated Most research development activities translation of Doctoral student seminars initiated; had robust attendance, but for some there knowledge summer grant writing course offered was limited attendance due to (2009 and 2010) conflicts with faculty teaching schedule Research development activities offered: Research Colloquia Summer Writing Workshop Visiting Scholar Lecture series Three day visiting scholar from NIH UHM Scholar Lecture series Copy editor hired Prospects for sustainability P-20 supplement submitted and awarded Difficult to conceptualize as the center is still developing Campus and school commitment to Support the Center as the focal point for nursing research New dedicated facility for research Projects and infrastructure support was acquired U13 grant submitted and awarded for Community based research capacity building for nursing in the USAPI and HI Collaboration between the SONDH and other research units initiated; U01 grant submitted Developed a system for dissemination of grant opportunities (vetted) to All nursing faculty and graduate students Implemented a secure electronic storage System that facilitates data sharing These were addressed by offering a writing retreat, hiring of a copy editor, matching PIs with mentor faculty and providing faculty summer overload support.
COSMCI is viewed as a project that is developing an infrastructure and culture of research at the SONDH. Participants indicated that the COSMCI was fulfilling its objectives in: (1) conducting and supporting activities in research for nurse investigators by providing many research related training and support opportunities for all Center researchers, other SONDH faculty and other departments; (2) extending interdisciplinary research collaborations with other UH departments and (3) implementing evaluations of Center activities and including external members for feedback. Table 2 summarizes the key summative outcomes including successes and challenges for each of the five evaluation focus areas.

Summative evaluation findings:
When examined in totality at the mid-point in the COSMCI project, the majority of key outcomes had been achieved.
External advisory committee: The External Advisory Committee (EAC), a multidisciplinary group of experienced national and local researchers met and provided helpful recommendations between the first year key and mid-point interviews. Their role is consultative and recommendations were critical to the evaluative process.

DISCUSSION
Overall, during the first two and a half years of the COSMCI project, the key aims and objectives of the COSMCI were met. Utilizing a systematic approach for ongoing evaluation provided an opportunity for timely recognition of successes and challenges and action steps where necessary. Themes that emerged from our evaluation included inclusion, timelines, realistic expectations, ongoing evaluation and preparing for team changes.
One of the most important lessons learned was that of inclusion. The COSMCI initiation, mission and vision must be marketed to the entire faculty and support staff of the School, as Center success is dependent upon not only the COSMCI investigators, but also from the School at large. We found that this was important to assure adequate School support for COSMCI. Once we began to market COSMCI more widely within the School, participation in the journal club improved, people became curious about the findings from the studies in progress and additional faculty expressed a desire to initiate chronic disease management research. We learned that Center establishment requires extensive involvement of participants beyond the official Center participants. It is a School endeavor.
Many lessons were learned about timelines as well. Timelines for scheduling meetings and evaluation activities, completion of projects and authorship of publications were significant challenges. Establishment of a new Center requires administrative leadership, which has the capacity to keep faculty investigators from multiple disciplines on target and on track. Scheduling meetings at pre-set intervals helped with creating routines that were easier to adhere to. Some of the projects encountered difficulties with recruitment, IRB application, procurement of supplies and equipment and reimbursement processes. Such difficulties are to be expected and strong administrative leadership was required to work through each of these very different issues. The importance of a strong center administrator cannot be over-emphasized.
When a new Center is being established, such as the COSMCI, realistic expectations must be set. All must recognize that the purpose of funding for such a Center is to establish a program, work through challenges that arise and build a team composed of researchers from different disciplines and schools. Expect that not all will go as planned. As such, issues must be recognized, acted upon and then all must move on. Also, successes must be celebrated as much as failures recognized and addressed.
The use of an external evaluator is an essential component to obtain valid feedback. We learned that someone who is not connected to the investigators or the project itself is more likely to be able to identify occult issues. The year 1 evaluation was conducted by an outside student and the mid-point evaluation was conducted by a faculty member from the University who was not a full member of the COSMCI. For the final evaluation, the COSMCI team will use an external evaluation team. This will serve to preserve the anonymity of the informants to enhance disclosure.
Finally, as with any other long-term project, it can be expected that team members may change for a variety of reasons. The COSMCI team learned that with good record keeping and succession planning, the departure of some team members and addition of new members do not have to disrupt the operations of the project. Developing a system for orienting team members also builds capacity for long-term COSMCI sustainability.

CONCLUSION
Effective development of a successful Center is highly dependent upon having a strong evaluation process that can inform ongoing development and take stock of overall accomplishments. The findings from our formative and summative evaluation activities were used to revise the project during the early development stage, illustrated outcomes and key lessons learned. An exploratory research center requires ongoing evaluation that allows for celebration of successes, as well as early identification of problems and rapid response.
A formal structure to support developing publications, creating interdisciplinary collaboration, submitting publications and more fully integrating evaluation into COSMCI will allow the Center to continue to address its research and evaluation cores' aims. The upcoming summative evaluation will highlight quantitative outcomes specified in the grant's aims. Focusing on evaluation, interdisciplinary collaboration and publishing creates the foundation toward sustainability for long-term research success.

ACKNOWLEDGEMENT
The project described was supported by Award Number P20NR010671 from the National Institutes of Nursing Research. The content is solely the responsibility of the authors and does not necessarily represent the official views of the National Institute of Nursing Research or the National Institutes of Health.
The researchers acknowledge the contributions of Rosa Castro, Chun-I Li and project staff at the University of Hawai'i at Mānoa School of Nursing and Dental Hygiene Center for 'Ohana and Self-Management of Chronic Illnesses in Hawai'i (5P20NR010671) and the Office of Research Development Extramural Associates Research Development Award (G11HD054969) for their assistance. We also thank Ashley Tsumoto and Marla Acosta for editorial assistance.