Moderators of the Effectiveness of Adult Learning Method Practices

Problem statement: The purpose of the analyses reported in this study was to identify the moderators of the effectiveness of evidence-based a dult learning method practices. Approach: The particular practices that were found most effective in 58 randomized controlled design studies of four adult learning methods were examined to identify th e conditions under which learner outcomes were optimized. Results: Thirteen different practices had moderate to large eff cts on learner outcomes. The particular practices that had positive learner benefits were ones that actively involved learners in acquiring new knowledge or skills. Further analysis showed that optimal learner benefits were realized when 4 or 5 different practices were used in combin atio with fewer than 40 participants in applied settings and the instruction or training lasted mor e than 20h and was conducted on multiple occasions. Conclusion: Implications for professional development are desc ribed in terms of taking the moderators of adult learning practices (number of a dult learning practices, hours of instruction or training, real-life settings and number of learners ) into consideration to ensure optimal learner bene fits.


INTRODUCTION
The term adult learning refers to a collection of theories and models for describing the processes and conditions under which learning beyond secondary school education is optimized (Merriam, 2001;Smith and DeFrates-Densch, 2009;Yang, 2003). Adult learning is premised on a learner's readiness-to-learn, self-directedness, active participation in the learning process and a solution-centered approach to knowledge and skill acquisition (Knowles et al., 2011). Central features of nearly all adult learning theories and models are active learner participation in acquiring and mastering new knowledge or skills (Kolb, 1984) and reflection and self-assessment of the mastery of the knowledge or skills (Schon, 1990).
The extent to which different adult learning methods are effective in terms of influencing learner knowledge and skills, attitudes and beliefs and the adoption and use of innovative or novel practices, has been the focus of both narrative and systematic reviews of adult learning research studies (Smith and Gillespie, 2007;Taylor, 2007;Tusting and Barton, 2006). Dunst et al. (2010a;2010b) recently completed a meta-analysis of four adult learning methods where the focus of analysis was unpacking (Dunst and Trivette, 2009b;Kelly and Perkins, 2012) and unbundling (Lipsey, 1993) the learning methods to identify which characteristics of the methods best explained learner outcomes. This study includes further analyses of the studies in the (Dunst et al., 2010a) meta-analysis to identify the adult learning method practices that mattered most in terms of explaining changes and improvements in learner knowledge, skills, attitudes and self-efficacy beliefs.

Adult learning methods:
The four adult learning methods that were the focus of analysis were selected because they have been used to facilitate or improve learner outcomes and each has been investigated using randomized controlled designs. The studies compared participants who received either instruction or training using the adult learning methods with participants who received no intervention, or compared participants who were taught using the adult learning methods with participants who were taught another type of instruction or training.
The four adult learning methods were accelerated learning, coaching, guided design and just-in-time training. Accelerated learning includes procedures for creating a relaxed emotional state, an orchestrated and multi-sensory learning experience and practices for promoting active learner engagement in the learning process (Meier, 2000). Coaching is a "method of transferring skills and expertise from more experienced and knowledgeable practitioners…to less experienced ones" (Hargreaves and Dawe, 1990) and includes procedures for joint planning and goal setting, learner information gathering and application and learner reflection and coach feedback (Leat et al., 2006). Guided design is characterized by a decision-making and problem-solving process that includes procedures for using real world problems for mastering learning content (Wales and Stager, 1978). Just-in-time training includes different methods and strategies used in the context of real-life challenges in response to learner requests for guidance or mentoring (Beckett, 2000). The four methods have been the focus of review or synthesis (Moon et al., 1986;Showers et al., 1987;Trivette, 2007) but not at a micro-level where disaggregation of the methods was the main purpose of investigation.

MATERIALS AND METHODS
Search strategy: Fifty-eight studies were located by controlled vocabulary and natural language searches of 11 electronic databases (ERIC, PsychInfo, MEDLINE); hand searches of the reference sections of published and unpublished adult learning method papers located by the electronic database searches; hand searches of journal articles, book chapters and books on adult learning in our Institute's library; searches of websites dedicated to each adult learning method; Google, Google Scholar and Ingenta searches; and Social Sciences Citation Index author searches of seminal papers by individuals closely aligned with each adult learning method. The studies included 2,095 experimental group participants and 2,213 control or comparison group participants. The complete list of studies, the characteristics of the study participants, which studies included which adult learning method practices and the outcomes in each of the studies, can be found in (Dunst et al., 2010b).
Data coding: Each of the adult learning methods was coded in terms of the practices used to present new information or knowledge to learners, engage the learners in application of the information or knowledge and promote learner deep understanding of the knowledge or practice (Bransford et al., 1999). Presentation included the methods and procedures used by instructors or trainers to: (1) Introduce new knowledge, material or practices and (2) illustrate or demonstrate the use of the knowledge, material or practices. Application included the methods and procedures to engage learners in: (1) The use of the knowledge, material and practices and (2) evaluation of the outcomes or consequences of application. Deep understanding included the methods and procedures for: (1) Engaging the learner in reflection on his or her learning experience and (2) self-assessment of mastery of the newly acquired learner knowledge or practice. Twenty-two adult learning method practices were used to introduce new knowledge and practices, have learners apply the new knowledge and skills and promote learner deep understanding of the knowledge or practice. Table 1 lists the practices that were used in the studies.
The studies included one or more of the following outcomes: Learner: (1) knowledge, (2) skills or practices, (3) attitudes and (4) self-efficacy beliefs. The knowledge and skills outcomes were combined and used as a measure of learner competence and the attitudes and belief outcomes were combined and used as a measure of learner confidence. This tactic was used in order to have a sufficient number of effect sizes to disaggregate the results.

Data analysis:
Cohen's d effect sizes for the mean difference in the post-test study outcomes between the experimental and control or comparison groups were used to identify the most effective adult learning method practices . The average weighted effect size for each practice as well as the 95% confidence intervals for the average effect sizes was used for substantive interpretation of the findings. The effect sizes were examined in a number of different ways to identify which practices under which conditions were associated with optimal learner outcomes. Q BET was used for between group comparisons and χ 2 for linear trends was used to test for incremental increases in effect sizes as a function of different combinations of practices ).

Main findings:
All but a few of the adult learning method practices listed in Table 1 were significantly related to the study outcomes see (Dunst et al., 2010a). The sizes of effect for the individual practices however varied considerably. Table 2 shows which practices for each adult learning method characteristic were associated with the largest average effect sizes. Several things can be discerned from these results. First, the average effect sizes were all medium to large, indicating that the practices were effective in terms of changes or improvements in learner outcomes. Second, the effect sizes for practices actively involving learners in the learning process (practicing, evaluation, reflection and mastery) were almost twice as large as those for the two instructor characteristics (introduce and illustrate). Third, two of the three practices used to introduce learners to new knowledge or practice also actively involved the learners in introduction of the learning content or practice (out-of-class learner activities and pre-class learner exercises). The results highlight the importance of active learner participation in acquiring new knowledge or skills.
The investigators of the studies used from none to five of the practices shown in Table 2. A 5-Between Number of Practices comparison was statistically significant, Q BET = 126.06, df = 4, p<0.00001. The average effect size for studies using none of the practices was d = 0.02 (95% CI = -0.05-0.08) and the average effect size for studies using five practices was d = 1.22 (95% CI = 0.91-1.53). A test for a linear trend found an incremental increase in the average effect sizes for the relationship between number of practices and sizes of effect,χ 2 = 60.83, df = 1, p<0.0001, indicating that the use of a combination of practices was associated with optimal positive learner outcomes. Figure 1 shows the relationships between three different combinations of practices and two types of learner outcomes (knowledge/skills and attitudes/beliefs). There were statistically significant between practice group differences for both knowledge/skills, Q BET = 48.01, df = 2, p<0.00001 and attitudes/beliefs, Q BET = 20.67, df = 2, p<.00001. Tests for linear trends found incremental increases in the average effect sizes for the relationships between the number of practices and both learner knowledge/skills, χ 2 = 30.31, df = 1, p<0.0001 and learner attitudes/selfefficacy beliefs, χ 2 = 17.53, df = 1, p<0.0001. The use of 4 or 5 practices was associated with large effect sizes for both learner competence and confidence.

Moderator analyses:
The extent to which the relationship between the practices shown in Table 2 and learner outcomes was moderated by length of instruction or training, the setting where the instruction or training took place and the number of participants who received the instruction or training, was also investigated. Instruction or training was provided on one to more than 30 occasions lasting only 1 to more than 200 h. The instruction or training was conducted in either university classrooms or in settings where the study participants worked. The number of participants who received instruction or training ranged from 10 to more than 200 in any one study. Figure 2 shows the average effect sizes for both instruction or training lasting less than 20 h (Mean = 8.60, SD = 6.56) and more than 20 h (Mean = 45.45, SD = 17.57). There were statistically significant between practices differences for instruction or training afforded less than 20 h, Q BET = 58.26, df = 2, p<0.0001 and more than 20 h, Q BET = 34.05, df = 2,p<0.00001.   As can be seen in Fig. 2, instruction or training lasting fewer than 20 h where none or only one of the practices were used was not effective in influencing learner outcomes. In contrast, instruction or training lasting more than 20 h where 4 or 5 of the practices were used was associated with the largest average effect size. The pattern of results was very much the same for the knowledge/skills and attitude/belief learner outcomes. The extent to which the study outcomes were influenced by setting was examined by comparing instruction or training conducted in university classrooms with instruction or training conducted in the participants' work environments. There were between practice group differences in the sizes of effect for the learner outcomes when more of the practices listed in Table 2 were used in either university, Q BET = 4.32, df = 2, p<0.04, or nonuniversity, Q BET = 24.64, df = 2, p<0.00001, settings. The average effect sizes, however, were more than twice as large when the instruction or training was conducted in the participants' work settings. The average effect size for instruction or training in university settings was d = 0.30 (95% CI = 0.26-0.34) and the average effect size for instruction or training in nonuniversity settings was d = 0.65 (95% CI = 0.59-0.71), Q BET = 32.55, df = 1, p<0.00001. The results were very similar for both the knowledge/skills and attitude/beliefs learner outcomes. These findings indicate that learning afforded in settings where there was an immediate opportunity to apply newly acquired knowledge or skills was more effective than learning in settings where there were few or no such opportunities. Figure 3 shows the relationship between number of practices and learner outcomes for instruction or training including 40 or fewer participants (Mean = 25.66, SD = 9.52) and more than 40 participants (Mean = 86.25, SD = 57.31). There were significant differences between number of practices and learner outcomes for instruction or training including fewer than 40 participants, Q BET = 34.83, df = 2, p < 0.00001 and for instruction or training including more than 40 participants, Q BET = 26.79, df = 2, p < 0.0001. Tests for linear trends, however, found incremental increases in the average effect sizes for instruction or training of less than 40 participants, χ 2 = 34.82, df = 1, p < 0.00001, but not for instruction or training of more than 40 participants, χ 2 = 3.18, df = 1, p = 0.07. As can be seen in Fig. 3, the more practices that were used with a smaller number of participants, the larger the effect sizes.

DISCUSSION
The findings from the disaggregation of the effect sizes in the meta-analysis showed that practices that more actively involved learners in the learning process had larger effects on learner knowledge, skills, attitudes and self-efficacy beliefs and that a combination of practices had optimal effects on learner outcomes. The influences of the adult learning method practices were also optimized when the practices were used with a small number of learners where instruction or training was afforded in the participants' work settings for more than 20 h on multiple occasions. The results show the conditions under which the adult learning methods had maximum positive benefits.
The practices found most effective in the metaanalysis have been used in a number of intervention studies where our colleagues and ourselves promoted early childhood practitioners' adoption and use of family-systems intervention practices (Dunst et al., 2011a) and early childhood classroom practices (Dunst and Raab, 2010). In both of these studies, training provided to small numbers of practitioners on multiple occasions where the practitioners were actively involved in immediate application and use of the newly learned knowledge and skills were more effective than more traditional types of training (e.g., presentations or workshops). The results from a recently completed meta-analysis of the effectiveness of training provided to professionals and parents to promote their adoption and use of assistive technology and adaptations with preschool-aged children with disabilities were remarkably similar to those reported in this study (Dunst et al., 2011b).
The results from the meta-analysis described in this study as well as elsewhere (Dunst et al., 2011a;2010a), together with the results from the studies briefly described above, were used to develop an evidencebased approach to adult learning called PALS (Participatory Adult Learning Strategy; Dunst and Trivette, 2009a). PALS includes both instructor/trainer and learner practices for each of the six adult learning method characteristics examined in this study (Table 1 and 2) to ensure that a combination of evidence-based practices are used to develop instruction or training and incorporated into how the instruction or training is conducted. The PALS model has direct implications for professional development inasmuch as it can serve as a model or framework for developing instruction or training that includes practices found effective in promoting practitioners' acquisition of new knowledge and skills.

CONCLUSION
Findings showed that the effectiveness of evidence-based adult learning method practices were moderated by both the hours of instruction or trianing and the number of learners taught using the adult learning methods in real-life settings. The number of adult learning method characteristics incorporated into the learning process also moderated the effectiveness of the adult learning method practices. The results from the analyses disaggregating the effectiveness of adult learning method characteristics indicate a need to consider the moderators as part of instruction or training to ensure optimal learner benefits.

ACKNOWLEDGMENT
The preparation of this article was supported by the Research and Training Center on Early Childhood Development, a grant funded by U.S. Department of Education, Office of Special Education Programs, Research-to-Practice Division (H324K010005). The opinions expressed, however, are those of authors and no endorsement by either the Department or Office should be inferred.