Researchers Without Borders

Gersten, R., Chard, D., & Baker, S. (2000). "Factors enhancing sustained use of research-based instructional practices." Journal of Learning Disabilities, 33(5): 445-457.

This article reviews key findings from school-reform studies of the 1980s on sustained use of effective teacher practices in the classroom and explains their relevance to special education. Recent findings are also highlighted and unresolved issues are identified. An appendix lists sustainability factors and issues.

Greenhalgh, T., G. Robert, et al. (2004). "Diffusion of Innovations in Service Organizations: Systematic Review and Recommendations." The Milbank Quarterly, 82(4): 581-629.

This article summarizes an extensive literature review addressing the question, How can we spread and sustain innovations in health service delivery and organization? It considers both content (defining and measuring the diffusion of innovation in organizations) and process (reviewing the literature in a systematic and reproducible way). This article discusses (1) a parsimonious and evidence-based model for considering the diffusion of innovations in health service organizations, (2) clear knowledge gaps where further research should be focused, and (3) a robust and transferable methodology for systematically reviewing health service policy and management. Both the model and the method should be tested more widely in a range of contexts.

Han, S. S. and Weiss, B. (2005). "Sustainability of teacher implementation of school-based mental health programs." Journal of Abnormal Child Psychology, 33(6): 665-679.

Evidence-based prevention and intervention programs are increasingly being implemented in schools and it therefore is becoming increasingly important to understand the complexities of program implementation under real-world conditions. Much research has focused on the contextual factors that influence program implementation but less work has attempted to provide an integrated understanding of mechanisms (e.g., teacher-training processes) that affect teachers' program implementation. In this paper, we review literature on factors related to teachers' implementation of school-based prevention and intervention programs, then from this review abstract what we believe are four basic ingredients that characterize potentially sustainable teacher-implemented classroom programs. Finally, we present a sequential model, based on these ingredients, of the naturalistic processes underlying sustainability of teachers' program implementation and describe how this sustainability can be enhanced through provision of teacher training and performance feedback from a classroom consultant.

Hargreaves, A. and Goodson, I. (2006). "Educational change over time? The sustainability and nonsustainability of three decades of secondary school change and continuity." Educational Administration Quarterly, 42(1): 3-41.

Johnson, K., Hays, C., Center, H., & Daley, C. (2004). "Building capacity and sustainable prevention innovations: a sustainability planning model." Evaluation and Program Planning, 27(2): 135-149.

This article presents an informed definition of sustainability and an associated planning model for sustaining innovations (pertinent to both infrastructure and interventions) within organizational, community, and state systems. The planning model stems from a systematic review of the literature and from concepts derived from a series of ‘think tanks’ made up of key substance abuse prevention professionals. The model assumes a five-step process (i.e. assessment, development, implementation, evaluation, and reassessment/modification) and addresses factors known to inhibit efforts to sustain an innovation. One set of factors concerns the capacity of prevention systems to support sustainable innovations. The other pertains to the extent to which a particular innovation is sustainable. A sustainability action strategy is presented that includes goals with corresponding sets of objectives, actions, and results that determine the extent of readiness to sustain an innovation. Sustainability tools to assist in implementing the planning model are illustrated, and next steps for the model are discussed. This planning model provides a conceptual and practical understanding of sustainability that can lead to further investigation.

Mancini, J. A. and Marek, L.I. (2004). "Sustaining community-based programs for families: Conceptualization and measurement." Family Relations, 53(4): 339-347.

A conceptual model for evaluating community-based program sustainability is presented, along with a 53-item Program Sustainability Index (PSI). Structural equation modeling (SEM) and principal axis factor analysis indicate strong support for each of the seven elements of the PSI. When considered as an overall model, six of the seven framework elements were retained. Internal consistency for each subscale was acceptable, and acceptable performance validity was demonstrated when the subscales were contrasted with middle-range program results. These findings are discussed with regard to next steps in the conceptualization and measurement of program sustainability, as well as implications for planning and implementing community-based programs for families.

Pluye, P., Potvin, L., & Denis, J.L. (2004). "Making public health programs last: conceptualizing sustainability." Evaluation and Program Planning, 27(2): 121-133.

In public health, programs constitute an important method of improving health, and program sustainability is critical. Knowledge on sustainability raises nevertheless two major issues. The first concerns the social structures within which programs are sustained. The literature suggests different structures however only organizational structures, namely routines, are used for analysis. The second issue concerns the temporal aspect of sustainability that is typically conceived as the final phase of program development after the planning, implementation, and evaluation phases. This [`]stage' model does not allow one to consider that sustainability must be prepared in advance, concomitantly with implementation. These structural and temporal dimensions ground our proposal to re-conceive sustainability. The literature on organizations defines two relevant social structures, one organizational (routines), and one institutional (standards). This in turn suggests three degrees of sustainability. We then emphasize how sustainability is concomitant with the implementation process, by exploring events that characterize these processes.

Shediac-Rizkallah, M. C. and Bone, L.R. (1998). "Planning for the sustainability of community-based health programs: conceptual frameworks and future directions for research, practice and policy." Health Education Research, 13(1): 87-108.

Attention to the sustainability of health intervention programs both in the US and abroad is increasing, but little consensus exists on the conceptual and operational definitions of sustainability. Moreover, an empirical knowledge base about the determinants of sustainability is still at an early stage. Planning for sustainability requires, first, a clear understanding of the concept of sustainability and operational indicators that may be used in monitoring sustainability over time. Important categories of indicators include: (1) maintenance of health benefits achieved through an initial program, (2) level of institutionalization of a program within an organization and (3) measures of capacity building in the recipient community. Second, planning for sustainability requires the use of programmatic approaches and strategies that favor long-term program maintenance. We suggest that the potential influences on sustainability may derive from three major groups of factors: (1) project design and implementation factors, (2) factors within the organizational setting, and (3) factors in the broader community environment. Future efforts to develop sustainable health intervention programs in communities can build on the concepts and strategies proposed here.

Adelman, H.S. and Taylor, L. (1997). Toward a scale-up model for replicating new approaches to schooling. Journal of Educational and Psychological Consultation, 8(2), 197-230.

The ideas presented in this article are meant to stimulate work on the problem of replicating new approaches to schooling. The general framework provides an overview of the diffusion process; the specific steps provide enough specifics to guide planning and implementation of institutional changes and evaluation of their relation to the ultimate efficacy of prototypes. To ensure that this model is directly applicable for efforts to make systemic change in schools and school districts, we are continuing to hone it in such arenas.

Aladjem, D., LeFloch, K., et al. (2006). "Models Matter—The Final Report of the National Longitudinal Evaluation of Comprehensive School Reform." Washington, D.C.: American Institutes for Research.

The National Longitudinal Evaluation of Comprehensive School Reform (NLECSR) is a quantitative and qualitative study of behavior, decisions, processes, and outcomes. It employs a quasi-experimental design with matched treatment and comparison schools. NLECSR seeks to determine the effects of CSR models on student achievement in about 650 elementary and middle schools (grades 3-8), identifying the components of CSR models that are most effective overall, as well as describing the situations and populations for which specific CSR models are most effective. This report presents the results of a federal study conducted by the American Institutes for Research, which utilized the NLECSR to examine comprehensive school reform (CSR) implementation, its impact on schools, and its progression from adoption to implementation to sustainability. It describes: (1) the birth of school reform, that is, the process whereby schools adopt comprehensive school reform models; (2) the design of and findings from a survey measure of implementation fidelity; (3) changes to the reform process within schools, that is, the process of implementation and the contextual factors affecting implementation; and (4) the sustainability of reform within schools. Important indicators of "school health" are also examined along the way--the relationship between model implementation and student academic achievement as well as the relationship between implementation and social capital. Thus, the metaphor of a life cycle of reform is employed. Findings are organized around the major life cycle phases and major outcomes of school-level reform. A brief review of the research on CSR provides a context for reporting the findings. Appended are: (1) NLECSR Analytic Rubric; (2) Variables and Scales; and (3) Additional Tables of Results

Aubusson, P. (2002). "An ecology of science education." International Journal of Science Education, 24(1): 27 - 46.

This article reports on a 15 month study of attempted innovation in school science. The teachers in an Australian secondary school were attempting to introduce a constructivist approach to their teaching of science. The change attempt is interpreted through analogical transfer. In this method of analysis, the school science system is mapped against an ecosystem. That is, the science education system is conceptualized as an ecosystem; a self-sustaining, homeostatic, yet evolving, system of interacting influences. This ecological view of science education provides a way of interpreting the findings of this case study by using biological features of ecosystems, such as succession, evolution, selection and adaptation, to explain stagnation, degradation and change in school science. Implications of this interpretation of school science are considered including a proposed mechanism to promote innovation, such as a constructivist approach, through successive stages and the production and communication of knowledge.

Berwick, D. M. (2003). "Disseminating innovations in health care." Jama-Journal of the American Medical Association, 289(15): 1969-1975.

Health care is rich in evidence-based innovations, yet even when such innovations are implemented successfully in one location, they often disseminate slowly—if at all. Diffusion of innovations is a major challenge in all industries including health care. This article examines the theory and research on the dissemination of innovations and suggests applications of that theory to health care. It explores in detail 3 clusters of influence on the rate of diffusion of innovations within an organization: the perceptions of the innovation, the characteristics of the individuals who may adopt the change, and contextual and managerial factors within the organization. This theory makes plausible at least 7 recommendations for health care executives who want to accelerate the rate of diffusion of innovations within their organizations: find sound innovations, find and support "innovators," invest in "early adopters," make early adopter activity observable, trust and enable reinvention, create slack for change, and lead by example.

Billig, S. H., Sherry, L., & Havelock, B. (2005). "Challenge 98: sustaining the work of a regional technology integration initiative." British Journal of Educational Technology, 36(6): 987-1003.

In this article, we offer a research-based theoretical framework for sustainability, describing those proven qualities of project and innovations that support their sustained existence over time. We then describe how a United States Department of Education Technology Innovation Challenge Grantee, working to promote technology integration in a socio-economically disadvantaged region of the state of Texas, succeeded in creating a sustainable set of activities around its work to support educators’ uses of technology. We examine the factors that served to nurture and facilitate sustainability of the practices associated with technology integration to promote student achievement. We take the tact that it is not the project but rather, the change in practice that is important.

Century, J.R., and Levy, A.J. (2002a). Sustaining change: Study of nine school districts with enduring programs. Paper presented at the annual meeting of the American Education Research Association, New Orleans.

Century, J.R., and Levy, A.J. (2002b). Researching the sustainability of reform: Factors that contribute to or inhibit program endurance. Boston: Education Development Center, Center for Science Education.

The completed Researching the Sustainability of Reform (RSR) project focused on the question of how to maintain the gains of an initial educational change process and support continuing reform over time. Within the broader study of sustainability, the research paid particular attention to systemwide approaches to science education reform as well as to the role that external funds can play in initiating reforms that are sustained.

Colbeck, C. L. (2002). "Assessing institutionalization of curricular and pedagogical reforms." Research in Higher Education, 43(4): 397-421.

Stakeholders often want evidence that curricular and pedagogical reforms will endure, but institutionalization of reforms is typically assessed superficially, if at all. This study involved developing and testing an Institutionalization Process Model. The model was developed from literature on institutional theory and a qualitative investigation of factors influencing institutionalization of externally funded curricular and pedagogical reforms at 7 engineering schools. The reforms focused on content (design), method (group projects), and improving the climate for students underrepresented in engineering. The model posits that regulative, normative, and cognitive institutionalization processes affect the likely diffusion of curricular and pedagogical reforms beyond faculty members directly involved in the reform effort. Subsequently, institutional data and a faculty survey conducted at the seven engineering schools were used to test the model using logistic regression. Findings showed that cognitive institutionalization indicators had a stronger influence than regulative or normative indicators on diffusion of design and group projects. The normative indicator of perceived support for teaching was the only significant predictor of increased sensitivity to the needs of underrepresented students.

Elmore, R.F. (1996) "Getting to Scale with Good Educational Practice" Harvard Educational Review, 66(1).

School organization and incentive structures help thwart large-scale adoption of innovative educational practices. Evidence from the progressive movement and past curriculum reform efforts suggest that wide-scale reforms are ineffective under current conditions. Change requires external normative structures, organizations that focus intrinsic motivation, intentional replication processes, and support for new learning.

Emory, R. (1981) Institutionalization: How Can We Continue Good Practices and Functions When Funding Ends? Part I: A Synthesis of Findings. Portland, OR: Northwest Regional Educational Lab.

Prepared for a meeting of the Northwest Regional Exchange (NWRx) Advisory Board and its staff and consultants, this document first reviews nine reports' findings on nine hypotheses concerning the institutionalization of educational change or innovation, especially as institutionalization relates to the dissemination functions of state organizations in NWRx. For each hypothesis, supporting statements are provided from three or more of the reports. The nine hypotheses concern the role in institutionalization of the following incorporation or routinization of a project, a project's base of support and flexibility, location of administration, administrator involvement, contacts with influencial persons and the public, leadership quality and type, relationship to current practices and values, political environment and funding changes, and training of practitioners in the new function. The document then summarizes each of the nine reports. The reports include a theoretical model of institutionalization, a practitioner handbook, studies of innovations' institutionalization in urban bureaucracies and in a U.S. Marine Corps human relations program, and five studies of federally funded educational projects, involving the Elementary and Secondary Education Act, vocational education, reading, the Research and Development Utilization Program, state education agencies' roles in dissemination, and 250 Massachusetts projects.

Fishman, B., and Krajcik, J. (2003) "What does it mean to create sustainable science curriculum innovations? A Commentary." Science Education, 87: 564-573.

Two key issues for research on the creation of sustainable science curricula are the following: (1) How do we create curriculum innovations that will be used in classrooms after the developers or researchers depart; and (2) How do we create curriculum innovations that are scalable, such that teachers who never have direct contact with the developers may successfully enact them? We introduce the idea of usability as a guiding principle for the successful design of sustainable and scalable curricula, present a framework for examining usability in school contexts, and critique the ideas presented in this issue of Science Education using this framework.

Florian, J. (2000). “Sustaining Education Reform: Influential Factors.” Aurora, CO: Mid-Continent Research for Education and Learning.

Organizational change in educational systems' research has provided valuable lessons that address the improvement of schools. School-improvement programs specifically target schools with underachieving and high-poverty populations of students with the intention of improving educational inequities. While these and other current reform initiatives have incorporated recommendations from research regarding the implementation of innovative programs, less is known about enhancing the sustainability of such programs after the initiatives end. To investigate the sustainability of reform initiatives centered in research-based practice, a study of districts was conducted, termed the "Enhancement Initiative," that had initiated a state-sponsored reform effort 10 years prior to the investigation. It established seven focus areas to be addressed by a district involving a sample of its schools. The Enhancement Initiative promoted locally designed, developed, and implemented reform that focused on identifying measurable student learning goals, research-based and innovative instructional strategies, and school-community partnerships. Four districts that had participated in the Enhancement Initiative from 1990 to 1994 were studied nine years after the reform's beginning to determine whether changes that were implemented were sustained. District representatives were questioned about reform changes that had and had not been sustained, and factors that promoted or hindered the retention of changes made under the reform program. The report presents a review of the pertinent research, and then presents the study methodology and results of factors influencing the sustainability of reform. An appendix includes instruments used for case-study interviews.

Goodman, R. and Steckler A. (1989). "A framework for assessing program institutionalization." Knowledge, Technology & Policy, 2(1): 57-71.

This article presents a framework for assessing the institutionalization of programmatic innovations in nonprofit community agencies and in schools. Institutionalization is the attainment of long-term viability and integration of innovations in organizations. Institutionalization is often characterized as the final stage in an innovation diffusion process. The framework was derived from a qualitative study of ten health promotion programs that were innovations in their host organizations. The framework is a two dimensional matrix: one dimension consists of organizational subsystems; the other consists of levels of institutionalization termed passages, routines, and niche saturation. The cells of the matrix are the basis for assessing program institutionalization.

Harvey, G. and Hurworth, A. (2006). "Exploring program sustainability; Identifying Factors in Two educational initiatives in Victoria." Evaluation Journal of Australasia. 6(1): 36-44.

This paper examines two recent successful school-based health initiatives in Victoria, particularly in relation to factors that seem to foster program sustainability. These programs, dealing with drug education and healthy eating, are described before presenting two different methods (individual and group) used to determine elements that allow for the continuation of such projects. The findings on sustainability from each program are discussed using the broad areas of factors associated with the programs themselves; those associated with the context in which the programs were implemented; and finally, those factors external to the programs and their implementation contexts. These results indicate a strong congruence with factors identified in the literature but also highlight the influence of the use of change theory in strengthening sustainability approaches in program development as well as the need to focus on funding options in forward planning. The possible roles for evaluators in assisting program development and supporting the integration of factors supporting sustained use are also discussed.

Harvey, S. and Peacock, A. (2001) "The Lifecycle of a South-African Non-Governmental Organization: Primary Science Program." Comparative Education, 37(2): 213-229

The case study traces the lifecycle of the Primary Science Programme (PSP) in South Africa, from its initiation in 1983 to its demise in 1999. The case is considered to be representative of the experiences of many South African non-governmental organisations (NGOs), insofar as PSP developed co-terminously with numerous other NGOs within the same political policy environment, serving the same clientele of disadvantaged teachers and depending on the same corporate, voluntary and external funding sources. It illustrates how the social and economic environment shaped PSP's development and demise. The conclusions highlight tensions between quality and quantity, subject versus holistic focus, participatory versus authoritarian management structures, and between styles of relationships with departmental structures. Lessons are drawn concerning the nature of effective InSET and the challenges inherent in taking education programmes to scale. An analogy with the ecological study of a plant's lifecycle is used to illuinate the account.

Hutchinson, J. R. and Huberman, M. (1994). "Knowledge dissemination and use in science and mathematics education: A literature review." Journal of Science Education and Technology, 3(1): 27-47.

A synthesis of the literature on knowledge dissemination and use in education, notably in science and mathematics, is presented. Perspectives have changed in the ways in which knowledge and products are seen to reach potential users. From the top-down, linear models, we have come closer to bottom-up approaches and to the crucial role of linking agents. At present, the most influential approach is a constructivist one, whereby research and other kinds of specialized knowledge is exchanged between researchers and professionals in a mutually constructed social context. While there is still debate over the best predictors of successful knowledge use, the scope of the field has been considerably enlarged by including users' perspectives. To some extent then, specialists in this field are now working in a new paradigm.

Nilsen, P., Timpka, T., Nordenfelt, L., & Lindqvist, K. (2005). "Towards improved understanding of injury prevention program sustainability." Safety Science, 43(10): 815-833.

As policy makers and funders have become more concerned with allocating scarce resources effectively, attention to the sustainability of health intervention programs has increased. However, the empirical knowledge base about factors facilitating or working against sustainability remains at an early stage. The aim of this study was to contribute to improved understanding of the conditions under which community-based injury prevention programs are most likely to attain sustainability. Ten Swedish community-based injury prevention programs were analysed with respect to factors that contribute to or detract from program sustainability. All the programs are integrated within existing municipality structures. Data were collected by means of semi-structured telephone interviews with key informants. The results suggested that different factors are interrelated, with no one factor being either primary or by itself sufficient for program sustainability. Financial, human, and relational resources lay the groundwork for the long-term operation of a program. The "integrated" program model appears to facilitate sustainability, but program intensity is vulnerable to changes in the financial status of the municipality and the priority-setting by municipality political decision makers. Sustainability may be compromised if a program becomes too dependent on a few key individuals. In contrast to financial, human, and relational resources, structural resources (e.g., injury surveillance and goals) appeared to have limited influence on sustainability. The programs were sustained with little evidence of effectiveness, resulting in limited feedback about how to improve a program in order to achieve and maintain long-term effectiveness.

Pluye, P., Potvin, L., & Denis, J.L. (2004). "Making public health programs last: conceptualizing sustainability." Evaluation and Program Planning, 27(2): 121-133.

In public health, programs constitute an important method of improving health, and program sustainability is critical. Knowledge on sustainability raises nevertheless two major issues. The first concerns the social structures within which programs are sustained. The literature suggests different structures however only organizational structures, namely routines, are used for analysis. The second issue concerns the temporal aspect of sustainability that is typically conceived as the final phase of program development after the planning, implementation, and evaluation phases. This [`]stage' model does not allow one to consider that sustainability must be prepared in advance, concomitantly with implementation. These structural and temporal dimensions ground our proposal to re-conceive sustainability. The literature on organizations defines two relevant social structures, one organizational (routines), and one institutional (standards). This in turn suggests three degrees of sustainability. We then emphasize how sustainability is concomitant with the implementation process, by exploring events that characterize these processes.

Pluye, P., Potvin, L., Denis, J.L., Pelletier, J., & Mannoni, C. (2005). "Program sustainability begins with the first events." Evaluation and Program Planning, 28(2): 123-137.

This study examines the process by which a program becomes sustainable. In health promotion, sustainability is usually modeled as the final stage of a program's evolution. In practice, however, this stage model appears deceiving. This article proposes that processes for implementing programs and for ensuring their sustainability are concomitant. Using a multiple-case study methodology, we examined routinization--the sustainability process in organizations--of the Quebec Heart Health Demonstration Project in five community health centers over 10 years. Data came from documents and interviews. The analysis considered themes using the Critical Incident Technique. Our results suggest routinization-related events occurred as soon as the project began, and the occurrence of specific routinization events foresaw the presence of program-related routines. These events were concomitant with those associated with implementation. This supports the proposition. It suggests health promoters should consider program sustainability from the very beginning, and can take inspiration from the events presented herein when looking for ways to influence the sustainability of their programs.

Racine, D. P. (2006). "Reliable effectiveness: A theory on sustaining and replicating worthwhile innovations." Administration and Policy in Mental Health, 33(3): 356-387.

While many health and human service innovations are sustained and replicated, it has been a puzzle how to sustain and replicate the performance of the better ones. What knowledge, skills, and conditions are required to reproduce across space and time the effectiveness of those innovations that are the most worthwhile? An extensive body of literature and experience is reviewed to suggest a comprehensive conceptual framework of programmatic, organizational, and environmental factors that may shape the circumstances for sustaining and replicating effectiveness.

van den Berg, R., Vandenberghe, R., & Sleegers, P. (1999). "Management of innovations from a cultural-individual perspective." School Effectiveness and School Improvement, 10(3): 321-351.

Although educational improvements have occurred over the past 30 years, few have had optimal effects. Three functional areas (teacher-organization-intervention) jointly influence an innovation's success. Realization of innovations appears to be more influenced by subjective experiences and culturally bound opinions than by well-managed social-change processes.