Back to resources

FAQs

FOI

1. What is FOI?

There is no clear agreement in the field about how to describe or define fidelity of implementation (FOI) in great detail, but there is a general consensus about what it is. Most definitions, particularly in education, are grounded in the intervention (e.g., instructional materials, professional development program) developer’s point of view. An example of this is “The degree to which instructional materials are being used in a classroom setting as intended by their developers” (Bodzin et. al, 2003). However, there are other definitions that do not seem to be limited to the developer’s point of view. For example, Brekke and Wolkon suggest that FOI is “Whether or not the treatment was implemented as intended” (1988) but do not clarify whose intentions (e.g., the developer’s, the teacher’s, the school district’s). Regardless of the definition of FOI, many agree that the ideas of critical components and multidimensionality are essential to understanding this phenomenon.

To further examine definitions of FOI:

Critical Components

Much of the implementation literature, first in health and then in education, includes an approach to understanding FOI based on the identification and definition of program components that are more critical and less critical achieving desired outcomes (Bond et al., 2000; Ruiz-Primo, 2005). In this approach, critical components are the variables or constructs researchers need to measure in order to determine a program’s FOI. Leithwood and Montgomery (1980) suggest a strategy to identify critical components, that includes getting information from the program developers, written materials produced by the developers, those involved in the implementation of the program.

For more information on early conceptualizations of program components, look at Wang et al (1984) who discuss the idea of structure and process types of critical components. Mowbray, Holter, Teague, & Bybee (2003) and Lastica & O’Donnell (2007) offer more recent perspectives on structure and process. Also look at Hall and Hord (1987) who address the analysis and organization of critical components into innovation configurations.

FOI is multidimensional

Early conceptualizations of FOI characterized implementation with broad terms. Hall and Loucks (1977), for example, focused on “levels of use,” and Fullan & Pomfret (1977) characterized FOI by suggesting that there are five aspects of implementation in practice: changes in materials, structure, behavior, knowledge, and value internalization. Since then, researchers, particularly those working in the field of health, have developed more refined approaches for characterizing and assessing FOI that account for the “complexity of implementation” and the fact that it is “multidimensional” (Fullan, 1983; Gersten et al., 1981; Wang et al., 1984). Most notably, Dane and Schneider (1998) conducted a review of prevention programs and identified five dimensions of FOI that were measured in the studies: adherence, exposure, quality of program delivery, participant responsiveness, and program differentiation (Ruiz-Primo, 2005). Over the last several years, educators have built from this foundational work in health toward measuring FOI of educational programs with a level of specificity that allows the identification of program components that are more and less critical for achieving desired outcomes (Bond et al., 2000; Ruiz-Primo, 2005).

Back to top

2. Why measure FOI?

Health, education and other interventions operate in complex and highly variable environments, and if research is to lead to answers with meaningful applications, we must use tools that help us describe and understand if and how that complexity affects the interventions and their outcomes. Careful description and measurement of implementation are essential if we are to know whether poor outcomes are due to shortcomings in the intervention itself or if it was not implemented as the model intended.

But fidelity of implementation measures are important for more than telling if an intervention is effective. Smaller scale efficacy studies also require FOI measures to help further establish the impact of and improve interventions under development. Developers may also use them in their development processes to improve their programs or improve the implementation of their programs (Bodzin et al., 2003).

Others might use FOI measures because they are interested in understanding the factors that influence implementation. For them, FOI is a dependent variable, and as they look at issues of culture and context, support, resources, and other variables that can affect implementation, strong FOI measures can help them more accurately understand the impacts that these factors have on intervention use.

And finally, FOI measures can be useful for practitioners to inform planning for professional development or as tools for self-analysis and reflection. By understanding how programs are being used, practitioners or professional development providers can seek out or tailor supports to fit their specific implementation needs.

For more information about the importance of measuring FOI, see:

Back to top

3. What should I consider when looking at fidelity?

Depending on your purposes for looking at FOI, there are a number of things to consider. First it is important to clarify the implementation indicators of the program you are measuring. In many cases, these will be the components of the program itself. For example, if you are measuring FOI of an instructional materials program, you may measure the teacher enactment of the program structures (e.g., the use of the sections within the lesson, writing structures, assessments). You also may want to measure the teacher's use of intended instructional strategies or other elements of the program (e.g., professional development, online resources).

During this exercise, you should consider the level of specificity with which you want to measure these elements. This includes considering the frequency with which these components are used (e.g., students use their journals daily, students use their journals once a week, students use their journals at all) and the level of detail of these components (e.g., the use of discussion, the use of questioning within discussion, the use of wait time within questioning).

After determining what you are measuring to determine fidelity, you should consider the best sources of data for those indicators. In the case of educational interventions, potential data sources could include firsthand experience (i.e., observations), teachers, students, school leaders, district leaders, or records. While looking at potential sources, it is important to consider which sources will yield the most reliable data. For example, a school leader might not be the best person to ask about the extent to which students are completing assignments in each classroom, but might be more knowledgeable about the amount of time teachers spend on a unit.

Finally, as you are measuring FOI, you should consider the factors that might affect program implementation (e.g., resources, presence of a specialist, behavior disruptions) and if and how you want to account for and measure them.

For more information on what to consider when measuring FOI, see:

Back to top

4. How can I measure FOI?

Researchers have created many different approaches to measuring FOI. Some look at the number of critical components that were enacted and then conduct analyses to create single scores of fidelity (Bond et al., 1997;Bruns et al., 2004; Balfanz et al, 2006). Others focus on more specific indicators such as the percentage of worksheets students completed and the number of standards covered and objectives mastered (Songer and Gotwals, 2005; Ysseldyke et al., 2003). And still others have used qualitative information gathered from observations, interviews, and questionnaires to categorize types of implementers (e.g., low, medium, and high) (Spillane & Zeuli, 1999; Stonewater, 1996).

For more information on how to measure implementation, see:

Back to top

5. What tools are available to measure FOI?

Many of the instruments for measuring fidelity of implementation were made for specific programs and used for purposes of formatively evaluating programs or measuring implementation studies of program efficacy and effectiveness. They include a variety of tools, including observation protocols, questionnaires, logs, interview protocols, document review protocols, walk-through protocols, and others.

Still, there are similarities across instruments that are used for different programs. One of the most often used instrument organizational formats is what Hall and Hord refer to as “Innovation Configurations” with each configuration comprising combinations of critical program components enacted in particular degrees (1987). Many other instruments, as well as the CEMSE suite measure the presence and extent of use of individual program components.

Specific instruments can be found in the RWB database.

Back to top

6. Is more fidelity better? Is it okay to adapt?

Some believe that it is best to implement an innovation with as much fidelity as possible, particularly innovations that have been shown to be effective. Others believe that in order to be most effective, innovations must adapt. Those who take a fidelity perspective contend that implementation should occur as intended by developers, whereas the adaptation perspective allows for changes to occur to fit specific contexts. Regardless of which perspective is taken, research suggests that when users enact programs, the programs change as a result of operating in different settings with different contexts (Bodzin, et al., 2003; Buston, et al., 2002).

Indeed, it is widely recognized that there will always be variation in implementation of educational innovations when they are implemented in various sites, and many even suggest that adaptation is necessary for a program to be successful. Some feel that adaptation is too much when the innovation can no longer be recognized. Others, however, may argue that the important parts of an innovation may not be the recognizable structures but rather the ways of thinking and acting that create the desired outcomes.

Data generated through FOI studies can help inform the long standing question of how much adaptation is acceptable when implementing a program.

For more information on the tension between fidelity and adaptation see:

Back to top

Sustainability

1. What is sustainability?

A review of the literature suggests that the term sustainability is usually used along side or in place of words such as maintenance, institutionalization, and scale-up. Although there is no agreed upon definition in the field, sustainability is usually characterized as the lasting of an innovation through the capacity to evolve and adapt to changing conditions over time. While sustainability is certainly related to the implementation of a reform or innovation, an open question remains as to whether sustainability is simply a stage in implementation. Additionally, researchers must consider what is being sustained. The thing being sustained could be the effects of a program, the structures of an innovation, or merely the name of a reform, as well as others.

Resources:

2. What are the mechanisms and processes that affect sustainability?

Many processes and mechanisms affect sustainability and, like other aspects of understanding sustainability, they suffer from a lack of shared language and definitions. Most commonly, people talk about the processes of dissemination and diffusion. Many turn to Rogers (2002) definition of diffusion, “the process through which an innovation is communicated through certain channels over time among the members of a social system.” However, more generally speaking and perhaps more significant in this context, many suggest that diffusion is the process of spread that happens in a sense, somewhat passively as an innovation (or elements of it) move from one use to another. In contrast, dissemination is more frequently viewed as having more intention to the movement. Basch et. al. (1985) suggest that while diffusion is the spread of new knowledge to a population, dissemination is a deliberate effort to spread that innovation.

In addition to these process of movement, others also talk about mechanisms of movement such as replication, translation and adaptation. Replication implies that there is an exact duplication of the innovation, while translation and adaptation suggest that the innovation changes either intentionally or unintentionally (or both) during movement. These mechanisms lie at the crossroads of the sustainability issue and the fidelity of implementation issues. Understanding them helps address the long-standing question: as innovations spread, how much adaptation is acceptable over time?

Finally, another common approach used to discuss mechanisms and processes of sustainability are phases. Many have created models that indicate the phases of movement of an innovation. Some of these are linear; others are more dynamic and cyclical. While these approaches can be helpful, their utility varies depending on where in the process the innovation is and the length of the long-and short-term time horizons.

3. What are the factors affecting sustainability?

Many researchers have identified factors that affect sustainability and its related terms. A review of the literature indicates that many of the factors that researchers have identified as affecting sustainability also have been found to affect related concepts, such as implementation, adoption, dissemination, diffusion, and use of practices. Additionally, the same factors were identified across different fields (e.g., health, education, business) and different types of innovations (e.g., whole school reform, evidence-based practices, Total Quality Management manufacturing system).

It is important to note that these factors have been found to be both positive and negative indicators of sustainability, depending on the context. For example, having a collaborative change process is found to be an important part of sustaining innovations, but having too much collaboration can stymie an innovation.

After surveying the literature on sustainability, we identified six organizing categories and 28 factors that previous researchers have identified as sustainability factors. Those factors and categories are listed in the table below.

Category Factors
Characteristics of the Innovation Adaptability; Scope; Understandability; Specificity; Effectiveness; Complexity
Elements of the Environment Inside the Organization (Internal Social Climate; Intra-organizational Networks; Embedded Consultant; Internal Organizational Structure)
Outside the Organization (External Political Climate; External Social Climate; Extra-Organizational Networks; External Consultant) Resources
Opportunities for Learning
Locus of Decision-Making
Characteristics of People in the Organization Characteristics of the Implementer; Experiences of the User; Leadership
Strategies Planning; On-going Evaluation; Collaborative Change Process
Fit Fit With Values and Beliefs; Fit With Needs; Fit With Current Practice
Emotional Mediators Incentives; Trust

Resources: