“I’m reluctant to venture [how PVOs/NGOs should promote sustainability] because it would call for them to come together and to begin to define what their shared vision is. If they can’t agree on that, then how is it going to be operationalized?”

Informant, Sustainability Initiative
 

“Systems of performance management and progress assessment are important to effective management of human activity, but just because good measures of a given issue are not available, it does not necessarily follow that the issue should be ignored.”

International Institute for Sustainable Development (1).

This volume presents the Child Survival Sustainability Assessment (CSSA) methodology: An evaluation framework and process to systematically approach Child Survival (CS) interventions from the standpoint of sustainability.  The CSSA is an outcome of the CORE–CSTS+ Sustainability Initiative, a qualitative research effort led by the Child Survival Technical Support (CSTS+) project and the Child Survival Collaborations and Resources Group (CORE) with the private voluntary organization (PVO) CS community.

The CSSA is presented as a tool helping CS interventions, notably PVO CS interventions, better integrate their plans and monitoring and evaluation (M&E) systems under the overarching purpose of achieving sustainable child health gains.  It seeks to do so through a realistic and contextually relevant systematic approach, yet expecting to increase the ability of the CS community as a whole to be accountable, to learn about and to communicate our common responsibility to the children today and tomorrow.

Background:  a need for improving sustainability and its definition in the context of Child Survival

The CORE–CSTS+ Sustainability Initiative is presented in details in the background volume for this document.[1]  The study stemmed from a range of observations and reflections:

·         On one end, CS projects have considerably evolved over the years, moving away from direct implementation to work through strategic partnerships, capacity building, and efforts to enhance the financial sustainability of basic services.  The projects funded by USAID’s Child Survival Grants Program (CSGP) have increasingly been asked to account for their capacity building and sustainability strategies, from the application stage to the final evaluation (2;3).  Qualitative observational studies (4) and examination of case studies (5;6) also describe meaningful contributions of the PVO community to the goal of sustaining child health.

·         On the other end, echoing the recurrent literature’s questioning of sustainability of Primary Health Care programs (7–10), a recent review of CSGP projects found that most projects had not satisfactorily addressed the problem of sustainability of health services and functions by the end of their grant period, according to their own evaluation reports (3).

·         The development of a research agenda has been hindered so far by the lack of conceptual clarity that has clouded the evaluation of sustainability in CS programs (11).

The CORE–CSTS+ Sustainability Initiative

In September 2000, CSTS+ and CORE launched the CORE–CSTS+ Sustainability Initiative in collaboration with the Johns Hopkins School of Public Health, a qualitative research that included the following steps (Figure 1):

·         A systematic review of the literature,

·         Content analysis of 21 interviews conducted with recognized CS practitioners in the PVO community,

·         A questionnaire—the Critical Issues Survey—administered to 50 CS professionals associated with CORE or the PVO community.

·         A project sustainability self-assessment questionnaire sent to two groups of CS project managers.

 

The study provided many lessons and valuable insights described in the background document (and briefly summarized in the introduction of this volume). Some of the main lessons of the Sustainability Initiative, in terms of improving the parameters of sustainability evaluation in CS interventions, are summarized in Box 1.

Box 1:  Key lessons from the Sustainability Initiative on the evaluation of sustainability in CS interventions

Although it sounds like a tautology, the finality of child survival—improving the health of children, particularly children living in poverty—is a cornerstone of any health intervention claiming to be sustainable. There is not one linear model, but a number of approaches to achieve sustainable results. A final “sustained impact” is the result of complex and multidimensional interplay. There are strong external factors outside the reach of projects and PVOs/NGOs that influence sustainability.  Measuring progress on these external conditions is a crucial part of assessing the prospect of sustainability in CS interventions. “Sustainable results” can often not be reliably predicted. They seem to be due to successful local “negotiations” supported by favorable conditions, which a project can support, but not necessarily control. Elements of definition for sustainable programs that gather a strong consensus from PVOs/NGOs fit within the general heading of “creating an enabling environment” and include “building functionality,” “creating opportunities,” or “developing relations and interdependency.” Capacity building in local partners is essential, but sustainability depends on many other factors.  Increasing the viability of local organizations is another important element, whether it relates to financial viability or other elements of an organization’s “profile of dependency” such as  organizational linkages and support relationships, advocacy coalitions, access to information and technical assistance, and accountability. Improvements in social cohesion (e.g., accountability) or community competence and capacity need to be better understood and better evaluated, but are cornerstones of sustainability. The processes through which health information is diffused or services are provided are extremely important to sustain health gains. Quality, equity, efficiency, or technological appropriateness all contribute to (or constrain) the durability of these benefits. Helping a local system progress toward sustainable health becomes the pertinent role of projects. Sustainability planning, at the Child Survival project level, must find its place within the larger issue of sustainable development. Although projects are only contributors to progress toward the next transitional stage, this contribution is essential in favoring or hindering lasting impact.

 

Offering a definition of sustainability relevant to CS projects

Based on the lessons of the Sustainability Initiative, the following definition of sustainability as it relates to the CS projects can be offered:

Sustainability in Child Survival projects is a contribution to the development of conditions enabling individuals, communities, and local organizations to express their potential, improve local functionality, develop mutual relationships of support and accountability, decrease dependency on insecure resources (financial, human, technical, informational), in order for local stakeholders to negotiate their respective roles in the pursuit of health, wellness and development, beyond a project intervention. The individuals, communities and local organizations constitute a local system with their environment, and it is ultimately their coordinated social interactions and efforts, based on the understanding of their own health and development that will lead to lasting health impact.

The logic of this definition encompasses the loss of control over local processes inherent to project approaches, which places the immediate determinant of sustainability—a local process of negotiation, role definition, and engagement—outside of the full control of a PVO. The responsibility of a PVO is not lessened by this recognized loss of control.  CS projects are in a critical position to advance key conditions in the local system where they intervene, if not directly, then by helping the local communities and stakeholders address these conditions.  Planning and evaluating for sustainability in CS project, hence, requires a new model, taking into account different dimensions in an integrated and systematic approach.  This is the function of the proposed Child Survival Sustainability Assessment methodology presented in this volume.

The Child Survival Sustainability Assessment (CSSA): Toward a shared sustainability assessment framework for CS projects

The CSSA methodology proposes both a framework, which allows approaching systematically the shared dimensions of evaluation on which progress can be measured, and a process for a participatory sustainability assessment with communities and local partners. The process starts with the consideration of the communities, institutional stakeholders, and environment, which define a “local system” expected to own the process of improving health beyond the life of a project.  This systematic approach allows framing a vision and defining consistent goals for sustainability along dimensions shared by all projects, but identifying locally the contextually relevant issues within these common dimensions. It can guide further planning of project activities on the results of the assessment.

The framework’s three main dimensions and their respective components are presented in Box 2 and displayed in figure 1.

Box 2:  The three dimensions of the CSSA and their components

1.   The first dimension consists of elements reflecting the primary health goals of the local system: The first component is the population’s health status (or proxies, such as immunization coverage). The second component consists of elements in the health and social services approach and quality, which will influence the durability of any health improvement, such as access, effectiveness, equity, appropriateness and fit of the activities. 2.   The second dimension consists of elements reflecting local organizational capacity and viability: The first component of this second dimension represents the organizational capacity, which needs to exist in the local partner(s) to maintain performance. The second component represents the organizational viability or the profile of dependency of this key local partner.  Dependency relates not only to financial viability, but also to the other essential types of support on which an organization may depend to continue existing and fulfilling its mission. 3.   The last dimension addresses the conditions in the community and the social ecological systems in which the project evolves: Its first component refers to community capacity and the overlapping elements of cultural acceptance and social cohesion. All these elements can be viewed under the umbrella concept of community competence (12). The second component includes a number of elements within the environment of the project in the largest sense: national policies, the economic and political environment, and the environmental and human development situation.  These elements are frequently, but not always, outside of a project’s scope of intervention.  They may, however, be relevant to a sustainability assessment within a CS project, as they indicate important transitional stages of development, which PVOs/NGOs cannot ignore. 4. Completing this framework is an added dimension of threat identification. Some issues are far beyond the control of a PVO and its partners and can place threats on even the best plans for sustainability. These risks need to be understood for what they are and may warrant contingency plans.

For each component within the three dimensions, the framework suggests issues that a given project may want to include in its assessment, as it builds a coherent picture of how sustainability ought to be addressed in its context.

The essential element of validity of the framework is that progress along these dimensions—as defined through locally meaningful indicators—should describe an improvement in the conditions under which durability has an increasing prospect, while lack of progress along these dimensions indicates a decreasing prospect for durable health impact.

While the framework may be used differently by different organizations, the CSSA suggests a six-stage participatory process[2] to build and implement an evaluation plan.

The six suggested stages are as follows:

1.       Define the system to be assessed, its vision and goals 2.       Identify elements / general objectives for the local system 3.      Choose indicators and performance criteria measuring progress on the determined elements 4.       Measure and map the status of the indicators combining the appropriate evaluation tools 5.       Combine the indicators and build indices as needed 6.      Review results and propose programmatic intervention (including specific project objectives) or policies

The CSSA does not offer directives, or ready-made indicators for project sustainability, but it supports the systematic development of a “dashboard” of sustainability, within which practitioners will develop, experiment with, and refine the necessary measurement tools.

Identifying measurement tools and indicators in the dimensions of the CSSA

The CSSA is strongly based on PVO-shared values and experience.  It does not offer a new measurement tool, but seeks to integrate assessment tools already in use in the CS community.

This volume presents an overview of the types of evaluation tools available to make measurements in the different dimensions.  Some elements of evaluation (e.g., health outcomes) have standardized quantitative indicators and widely available survey tools (13).  But many other elements of evaluation (e.g., organizational capacity and community processes) require qualitative indicators obtained from nonstandardized measurement tools.  Efforts are still ongoing to refine the evaluation of these elements throughout the Public Health community. 

For the various types of measurement tools available, the CSSA suggests building performance criteria describing stages of progress on any given indicator from “minimal,” “emerging,” “medium,” “promising,” to “strong” contribution to sustainability.  Given the multidimensionality of the questions raised by sustainability, the complexity of the issues and the diversity of measurements that can be made, the development of performance criteria will help managers and evaluators:

·         Synthesize the information about a given dimension, if appropriate, by combining indicators into an index score,

·         Compare progress toward sustainability on elements of a different nature, assessed through different tools, thus deriving programmatic implications,

·         Establish comparisons across sites and projects for the sake of cross-learning, benchmarking and improving evaluation tools, and research questions.

Potential of the CSSA as a tool

The approach of the CSSA is congruent with other evaluation trends in Public Health (14), Sustainable Development (15), and business management (16).  One of the strengths of this process is that it has been used successfully in rural development with communities and local partners, through a participatory process familiar to most PVOs/NGOs (17;18).

The CSSA has been presented to PVO staff on different occasions, including in the field.  A workshop report is available about the first training conducted in Mali in November 2001 on planning for sustainability using the CSSA (19).  The dimensions of evaluation and their content, as well as the participatory process and system approach to assessment, have generally been well accepted by those who have been introduced to the framework.  Some questions have been raised about the balance between the comprehensiveness and the simplicity of the tool, usually accompanied by recommendations that sustainability planning focuses on one specific element, which varies with the commentator’s experience and specific concerns (such as institutionalization, capacity, or financial viability). This goes back to the original motivation for the Sustainability Initiative: the fragmentation of models.  As it stands, the CSSA allows focusing on financial viability issues, or institutionalization, or ownership at the community level, or the quality-demand equation of service delivery, depending on the situation.  However, it forces planners to think systematically and integrate evaluation plans.  It can also help evaluators look for critical gaps in a sustainability strategy.

Valid indicators and reliable measurement tools are still needed in most dimensions of the framework as it stands.  Because it integrates measurement tools, instead of adding new “metrics” for sustainability, however, the CSSA will be able to benefit from current and future advances while providing a common structure for communication and exchange of experiences.  Because the CSSA is a flexible methodology, PVOs/NGOs and their local partners may use it to bring to light larger issues from the “global agenda,” such as the Millennium Development Goals (20) or the Child Rights’ agenda (21), when this is contextually meaningful and feasible.

Improving sustainability evaluation in Child Survival, what for?

Sustainability—in spite of cyclical fads in the concerns expressed about capacity versus immediate results in health and development work—remains an unavoidable priority because of a group of issues.  These issues are:  the new relative threats to Child Survival (7), the fate of the many more children who will come after those targeted by today’s programs (22), and the “wellness” of these surviving children as suggested by Foster (11).  Reason dictates that what has been achieved must be maintained, while new threats are addressed.  True impact is measured over time, and interventions that are not based on durable models will not reach true impact.  Similarly, the growing concern with going to scale will have limited relevancy if program models are not sustainable at the initial project stage.

At this stage of the health transition in many developing countries, improving sustainability may be the critical determinant in achieving true impact. Improvements remain limited and isolated without good evaluation.

Next steps for CS managers, researchers, and policymakers

There are two distinct ways to consider the potential evolution of the CSSA as a contribution to improving sustainability evaluation in Child Survival: First as a tool for project management and accountability, then as a guide to policy and research.

In terms of PVO project management and accountability, improving evaluation through the use of a systematic methodology can have certain benefits:

·         The first potential benefit of the CSSA is to shed more light on PVO contributions and improve how they are valued.

·         A clear methodological approach to sustainability assessment will allow PVOs/NGOs to be both realistic about what can be achieved and accountable to all their constituents.

·         Improved evaluation in all relevant dimensions improves the accountability of all stakeholders, local actors, host countries, PVO grantees and donors, by being more explicit about achievements and constraints.

·         Finally, a systematic system assessment approach will improve programmatic and management decisions.

Good policy and program development require good evaluation and valid indicators.  But “when indicators are chosen in a conceptual vacuum, it is very difficult to tell how important or how relevant they are to what people want to achieve” (18).  A shared evaluation model can help us move from assumptions to evidence and practical learning to improve both research and policy.

Some of the research questions will have to address the timelines for observing different transitions in different contexts; the types of capabilities at various levels, which best predict sustained health gains; and the critical stages (thresholds) in local development process, which increase the predictability of sustainable health.  The possibility and the benefits of the postintervention studies sometimes advocated (11) will be enhanced if evaluation and data collection are systematic and share common points of reference.  For the moment, we have little empirical evidence on which to base project plans for phasing out.  Only progress in evaluation can help answer questions about effective phase-out strategies.

Advancing Sustainability evaluation in Child Survival, beyond the “ridiculous” and the “sublime”

In conclusion, exploring such a complex issue as sustainability and proposing a tool for improving its evaluation force us to step back and consider it again in the current context of child health in developing countries.  In practitioners’ discussions about sustainability, there is a tendency to vacillate between two extremes:  On one side, always asking for more sustainability from PVOs/NGOs operating with short timeframes and limited funding (the “sublime”); on the other side, dismissing sustainability as simply irrelevant to project approaches.

The case for the relevance of sustainability as a condition to achieve true Public Health impact is strong (7;22–24) When sustainability is appropriately defined, the argument is still strong, even at the project level, since external resources are finite and are generally conditional on larger and unpredictable geopolitical shifts.

A balanced policy question about sustaining primary health care and Child Survival might be, “Are we currently living with unrealistic expectations within nonvalidated guidelines?”  The recent report to the World Health Organization (WHO) on macroeconomics and health (25) states that “the highest priority is to create a service delivery system at the local (“close-to-client”) level . . . that can reach the poor” (where PVOs/NGOs have demonstrated their competitive advantage!)  It is then essential to be more systematic in demonstrating what can be done by PVO projects, through what strategies, for what long-term gains, at what speed, and at what cost.  Bringing evaluation to the level where it can inform policy decisions is a necessary step to move from debate to learning.  We cannot ignore fundamental evaluation dimensions simply because we still struggle with their measurement or because they require us to examine the benefits of health promotion interventions beyond health outcomes (26), as proposed in two dimensions of the CSSA framework.

A final reason to encourage planning for sustainability truly from the onset of projects (and improving our evaluation systems) comes from the communities themselves, who have shown that their support for health promotion, in a general sense, increases when it is linked to their overall long-term development.  Being teachable and accountable about what progress we are able to contribute as partners of communities struggling for a healthy, viable future, firmly planted in their own hands, is a moral as well as a programmatic imperative.

---