The CORE/CSTS+ Sustainability Initiative

This report describes in detail the context, design and results of the CORE-CSTS+ Sustainability Initiative—a qualitative research effort led by the Child Survival Technical Support Project (CSTS+) and the Child Survival Collaborations and Resources Group (CORE) to improve the approach to sustainability, its evaluation and planning, in the PVO (Private Voluntary Organization) Child Survival community.

Sustainability in the Context of Child Survival

Sustainability has been a major concern of international health program professionals for a long time, but most notably since the 1990s. (3-5) It is of particular importance to the Child Survival projects operated by Private Voluntary Organizations (PVOs/NGOs) under the United States Agency for International Development (USAID) Child Survival Grants Program (CSGP), as these projects target child mortality in some of the poorest countries and regions of the world. Private Voluntary Organizations (PVOs/NGOs) have been implementing child survival (CS) projects in developing countries through the CSGP since 1985.(6;7) The need for effective community-based Primary Health Care (PHC) programs to become sustainable and reach scale has led to increasing emphasis on project sustainability. (6;8-11) This increased emphasis has, however, met unequal results: (6;7)

• On one end, CS projects have considerably evolved over the years. They have moved away from direct implementation to work through strategic partnerships, capacity building, and efforts to enhance the financial sustainability of basic services. (7) End-of-project (11;12) and post-intervention case studies(13) also describe some positive results of PVO projects in terms of sustainability.
• On the other end, echoing the recurrent literature’s questioning of sustainability of Primary Health Care programs (3-5;14), a recent review of CSGP projects found that most projects had not satisfactorily addressed the problem of sustainability of health services and functions by the end of their grant period, according to their own evaluation reports. (7)

Finally, the advancement of a research agenda and the improvement of evaluation systems have been hindered so far by the lack of conceptual clarity that has clouded the evaluation of sustainability in CS programs.(15)

The CORE-CSTS+ Sustainability Initiative

These observations gave CSTS+ and CORE the impetus to launch the CORE-CSTS+ Sustainability Initiative, a qualitative research which included the following steps:

• A systematic review of the literature,
• Content analysis of 21 interviews led with leading CS practitioners in the PVO community,
• A questionnaire—the Critical Issues survey—administered to 50 CS professionals associated with CORE or the PVO community,
• A project sustainability self-assessment questionnaire sent to two groups of CS project managers.

Literature overview

The literature makes a strong case for the relevance of sustainability in child health interventions, with the general observation that sustainability of health programs in developing countries remains an unresolved issue in many ways. (4;14;16) The value of sustainability is based in large part on concerns for the welfare of future generations in a universe with finite resources at our disposal, very much in line with the thinking that has emerged about sustainable development.(1;3;17-19) This somewhat self-evident reason to give proper emphasis to the issue of sustainability is amplified by considerations specific to health programs in developing countries. (16)

A good deal of effort has been placed on clarifying definitions and proposing models for the exploration of sustainability. (3;4;16;20;21) Different conceptual approaches (definitions) support the development of different models. Models are essential to representing a complex reality, exploring relationships and predicting outcomes. (22) They are essential to the development of meaningful research questions, or the definition of indicators of complex processes such as sustainability. (1;3;18;19)

The first definitions and models—largely based on Bossert’s landmark research—focused on institutionalization. (4;23) Subsequent definitions have evolved to become more relativist, open to a greater variety of mechanisms for maintaining health benefits to a population. (5;24-26)

Some of the models, for example those influenced by the health promotion movement(27), emphasize community processes of appropriation as the main driver of sustainability. (28-30) We found the model of Shediac-Rizkallah to be the most appropriate to start our investigation because of its openness and clarity. (16)

While all these models offer some insights into programmatic approaches to sustainability and may be applicable to Child Survival projects to a greater or lesser extent, they have, however, very seldom been used effectively for evaluation and research purposes.

In terms of evaluation research, our review of the literature has identified two groups of publications of interest: first, landmark studies of the sustainability of health programs in developing countries and, secondly, studies that provide insights into original methodological directions for evaluation and research on sustainability.

Bossert and Lafond identified national program-level and external aid factors of sustainability. (4;5) Different authors provide additional and overlapping insights into factors of sustainability in community-based interventions, through individual case studies or reviews of groups of projects. (12;31-42)

Researchers have not only tried to identify the determinants of sustainability, but also to demonstrate that sustainability can be observed at all. Of note is the recent post-intervention sustainability study in Bolivia and Bangladesh, where Seims found evidence of sustainability from PVO operated CS interventions: (13)

• continuation of project activities as long as ten years after the end of funding,
• capacity built in the (US) PVOs/NGOs with expansion of activities in new countries,
• creation and development of (local) NGOs,
• institutionalization of lessons within the countries’ Ministry of Health (MOH),
• continued activities of community organizations and volunteer health workers,
• sustained impact in communities.

The last level of research and analytical efforts addresses measurement and predictability. Most research on sustainability applies case studies, reviews, and exploratory qualitative approaches. Very few studies have been designed to make use of statistical tools of investigation. Such studies face obvious limitations in developing a satisfactory sample size, when projects are the units of analysis. They are also constrained by the limited number of variables that can be managed in complex analytical models, especially with small sample sizes, and the generation of valid quantitative variables to represent the elements of the analytical model.

Some of the efforts to improve evaluation standards and provide some means of predicting the likelihood of sustained outcomes have been directed at the development of sustainability indexes, including through statistical regression models. (43;44) Building indexes is a possible step for bringing sustainability studies from exploratory research to evaluation, and to tackle the question of predictability: how likely is it that a program assessed today will have sustained results tomorrow? Experiences of this type have been conducted in family planning programs (43), but not in Child Survival.

Overall, there is still only limited empirical research on the sustainability of primary health care and community-based health programs in developing countries, and the existing models have seldom been used to advance evaluation research or planning efforts. Two observations can be made about the advancement of a "sustainability agenda" in primary health care:

• Tools are slowly being developed, in and out of the PVO community, to address issues related to sustainability (for example institutionalization and capacity building (45;46), community ownership, participation and competence (47-49)).
• At the same time, it is apparent – both from our literature review and from debates within the PVO community – that organizing the questions in a manageable framework and allowing comparisons and heuristic learning has become a necessity if progress is to be made on the research questions being asked by CS professionals. (15;33) Specific yet shared models are needed, if only because "when indicators are chosen in a conceptual vacuum, it is very difficult to tell how important or how relevant they are to what people want to achieve."(50)

The CORE-CSTS+ Sustainability Initiative study was initiated in September 2000 and conducted in the context of a daily involvement in the experience of the PVOs/NGOs through CSTS+’ relationship with CORE, individual PVOs/NGOs, and the CSGP as a whole. The research aims were exploratory and the approach highly participatory, from the definition of the study aims and questions, to the development of the tools. Its central question was, Can a common framework be developed, allowing for the expression of diversity, yet allowing PVOs/NGOs to assess performance on sustainability, share lessons, and have a leading role in the sustainability agenda?

Results are presented for the informant interviews and survey on critical issues, on one end, and for the project sustainability self-assessment on the other, before synthesizing the salient lessons that can be applied to the development of a tool.

Results of the informant interviews and survey on critical issues

The analysis of our informant interviews and responses to the critical issues questionnaire provide useful lessons on the relevance of and questions about sustainability in our community of investigation, on PVO strategies and results which advance sustainability, and on the parameters of evaluation that should be taken into consideration.

Relevance of sustainability

The relevance of sustainability to the Child Survival agenda is unanimously recognized and it is strongly rooted in the finality of CS projects’ service to communities in need. For example, respondents to the critical issues questionnaire almost unanimously agreed with statements making sustainability part of what makes a project "truly effective."

It is not the importance of sustainability that leads to expressed skepticism, but our ability to address it programmatically through projects in a meaningful fashion and in all circumstances. Forty-three percent (43%) of the critical issues (CI) survey respondents think that, "a project approach to health programs in developing countries is not compatible with high expectations for the demonstration of sustainable results."

Reservations are expressed about what the focus of a project should be in different situations. The sense of the community’s needs, for example, comes unsurprisingly as a foundational value of our PVO respondents and informants, even for some of those endorsing sustainability as a sine qua non of project funding. A small majority of respondents think that PVOs/NGOs should not work in situations where sustainability is "clearly an unreachable goal." A larger majority considers that "direct implementation of interventions by PVOs/NGOs is necessary when all other stakeholders are either unable or unwilling to serve the needs of a specific disfavored population."

Additional concerns are expressed about the risk of a disconnect between what sustainability should mean for the communities and how it can be translated in bureaucratic requirements.

Strategies and results

Working through partnerships and building capacity are central strategies for US-based PVOs/NGOs trying to achieve sustainable health results. Three primary types of partners are essential to consider—Ministry of Health (MoH) structures, local Non Governmental Organizations (NGOs), and communities (including community structures).

Partnership with MoH structures emerges both from necessity and opportunity: the necessity to overcome the risk of gridlock in project implementation and the opportunity it creates for sustained results. For most respondents, partnership with health districts is strongly motivated by the pursuit of sustainability. But immediate pragmatic concerns for implementation gridlock is at least equally important issue for half of our respondents.

Our respondents also recognize the need to coordinate CS interventions to national policies and priorities. This does not translate, however, into an unconditional demand for an alignment of PVO interventions behind government structures, particularly when the commitment of the latter is questionable.

Partnering and developing the capacity of local NGOs is recognized as a key strategy for sustaining child health, and this goes far beyond transferring basic technical skills for the delivery of services:

• As a strategy, NGO capacity building fits within the larger development of civil society and democracy, which will create the conditions that will improve the sustainability of Child Survival.
• Ninety percent (90%) of respondents believe that "building vision and commitment" in local NGO partners is "equally or more crucial to the prospects of sustainability" than developing technical and managerial capacity.

But capacity building for partner organizations in and of itself does not summarize sustainability.

Capacity building plays its part along with other developmental changes: Developing accountability between service providers and communities, for example, is a condition for improving the sustainability of interventions through supply and demand mechanisms (e.g. improving the quality of services). With this, comes a strong consensus on the fact that building relationships between stakeholders justifies a specific allocation of resources.

Improving the viability of local organizations is another important element influencing CS work in the long term. This relates to financial viability, but also to other issues defining a dependency profile of these organizations: organizational linkages and relationships for support, advocacy, access to information and technical assistance, etc.

The three questions from the critical issues survey—below—obtain a high percentage of agreement and illustrate some of the complexities that come in partnership for capacity building efforts.

In terms of working with communities, our study participants—informants and respondents—generally convey agreement about four main ideas:

1. That whatever the level of enthusiasm for community participation may be, "buy-in" and ownership by the community is essential to sustaining efforts in Child Survival interventions.
2. That development in non-health sectors creates conditions favorable to sustainable health.
3. That communities are more likely to support health interventions strategies that are linked to their perceived development needs, health related or not.
4. That the development of community capacity, through community organizing and development, is essential to maintaining individual healthy behaviors and community engagement behind health issues.

There is a large consensus on the idea that using community participation to gather support for project activities without a true community development approach can be just as unsustainable as any other approach. This is translated in different ways, as illustrated by the general agreement to the two following questions.

There is first—of course—recognition that the general conditions addressed by community development (poverty, education, agriculture, environment, community organization, etc.) have their own impact on achieving and maintaining health benefits. Our informants refer to literacy, agricultural and food diversification, and water and sanitation programs as examples of complementary development interventions which have increased the prospect for sustainable health benefits.

There is also an underlying belief in the potential of communities, a potential which can be enhanced or revealed through appropriate approaches, and which can sustain the health benefits at the community level. The following statement from one of our informants would be widely supported by others: "with a little guidance, [if] you give people the opportunity, they’ll come up with plans better for them and their communities than we could come up with."

This leads of course to discussion of both participation and capacity building at the community level.

Community participation and ownership are mentioned numerous times in our interviews and are generally considered essential to the effectiveness and sustainability of interventions. The social and cultural changes represented by "community ownership" are essential to the sustainability of community-based health interventions. In projects involving Community Health Workers (CHWs), for example, community development work to establish community ownership, accountability, and incentives, is for many participants equally or more important than technical training and support of the CHWs. Sixty percent of our respondents consider that "long term community support and incentives need more efforts [to maintain the performance of CHWs over time] since technical supervision and training are insufficient."

In terms of community capacity building, respondents were asked to state—on a scale—which of two different strategic approaches was most likely to improve the sustainability of a health intervention at the community level. Overall, 84% of our respondents supported community organizing as an approach more important (58%) or as important (26%) to sustainability than IEC (information education and communication) efforts to promote the adoption of appropriate health behaviors. This strong consensus behind the value of general community development to support long term health progress is reflected in the following quote from one informant:

A set of questions and comments from our informants specifically addressed the evaluation of sustainability.

Measurement is still problematic because of a perceived intangibility of sustainability, and because of the multiple dimensions and pieces that play a part in making health achievements durable. A difficulty in measuring sustainability is also identified by the agreement of a majority of our respondents (80%) to the proposition that long-term results of projects "may only become obvious long after end-of-project evaluation has taken place." Finally, not only is sustainability difficult to measure, but—in the experience of our informants—it is sometimes quite unpredictable: a single uncontrollable event can undermine an entire strategy. On the other hand, unplanned-for local processes can dramatically and positively affect long-term prospects.

Key directions from our participants for assessing sustainability can be summarized:

• Sustainability is inseparable from progress toward high level health and development goals.

• Sustainable results represent progress between transitional stages, rather than the achievement of a fixed state.
• Any model of sustainability "has to work" for the local stakeholders. The best mechanisms for maintaining activities have no value unless they are meaningful in the local context and to the local stakeholders.

• Attribution of sustainable results to a single agent, intervention, or project, is unlikely to be realistic. Sustainability evaluation is strongly process-oriented and looking at projects’ contributions to larger processes. There is relative consensus (77% agreement) on a complex statement offering that sustainability evaluation should try to assess a project’s "contribution" to sustainability, and that attribution of results to "one single intervention or stakeholder" is a very difficult task.

A fundamental contribution of projects to processes that will sustain health gains is the development of enabling conditions and local opportunities. A series of nine questions in the CI survey asked directly which elements should be taken into consideration to declare a health intervention "sustainable." The stronger agreement is for general conditions created by the project. The elements that received the highest level of recognition represent conditions improved by the project, which create an enabling environment for greater and lasting health impact (improving the functionality of local systems, creating opportunities, fostering interdependency and relations in local systems), rather than for those elements referring to immediate resource dependencies. Another illustration is that a majority of respondents support that sustainability means leaving behind a more functional (local) organization, with a greater ability to conduct its overall mission, and not just maintain the performance directly linked to the initial project activities (Table 1).

Results of the self-assessment

The Child Survival Project Self-Assessment Survey (SA survey) is the third and final component of the Sustainability Initiative. It targeted CS projects for which an evaluation phase was planned. Responses came back from 22 projects out of 42 (52.4 % response rate). Its aim was also exploratory, to examine project managers’ expectations about the sustainability of their intervention, and the perceived achievements on pertinent intermediary results.

The organization of the responding CS project has been working in the region of intervention in health programs for 8.6 years on average. Most projects (20/22) are planned within at least a two-grant cycle timeframe in a rural environment. The survey explored factors affecting achievements (project and organizational factors, as well as country and community of intervention factors), key project strategies, expectations for the sustainability of activities and benefits, and achievements on intermediary results.

On a scale from minus five (very negative) to five (very positive), project and organizational factors are assessed with a median score of respectively 2.4 and 2.2, while country and community factors have a lower median score of 0.7 each.

Eighteen out of 22 projects recognize a strong commitment to the project by the MoH. Some factors in the country and community of interventions are judged critically for a number of projects:

• Fourteen projects recognize a high level of instability among essential staff in its structure.
• Sixteen projects agree that "MoH cadres face a lot of external pressures competing with rational resource allocation."
• Sixteen projects work in a context of "scarcity of qualified human resources."
• Projects have faced internal political instability (10 projects), non-peaceful international relations (4 projects), or civil unrest (7 projects) during their implementation of the intervention.
• A little under half of our respondents (10 projects) identifies as a constraint the "diversion of funds or resources in organizations involved in health care and social services in the country."
• The community of intervention did not have a "culture of participation and effective community organizing pre-existing the project" (9 projects) or a "culture favorable to addressing issues of gender in development" (13 projects).
• In terms of the economic situation perceived by project respondents, none can identify with conditions of growth and economic improvement for the poor, and 12 projects judge their situation as "poor with recent aggravation of economic conditions."
• Conditions were generally perceived as very low on human development and modest on the respect of human rights.

In terms of strategies, all participating projects consider a wide range of stakeholders with a direct responsibility or supportive role for sustaining their accomplishments:

• Projects identify from five to 15 "active partners" (eight on average) as currently working with them.
• 20 out of 22 projects identify more than three groups as having a role in maintaining activities and benefits after the grant period.
• All projects see their role at least equal or more important in capacity building than in direct implementation. Four out of 22 projects do not take any direct implementation role at all. They engage in multiple partnerships in order to advance sustainability, in which local NGOs and CBOs play a key role.
• Most projects (18/22) see their capacity building role equally divided between technical and management issues.

An additional series of questions referred to the sustainability expectations of the projects and their intermediary achievements.

Overall, seventeen projects expect both activities and benefits to be maintained (Table 2).

The maintenance of activities beyond the grant period relies heavily on governmental structures and, in equal proportion, on local NGOs or CBOs. An important supportive role is given to social networks and community structures (cooperatives, schools, etc.) for maintaining the health gains. The role of these different structures is, however, mutually supportive and complementary, often sharing the responsibility of the activities in the long run (Figure 1).

The self-assessment addressed the project’s achievements on intermediary results influencing the sustainability of their work. The questions were about the level of capacity achieved--in the project, in local organizations and communities—about the cultural acceptance of the intervention, and about the securing of long-term financial resources and equipment. Responses are generally skewed toward positive perceptions. Three of these elements, however,—partner capacity, securing resources, and maintaining equipment and structures—are assessed more critically by most respondents.

At the organizational level, there is a clear gap in assessed capacity between project and local partner (CBOs, NGOs, or health districts). Median capacity scores on a scale from minus five to five are respectively 2.7 and 0.3 for project and local partner. Among six organizational capacity areas, the most important gap between project and partner is in financial management.

At the community level, projects generally report improvements in terms of capacity and acceptance of the child survival behaviors. Most respondents, however, consider projects as sharing the responsibility for these community changes with other stakeholders.

Eighteen out of 22 projects report some measure of achievement toward financial sustainability through one of the following mechanisms:

• developing the experience of local partners in grant application and management,
• diversification of the sources of funding through a local NGO or CBO,
• developing the experience of local partners in fund-raising,
• increasing cost-recovery from service users,
• cost sharing of key activities by different stakeholders,
• improving the collaboration with the private (for profit) sector,
• improving government funding for key activities.

But only eight of these 18 projects feel strongly confident about these achievements. Overall, the median score for securing financial resources is low (0.4).

The essential findings of the self-assessment are the generally positive expectations for sustainability of a majority of respondents, in spite of difficult contextual conditions. Respondents generally have positive perceptions of their immediate project and PVO organizational environment, and report active and multiple capacity building efforts, at times disengaging their project totally from the implementation of activities. Achievements on building local partner capacity and in securing long-term resources are, however, generally assessed modestly and put respondents’ expectations in question.

Additionally, although measures were limited to very exploratory self-assessment questions, these questions were relevant to all responding projects and revealed common critical areas of performance affecting maintenance of results.

Conclusion: applicable lessons from the study

This study provides many lessons and insights, from the opinions expressed in the interviews, areas of consensus and debate in the critical issues survey, and in the responses to the self-assessment. Diversity, limited predictability and multidimensionality characterize the range of PVO experiences with sustainability. Shared values and common elements of definition emerge, however, to give meaning to the concept in the context of Child Survival.

Some of the key lessons of the three study components can be summarized:

• Although it sounds like a tautology, the finality of child survival – improving the health of children, particularly children living in poverty – is a cornerstone of any health intervention claiming to be sustainable.
• There is not one linear model, but a number of approaches to achieve sustainable results. A final "sustained impact" is the result of complex and multi-dimensional interplay.
• There are strong external factors outside of the reach of projects and PVOs/NGOs that influence sustainability. Measuring progress on these external conditions is a crucial part of assessing the prospect of sustainability in CS interventions.
• "Sustainable results" can often not be reliably predicted. They seem to be due to successful local "negotiations," supported by favorable conditions which a project can support, but not necessarily control.
• Elements of definition for sustainable programs that gather a strong consensus from PVOs/NGOs fit within the general heading of "creating an enabling environment": "building functionality," "creating opportunities," or "developing relations and inter-dependency."
• Capacity building in local partners is essential, but sustainability depends on many other factors. Increasing the viability of local organizations is another important element, whether it relates to financial viability, or other elements of an organization’s "profile of dependency": organizational linkages and support relationships, advocacy coalitions, access to information and technical assistance, accountability, etc.
• Improvements in social cohesion (e.g. accountability) or community competence and capacity need to be better understood and better evaluated, but are cornerstones of sustainability.
• The processes through which health information is diffused or services are provided are extremely important to sustain health gains. Quality, equity, efficiency, or technological appropriateness all contribute to (or constrain) the durability of these benefits.
• Helping a local system progress toward sustainable health becomes the pertinent role for projects.
• Sustainability planning, at the Child Survival project level, must find its place within the larger issue of Sustainable Development.
• Although projects are only contributors to progress toward the next transitional stage, this contribution is essential in favoring or hindering lasting impact.

Considering the wide distribution of responsibilities for the maintenance of health outcomes, these lessons suggest a system perspective assessing and defining sustainability in Child Survival projects.

For projects, sustainability can be defined as a contribution to the advancement of certain conditions which enable actors of a local system to negotiate roles and responsibilities in order to achieve lasting health gains.

The individuals, communities and local organizations constitute a local system with their environment, and it is ultimately their coordinated social interactions and efforts, based on the understanding of their own health and development, which will lead to lasting health impact.

The logic of this definition takes into account the loss of control over local processes inherent to project approaches, which places the immediate determinant of sustainability—a local process of negotiation, role definition and engagement—outside of the full control of a PVO. The responsibility of a PVO is not lessened by this recognized loss of control. CS projects are in a critical position to advance key conditions in the local system in which they intervene, if not directly then by helping the local communities and stakeholders address these conditions.

Planning and evaluating for sustainability in CS project requires a model, based on a system perspective and taking into account different dimensions in an integrated and systematic approach. It should, however, make ample room for representing and measuring the diversity of issues PVOs/NGOs must address on a case-by-case, contextual basis.

This is the purpose of the Child Survival Sustainability Assessment (CSSA) methodology presented in the companion volume, "The Child Survival Sustainability Assessment (CSSA). For a shared sustainability evaluation methodology in Child Survival interventions."

Back to main sustainability page