Documents - Workshop

Twenty-five representatives of private voluntary organizations (PVOs/NGOs), cooperating agencies (CAs), the Child Survival Collaborations and Resources Group (CORE), and the United States Agency for International Development (USAID) met together at Macro International in Calverton, Maryland to discuss the recently published CORE/CSTS+ sustainability study, its implications, and next steps.

A first Sustainability Dialogue meeting was organized by CSTS+ and CORE in March 2000, and effectively set the stage for the Sustainability Initiative which was to follow. A report of the March 2000 meeting (in MicroSoft Word) is available by clicking here. You can also download a copy of the recommendations (also in Word) from that meeting.

During the morning session, various speakers discussed what had been learned through the sustainability initiative study and current questions being tackled by the PVO community and its institutional partners. Speakers included Leo Ryan of the Child Survival Technical Support Project (CSTS+), Sheila Lutjens of USAID's Child Survival and Health Grants Program (CSHGP), Karen LeBan of CORE, Eric Sarriot of CSTS+, and Eric Swedberg of Save the Children.

CSTS+ presented a tool combining online support to sustainability evaluation design in child survival, and access to a database of indicators and tools. During the afternoon sessions, participants split into small groups to discuss the proposed tool, and made recommendations to increase its usability and value to the PVO community.

This report presents a a brief summary of the morning presentations and the day’s discussions, including key recommendations made by participants. It also presents the history of the CORE-CSTS+ Sustainability Initiative.

Read the day's agenda

Read the participant list

If it has been a while since you've read the Sustainability Initiative papers and need a review, click here for "A Quick Reminder about the Child Survival Sustainability Assessment Framework"

Introductory presentations

Although each of the morning speakers addressed the topic of sustainability from a different angle, several key themes overlapped in the presentations:

• Sustainability still matters to USAID; so does documentation.

Sheila Lutjens reminded participants that although there is not a separate section called

Sheila Lutjens--Chief, USAID Child Survival and Health Grants Program

“sustainability,” in the latest CSHGP request for applications, a discussion of sustainability is scattered throughout the document and is an integral part of program considerations. Lutjens also emphasized that documentation is not just writing a report. “What is the analysis that went behind the intervention?” she asked. "Why did it work? Why did it not work?" “That will be critical this next year,” she summarized. “We want to be able to talk about the PVOs/NGOs and what they’ve accomplished, but we need real hard data when we present our case.”

• Sustainability and documentation also matter to CORE

“We still don’t work together as well as we can to demonstrate the overall impact, rather than just the [impact of the] organization,” Karen LeBan explained. “We have to demonstrate that what we’re doing really works--that our approaches can be brought to and maintained at scale, that they reach the most disadvantaged, and they have more lasting benefits than the alternatives--that’s the documentation we have to be able to show.”

• Sustainability is still confusing and not valued enough as a specific contribution of PVOs/NGOs

The Sustainability Initiative has shown that sustainability does indeed mean something to the PVOs/NGOs, and that it can be presented in a coherent manner that respects their pro-community, pro-poor, pro-equity approaches. But we are still faced with confusing concepts of sustainability. Expectations on, and judgments of, approaches are often based on individual assumptions and biases. PVO approaches cannot be valued until they can communicate clearly the benefits of their work for the long-term. And this brings us back to the question of evaluation.

• There are quite a few lost opportunities for PVOs/NGOs to learn

PVOs/NGOs have documented "promising practices," but when it comes time to formulate conclusions and generalizations about promising practices,” LeBan explained, “we fall to pieces. We have very little reflective documents within and across PVOs/NGOs. We have to concentrate on this.”

One area in which she sees the possibility for improvement is in the learning cycle itself. “We’re caught in a closed loop learning cycle, as opposed to a spiraling learning cycle,” she explained. “Learning is the key to improving sustainability. We need to break the thinking that we’re all so unique and find the underlying patterns which are key to learning. What are the patterns that cut across the different PVOs/NGOs?” she asked.

• These lost opportunities for learning translate into lost opportunities for demonstrating the value of PVO approaches

Karen LeBan--Executive Director, CORE, Inc.

“We need to learn better how to distill lessons learned,” she continued. “We need data, and we really need to be able to share these practices more with each other. This is especially true with 'underdeveloped practices.' "How do we make something that appears to be sustainable even more effective?” Each one of these practices require data, analysis and reflection across the PVO community.

With the increase in investment in the NGO sector, there is an urgency in PVOs/NGOs deciding on what their role will be. "However,” she stated, “we can only accomplish this if we have an aggregated view to be able to guide us and advocate for what’s needed. This is what I hope we can start moving toward with the Child Survival Sustainability assessment tool.”

• We have difficulty in measuring all dimensions relevant to sustainable child health, including those being tackled by current child survival projects

“We want to know which determinants have changed, so we can have some idea of which activities might have succeeded,” stated Eric Swedberg. "We want to be accountable for our choice of strategies. . . .We work at various levels in order to be more sustainable. If you work just at the individual level you might have limited health improvement. If you work only at the social systems level, you have increased potential for health improvement, but when you combine the two is where you have self-sustained health improvement. By combining them, you have the ultimate impact on health.”

Read presentation summaries and download the PowerPoints

Morning question-and-answer session

Various points were brought out in the question-and-answer session following the presentations.

Six main points were made and are summarized below.

1. If we want to become more effective in advocacy, we need to have the right information (data) and learn to value different types of data.
2. There can be an invisible tug-of-war between competing values such as coverage and quality, depth and breadth and equity.
3. There is a lot in the external environment that is frequently out of our control, but that we can influence.
4. Every evaluation comes at a cost. How much are we willing to do?
5. We need more funding and energy for research in addition to evaluation.
6. We need more data, yes, but we also need to make sure that it’s quality data.

Read more from the morning discussion.

Overview of the proposed Online Assistant and Database

The proposed Online Assistant and Database (CSSA-OAD) consists of three interactive parts:

1. An online assistant tool to assist users in developing a systematic evaluation plan, focusing projects on progress toward sustainability
2. An indicators (and tools) database, providing users of the assistant with links to tools, and existing indicators they may use (considering the many “soft” issues we need to address, the indicators’ database needs to be an expanding database, capturing the innovations of PVOs/NGOs and projects as they define their own evaluation plans
3. An expanded project database, capturing the evaluation plans of individual projects (and feeding into the indicators’ database) and collecting data from different evaluation phases (this is what takes place already with the standardized Rapid CATCH indicators, but would include a wider range of measures used by projects).

Users--PVO backstop and managers, possibly local partners--would access the tool through a computer interface.

The purpose of the Child Health CSSA-OAD is threefold:

1. Assist individual CSPs in developing an integrated evaluation plan toward sustainable child health through:
   • Access to an indicator/tools database (as a first illustration, a first presentation of assessment tools in the main dimensions of the Child Survival Sustainability Assessment (CSSA) is available on http://www.childsurvival.com/documents/CSTS/sustappendix.cfm);
   • Guidance through an online assistant or wizard.
2. Improve cross-learning by:
   • Sharing projects’ evaluation approaches and indicators through expanding the project database, through individual projects’ contributions of measures, tools and indicators that they use to the indicators’ database.
   • Benchmarking evaluation tools and indicators against each other over time.
   • Improving project evaluation research designs based on proactively collected information. (Over time, as indicators and measures are improved, the validity of the data collected would allow better and better study designs, including post-intervention studies.)
3. Build information base and improve accountability at the Grants Program level through time trends on project achievements in sustainability processes and key outcomes.

The tool would not require additional assessments from what PVOs/NGOs are already doing, but suggest resources, and capture project level information.

It would not prescribe anything more than the limited number of dimensions (three) and components (six) included in the CSSA. Local processes and local decisions would determine the specifics of what elements these components should include and how they would be assessed. In this way, the OAD is essentially a collective learning and capacity development tool.

Given that the tool builds on the shell of the CSSA, a quick reminder of the CSSA internal logic is provided here.

For the child survival and health community (PVOs/NGOs, CORE, USAID, and CA partners), the long-terms benefits of such a tool can be:

• Improved integration of sustainability in CSP monitoring and evaluation plans
• Increased demonstration of achievements and accountability
• Learning by benchmarking tools, indicators and measures
• Expanded knowledge base for more demonstrative research designs based on collective PVO experience
• A powerful advocacy information base.

Such an effort will however require efforts from the different partners:

• CORE Working Groups:
  • Buy-in,
  • Clear articulation of overlaps between sustainability and current operational frameworks (C/HH-IMCI; BEHAVE),
  • Defining a research & advocacy agenda, participating in validation studies, setting guidelines;
• PVOs/NGOs
  • Contributing experience and project information through a new interface,
  • “Early adopters" can be considered for early testing;
• CSHGP
  • Continued dialogue with PVOs/NGOs to define the boundaries of what can be expected from individual projects in terms of evaluation and what other resources are needed
  • Commitment to sustainable child health agenda with CORE and PVOs/NGOs
  • Communication of efforts and results to high level stakeholders
• CSTS+
  • Tool development, PVO support, collaboration with CORE and its WGs

Participant reaction to proposed tool

During the afternoon session, participants worked in small groups and discussed their thoughts on the proposed tool and their recommendations for improving it. Each group listed their likes and dislikes about the tool as it was proposed.

Likes
Overall, participants liked the user-friendliness or menu driven approach, and the categorization of indicators (and tools). The guidance offered by the OAD in selecting indicators was valued particularly on issues of capacity building at the organizational and community levels. The flexibility in the use of the tool to design specific plans seemed appreciated, along with the fact that it remains “open” to expansion and evolution. Its accessibility, particularly for PVO headquarters staff with easy access to the Internet, was considered a plus. The comprehensiveness of the tool and the way it helps address “abstract” evaluation questions was presented as something of value.

Other comments emphasized the value the OAD gives to major components of a complex issues, and its potential to be an “empowering” tool, even at the community level. Finally, participants recognized it as a forum to share ideas and a mechanism to “get successes out.”

Dislikes
There was some confusion attributed to this first presentation. Participants also felt that, as any new tool, it would require some time investment and commitment to be appropriated by the NGOs. The dependency on technology was also noted as having a flip side with regards to field applicability in some cases.

Other dislikes referred to the name of the tool, and to questions about the underlying model (sustainability framework).

Read the “likes and dislikes” list generated by the participants.

Participant comments and questions on proposed tool

Participant questions and discussions addressed both the proposed online assistant and database and its underlying model, the Child Survival Sustainability Assessment framework (CSSA).

Read a summary of the afternoon’s comments and discussions on the proposed tool

Recommendations from the meeting and the proposed tool

Participants at the meeting made a number of recommendations related to:

• The sustainability assessment framework,
• Testing the tool,
• Development and Implementation of the tool.

Action has already been taken on some of these recommendations. Where this is the case, this has been noted in the comment box below the recommendation. Useful information related to specific points are also noted in the comment box.

Recommendations on the framework itself

The sustainability assessment framework (CSSA) itself and linkages within it should be made clearer:

• Provide a 10-15-page summary showing what the purpose and the thoughts behind the tool are.
• Conduct more presentations of the framework: CORE Meetings, etc.

• Increase the language in the tool to include MOH/other governmental organizations.
• Review framework for linkages to key concepts of prevention, and the multi-sectorial platform of the C/HH IMCI framework, Q (quality), A (availability), A (accessibility,) and D (demand).
• Try to emphasize the issue of working together. Include coalition-building among PVOs/NGOs as another element in dimension three.

Recommendations on testing the tool

The CSSA needs to be tested with PVO projects before developing the online assistant (OAD)

• Adapt the tool for evaluation of previous projects in order to choose/improve future viability/applicability of the CSSA. (Recommendation to CSTS+/PVOs/NGOs)
• Field test to look at actual use.
• Pilot test the CSSA-OAD extensively in the field with PVOs/NGOs.

The resource implications of the tool need to be assessed.

• Do a pilot study on usability including level of effort (time, labor, and funds) before full-blown implementation. (Prototype)

In order for this initiative to be most helpful and useful there needs to be a well-defined research agenda.

• Define research questions and use the proposed tool to answer these questions. For example, Which are more effective, horizontal or vertical programs?
• The tool could demonstrate the importance of capacity building by PVOs/NGOs. Adapt the tool to provide numbers for this argument.

Plan for case studies and post-intervention studies.

• Plan for post-project sustained behavior study of promising cases. Case studies might be useful, especially for local learning about what processes have worked or not.
• Include an examination of [dissemination in] other sectors outside of health for example, dissemination of health education messages over time and different channels, or areas in which CHWs stopped working, but sanitation facilities are maintained, etc.
• PVOs/NGOs working in a country could collaborate on these dimensions and look at PVO-wide impact over time.

USAID should consider post-intervention assessment funding support.

Recommendations on the development and implementation of the proposed OAD tool:

Partners need to be involved at the field level.

• Partners should be involved in tool refinement and development of specific content.
• PVO/NGO contributions must be communicated to the national MOH. CSSA results should be used in this way, not just for USAID, etc.
• Results have to go to the national level. This is very important if you want to run a program. If a consortium goes to the national level they are much more likely to be heard.

Collaboration with other agencies at the US level for the development of the OAD should not be ignored:

• Sustainability needs to be tied to USAID’s priorities and opinion of sustainability, e.g., now AID is trying to stop disability.
• USAID and organizations working in child survival (CORE, CSTS+, and PVOs/NGOs) should meet and discuss these issues with organizations working with civil society/democracy and governance to share their experience with indicators and measurement.
• We need to define the role of cooperative agencies in sustainability.

Consider the trade-offs and competing interests in deciding the level of flexibility in the database.

• Make it clear(er) that we can customize our dashboard.
• Keep flexibility and customization in the tool to capture the innovations.
• Keep the tool simple enough that there are not too many indicators.
• Define a few key strategic sub-element areas for learning and reporting e.g., CHWs under community competency/capacity.

At the end of the day there needs to be a field-friendly version of the tool.

• Make the CSSA-OAD available off-line on CD-ROM for field project managers and in multiple languages.
• Make it a menu-driven program (so as not to forget that the goal is to improve health).
• Develop tool that is easily transferable from HQ to field and so on, given high turnover rates within PVOs/NGOs and within CSPs.

Communication of the benefits and continued participation are vital.

• The short-term benefits of the tool should be communicated better.
• If PVOs/NGOs do not use it, it will not be effective. Therefore, we should think of ways to encourage people to use it.
• Create a working group for CSSA-OAD to ensure PVO input and commitment.

Confidentiality is key.

• Who would have access to project-specific data? We need a clearly written statement of the limits of confidentiality, and should study the balance between open use and access and confidentiality. Maybe information could be entered without directly linking it to a PVO.

Continue technical assistance through face-to-face interactions.

• Budget/plan for regional training once the pilot test is complete.

Conclusion

At the end of the day, participants agreed to remain engaged as an “interest” or “affinity” group within CORE. There was reluctance to consider a new Working Group, considering how current WG members are already stretched. Some PVOs/NGOs voiced interest in implementing some of the recommendations, such as “retro-fitting” the model, or conducting post-intervention studies with serious methodological standards.

The discussions on the cost, applicability, format, and purpose of the online assistant tool will be ongoing between CSTS+, PVOs/NGOs, CORE, USAID and other potential partners.

Next steps for advancing thinking about the topic have been identified and specific sessions will be organized during the APPLE CORE Spring Meeting, and the Child Survival and Health Mini-University.