Last week I looked at the poor state of program evaluation among non-profit organizations. Pressured to do more for clients with less money, some organizations can't resist taking money earmarked for evaluation and spending it on services.
Unfortunately, this is a short-sighted approach. Without adequate data to show program effectiveness -- or to provide important lessons for improving programs -- non-profits will find it increasingly difficult to garner critical external support.
There is plenty of blame to pass around for the lack of quality evaluation. Much of it falls on the corporations and foundations, which generally have not insisted on solid measures of performance. In other cases, non-profits themselves are at fault. Evaluation is viewed as a distraction.
But by not attending to evaluation, many fine non-profits are neglecting to build a strong case for their good works. In today's highly competitive environment, external funding suddenly becomes harder to obtain.
Look at the situation from the funding side of the equation. Given two agencies applying for a limited pot of money, which should be funded?
When I am called in by companies to help make such decisions, one of the first areas I teach them to examine is prior program evaluations. Is the non-profit committed to assessing its strengths and weaknesses? What does it do with the results of prior evaluations? And, most importantly, does it cover up its previous failures or does it announce up front what they were, what the non-profit learned from them, and how it intends to improve based on them?
That brings up a critical point in evaluating programs. In my work with non-profits, I have seen time and again the attempt to hide negative data with piles of the good stuff. Of course, an organization must put its best foot forward to obtain money. Data showing a significant improvement in grade point average for at-risk youth, for example, should be touted. However, organizations should not shy away from presenting negative data, especially if it offers valuable lessons. The challenge is to present the data properly.
I remember reviewing the evaluation of an inner-city tutoring program that showed exceptional gains in school achievement and behavior for all youth involved, except for those drawn from one school area. Upon closer examination, we found that the program, a model of cooperation between a school system and the non-profit, had located one of the after-school sites in the school itself, rather than at a youth agency location.
Burying the data within the total project would have still proven the program to be successful. And it would have triggered a grant renewal from a local foundation.
But the data were tantalizing. Why had the children in this program performed at a lower level than the other youth?
The agency found that the kids responded better to a change in environment after school. The principal, a dedicated man, hung out in the program and intimidated the youths. A social factor also was at play. Youths leaving the school taunted the children in the program for having to "stay after school."
Armed with this "negative" information, the agency recommended changes, which resulted in increased funding for the second and third years of the grant.
Finally, there is the problem of "soft" evaluation. Unlike scientific laboratory data, data on people are always variable, often unpredictable, and usually messy. This has often put social agencies on the defensive. How do you document changes in attitudes toward art? How do you verify changes in sexual behavior as a result of AIDS educational programs?
Such questions are not easy to answer. But they have been addressed successfully by professional evaluators nationwide.
Donors must recognize the need for good program assessments -- and be willing to provide the extra funding necessary to see that it is done. If not, we will never know which programs most effectively address our pressing social problems.
Les Picker, a consultant in the field of philanthropy, works with charitable organizations and for-profit companies.