The Fundamental Question
~ Caroline Marshall
Back in 2002 James Allen Smith addressed the Museum Trustee Association and Getty Leadership Institute and asked "What do economics have to do with culture?" He noted the "pressure we are under to justify our work in instrumental or utilitarian terms," and how when "cultural critics talk about economics, or economists talk about culture, smart people can end up saying ridiculous, confusing things." In the talk that followed he offered a comprehensive analysis of the trends in cultural sector economics that are increasing the pressure to evaluate, and described the issues behind the fundamental question: "In what language should we answer when policy makers and foundation funders speak their utilitarian prose and expect quantitative answers?" It is a talk well worth reading in its entirety.

Money Well Spent
~ Thomas Wolf
Of all the foundation presidents active today, Paul Brest of the William and Flora Hewlett Foundation is one of the most respected. He has written extensively on evaluation and accountability, most recently in his book, Money Well Spent, co-authored with Hal Harvey. In a recent conversation, I asked him how his foundation balances the need to come up with significant dollars to do effective evaluation with the desire to put as much money as possible into supporting programming and mission-related activities. Brest replied that this is the wrong way to frame the question. Foundation dollars are intended to drive towards impact. Evaluation is critical in helping to achieve that goal. It is not a separate add-on expenditure, but fundamental to the grant because it is crucial to know if the goals of the effort are being achieved. There is no point in funders making grants, he suggests, if they do not know in some kind of objective way if they are being successful. Indeed, Money Well Spent was written partly to change the paradigm in funder thinking that evaluation takes money away from program funding. Both are needed if outcomes are to be achieved.

Taking on What Seems a Burdensome Task
~ Jane Culbert
As part of our research, we have been speaking with executive directors and development directors of state arts agencies and cultural organizations. For most of these people, evaluation and assessment is taking a back seat to the more urgent work of survival. While all agree that ticket sales and attendance metrics do not convey the real outcomes of arts programs, lack of staff time and other resources keep even those who are supportive of evaluation from moving beyond these rudimentary statistics. Where more extensive evaluation is happening, it is primarily because a funder has required it, and, in many cases, is funding it. But there are exceptions. For example, one contemporary arts organization is building evaluation into some of its programs, such that programming will change from one event to the next based on the results of evaluation efforts - sometimes to the discomfort of staff. In general, there is interest in moving beyond "counting heads," but in a time of scarce resources, doing so remains a vague dream.

Making Evaluation Metabolic
~ Laura Mandeles
Our research also looks beyond the cultural sector to other kinds of nonprofits. Recently, I spoke with an education research and evaluation organization and a venture philanthropy investment group. A common theme stood out: the process of gathering information for evaluation needs to be embedded in the functioning of an organization or project. Otherwise evaluation becomes a burdensome add-on. As Michael Gilbert argues in Integrated Program Evaluation: A Three Part Vision for Better Leadership, Planning, and Effectiveness, "good evaluation is integrated evaluation." The implication of course, is that the organization has to have the capacity in its leadership and staff to conceptualize strategic outcomes, and the systems to gather, analyze, and summarize data. The venture philanthropy group I spoke with takes a serious and far-sighted approach, funding organizations to create a theory of change, define outcomes, measure outcomes, and train staff to manage performance measurement.