Tag Archives: evaluation

A Challenging Question

by Julie Plaut

How can we document meaningful short-term outcomes, when much of what we’re aiming for takes a long time to achieve?

We’ve probably all heard the grudging joke about needing to wait to read our students’ obituaries – hopefully decades from now – to know if we really graduated actively engaged, informed and responsible citizens. Major change in individuals and in communities often takes years, yet it’s critically important to know in the shorter term to what extent we’re achieving our goals, or at least contributing to long-term movement toward them.

A common framework categorizes knowledge and skills as outcomes that can be measured in the short term; actions or behaviors as intermediate outcomes; and values, conditions, and status as long-term outcomes. Thus a campus-community partnership focused on increasing college access might track: middle school students’ understanding of key steps to attend college, and college administrators’ knowledge of effective strategies for increasing access; the same students’ enrollment in a rigorous set of high school classes, and changes in college admissions and financial aid policies or practices; smaller gaps in high school graduation and college enrollment rates by race, income, and parents’ level of education, and a shared commitment among educators to support all students’ educational success.

For institutions and individuals committed to developing engaged citizens, determining exactly what we seek to accomplish and measure may be the fundamental challenge. We can certainly draw on indicators from the national Civic Health Index, the VALUE rubrics, and other resources, including those noted in MNCC’s 2009 Civic Engagement Forums report. Yet there is neither extensive research on the results of different experiences and contexts, nor often a strong link between existing knowledge and practice. In analyzing alumni survey data, for instance, do we take into account political scientist Laura Stoker’s work on life-cycle patterns, so we don’t judge our success by looking at a typically low point in adults’ civic engagement? Some useful reflections appear in How Young People Develop Long-Lasting Habits of Civic Engagement, the result of a conversation to inform the Spencer Foundation’s Civic Learning and Civic Action initiative, which is a source of research grants that will surely inform future practice.

Thoughtful consideration of the desired outcomes for communities, students, and institutions, drawing on multiple disciplines and types of knowledge, can be a civic act in itself—developing participants’ capacity for dialogue, strategic judgment, commitment to engage over time, and sense of accountability for results as well as intentions or actions.

Reprinted from “Outcomes,”our assessment brief which is published twice a year and available at www.mncampuscompact.org/assessment

Assessing the Outcomes of Civic Engagement: Why Bother?

by the MNCC Assessment Leadership Team

Assessment is like flossing. We all know it’s a good thing to do, but we don’t necessarily act on that knowledge unless there’s an external push—a looming dentist visit or a funder’s requirement. Civic engagement practitioners, like most people in higher education, assess the results of their work primarily in response to others’ demands. Our daily lives are full. Yet assessment helps us track progress towards our goals and understand what factors contribute to success. It allows us to tell powerful stories and identify where we might best invest our time and money. It keeps us accountable to our own values as well as our partners. It thus helps us do our jobs better, advancing our institutions’ civic missions and broader movements for positive change.

So how can we fully commit to actions that are healthy for us in the long run? One step is simply recognizing assessment’s benefits as a matter of compelling self-interest. Another critical step is focusing on our strengths—taking the asset-focused approach we so often advocate in community partnerships. We have access not only to all sorts of useful resources and sample instruments, but also to institutional researchers and others with relevant responsibilities, skills, and interests. IR people, in particular, are expected to document and disseminate progress toward the institution’s mission, strategic priorities, and accreditation standards. They are increasingly being asked to explain the institution’s relationship with and impact on the community, as well as the meaning of a degree to alumni years after graduation. Civic engagement practitioners represent a source of valuable connections and knowledge for them, just as they offer quantitative research capacity and a fresh perspective on key questions and outcomes.

Assessment of civic engagement is ideally collaborative, involving multiple stakeholders within an institution, its partner organizations, and sometimes other campuses. It’s cross-cultural work too, as we build relationships, mutual trust and respect, and a sense of common purpose across differences. Along the way, we’ll practice and develop our reflection skills. Only with reflection will collecting data and stories lead to wisdom and greater insight into what works and why. Really looking closely at outcomes for communities and for students is an act of courage. It means being facing the risk of negative or neutral findings—and being willing to change and grow. Courageous leadership is something we seek to cultivate in our students, and we’ll teach it best when we model it too.

The Minnesota Campus Compact Assessment Leadership Team is a small group of civic engagement practitioners, institutional researchers, and faculty assessment coordinators committed to supporting enhanced assessment of civic engagement’s outcomes, grounded in this constructive vision and spirit. Our goal is to produce, in the coming months and years, brief pieces that highlight particular assessment tools and what was learned through their development and application. We’re also researching other resources and considering what opportunities we might offer for collaborative planning and assessment.

Reprinted from “Outcomes,”our assessment brief which is published twice a year and available at www.mncampuscompact.org/assessment

Developing Indicators of Success

By John Hamerlinck

Here is just a quick post while this is still on my mind. Recent conversations at our forums on assessment and evaluation and an inquiry from a colleague regarding indicators reminded me to share a few useful tips that I’ve collected over the years. None of these are my original ideas, but I can’t recall the individual sources – except for the first one, which I got from Geralyn Sheehan.

  • Never a story without a number – never a number without a story
  • Indicators arise from values. We measure what we value. We value what we measure.
  • Indicators must be: 1) meaningful, 2) robust and technically sound, 3) sensitive and 4) capable of being reported regularly.
  • Measure what you want to be.
  • Measure the cause not just the effect.
  • Don’t fall into the trap of measuring what is easily measurable rather than what is important.
  • Effective indicators give you information while there is still time to act.

Getting Beyond “Either/Or”

By John Hamerlinck

Ten years ago, I was working for a government agency that was actively involved in remediation of the Y2K computer problem.  Now days most people look back at that time and remember some rather scary predictions and a relatively uneventful New Year’s Day, 2000.

This result was not simply a case of unfounded crisis hype. The cause for the reasonably humdrum January 1, 2000, was countless hours of work by folks in every sector of society, acting to ensure that problematic systems were updated. The smooth Y2K rollover, however, was not the only thing that they achieved.

We live in a culture where we are constantly presented an overly-simplistic, “either/or” view of the world. People are seen as only liberal or conservative; ideas are only deemed to be either good or bad (as if the “both/and” versions of these options didn’t exist). Because of this, we tend to look at events like Y2K simply as something that either did or did not happen.  A little additional analysis, however, reveals that numerous benefits resulted from all that hard work.

Cities, hospitals, businesses and schools everywhere suddenly had disaster preparedness plans that could be (and have been) implemented during all sorts of natural and human-caused catastrophes.  Widespread computer hardware updates resulted in increased productivity as organizations replaced slow, inefficient machines. Perhaps most significantly, people whose lives and enterprises moved along seemingly oblivious to the larger world were suddenly required to gain a deeper understanding of the world’s interconnectedness and interdependence.

As we engage in the work of community-building through civic engagement it is important to avoid the trap of dualism. Overcoming the common challenges we face will be more difficult if we see concepts like leadership and power in limited, “either/or” terms. We can all find lots of opportunities to demonstrate individual and collaborative leadership. As for power, it is an unlimited and renewable resource.

When we embrace a broader, multi-variant view of engagement, assessing and evaluating our work becomes more effective. We begin to recognize more unintended consequences, more unforeseen benefits and more opportunities to find underdeveloped capacities in the gray areas between the black and white surfaces.