Story

Using Story in Evaluation

Evaluating programs through the use of stories is a powerful way to personalize the experiences of those you serve for your funders and other stakeholders. And although it’s often more work, the experience of gathering stories and reflecting on them allows you to truly understand a service up close and personal from your client’s perspective.

My colleague Kim and I were recently involved in an evaluation project that involved gathering ‘Most Significant Change’ (MSC) stories from individuals and service providers that had been impacted by a series of health care initiatives. The stories were incredibly revealing. Some were very raw. They pointed to the harsh realities of dealing with persistent substance misuse and mental health issues, often rooted in past traumas. They also highlighted the daunting and overwhelming challenges of trying to navigate systems that aren’t designed to be easy or accessible to anyone other those with a lot of social or financial capital.

The experience of gathering and then writing and presenting the stories brought me back to my dissertation work. I utilized a qualitative research methodology called Grounded Theory to try to understand how leadership in social service programs impacts the outcomes that individuals achieve through those programs. As I wrote about in another blog post, it ended up being about trust and autonomy. What I realized in reflecting on those experiences is that I have often gravitated towards numbers rather than stories in my work evaluating programs. For the sake of convenience and to meet funder requirements, it’s been easy to just gather up survey data or data on service utilization patterns and then tell a narrow story of what happened. Utilizing qualitative methodologies to gather stories takes more time, but I believe has amazing payoffs in terms of gaining a more in-depth understand of patterns.

In the end, I think the best evaluations actually use both quantitative data and stories. Some practical tips that will make it easier for you add in stories to your evaluation process include;

Use an outsider to gather the stories. Program staff or management will bring their own lens to the process that may not be helpful, and those being interviewed may feel some social pressure to answer in certain ways. If having an outside evaluator isn’t in your budget, you can use staff from other programs or students from a local social service program.
Consider what questions might help round out or more fully tell a story about the different aspects of a program or service. For example, you can ask questions about what people felt was the most powerful or beneficial aspect of a program and use that to bring life to quantitative outcomes data from surveys.
Keep the sample small but diverse. You likely won’t need a massive number of stories to gain a more in-depth understanding as long as you sample for diversity in terms of people’s characteristics (e.g., age, gender identity, cultural background) or their experiences with services (e.g., those that completed the program vs. those that didn’t).
Choose your method for gathering the data carefully. Interviews typically provide richer data, but can be more work than using a focus group methodology. If you wish to use focus groups, I would consider adding a few interviews as well. And while phone interviews will suffice, face-to-face is best if you can manage it.
Ask permission to record the interview or focus group. By recording, you free up your mental energy to really focus on what people are saying and ask good follow-up questions. Promise people that you are only using it to make sure you get the story right and that you will both maintain it securely and destroy it once you’re done. The recording will allow you to capture verbatim quotes, which can be unbelievably powerful in storytelling.
Open up and lean in to the experience! I’ve found that listening and really engaging with other people’s stories can be profound and enriching. It’s a real honour to bear witness to people’s stories. They often feel incredibly grateful for being listened to – particularly people who have been made voiceless through marginalizing experiences. And those stories can help propel your program forward in its growth and evolution towards better outcomes for those very individuals.

The Power of Story

I believe we are storied beings. Our lives have a beginning, middle and end. Within that life, there are numerous chapters, plots and sub plots. There have undoubtedly been some unexpected twists. The characters include heroes and villains; perhaps some characters turn out to be a bit of both! And while this all might seem self-evident, the true impact of our need and reflex to story our lives was a relatively recent revelation for me.

Two books that I read recently – Brené Brown’s “Rising Strong” and Bessel van der Kolk’s “The Body Keeps the Score” – have really driven home the impact of how we story our lives. Brené’s work is all about story and about owning our stories. It provides a roadmap for how to live through – and rise from – those ‘face down’ moments in our lives. Van der Kolk’s book isn’t specifically about story. It describes how traumatic events in our lives – both the sudden and extreme ones as well as the repeated ones that traumatize us over time – are stored in our brains. Both authors point towards the need to understand how we’ve created (and in some cases, made up by filling in the blanks) the story of those events. In the case of truly traumatic events, it involves opening pathways for the cognitive processing part of the brain (the neo-cortex) to have access to (and help to sort out) the painful memories stored like flashcards in the more ancient limbic part of our brain. The message for me from both books; having an honest and authentic account of our stories and being able to fully own (or integrate) them is critical to our happiness and success in life.

That revelation got me to thinking about the stories we have and hold about the organizations we work in. I believe that the act of owning and integrating our organization’s story can be just as powerful for an organization’s success. After all, once you strip away the buildings and equipment and policies and titles, organizations are nothing more than networks of relationships. So I decided to find ways to integrate the concept of ‘story’ more fully in my work. Along with my colleague Kim Lyster, I used the story metaphor as part of a recent strategic planning event. We helped the organization tell ‘the story up until now’, which included some of the traditional elements of a SWOT analysis. That process confirmed that the stories people made up about the organization were often only partially accurate or complete. We teased out, listened to, and valued all of the stories that people carried about the organization and we rumbled with the differences in those stories. Stories are very real in their consequences; they inform actions and behaviors. So the act of ensuring that the stories are consistent, accurate and fully owned by everyone is critical to building a foundation for the future. The work also included a heavy emphasis on values identification, because I firmly believe that knowing our values is core to us understanding and owning our stories, both as individuals and in our organizations. Having established a firm base of a shared story and common values, the second half of the planning event involved helping the organization to “write the next chapter”. We supported them to clearly define an authentic future state and the steps needed to walk a path towards that desired end based on what they learned through rumbling with their stories. Although we need to continue to refine this approach, I’m very encouraged by our initial experiences and plan to continue to evolve it in some upcoming strategic planning events.

Beyond strategic planning, I’m convinced that the work to give people tools to reality check their stories and rumble with differences will transform organizations. So as you walk your walk everyday in your organization or company, I encourage you to think about the ways in which the people you are interacting with are constructing their stories and how that impacts the work. By remaining curious and honouring those stories, we can open a space to have courageous and sometimes difficult conversations about where we want to go and how to get there. And those conversations can be powerful drivers of positive change.

PS – I plan to discuss how story can be a bigger part of evaluation work in an upcoming post, so stay tuned!

Using Story in Evaluation

The Power of Story

Subscribe Free Resources

Categories