Action research and evaluation on line
Session 11: The Snyder process
This is Session 11 of areol, action research and evaluation on line, a 14-week public course offered as a public service by Southern Cross University and the Southern Cross Institute of Action Research (SCIAR).
...in which an overview of the Snyder evaluation process is given, incorporating the three phases of process, outcome and short-cycle evaluation
Take a moment to consider -- how good is the feedback you receive on your performance? What sort of feedback would you find most useful? How might it be provided so that you would use it?... and then consider further -- how much do you know about what people think of you and what you do? What would you like to know about this? In what form would you prefer to receive it
In this session
- Introduction
- A
systems model
- An
overview of the process
- In
partial summary
- Notes
- Archived
resources
- Other
reading
- Activities
- Conclusion...
This, the second of the sessions on evaluation, takes up a particular evaluation process: the Snyder process. 1 I begin by describing the systems model which underlies parts of the process. I then give an overview of the three-phase process itself.
I should mention that this is by no means the only approach to evaluation. Various other approaches, especially those that are both qualitative and participative, are likely to resemble action research. Many action research processes can be pressed into service for evaluation purposes. In addition, there are many approaches beyond those that resemble action research.
I choose this approach because I use it often. Also, it lends itself to participative approaches, which I prefer. It includes a variety of approaches within itself: this can be a benefit. It can be integrated easily and effectively with strategic planning, and may even rescue that often-misused process from its less intelligent applications. Most of this will become more apparent as we proceed.
Above all, it's an approach for practitioners. I've had good results using it participatively to bring about change. The group being evaluated usually begin to change their behaviour by the third or fourth step into the first phase.
I invite you to notice the way it builds understanding as it proceeds. This, I think, is an important reason for its effectiveness. As people come to understand how they achieve their objectives they become more effective at achieving their objectives.
As a bonus, you'll find it easy enough to learn if you don't get lost in the detail.
A systems model
The Snyder process uses systems concepts. It assumes that the organisation or unit or project or program being evaluated can be viewed as a "system". Systems models treat a program (or whatever) as something which transforms inputs into outputs. Resources into achievements.
By monitoring the achievements you provide yourself with feedback. The feedback allows you to make better choices: about inputs to use; about activities; about outputs to pursue.
The Snyder process uses three levels of output. They take place over different time spans. The five elements are:
resources -> activities -> effects -> targets -> ideals
| | | |
| inputs | processes | outputs |
In more detail:
resources These are the inputs. They consist of anything (time, money, materials, etc.) consumed by the activities. (They also include anything, like skills, which are not "consumed" but are required and are not plentiful.)
activities These are the processes. They are the activities and operations carried out as part of the program (or whatever it is being evaluated). They include what people do, day by day and moment by moment.
effects (An abbreviation for "immediate effects".) These are the outcomes which result as the activities are carried out: the immediate results of the activities. There are intended and unintended effects.
targets (Sometimes called objectives.) These are the identified outcomes which the program pursues. (In the corporate literature they go by a variety of names: goals, objectives, and the like.)
ideals (Often called "vision".) This is the "better world" to which the activities are presumed to contribute in the long run. The ideals are future, and general. They are more something to aim for rather than something that will actually be achieved.
For convenience, you can think of targets as being tied to the planning cycle of the program. This is often a year. For some purposes it may be more or less than this.
In comparison the immediate effects are short term. The ideals are very long term, and probably unachievable: the vision which guides and motivates the program.
The diagram provides a partial summary.
Why don't you try it out? Choose some important activity you engage in. Identify the other components, from resources through to idealsOr choose your present activity. You're reading this, perhaps on a computer screen. What are the resources and immediate effects? What is your best guess about the targets I think you might have achieved, and the vision of a better world which encourages me to provide this material?
This systems model is a categorisation. It helps to manage the data collected.. It also provides a framework. It underlies each stage of the process, which is now described.
The Snyder process sets up feedback loops. These allow stakeholders to better understand and manage their activities and resource use. The stakeholders are thus better able to achieve worthwhile and desired goals.
An overview of the process
The Snyder process is designed as three phases. Each addresses a separate form of evaluation. Each builds on the preceding phase or phases.
Please note: In this and subsequent sessions, I address some important issues only very superficially. Entry, contracting, identifying stakeholders, and building relationships are given only a cursory treatment. However, these are at least as important as in other action research processes. Given the scepticism about evaluation in many quarters, they may well be more important. I assume you will fill in the gaps from earlier sessions.
In addition, the process as described below doesn't pay explicit attention to what the program or unit environment is doing. You may therefore want to include, as part of the "ideals", an examination of the changes going on in the relevant environment. (Many vision-seeking exercises include this, and may be pressed into service.)
In the following descriptions, I assume a participative approach to evaluation. The "evaluator" acts more as a process consultant, guiding the process used for evaluation. The actual evaluation is done by the participants.
This is done on the grounds that it is usually participants who have to turn the evaluation into change. Their understanding and commitment matter more than that of the "evaluator".
On occasion, you may well have reason to do an evaluation non-participatively. If so, the same overall process can be used.
On yet other occasions you may be working with participants who are, or can become, sufficiently skilled in the processes. You can then usefully involve them as co-researchers.
Each of the three phases of the Snyder process is a different form of evaluation seeking to answer different questions. The phases are described below.
A. The process evaluation component 2
The goal is for the stakeholders to understand the process by which they achieve what they achieve. Unintended achievements, good or bad, are important.
The questions it seeks to answer are to do with the way the program or unit operates. What resources are consumed by what activities? Are these the activities which appear to contribute most to targets and vision? What unintended effects do they have?
The process evaluation tries to answer these questions by addressing the links between the elements. It operates by trying to identify:
- which activities consume which resources;
- which activities produce which immediate effects, intended and unintended;
- which immediate effects contribute to which future targets; which immediate effects hinder achieving those targets;
- which targets are likely to contribute to which ideals.
It does this, in general, by identifying and comparing adjacent elements -- for example, ideals and targets. It then analyses which targets and ideals are associated. Mismatches between targets and ideals then become the catalyst for changing targets, ideals, or both.
It is possible to do only a process evaluation, and then switch to action planning. However, I think the Snyder process is most valuable when it also includes all three phases.
In short, the process evaluation phase seeks to understand how the unit or program operates. It does this by examining the links between the elements:
resources -> activities -> effects -> targets -> ideals
^ ^ ^ ^
B. The outcome evaluation component
The goal of this phase is to develop performance indicators. These allow the achievements of the program or unit to be monitored.
Outcome evaluation often seeks to answer the question: Is this program achieving its goals? Or, Is this program better than program X?
My personal view is that these questions are seldom answerable. Therefore, in the Snyder process, this phase seeks instead to determine how performance can be monitored. The performance indicators developed in this phase can then be used to set up feedback loops.
In general, the Snyder process does this by finding present indicators of future targets and ideals. Ideals are not evaluable in any real sense. Targets can be evaluated when you get there; but that may be too late. Indicators are here in the present.
The process for identifying indicators is simple enough. You start with the ideals, and follow them back until you find something present that you can evaluate.
To anticipate the topic of a following session... The criteria I will be suggesting for indicators is that they are an adequate sample of:
- the ideals,
- resource use,
- intended immediate effects, and
- unintended immediate effects.
As far as possible, I suggest you don't use performance indicators of activities. That assumes that there is one right way to do something. It may well be that different people can achieve outcomes in different ways.
Activities are often easily measured. Unfortunately, the activities which work best for some people may not be the best way for others to achieve the outcomes. Activity-based indicators may constrain people, and ignore individual differences.
In short, the outcome evaluation phase builds on the understanding achieved from the process evaluation phase. It uses this understanding to identify performance indicators which can serve as proxies for future targets and ideals: