Action research and evaluation on line
Session 12: The Snyder process (2)
This is Session 12 of areol, action research and evaluation on line, a 14-week public course offered as a public service by Southern Cross University and the Southern Cross Institute of Action Research (SCIAR).
...in which I expand briefly on some features of the Snyder process evaluation phase, and examine in more detail the use of performance indicators in the outcome evaluation phase
How do you monitor your performance in practice? If you could use only one indicator of your long term effectiveness, what would you choose? If you were using it only for your own insight, how effective would it be? If you set out to defeat it -- to use it to appear to be effective -- how would you go about that? And in the light of your last answer, what could you do to develop a more robust indicator?
In this session
- Introduction
- Process
evaluation
- The
function of performance
indicators
- Indicators,
not measures
- Developing
indicators
- Short-cycle
evaluation and other processes
- Notes
- Archived
resources
- Other
reading
- Activities
- In
summary...
The previous session dealt, in overview, with the Snyder evaluation process. In this session I want to focus on the identification and use of performance indicators. Before then, I'll summarise the Snyder process, and add a little more detail to the discussion of process evaluation.
You will recall that the Snyder process uses a particular systems model:
resources --> activities --> effects --> targets --> ideals
with the addition also of a feedback loop back to resources.
The three phases of the process are defined in terms of this model.
The purpose of it all, if you carry out the whole process, is to improve whatever it is you are evaluating. More importantly, it allows the improvement to be ongoing.
Process evaluation
The first phase of the Snyder process is process evaluation. It examines the links between adjacent elements: between vision and targets; between targets (or vision) and immediate effects; and between resources and activities.
At the end of a participative process evaluation, participants understand better how activities change resources into outcomes. They know more, too, of the unintended outcomes of their activities, and of the long term consequences.
Let me now add some detail which will help you to turn the descriptions into practice. The examination of links proceeds somewhat differently, depending on which links you are examining:
- In
comparing targets to ideals, look for orphans: targets which
don't connect well to ideals, and ideals to which few targets
contribute. Make appropriate changes to targets, or
ideals, or both. In other words, improve the alignment of
targets and ideals.
- In
comparing immediate effects to targets, again look for orphans.
Again make appropriate changes. In effect you are asking,
how well aligned are immediate effects and ideals? -- and this
is really asking, how well aligned are activities and
ideals?
- As the
immediate effects are derived from the activities, there is not
much point in a comparison. However, you can examine the
unintended as well as the intended effects of each
activity. In other words, what are the unintended (and
often unseen) consequences of what we do? Are they beneficial
or harmful?
- The list of resources is similarly derived from the activities. Here, you are checking to see if the most resource-consuming activities are also the activities which contribute most to the ideals. Are we spending our resources where they will do the most good?
Now to the core topic of this session, performance indicators. You will recall that these were generated by the second phase, and used to provide ongoing feedback in the third phase.
The function of performance indicators
There are many reasons why you might want to carry out an evaluation; but they tend to boil down to three:
- To
decide, simply, if the program (or whatever) is achieving what
it is supposed to. To my mind, this is very difficult to
achieve. I have decided that I'm not often prepared to
put in the effort for such doubtful gains. Others might
(and do) decide differently.
In any event, the Snyder allows you to attempt this if you do think it's worthwhile. It's done using the first and second phase of the process.
- To devise
ways, in a one-time exercise, of improving the program. This is
achievable, though it begs the questions of what an improvement
might be. The Snyder process sidesteps this issue by
letting the stakeholders define the outcomes. The first
phase of the process addresses this. As it can be
difficult for stakeholders to reach agreement you may wish to
use a more elaborate first step to develop a
vision.
- To set up within the program some means for ongoing monitoring and improvement. In other words, the intention is to create a self-improving system. This achieves what the second achieves, and then keeps on doing it. It requires all three phases. 1
Performance indicators provide one of the vehicles by which outcome evaluation and ongoing improvement are achieved.
Think back to the "playing darts blindfold" thought experiment. 2 Performance indicators are what allow the people in the program to get feedback which is as direct and immediate as possible.
In effect, performance indicators serve as proxies for ideals. Ideals can't be perceived directly, being general and in the future. Performance indicators can be thought of as necessary (though not sufficient) proxies that indicate that you are taking steps on the way to those ideals.
Indicators, not measures
The label "indicators" is well chosen, in my view. Performance indicators indicate or signal that you may or may not be on track. They don't measure achievement.
Suppose some indicators decline. That doesn't mean that achievement has necessarily also declined. It may have. Or it may be that something else that we don't understand has occurred.
Declining indicators signal a need for research, not a need for blame.
Your experience probably informs you that people become skilful at delivering the indicators without the performance. They devise ways of defeating the feedback system. Here are four strategies for dealing with this. (I think that, if you have to use these, you are already in trouble. 3 )
First, we try to cover all of the ideals with our indicators. It is then more difficult for people to focus on those ideals which are signalled by indicators, and ignore the rest.
Second, we recognise that any one indicator may bear only a rough relationship to the ideal it signals. Packages of multiple indicators will confer some assurance that the indicators are a reasonable proxy.
Third, we monitor resource use and unintended consequences as well as intended consequences. People may otherwise drive up apparent performance by driving up associated (but unsignalled) costs and unwanted consequences. (We also bear in mind that many of the unintended consequences may be less tangible, and less immediate.)
Fourth, the indicators are developed and used by those people whose performance is being indicated. In other words, we assume that most people would prefer to do a good job than a poor job. We give them access to the information that allows them to do so.
This is sufficient background for us to return to the topic mentioned briefly in the previous session: developing indicators.
Developing indicators
You will recall that the previous session stated, briefly, that effective indicators were an adequate sample of:
- the ideals;
- intended immediate effects;
- unintended immediate effects; and
- resource use.
I can now add: for each ideal, a package of indicators is safer than a single indicator.
For example, labour turnover is an indicator of morale. By itself, however, it isn't a very good one. It may go up or down because, say, employment levels have changed in the economy. Absenteeism is also an indicator of morale, again not a good one. It may be more sensitive to the incidence of colds and flu than the level of morale.
Use both turnover and absence, and you've improved the quality of the indicator. Add quality of goods produced, variability in productivity, and timekeeping, and you've improved it further.
It is self-evident, I think, that it's better to sample all of the ideals if you can. The result, otherwise, is to channel performance into those ideals which the performance indicators point to.
I can offer an example from academia. The number of publications in refereed journals is sometimes taken as a prime indicator of academic performance. Many academics therefore seek to achieve as many publications as possible from each study.
You might reasonably argue that the outcomes would be better if they put more of their effort into doing their research, and reported it as economically as possible.
Now consider the three categories of indicator that I'm advising: intended effects, unintended effects, and resource use.
If you omit intended effects, you're ignoring the indicators which usually point most directly to the ideals. If you omit either of the others, you make it easy for people to lift apparent performance by increasing unintended effects or by consuming more resources. You're lifting apparent performance by increasing costs.
Summarising so far... We can say that the Snyder process seeks to create:
a package of indicators ...... which adequately sample ideals, intended and unintended immediate effects, and resource use ...
... and are developed and used by those whose performance is under consideration.
So, why not use activities as indicators?
In fact, sometimes those are the only indicators that are accessible enough in the short term to be feasible. But there is a trap in their use. It assumes that there is one best way to turn resources into outcomes. In practice, different people may be able to achieve the best results in different ways.
From another perspective, it really isn't the boss's or the system's business how a person or a team achieves outcomes. If the desired outcomes are achieved within the resource constraints, the system has what it needs. Any other responsibility can be devolved to the individual or team.
Think of it in systems terms. The inputs (that is, the resources) and the outputs (that is, the effects and targets) affect the system. The activities, of themselves, don't. 4
Now consider the choice of people to identify the indicators. There are actually two choices to be made here. First, who devises the indicators? Second, who uses them?
I've described the Snyder process used participatively -- the stakeholders do the analysis, perhaps with guidance from a facilitator. There are obvious reasons for doing this:
- the stakeholders are often the people who have a sufficient understanding of the system; and
- there will be less resistance to the system than if it is imposed.
There is a further advantage, and to my mind a more important one. If the stakeholders devise the indicators in the first place, they are more likely to revise them if they don't work. In other words, there is a reasonable chance that the stakeholders will own the process for developing the indicators as well as owning the indicators themselves.
In short, there are advantages of having the indicators devised and used by the people whose performance is indicated.
(Apart from this, there are values involved. You may prefer, as I would, to involve people because it is consistent with your values to do so.)
Short-cycle evaluation and other processes
It is probably apparent from my description that the Snyder process implements many of the action research techniques that I mentioned in earlier sessions. (For that matter, I haven't described it in enough detail for you to carry it out without drawing on the earlier material.)
Notice too, however, the result of short cycle evaluation: it sets up a feedback system which people can use to carry out an action-research-like activity thereafter. The action research doesn't stop when the Snyder is completed -- the participants become action researchers of their own performance.
The result can be continuous improvement: "kaizen", as some of the quality management literature calls it. 5 In fact, you could use the Snyder process as a qualitative version of Total Quality Management or TQM.
Done participatively, it bears a strong resemblance to the form of TQM recommended by Deming. 6 Genuine responsibility is devolved to the people who carry out the activities, and they continue to manage the process. In this regard, it also serves a function similar to that of appraisal: it helps to manage performance. However, notice the differences from many performance appraisal systems. Notice especially:
- that
people appraise themselves; and
- that the appraisal is of a team, not of an individual.
In these respects, it overcomes the objections to appraisal which Deming voices.
- You may
think there is an apparent contradiction is this. If you
can't define outcomes, then by what criteria are you going to
monitor improvement? My answer is that you can't measure
outcomes and their causes well enough to decide if the
program is doing what it is supposed to. Even if you can
measure the outcomes reasonably well (and often you can't), you
can seldom attribute them confidently to the program. Too
much else will have influenced them at the same time.
However, you can usually define the ideals, roughly, well
enough to derive performance indicators for them. The
indicators can tell you if you're headed in the right
direction. And you can continue to polish the indicators
over time.
Note, too, the insistence that we are here dealing with indicators, not measures. We're not trying to determine in any definitive fashion if someone is performing. We're trying to give them access to unthreatening feedback which they can use to monitor their own performance. [ back ]
- For those
who don't have the earlier session... The "darts" thought
experiment is one where people were asked to imagine playing
darts blindfold, without instruction and with general and
occasional feedback. They were then invited to consider
their present job (for example) and decide if it was like
playing darts blindfolded, or like learning darts with skilful
and supportive feedback.
[ back ]
- As I said
earlier, you can use evaluation processes to enable people, or
to control them. If people are trying to defeat the
evaluation system, you may find it useful to ask yourself: Why
are they doing so?
[ back ]
- There are
more complex issues here, too. This isn't the place to go
into them in detail. Suffice it to say that an
organisation which controls people's activities is a very
different kind of organisation, with very different management
styles, to one which controls only inputs and outputs.
Increasingly, the structures which work effectively are those
which devolve as much responsibility as possible for the work
to those who are doing the work. You can adopt the
strategy of controlling activities in a stable
environment. It works well enough then. It breaks down
quickly in unstable environments.
[ back ]
- Imai,
M. (1986) Kaizen: the key to Japan's competitive
success, New York: McGraw-Hill.
[ back ]
- Deming, W.E. (1986) Out of the crisis. Cambridge, Mass.: MIT Centre for Advanced Engineering Study. [ back ]
Archived resources
The most relevant archived resources are those already mentioned: "snyder" and "snyder-b". These consist of two more-detailed descriptions of the process. I suggest you download and examine them before attempting the exercises below.
http://www.uq.net.au/action_research/arp/snyder.html ftp://ftp.scu.edu.au/www/arr/snyder.txthttp://www.uq.net.au/action_research/arp/snyder-b.html ftp://ftp.scu.edu.au/www/arr/snyder-b.txt
Other reading
The works mentioned in the previous session are again relevant.
Activities
A thought experiment
Continue the "darts" experiment. You've already rated your overall job in terms of the feedback you receive. Now look separately at the components of the job. (All but the simplest jobs consist of several different tasks which can be separately analysed.)
Imagine a typical day. Identify the major activities you engage in. For each of them, give a rating on a 10-point scale, where
1 is like playing darts blindfold 10 is direct, immediate, accurate, non-threatening feedback.
What could you do to improve the feedback for the most important activities?
An individual activity
People usually think of their work, or their social or family life, as a sequence of activities. Thinking of it in terms of resources and immediate effects is more difficult. Yet these are what affect others.
In the previous session you analysed a typical day in terms of the costs and benefits. Carry that analysis further: notice who bears the costs, and who enjoys the benefits. Identify the feedback that would most effectively help you to monitor the performance.
If you wished, you could even approach those people and ask them for relevant feedback. I suggest that you do your own analysis first, then check it out against their perceptions.
For your learning group
In the previous session you were invited to begin helping each other to choose a suitable project to which you could apply the Snyder process, at least in imagination. Help each other to continue that process. In turn, imagine how you might actually facilitate a complete Snyder process applied to the project of your choice.
Help each other to give particular attention to the development and use of performance indicators.
In summary ...
In this session, we examined in a little more detail two aspects of the Snyder process: the comparisons within the process evaluation phase, and the development and use of performance indicators. We also briefly compared the Snyder process to Total Quality Management, and to appraisal processes.
Next session we examine a different systems model: soft systems methodology. Although not usually thought of as an evaluation process, it serves that purpose very well. See you then -- Bob
Let's practice action research on areol. What ideas do you have for improving this session? What didn't you understand? What examples and resources can you provide from your own experience? What else? If you are subscribed to the email version, send your comments to the discussion list. Otherwise, send them to Bob Dick
_____
Copyright © Bob Dick 2002. May be copied provided it is not included in material sold at a profit, and this and the following notice are shown.
This document may be cited as follows:
Dick,
B. (2002) The Snyder process (2).
Session 12 of Areol - action research and evaluation on
line.
URL
http://www.uq.net.au/action_research/areol/areol-session12.html