Action research and evaluation on line

Session 6: Rigour in action research

 

 

This is Session 6 of areol, action research and evaluation on line, a 14-week public course offered as a public service by Southern Cross University and the Southern Cross Institute of Action Research (SCIAR).

...in which I discuss ways of increasing rigour in action research without sacrificing the flexibility which allows action research to achieve change in field settings

 

You are no doubt often given information as you talk with someone.  How much trust do you place in it? How do you work out what it means? And how much do you trust the meaning you give to it? How would you go about testing the accuracy of the information and the attributed meaning?

In this session:

 

To recap on some of the previous material ...

Action research can be used in a variety of settings, and for several purposes.  Its nature is to some extent determined by its primary aim: to allow the simultaneous achievement of change and understanding.  As I've said before, action research is action and research.

To achieve change, action research has to be responsive to the situation in which it is used.  It has to exhibit flexibility. Action-oriented research in field settings depends to a large extent on skill and technique.  In my view it also is, to a large extent, a performing art.  (But then, in my view, so is most good research.)

 

Rigour and flexibility and commitment?

It seems to me that these features, responsiveness and flexibility, are essential qualities.  They are much of the reason why action research is able to do what it can do.  At the same time, they make it hard for you to achieve rigour in conventional ways. Compared to conventional research, action research must develop its own sources of rigour.

In addition, change occurs most easily and effectively when those who are to carry it out are committed to it.  That is best done, I believe, through involving them as directly and deeply as the situation allows, and they are willing.  This, too, does not fit well with rigour as conventionally defined.

 
In this session I begin to explain how adequate rigour may be achieved.  I describe how it can be done without sacrificing the commitment to change, or to the responsiveness and flexibility (and participation) on which change depends.

It will be seen that many of its features contribute to both effective change, and good research.  Far from being obstacles to rigour, commitment and flexibility can be used as a foundation for it.

 

The cyclic nature of action research

As I've said before, at the heart of most versions of action research you will find a cycle.  There are a number of ways of describing it.  The simplest is as an alternation between action and critical reflection.

action --> reflection --> action --> reflection ...

(For me, critical reflection is one of the important defining features of action research.  I use "critical" in the sense of "evaluative", not necessarily as used by the Frankfurt school of philosophy.)

The action research cycle allows change and understanding to be achieved at the same time.  The action produces the change.  The critical reflection serves two purposes.  It draws understanding from the experience of the action.  It then allows the development of plans to turn understanding into action.

Accordingly, reflection has two main parts.  Critical review occurs after the action.  Planning comes before the next action.  You might therefore characterise the action research cycle, simply, as

intend --> act --> review

It seems to me that most versions can be related to this.  In fact this is equivalent to Kurt Lewin's own cycle of

plan --> act --> evaluate

(You will notice the similarity to some other cycles such as those for experiential learning or continuous improvement.)

The best known cycle here in Australia is that of Stephen Kemmis and Robin McTaggart 1 

plan --> act --> observe --> reflect

You might think of this as reflection before action, action, reflection during action, and reflection after action.

Some versions of action research depend upon frequent cycles.  These offer great flexibility.  At each point you need only enough information to take the next step -- and that step leads in turn to the step following.

Here is where part of the rigour comes in.  Each cycle gives you yet another chance to challenge the data and interpretations of the previous cycles.  More cycles, more challenge.  And more assurance that the results are valid.

There are some action research processes which appear to use fewer and longer cycles.  Soft systems methodology is one.  2   Such processes can and often do contain many small cycles within the larger cycles, and are strengthened by them.  Action research can consist of cycles within cycles within cycles, perhaps within cycles.

Within each cycle you can...

Intend (or: reflect before action)

This allows you to decide what you want the next step to achieve.  In addition it allows you to think about what actions might achieve the desired outcomes, and why.

An important part of the "intend" step is to work out how you can most strenuously test the data and emerging interpretations from previous cycles.

Your plans may include changes to your questions, your methods, the people involved, and perhaps other aspects of the project as well.

Act (and: reflect during action)

This allows you to check that you are doing what you intended.  You may find, for example, that you lack the skills to put your plans into action.  Or they may not work as well in practice as you expected.

Also, you can monitor whether or not you are achieving your intended outcomes.  You can change your action in the light of your experience.

Review (or: reflect after action)

Now you can recollect your actions, and those of other people.  You can reflect critically on the assumptions that underpinned your intentions.  You can try to make sense of the experience.

It is useful to review:

  • the goals you are pursuing
     
  • the data you collected
     
  • your interpretations
     
  • the methodology and methods you are using, and how well they are working
     
  • the people who are involved as participants or informants.

You then move into the second half of the reflection session, by deciding your intentions for the next cycle.

If the constraints allow, all three stages are likely to be more effective if they are done by all the players, acting as co-researchers with you.

 

Triangulation

There are also other strategies you can use to increase your confidence in your data and interpretations.  Most important among them is the use, preferably within each cycle, of multiple sources of information.

There are several ways of doing this, some examined below.

Different methods of data collection

This is commonly called "triangulation" (some people use the term for any technique which uses multiple sources).

For example, you might use both interviews and focus groups, as in the earlier case study (archived as "case1").

Better still, you could choose several very different forms of data collection, such as interviews, analysis of documents, and observation.

Different methods of data interpretation

You might collect interpretations as part of a focus group, and also analyse the transcript of the focus group with the help of suitable software.  (Transcribing whole sessions does lead to a lot of time and expense, so you may prefer to find some other means of interpretation.)

You can also use theories to help you make sense of data.  Soft systems methodology, for example, uses "systems theory" 3  as a framework for data interpretation.  Within the same process, you could use some other theory as well.

Different informants

It is usual in experimental research to try to compile a sample which is randomly chosen from the population being studied.  In action research, it may often be more useful to put together a sample which is as diverse as possible.

Such a sample is more likely to contain apparently contradictory information.  It is often such information which leads you to ask the most useful questions in the next round of data collection.

Multiple case studies

There may be occasions when you can set up action research projects in different situations as two or more independent case studies. (Action research methodologies and case study methodologies go well together.)

Imagine that someone has asked you to help introduce multi-skilling to an organisation.  You could find some other organisation also interested in multi-skilling, and study it at the same time.

Equivalent information using different questions

You can ask different questions which pursue the same information of the same informant.  For example, asking for dissatisfactions and suggestions for improvement obtains the same information from different perspectives.

A common interviewing technique: after an informant offers her own opinion, ask her what opinions others hold.

Overlapping information through
different questions

This is related to the previous point.  You can ask an informant related questions which yield overlapping information.

This is an important source of rigour in the Snyder evaluation process, described in later sessions.  For example, the participants are asked to define a long-term vision; and then to identify intermediate-term targets or objectives.  When they analyse which targets contribute to the vision, there are usually gaps and mismatches.  This helps them to re-order their priorities.

Different researchers

One of the advantages of involving participants as co-researchers is that they provide a different perspective which can challenge your own.

You can also involve colleagues as researchers, interpreting information separately and then comparing notes.

For example, in convergent interviewing 4  I recommend that two (or more) interviewers carry out independent interviews.  They then meet to compare results.  It's sometimes possible to use two pairs.  Within each pair, two interviews are compared at a time.  Between pairs, the interviews from whole samples can be compared.

 

Creating a climate

Chris Argyris has spoken compellingly about the ways that threats to valid data can arise in many settings.  5  He has offered his own alternative, which creates a climate of open inquiry within the group of people who are researchers and participants.  6 

(Some find his work hard to read.  His critical analysis, in my opinion, is good enough to repay the effort.)

An earlier session spoke of the importance of beginnings.  It mentioned some material on entry and contracting, and suggested some activities to help in building open relationships.  You may be able to build an effective participant group in this way.

Your own willingness to be confronting and supportive at the same time will do much to provide a model for other people to follow.

 

Using the literature

The literature is another valuable source of information.  This is especially true of literature which reports relevant experimental or quasi-experimental research findings.  Its methods are sufficiently different that, if they yield similar conclusions, you can have more confidence in those findings.

(If the literature disagrees with your own findings, don't assume that you are the one who is mistaken.  Research the issue further.)

You face some important choices in deciding how to use the literature.

On the one hand, extensive prior reading of the literature can help you to avoid re-inventing the wheel.  On the other, it's hard to know at the start of a study which literature will be relevant.  As well, if you develop your conclusions only from the data, then the literature later provides a more stringent challenge.

You might consider this compromise strategy...

Use the methodological literature to help you choose a robust approach.  But then, don't assume you have it right.  Review your methodology continuously as you conduct the project, searching out improvements.

Read the literature which is clearly relevant to the content of your study.  But stay focused.  Don't attempt to read as widely as you might for an experimental study.

You will then find that other literature becomes relevant as you continue the research.  Much of this literature allows you to assess the extent to which your results also apply to other situations.  7

 

Monitoring change

I suggest that you look for chances to move into action as soon as you reasonably can.  Don't plan a large change project and only then implement it.  Plan the first step, implement it, and then review what you have done.  In that way, each change provides a potential test of your plans and their underlying assumptions.

As I describe it here, action research is a pragmatic process.  It is intended to work: that is, to produce change.  If it doesn't, that indicates that your actions or assumptions were not appropriate.

 

In partial summary, here is a list of the methods suggested above...

You can use a variety of ways of increasing the rigour of an action research project:

  • use multiple cycles, so that later cycles can test the results of the earlier ones
     
  • combine data collection and interpretation within each cycle, so that interpretations as well as data can be challenged in later cycles
     
  • use different methods of data interpretation, or interpret the data against several relevant but different theories or models
     
  • involve different participants and informants, such as through the use of maximum-diversity samples
     
  • carry out two or more independent action research studies on similar topics
     
  • give different questions which pursue equivalent information to the same informant;
  • ask questions which yield overlapping information
     
  • use different researchers, including participants as co-researchers
     
  • involve a variety of participants, and create a climate in which they are encouraged to challenge your ideas
     
  • use the literature as a further source of disconfirming evidence
     
  • monitoring the achievement of planned changes; each change is a test of the assumptions and plans that led to it.

I believe these are important for research, whether mostly action-oriented or mostly research-oriented.  They demand attention when the study is being done as a thesis, or for publication. 8 

Whatever your reason for pursuing them, they can be summed up quite simply:

  • use a cyclic procedure
     
  • within every cycle vigorously seek out disconfirming evidence in as many ways and from as many sources as practicable.

Assume at all times that your data, your interpretations, your participants, your methods are all inadequate.  Seek to improve them.

 

Notes

  1. For example, Kemmis, S.  and McTaggart, R.  (1988) The action research planner, third edition.  Victoria: Deakin University. [ back ]
     
  2. Soft systems methodology is briefly described in a later session. There is a growing literature.  As a starting point, see Checkland, P.  and Scholes, J.  (1990) Soft systems methodology in action.  Chichester: Wiley.  For a similar and in some respects more elaborated approach see Bob Flood's description of his "Total Systems Intervention".  He has described it in  Flood, Robert A.  (1995) Solving problem solving: a potent force for effective management. Chichester: Wiley.  [ back ]
     
  3. Yes, I realise "systems theory" is more of a logic or taxonomy than a theory.  That doesn't really change the point I'm making. [ back ]
     
  4. A later session describes convergent interviewing in some depth, as it illustrates the use of conflicting evidence in devising more specific questions ("probes") for later interviews.  For more detail see my 1990 monograph Convergent interviewing, version 3 (Brisbane: Interchange). [ back ]
     
  5. Argyris, C.  (1980) Inner contradictions of rigorous research.  New York: Academic Press. [ back ]
     
  6. Argyris, C., Putnam, R.  and Smith, D.McL.  (1985) Action science: concepts, methods and skills for research and intervention, San Francisco, Ca.: Jossey-Bass. [ back ]
     
  7. Action research values local relevance over generalisability.  It is therefore often argued that you can't generalise from action research.  I hope this illustrates that that need not always be true. [ back ]
     
  8. I have archived some documents specifically on the writing of theses and publications.  See files "phd" and "research" on the arlist archive (more details below). [ back ]

     

Archived resources

Two files on the arlist archive directly address the topic of rigour in action research.  There are also two files specifically on action research for theses and publications.

In more detail...

rigour describes, from a practitioner's perspective, why experimental and quasi-experimental methods are often hard to apply in field settings

http://www.uq.net.au/action_research/arp/rigour.html
ftp://ftp.scu.edu.au/www/arr/rigour.txt

rigour2 explains how rigour and economy of effort can both be improved without sacrificing flexibility

http://www.uq.net.au/action_research/arp/rigour2.html
ftp://ftp.scu.edu.au/www/arr/rigour2.txt

rigour3 is a paper presented at the Association for Qualitative Research Conference "Issues of rigour in qualitative research" at the Duxton Hotel, Melbourne, Victoria, 6-10 July 1999.  It was written primarily for people with little familiarity with action research

http://www.uq.net.au/action_research/arp/rigour3.html
ftp://ftp.scu.edu.au/www/arr/rigour3.txt

phd is a brief account of one approach to action research for thesis purposes; it offers some suggestions for checking that your study is adequate both as a change process and a research process

http://www.uq.net.au/action_research/arp/phd.html
ftp://ftp.scu.edu.au/www/arr/phd.txt

research discusses action research for publication and for theses, with particular attention to documentation and writing-up.

http://www.uq.net.au/action_research/arp/research.html
ftp://ftp.scu.edu.au/www/arr/research.txt

 

Other reading

See the material by Argyris, listed in the Notes (above).  Then try the material below.  I'd suggest you start with Kirk and Miller, which is readable and useful.  Then perhaps try Smith, though it isn't an easy read.  Gareth Morgan's book is a collection of papers; each of them is informative and many are thought-provoking.

Some of the grounded theory literature is relevant here too. It was a pleasant surprise to me when I discovered the grounded theory makes many of the same assumptions I argue for above. (This is especially true of Barney Glaser's writing, less true of Anselm Strauss's.)

Dick, Bob (1999) Rigour without numbers: the potential of dialectical processes as qualitative research tools, second edition.  Brisbane: Interchange.

Glaser, Barney (1992) Basics of grounded theory analysis:  emergence vs forcing. Mill Valley, Ca.: Sociology Press.

Kirk, Jerome and Miller, Marc L.  (1986) Reliability and validity in qualitative research.  Beverly Hills, Ca.: Sage.

Morgan, Gareth, ed.  (1983) Beyond method: strategies for social research.  Beverly Hills: Sage.

Smith, John K.  (1993) After the demise of empiricism: the problem of judging social and educational inquiry.  NJ: Ablex Publishing.

 

Activities

A thought experiment

A friend has just passed on some information to you, suggesting that there is a spectacular opportunity for investment.  Ask yourself...

  • what determines the credibility of the information for you?
     
  • what determines its accuracy?
     
  • are credibility and accuracy positively related, or negatively related, or independent?

    An individual activity

(You may be working on a thesis, or a project for publication.  If so, use that as a basis for this activity.)

If you are more of a change agent than a researcher...

Take a process for collecting and interpreting information within a change program.  Analyse it, and devise ways of improving the rigour of the data collected and the interpretations of it without damaging its effectiveness for change management.

If you are more of a researcher than a change agent...

Take a typical field research process with a strong research emphasis.  Analyse the sources of its rigour.  Try to improve its ability to bring about change without undermining its rigour.

If you are a novice in both fields...

Find a process which is used for both action and research -- for example the structured focus groups described in the archived file focus.  Analyse the sources of rigour and change in the process.  How can you improve them?

http://www.uq.net.au/action_research/arp/focus.html
ftp://ftp.scu.edu.au/www/arr/focus.txt

For your learning group

Help each other analyse your projects.  Identify the main sources of flexibility, of commitment, and of rigour.  (If you are doing action research for thesis or publication, use that.) Help each other improve the rigour, using the strategies described above, without undermining the responsiveness and flexibility.

 

In summary

Action research is a methodology which pursues action and research.  The action is helped by flexibility and commitment.  It is possible to design action research processes in such a way that flexibility, commitment and rigour enhance each other.  This is made easier by using

  • a cyclic process in which data and interpretation are integrated
     
  • multiple sources of data and interpretation within each cycle
     
  • and at all time a vigorous pursuit of disconfirming evidence.

The few immediately-preceding sessions focused on participation -- an "action" aspect of action research.  This session took up the issue of rigour, which is a "research" aspect.

Growing numbers of people are using action research for theses, and for publishable research.  For them, in particular, the research aspects have to be primary.  But rigorous methods are a benefit for all action research.

The next sessions examine some specific methods which have been designed to increase the rigour of data collection and interpretation.  See you then.  -- Bob

Let's practice action research on areol.  What ideas do you have for improving this session? What didn't you understand? What examples and resources can you provide from your own experience? What else? If you are subscribed to the email version, send your comments to the discussion list. Otherwise, send them to Bob Dick

_____

 

Copyright © Bob Dick 2002.  May be copied provided it is not included in material sold at a profit, and this and the following notice are shown.

This document may be cited as follows:

Dick, B.  (2000) Rigour in action research.  Session 6 of Areol - action research and evaluation on line.
URL http://www.uq.net.au/action_research/areol/areol-session06.html

 

 

 

  

Maintained by Bob Dick; this version 11.02w; last revised 20020712

A text version of this file is available at
ftp://ftp.scu.edu.au/www/arr/areol-session06.txt