Data analysis: how to measure critical reflection?

The process is slow but I finally feel as if I am getting somewhere with the qualitative analysis part of my thesis. I have been analysing participants' reflection assignments using the Levels of Reflection Taxonomy - descriptive, explanatory, supported, contextual and critical using NVIVO - which I developed way back in August 07.

Since pulling out the textual references at each category and sub-category for the five levels of reflection, I have annotated them and put them in tables so I can pull out the themes and patterns both for each participant and across the group. I have also calculated frequencies for each category and sub-category for each level of reflection so I can compare participants' writing.
A dilemma came up when trying to look for the higher levels of reflection - contextual and critical - it is impossible to gauge these levels by using a reduction methodology i.e. coding each sentence. The unit of a sentence I have used for coding in NVIVO works for descriptive, explanatory and supported reflection. To ascertain whether contextual or critical reflection is occurring, I will need to analyse the data more holistically, i.e. look for paragraphs and chunks in each assignment and rate them for contextual or critical reflection using criteria. So to do this I need to go back to the theoretical principles I used to create the taxonomy in the first place.

It was based on the following people's work:

Allen, J. & Jeffers, G. (2000). Teaching the dialectic process to preservice teachers in an educational psychology class. Annual meeting of the American Educational Research Association. New Orleans, LA, April 24 – 28.

Hatton, N. & Smith, D. (1995). Reflection in teacher education: Towards definition and implementation. Teaching and Teacher Education, Vol 11 (1), p33-49.

The five level approach developed by Hatton and Smith (1995) has reflection-in-action placed at the highest level. In summary, the model has the following components:

1. technical - decision-making about skills

2. descriptive - seeking best possible practice

3. dialogical - exploring alternative solutions

4. critical reflection - goals and ethical practices in wider context

5. reflection-in-action - contextualisation of multiple viewpoints - drawing on

the other four levels.

Sparks-Langer, G., Simmons, J., Pasch, M., Colton, A (1990). Reflective pedagogical thinking: How can we promote it and measure it. Journal of Teacher Education, Vol. 41, No. 5, 23-32.

I have also started organising the findings to see how I can answer the main research question as below:

1. What type of reflective strategies support practitioners when they develop and present an electronic portfolio?

a. What sort of approaches do participants take when writing reflectively?

My next step is to explore critical reflection further. Here is a paragraph I like about critical reflection:

“Critical reflection involves thinking and problem solving (Copeland, Birmingham, De La Cruz, & Lewin, 1993; Dieker & Monda-Amaya, 1997). Problem solving is a process in which capable individuals attempt to make sense of a challenging situation, identify areas of practice needing scrutiny, define goals for improvement, and pursue actions to accomplish them.

Reflective practitioners use this process to modify and enhance their understanding of professional practice. The end result of problem solving is reconstruction of knowledge. Ginsburg (1988) notes that critical reflection must involve practical experience as well as a knowledge base for thinking: Critical praxis is the process of combining critical thinking and critical practice. The issue here is to retain a critical stance toward society, schooling, and teacher education, while developing and refining strategies for action, for intervening in political, economic, and ideological arenas. It is not just a question of theory informing practice, but also practice informing theory (p. 202).” An Examination of the Construct of Critical Reflection: Implications for Teacher Education Programming in the 21st Century."

Author(s):Deborah S. Yost, Sally M. Sentner and Anna Forlenza-Bailey.

Source:Journal of Teacher Education 51.1 (Jan 2000): p39. (6488 words)