meeting 24 march and progress stunted by Womad

Image: Moana & the Tribe - WOMAD Taranaki 2009
At the moment I am chugging along with the individual cases. Womad sort of got in the way but now I am on the wagon again with the thesis. I am almost finished the third case. My supervisors have the draft general results and the draft methodology. They are on the shelf at the moment as the priority is to finish the individual cases so I can prepare a discussion about them.

Meeting 24 March 24, 2009

S & L are not sure about adding in extra description for case three. I have added more description about the style of writing in the section called: use of the reflective framework. The reason is because the reader/examiners will need to compare all the cases by headings which need to be consistent – if extra headings and extra writing may confuse examiners. However, I may have discovered a new heading which is needed in all the cases even if no data is available for some because they did not write their reflections using the template headings.

B to do

1. Send third case when done.

2. Put all three cases together (first one with feedback) and send to S & L.

3. Delete results chapter with S’s feedback and leave until later. Tempted to look.

4. Start on next case in a different doc. Add to finished cases and send as draft doc – this will help keep everything together and enable comparisons to be made more easily, plus control versioning of the document a bit better. S & L will keep the same version with their feedback on it.

5. Aim for two weeks to have all cases finished and sent to S & L.

6. Work on a discussion about themes across all cases.

Doctorate overview of findings presented to Spotlight Tertiary Teaching and Learning Colloquium for the Southern Region

On 20 November I gave a presentation about my Doctorate research to the Spotlight on Tertiary Teaching and Learning Colloquium for the Southern Region. The presentation is below or you can link to all functions for the presentation - pdf download, full screen etc - on Myplick. The Three-Step Reflective Framework diagram and template can be accessed here.

I have a Creative Commons by attribution licence on my work so as long as you ensure you give attribution to me, you are free to use my work and modify it to suit your situation.

BHSpotlight presentation three step framework2

Data analysis: how to measure critical reflection?

The process is slow but I finally feel as if I am getting somewhere with the qualitative analysis part of my thesis. I have been analysing participants' reflection assignments using the Levels of Reflection Taxonomy - descriptive, explanatory, supported, contextual and critical using NVIVO - which I developed way back in August 07.

Since pulling out the textual references at each category and sub-category for the five levels of reflection, I have annotated them and put them in tables so I can pull out the themes and patterns both for each participant and across the group. I have also calculated frequencies for each category and sub-category for each level of reflection so I can compare participants' writing.
A dilemma came up when trying to look for the higher levels of reflection - contextual and critical - it is impossible to gauge these levels by using a reduction methodology i.e. coding each sentence. The unit of a sentence I have used for coding in NVIVO works for descriptive, explanatory and supported reflection. To ascertain whether contextual or critical reflection is occurring, I will need to analyse the data more holistically, i.e. look for paragraphs and chunks in each assignment and rate them for contextual or critical reflection using criteria. So to do this I need to go back to the theoretical principles I used to create the taxonomy in the first place.

It was based on the following people's work:

Allen, J. & Jeffers, G. (2000). Teaching the dialectic process to preservice teachers in an educational psychology class. Annual meeting of the American Educational Research Association. New Orleans, LA, April 24 – 28.

Hatton, N. & Smith, D. (1995). Reflection in teacher education: Towards definition and implementation. Teaching and Teacher Education, Vol 11 (1), p33-49.

The five level approach developed by Hatton and Smith (1995) has reflection-in-action placed at the highest level. In summary, the model has the following components:

1. technical - decision-making about skills

2. descriptive - seeking best possible practice

3. dialogical - exploring alternative solutions

4. critical reflection - goals and ethical practices in wider context

5. reflection-in-action - contextualisation of multiple viewpoints - drawing on

the other four levels.

Sparks-Langer, G., Simmons, J., Pasch, M., Colton, A (1990). Reflective pedagogical thinking: How can we promote it and measure it. Journal of Teacher Education, Vol. 41, No. 5, 23-32.

I have also started organising the findings to see how I can answer the main research question as below:

1. What type of reflective strategies support practitioners when they develop and present an electronic portfolio?

a. What sort of approaches do participants take when writing reflectively?

My next step is to explore critical reflection further. Here is a paragraph I like about critical reflection:

“Critical reflection involves thinking and problem solving (Copeland, Birmingham, De La Cruz, & Lewin, 1993; Dieker & Monda-Amaya, 1997). Problem solving is a process in which capable individuals attempt to make sense of a challenging situation, identify areas of practice needing scrutiny, define goals for improvement, and pursue actions to accomplish them.

Reflective practitioners use this process to modify and enhance their understanding of professional practice. The end result of problem solving is reconstruction of knowledge. Ginsburg (1988) notes that critical reflection must involve practical experience as well as a knowledge base for thinking: Critical praxis is the process of combining critical thinking and critical practice. The issue here is to retain a critical stance toward society, schooling, and teacher education, while developing and refining strategies for action, for intervening in political, economic, and ideological arenas. It is not just a question of theory informing practice, but also practice informing theory (p. 202).” An Examination of the Construct of Critical Reflection: Implications for Teacher Education Programming in the 21st Century."

Author(s):Deborah S. Yost, Sally M. Sentner and Anna Forlenza-Bailey.

Source:Journal of Teacher Education 51.1 (Jan 2000): p39. (6488 words)

looking ahead in 2008

The pic is a reminder of my wonderful camping holiday on the west coast and Golden Bay - the holiday memories will keep me going in 2008. It certainly cleared my mind ready for the onslaught this year - now I am ready to focus.

This is a pic I took of Kaihaihai beach from the zig zag track at the start of the Heaphy track near Karamea - January 2008.

Data collection and analysis

To date data has been collected from seven participants:
1. survey questions - demographic, professional role and previous experience with reflective writing;
2. assignments - reflections prepared using the RF and supporting statements which accompanied the learning objects;
3. interviews with seven participants and the subject lecturer.

Survey responses were used to prepare semi-structured interview questions.

To date the following analysis of the data has occurred:
1. Survey data has been entered on a spreadsheet;
2. Reflections have been coded in hard copy to develop a hierarchy and categories for coding in NVIVO;
3. Levels of reflection have been coded using NVIVO;
4. Content coding analysis has begun.

All participant interviews have been transcribed.

The plan for 2008 is to have a draft of my thesis by the end of this year! Here is the list of milestones:
Autumn Session 2008
31 January - data analysis - all codes checked for assignments and investigation of relationships between data commenced, themes and patterns investigated.
29 February - interviews coded. Methodology written up.
31 March - interview codes checked and relationships, themes and patterns extracted.
30 April - results ready to write up more formally from project notes.
31 May - results completed.
30 June - literature review added to and compiled for final thesis. Reference list continues to be added to.

Spring Session 2008:
31 July - Discussion of results commenced.
31 August - Discussion continues to be written.
30 September - Discussion completed.
31 October - Introduction commenced.
30 November - Introduction continues.
31 December - Introduction completed, draft thesis.

For the next meeting with my supervisors, I need to have finished the code checking of the reflections - levels of reflection and also content coding if possible AND prepare a short synopsis. The annual report has been completed and signed off. Still waiting to find out if an ethics progress report is required. This year we will be having fortnightly meetings to keep the progress happening. Each day I need to spend a concentrated amount of time on the work.

In preparation for the 3/12 write up time in the second half of the year, I need to organise my work life carefully so any projects are neatly tied up when I go on leave apart from the research project which I can keep ticking over.

Free nodes and more categories

Analysis of the data is going to be around not only levels of reflection but also categories and sub-categories illustrated in the mind map here. For example, multimedia design, challenges, professional. These were some of the words which jumped out at me when I was working through the participants' reflections. They were mentioned frequently and in relation to words such as skills, technology, roles, goals and learning. If you click on the diagram you can view a larger size.

A month of qualitative analysis

October is the month of qualitative analysis. I have developed some codes to use in NVivo on the levels of reflection and have set up tree nodes using the software. I also had a play with free nodes. NVivo uses a hierarchy system with tree nodes.

I found it easiest to create a concept map first of the main categories which came up when I worked through the data in hard copy e.g. learning, MM design, professional, teaching, ed tech tools, challenges, readings. Watch this space!

The intention at present is to focus on the levels of reflection in five categories:
1. descriptive reflection - superficial;
2. explanatory reflection - with rationale;
3. supported reflection - with evidence;
4. contextual reflection - different perspectives;
5. critical reflection - applied.

The levels of reflection apparent in the data don't necessarily directly match the categories in the three-step reflective framework though I have incorporated them. So far, I am working through the documents and coding levels of reflection in NVivo. Once this is done for one data set, I will code the other categories and sub-categories. All the reflections are now coded with codes for levels of reflection. I have to run some coding queries and see how the text I have coded looks regarding levels of reflection. I also have to figure out how to measure the instances of reflection and range of levels. At the moment the highest levels of reflection - contextual and critical are not very common. Explanatory reflection (where rationale is given) and supported reflection (evidence from theory & the literature) and descriptive reflection (no rationale) are the most common.

Some questions:

• What is critical reflection really for the purposes of this research?
• Has the reflective framework helped with critical reflection?
• Is it okay to code one paragraph or sentence with more than one code?

So far I have defined critical reflection as: Writing demonstrates awareness of multiple theories and principles and multiple perspectives – ideas which go beyond the subject, the project and the self. The application to professional practice is evident. What was learned and how it can be used is outlined in the writing. There are a few examples of critical reflection emerging from analysis of the reflections.


To do: The idea of setting up a matrix to look for relationships is something I need to explore further.

The next thing to do is run some queries on the reflections using the free nodes I have set up. Plus explore them to make sure I have coded thoroughly. some of the free nodes will become tree nodes eventually. I am still not entirely sure about the hierarchy in this system.

NVivo tutorials
Luckily I found some free tutorials on the web for NVivo as the tutorials which come with the software are pretty slim pickings. They can be found from Lyn Richards home page. She has spent years running NVivo workshops and has developed some pretty comprehensive tutorials.

Research articles
When pondering where to code goals as a sub-category in NVivo, I found an interesting looking conference paper called: Nicholls, H. (2003). Cultivating “The Seventh Sense” – metacognitive strategising in a New Zealand secondary classroom. International Education Research Conference AARE - NZARE, Auckland, New Zealand. Available at: http://www.aare.edu.au/index.htm as a pdf

I will read and annotate as it may help me get some clarity about the whole coding thing.

self-regulation and teacher feedback

It is a while since I made an entry. At present I am busy collecting data via interviews and written reflections. One of the things I am interested in, apart from how the three-step reflective framework was used, is how participants have been reflecting in their studies and professional lives normally and how they may have changed their approach since using the reflective framework. One of the variables in the reflective process may also be the amount of feedback they were given on their assignments.

As part of the interview, I asked participants whether lecturer feedback was helpful in helping them reflect - so far they have said yes. Then I was listening to a keynote from the REAP assessment conference I found some interesting material which suggests how important feedback is in the process of reflection. In the keynote called: Principles of good assessment and feedback: Theory and practice, David Nicol, University of Strathclyde- the emphasis was on self-assessment and feedback. The following statement relates closely to the way in which teacher feedback can assist reflection:

..."the act of using teacher feedback implies that self assessment is present. To use teacher feedback students must decode the message, internalise it and use that feedback to make judgements about their own work and then act on those judgements – but this already implies internal processes of comparison and self-assessment. Hence self-assessment is actually embedded in the use of teacher feedback. And if the use of feedback involves active self-assessment would it not be better if we strengthened this capability rather than only focus on improving teacher feedback. So one question for me was: How can we scaffold students’ learning so that they become better at self-regulation?". In the article accompanying the keynote I found some principles which were linked to feedback and reflection.

Four principles relate to feedback as an assessment tool. But note that only one specifically mentions reflection. I would suggest that feedback of any kind stimulates some sort of reflection anyway.

The principles 3 to 6 are as follows:
3. Deliver high quality feedback information that helps learners self-correct.
What kind of teacher feedback do you provide – in what ways does it help
students self-assess and self-correct?
4. Encourage positive motivational beliefs and self-esteem.
To what extent do your assessments and feedback processes activate your
students’ motivation to learn and be successful?
5. Encourage interaction and dialogue around learning (peer and teacherstudent.
What opportunities are there for feedback dialogue (peer and/or tutor-student)
around assessment tasks in your course?
6. Facilitate the development of self-assessment and reflection in learning.
To what extent are there formal opportunities for reflection, self-assessment or
peer assessment in your course?

Some of the resources associated with the REAP project can be found on the project site.

This self-assessment idea and feedback also relates to e-Portfolios and blogs - think about how we reflect on what we have written when someone gives us feedback on our blogs. Feedback on e-Portfolios, however, could be a more spurious thing because they are rarely set up to receive dynamic and ongoing feedback - generally the feedback is more finite i.e. whether you get a pass grade or the promotion, or the practising certificate or the job for which you created the e-portfolio. So the question is..the reflective framework is designed to assist people reflect as they prepare evidence for an e-Portfolio or blog, but how can it assist in the feedback process...this may be something I need to look into further....feedback well positive feedback can be very motivating. Negative feedback on the other hand can drive us underground or into retaliation...ah but that is another story.

Does feedback on a reflective blog help people reflect even further and alter the course of their learning?