Reflections from the workshop: Towards comprehensive evaluation for health and development: promoting the integration of evaluation methods
A workshop funded by the British Council Researcher Links programme and the EC SDH-Net consortium
By Melisa Martinez-Alvarez (LSHTM)
Mixed methods research is recognised to be essential to doing robust evaluation of complex interventions. It isn’t sufficient to know whether a programme works but also how and why it works, requiring the mixing of qualitative and quantitative and sometimes participatory methods.
The morning started with a presentation from Joanna Busza (a Senior Lecturer at the London School of Hygiene and Tropical Medicine) outlining the differences between quantitative, qualitative and participatory research methods in evaluation, in their use, data collection and analysis methods:
1. Quantitative methods are used to assess the degree of exposure of an intervention, the distribution of an intervention across population groups, to identify the associations between activities and behavioural and health outcomes, and to assess the impact of an intervention using statistical methods. Quantitative data collection uses standard approaches, through clear and easy to use instruments, where data entry must be systematic. The analysis of quantitative data is deductive, and involves testing hypotheses, conducting rigorous statistical analysis, with great emphasis on data being collected in an unbiased manner.
2. Qualitative methods are used to understand the local context, investigate community perceptions and knowledge, identify case studies, understand the relationships between providers and patients, and to provide an in-depth understanding of the story behind the “numbers”. Qualitative data collection methods are unstructured and informal, and have a high degree of flexibility, where there is a topic guide rather than specific questions (and questions can be modified during the data collection process). Qualitative data collection requires good observation and documentation, where the context and the atmosphere should be captured and taken into consideration. Qualitative methods therefore require a different skillset, using probing and creative ignorance. Qualitative data analysis tends to be inductive, with the aim of building theory rather than testing hypotheses. It requires a process of coding and identification of themes. Qualitative research sees the world as constructed and interpreted by people in specific social and cultural contexts and relies on the ability of the researcher to identify and understand the bias introduced in the study (rather than to try to eliminate it).
3. Finally, participatory methods are used to actively involve the research population in the research process. They are used to undertake health promotion and to understand community perceptions and priorities. They combine research, education and action, and support commitments of empowerment and community mobilisation. In participatory methods the process of data collection is as valued as the content, with tools often relying on visual representation.
Mixing qualitative methods in evaluation
Joanna’s talk was followed by a presentation by Jennie Gamlin (Senior Research Fellow, Institute of Global Health, UCL), who showed that mixing of methods does not necessarily need to involve quantitative methods. In evaluations, often different qualitative methods are used, depending on the question is being asked. For instance, unintended consequences can be explored through the use of narrative analysis and semi-structured interviews, comparisons (before and after and between different aspects of intervention) can be done by ranking activities through the use of pictures, photographs and photo-voice, and quality assessments, including the completion of activities, can be done through observations, guided walks and internal document revisions
Potential pitfalls of mixed methods research:
Despite the potential advantages of mixed methods evaluations, Joanna highlighted some potential pitfalls of using mixed methods, particularly when not done well:
1. Methods are not integrated, but are instead reported separately. In this case the sum will not be greater than the parts and the bigger more complex picture will be missed.
2. Qualitative research is just “added on” to illustrate findings. There are misconceptions that qualitative research is cheap, easy and quick and that anyone can do it, whereas appropriate training, planning and resources are needed to undertake rigorous qualitative research.
3. Qualitative data are quantified and therefore the findings are diluted and simplified.
4. Participatory research may be undertaken in a weak and tokenistic manner.
How to effectively mix or integrate different methods?
Joanna highlighted that very few people are actually good at both quantitative and qualitative methods, so when discussing mixing methods, it is the mixing of teams rather than research methods that is discussed. There are different strategies to mix methods, but regardless of which strategy is used, it is essential to have a plan for mixing the methods right at the beginning of the evaluation, from the development of the theory of change.
Joanna proposed two methods to integrate the different methods:
1. Qualitative exploration → Quantitative data collection → Participatory action
An example of this method would be to use qualitative methods to undertake the needs assessment and explore what are the problems that need addressing. Quantitative data could then be collected to assess how widespread and where are the particular problems identified are more prevalent, with participatory action taken to identify why and what can be done.
2. Quantitative overview → Participatory analysis → Qualitative research
Using this strategy, quantitative data could be collected to identify which problems are the most common, participatory analysis would then be used to understand the local perspective and develop a theory of change and qualitative research can then be undertaken to find out how health promotion messages can best be disseminated.
In terms of practical advice on how to mix different methods, Jennie suggested completing a framework to design a mixed methods evaluation using three headings: (i) questions, (ii) who and how, and (iii) what is needed. In addition Joanna suggested the following steps should be taken to successfully integrate different methods:
(i) Identify all stages in designing and evaluating interventions
(ii) Determine what information is required for each component
(iii) Identify who will be involved and how for each step
(iv) Decide on the order of the steps
Alternatively, Jennie described the process of triangulation of methods and people, which can recognise and explain bias, where mixed-methods research can provide a “mosaic evaluation”, with each method filling a piece of the mosaic.
Figure 1. Triangulation triangle, adapted from Jennie
There was also some discussion within the group on whether quantitative and qualitative data collection should be concurrent (at the same time) or sequential (one method after the other as in the examples above). It was agreed that either strategy would be valid, but the most important thing should be to decide on the method of integration at the design stage of an evaluation.
From her experience, Pilar recommended that quantitative and qualitative research teams keep a regular dialogue, but at the same time give each other space so they do not influence each other (a happy medium would be for teams to meet every two weeks). In addition, she recommended having a pre-defined analysis plan and for both quantitative and qualitative research leaders to have the same level of authority within the evaluation team.
Finally, Ietza Bojorquez (Professor, El Colegio de la Frontera Norte, Mexico) highlighted the importance of selecting an appropriate paradigm when mixing methods, and recommended considering qualitative paradigms, which assume reality is constructed, which was more likely to allow for the inclusion of quantitative methods than a positivist paradigm (a more quantitative paradigm) to be useful for qualitative methods.
Advantages of mixing methods
If done well, mixed methods evaluations allow for the generation of a more complete and complex picture of an intervention, where the sum is bigger than its parts. Ietza Bojorquez presented on a fascinating study on drug services in the Baja California state of Mexico, where she explored the different models of drug support between state medical facilities and centres run by the evangelical church. Ietza was asked to add a quantitative element to the study, and used a meta-narrative approach to combine the narrative from both groups. She found that by mixing both quantitative and qualitative methods, she could better engage with the different groups. This was because religious groups were more likely to be interested in the results from ethnographic methods, whereas medical professionals were more interested in the quantitative results.
Pilar Torres presented the results from a mixed methods evaluation of the Mexican national programme for the elderly (70+), undertaken by the National Institute of Public Health. The programme is an unconditional cash transfer of $66 every two months that aims to reduce vulnerability and increase living conditions of the rural old population. Two teams undertook the evaluation: one quantitative and one qualitative. The teams worked together from the outset and made efforts to share objectives and vision. Pilar showed how both methodologies were complementary and for instance, whilst quantitative methods could show how many beneficiaries were reached or the percentage decrease in depression, qualitative methods showed the method of collecting the cash transfer could be improved and that the decrease in depression was a result of empowerment of the beneficiaries within their household.
How should mixed methods research be written up?
Quantitative methods test hypotheses and are therefore usually written up using an Introduction-Methods-Results-Discussion format, whereas qualitative methods do not test hypotheses and are often written in a format of Introduction/Methods–Results/Discussion–Conclusion. It is therefore difficult to know how to write up mixed methods studies, and there is no agreed way of doing so in the literature. Some options presented by Jennie include:
(i) Follow the evaluation framework in a question and answer format.
(ii) Narrative report in the first person where the findings from the results are discussed alongside one another, with quotes and visual aids.
(iii) Validate numerical data with narrative or explain why changes have been observed by describing the context. This approach is based on the quantitative writing style and allows for the discussion of results when quantitative and qualitative methods are not in agreement.
Pilar Torres (Researcher at the National Institute of Public Health, Mexico) also advised on the importance on writing up both methods in the same report. This will not only strengthens the evaluation report, but also ensure that the qualitative results are read by policymakers and other actors who may not otherwise prioritise qualitative research methods. However, doing so can be challenging as it requires qualitative and quantitative researchers to reach agreement on their findings which may not always align. Pilar recommended involving a third party to help facilitate the integration of findings.
Useful mixed methods references
1. Morag C Farguhar, Gail Ewing, Sara Both (2011) Using mixed methods to develop and evaluate complex interventions in palliative care research. Palliative Medicine. 205(0(80) 714–81–0757
2. John W. Creswell, Michael D. Fetters, and Nataliya V. Ivankova (2004) Designing A Mixed Methods Study In Primary Care. Ann Fam Med. 2004 Jan; 2(1): 7–12.
3. David L Driscoll, Afuah Appiah Yeboah, Philip Salib, Douglas J. Rupert (2007) Merging Qualitative and Quantitative Data in Mixed Methods Research: How To and Why Not. Ecological and Environmental Anthropology. Paper 18.
4. Moore G, Audrey S, Barker M, Bond L, Bonell C, Cooper C, Hardeman W, Moore L, O’Cathain A, Tannaze T, Wight D, Baird J. Process evaluation of complex interventions. Medical Research Council guidance. BMJ 2015 350:h1258.
5. Cooper C, O’Cathain A, Hind D, Adamson J, Lawton J, Baird W. Conducting qualitative research within Clinical Trials Units: Avoiding potential pitfalls. Contemporary Clinical Trials 2014; 38:338–343.
6. O’Cathain A, Thomas KJ, Drabble SJ, Rudolph A, Goode J, Hewison J. Maximising the value of combining qualitative research and randomised controlled trials in health research: the QUAlitative Research in Trials (QUART) study – a mixed methods study. Health Technol Assess 2014; 18(38)
7. O´Cathain A, Murphy E, Nicholl JP. The quality of mixed methods studies in health services research. Journal of Health Services Research and Policy, 2008;13:92-98. DOI:10.1258/jsrp.2007.007074
8. O´Cathain A, Murphy E, Nicholl JP. Why, and how, mixed methods research is undertaken in health services research: a mixed methods study. BMC Health Services Research 2007; 7:85.
9. John W Cresswell, Vicki L. Plano Clark (2011). Designing and Conducting Mixed Methods Research. 2nd Edition, SAGE.
10. James D. Fearon and David D. Laitin. Integrating Qualitative and Quantitative Methods. Oxford handbook of political methodology. 2009
11. David L. Morgan. Integrating Qualitative and Quantitative Methods – A pragmatic approach. SAGE Publication, 2014.
12. O’Cathain A. Assessing the quality of mixed methods research: toward a comprehensive framework. In Tashakkori & Teddlie (eds). Handbook of Mixed Methods Research, Second Edition, 2010, pp531-555.
13. O’Cathain A. Mixed methods involving qualitative research. In Bourgeault I, Dingwall R, DeVries R (eds). The Sage Handbook of Qualitative Methods in Health Research. Sage Publications, 2010, pp 575-588.
14. O´Cathain A, Thomas K. Combining qualitative and quantitative methods. In Pope C, Mays N (eds). Qualitative Research in Health Care. Blackwell Publishing Ltd: Oxford, 2006, Third Edition, pp102-111.
15. Abbas Tashakkori and Charles Teddlie (2010) Mixed Methods in Social & Behavioral Research. SAGE Handbook. Second Edition.
16. O´Cathain A, Thomas K. Combining qualitative and quantitative methods. In Pope C, Mays N (eds). Qualitative Research in Health Care. Blackwell Publishing Ltd: Oxford, 2006, Third Edition, pp102-111.
17. Abbas Tashakkori and Charles Teddlie (1998) Mixed methodology: combining qualitative and quantitative approaches (applied social research method). First Edition.