An introduction to educational research methods. Введение в образовательные исследовательские методы Білім беру-зерттеу әдістеріне кіріспе

Reliability And Validity In Qualitative Research By Teacher Researchers

331

or other potential sources of evidence of the processes of learning or teaching. Furthermore, 

these phenomena are often not taken as given (i.e. they are not seen as pre-existing realities 

waiting to be uncovered), but as complex, developing, multifaceted, intertextual and largely 

dependent on the process of interpretation used in the investigation. In this way, one cannot 

divorce a discussion of qualitative methodology from the epistemological assumptions that 

underpin it. The epistemological paradigm on which qualitative research is predicated rejects 

the positivist view of knowledge and instead sees empirical data as being the product of 

‘multiple constructed realities’ (Lincoln and Guba, 1985: 295). This is why it is important to 

distinguish between qualitative or quantitative research design and qualitative or quantitative 

research  methods  since  it  is  quite  acceptable  for  a  qualitative  study  to  include  the  use 

of ‘quantitative instruments’ such as questionnaires, or tests, but the ultimate purpose of 

their use is as a contribution to understanding the particular constructions, beliefs and 

understandings of the subjects being researched. The range of instruments most frequently 

used to elicit relevant qualitative data consists of interviews (of which there are several 

types, serving different purposes), lesson observation notes and recordings, diaries (or other 

forms of written narrative), as well as information captured through non-verbal means.

Despite this focus on the particular, the ‘constructed’ and the context-dependent, qualitative 

research in education also seeks to produce knowledge, hypotheses, theoretical frameworks 

or case studies which will contribute to the systematic understanding of aspects of the 

processes of education.

One can infer from my above summary that qualitative research can be grouped under 

two broad categories: observational and introspective. Observational perspectives within a 

research design involve investigating phenomena which occur under ‘natural’ conditions, as, 

for instance, in the case of a teacher examining aspects of interactional dynamics between 

pupils  working  on  group  tasks  in  lessons.  Unlike  the  conditions  of  quasi-experimental 

research, which usually involve some specially designed set up, with a control group acting as 

a comparison, a qualitative approach would be to examine the phenomenon as it naturally 

occurs in lessons. This is why the term ‘qualitative research’ is often used interchangeably 

with the term ‘naturalistic research’. The emphasis of introspective research, most commonly 

based on the use of interviews, is an examination of how individuals or groups make sense 

of or conceptualize phenomena related to teaching or learning. In this case, the researcher 

can be dealing with two layers of evidence: the informant’s verbalized introspections and the 

phenomena in question.

Let us look at two hypothetical examples of qualitative studies conducted by classroom 

practitioners at an early stage of their career: one observational study and one introspective. 

We should note, of course, that it is often advisable to combine observational and introspective 

strategies within the same enquiry, as data from both sources can be complementary, but it 

is nevertheless important for the practitioner research to be clear which perspective is the 

focus of their study.

Reliability And Validity In Qualitative Research By Teacher Researchers

332

Example A

John Smith is a PGCE student teacher who specializes in the teaching of geography at

secondary school level and who has begun his main professional placement at a mixed

comprehensive school. As part of his training, John is required to carry out an indepth

subject studies investigation into some aspect of his school-based experience.

The guidelines for the assignment require him to engage in a study that serves three

distinct purposes: pedagogical, intellectual and developmental. What this means is

that the study should be useful in terms of ultimately contributing to improved teaching

in the area selected; it should also explore a legitimate theme which has been

discussed by others within the subject and/or educational literature; and the experience

of the research should be integral to the developmental training and not just an

artificial add-on. How can a qualitative study serve these purposes? Following a discussion

with his faculty tutor and school mentor, John decides to conduct an enquiry into

teachers’ and Y7 pupils’ perceptions of the Key Stage 2/3 transition with regard to the

teaching and learning of geography.

Over a period of a few weeks, John carries out individual interviews with the head of

department and two other teachers of geography at his school. He also interviews five

Y7 pupils individually and a focus group of six other Y7 pupils. He also teaches a Y7

geography class and keeps an ongoing record of relevant data emerging from this.

Example B

Jane Collins is in her second year of teaching at a specialist language college, teaching

French and Spanish to children across the 11–18 age range. She has registered on a

Masters course aimed at teacher researchers at her local university and needs to design

an empirical study based on some aspect of her current practice. Again, the rationale for

the choice of topic is that it should be an appropriate intellectual investigation of an

issue raised in the relevant literature, and contribute to an informed development of

some aspects of the teacher’s professional work. So the choice of topic will normally be

linked to a problem or issue which the teacher has encountered within their own 

classroom

teaching or other sphere of work within their school. Jane has found in her experience

of language teaching that target language interaction takes place predominantly in

teacher–whole class mode. Group or pair work interaction has proved difficult to fit into

her normal work scheme schedule, but there have been signs that the pupils would be

eager to interact amongst themselves in the target language, given the right conditions

and support. So with appropriate preparation and task design, Jane introduces 

opportunities

for small group work within her lessons with a Y10 French class over several

weeks. What interests her is how the pupils collaborate in learning through the group

interaction, the quality of the French produced, and when and why the pupils code

switch from French to English and back again. The interactions of each group are recorded 

and an analysis of the transcripts is based on a separate coding framework for the three 

Reliability And Validity In Qualitative Research By Teacher Researchers

333

areas of focus.

These two examples of qualitative studies may well provide the teachers conducting 

them with valuable anecdotal information and insights regardless of the quality of the 

study as an instance of systematic research. However, this value is largely hit-or-miss 

and will not normally be of much use in contributing to the body of knowledge on the 

subject. In order to measure the quality of the research, two criteria are available to the 

educational researcher: reliability and validity. It is worth noting that to some extent these 

are conceptual labels, each enclosing a variety of different types of measurement, not all 

of which are applicable in the qualitative context. What follows is a description of those 

which are applicable in this context, and their value will be illustrated where relevant 

through reference to the two examples described above.

RELIABILITY

The notion of reliability refers to the rigour, consistency and, above all, trustworthiness of 

the research. In this sense, reliability is ‘a precondition for validity’ (Lincoln and Guba, 1985: 

292). If a study is unreliable, then it cannot be said to be valid, if by the latter (as we shall 

see in the next section) we mean the extent to which a study measures what it intends 

to measure. However, the reverse does not hold: a reliable study can conceivably still be 

invalid, as in the case of research which is rigorous and trustworthy in its procedures but 

ends up answering a different set of questions to the ones it claims to do.

Given the importance of the construct, how can we estimate whether a research study is 

reliable or not? As Denscombe (2010: 213) notes, in qualitative studies, this is commonly 

done by asking the following question in relation to a particular study: ‘If someone else 

did the research would he or she have got the same results and arrived at the same 

conclusions?’ This operation therefore consists of holding the research as a constant 

variable and substituting the researcher, so that if the results turn out to be the same, we 

can eliminate the possibility of unreliability due to researcher bias. More specifically, there 

are two ways in which such a reliability test can be carried out. If ‘someone else’ (more 

usefully described by Nunan (1992: 17) as ‘an independent researcher’, since if it were a 

friend or colleague then there might be a transference of the initial bias) carried out a 

replication study in the same or different context and reached the same conclusion, this 

would verify the ‘external reliability’ of the original study. If the independent researcher re-

analysed the data from the original study and reached the same conclusions, this would be 

an indication of the ‘internal reliability’ of the research.

However, while these operations are feasible in the context of quantitative research, they 

are much more problematic when applied to qualitative studies. While checking internal 

reliability is feasible as a procedure, it is limited in the context of qualitative research. 

Confining one’s judgement exclusively to the evidence of re-analysis is to focus solely on 

the data produced and to ignore everything that led to the output. As we have

seen, process and product are intricately linked in qualitative research, and an assessment

of the latter cannot be made without also taking account of the former. Establishing

external reliability is, on the other hand, totally problematic. It is obvious that the

chances of a qualitative study ever being repeated in the same context using the same

informants and procedures by an independent researcher are remote or even, some

Reliability And Validity In Qualitative Research By Teacher Researchers

334

would say, impossible, since the context-dependent conditions of qualitative research

are constantly subject to change: time passes, situations change, pupils are exposed to

new learning and experiences, and so on. An exact repetition, which logically is what is

needed for verification, thus becomes impossible. Replication based in a different context

with different pupils would introduce even more new, extraneous factors, potentially

influencing the findings.

A further problem relates to the fact that qualitative research (particularly in the

case of an ethnographic study carried out by a teacher researcher) needs to

acknowledge the role of the researcher in the research: this role is not merely an

instrumental or procedural one but constitutes an integral part of the findings themselves,

as interaction between the researcher and the researched is seen ‘as an

explicit part of knowledge production’ (Flick, 2002: 6). In the case of action research

by teachers, it is also quite likely that the responses and behaviour of pupils (or for

that matter colleagues) being researched will consciously or subconsciously be influenced

by the fact that the person interviewing or observing them is their own

teacher (or colleague).

So replicability as a criterion for measuring the reliability of a qualitative study is here

usually confined to the realm of the virtual or the hypothetical. The importance of the

criterion therefore shifts to the strategies and processes that the researcher puts in

place to allow such a hypothetical replication of his or her study.

In John’s case, there are three main ways in which he can be explicit about his processes

so that others can (in theory) do a repeat performance.

Firstly, it is important to situate the study within the context of an existing wider,

intellectual and professional debate. John needs to provide a clear account of the aims

and objectives of his research. These should not merely be listed, but the rationale

behind them should be presented and linked to prior discussion in the literature. The

latter point is important for the purpose of reliability because it allows the researcher

to connect with ideas, arguments, theories and the research of others and uses aspects

of this shared resource and shared understanding as a basis or starting point for the

individual study. In so doing, it allows for scrutiny of the aims and objectives, at least,

on a more generalized and ‘objective’ scale. This connection could be made in different

ways, including, for instance, by alluding to the literature on relevant educational policy-

making such as Ofsted’s (2003) view that secondary schools fail to ensure that their

‘pupils build on what they learned at primary school’. In addition, John’s aims and

objectives can link to prior analyses of pupils’ perceptions of learning geography at KS2

and KS3.

Secondly, the reliability of a study depends on the degree of transparency of research 

rationale and decisions guiding the selection of the research sample. John needs to be 

explicit about how his research was undertaken. This information would refer to the 

selection of the pupil sample as well as to the educational profiles of the people involved. 

The rationale for the sample selection needs to be given: why were the pupils chosen 

from one Y7 class? Did the group include pupils who had gone to different primary 

schools? Why was this important as opposed to selecting pupils with the same primary 

school history? What was the basis for choosing to interview those teachers in particular?

Finally, reliability depends on the accessibility of data collection and analysis procedures. 

Reliability And Validity In Qualitative Research By Teacher Researchers

335

These will involve keeping transcripts of interviews and records of lesson observation and 

other fieldwork notes. Questions used at interview need to be justified in relation to the 

specific study aims. This is important, even though in qualitative studies, especially when 

interviewing children, one is likely to use semi-structured interviews which are flexible and 

allow individual interviewees the scope to follow their own thread of thinking rather than 

being constrained by closed, structured questions.

VALIDITY

Validity too is an umbrella term which encompasses a variety of different qualities that have 

been applied to educational research. As with the concept of reliability, a study’s validity is 

often identified through reference to the following binary contrast: external validity and 

internal validity.

External validity has been defined as the extent to which the research design succeeds in 

allowing one to ‘generalize beyond the subjects under investigation to a wider population’ 

(Nunan, 1992: 17). In other words, external validity is an indication of the generalizability or 

‘transferability’ (Miles and Huberman, 1994: 279) of the study’s findings. In quantitative studies, 

researchers use randomized samples from a given population in order to strengthen the 

generalizability claims of their findings. However, given that qualitative research is essentially 

context-bound and ethnographic, external validity is generally viewed as inappropriate 

since it contradicts the epistemological and methodological perspectives in this approach. 

However, Lincoln and Guba (1985: 301) suggest that ‘thick description’ should be provided 

‘to enable someone interested in making a transfer to reach a conclusion about whether 

transfer can be contemplated as a possibility’. The term ‘thick description’, popularized by the 

anthropologist Clifford Geertz who borrowed it from the philosopher Gilbert Ryle, refers 

to the close analytical account and interpretation of the ‘structures of signification’ (Geertz, 

1973: 9) inherent in events and social behaviour.

Internal validity, on the other hand, is highly applicable to the qualitative context, and it is very 

important for the researcher to consider since in its most fundamental form, it refers to the 

extent to which a study ‘actually investigates what it purports to investigate’

(Nunan, 1992: 14). In other words, this quality refers to the ‘authenticity’ or ‘credibility’

of the findings (Miles and Huberman, 1994: 278). Checking internal validity involves different

strategies, as we shall see, but the key element is that of evidence of a persuasive

connection between the conclusions made in the outcome of the study, and the procedures

and methodology used in collecting and analysing the data. Maxwell (2005) usefully

warns the researcher against the automatic use of strategies to offset threats to the

study’s validity. Masters students sometimes describe an inventory of strategies they

have used, quoting authorities from the literature but without reference to the specific

threats within their own study, as if this automatically endorses the internal validity of

the research. ‘Validity threats are made implausible by evidence not methods’ (Alexwell,

2005: 105), and therefore it is important to identify the potential areas under threat in

the proposed study first, and then to implement appropriate strategies to counter them.

The methodology chapter in the report (or thesis) should discuss that connection

explicitly, and should be honest about the extent to which the strategy is successful in

reducing the particular threat.

What follows is a selection of strategies which are particularly applicable in the context

Reliability And Validity In Qualitative Research By Teacher Researchers

336

of school-based teacher research, and illustrated through reference to the study described

earlier in Example B.

Long-term involvement

This is one strategy where the teacher researcher is at an advantage over the external

researcher. The purpose of this strategy is to overcome weaknesses, errors of judgement

and patchy knowledge of the field due to the researcher’s status as an outsider,

and therefore it will enhance the credibility of the study’s findings. However, when

teachers are researching aspects of their own practice at their own school, an ethnographic

‘feel’ for the context in which the data is generated, the contextual background

of the research, can be more easily achieved.

Why is it necessary?

Lincoln and Guba identify two aspects of long-term involvement (1985: 301). The first

they call ‘prolonged engagement’ which serves the purpose of making the researcher

‘more open to the multiple influences of the phenomenon being studied’. In the case

of Jane Collins’s study (Example B), the researcher could rightly claim that she has a

good knowledge of the pupils’ attitude and performance in French since she is teaching

them over the academic year, and may have taught some at least in her previous first

year at the school. She also has access to information relating to the pupils’ personalities,

behaviour and general academic performance through other colleagues and extracurricular

contact. This first-hand contact developed over time will help to build trust

between Jane and her pupil-subjects, and will allow her to understand the ‘culture’ of

the school and how to interpret it. It is unlikely that the validity of her research will be

threatened by insufficient knowledge in this area.

The second aspect of long-term involvement which is identified by Lincoln and Guba is 

what they call ‘persistent observation’ and this does require proactive planning by Jane. The 

authors argue that this strategy serves the purpose of identifying ‘those characteristics and 

elements in the situation that are most relevant to the problem’ being investigated.

How can the threat be reduced in Jane’s study?

Jane’s study involves the analysis of small group interaction on pupils’ target language use. The 

purpose is to see how this kind of learning situation, to which the pupils are not accustomed, 

generates their use of spoken French. Insufficient opportunities for pupils to engage in such 

tasks will not lead to valid findings: two or three lessons over the term could produce results 

which may be invalid because insufficient time has been allowed for the pupils to overcome 

their lack of familiarity with the learning task and situation. Pupils who are reluctant to speak 

French in whole-class situations may need time to adapt to group interaction. As well as 

its importance in allowing her pupils to adapt to their new learning situation, the strategy 

of ‘long-term engagement’ will help Jane to develop a clearer research focus through her 

preliminary observation of the interactions. She can do this through a pilot phase which 

serves the purpose of identifying what features of the pupils’ interaction seem to be the 

most salient, what the obstacles are to her investigation and where the promising kinds of 

evidence in terms of her research aims seem to arise. In this way, the specific focus of Jane’s 

observation and of her eventual analysis emerge from the research itself.

Reliability And Validity In Qualitative Research By Teacher Researchers

337

жүктеу/скачать 32,4 Mb.

Достарыңызбен бөлісу: