ASCD / www.ascd.org 1
Chapters 7 and 8 introduced a variety of viable data collectiontechniques. However, employing proven techniques doesn’t guaranteethe quality of the findings that emerge. The reality is, action researchsimply isn’t worth doing unless it is done well. Although that may soundlike just an old refrain, it is far more. The imperative for maintaininghigh standards of quality is a truth learned and sometimes painfullyrelearned by teacher researchers. There are three fundamental reasonswhy you as a teacher researcher should hold yourself to the highestquality standards possible:
1. Your obligation to students2. The need for personal and collective efficacy3. The need to add to the professional knowledge base
The first reason, your obligation to students, rests on the premisethat the education of the community’s young is a sacred trust placedupon you as a educator. Therefore, the decisions you make on behalf ofstudents are actions of no small consequence. No one, least of all teach-ers, would wish to see students victimized by malpractice. When youmake teaching decisions on the basis of sloppy research, you place yourstudents at risk.
A second reason to hold your action research to the highest stan-dards of quality is that understanding your influence on educational out-comes can significantly enhance your personal and collective feelings ofefficacy. However, before you can take credit for the success reflected inyour data, the quality of that data must withstand the scrutiny of theworld’s most critical jury—your own skeptical mind. Ultimately, if youdoubt your own conclusions regarding the contribution you have made
109From Guiding School Improvement with Action Research by R. Sagor. © 2000 by ASCD. Reproduced with permission. All rights reserved.
2 ASCD / www.ascd.org
110 Guiding School Improvement with Action Research
to your students, those findings won’t have much impact on your feelings of self-worth.
The third factor, adding to the knowledge base, may not seem im-portant if you are a teacher researcher practicing in a one-room school or you find yourself in a school culture that emphasizes individualism. However, it should be extremely relevant to the vast majority of teach-ers—those of you who tend to share what you’ve learned with your col-leagues. Not infrequently, one of the unspoken reasons for conducting action research is to persuade or entice your skeptical colleagues to con-sider “your” perspective on an issue. When you present your research to peers who are skeptical about the theory you are following, you should expect a similar skepticism about the research findings you produce con-cerning those theories. If your pedagogical opponents can find fatal flaws in your action research data, all future efforts at persuasion become that much more difficult.
The criteria used to establish the quality of action research should be no different from those used with other forms of research. Topping any re-searcher’s list of quality criteria are the twin pillars of science: validity and reliability, first introduced in Chapter 1. These concepts are so criti-cal to the quality of action research that it is worth taking some time to discuss and explore each of them.
As you no doubt recall from Education Psychology 101, validity re-fers to the essential truthfulness of a piece of data. By asserting validity, the researcher is asserting that the data actually measure or reflect the specific phenomenon claimed. Scientific history is full of examples of re-search findings that were discredited because they were shown to lack validity.
A mercury thermometer is an example of a valid instrument yielding valid data. The height reached by the fluid in an accurate thermometer is a valid and appropriate measurement of air temperature. Similarly, the movement of a membrane in a barometer is an appropriate and valid way to determine barometric pressure. A ruler can be a valid way to measure length, and unfortunately (for those of us who are weight conscious) a bathroom scale can be a valid measure of weight.
ASCD / www.ascd.org 3
Nothing has helped me understand the importance of attending tovalidity as much as my experience with performance assessment. One ofthe great accomplishments of the modern assessment movement hasbeen drawing teacher attention to the value of authentic work products.Although bubble-sheet tests can, in many cases, produce valid data,teachers’ preference for authentic work products is understandable. It isanalogous to historians’ preference for “primary source material” over“secondary source material.” Intuitively, we all know that words fromthe horse’s mouth are more believable than words related by the horse’strainer. Similarly, a piece of actual student writing has more validitythan a score obtained on the language section of a standardizedmultiple-choice exam. A performance by the school band is a better in-dicator of students’ ability to execute a musical piece than are the stu-dents’ grades in band.
However, even given the deserved popularity of performance andportfolio assessments, these types of data are not exempt from concernsregarding validity. For example, how should we react to the use of a writ-ten lab report as a means to assess student understanding of the scientificmethod? Should a lab report written in standard English be accepted as avalid indicator of a student’s understanding of science?
Suppose you answered yes. Would you still accept that lab report as avalid indicator if you learned that the student lacked fluency in English?Probably not. This is because the English-language proficiency neededto complete the report introduced what scientists call an intervening andconfounding variable. In the case of assessing the proficiency in science ofa student with limited English proficiency, the written aspect of the re-port intervenes and thereby confounds the accuracy of the assessment.Intervening and confounding variables are factors that get in the way ofvalid assessment. This is why when conducting assessments on studentlearning and collecting data for action research, it is important to ask:
Are there any factors or intervening variables that should cause me todistrust these data?
Reliability is a different but no less important concept. Reliabilityrelates to researchers’ claims regarding the accuracy of their data. A fewyears ago, when a police officer issued me a ticket for speeding, I didn’tquestion the validity of his using an expensive, city-issued speedometer.I was willing to concede to the officer the validity of measuring vehicular
Data Collection: Building a Valid and Reliable Data Collection Plan
111
4 ASCD / www.ascd.org
• Is this information an accurate representation of reality?• Can I think of any reasons to be suspicious of its accuracy?
To appreciate the concepts of validity and reliability and how you mightestablish them, consider how you would behave as a juror deliberating ina criminal trial. Lawyers for both sides would argue their cases as persua-sively as possible. Your task as a juror is to determine which of the argu-ments to believe. In deciding if a lawyer had “proved the case,” youwould probably ask these questions regarding validity: Are these claims
112 Guiding School Improvement with Action Research
speed with a speedometer. However, I urged him to consider my thesis regarding the reliability of his speedometer. I respectfully suggested that although I knew he sincerely believed that his speedometer was accu-rate, he ought to consider the possibility that it could be damaged. I ar-gued that if it were broken it wouldn’t produce an accurate, credible, and reliable measure of my speed. What I was suggesting was that although speedometers are valid measures of speed, they aren’t always reliable.
Unfortunately, I lost that argument. I fared no better when I pre-sented the same “reasonable doubt” plea to the judge. Unbeknownst to me, the state police regularly establish the reliability (accuracy) of their speedometers by testing the speedometer on each patrol car every morn-ing. In the end, I had to pay the fine. But in return I learned a memorable lesson on the value of establishing reliability.
Reliability problems in education often arise when researchers over-state the importance of data drawn from too small or too restricted a sample. For example, imagine if when I was a high school principal I claimed to the school board that I had evidence that the parents love our school’s programs. When the board chair asked me how I could make such a claim, I responded by defensively asserting it was a conclusion based on “hard data”—specifically, a survey taken at the last winter band banquet. The board chair might respond that because that event was at-tended by only 5 percent of the school’s parents and all the parents who attended had one thing in common—they had children in band—my conclusions were “unreliable.” He would be right. Claiming that such a small and select sample accurately represented the views of a total popu-lation (all the school’s parents) stretches the credibility of my assertion well beyond reasonableness.
To enhance the reliability of your action research data, you need to continually ask yourself these questions when planning data collection:
ASCD / www.ascd.org 5
credible? Can I truly believe that this evidence means what these wit-nesses and lawyers say it does? To determine the reliability of the evi-dence, you would ask questions such as these about the accuracy of thewitnesses’ recollections and testimony: Can I trust the accuracy of theireyes and ears? Could time or emotions have played a trick on theirmemories?
So how do legal “researchers”—defense lawyers and prosecutors—convince a jury of the essential truth and accuracy (validity and reliabil-ity) of their cases? They do it through the twin processes of corroborationand impeachment. When they want the jury to believe what one of theirwitnesses said, they bring in other independent witnesses. If an addi-tional witness corroborates everything the first witness said, it increasesthe confidence a juror will have in the initial testimony. The more inde-pendent pieces of evidence a lawyer can place before a jury, the more thejurors will trust the truthfulness and accuracy of the claims. Conversely,if lawyers want the jury to doubt the truth and accuracy (validity and re-liability) of the other side, they try to impeach (challenge the credibilityof) the testimony of the other side, by, for example, entering into evi-dence alternative or irreconcilable reports on the same phenomenonfrom several independent sources.
Action researchers use a similar process to that used by lawyers. It iscalled triangulation, and, as was discussed in Chapters 1 and 2, it involvesthe use of multiple independent sources of data to establish the truth andaccuracy of a claim.
There are ways to develop valid and reliable instruments withouttriangulation, but these methods are often problematic. First, they aretime-consuming and frequently prohibitive in terms of cost. This is be-cause significant field-testing is required to establish the validity and re-liability of a measuring instrument. Just consider the many millions ofdollars invested by publishers to support the validity and reliability oftheir standardized tests. But even if teachers were willing to invest thetime, money, and energy required to establish technical validity (con-struct and content) for their home-grown instruments, they probablywouldn’t be happy with what they produced.
For good reason, educators are intuitively unimpressed with “singleinstrument measures.” They tend to question whether any single toolcould ever capture the full reality of any meaningful educational out-come. Occasionally I will meet a layperson who believes that SAT scores
Data Collection: Building a Valid and Reliable Data Collection Plan
113
6 ASCD / www.ascd.org
Is Sagor HighSchool a school?good
Parent Surveys
Student Surveys
Teacher Surveys
GraduateFollow-Ups
CollegeAdmissions
SAT Scores
Drop-Out Rates
114 Guiding School Improvement with Action Research
alone (or another piece of seemingly compelling data, such as college ad-missions data or discipline referrals) provide an accurate picture of a school’s quality. But I have never met a knowledgeable educator who is willing to make a judgment based upon any of those same valid and reli-able instruments. This is because educators know that what these “valid and reliable” instruments reveal is simply too narrow to justify conclu-sions regarding educational quality.
This is not to say that these instruments (SAT scores, college admis-sions, discipline referrals, and so forth) aren’t valuable windows into the larger phenomenon (the quality of a school), but before conclusions can be drawn about the big picture, those findings need to be corroborated by looking at the phenomenon through a variety of other windows.
Figure 9.1 illustrates what a plan for triangulated data collection might look like to answer a question on the quality of a high school.
Although we might be skeptical about drawing conclusions regard-ing a school’s quality from any one of the success indicators in Figure 9.1, if all of these instruments painted a similar picture, we would, no doubt, feel confident in declaring the school “good.”
FIGURE 9.1
A Plan for Triangulated Data Collection
ASCD / www.ascd.org 7
Chapter 6 presented guidelines for producing a written problemstatement/research proposal (Implementation Strategy #6). The sampleproposal written by Richard and Georgia, although short, contained allthe items expected from a formal research proposal except the data col-lection plan. Chapter 2 described the triangulation matrix as a helpfulplanning tool (Figure 2.3, p. 21). Figure 9.2 shows the triangulated datacollection plan, in the form of a matrix, that Richard and Georgia usedto answer their research questions. Implementation Strategy #10 canhelp you complete a triangulation matrix.
Data Collection: Building a Valid and Reliable Data Collection Plan
115
1. Could wemotivate our 8thgraders to con-duct and com-plete RealWorld Advo-cacy Projects?
Teacherjournals
Student surveys Grade bookrecords
2. What wouldbe the quality ofthe projectsproduced byour students?
Teacher assess-ments using aproject rubric
Student self-assessmentsusing the samerubric
Assessment bycommunitymembers usingthe rubric
3. Would thecompletion ofReal World Ad-vocacy Projectsresult in en-hanced feelingsof social effi-cacy for ourstudents?
Surveys of stu-dents’ otherteachers
Interviews withrandom sampleof students
Interviews withrandom sampleof parents
8 ASCD / www.ascd.org
Implementation Strategy #10—Building a Triangulated Data Collection Plan
WHAT:Constructing a data collection plan with high probability of producingvalid and reliable answers to your research questions
HOW:1. Prepare a four-column data collection matrix with separate rows foreach research question (see Figure 9.2).
2. Write your research questions in column 1 of your matrix.
3. For each research question, ask yourself the following: What is onesource of data that could help answer this question? Write your answer incolumn 2 next to the research question.
4. Ask the question two more times to determine a second and thirdsource of data, and write your answers in columns 3 and 4, respectively.*
5. Repeat this process for each research question.
6. Review the completed matrix and ask yourself the following question:Are these the best sources of data I/we could collect in answer to each ofthese questions? When you are satisfied with your answer to this ques-tion, you have a completed data collection plan.
*Although this strategy suggests collecting three types of data to answer aresearch question, it is perfectly permissible to collect more than threetypes.
Once you have developed a triangulated data collection plan, you haveaccomplished much of the hard work of action research. Most doctoralstudents report that the hardest aspect of completing a doctorate is get-ting a comprehensive research proposal through their dissertation com-mittee. Once the rationale for their research has been established and amethodology (the data collection plan) for answering their researchquestions has been put in place, all that is left is to carry out the proposal.
116 Guiding School Improvement with Action Research
ASCD / www.ascd.org 9
If you, alone or with colleagues, have followed the steps outlined in thisbook thus far, you are ready to proceed. Now all you have to do is carryout your plan.
Unfortunately, many beginning action researchers stall at thispoint, usually because completing the next stage, data collection, re-quires budgeting time from an already packed schedule. To get over thishurdle, it is helpful to commit to a time line and a process for completingthe work of data collection. The rationale for formalizing this commit-ment is to keep the demands of a hectic work life from getting in the wayof completing what should prove to be a most satisfying piece of work.Implementation Strategy #11 takes only a few minutes to complete, butdoing so will help ensure that you get over the time hurdle and maintainyour momentum for completing your research.
Implementation Strategy #11—Data Collection Time Line/To-Do List
WHAT:Making a commitment to a plan for completing the data collection por-tion of your action research
HOW:1. Make a four-column list on a sheet of chart paper.
2. Brainstorm (either individually or, if your research is a team effort,with your colleagues) a list of each thing that needs to be accomplishedin order to complete your triangulated data collection plan. List theseitems (roughly in chronological order) in the left-hand column on thechart paper.
3. In the second column, write the date that each should be accom-plished. Then ask yourself if it is realistic to complete this item by thatdate. If the answer is yes, go to the next item. If the answer is no, deter-mine the earliest “realistic” date.
4. If working individually, go on to the next step. If working as a team, gothrough each item on the list and determine who is willing to be respon-sible to see that this item is accomplished by the agreed upon date. Writethat person’s name in column 3.
Data Collection: Building a Valid and Reliable Data Collection Plan
117
10 ASCD / www.ascd.org
5. Ask yourself (or ask the team) the following question: What types ofsupport or help might I/we need to complete each of these items? Per-haps you will need some support from your principal or some help from aprofessor at a local university. Write the name of the person or organiza-tion whose help you anticipate needing in the last column and committo a time for making contact with these “critical friends.”
6. One last time, ask yourself or your team if this plan is realistic. If you an-swer yes, you are ready to proceed. If you answer no, repeat this strategy.
Chapters 10 and 11 explore the three remaining steps in the actionresearch process: data analysis, reporting, and action planning. Chapter12 discusses a number of important ethical and methodological issuesthat will be particularly helpful for beginning researchers. If you intendto conduct your data collection before reading the rest of this book, Istrongly recommend that you read Chapter 12 first.
118 Guiding School Improvement with Action Research
