Preface

The dictionary defines research as "careful, patient, systematic, diligent inquiry or examination in some field of knowledge, undertaken to establish facts or principles" (McKechnie, 1983, pp. 1538-1539). When Webster's definition of research is paired with the word, education, the results seem almost oxymoronic. Educators often make decisions at the last minute, under extreme pressure, and with more regard for what is popular or innovative, than for what is established, factual, or truthful. For many educators, any consideration of the rigors of research ended when they sold their statistics books to the used bookstore—grateful that they would never again have to "crunch numbers" or decipher data. Unfortunately, they regarded research as an obstacle to surmount on the way to an advanced degree, not an essential aspect of informing their day-to-day practice. Research was for the ivory tower types—not for "down-in-the trenches" practitioners.

We can no longer afford this shortsighted approach. One word has changed the educational landscape—accountability. Now that educators are increasingly accountable for the outcomes of their efforts, the decisions they make regarding instruction and curriculum have taken on more gravity. Good intentions and hard work no longer count. Politicians, parents, and the press want results and research can help us get them. Make no mistake; research cannot provide recipes or prescriptions. Schooling is far too idiosyncratic to respond to simplistic formulas. However, when carefully read and thoughtfully considered, quality research can inform, enlighten, and provide direction to practitioners that will save us time and money, but more importantly enhance the effectiveness of our schools and increase the opportunities for our students.

We, the authors of Making Sense of Research, are related to one another, as you may have already surmised. We are a mother-son team—an unusual writing combination to be sure. But we think you will find that our unique blend of training and experiences provides a helpful perspective as you attempt to make sense of education research. If you have not already done so, take a moment to skim our biographies in About the Authors. We bring the best of two worlds to the writing of this book—the "real" world where education is practiced daily and the "research" world where the "disciplined search for knowledge" (Smith & Glass, 1987, p. 6) is ongoing. Ideally, these two worlds would be well-connected with the key findings regarding educational policies and practices making their way straight from the pages of refereed journals to the in-baskets of education's principal stakeholders and decision-makers. However, as one writer pointed out, "The [education] research-to-practice pipeline has sprung many leaks" (Miller, 1999, p. A17). We aim to put our collective fingers in those leaks.

In our experience, the relationship between education researchers and practitioners is a tenuous, if occasionally non-existent one. Researchers are often guilty of shutting down practitioners with esoteric arguments, while practitioners are no less guilty of putting up their own smoke screens to defend their distrust of data. "When was the last time you were in a classroom? You ought to get out into the real-world," they assert. The authors have certainly been guilty of engaging in these arguments from time to time—even with each other. In a way, our authorship is a metaphor for the larger questions: How can education research improve practice? How can educators make sense of the complexities and even incongruities of research? Conversely, how can researchers tap into the rich knowledge base of educators regarding the implementation of research in authentic settings?

THE GOALS OF THIS BOOK

Our mission is set forth in the book's title. We want our readers to be equipped with the conceptual understandings they need to make sense of education research in order to achieve the educational goals of their particular classroom, school, or district. Making sense of research is not just about reading and understanding research done by others. In our opinion, making sense of research is also about doing your own site-specific, user-driven research as a way of sustaining school improvement, keeping the vision alive, and attaining your mission.

If your eyes glaze over whenever you read a research study because you lack the tools to comprehend what it means, then you are being held hostage to someone else's interpretation of the findings. If you read only the introduction and the conclusion to a research study, ignoring everything in between, you may as well not read it at all. By the time you finish reading Making Sense of Research, you will be able to judge the quality of research for yourself and have confidence in your judgment.

WHO THIS BOOK IS FOR

We have written this book for educators and educational stakeholders who want to be informed and active participant in discussions regarding curriculum, instruction, and policy. This book is for those who make decisions—from the seemingly smallest teacher-made decision regarding time allocation during reading instruction to major statewide policy decisions such as reducing class size. The following individuals will find Making Sense of Research to be helpful:

1. Concerned and conscientious teachers who recognize that their efforts are not bringing about the desired results and want to be more effective in their classrooms.
2. Principals who feel increasing pressures to bring students to mastery of national, state, or local achievement standards and are frustrated by the often haphazard way in which program decisions and choices are made.
3. Teams of teachers and administrators who are charting school improvement initiatives and need the tools to make quality decisions.
4. District administrators who are faced with large-scale budgetary and curricular decisions and need direction in the allocation of resources.
5. University professors who want their students to become well-informed and knowledgeable consumers of education research.
6. Educational consumers and policy-makers who want to base their decisions on sound research.

WHAT THIS BOOK IS NOT

Making Sense of Research is different from other books you may have purchased or read about education research in the past. It is not a book that explains how to do research. Neither does this book methodically summarize research findings regarding a laundry-list of educational innovations, instructional practices, or well-known reforms so that you won't have to do any thinking for yourself. You have listened to the "experts" for too long. It is time to do your own thinking. While we do provide a variety of interesting examples, illustrations, and case studies from current research findings, we do not provide recommendations about the best practices or the proven methods. Books of this nature become outdated very quickly with the appearance of newer programs and additional research.

Actually, one of the most important lessons we can learn from research is that all-purpose solutions do not exist. What may work for one set of students and teachers in a particular setting may not be as applicable or effective in another classroom or school. It is our belief that educators are intelligent enough to read the research for themselves, evaluate its trustworthiness as well as applicability in their own setting, and then make informed decisions. Our goal is to empower you to be confident and accountable regarding your instructional, curricular, and policy decisions in a variety of settings and job roles.

OVERVIEW OF THE CONTENTS

Chapter 1 introduces five broad questions that should change the way you read and think about research:

1. The causal question: Does it work?
2. The process question: How does it work?
3. The cost question: Is it worthwhile?
4. The usability question: Will it work for me?
5. The evaluation question: Is it working for me?

Chapters 3 and 4 will explain how to find answers to the first big question: does it work? This question has preoccupied researchers in every branch of the social sciences for many years. Their ingenuity has led to the development of numerous methods to determine whether there are causal effects of a specific treatment or policy on the students, teachers, schools, districts (or even states and countries) in which it was tested and observed. The best of these methods are referred to as experimental, and they are described in Chapter 3. Other methods, loosely referred to as quasi- or non-experimental, are examined in Chapter 4.

Chapter 5 looks at the second big question: how does it work? In order to answer this question, which concerns process rather than outcomes, we will turn our attention to qualitative data. While there is no doubt that quantitative data (e.g., achievement test scores) are an essential aspect of results-based decision-making, we will demonstrate that education research is not an "either-or" endeavor. We must also value, integrate, and utilize information that is collected through observations, interviews, and the analyses of documents (e.g., memos, letters, vision statements, student work, or teachers' journals).

Chapter 6 takes on the fourth question: is it worthwhile? Though frequently ignored in debates over education research, one must consider the costs of research-based decisions. Do the costs of a program or policy render it infeasible? Is there another, less costly means of accomplishing the same goals?

In Chapter 7 we will examine the fourth big question: will it work for me? Just because a method or program is shown to have an effect on a group of students or schools in an experimental study is no guarantee that it will work for you. We will explore how you can determine the generalizability of research to your unique setting using a series of rules-of-thumb.

Finally, in Chapter 8 we'll consider the last, but by no means the least, question: is it working for me? Although published research is an essential guidepost for practitioners, it cannot provide all of the answers once you have made a decision to act upon that research finding. The methods that characterize good research by academics are also powerful tools for evaluating implementation in your school or district. User-driven research can aid in answering the question: is it working for me?

A FEW WORDS OF EXPLANATION

As authors we debated the necessity of including chapters that define and explain (yet again) research and statistics. We have chosen to steer clear of this "kitchen sink" approach to writing about research methods. Our suspicion is that these chapters are rarely read with the attention they deserve. When they are read, the forest is completely lost for the trees by beleaguered students who scramble to memorize obscure formulas. The message of this book is not that statistics and other research methods are unimportant—quite the contrary. They are exceedingly important, but they are nothing more than tools to be marshaled in answering the questions described in the last section. If the questions are poorly understood or imprecisely formulated, then understanding a set of methodological tools (or not) is of little consequence. As a compromise, however, we include intuitive discussions of techniques used by education researchers—or tools that we wish were used by education researchers. These are complemented by short sections at the end of every chapter with suggestions for additional reading.

As you read you will note the use of the term education research rather than the more commonly used educational research. While the majority of authors who have written on this subject use the designation educational research in their book titles and texts, we prefer the example of Lagemann (2000) and Lagemann and Shulman (1999). We must further acknowledge our tremendous intellectual debt to the methodological writings of Shadish, Cook, and Campbell (2002). The newest edition of the Cook and Campbell (1979) classic has alternately inspired, encouraged, and challenged us with its crisp writing, coherent explanations, and thought-provoking questions. The reader who is intrigued by our discussion would do well to consult these volumes.

ACKNOWLEDGEMENTS

Winston Churchill captured the essence of what it means to write a book when he said, "Writing a book is an adventure. To begin with, it is a toy and an amusement; then it becomes a mistress, and then it becomes a master, and then a tyrant. The last phase is that just as you are about to be reconciled to your servitude, you kill the monster, and fling him out to the public" (Gilbert, 1991, p. 887). Before we fling our joint effort out to the reading public, we must acknowledge the contributions of others to our efforts.

I (EKM) am grateful to Patrick McEwan for writing a book with his mother. It isn't every son who would have the patience with and confidence in his mother to undertake a project such as this. I am proud of his scholarship and commitment to excellence. I also owe an enormous debt of gratitude to James Heald, my academic mentor and long-time friend. Without his confidence in me, aided by a positive recommendation to an editor friend, my educational writing career might never have been launched. Most especially, I offer my heartfelt thanks to my husband and business partner, E. Raymond Adkins, for his support, encouragement, and honest appraisal of my work.

I (PJM) am indebted to Elaine McEwan, who constantly prodded me—whether she realized it or not—to link my research endeavors to the "real world" of education practice (every education researcher should have an award-winning principal to ask the all-important question: "so what?"). I also am deeply grateful to my colleague and mentor, Henry M. Levin, who thoughtfully combines research and practice in education—and who unstintingly shares his lessons with others.

Finally, we both appreciate the frequent readings of this manuscript by our daughter and sister, Emily McEwan-Fujita. Although engaged in personal research and writing, she always took the time to offer helpful comments and suggest excellent alternatives.

For a complete list of all of Elaine's titles visit The Bookshelf.