8 Chapter 8: Putting It All Together: Understanding and Assessing Criminal Justice Research

In This Chapter You Will Learn

The steps in the research process

The structure of research articles and reports

What questions to ask and answer about each section of a research article

About external indicators of research quality

How to assess the internal indicators of research quality

How to critique a research article

Introduction

We hope your adventure through research methods has been an enjoyable one. It is now time to put the material you have learned into practice. In this chapter, we will revisit the steps in the research process covered in Chapter 1. Since the past several chapters have focused on specific research tools, it is important to once again see the big picture of research methods, so we will review the steps in the research process once again. After this, we will discuss the structure of research articles and reports so you will be prepared to read research articles. Understanding the structure of these documents will greatly assist you as you move forward as an educated consumer of research. Along with an overview of the structure of research articles and reports, a set of questions that you should ask as you read each section of a research article will be presented. The questions will serve as a helpful guide as you move forward in evaluating research documents and putting what you have learned in this book and in your class into practice.

The next two sections of this chapter will discuss how to determine the quality of a research article. In other words, how do you know if a research article is “good” and how do you go about assessing the specific details provided in research articles? This section is the culmination of your efforts throughout this semester. Now that you have the knowledge necessary to be an educated consumer of research, this section will provide you the specific tools necessary to know what questions to ask and what documentation to look for as you review and assess research articles. The sections are divided into two categories: external indictors of quality research and internal indicators of quality research. Finally, the last section of this chapter will provide an example of a research article critique. Using experimental design as the template, we will ask and answer a set of questions regarding the Minneapolis Domestic Violence Experiment (also covered in Chapter 5). The questions and answers will allow you to make an educated decision regarding the quality of the research. This will serve as a practical example of what you can do in the future as you put your knowledge of research methods into practice in both your academic and professional careers and your personal life.

The Research Process

Although this section was previously covered in Chapter 1, it is important to, once again, review the steps in the research process. Over the past several chapters, we have reviewed several specific research tools, designs, and methods. As you studied the specifics of research methods in each separate chapter, you may have lost focus of the “big picture” and how these different pieces fit together to form a comprehensive understanding of research methods. At the beginning of this chapter, it is important once again to look at the “big picture” to see how all these different tools, designs, and methods come together in the research process. When you read this information the first time in Chapter 1, you were in your first week of class and still may have been quite anxious about understanding this material. Now that you have almost completed the course, the steps should be easy to understand. What was once possibly a foggy understanding of the steps in the research process in Chapter 1 should now be crystal clear.

One of the nice things about studying research methods is it is about learning a process. Research methods can be seen as a sequential process with the first step being followed by the second step, and so on. There are certainly times when the order of the steps may be modified, but researchers typically follow the same process for each research study they complete regardless of the research topic. Very simply, a research problem or question is identified, and a methodology is selected, developed, and implemented to answer the research question. This sequential process is one of the advantages of understanding research methods, because once you understand the process, you can apply that process to any research question that interests you. In addition, research methods are the same across disciplines. So, sampling is the same in business as it is in health education as it is in criminal justice. Certainly the use of a particular method will be more common in one discipline in comparison to another, but the protocol for implementing the method to complete the research study is the same. For example, field research (discussed in Chapter 6) is used much more frequently in anthropology than in criminal justice. However, the research protocol to implement field research is the same whether you are studying an indigenous Indian tribe in South America in anthropology or a group of heroin users in St. Louis in criminal justice.

Some authors have presented the research process as a wheel or circle, with no specific beginning or end. Typically, the research process begins with the selection of a research problem and the development of research questions or hypotheses (discussed in Chapter 2). It is common for the results of previous research to generate new research questions and hypotheses for the researcher. This suggests that research is cyclical, a vibrant and continuous process. When a research study answers one question, the result is often the generation of additional questions, which plunges the researcher right back into the research process to complete additional research to answer these new questions.

In this section, a brief overview of the research process will be presented. Although you will probably not be expected to conduct a research study on your own, it is important for an educated consumer of research to understand the steps in the research process. The steps are presented in chronological order and appear neatly presented. In practice, the researcher can go back and forth between the steps in the research process.

Step 1: Select a Topic and Conduct a Literature Review

The first step in the research process is typically the identification of a problem or topic that the researcher is interested in studying. Research topics can arise from a wide variety of sources, including the findings of a current study, a question that a criminal justice agency needs to have answered, or the result of intellectual curiosity. Once the researcher has identified a particular problem or topic, the researcher assesses the current state of the literature related to the problem or topic. The researcher will often spend a considerable amount of time in determining what the existing literature has to say about the topic. Has the topic already been studied to the point that the questions in which the researcher is interested have been sufficiently answered? If so, can the researcher approach the subject from a previously unexamined perspective? Many times, research topics have been previously explored but not brought to completion. If this is the case, it is certainly reasonable to examine the topic again. It is even appropriate to replicate a previous study to determine whether the findings reported in the prior research continue to be true in different settings with different participants. This step in the research process was discussed in Chapter 2.

Step 2: Develop a Research Question

After a topic has been identified and a comprehensive literature review has been completed on the topic, the next step is the development of a research question or questions. The research question marks the beginning of your research study and is critical to the remaining steps in the research process. The research question determines the research plan and methodology that will be employed in the study, the data that will be collected, and the data analysis that will be performed. Basically, the remaining steps in the process are completed in order to answer the research question or questions established in this step. The development of research questions was discussed in Chapter 2.

Step 3: Develop a Hypothesis

After the research questions have been established, the next step is the formulation of hypotheses, which are statements about the expected relationship between two variables. For example, a hypothesis may state that there is no relationship between heavy metal music preference and violent delinquency. The two variables stated in the hypothesis are music preference and violent delinquency. Hypothesis development was discussed in Chapter 2.

Step 4: Operationalize Concepts

Operationalization involves the process of giving the concepts in your study a working definition and determining how each concept in your study will be measured. For example, in Step 3, the variables were music preference and violent delinquency. The process of operationalization involves determining how music preference and violent delinquency will be measured. Operationalization was discussed in Chapter 2.

Step 5: Develop the Research Plan and Methodology

The next step is to develop the methodology that will be employed to answer the research questions and test the hypotheses. The research methodology is the blueprint for the study, which outlines how the research is to be conducted. The research questions will determine the appropriate methodology for the study. The research design selected should be driven by the research questions asked. In other words, the research questions dictate the methods used to answer them. The methodology is basically a research plan on how the research questions will be answered and will detail:

1. What group, subjects, or population will be studied and selected? Sampling was discussed in Chapter 3.

2. What research design will be used to collect data to answer the research questions? Various research designs were covered in Chapters 4–7.

You need to have familiarity with all research designs so that you can become an educated consumer of research. A survey cannot answer all research questions, so knowing a lot about surveys but not other research designs will not serve you well as you assess research studies. There are several common designs used in criminal justice and criminology research, and brief descriptions of some of them are presented next. At this point in the semester, you should be completely familiar with each of these designs since they have been detailed in prior chapters.

Survey Research is one of the most common research designs employed in criminal justice research. It obtains data directly from research participants by asking them questions and is often conducted through self-administered questionnaires and personal interviews. For example, a professor might have her students complete a survey during class to understand the relationship between drug use and self-esteem. Survey research was discussed in Chapter 4.

Experimental Designs are used when researchers are interested in determining whether a program, policy, practice, or intervention is effective. For example, a researcher may use an experimental design to determine if boot camps are effective at reducing juvenile delinquency. Experimental designs were discussed in Chapter 5.

Field Research involves researchers studying individuals or groups of individuals in their natural environment. The researcher is observing closely or acting as part of the group under study and is able to describe in depth not only the subject’s behaviors, but also consider the motivations that drive their behaviors. For example, if a researcher wanted to learn more about gangs and their activities, he may “hang out” with a gang in order to observe their behavior. Field research was discussed in Chapter 6.

Case Studies A case study is an in-depth analysis of one or a few illustrative cases. This design allows the story behind an individual, a particular offender, to be told, and then information from cases studied can be extrapolated to a larger group. Often these studies require the review and analysis of documents such as police reports and court records and interviews with the offender and others. For example, a researcher may explore the life history of a serial killer to try and understand why the offender killed. Case studies were discussed in Chapter 6.

Secondary Data Analysis occurs when researchers obtain and reanalyze data that was originally collected for a different purpose. This can include reanalyzing data collected from a prior research study, using criminal justice agency records to answer a research question, or conducting historical research. For example, a researcher using secondary data analysis may analyze inmate files from a nearby prison to understand the relationship between custody level assignment and disciplinary violations inside prison. Secondary data analysis was discussed in Chapter 7.

Content Analysis requires the assessment of content contained in mass communication outlets such as newspapers, television, magazines, and the like. In this research design, documents, publications, or presentations are reviewed and analyzed. For example, a researcher utilizing content analysis might review true crime books involving murder to see how the characteristics of the offender and victim in the true crime books match reality as depicted in the FBI’s Supplemental Homicide Reports. Content analysis was discussed in Chapter 7.

Despite the options these designs offer, other research designs are available and were discussed in previous chapters. Ultimately, the design used will depend on the nature of the study and the research questions asked.

Step 6: Execute the Research Plan and Collect Data

The next step in the research process is the collection of the data based on the research design developed. For example, if a survey is developed to study the relationship between gang membership and violent delinquency, the distribution and collection of surveys from a group of high school students would occur in this step. Data collection was discussed in several chapters throughout this text.

Step 7: Analyze Data

After the data have been collected, the next phase in the research process involves analyzing the data through various and appropriate statistical techniques. The most common means for data analysis today is through the use of a computer and statistically oriented software. Data analysis and statistics are discussed in Chapter 9.

Step 8: Report Findings, Results, and Limitations

Reporting and interpreting the results of the study are the final steps in the research process. The findings and results of the study can be communicated through reports, journals, or books. At this step, the results are reported and the research questions are answered. In addition, an assessment is made regarding the support or lack of support for the hypotheses tested. It is also at this stage that the researcher can pose additional research questions that may now need to be answered as a result of the research study. In addition, the limitations of the study will be described by the researcher as well as the impact the limitations may have on the results of the study. All research has limitations, so it is incumbent on the researcher to identify those limitations for the reader.

Structure of Research Articles and Reports

As noted in the prior section, research methods can be seen as a sequential process as depicted in the steps in the research process. The question then becomes, what do researchers do with all the data and information they collect in their research projects? The answer is they write journal articles, research reports, and books. One of the advantages of reading research articles is their structures are usually very similar. It is helpful as an educated consumer of research to understand the common structure of research articles and reports. You will be able to easily identify these sections as you review research articles and reports in the future.

Before we discuss the structure of research articles and reports, it is necessary to mention the three main publication outlets for research articles. First, a common outlet for a researcher’s study and results is a journal article. Journal articles appear in thousands of journals covering the gamut of academic disciplines from accounting to zoology. Researchers write manuscripts based on their research projects and then submit them to journals for publication. Later in this section, we will discuss how to assess the quality of a journal. For now, it is important to recognize that journals are a critical outlet for research articles. Second, another place to find research is in reports that are published by government agencies, private companies, and businesses. For our specific purposes, this includes government documents produced by state and federal agencies such as the U.S. Department of Justice and local, state, and federal criminal justice agencies. Thousands of documents each year are published as reports with many available online. Reports are similar in structure to journal articles; however, reports are usually more detailed than journal articles, which have limited space for details such as the research methods used in the study and the results. Furthermore, reports typically do not include a literature review. Due to these similarities, reports can be evaluated in much the same way as journal articles. Third, books are sometimes used as a primary source to document a research study. Due to the magnitude of the study, a book-length manuscript is developed to document the research methodology and findings. University presses (e.g., Cornell University Press and University of Texas Press) commonly print research projects and their results as books.

Most published research follows a basic format, which will be reviewed in this section. Although the structure described next will vary slightly depending on the research methods utilized and will vary by discipline and the requirements of the publication outlet, most research reports that are published follow a similar format. Consumers of research grow accustomed to seeing the same basic information in each published research article. By knowing the structure of a research article and the general information that is included in each of the sections, research consumers can be less anxious about reading published research. This format also provides a structure for researchers to follow when writing up their research. This format includes a title, abstract, introduction, literature review, methodology, findings, discussion, conclusion, and references. Each will be described next along with questions that an educated consumer of research should ask regarding each part of a research article.

Title

Titles help research consumers quickly identify if the journal, article, or book may be of interest to them. The title should provide the research consumer with a general idea of the article topic and perhaps the main variables assessed in the study. Some evaluative questions to ask regarding the title of a research article include:

1. Is the title concise and specific?

2. Are the main variables and research subjects included in the title?

3. Is the title free of jargon and acronyms that may not be generally recognized by research consumers?¹⁰

Abstract

The abstract is typically limited to 150–200 words, depending on the publication outlet. The abstract gives a short synopsis of the article, including the topic covered, the methodology, and the major findings of the research. The abstract provides a quick way for the research consumer to get an overview of the research study to see if more in-depth exploration is warranted. Some evaluative questions to ask regarding the abstract of a research article include:

1. Is the purpose of the study clearly stated in the abstract?

2. Does the abstract include the main points of the methodology used?

3. Does the abstract include the main findings?¹¹

Introduction

In the introduction, the researcher should establish for the reader the importance of the research topic, the historical background of the problem, and the need for research in the area. For example, several strategies have been developed to reduce recidivism among convicted sex offenders (e.g., sex offender registration, community notification, residency restrictions, and civil commitment). This is obviously an important topic, and researchers want to find out which strategy works best at reducing recidivism. After the researcher has introduced the topic and its significance, the introduction should be narrowed down to the specific research project discussed in the article. Some evaluative questions to ask about the introduction include:

1. Does the researcher identify a specific problem area in the introduction?

2. Does the researcher establish the importance of the problem area?

3. Does the introduction start broadly and then narrow down to the specific research topic?

Literature Review

The literature review section provides a framework for the research by assessing the prior studies that have already been conducted on the topic. Basically, the literature review includes the pertinent prior studies that have addressed the topic of your research and have answered your research questions. The literature review provides a compilation of what is currently known about the topic under study. The literature review should also be critical and note the limitations in the prior research and identify the gaps in the literature where adequate research has not been conducted. The literature review should contain the most recent research studies on the topic as well. However, the literature review should not dismiss history, but instead should discuss the seminal studies on the topic. For example, if you are conducting a study on the factors that impact the decisions of police officers to arrest juvenile offenders, you would definitely want to include the article, discussed later in this chapter, by Piliavin and Briar: the seminal work on police discretion and juveniles.

The review of the literature may also include a description of the theoretical framework that was used for the study. The theoretical framework can put the development of the researchers’ hypotheses into context. To demonstrate the link between the theoretical framework and hypothesis construction, some publications will place the hypotheses at the end of the literature review instead of at the beginning of the methodology section. Some evaluative questions to ask about the literature review include:

1. Does the literature review include current as well as seminal citations?

2. Is the literature review critical of the prior literature by noting its limitations?

3. Have gaps in the current literature been identified in the literature review?¹⁵

Methods

The methods section is where the material discussed in the prior chapters of this text is put into action to let readers know the details of how the research was conducted. It should provide details on the methods used by the researchers to gather data and answer the research questions and/or test the hypotheses. The methodology section includes a description of the research design, data sources, sampling strategy, variables and how they were measured, and data collection instruments used. Methods sections are often longer in quantitative studies than in qualitative studies. Explicit details of the methods used should be provided, because this section of the article provides much of the material needed by the consumer of research to evaluate the quality of the article. Unfortunately, this detail is not always provided, which leaves the consumer to make assumptions about the methods and their limitations.

Depending on the publication outlet, the limitations of the research study may also be included in the methods section of the research article. If the limitations are not discussed in the methods section, then they will be reviewed in the discussion or conclusion section. Complete transparency and full candor are expected as the researcher shows the reader that he acknowledges the limitations of the research and has thought through how the limitations can impact the results. For example, if the experimental design had several participants drop out of the program between the pre-test and the post-test, then the researcher needs to discuss how subject mortality can influence the results of the study. Similarly, if the survey researcher discovers some wording issues with one of the questions on the survey, then he should address this reliability issue and how it may impact the results of the study. No research is perfect, so it is incumbent upon the researcher to state the limitations and flaws in the research and to provide the information needed by the educated consumer of researcher to decide the magnitude of the limitations and their cumulative impact on the research findings.

In the methods section, the researcher should also discuss the statistical analysis of the data to be performed and reported in the results section of the article. Although the statistical tests performed can become quite technical and complex, it is important for the researcher to present information about the statistical tests so the consumer can become familiar with the techniques used in analyzing the data. Statistics will be discussed in Chapter 9.

It is impossible to develop a single checklist of the topics that need to be addressed in a methods section because the details that need to be provided depend on the methods used. For example, the response rate only needs to be included in the methods section when a survey has been conducted. Likewise, the sampling protocol is frequently a component of the methods section, but sampling is irrelevant when a case study is completed. Specific evaluative questions about the methodology section will not be presented here, but instead will be provided when the internal indicators of research quality are presented later in this chapter.

Results/Findings

The results section is a straightforward discussion of the research findings, but it is frequently the most difficult part of a research article to understand due to the statistical tests that may be used, especially for a newly educated consumer of research. At the beginning of the results section, a discussion of the sample is provided, along with appropriate statistical description, so the reader can understand the characteristics of the research subjects. The results section presents the findings of the research and answers each research question and/or tests each hypothesis.

For quantitative studies, the statistical analysis presented in the results section is typically presented in tables, charts, and graphs as well as described in the text. The tables often provide greater detail than what is described in words and also allow the reader to follow the description of the results provided in the article. In the results section, the researchers will also state whether there were any unexpected findings or findings that have not been discovered in the prior literature. The researcher will wait until the discussion section to discuss the reasons for the anomaly in detail, but she will typically point out the interesting finding when it is initially presented in the results section.

The presentation of data flows logically and a straightforward discussion of the main findings and their significance are presented in the results section. In qualitative research, the data are organized in a manner that illustrates a line of reasoning or tells a story. Some evaluative questions to ask about the results include:

1. What findings are statistically significant? (discussed in Chapter 9)

2. Does the author answer each research question and/ or address each hypothesis?

3. If there are tables, are the highlights discussed in the narrative of the results?

4. Are the results of qualitative studies adequately supported with examples of quotations or descriptions of observations?¹⁶

Discussion

Some research articles combine the discussion section with the results section, while others combine the discussion section with the conclusion section, while still others keep all three sections separate. We will present them as separate sections. In the discussion section of the article, the researcher is expected to interpret the findings of the research study and discuss their implications. The major findings are discussed and examined for agreement or disagreement with prior research findings as presented in the literature review portion of the article.

The researchers basically answer the following question: “Did the findings support the results of prior literature?” If yes, the author will note that his findings are consistent with prior research. If the answer is no, then the researcher will identify potential reasons for the unexpected finding. The unexpected finding may be due to a research limitation or the researcher may provide an analytical or theoretical framework to explain the odd finding. In a sense, the research article is presented in a circular manner. The literature review is presented followed by the methods and results section, and then in the discussion section, the findings are tied back to the prior literature presented in the literature review.

Depending on the type of research completed, the researchers will present in the discussion section the implications of the findings for prior literature, theory development, and policy and practice in the criminal justice system. Depending on the publication outlet, some research articles may include a few paragraphs on the policy implications of the research findings. Some evaluative questions to ask about the discussion include:

1. Do the researchers acknowledge specific methodological limitations and their potential impact on the findings (limitations may be covered in methods section)?

2. Are the results discussed in terms of the literature cited in the literature review?

3. Are the results discussed in terms of any relevant theories?¹⁷

4. Does the author offer speculations about what the results may mean and their implications?

Conclusion

In the conclusion, the researchers will first briefly summarize the research problem addressed in the study, the major results, and any significant unexpected findings. Then, the researchers provide the reader a broad perspective on the research findings and put the findings into context by demonstrating where the study results fit into the larger literature. The researchers will explain the benefits derived from the research and identify areas in need of further research.

Also, the researcher will discuss the generalizability of the findings. Can the findings be generalized to other populations, other settings, and other contexts? If so, the researcher will justify why the findings can be generalized within the selected parameters. If not, the researcher will discuss this limitation and its causes along with providing an explanation of what research needs to be done in the future to improve the generalizability of the results. Some evaluative questions to ask about the conclusion include:

1. Are specific implications discussed?

2. Are suggestions for future research specified?

3. Does the research extend the boundaries of the knowledge on a topic?¹⁸

4. Do the researcher’s conclusions concur with the results that were reported or does the author overgeneralize the findings?

References

The reference section of a research article gives the reader complete information about the previous research that was used in the study. Researchers need to provide citations for each source that was used in the research article. Many of the sources will have been used in the literature review section. The reader can use the reference list to obtain additional primary sources on the research topic. If you compare the references of several research articles on the same topic, you will be able to determine what seminal and critical articles exist on the research topic.

Some evaluative questions to ask about the references include:

1. Has the most recent research on the topic been included in the references?

2. Have the seminal articles on the topic been included in the references?

External Indicators of Research Quality

This section will focus on the external indicators of research quality. The external indicators can be identified without reading the research article. It is important to recognize that reading the article and identifying the internal indicators of research quality, which will be discussed in the next section, is a more valid measure of, and therefore a better way to assess, research quality. However, the external indicators can serve as a shortcut method to determine if the research article is worth further attention and assessment.

As previously discussed, the three main publication outlets for research articles are journal articles, reports, and books. Most of the focus of this section will be on journal articles since academic journals are the primary way that researchers disseminate the results of their research. Some of the indicators discussed next, such as university affiliation and researcher reputation, apply equally to journal articles, books, and reports, but a few comments about the indicators of quality books and reports are necessary before we delve into those topics.

For this section, we are referring to books used as a primary source to document a research study. In other words, due to the magnitude of the study, a book-length manuscript is developed to document the research methodology and findings. Secondary sources include textbooks, encyclopedias, and news reports. Secondary sources usually involve summaries of primary source work and, while they may offer some information related to the research methods used in the study, they often do not provide the detail you need to assess the quality of the research article without tracking down the original article (i.e., primary source).

For books as a primary source, one of the main factors to consider when determining the quality of research reported is to look at the book’s publisher. Books published by university presses (e.g., Cornell University Press and University of Texas Press) are typically high-quality manuscripts involving high-quality research. University presses, similar to some of the academic journals discussed next, have stringent manuscript review processes, so only the best manuscripts are printed by these publishers.

Similarly, for reports, you need to look at what organization published the research. If the report is published by a federal agency such as the U.S. Department of Justice, then it should be considered high quality. The U.S. Department of Justice and other federal agencies have full-time researchers that write research reports on data obtained by the agency, so the research is conducted and written by trained researchers. The same can be said for reports published by state agencies. There are also private companies (e.g., Rand Corporation) that routinely conduct research in criminal justice agencies and then publish the reports, which are frequently available on their websites. Much of this work completed by respected companies is high quality as well because trained, full-time researchers are conducting the research and reporting the results. The theme is that if the research is completed by a full-time researcher, then it is probably a quality piece of research. Of course, there will be exceptions to the general rule, but it is a good piece of information to remember as you move forward in assessing research articles.

You should be more skeptical of research reports completed by city or county criminal justice agencies. Most of these agencies do not have full-time researchers, so there is concern about the capabilities of the person who completed the research. That does not mean that a report completed by a county or city criminal justice agency cannot be high quality but rather that you should not assume it to be high quality based on the agency that produced it. These reports need further assessment to determine their quality. We will discuss three external indicators of quality research: researcher reputation, institutional reputation, and journal quality.

Researcher Reputation

Some researchers consistently produce top-quality research, so the research articles they write—whether as a book, report, or journal article—can be assumed to be high quality. There are researchers that are the best in their particular area of expertise and everything they produce is high quality. Often, the largest body of research in a particular area is done by the same person or the same group of people at a few different institutions. We will not name specific researchers who meet these requirements, but maybe your professor will. If you or your professor does not know the researcher, you can search for information about her on the Internet. You should be able to easily find the researcher’s curriculum vita (like a resume) online and look at her prior publications to see if she has written in this area before and where the prior research articles have been published.

Institutional Reputation

Another external indicator is the institutional affiliation of the researcher and its reputation. Just like journals discussed in the following section, some universities are better than others when it comes to research. Researchers who work at top-ranked research universities are going to have strict requirements to produce high-quality research and high-quality publications in order to maintain their employment. Although even the top-ranked research universities have some deadwood (we definitely are not going to name names), institutional reputation is a quick way to determine if the research article is high quality.

Journal Quality

As previously stated, academic journals are the primary way that researchers disseminate the results of their research. The journals vary in quality though. One measure of the quality of a journal is their manuscript review process. There are two types of journals: refereed and nonrefereed journals. Refereed journals use a peer-review process to review manuscripts. Peer review is a form of quality control most commonly related to academic journals. If a journal follows a peer review process, when a manuscript is submitted to the journal, it is sent out to be reviewed by two or more experts in the field to determine whether the manuscript is suitable for publication. The review is almost always a blind review, which means the reviewer does not know the name of the researcher who wrote the manuscript. This limits bias in the review process.

When a manuscript is submitted to a refereed journal, the journal editor will review the manuscript to see if it is of sufficient quality to send out for external review. Those that are not of sufficient quality are rejected by the journal editor and returned to the author. For those that are good enough to be externally reviewed, the editor will send the manuscript to two or more reviewers. Sometimes, these reviewers are members of the journal’s editorial board. Each reviewer evaluates the article, provides written comments to the editor, and recommends to either accept, reject, or send the manuscript back to the authors for revisions and reassessment after the revisions are made (commonly called “revise and resubmit”). After the reviews are completed, the editor makes the final decision on whether the manuscript will be accepted for publication, rejected, or sent back to the authors with a request to “revise and resubmit.”

Nonrefereed journals do not utilize the peer review process. The articles in nonrefereed journals will be reviewed by the journal editor and published in the order in which they are received or according to a fee that the author pays. Journals that are not peer reviewed do not garner as much respect from fellow researchers as there is less quality control of the material published. If you are not familiar with a particular journal, you can usually determine if it is a peer-reviewed journal by reading the material at the front of the journal. There is a section at the beginning of the journal (sometimes it is in the back) that will be titled something like “information for potential contributors,” which will provide information on the manuscript review process for the journal.

As an external indicator of research quality, research articles appearing in refereed journals will be higher quality than articles appearing in nonrefereed journals. Of course, there are exceptions to this statement, but it is a good principle to remember as you move forward as an educated consumer of research.

You have identified that the journal utilizes a peer-review process and so it is a refereed journal. Now what? Well, all refereed journals are not the same quality; some are better than others. So, how do you determine if the refereed journal is a high-quality journal? One indicator is the journal’s acceptance rate. The acceptance rate is the number of manuscripts accepted for publication divided by the number of manuscripts submitted to the journal for publication consideration. The lower the acceptance rate, presumably the better the quality of the journal. The acceptance rate can be difficult to obtain without directly contacting the journal editor, so it is not always available to the consumer of research.

Another indicator of the quality of the journal is the reputation of the journal editor. As stated previously regarding the reputation of the researcher, if the journal editor is one of the top researchers in the area covered by the journal, then that is an indication that the journal is high quality as well. Similarly, the reader can review the names of the editorial board members listed at the beginning of the journal. The better the reputations of the editorial board members, the better the reputation of the journal. In addition, the university affiliation of each editorial board member will be presented in the journal. If the journal has several editorial board members that come from major research universities, then this is an indication that the journal is of high quality. These three indicators (acceptance rate, editor reputation, and editorial board composition) are used by research consumers to assess the quality of the journal, but a new indicator has been developed in recent years to measure journal quality: the impact factor.

Impact Factor The impact factor is a measure of the frequency with which the average article in a journal has been citied in a particular year. It is an evaluation tool that is annually provided by the company Thomson Reuters in the Journal Citation Reports. A journal’s impact factor is the average number of times articles from the journal, published in the past two years, have been cited. The latest year available at the time of this writing is 2011. The impact factor for 2011 was calculated as the number of citations in 2011 to articles published in the journal in 2009 and 2010 divided by the number of articles published in the journal in 2009 and 2010. The higher the impact factor, the higher the journal quality. The idea behind the impact factor is that if other researchers are citing the material from the journal’s articles, then this is an indication that the journal’s articles are high quality and thus the journal is a quality venue for published research.

The top-ranked journals of the 2011 Journal Citation Reports in the criminology and penology category which is where criminal justice and criminology journals are ranked, are:

1. Trauma Violence and Abuse (impact factor—3.265);

2. Criminology (impact factor—2.467);

3. Journal of Research in Crime and Delinquency (impact factor—2.231);

4. Crime and Justice: A Review of Research (impact factor—2.188), and;

5. British Journal of Criminology (impact factor—2.130)

A list of the top 50 journals in criminal justice and criminology are presented in Figure 8–2. You can assume that the articles in the higher ranked journals are high quality before you even begin to read the journal article.

When judging the quality of a research article using external indicators, the bottom line is to look at it from as many different perspectives as possible, including the author’s credentials, the author’s professional affiliation, and where the work is published.

FIGURE 8.2 | Criminal Justice and Criminology Journal Rankings by Impact Factor–2011

Internal Indicators of Research Quality

Although the external indicators of research quality serve as a quick screening mechanism for research articles, it is not a substitute for a detailed, internal review of the article. Even though many journals utilize a peer-review process and some have low manuscript acceptance rates, articles within those journals will vary in quality, so an internal assessment of research quality is essential. A reader who critically assesses a research article will evaluate the methodology used by the researcher, assess the analysis performed, and verify that the conclusions drawn by the authors are supported by the research findings. When we reviewed the structure of research articles earlier in the chapter, we provided evaluative questions you should ask about the title, abstract, introduction, literature review, results, discussion, conclusion, and references. We did not provide any evaluative questions for the methods section at the time, but now we’re ready to address those in this section, in which the internal indicators of research quality will be presented. These indicators will assist in answering the question “How good is this research article?” We will focus on the methods section of the article.

Throughout this book you have been provided with the tools necessary to critically evaluate research. We have covered a comprehensive list of research methods topics and have consistently discussed issues and challenges faced by researchers when conducting research. For example, in Chapter 3 we discussed probability and non-probability sampling and the issues of representativeness, generalizability, sampling error, and sample size. In Chapter 4, we discussed surveys and issues such as nonresponse, social desirability, ensuring truthful responses, and ensuring consistent question meaning for all respondents. We also provided a comprehensive discussion of reliability and validity, focusing on mechanisms to assess and increase each. In Chapter 5, we discussed experimental and quasi-experimental designs, focusing on the critical importance of the pretest, control group, and random assignment. In addition, we discussed several threats to internal validity that may impact the results found utilizing these designs. In Chapter 6, we discussed field research and the issues of going native, reactivity and the Hawthorne Effect, maintaining objectivity, and ethical dilemmas faced by field researchers. We also discussed case studies and the issues of generalizability and researcher bias. In Chapter 7, we discussed unobtrusive measures, such as secondary data analysis and the issues of having no control over the original data collection and problems with the validity of agency records. We also discussed the use of archival data and the reliability and validity issues that arise from this method, along with content analysis and the challenge of interrater reliability.

At this point, you may be thinking that no piece of research is going to withstand the rigors set forth in this chapter and book. We have stated it before, but it should be clear after the last paragraph that there is no perfect research. Researchers make a series of decisions when conducting a research study and not everyone will agree with those decisions. Each decision creates the potential for error, and each step in the research process creates the potential for error as well. There is plenty of high-quality research, though. The issue is not whether the research has limitations but how the flaws impact the research results. Are the errors of the magnitude that they can significantly impact the results of the study, which then leads you to question the results?

Your ability to read and critically evaluate research articles will improve with practice by reading many journal articles, books, and reports, but it does take practice. Remember, research involves a process and research articles are structured in a similar manner each time. Once you are comfortable with the process for conducting research and the structure of research articles, your review of articles will become easier and more routine. Once you have developed your ability to critically assess research articles, those skills will remain with you forever. You will not have to relearn the research process and evaluative issues every five years in order to maintain your skills. The reality is research methods do not change that much. Yes, it is true that new techniques are developed (e.g., Internet and e-mail surveys), but the foundation of research methods developed decades ago will be the same foundation in 100 years.

We will present several evaluative questions next. As you put your research methods knowledge into practice and review research articles, revisit the earlier chapters. But, do not think of research quality as a dichotomy of good and bad. In reality, research quality should be reflected as a continuum, and every research article can be placed somewhere on the continuum.

Questions to Ask about Research Articles

Following are some evaluative questions you should ask about a research article. Some of the questions will differentially apply, depending on the methods used in the research study. The questions will be categorized by chapter so you can easily refer back to the chapter for additional information.

Evaluative Questions from Chapter 2 Material—Ethics and Terminology

What is/are the basic research question(s)?

What were the major concepts in the research study?

How were the concepts operationalized and measured?

Would other measurements have been better? Why or why not?

What were the independent and dependent variables in the study?

Were any hypotheses stated?

Were these hypotheses justified adequately in terms of prior research?

Did the study seem consistent with current ethical standards? Explain how ethical standards were upheld.

Did the author adequately represent the findings in the discussion and/or conclusions sections?

Did the researcher overgeneralize the findings? If yes, how?

Evaluative Questions from Chapter 3 Material-Sampling In the methods section of a research article, the researcher should describe the sampling procedures used in the study and describe the subjects who participated in the research. When you are reading about the sampling procedures, be sure to evaluate who the research subjects were, taking into account available demographic characteristics such as age, race, ethnicity, and gender. Be sure to assess whether the composition of the sample will limit the generalizability of the results. Some evaluative questions about sampling follow.

Was a sample or the entire population used in the study?

What type of sampling method was employed?

Was a probability sampling method used?

If a non-probability sampling method was used, does the researcher justify its use and discuss its limitations?

Did the authors think the sample was generally representative of the population from which it was drawn? Why or why not?

Do you think the sample was generally representative of the population from which it was drawn? Why or why not?

Was the sample size large enough? Why or why not?

Overall, is the sample appropriate for generalizing to the population?

Evaluative Questions from Chapter 4 Material—Surveys, Validity, and Reliability

Was survey research used? Was this design suited to the research question posed? Why or why not?

Was the response rate reported? If yes, what was it?

Does it appear likely that those who did not respond or participate were markedly different from those who did participate? Why or why not?

What is the likelihood that characteristics of the nonrespondents introduced bias into the results? Did the authors adequately discuss this issue?

What evidence did the authors provide to show the representativeness/generalizability of the sample to the population? What comparisons between the respondents, sample, and population were presented?

Did the survey used, the measures of the variables, seem valid and reliable? Why or why not?

How did the author attempt to establish the validity of the survey?

When delving into sensitive matters (e.g., past victimization or criminal/delinquent history), is there reason to believe that accurate data were obtained?

Do you believe that social desirability is an issue in the study? Why or why not?

How did the author attempt to establish the reliability of the survey?

If an index was used in the survey, does it have adequate internal consistency?

How did the researcher ensure consistent meaning of the questions?

How was the survey distributed? What factors influenced the author’s decision to distribute the survey in this manner?

Evaluative Questions from Chapter 5 Material—Experimental and Quasi-Experimental Designs

What type of research design was used in this study?

What were the parameters/criteria for inclusion in the study?

What cases were excluded from the study?

What was/were the treatment(s)?

What was/were the dependent variable(s)?

What pre-test(s) was/were conducted? Be sure to discuss how each dependent variable was measured and the time frame.

What post-test(s) was/were conducted? Be sure to discuss how each dependent variable was measured and the time frame.

What was/were the experimental/treatment group(s)?

What was/were the control/comparison group(s)? Was the control/comparison group adequate?

Were the groups established through randomization or matching? Be sure to discuss the procedures used to establish and monitor equivalence.

What threats to internal validity could impact the results of this study? For each threat to internal validity identified, define the threat, describe how it applies to this study, and explain how it could impact the results of this study.

Were any causal assertions made or implied in the hypotheses or in subsequent discussion? If yes, were all three criteria for establishing causal relationships addressed? How did the author establish the existence of the three criteria for causality? What, if any, variables were controlled in the analysis to reduce the risk of spurious relationships?

Evaluative Questions from Chapter 6 Material—Field Research and Case Studies

Was field research or a case study used in this study? Was the design suited to the research question posed? Why or why not?

How much interaction was present between the researcher and the research subjects?

What steps were taken to keep the researcher from influencing the behavior of the research subjects?

Were any ethical dilemmas faced by the researcher? If yes, how did the researcher handle the dilemmas?

Did the researcher maintain objectivity? If not, how did this influence the results of the study?

Did the researcher acknowledge any biases or reactivity in the study? If so, how did they impact the results?

Did the researcher discuss the generalizability of the findings?

Did the instruments used, the measures of the variables, seem valid and reliable? Why or why not?

How did the author attempt to establish the reliability of the measurement?

How did the author attempt to establish the validity of the measurement?

What more could have been done in the study to establish reliability and validity?

Evaluative Questions from Chapter 7 Material—Unobtrusive Methods

Was an unobtrusive method such as secondary data analysis used? Was this design suited to the research question posed? Why or why not?

Did the instruments used, the measures of the variables, seem valid and reliable? Why or why not?

How did the author attempt to establish the reliability of the measurement?

How did the author attempt to establish the validity of the measurement?

What more could have been done in the study to establish reliability and validity?

Article Critique: Minneapolis Domestic Violence Experiment

Sherman, Lawrence W., and Richard A. Berk. (1984). “The specific deterrent effects of arrest for domestic assault.” American Sociological Review. 49, 261–272. Portions of the article are included in Appendix A.

As the final section in this chapter, we will present an example of an article critique utilizing the internal indicators of research quality discussed in the previous section. Recall that it is impossible to develop a single checklist of the topics that need to be addressed in a methods section of a research article, because the details that need to be provided depend on the methods used. Similarly, it is impossible to develop a single set of questions that can be asked and answered to comprehensively assess every research article, because the questions depend on the methods used in the article. Therefore, we will present an example of an article critique tailored to the experimental and quasi-experimental designs discussed in Chapter 5. We selected the Minneapolis Domestic Violence Experiment (MDVE) as the example, because it is probably the best known piece of research in criminal justice and criminology. You not only hear about the results of the MDVE (arrest is most effective in reducing domestic violence recidivism) in research methods, but also hear about it in policing classes and policy courses. The MDVE is to research methods what Miranda v. Arizona is to criminal procedure: the most recognized example on the topic.

Next, we ask and answer several questions regarding the MDVE. Portions of the article are in Appendix A. You can review the article to see where we obtained the information to answer each question. This is the type of activity you will be able to perform, although probably not as structured as the questions and answers that follow, as you begin to apply your knowledge as an educated consumer of research.

1. In one sentence, state the purpose of this study.

The purpose of the MDVE was to determine which of three police responses (arrest, separation, or mediation) produced the lowest recidivism rates for domestic violence offenders.

2. What type of research design was used in this study?

A randomized experimental design was used in the MDVE.

3. What were the parameters/criteria for inclusion in the study?

In order to be included in the study, the calls for police service had to involve a misdemeanor (simple) domestic assault, and both the suspect and the victim had to be present when the police arrived.

4. What cases were excluded from the study?

The following cases were excluded from the study: (1) cases involving felony domestic assaults; (2) cases where both parties were not present when the police arrived; (3) cases where the suspect attempted to assault police officers; (4) cases where a victim persistently demanded an arrest, and (5) cases involving injuries to both parties.

5. What was/were the treatments)?

The treatment was the police response to domestic violence, which included arrest, separation, or mediation.

6. What was/were the dependent variable(s)?

The dependent variable was repeat domestic violence (i.e., recidivism).

7. What pre-test(s) was/were conducted? Be sure to discuss how each dependent variable was measured and the time frame.

Since the dependent variable was recidivism, no pre-test was conducted in the MDVE.

8. What post-test(s) was/were conducted? Be sure to discuss how each dependent variable was measured and the time frame.

The post-test involved a 6-month follow-up period to measure the frequency and seriousness of domestic violence after each police intervention of arrest, separation, or mediation. The dependent variable (i.e., repeat domestic violence) was measured in two ways. First, the researchers obtained police offense and arrest reports for domestic violence that mentioned the suspect’s name during the 6-month follow-up period. Second, victim interviews were completed 2 weeks after the police intervention and every 2 weeks thereafter for 24 weeks. The first victim interview was face-to-face while each subsequent interview was conducted over the telephone. The victim interviews allowed researchers to examine domestic violence offenses that may have occurred but did not come to the official attention of police.

9. What was/were the experimental group(s)?

Since three interventions were used in the MDVE, there are three different groups of people, totaling 314 cases. There were two experimental groups. First, the 136 people arrested during the study comprised one experimental group. Second, the 89 people separated during the study comprised the second experimental group.

10. What was/were the control group(s)?

Since the common police practice at the time for cases of misdemeanor domestic violence was to advise and mediate the situation, leaving both parties at the location, the control group was the 89 people advised/mediated during the domestic violence calls for service.

11. How was random assignment established? Be sure to discuss the procedures used to establish and monitor equivalence.

The three different police actions were randomly distributed based on color-coded report forms carried by the officers. Upon a misdemeanor domestic violence call with both parties present, the officer’s action (arrest, separation, or mediation) was predetermined by the order of report forms in the officer’s notebook. The colored report forms were randomly ordered in the officer’s notebook and the color on the form determined the officer response at the scene. All colored report forms were randomly ordered through a lottery assignment method. The result was that all police officer actions in misdemeanor domestic violence calls were randomly assigned. To ensure the random assignment was carried out as planned, research staff participated in ride-alongs with officers to ensure that officers did not skip the order of randomly ordered forms. In addition, the research staff logged the domestic violence reports in the order received to make sure the sequence corresponded to the original assignment of police interventions. Table 1 in the research article located in Appendix A shows the degree to which the treatments were delivered as designed.

12. Identify two threats to internal validity that may impact the results of this study. For each threat to internal validity identified, define the threat, describe how it applies to this study, and explain how it may impact the results of this study.

The first threat to internal validity is selection bias. As depicted in Table 1 in the article in Appendix A, the officers did not always follow the random assignment protocol. For example, some officers decided to violate the random assignment protocol, which violated the randomized experimental design, because they found the upcoming treatment inappropriate for the situation. The officers did not always follow the randomly assigned action (arrest, separation, or mediation) as a result of other circumstances that occurred at the scene. In fact, some officers simply ignored the assigned action if they felt a particular call for domestic violence required another action. Selection bias generally indicates that the experimental group is somehow different from the control group on a factor that could influence the post-test results. Since the officers violated the random assignment protocol and self-selected some subjects for arrest, the arrested group may be different from the other two groups on factors that may impact repeat domestic violence.

The second threat to internal validity is attrition or subject mortality. Attrition refers to differential loss in the number or type of subjects between the experimental and control groups. In the MDVE, attrition occurred with the victim interviews. Only 205 of the victims (65% response rate) could be located and initial interviews obtained two weeks after the police intervention. Only 161 victims (51% response rate) provided all 12 follow-up interviews over the 6-month follow-up period. The question then is, “Are there differences between the respondents and the nonrespondents that could then impact the results of the MDVE?” We do not know how the two groups differed so are unable to adequately answer the question. We do know from Chapter 4 that as the response rate decreases, it becomes more likely that the nonrespondents will differ from the survey respondents. When there are significant differences between respondents and nonrespondents, each group represents biased subgroups of the total population. Bias means that in some systematic way the individuals responding to a survey are different from the overall population. Therefore, the results based on the survey respondents are not reflective of the overall population.

Chapter Summary

In this chapter, we reviewed the steps in the research process to provide the “big picture” of research endeavors. We also reviewed the structure of research articles. It is helpful as an educated consumer of research to understand the common structure of research articles and reports. You will be able to easily identify these sections as you review research articles and reports in the future. We specifically discussed three external indicators of research quality, including researcher reputation, institutional reputation, and journal quality. The external indicators can serve as a short-cut method to determine if the research article is worth further attention and assessment. Although the external indicators of research quality serve as a quick screening mechanism for research articles, it is not a substitute for a detailed, internal review of the article. A reader who critically assesses a research article will evaluate the methodology used by the researcher, assess the analysis performed, and verify that the conclusions drawn by the authors are supported by the research findings. We reviewed the internal indicators of research quality in this chapter. The discussion was a culmination of the tools provided throughout this book to critically evaluate research. Finally, this chapter provided an example of a research article critique to demonstrate how the internal indicators of research quality can be used to assess a research article. Now that you have finished this chapter, you have the tools necessary to thoroughly review and evaluate research, with one exception: statistics. Statistics will be covered in the next chapter.

Critical Thinking Questions

1. What are the steps in the research process, and what activities occur at each step?

2. What are the primary sections of a research article, and what material is covered in each section?

3. What are the three main external indicators of research quality? How is journal quality determined?

4. How would you apply the internal indicators of research quality to a research article? Be sure to provide the types of questions you would ask in your assessment.

Endnotes

1 Grogger, Jeffrey, and Greg Ridgeway. (2006). “Testing for racial profiling in traffic stops from behind a veil of darkness.” Journal of the American Statistical Association 101, 878–887.

2 Worden, Robert E., Sarah J. McLean, and Andrew R Wheeler. (2012). “Testing for racial profiling with the veil-of-darkness method.” Police Quarterly 15, 92–111.

3 Ibid.

4 Hamermesh, D. S. (1996). Workdays, workhours, and work schedules: Evidence for the United States and Germany. Kalamazoo, MI: W. E. Upjohn Institute for Employment Research.

5 Worden, et al., 2012.

6 http://www.racialprofilinganalysis.neu.edu/background/jurisdictions.php. Retrieved on July 28, 2012.

7 Ibid.

8 Worden, et al., 2012.

9 http://www.forbes.com/sites/jacquelynsmith/2012/07/20/how-road-rage-endangers-your-commute-and-how-to-prevent-it-in-yourself/; http://www.careerbuilder.com/share/aboutus/pressreleasesdetail.aspx?sd=7/l8/2012&id=pr708&ed=12/31/2012

10 Pyrczak, Fred. (2008). Evaluating research in academic journals: A practical guide to realistic evaluation. Glendale, CA: Pyrczak Publishing.

11 Ibid.

12 McReynolds, Larkin S., and Jeffrey C. Sandler. (2012). “Evaluating New York State’s Sex Offender Management and Treatment Act: A matched historical cohort analysis.” Criminal Justice Policy Review 23, 164–185.

13 Sandler, J. C., N. J. Freeman, and K. M. Socia. (2008). “Does a watched pot boil? A time-series analysis of New York State’s sex offender registration and notification.” Psychology, Public Policy, and Law 14, 284–302; Zgoba, K., P. Witt, M. Dalessandro, and B. Veysey. (2008). Meghan’s Law: Assessing the practical and monetary efficacy. Washington, D.C.: U.S. Department of Justice.

14 Duwe, G., W. Donnay, and R. Tewksbury. (2008). “Does residential proximity matter? A geographic analysis of sex offense recidivism.” Criminal Justice and Behavior 35, 484–504.

15 Pyrczak, 2008.

16 Ibid.

17 Ibid.

18 Ibid.

19 Dance, Amber. (August 7, 2007). “Videos as a baby brain drain: A study finds infants who view some popular educational products learn fewer words.” Los Angeles Times. http://articles.latimes.com/2007/aug/07/science/sci-babyeinstein7. Retrieved on July 28, 2012; Park, Alice. (August 6, 2007). “Baby Einsteins: Not so smart after all.” Time Science. http://www.time.com/time/health/article/0,8599,1650352,00.html. Retrieved on July 28, 2012.

20 Zimmerman, Frederick J., Dimitri A. Christakis, & Andrew N. Meltzoff. (2007). “Associations between media viewing and language development in children under age 2 years.” Journal of Pediatrics 151, 364–368.

21 Ibid.

22 http://www.washington.edu/alumni/uwnewslinks/200709/videos.html. Retrieved on July 28, 2012.

23 http://www.seattlepi.com/local/article/The-full-text-of-Walt-Disney-Co-s-letter-1246455.php. Retrieved on July 28, 2012.

24 Ibid.

25 uwnews.washington.edu/ni/article.asp?articleID=36148. Retrieved on July 28, 2012.

26 Piliavin, Irving, and Scott Briar. (1964). “Police encounters with juveniles.” American Journal of Sociology 70, 206–214. This study was also included in Amy B. Thistlethwaite and John D. Wooldredge. (2010). Forty studies that changed criminal justice: Explorations into the history of criminal justice research. Upper Saddle River, NJ: Prentice Hall.