A scholarly search for the truth, obtained by observation and experimental studies, often involves obtaining detailed data to test one or more hypotheses. Ideally, experimental studies answer a research question in a complete and unambiguous manner that is consistent with other known results. Research always is chancy, and the expected results are not always obtained even when well-designed experiments are conducted by experienced scientists.
Good research uses well-designed experiments, includes adequate controls, and leads to solid interpretations. The conclusions drawn from good research enable accurate predictions to be made, and can easily be related to existing bodies of other knowledge. Future experiments can build successfully upon what is established from good research.
Bad research is the opposite of good research. It results from poorly designed experiments, and can feature incomplete or inadequate controls. The conclusions drawn from bad research usually are later shown to be completely or partly invalid; they make only incorrect predictions, and are inconsistent with other bodies of knowledge. The results from bad research often are not repeatable, and form a defective basis for any further studies.
Good versus bad research.
All scientists hope to conduct good research. Typical questions for judging research quality include the following: (1) are the experiments well-designed and properly conducted; (2) are the controls fully adequate; (3) are the data complete; (4) are the data and their interpretations self-consistent; (5) do the experimental data support the conclusions of the research study; (6) are the conclusions consistent with other data and known facts; and, (7) do these experiments answer the selected research question(s)? Failure or insufficiency in any of these parameters is a typical sign of bad research.
The judgemnent of research quality needs to be distinguished from several related evaluations. Quality of research is distinguished from quality of the research subject (e.g., either good or bad research investigations can be conducted on how to add multivitamins to a metropolitan water supply), and from good or bad usage of the research findings (e.g., good chemical research might later be utilized to make some extremely toxic new complex). Experimental results supporting a well-known theory or popular concept do not necessarily mean that this research is good; similarly, experimental studies that contradict or do not agree with some well-established theory are not necessarily bad.
Research in any branch or category of science can be judged to be good or bad. In general, judgements of research quality do not have any intermediate levels. These determinations are made in basic or applied research, theoretical or experimental research, small or giant studies, field or laboratory research, simple or complex research, etc. As one example, consider a modern research study of butterflies inside Columbia, which finds that one species there is simultaneously present in Argentina. Assume here that detailed morphological measurements, molecular genetics, and field observations were conducted properly, etc., and that all data show complete taxonomic identity, while other species in Argentina lack identity. Although there is no obvious usefulness in this discovery, it is a clear example of good research in basic science.
Who exactly best determines.whether research is good or bad? Here, a critical judgement is sought, and not a casual opinion. Since the necessary very careful evaluation of the experiments involved in any research project can be quite complex, this determination is best made by knowledgeable experts (i.e., other scientists). This judgement must be made objectively without regard to personal interest or emotional preferences.
Who utilizes the judgement of good vs. bad research?
The critical evaluation of research quality is part of several major job activities for university scientists, including determining priority scores for research grant applications and proposals, and, examination of manuscripts submitted for publication in a science journal. In both cases, peer review utilizes the evaluation by scientists who have expertise in the same area as the applicant or author.
Peer review of proposals and applications for financial support of research aims to make judgements be as objective as possible . To determine fundability, the design of experiments, adequacy of controls, methods for data analysis, and ability to answer the research questions proposed first are evaluated. The final conclusion for fundability also utilizes certain other criteria besides determining whether the research is good or bad (e.g., capability to answer the selected research questions, chances for success of the project in the time period proposed, previous training and experience with the methodologies used, atmosphere at the institution, track record of the applicant for success in previous research projects, relevancy to program targets, use of undergraduate students or special groups of people, research safety considerations (e.g., exposure to disease agents, toxins, or radioactive materials, etc.). A listing of official criteria for evaluating merit in the very numerous research grant applications sent to the National Institutes of Health (see: http://grants.nih.gov/grants/peer/critiques/rpg.htm ) or to the National Science Foundation (see: http://www.nsf.gov/nsb/publications/2011/meritreviewcriteria.pdf ) are published at periodic intervals.
Not all manuscripts submitted to science journals are accepted for publication. To determine publishability, the journal editor and assigned referees first take a critical look at whether the research reported is good or bad, and then examine the conclusions drawn from the experimental data. If their evaluations conclude that something is missing, the experiments are poorly designed, controls are inadequate, interpretations are not supported, data are incomplete, the subject area is not relevant to the journals’s focus, etc., then a manuscript will be rejected. The critical comments are relayed to the authors so they can try to make the needed additions, deletions, and other changes; after consideration of the revised manuscript, a final decision about publishability then is made and reported to the authors.
What can go wrong with judging good vs. bad research?
There are quite a few possibilities where the examination of research quality can go wrong. Selection of reviewers with insufficient expertise excourages mistakes to be made. Selection of scientists as reviewers who are unable to put aside the fact that they are competing with the applicant for research grant awards also leads to unfortunate mistakes. In the modern era, time is very precious for all research scientists working at universities; doing a rush job with evaluating research quality saves time, but increases the chance of making mistakes. As personal integerity decreases, there is increased likelihood that rigor of this important task for making objective evaluations is not maintained (e.g., ignoring some defect for a friend, colleague at the same institution, or former associate). In other cases, rigor is undercut by the unethical desire to please someone or to trade favors (e.g., “I will overlook this mistake in your manuscript if you do the same when you review my manuscripts!”). The agencies awarding research grants take explicit steps to try to preclude these improper diversions from good ethical practices; most professional science journals require at least two independent expert reviewers to critically examine each manuscript, in order to decrease the chance that any mistaken or improper judgement will be made.
Determination of good versus bad research can be made readily using standardized criteria for evaluating the quality of the experiments, particularly if this review is performed by several experts. These detailed evaluations must be done very carefully, and demand the critical capabilities of other expert scientists working in the same area. These peer evaluations constitute a major part of the review process for applications seeking research grant support, and of manuscripts submitted to science journals for publication. Determining the quality of research is not identical to determining the quality of science (i.e., good research can be part of bad science, and vice versa). Critical determinations of research quality are important to help science be rigorous, objective, and meaningful.
GO BACK TO HOME PAGE OR SCROLL UP TO MENU
UNDER THE WEBSITE TITLE