EVALUATING THE QUALITY OF PUBLISHED RESEARCH
Dr Chris van Dijk
‘Research” is a tool that is widely used, and in some instances misused, to support specific claims of product performance and to imply other benefits to be derived from a product. In the clutter of claims that surround a product, it can be difficult to know what is truly based on solid scientific research and what is based on questionable research done or interpreted specifically to support a marketing objective.
One of the standard measures of research validity is its publication in a scientific journal. Many people do not understand, however, the different levels of review that are required for research to be accepted for publication. Separating the wheat from the chaff in scientific research can be accomplished by evaluating the way in which the original research was conducted and the review process it has undergone prior to being published.
Journal articles can be categorized as “refereed”, “peer-reviewed” or “non-reviewed”, and the reputation of a journal within the scientific community is generally tied to the stringency of this scientific review process. Thus, information in a referred journal is generally given greater authority and legitimacy than information in a peer-reviewed journal, and considerably greater validity than information in a non-reviewed journal.
- Refereed papers are generally reviewed by scientists active in the field.
- Peer reviewed papers are reviewed by someone knowledgeable in field.
- Non-reviewed papers are often proceedings or abstracts from meetings and seminars, or papers submitted primarily for the purpose of putting the information out into an arena for presentation and discussion.
The Review Process
Understanding the publication review process can help veterinarians and other readers give proper weight to the research they read.
Review Process: Refereed Journals
- Author submits manuscript
- Editor or section editor assigns reviewers (usually two): reviewers are independent and will not know who else is reviewing the material.
- Reviewers make line by line comments on the manuscript and then either recommend that the article be accepted, accepted with revisions, or rejected.
- The article almost always goes back to the author for clarification, additional information or to allow the author to respond to or refute comments and critiques.
- Article may go back to reviewer for second review; the publication may choose a third reviewer if there is substantial disagreement between reviewers.
- Editor makes the final decision.
- If reviewers concur, the recommendation is usually accepted by the editor.
This process explains why it can take 6-12 months from submission to publication.
Criteria for Acceptance by Reviewers
Various journals have their own instructions for reviewers, which vary to some extent. In evaluating research for publication, reviewers will generally use these criteria:
- Is the material appropriate for this journal?
- Is it original?
- Does it contribute new information to the knowledge base? (This is an area where refereed journals and peer-reviewed journals can differ.).
- Is the experimental design appropriate for the hypotheses?
- Are the statistical methods appropriate?
- Are the conclusions supported by the data?
Review Process: Peer Reviewed Literature
With a peer-reviewed publication, the editor receives a paper, and often asks just one person from the editorial board to read the paper and provide comments. The editor then works through any comments directly with the author and publishes the paper.
Review Process: Proceedings and Abstracts
Although they qualify as “published” research, proceedings and abstracts generally have no review requirement, except perhaps from a program committee or editorial committee. However, the reviewer does not evaluate the validity of the research conducted. Readers of abstract should understand that the information presented has not been through a research review process.
Evaluating Research Quality
In a study in the Bovine Practitioner in 1997, Dr. Louis Perino and Dr. Breck Hunsaker conducted a literature search of abstracts from 1972-1996 to review field efficacy of bovine respiratory viral vaccines based on research reported in peer-reviewed publications. The researchers found that the vast majority of these studies did not use sound research methods, did not have clinically relevant outcomes or did not include sufficient information to confirm statistical results.
The following research recommendations came out of that analysis:
- Include a valid control group
- Use an externally relevant population
- Use a clinically relevant treatment regimen
- Use random treatment assignment
- Assessors should be blinded to treatment assignment
- Use a field challenge in an externally relevant production setting.
- Ensure adequate follow-up
- Have a adequate statistical power to detect treatment effects
- Control for confounding variables.
- Measure clinically relevant outcomes to determine if differences are important.
- Use appropriate statistical evaluation to determine if differences are real.
Many veterinarians also want to know how well a given study might predict the outcome in another herd. They are reluctant to rely on research unless the study sample duplicates conditions in the herds and operations they are dealing with. Key factors in making this determination are:
- Geography, Nutritional Environment, Season and Breed
When presented with a new product or vaccine ask yourself the questions as above.
Zoetis’ Commitment to Quality Research
The criteria listed by Drs. Perino and Hunsaker match those that Zoetis uses in establishing research protocols, writing final reports, evaluating the statistical significance of research results, and applying the results to marketing and promotional materials. Zoetis’ “biometrics groups”, which are responsible for the statistical analysis and subsequent release of research, have exceptionally high standards, which is what ensures validity of our research process.
Zoetis’ research standards are designed to promote and protect research quality.
Disclaimer: Zoetis takes no responsibility for any claims that may arise from information contained in this information sheet. Individual situations may vary from location to location and it is recommended that you consult your veterinarian before any management or treatment decisions are implemented.
Vrywaring: Zoetis neem geen verantwoordelikheid vir enige eise wat mag voortspruit uit inligting vervat in hierdie inligtingsdokument. Individuele situasies varieer van plek tot plek en dit word voorgestel dat u eers u veearts kontak alvorens enige bestuurs- of behandelingsbesluite geïmplementeer word.