Reading news stories of medical research

The overall goal of debates around peer review and the appearance of new publication platforms and approaches is to create a united front of authors

Wondering if that latest study finding is too good to be true, or whether it's as bad as we're told? Here are five questions to ask to help you assess the evidence. Who doesn’t want to know if drinking that second or third cup of coffee a day will improve your memory or if sleeping too much increases your risk of a heart attack?

We are invested in staying healthy and many of us are interested in reading about new research findings to help us make sense of our lifestyle choices. But not all research is equal, and not every research finding should be interpreted in the same way. Nor do all media headlines reflect what was actually studied or found. So how can you tell? Keep these five questions in mind when you’re reading media stories about new studies.

1. Has the research been peer-reviewed?Peer review is a process by which a study is checked by experts in the discipline to assess the study’s scientific validity. This process involves the researcher writing up their study methods and results and sending this to a journal. The manuscript is then usually sent to two to three experts for peer review.If there are major flaws in a study, it’s either rejected for publication, or the researchers are made to address these flaws. Although the peer-review process isn’t perfect, it shows a study has been subjected to scrutiny.

Peer review is the central foundation of science. It’s a process where scientific results are vetted by academic peers, with publication in a reputable journal qualifying the merits of the work and informing readers of the latest scientific discoveries. But peer review sometimes gets a bad rap – criticized for a purported lack of transparency, low accountability, and even poor scientific rigor. There’s now a considerable movement towards tweaking or even remodeling the peer review system. Key areas of focus include making journal editors more directives in the process, rewarding reviewers, and improving the accountability of editors, reviewers and authors.

Peer review relies on volunteers: The peers in the peer review system are volunteer academics with expertise relevant to the paper being considered. But it’s hard to find suitable volunteers. Reviewing is more complex and onerous than just rejecting or accepting a manuscript. More often than not, a reviewer suggests additional experiments that authors have overlooked or challenges the interpretation of some of the data. This initiates a dialogue between author and reviewer aimed at improving the integrity and scientific merit of the paper.

More accountability from editors: We should demand more of our journal editors. Editors can become more proactive by rejecting articles that are not at publication standard upon submission, rather than placing the arduous task on a reviewer to be both a scientist and copy checker. To retain and train novice reviewers, clearer evaluation criteria from editors would vastly improve the reliability and quality of submitted papers.

Recognition of reviewers: Traditionally, editors are held up to be a revered part of the peer review process, and reviewers are simply not acknowledged for their contributions. But this is changing. To promote increased transparency, greater accountability and fairness, open peer review processes list reviewers and editors in addition to authors in each publication. This is happening now in newly established online journals such as eLife. Independent platform Publons reward reviewers by listing all peer reviewing and editorial activity to provide evidence of a reviewer’s expert contributions in their field. Publons also runs a reviewer awards program.

Post- vs pre-publication peer review: It’s now becoming clear that scientific dialogue does not need to stop at the endpoint of publication, and that not all problems within a manuscript may be identified at the time of peer review.

Time to try something new.Peer review is not ready to be retired – but it is primed to change. A recent trial by eLife intends to radically transform the roles of editor, reviewer, and author. According to this model, if a senior editor deems a publication worthy of going to review, this paper immediately qualifies for publication.

Once under review, an open dialogue between author and reviewers takes place. Upon receipt of reviewers’ recommendations, the authors can decide to continue experiments if advised, retract the paper or publish it. This leaves the author’s decision to the scrutiny of the general scientific community.

This innovation may greatly improve the transparency of open peer review, increase accountability on behalf of all participants and reduce the burden on the peer review system. It addresses the three major strategies required for the improvement of the peer review system. But is it a step too far, too soon? Time will tell.

The overall goal of debates around peer review and the appearance of new publication platforms and approaches is to create a united front of authors, reviewers and editors to uphold scientific integrity. This is vital not just within academic circles, but also to maintain the reputation of science in the broader community. Any reported findings that haven’t been peer-reviewed should be read with a degree of reservation.

2. Was the study conducted in humans? Findings from studies conducted in animals such as mice or on cells in a lab (also called in vitro studies) represent the earliest stage of the scientific discovery process. Regardless of how intriguing they may be, no confident claims about human health should ever be made based on these types of study alone. There is no guarantee that findings from animal or cell studies will ever be replicated in humans.

3. Are findings likely to represent a causal relationship? For a study to have relevance to our day-to-day health, the findings need to reflect a causal relationship rather than just a correlation. If a study showed that coffee drinking was associated with heart disease, for example, we want to know if this was because the coffee actually caused heart disease or whether these to things happened to occur together.

In a number of studies that found this association, researchers subsequent found that coffee drinkers were more likely to be smokers and therefore, these results were more likely to reflect a true causal relationship between smoking and heart disease.

In observational studies, where researchers observe differences in groups of people, it can sometimes be difficult to disentangle the relationship between variables.

The highest level of evidence regarding causality comes from a double-blind placebo controlled randomized controlled trials (RCTs). This experimental type of study, where people are separated into groups to randomly receive either an intervention or placebo (sham treatment), is the best way we can determine if something causes disease. However it, too, is not perfect.

Although other types of studies in humans play an important role in our understanding of health and disease, they may only highlight associations that are not indicative of causal relationships.

Interpreting health (or any other) risk estimates reported in the media is not straightforward. Even health professionals can get tripped up trying to make sense of these statistics, so it is no wonder the public can easily be confused or misled. Often there is a tendency to overreact to risk estimates, so it’s worth unpacking what these really mean.

Until we have a significant body of evidence that is in agreement, we have to be very careful about our interpretation of the findings from any one study.

What if these questions aren’t answered? If the media report you’re reading doesn’t answer these questions, consider changing news sites or looking at the original paper. Ideally, this would be linked in the news article you’re reading, or you can search PubMed for the article using a few keywords.

The journal article’s abstract should tell you the type of study, whether it was conducted on humans and the size of the effect. If you’re not blocked by a paywall, you may be able to view the full journal article which should answer all of the questions you have about the study.

As a medical sociologist and research ethicist,I live and breathe science. Much of what I read and hear is couched in the language of science which to outsiders can seem little more than jargon and gibberish. But one word is rarely spoken or printed in science and that word is “proof”. In fact, science has little to do with “proving” anything.

The writer is  former Head, Department of Medical Sociology,

Institute of Epidemiology, Disease Control & Research (IEDCR)

Dhaka, Bangladesh

E-mail: [email protected]

Reading news stories of medical research

The overall goal of debates around peer review and the appearance of new publication platforms and approaches is to create a united front of authors

Wondering if that latest study finding is too good to be true, or whether its as bad as were told? Here are five questions to ask to help you assess the evidence. Who doesnt want to know if drinking that second or third cup of coffee a day will improve your memory or if sleeping too much increases your risk of a heart attack? We are invested in staying healthy and many of us are interested in reading about new research findings to help us make sense of our lifestyle choices. But not all research is equal, and not every research finding should be interpreted in the same way. Nor do all media headlines reflect what was actually studied or found. So how can you tell? Keep these five questions in mind when youre reading media stories about new studies. 1. Has the research been peer-reviewed?Peer review is a process by which a study is checked by experts in the discipline to assess the studys scientific validity. This process involves the researcher writing up their study methods and results and sending this to a journal. The manuscript is then usually sent to two to three experts for peer review.If there are major flaws in a study, its either rejected for publication, or the researchers are made to address these flaws. Although the peer-review process isnt perfect, it shows a study has been subjected to scrutiny. Peer review is the central foundation of science. Its a process where scientific results are vetted by academic peers, with publication in a reputable journal qualifying the merits of the work and informing readers of the latest scientific discoveries. But peer review sometimes gets a bad rap criticized for a purported lack of transparency, low accountability, and even poor scientific rigor. Theres now a considerable movement towards tweaking or even remodeling the peer review system. Key areas of focus include making journal editors more directives in the process, rewarding reviewers, and improving the accountability of editors, reviewers and authors. Peer review relies on volunteers: The peers in the peer review system are volunteer academics with expertise relevant to the paper being considered. But its hard to find suitable volunteers. Reviewing is more complex and onerous than just rejecting or accepting a manuscript. More often than not, a reviewer suggests additional experiments that authors have overlooked or challenges the interpretation of some of the data. This initiates a dialogue between author and reviewer aimed at improving the integrity and scientific merit of the paper. More accountability from editors: We should demand more of our journal editors. Editors can become more proactive by rejecting articles that are not at publication standard upon submission, rather than placing the arduous task on a reviewer to be both a scientist and copy checker. To retain and train novice reviewers, clearer evaluation criteria from editors would vastly improve the reliability and quality of submitted papers. Recognition of reviewers: Traditionally, editors are held up to be a revered part of the peer review process, and reviewers are simply not acknowledged for their contributions. But this is changing. To promote increased transparency, greater accountability and fairness, open peer review processes list reviewers and editors in addition to authors in each publication. This is happening now in newly established online journals such as eLife. Independent platform Publons reward reviewers by listing all peer reviewing and editorial activity to provide evidence of a reviewers expert contributions in their field. Publons also runs a reviewer awards program. Post- vs pre-publication peer review: Its now becoming clear that scientific dialogue does not need to stop at the endpoint of publication, and that not all problems within a manuscript may be identified at the time of peer review. Time to try something new.Peer review is not ready to be retired but it is primed to change. A recent trial by eLife intends to radically transform the roles of editor, reviewer, and author. According to this model, if a senior editor deems a publication worthy of going to review, this paper immediately qualifies for publication. Once under review, an open dialogue between author and reviewers takes place. Upon receipt of reviewers recommendations, the authors can decide to continue experiments if advised, retract the paper or publish it. This leaves the authors decision to the scrutiny of the general scientific community. This innovation may greatly improve the transparency of open peer review, increase accountability on behalf of all participants and reduce the burden on the peer review system. It addresses the three major strategies required for the improvement of the peer review system. But is it a step too far, too soon? Time will tell. The overall goal of debates around peer review and the appearance of new publication platforms and approaches is to create a united front of authors, reviewers and editors to uphold scientific integrity. This is vital not just within academic circles, but also to maintain the reputation of science in the broader community. Any reported findings that havent been peer-reviewed should be read with a degree of reservation. 2. Was the study conducted in humans? Findings from studies conducted in animals such as mice or on cells in a lab (also called in vitro studies) represent the earliest stage of the scientific discovery process. Regardless of how intriguing they may be, no confident claims about human health should ever be made based on these types of study alone. There is no guarantee that findings from animal or cell studies will ever be replicated in humans. 3. Are findings likely to represent a causal relationship? For a study to have relevance to our day-to-day health, the findings need to reflect a causal relationship rather than just a correlation. If a study showed that coffee drinking was associated with heart disease, for example, we want to know if this was because the coffee actually caused heart disease or whether these to things happened to occur together. In a number of studies that found this association, researchers subsequent found that coffee drinkers were more likely to be smokers and therefore, these results were more likely to reflect a true causal relationship between smoking and heart disease. In observational studies, where researchers observe differences in groups of people, it can sometimes be difficult to disentangle the relationship between variables. The highest level of evidence regarding causality comes from a double-blind placebo controlled randomized controlled trials (RCTs). This experimental type of study, where people are separated into groups to randomly receive either an intervention or placebo (sham treatment), is the best way we can determine if something causes disease. However it, too, is not perfect. Although other types of studies in humans play an important role in our understanding of health and disease, they may only highlight associations that are not indicative of causal relationships. Interpreting health (or any other) risk estimates reported in the media is not straightforward. Even health professionals can get tripped up trying to make sense of these statistics, so it is no wonder the public can easily be confused or misled. Often there is a tendency to overreact to risk estimates, so its worth unpacking what these really mean. Until we have a significant body of evidence that is in agreement, we have to be very careful about our interpretation of the findings from any one study. What if these questions arent answered? If the media report youre reading doesnt answer these questions, consider changing news sites or looking at the original paper. Ideally, this would be linked in the news article youre reading, or you can search PubMed for the article using a few keywords. The journal articles abstract should tell you the type of study, whether it was conducted on humans and the size of the effect. If youre not blocked by a paywall, you may be able to view the full journal article which should answer all of the questions you have about the study. As a medical sociologist and research ethicist,I live and breathe science. Much of what I read and hear is couched in the language of science which to outsiders can seem little more than jargon and gibberish. But one word is rarely spoken or printed in science and that word is proof. In fact, science has little to do with proving anything. The writer is former Head, Department of Medical Sociology, Institute of Epidemiology, Disease Control Research (IEDCR) Dhaka, Bangladesh E-mail: [email protected]