What are some common issues I find when reviewing algorithmic bioinformatics conference papers?

UPDATE: This post, in a slightly modified and improved form, is now published at https://doi.org/10.1371/journal.pcbi.1007742 . Please see there for the latest version.

As a PC member, I sometimes find it frustrating to see a paper that potentially has a great contribution be rejected because of the way it was written. I wish that I had the opportunity to tell the authors — hey, you forgot to do this really important thing, without which its hard to accept the paper, but if you could go back and fix it, you might have a great paper for the conference. In our conference format, this type of back-and-forth is usually not possible. This motivated writing this post, so that newcomers to the field have a chance to know in advance what a potential reviewer might look for in an algorithmic bioinformatics conference paper.

What do I mean by algorithmic bioinformatics conference paper? I am thinking of the subset of papers submitted to RECOMB/WABI/ISMB that take an algorithm-based approach to solving a bioinformatics problem. This is largely intended to contrast against papers more rooted in statistical methodology, where the standards are a bit different. I also focus on conference reviews, where the process is a bit different than for a journal. When reviewing a paper for a bioinformatics journal like Oxford Bioinformatics, there is of course an opportunity for the authors to address any limitations in a revision.

I want to also add a disclaimer that this is not in any way an official statement about what PC members would look for in a review. As far as I know, there is no such official policy, and the things that each PC member looks for do not completely overlap. There is a diversity of standards and that is why each paper has multiple PC members reviewing it. I cannot speak for others, though I hope that people can add their comments and feedback.

When I review, the first thing I try to identify is: what is the main novel contribution of the paper? Is it an idea, a theorem, an algorithm, or a tool (i.e. software) people can use? Sometimes a paper has all these components, but not all of them contribute to the novelty of the paper. Here are some examples:

1) The paper contains an algorithm and a tool that implements the algorithm. The algorithm itself may be a simple modification of what is previously known, but the algorithm is implemented in a novel software tool for an important biological problem. If the tool performance is an improvement over previous tools, then the tool is the main contribution. 

2) In another example, the main novelty is in the algorithm or in its analysis, and this is what the reader is intended to take away from the paper. The paper may have implemented a tool, but the intention of the tool is to only be a prototype to test the feasibility of the idea. The tool is not the main contribution.

3) Sometimes, the main contribution of the paper is novel biological findings, without any methodological (either algorithmic or software) novelty. This is not really within the scope of the RECOMB/ISMB†/WABI conferences, which has to be methodological. Certainly, having novel biological findings can serve to demonstrate the strength of the methodological contribution. But if you discover a cure for cancer by applying existing software, then it is probably outside the scope of RECOMB/ISMB†/WABI.

It is up to the authors to make the main contribution of the paper crystal clear to the reader. As a reviewer, I will then base my evaluation on what the authors claim. If the authors’ claim is not clearly stated, then I will do my best to guess what it is. But if I make a mistake, then I may end up evaluating the paper from a completely incorrect angle.

Here are some common issues I find with papers. This is not intended to be an exhaustive list and only includes issues that are both basic and that I’ve seen multiple times. In a competitive venue, a paper is usually accepted based on its strengths rather than a lack of weaknesses; however, in my experience, the weaknesses below typically ruin a paper’s chance of acceptance.

Context within prior algorithmic work is not given:  A common scenario where this happens is when the authors developed a method for a particular biological dataset, and there are no other tools designed specifically for this kind of dataset or problem. However, the problem and/or solution might be very similar to what has been previously studied. For instance, many problems come down to clustering of some data points (e.g. genes in a network or reads from a sequencing experiment) or to some version of sequence alignment. The algorithmic context of such a paper is, at least in part, clustering or, respectively, alignment algorithms. Sometimes the authors provide the biological context (e.g. what is the relationship to previous approaches to finding genes in a network) but leave out the algorithmic one (e.g. what is the relationship to previous clustering algorithms). Why is this particular problem or dataset different enough so that standard clustering or alignment techniques do not apply? If the authors present a clustering algorithm for the problem but do not answer this question in the intro, then their contribution is not placed in the algorithmic context — which makes it hard to evaluate its novelty.

Unclear writing: Some papers will contain many spelling and grammatical mistakes, or ambiguous notation and terminology.  I try to do the best I can to understand the contribution of the paper, and often I do understand it in spite of these problems. In such cases, it does not greatly influence my overall decision about the paper, and I generally trust the authors to clean up the paper before publication (if it is accepted). In other cases, I cannot understand the paper after a reasonable amount of time trying. In these cases, I simply cannot evaluate the paper’s contribution.

The paper is written in the style of a biology journal: In biology journals, the methods section is often written as a step-by-step manual necessary to reproduce the results (i.e. a pipeline of processing steps on the data). This type of presentation focuses on implementation details and reproducibility rather than highlighting the novelty of the algorithm. Even if the method is novel, when it is written in this style it is hard for the reader to identify and understand the novel parts. Another aspect of this is that for a biology journal, the results section comes before the methods section. Doing this for an algorithmic bioinformatics paper is not in it of itself a problem, but it usually correlates with not enough focus being given to the method.

Claims in the intro that are not supported by the rest of the paper: For example, the authors claim that their tool is the fastest to-date for a problem, but the results section only contains a comparison against one other tool or only on a narrow type of data. In such cases, I simply ask the authors to tone down their claims. However, sometimes the claims are central to the claimed importance of the paper, in which case this feels a bit disingenuous. Another example is the bait-and-switch, when the intro claims that the paper presents an algorithm for some interesting problem. But, what ends up being evaluated in the results is an algorithm for a slightly different problem. 

There is neither a strong theoretical contribution nor an experimental evaluation: Some contributions are theoretical — a powerful idea, a way of thinking about a problem, or a theorem which can be applied by other algorithm developers. These papers require a lot of work on the modeling or theoretical side, and it can be justifiable if experimental results are either not included or limited. However, in most other cases, experimental evaluation is essential to a paper. If this is missing or is inappropriate to the problem, it can make it impossible to evaluate the strength of the contribution.

There is no comparison against other work: The authors sometimes find it obvious that their method should work much better than anything else out there. They may be right, but it is important to demonstrate this in the paper by finding the most compelling alternative approach and comparing against it. 

Software: If the main contribution of the paper is a tool, then the tool should be usable. At the very least, I should be able to download the software, install it, and run it on a toy input that is provided in the download.  If I can see that the tool already has some users (e.g. through GitHub activity), then this is enough to demonstrate its usability and I may not bother to try it out myself. On the other hand, if the paper contains a tool that is only a prototype and is not the main contribution, then the usability of the software is not something I consider very important.

Correctness: Sometimes the authors present an algorithm or data structure for which they prove the correctness, or it is obvious through the construction. For example, it could be a data structure to represent and query some data. However, when they evaluate their tool for its e.g. runtime, it is still essential that the correctness of the algorithm is explicitly verified in the experiments. This can be a simple one line that says: e.g. we verified that the new data structure gives the same answers to queries as the previous one on all the evaluated datasets. However, without this check, how does the reader know that the algorithm is not twice as fast as the competition just because it has a bug?

No analysis of running time or memory usage: In most cases, it is important for an algorithmic bioinformatics paper to present the running time and memory usage of the algorithm, either through experimental evaluation and/or theoretical analysis. This is a very natural thing to do for computer scientists, but I sometimes find that researchers with a different background forget to include this. In other cases, the authors do not include any memory or time analysis because they know that it is tiny and besides the main point, but it may not be at all obvious to the reader. In such cases, a simple statement to the effect that the memory usage or running time is negligible would suffice. 

† ISMB is very diverse, with many different types of tracks and presentations. This blog only refers to those papers focusing on methods. Certain tracks may also feature papers focusing on the biology.

[ Subscribe to new posts via RSS feed. ]