Should we reject papers with bogus artifacts?

Yesterday I blogged about the accepted artifacts at ISSTA. I think it is worthwhile noting that out of the ten papers that got accepted and which had artifacts submitted there seven for which the artifacts checked out. That is a good thing!

What worries me, however, are the three papers for which Artifact evaluation failed. For those three papers, we were largely unable to reproduce their results, and yet the papers made it into the program. Moreover, for two of those three, the fact that they failed artifact evaluation was already known before the PC meeting, i.e., there would have been a chance to reject them.

The reason for why the PC did not is that only positive reviews were taken into account this time, in order not to discourage people from submitting artifacts in the future. We as a community should really think about whether we cannot find a way to make artifact evaluation the default so that people have no other chance than to submit all the evidence they have to back up their claims.

ICSE Distinguished Reviewer Award

I was very pleasantly surprised when I received the ICSE Distinguished Reviewer Award today, especially given that out of the 19 papers I reviewed only two were finally accepted. I guess there must also be a helpful way to reject papers after all. So thanks a lot to the ICSE authors for their positive vote! And more thanks go also to Alexandre Bartel, Mauro Baluda, Philipp Holzinger, Siegfried Rasthofer, Stephan Huber and Steven Arzt for assisting my reviews. Thanks a lot guys, we all share this award!

Boomerang accepted at ECOOP 2016

We are happy to announce, that a paper on our new algorithm for demand-driven context- and flow-sensitive points-to analysis, called Boomerang, has been accepted at the ECOOP 2016 conference. Download the paper here.

Boomerang is the first points-to analysis algorithm which on demand delivers in addition to computing points-to sets also the reverse information. Along with an allocation site, the analysis delivers all possible pointers in the current scope which point-to that particular allocation site. This feature is crucial for state-of-the-art clients such as taint and typestate analysis. On top of that, Boomerang enables client-driven context-resolution: The client can limit the search scope for the points-to analysis to the methods of interest.

For the evaluation we introduce PointerBench, the first benchmark suite to evaluate precision and soundness of points-to analyses. We hope for contributions to further enhance the benchmark suite and make points-to analyses comparable more easily.

Boomerang also received the artifact evaluation award.