acm sigcomm

The IMC Steering Committee has spent the last few months considering changing the conference to double blind reviewing. In doing so, we considered not only the recent literature on the subject, but also thoughts from many community members. We have had many conversations and exchanged many emails on this topic. Given the interest in this topic within the community, we wanted to distill the salient points of our discussion to give the community a better sense of the SC's thinking on the issue.

• We do not believe large scale, overt bias exists in the IMC review process.
• However, there is general evidence that suggests subconscious biases play a role when reviewing papers that include author identity. While we do not have direct evidence about how this pertains to IMC, we find it difficult to believe IMC would be immune from such biases.
• There is a subset of the IMC community that strongly views IMC's traditional use of single blind reviewing to be problematic.
• We found the idea of simply asking authors to anonymize their submissions problematic in the IMC context due to the fairly prevalent use of 'unique capabilities', such as apps, measurement platforms, datasets, vantage points, etc. In some instances, anonymizing these will degrade a submission because it cannot leverage previous work. On the other hand, naming them will not allow for anonymization.
• While small groups can anonymize papers under a double blind approach, large organizations can be unnamed, but often not anonymous (e.g., "a large CDN", "a global video content provider", etc.). Some community members from small groups feel this provides a reputational benefit to large players that they are not afforded when using some well-honed system that they have spent years developing.
• There is a subset of the community that strongly views not being able to reference and lean on a line of their previous work to be problematic.
• The opinions within the community---and in fact within the SC!---of the benefits and costs of double blind reviewing are fairly disparate. The community is not of a single mind on this issue.
• In general, conferences in IMC-related areas are moving towards double blind reviewing. While the SC was not compelled to simply "follow the crowd", we recognize that IMC could at some point suffer a reputational cost for continuing to use single blind reviewing in the absence of strong consensus within our community for single blind reviewing.
• Conversations about single-vs-double blind reviewing can readily animate some in the community. Our goal---which we believe we (mostly) achieved---was to have a dispassionate conversation about the pros and cons of various approaches.

Given the above, we found no clear consensus within the SC for single or double blind reviewing. This lead us to look for a middle ground, which we believe we have found. IMC 2019 will use double blind reviewing. However, crucially, there is an explicit exception for naming 'unique capabilities'. This is explained in detail---with examples---in the CFP [1] and therefore we do not repeat it here. This is how some community members have traditionally viewed double blind. I.e., that anonymization need not be absolute. However, this view is not universal and therefore we find it important to explicitly state this exception in the CFP so that authors and the program committee are all on the same page.
 
We hope the approach we have taken helps largely alleviate any subconscious biases present in IMC's process, while at the same time allowing researchers to leverage their long-standing measurement facilities. Our goal is to make the processes better, not perfect. We will assess the approach after the IMC 2019 process concludes and make changes as necessary for future years. As always, community feedback and input is welcome and encouraged.
 
IMC Steering Committee 
Mark Allman, Olivier Bonaventure, Anja Feldmann, Dina Papagiannaki, Darryl Veitch 
April 2019
[1] https://conferences.sigcomm.org/imc/2019/anon/