As announced in the previous issue, this is the last issue of Computer Communication Review to be printed on paper. It could become a collector in a few years, so keep it in a safe place once you've read it of course.
In practice, for experimental papers, describing all the experiments in enough details to ensure that they can be completely reproduced can be challenging given our page limits.
The ACM publications board has recently discussed this problem and came up with an interesting classification that applies to experimental papers. This classification provides a precise definition for three words: Repeatability, Replicability and Reproducibility that could be considered as synonyms by nonnative English speakers like me.
Starting from the July 2016 issue, CCR will only be available online. The papers will be archived on the ACM Digital Library. They will also be posted on http://www.sigcomm.org. We are exploring other delivery methods to improve your online reading experience. We hope that having an online publication will allow us to better serve the community.
The first level is Repeatability. A measurement described in an article is considered to be repeatable if the same team can obtain the same results with the same setup in multiple trials. For an experimental paper, this implies that the software used for the experiment produces the same results multiple times. This is the basic level and we expect that all CCR papers are repeatable.
This issue contains three technical contributions and two editorials. “On the Interplay of Link-Flooding Attacks and Traffic Engineering” discusses a specific type of denial of service attack where an attacker tries to disconnect some targets by overloading key links in their target’s neighborhood. “Attacking NTP's Authenticated Broadcast Mode” analyses the security problems that can occur when the Network Time Protocol (NTP) is used in broadcast mode. “Paxos Made Switch-y” proposes an implementation of the Paxos distributed consensus protocol in P4. The first editorial, “Global Measurements: Practice and Experience (Report on Dagstuhl Seminar #16012)” summarizes the lessons learned from a recent workshop on global Internet measurements. The second editorial, “Towards Considering Relationships between Values and Networks” looks at the interactions between human rights and the technology that we develop. It reminds us that when we decide to carry out research on a given topic, our research results may have a broader impact than simply a series of papers published in conference proceedings, journals or online libraries. Some of our work can influence, in one direction or another, the evolution of our society and some of our design choices may have a huge impact in the long term. I encourage you to read this editorial and then take some time to think about your ongoing work and the impact that it could have on values such as human
Repeatability, Replicability & Reproducibility
Scientific papers such as those published in CCR are expected to contain enough information to allow other researchers to obtain similar results. This is what differentiates scientific publications to blog posts or articles that appear in trade magazines.
The second level is Replicability. An article is considered to be replicable if a different team than the authors of the paper can obtain the same results as those stated in the paper by using the same software, datasets, etc. as those used for the paper. Replication of research results is obviously facilitated if the artifacts used to write the paper are available. The ACM Digital Library provides a permanent storage for all the papers published in CCR and our conferences. In addition to storing pdf versions of the articles and the associated metadata, it is now possible to associate artifacts to each published article. These artifacts contain additional material related to the article such as datasets, proofs for some theorems, multimedia sequences, software (source code or binaries), … These artifacts are important to ease the replication and the reproduction of our published research results. Some papers already include links to author's web pages for some of these artifacts. However, these links are rarely permanent and they often disappear after a few months or years. Starting with this issue of CCR, the authors of accepted papers will be encouraged to provide artifacts that will be linked to the paper in the ACM Digital Library. Two papers published in this issue already provide such artifacts.
The third level is Reproducibility. An article is considered to be reproducible if an independent group can implement the solution described in a paper and obtain similar results as those described in the paper without using the paper's artifacts. Reproducing an experimental paper is not a simple job since it often requires an engineering effort to implement the software used for all experiments. However, it is a very important step in the validation of new scientific results. As a community, we do not frequently encourage the reproduction of previous articles since we usually focus on original results. I believe that we could also learn a lot from articles that reproduce important results. I hope that future CCR issues will contain such papers.
Last year’s reviewers
CCR heavily depends on reviewers who agree to spend time to comment submitted papers. Their feedback is often very detailed and it clearly contributes to the quality of the papers that you read. While preparing this editorial, I checked the submission site and found that last year 150 members of our community agreed to review one or more papers for CCR: Cedric Adjih, Mohamed Ahmed, Mark Allman, Luigi Atzori, Brice Augustuin, Ihsan Ayyub Qazi, Jingwen Bai, Aruna Balasubramanian, Nicola Baldo, Sujata Banerjee, Theophilus Benson, Robert Beverly, Nevil Brownlee, Ed Bugnion, Giovanna Carofiglio, Antonio Carzaniga, Pedro Casas, Kai Chen, Chih-Chuan Cheng, David Choffnes, Antonio Cianfrani, Jon Crowcroft, Italo Cunha, Alberto Dainotti, Lara Deek, Shuo Deng, Luca Deri, Xenofontas Dimitropoulos, Ning Ding, Yongsheng Ding, Nandita Dukkipati, Alessandro Finamore, Davide Frey, Timur Friedman, Xinwen Fu, Erol Gelenbe, Aaron Gember-Jacobson, Minas Gjoka, Lukasz Golab, Andrea Goldsmith, Sergey Gorbunov, Tim Griffin, Arjun Guha, Saikat Guha, Deniz Gunduz, Chuanxiong Guo, Berk Gurakan, Gonna Gursun, Hamed Haddadi, Emir Halepovic, Sangjin Han, David Hay, Oliver Hohlfeld, Shengchun Huang, Asim Jamshed, R.C. Jin, Abdul Kabbani, Michalis Kallitsis, Naga Katta, Ethan Katz-Bassett, Eric Keller, Manjur Kolhar, Balachander Krishnamurthy, Kun Tan, Mirja Kuhlewind, Anh Le, Jungwoo Lee, Zhenhua Liu, Matthew Luckie, Sajjad Ahmad Madani, Olaf Maennel, John Maheswaran, Petri Mahonen, Saverio Mascolo, Deepak Merugu, Jelena Mirkovic, Vishal Misra, Radhika Mittal, Tal Mizrahi, Amitav Mukherjee, Dragos Niculescu, Nick Nikiforakis, Dave Oran, Chiara Orsini, Patrick P. C. Lee, Christos Papadopoulos, Dimitris Papadopoulos, Craig Partridge, Peter Peresini, Ben Pfaff, Guillaume Pierre, David Plonka, Ingmar Poese, Lucian Popa, Ihsan Qazi, Zafar Qazi, Feng Qian, Costin Raiciu, Bhaskaran Raman, Fernando Ramos, Ashwin Rao, Ravishankar Ravindran, Mark Reitblatt, James Roberts, Franziska Roesner, Dario Rossi, Michele Rossi, Mario Sanchez, Stuart Schechter, Fabian Schneider, Julius Schulz-Zander, Sayandeep Sen, Soumya Sen, Zubair Shafiq, Craig Shue, Georgos Siganos, Georgios Smaragdakis, Joel Sommers, Alex Sprintson, Stephen Strowes, Srikanth Sundaresan, Muhammad Talha Naeem Qureshi, Vamsi Talla, Boon Thau Loo, Brian Trammel, Martino Trevisan, Narseo Vallina-Rodriguez, Roland van Rijswijk-Deij, Matteo Varvello, Aravindan Vijayaraghavan, Stefano Vissicchio, Ashish Vulimiri, Mythili Vutukuru, Nick Weaver, Michael Welzl, James Westall, Erik Wilde, Walter Willinger, Craig Wills, Rolf Winter, Bernard Wong, Wenfei Wu, Matthias Wählisch, Di Xie, Teck Yoong Chai, Yan Zhang and Haitao Zhao. As you can see from this list, producing CCR relies on the efforts of a large number of members of our community. Our Associate Editors selected these reviewers. They usually serve for a period of three years. Dr. Hitesh Ballani has finished his tenure. I would like to thank him for all the efforts he put in handling CCR papers and I welcome Prof. Costin Raiciu from University Politehnica of Bucharest (Romania) who joins our Editorial board.