SIGCOMM Artifacts Evaluation Committee

SIGCOMM Artifacts Evaluation Committee

 

A growing fraction of the papers accepted by Computer Communication Review and the SIGCOMM-sponsored conferences includes artifacts such as simulation models, measurement datasets, software implementations, etc. These artifacts are an essential part of many of these papers, and artifacts’ availability encourages other researchers to build upon and extend previous results. The ACM SIGCOMM community showed to be sensible to artifacts availability and research reproducibility, as proven by the ACM SIGCOMM Reproducibility Workshop. Also, a survey conducted among the authors of the accepted papers at SIGCOMM’17, CoNext’17, IMC’17, and ICN’17 revealed that a significant fraction of the accepted papers contained artifacts.

ACM has created a badging system to assess the quality of these artifacts. Similarly to Artifacts Evaluation Committees (AEC) created in several SIGs or conferences, the ACM SIGCOMM Artifacts Evaluation Committee evaluates the artifacts of accepted papers and assign badges to these papers.

The objective of the committee is to assign badges to the papers accepted at the SIGCOMM-sponsored conferences. ACM has defined a range of badges that can be associated to papers. The evaluation will focus on three of these badges :

  • Artifacts Available: this badge indicates that author-created artifacts relevant to this paper have been placed on a publically accessible archival repository.

  • Artifacts Evaluated – Functional:  this badge indicates that the artifacts associated with the research are found to be documented, consistent, complete, exercisable, and include appropriate evidence of verification and validation.

  • Artifacts Evaluated – Reusable:  this badge indicates that the artifacts associated with the paper are of a quality that significantly exceeds minimal functionality.

Submission instructions

Every paper of 6 pages or more that has been published or accepted by CCR or any of the conferences sponsored by ACM SIGCOMM in 2018 can be submitted to the ACM SIGCOMM Artifact Evaluation Committee.

To submit their request for artifacts evaluation, authors will submit a revised version of their accepted paper that contains besides the full paper,  pointers to their publicly-available artifacts and in the appendix additional information to help the artifacts reviewers (e.g., release notes or installation guidelines for software, description of metadata for datasets…).

Paper registration and submission wil be done via HotCRP at:  https://sigcomm18ae.hotcrp.com.

Artefact Evaluation Committee

By nature artifacts are various and technical and their evaluation is complex and potentially time consuming. In order to provide the highest possible evaluation quality, the AEC continuously look for reviewers that will be ready to dedicate some of their time in reviewing artifacts and propose badging. We are looking for specialists from all fields covered by the SIGCOMM community.

 

If you would like to volunteer to be considered to serve on the Artefact Evaluation Committee, please complete this form.

Artifacts Review Process (ARP)

The Artifacts reviews are mainly conducted by members of the AEC, but the AEC chairs may decide to request external reviewers if the required expertise is not present in the AEC. The review process is single-blind but Artifacts reviewers may communicate anonymously (e.g., through a rebuttal) with the authors during the evaluation if they have practical problems when exercising the artifacts. Each artifact is reviewed by at least two reviewers who propose a badge according to the quality of the artifact (Artifacts Evaluated – Reusable being this highest one). The comments of the reviewers are returned anonymously to the authors if the paper receives an Artifacts Available badge or if the AEC decides to not assign a badge to the paper. If the reviewers decide to assign either an Artifacts Evaluated – Functional or an Artifacts Evaluated – Reusable  badge, they write together a public review for the evaluated artifact.

 

The AEC summarises the lessons learned from the evaluation of the submitted artifacts in an editorial that will be published in the January issue of CCR with all public reviews and the complete list of badged papers.

 

When possible,  the ACM Digital Library is updated to attach the assigned badges and the public reviews to all badged papers.

Important dates

  • September 13, 2018: Publication of the final Artefact Evaluation Committee.

  • September 13, 2018: Opening of submission site.

  • October 13, 2018: Submission deadline.

  • December 10, 2018: Artifacts reviews are returned to authors and AEC members prepare public reviews.

  • January 2019: publication of the AEC report in Computer Communication Review and publications of the badges.