acm sigcomm

SIGCOMM Program Committee, Best Current Practice. This is an informal and incomplete description of current accepted practice of the SIGCOMM Program Committee.

This is an informal and incomplete description of current accepted  practice of the SIGCOMM Program Committee. That does not mean that  nothing in this process can be changed by new Program Co-Chairs.  However, that does mean that there should not be gratuitous changes  in overall procedures from year to year, and that significant  departures from accepted procedures should be made only after  discussion with representatives from the SIGCOMM Executive Committee  and/or the Technical Advisory Committee.

Selecting the Program Committee: Best Current Practices for PC composition and the PC selection process are summarized here.

The Call for Papers:  The Call for Papers should clearly state that all manuscripts be  original material that has not been previously published nor is  currently under review by another conference or journal. It seems  that every year, there are some number of papers that are detected as being substantially similar to a paper under review elsewhere.  In such cases, the PC chairs should contact the authors and inquire  about the level of overlap. A useful point of view is to ask the  author to consider how the contribution of the submitted sigcomm  paper would be rated if all material under submission elsewhere were removed.

The Call for Papers should also state when the papers will be  available on-line (usually 2 weeks pre-conference via the DL). Future Program Co-Chairs might have to deal  with authors filing for a patent, for whom this advance posting of papers is problematic; this issue has not yet arised to date. This is the clause that should be included in the CFP:

AUTHORS TAKE NOTE: The official publication date is the date the proceedings are made available in the ACM Digital Library. This date may be up to two weeks prior to the first day of your conference. The official publication date affects the deadline for any patent filings related to published work. (For those rare conferences whose proceedings are published in the ACM Digital Library after the conference is over, the official publication date remains the first day of the conference.)

Assigning papers:  The list of paper titles and abstracts for submitted papers is sent
to the PC, and each PC member returns a list of the papers that  they are and are not qualified to review. From these lists, the  Program Co-Chairs have complete authority to assign papers to PC members as they see fit. Each paper is assigned to at least two  PC members to review, and more typically three, with additional
reviews if at least one PC member rates the paper acceptable.  (The only constraint is that papers will not be assigned to PC  members with conflicts, as described below.) The Program Co-Chairs also sometimes solicit outside reviews, for reviewers who are not members of the PC.

External reviewers:  Several PCs have used external reviewers for additional second-round reviews. This is particularly critical for those papers where the PC lacks the necessary expertise to evaluate the paper.

Paper Conflicts for PC members:  Members of the PC are allowed to submit papers. PC members are considered to have a conflict with papers they have coauthored,  with papers coauthored by members of their home institution, and with any other papers for which they consider themselves to have a conflict. When papers are discussed in the PC, PC members with conflicts must first be asked to leave the room. (This has the unavoidable side-effect of conveying information at that time to
the rest of the PC about the institutional affiliations of the authors of the paper.)

Paper Conflicts for Program Co-Chairs:  Best Current Practices for handling paper conflicts for Program Co-Chairs are summarized here.

Reviewing papers:  Papers will generally be reviewed by the PC members themselves.  In some cases, the PC member might get another reviewer from their
own institution to review a paper assigned to them, particularly  when that person is more qualified to review that particular paper.  However, in all cases the PC member will read all papers assigned  to them, and be responsible for all reviews, and be prepared to discuss all papers assigned to them at the PC meeting. Reviews by  students are strongly discouraged, unless the student reviewers  are very senior and expert in the field.

Paper reviewing is typically handled as a two-step process, with  a first phase of reviews (typically three reviews per paper) followed by a second phase of email discussion, identification of the need for additional reviews, and obtaining additional reviews.

The Early Decision category: The Early Decision category was initiated in SIGCOMM 2001. Each paper is assigned to a Lead Reviewer, and in the first week the
Lead Reviewer can propose a paper to the Co-Chairs for a Early Decision. Papers can be proposed for Early Decision because they are out of scope or clearly below traditional SIGCOMM standards. Papers approved by the Co-Chairs for Early Decision are rejected without further detailed reviews.

Responsible PC members: Each paper has a PC member assigned responsibility for that paper. The assigned PC member leads the discussion at the PC
meeting (if the paper is discussed at the PC meeting). The assigned PC member is also responsible for coordinating email discussion of that paper among the reviewers before the PC meeting, if such a discussion is deemed necessary.

Double-blind reviewing: The review process is double-blind (at least since 1994), meaning that the authors names are not included on the submission itself.
Outside SIGCOMM, authors are free to make their papers publicly available as they see fit. Thus, it is possible that some PC members will in fact know the authors' identity for a paper that they are reviewing, if the authors have chosen to make that paper publicly available.

Rebuttals: Rebuttals were initiated in SIGCOMM 2001, with first-round reviews  forwarded to the authors for a brief feedback for correcting errors or misunderstandings. There was a wide range of opinions at the end of the PC meeting about the usefulness of this process for future years. Perhaps authors will have feedback about the usefulness of the rebuttal process for them also.

Access to papers and to reviews: All PC members will have access to all submitted papers and to all reviews, for all papers for which they do not have a conflict, well
before the PC meeting.

Discussing papers before the PC meeting: Many of the SIGCOMM submissions receive extensive email discussion among the assigned PC members in the month before the PC meeting. The Lead Reviewer for the paper is responsible for coordinating the email discussion of the paper. The purpose of this process is
to clarify and possibly resolve disagreements among the reviewers, identify papers that need additional reviews, and identify papers that can be recommended for Reject without discussion at the PC meeting. This process is a double-edge sword and a process that needs to be carefully managed, i.e., by asking PC members to err
on the side of caution and be ready to resurrect papers at the meeting.

Discussions in the PC meeting: All members of the PC are free to read, comment on, and submit a review for any of the papers. However, in the group discussion
the assigned PC members for a paper will be identified, and the assigned PC members will open the discussion of that paper. The Program Co-chairs might restrict a PC member's access to non-assigned papers until that member has submitted reviews for all assigned papers.

What papers are discussed in the PC meeting: No paper will be accepted without first being discussed in the PC meeting. However, some papers might be rejected without first being discussed in the PC meeting. For any paper that is on the list to be rejected without discussion, any PC member is free to add that paper to the list of papers to be discussed (this is done via private communication to the Program Co-Chairs). Papers on the list to be rejected without discussion are declared as such in advance of the PC meeting, so PC members can have some time to consider whether or not they want a particular paper (that is a candidate for non-discussion) to be discussed. After some stage during the PC meeting itself, the Program Co-Chairs can decree that papers can only be added to the to-be-discussed list if there is at least one PC member who is ready to argue that that paper should be accepted.

Discussions in the PC meeting: The PC meeting generally starts with the papers separated into two categories, the Reject category and the Discuss category. The papers in the Reject category will not be discussed at the PC meeting, unless one of the PC members requests otherwise.

The calibration phase of the PC meeting: The discussion of papers begins with a calibration period of discussing some of the highest-and lowest-ranked Discuss papers. This calibration period is sometimes done by discussing a number of the strongest Discuss papers that are likely to be quickly accepted, and a number of the weakest Discuss papers that might be quickly rejected, spending only a short time on each one. Each paper is classified as Accept, Reject, or Accept If Room, or is left in the Discuss category to be discussed again later in the meeting.

The second phase of the PC meeting: After the calibration period, the papers in the Discuss category are discussed. Papers discussed during the meeting are ranked as Accept, Reject, or Accept If Room (+/-), or are left in the Discuss list to be revisited later in the day.

This discussion has often been done in two passes, with a first pass spending a limited time on each paper, and completing by the middle of the afternoon. At this point many papers should be ranked as Accept, Accept If Room, or Reject, with papers that need extended discussion still in the Discuss category. One advantage of the two-stage process is that the discussion of the more difficult or controversial papers can proceed with a more clear idea of the constraints in terms of the likely number of accepted papers.

There have been a range of procedures for the ordering chosen for the discussion phase. The SIGCOMM 2001 PC used numerical order for the discussion process, based on paper number.

PC members who are absent from the PC meeting: It is expected that all PC members will attend the PC meeting. However, every year there are unavoidably a few PC members who are not able to attend the meeting, due to illness or other pressing business. Special care must be taken with papers whose assigned PC members are not at the PC meeting. The prior email discussion of papers by the assigned PC members can help deal with this to some extent. Phone conferencing is sometimes attempted for absent PC members, with varying degrees of success.

Accepting papers: The numbers from the reviews are never used to make accept decisions; accept decisions are made only on the basis of the discussion at the PC meeting. The numbers from the reviews are sometimes used to essentially make a reject decision, by putting a paper on the list of papers not scheduled for discussion at the meeting itself.

Shepherding: Some papers might be accepted subject to shepherding. That is, the paper is accepted conditionally, and is included in the program only if the authors agree to make specific changes. (These changes must be changes that it can be reasonably expected that the authors can accomplish; the condition can not be that the authors come up with a correct proof for their difficult theorem.) For shepherded
papers, a PC member is assigned to be shepherd, and that PC member determines whether the authors have made the requested changes. The list of accepted papers is not announced until the authors have agreed to have their paper be shepherded. The authors have the option of instead withdrawing their paper from submission. To date, all shepherded papers have been shepherded successfully, in that they have actually appeared in the final program.

On-line access: All accepted papers are made freely available on-line shortly after  the final camera-ready copy is submitted by the authors.