# POPL 2023 Reviewers Guide

You can access your reviewing assignments at:

https://popl23.hotcrp.com/

Each of you is LEAD reviewer (called Discussion Lead in Hotcrp) for 3-4 papers. You'll see an "L" next to reviews on which you're Lead.

# Timeline

* Jul 9 to Jul 12 - Paper bidding; please submit at least 35 bids
* July 26 - indicate your expected expertise on each paper you're
assigned by posting a comment for that paper. Please spend 15 minutes
going through each paper to estimate your expertise. (See instructions
below on how to leave a comment before submitting a review.)
* July 29 - Lead reviewers: if we are not confident that the paper will
have at least one X and one Y review, the Lead reviewer should post
(as a comment) an ordered list of 3 suggested external expert
reviewers.
* Aug 22 AoE - Review Deadline for 7 out of 11 of your reviews -- these
7 should include all the papers on which you are Lead.
* Sep 4 AoE - All Reviews due (**hard** deadline)!
* Sep 6-9 - Author response period
* Sep 10-19 - Online Discussion, during which we will decide which
papers to reject; the rest will be discussed during the Zoom PC
meetings.
* Sept 13 - each reviewer must have offered at least one comment on each papers (taking the author response and other reviews into account)
* Sept 21-22 - Zoom PC meetings (see below for exact times)
* Sept 26th - All meta-reviews must be complete
* Sept 27 - Notification of conditional acceptance
* Oct 26-Nov 7 - Revision check, with Final Accept notification by
Nov 7.

# Checking Revisions

Do an initial check by **Nov 2nd** in case authors need to make additional adjustments. The revision check deadline is Nov 6th.

1. All reviewers should check the revisions and leave a comment saying what they are satisfied and not satisfied with. The Discussion Lead is responsible for doing a detailed check to ensure that all Mandatory revisions have been made and, as necessary, prodding the other reviewers (both PC and external) to check changes that they had requested.

2. Between now and November 6, reviewers can use hotcrp comments to discuss revisions with authors and ask for further changes that they'd like the authors to make. If you make such requests, please make it clear to authors that you'd like them to submit a revised version that addresses the request. (If the further changes are minor and you don't need to see another revision, just ask the authors to make the change in the camera-ready version.)

3. Please aim to do an initial check of the revisions and provide a first response to authors by November 2nd as authors will need some time to make any further changes you suggest.

4. Note that Conditional Accept really does mean *conditional* -- if authors have not made the requested mandatory revisions to your satisfaction, and don't manage to address your concerns during this 12-day period, then we should reject the paper.

5. Once all reviewers of a paper are happy with the revisions, the Decision Lead should leave a comment tag: #final-accept or #final-reject. (Note: click on Tags icon on lower left below the comment box.) The Associate Chairs and I will change the paper decision from ConditionallyAccepted to Accept or Reject as appropriate.

The following email was sent to authors:

Dear authors,

The revised version of your paper is 
due Wednesday, October 26, 2022,  AOE.

Please submit the revised version as a comment at:
  https://popl23.hotcrp.com/

Specifically, submit a comment with:
(1) the detailed description of the changes you have made 
    (as body of comment, not an attachment)
(2) the revised version of the paper (attach file to comment)
(3) a document showing the changes you have made relative
   to the original submission, produced using
   latexdiff or a similar tool (attach file to comment)

Here are the salient revision instructions from my last email:

- The revised version should NOT be anonymous.

- You may purchase up to 4 additional pages for $100
  per page for the final version, which would bring
  the page limit to 29 pages, not including references.

- Final acceptance is contingent on implementing the 
  mandatory revisions. If you do not make satisfactory
  revisions, the paper will be rejected. We trust that
  you will make the changes proposed in your response
  (if any), and will seriously consider the suggestions
  made in the reviews as you revise your paper.

- Your revised version should be accompanied by a
  detailed description of the changes you have made
  (document (3) in list above). If some expected
  changes were not implemented, please provide a
  justification.

- You may use HotCRP comments to ask reviewers for
  clarification about mandatory or suggested revisions, 
  but please do so sparingly.

- Decisions about final acceptance will be made by
  November 7, 2022.

# Post PC Meetings: 2 Action Items

- **ACTION ITEM 1** Please remember to update your review score to reflect your sentiment at the end of the PC meeting discussion. This is especially important for papers being rejected so authors see the sentiment shift, post response and post discussion, reflected in the score change.

As a reminder, the scores are not on a linear order, but AB means you prefer acceptance and CD means you don't; AD means you have a strong opinion and BC means you don't.

- **ACTION ITEM 2** Please complete Meta-Reviews by Sept 26th AOE.

- Discussion Leads should draft the meta-review and prod the other reviewers to give input and feedback. If the Lead is not expert enough, they should still be the one to prod one of the other reviewers to write it and make sure it gets written.
- Make the meta-review an author-visible comment -- for now you can edit author-visible comments as much as you like since authors cannot see them until a switch is flipped in hotcrp (which will happen as the notifications are sent out).
- Francois, Robby, and Amal will be doing a final read-through of meta-reviews and tagging the papers #ready based on the state of the meta reviews.

Notifications: Amal plans to send out notifications on **Sept 27th at 10am EDT**.

### Guidance on writing Meta-Reviews:

- For Rejected papers: Capture the key points and advice from the discussion that followed the author response. Summarize what we thought of the paper (strengths and weaknesses) and why it was rejected. If the paper was discussed at the PC meeting and then rejected, say it was discussed and summarize the main points of the discussion and reason for the decision. **If you believe a rejected paper has a future**, then add some encouraging words to the authors saying why and spill some ink to help them improve the work or the paper. Keep the tone positive or neutral. Avoid phrasing that sounds disparaging or overly critical.

- For Conditionally Accepted papers: The meta-review should consist of a Revision Summary that clearly identifies MANDATORY and SUGGESTED revisions, ideally in separate sections with those headings. Make sure you write the required changes in a way to make them easy to check later. When appropriate, the meta-review should also capture key points of the discussion from the comments and the PC meeting.

Let's give the authors some good final feedback. Thank you all so much for all the time and energy you've put into this year's POPL!

# Zoom PC Meetings

The plan is to have an initial discussion of all papers on Day 1 and table those that need further discussion; Day 1 should help us all get calibrated on where the bar is for this year's POPL. We'll return to tabled papers on Day 2.

These meeting times are written in the US East Coast timezone; you should have calendar invites to translate to your local timezone)

- September 21st: [zoom link](https://northeastern.zoom.us/j/97974964296?pwd=bnBKKzViRzk4cnhrRU43TGpPNEdWdz09)
- 10am-12pm EDT: I hope everyone will join this regardless of time
zone, so we can get started together
- 12:30pm-4:30pm EDT: For Americas and Europe. PC members in Asia will be
asleep
- 7pm-10pm EDT: For Americas and Asia. PC members in Europe will be asleep

- September 22nd: [zoom link](https://northeastern.zoom.us/j/93823937220?pwd=bnEwQkY1WnlkeXRseXNWZlBZRkRQQT09)
- 9am-12pm EDT: Everyone should join, but west coasters can join by
10:30am EDT
- 12:30pm-4pm EDT: For Americas and Europe. PC members in Asia will be asleep

* **Discussion Leads have changed** For papers being discussed at the PC meeting, we have assigned the most positive PC reviewer as the Lead. Please check NOW and make a note of which #discuss papers you are Discussion Lead for.

* **Please try to give *every* paper at least a tilt** before the PC meeting -- either 'probable accept' or 'probable decline'.

Adding a tilt to every paper is useful because good decisions are hard to make on the spot while on the clock at the PC meeting. You already have most, if not all, of the information you're going to get about a given paper -- pushing the decision off, even a really hard one, isn't very useful. At some point, we are going to have to bite the bullet, so bite the bullet today by giving the paper a tilt. If you find that while attending the PC meeting that you have "miscalibrated" and been either harsher or more lenient then you can certainly change your "tilt" (i.e., decision) during the meeting. If you find the meeting hasn't indicated you have "miscalibrated" then just stick with the default tilt. The discussions will go faster this way. We'll also avoid (some of) the problems of mistreating papers that come late in the PC meeting when people are tired and decisions become more random.

* We will discuss ALL papers in the running for a Conditional Accept decision. The plan is to try to discuss all papers on Wednesday, while **tabling papers that we can't converge on after several minutes of discussion**. Tabled papers will be discussed on Thursday.

* Meetings when papers will be discussed (assuming you have not sent me constraints):

- Papers whose Discussion Lead is based in Europe, India, Israel, and (often) North America will be discussed in Wednesday PC Meetings 1 or 2 (10am-12pm and 12:30pm-4:30pm US Eastern time). These will be tagged #discussW1 and #discussW2 (sometime on Sept 20th).

- Papers whose Lead is in Asia and (sometimes) North America will be discussed in Wednesday PC Meeting 3 (7pm-10pm US Eastern time). These will be tagged #discussW3 (sometime on Sept 20th).

- Papers whose discussion is tabled on Wednesday will be discussed in Thursday PC Meetings 1 and 2 (9am-12pm and 12:30pm-4pm US Eastern time). These will be tagged #discussT1 and #discussT2 before Thursday.

* Discussion of most papers on Wednesday should ideally be 2-5 minutes. If we can't reach agreement after 8 minutes or so, the paper will be tabled.

* Meeting tracker: During the meeting, I will run a **meeting tracker on HotCRP**. If you are logged into HotCRP you'll see a bar at the top telling you which paper we are currently discussing, and which two are coming up next **if** you don't have a conflict with them.

* **Conflicts**: When the tracker moves to a new paper, if you are a reviewer/Lead, please do not start talking right away. I will first ask all conflicted PC members to move to the "Conflicts" breakout room, and then give the go ahead to start discussion once all conflicts have left.

* We have **275 submissions**. We have about 100 papers to discuss at the PC meeting. We should conditionally accept roughly 65-70ish. But there are many more papers in contention so having a champion on a paper does not mean we will accept.

* I will send out the exact discussion order Tuesday evening (US Eastern time) as several papers are still being discussed.

* The PC meetings are on Zoom and the calendar invitations contain the Zoom link for each meeting. There is one Zoom link for all the Wednesday meetings and a different Zoom link for the Thursday meetings (as above). NOTE: **You have to be logged into a Zoom account to join the meeting.**

### The Timetable / Plan for Each Paper during the Zoom Meeting
If you are the Discussion Lead for a paper being discussed, you should
be prepared to:

- Take 1 minute to summarize the paper's contributions.

- If there is already consensus to accept, take 1 additional minute to tell us why we should accept the paper and summarize the views of all reviewers (or share the minute with them and have them tell us).

- For papers where we do not have consensus, we will take 5-8 minutes to discuss the pros and cons and to hear from other reviewers of the paper and other PC members.

[ Note: Other PC members should not hesitate to interject with questions or comments -- PC meetings are valuable because they allow us to benefit from the expertise and perspectives of other committee members. But, of course, let's be cognizant of time. ]

- If we do not converge at the end of 8 minutes, the paper will be tabled. We will put reviewers in a breakout room to discuss further and/or they should continue the discussion online overnight. We will bring back the paper for discussion on Thursday.

Note that time will be tight. Please be prepared to summarize the paper and the opinions of the other reviewers in the allotted time. It would also help if all reviewers can think beforehand if they currently advocate Accept or Reject, and why.

# Lead Reviewer Duties

* check that there is 1+ expert (ideally, at least an X and a Y) on each paper and suggest external reviewers by July 29;
* read reviews as they come in and suggest changes if warranted;
* lead online discussion and present the paper during zoom discussion;
* write a Meta-Review (summary) of all discussions for the authors; see the more detailed guidance below

# Making Decisions on Papers FAQ

## How many papers will we accept?

In the interest of transparency, I (Amal) want to answer this question so everyone goes into the PC meeting with the right mindset: I've been asked to aim for 23% to 25% acceptance (which would be 63 to 70 papers). That said, I do not want to set a quota. The point of the PC meeting is to help us calibrate. If we see during our discussion that we have spectacular papers, I'm fine with accepting more, and if we don't, then we'll accept less. But what we want to ensure is that we are making good decisions that seem fair across the board, that we aren't rejecting or accepting because we run out of time or attention.

## Will papers be Accepted or Conditionally Accepted?

- All 'acceptance' decisions sent out on Sept 27 will be Conditional
Accept and will be accompanied by a meta-review (provided as an
author-visible comment) that will specify Mandatory and Suggested
Revisions.

- Authors have to submit Revisions by Oct 26, which the reviewers will
check.

- This will be a REAL conditional accept: if the authors do not do the
Mandatory revisions to the reviewers' satisfaction, we will reject.

## What are "reasonable" revisions to ask for? If revisions are too
extensive, should we Discuss at PC meeting?

For a Conditional Accept:
- the Mandatory changes should be clearly stated, easy for the authors to perform in the next 4 weeks (i.e., by Oct 26), and easy for us to check
- it should be the case that we do truly want to accept the paper (i.e., we are not publishing a list of required changes in the hope that the authors will not be able to make these changes)

Please discuss if the revisions you want to see are too extensive to do
in the next month, or if the outcome of additional work you're asking
for is uncertain. If so, then let's not discuss at the PC meeting since
our time will be limited.

## Will there be 'Shepherds' for conditionally accepted papers? Who will
check revisions? What if the external reviewer is the expert?

No Shepherds. Often conferences assign a PC member to "shepherd" a
conditionally accepted paper with the expectation that this will be the
single person responsible for checking revisions. We will not have a
"shephard" designation. The changes we request are a collective effort
and having only one reviewer check them doesn't really work well.
Instead, we will ask the Discussion Lead to make sure that when the
revisions come in, all reviewers -- including our external experts --
check the paper to make sure their requested changes have been addressed
to their satisfaction. Authors will submit revised versions as
attachments to HotCRP comments, and HotCRP has excellent support for
reviewers and authors to anonymously discuss/suggest changes. Once all
reviewers have signed off, the Lead should let the chairs know that we
can move to final accept.

## Can an external expert please attend the PC meeting?

Sorry, no. External reviewers cannot attend the PC meeting. The
Discussion Lead is responsible for adequately understanding the external
reviewer's opinion and summarizing it for the committee.

## How will PC papers be handled?

Your PC invitation said that PC submissions "will be accepted if they
have sufficient support and no strong detractors". At the moment there
are very (very!) few PC papers with both a champion and a strong
detractor. The Chairs will monitor PC-paper discussions and if there is
a strong detractor, we will let the reviewers know that this is a PC
paper (but not who the authors are) so they can take that into account.
PC papers will be discussed at the PC meeting in the standard manner
along with all other papers.

# Writing Meta-Reviews

## Mechanics/process:

- Draft a comment with the meta-review -- ideally put "Meta-Review" (for
rejected papers) or "Revision Summary" (for accepted papers) on top.
- The other reviewers should provide feedback, say what they want
included, and sign off on the final version.
- Once all reviewers are happy, the Lead should make the comment Author
Visible (by Sept 26 AOE).

## Guidance on writing Meta-Reviews:

- For rejected papers: Capture the key points and advice from the
discussion that followed the author response. Summarize what we thought
of the paper (strengths and weaknesses) and why it was rejected. If the
paper is discussed at the PC meeting and then rejected, say it was
discussed and summarize the main points of the discussion and reason for
the decision. Give the authors some final advice on how to improve the
work or the paper. Keep the tone positive or neutral. Avoid phrasing
that sounds disparaging or overly critical.

- For conditionally accepted papers (which you'll write after the PC
meetings): The meta-review should consist of a Revision Summary that
clearly identifies MANDATORY and SUGGESTED revisions. It should also
capture key points of the discussion from the comments and the PC
meeting.

# Online Discussion Phase

Discussion Leads: please ensure reviewers have arrived at a consensus
by Sept 19 and post a comment for the paper chair. Chairs will mark
the paper reject or discuss.

- Please change your score to reflect how you feel about the paper
after the discussion and the author response. (If you lower your
score, please be sure to update your review to tell the authors why.)

- Note: Any paper with at least one A will be discussed at the
PC meeting, as well as papers with all Bs (unless the reviewers
agree otherwise).

NOTES ON PROCESS:

1. All Acceptance decisions will be made during the Zoom PC meetings.
No paper will be marked Accept without discussion at the PC meeting.

2. Meta-Reviews: During the online discussion period, the chairs will
assign one reviewer the task of writing a meta-review for the paper.
This will either be the Discussion Lead or the most expert reviewer.
If the authors did not submit an author response, we will not
provide a meta-review.

**Meta-reviews** will have to be completed by Sept 26 since decisions
will be sent out on Sept 27. When writing a meta-review:

- For Rejected papers: Capture key points and advice from the
post-author-response discussion (online and/or at PC meeting).
If the submission has worthwhile ideas, give the authors some
final advice on how to improve the work.
- For Accepted papers: Write a Revision Summary that clearly
identifies MANDATORY and SUGGESTED revisions. Also, capture key
points of the discussion from the comments and the PC meeting.

# Process for Soliciting External Reviewers

* Wait until all assigned reviewers for the paper have indicated their expected expertise (July 26th)
* Please invite an external expert only if we do **not** have: one expected X and one expected Y. (If a reviewer indicates they'll be X or Y, then let's be conservative and count that as a Y. Similarly, count a Y or Z as a Z.)
* If we need an external expert, please post an ordered list of 3 external reviewers by July 29. Ask the other reviewers for suggestions if you (the Lead) are not an expert.
* Use the Invite link on the paper page to submit the request for the external review. The chair responsible for this paper will be asked to approve or deny the review request -- they'll check for conflicts before approving.
* If the request is denied, please ask the next potential external reviewer on your list.
* When you submit a request for an external review in HotCRP, please write a personalized note that makes it clear that you are looking for an expert review, or if you aren't looking for an X review, say what aspect of the paper you'd like their expert opinion on. Note that the generated message already provides a review deadline of Sept 2nd. Here is a very basic (not sufficiently personalized) template you can tweak for your note:

Hi _external person_, We really need an expert review on this POPL submission and I think you'd be able to provide one. We'd really appreciate it if you could review the paper.

* Please keep track of *unanswered* requests, which tend to be, sadly, common. Email the reviewer (not using hotcrp) if we haven't heard back in 3-4 _(this said 5-7 days before, but that's too long)_ days and post a comment saying you've pinged them. The chairs will also keep an eye on this -- we know many PC members have vacation plans over the summer so we will ping the reviewer if the Lead hasn't and can invite the next reviewer on your list. If the external doesn't reply in a couple of days, please invite the next reviewer.

* If the external reviewer accepts/rejects, you will immediately receive an email. So if you haven't received an email, then the reviewer has not responded yet. In HotCRP, if you see things like "Review (not started)", this means that the request is still pending. Once the reviewer has accepted, it will show up as "Review (started)"

* Please limit external reviewer disucssion comments using the "PC visibility" setting in hotcrp.

# Writing Good Reviews

Please use the advice below as a guide as you work on your reviews. We would like you to reflect on the **bad reasons to reject** in the note below and avoid similar arguments in your reviews. On the other hand, we should not shy away from **good reasons to reject**. Additionally, we should avoid **bad style and tone** in our reviews as discussed below. As reviewers, while we are asked to be gatekeepers for what is published, our primary goal is to use the reviews to improve the quality of research in the community. Thus we have a responsibility to provide thoughtful and reasonable arguments for and against.

We ask you to do two things:
1. Write each negative-leaning review so it explains what the authors should fix/change to turn this paper into one that would be acceptable. The changes you suggest should be realistic (e.g., doable within the page limits) and reasonable (i.e., given normal practice for that kind of paper). Of course, there will be papers that you might think should never be accepted (see *good reasons to reject* below) but for the rest, please provide constructive feedback.
2. Read the other reviews for each of your papers and speak up if you think the review doesn't quite live up to the "good reviewing" standards we are trying to encourage. Use the comments to discuss with the other reviewers and encourage them to improve their argument against or reconsider their evaluation.

----
### Good and Bad Reasons to Reject Papers
[This section is largely derived from a note by Peter Sewell dated 2021-11-01]

As reviewers, what do we have to decide? Fundamentally, whether publishing the paper will advance the subject in some substantial way. In more detail:

- is the motivation real - does the paper address an important problem?
- would the claims it makes constitute substantial progress?
- are those claims backed up - is it technically solid?
- is it well-written - enough for readers (with the appropriate background) to understand?

Then, as our venues are typically competitive, we have to weigh the paper against other submissions (how competitive they should be is a question, but we won't go into that here) - so reviewers need some sense of the level of contribution appropriate to the venue, so that their scores are broadly comparable.

#### Bad Reasons to Reject Good PL Papers

Reviewing is essentially a judgement call. We've discussed our review processes at great length over the years, and those processes do matter - we've tuned them, and I think in many ways improved them - but, fundamentally, peer review relies on informed judgements from a suitably expert and sensible group of people. So this note is not about process. Instead, it identifies some of the bad forms of argument that one sees again and again. If one sees one of these, or if one finds oneself writing one, an alarm bell should ring...

- I could have done this better, if only I'd got round to it
- I can imagine some quite different research that I'd prefer
- I can imagine some quite different exposition that I'd prefer
- It's not self-contained/accessible to me, because I don't know the work it builds on
- It's not self-contained, because this project is too big for all the details to fit in the page limit
- I want more examples / discussion (fitting into the page limit by magic)
- I want extra evaluation (even though it does a decent job to support the claims)
- I just wasn't excited by it (even though it's a clear advance on an important problem)
- I'm assessing this as if it was about X, even though it's actually about Y
- I'm assessing this as if it was a paper of kind X, even though it's actually of kind Y (PL spans many kinds of papers, with different values and criteria)
- It's about language design
- It's too mathematical
- It's about the semantics of actual languages, which makes it complicated
- They didn't mechanise all the proofs (though they didn't claim that they did)
- A previous paper claimed to do this (though it doesn't really subsume this)
- It could do with another pass (and the authors will thank us for rejecting it)
- It presents a big project, not a single clever/cute idea that can be fully explained in a few pages
- The idea here is too simple (even though it's very useful, and no-one fleshed it out and published it before)
- It's incremental (even though it's a big increment - most research is advancing previous work)
- This feels more like a paper for venue X (even though it could perfectly well fit here)
- This should be a journal paper instead (for good and ill, PL is based on conference publication)
- The authors already put a version on the web
- I'm working on a competing project
- (and finally, the classic) It doesn't cite my paper

Many of these boil down to having due respect for the authors and the work they've put in. Remember, they've typically spent between one and ten person-years on this, while the reviewer has spent maybe a day. We're not awarding prizes for effort, and sometimes a reviewer will understand things better than the authors, but one should be wary as a reviewer of trying to require substantially different research or exposition.

Of course, none of them are absolutes - even the last reason above can be a legitimate complaint in specific circumstances, e.g. if that uncited paper renders the submitted work moot.

Another bad reason arises during discussion, after the first reviews have been written. At the end of the process, one has to arrive at accept/reject decisions, but during the process it's all too easy to regard the current scores as an objective assessment, e.g. saying "this is a `B' paper". The whole point of the discussion is to consider whether reviews are wrong or miscalibrated - otherwise we'd just order papers by the original scores.

#### Good Reasons to Reject Bad PL Papers

On the other side, not all papers are good, unfortunately, and we shouldn't shy away from rejecting poor-quality work, lest the subject be contaminated by bogosity. Returning to the above list, in order of decreasing importance:

- is the motivation real - does the paper address an important problem? (sometimes, simply identifying an important problem is a major contribution)
- would the claims it makes, if true, constitute substantial progress?
- are those claims backed up - is it technically solid?
- is it well-written - enough for readers (with the appropriate background) to understand?

A clear "no" for any of these should rule the paper out from any serious venue. In more detail:

- The motivation isn't explained - it doesn't clearly explain why anyone should care
- The motivational argument is bogus
- The work is technically correct but pointless (basically a rephrasing of the above)
- The claims (presuming they are substantiated) wouldn't significantly advance the subject (it really is a minor increment over previous work)
- ... or, so far, really insufficiently developed for this venue
- It really has been done before
- The claims are misleading: the work is over-sold and the authors aren't clear about the limitations, or about the relationship to previous work
- The claims are unsubstantiated: it doesn't give the actual proofs or data, without a good reason why not
- The claims are unsubstantiated: the evaluation is too limited or too flawed to support the claims
- It is substantially less rigorous (theoretically or experimentally) than normal practice in the area
- It's technically wrong (and isn't straightforwardly fixable)
- The exposition is so bad that it's hard (even for an informed reader) to understand what the authors have actually done

When arguing that a paper should be rejected, or summarising a PC decision for the authors, it may be useful to identify exactly which of these (or other) reasons justify that.

---

### Bad Style and Tone in Reviews

As a matter of respect for the authors, we should be mindful of how we write reviews. In particular, avoid the following kinds of bad style and tone in reviews:

- References to the authors' lack of understanding or knowledge of something, rather than to what is written in the paper: The review's subject is the submitted paper, not its authors.
- Praise or criticism of other work that the submission cites, builds upon or is related to: the related work is not subject to review.
- Tangential criticism in passing in the middle of a different point. This can confuse authors. If the tangential criticism is worth mentioning at all, write it as an explicit point with constructive suggestions.
- Sarcastic feedback or comments.

# Additional Guidance

### Paper Chairs

Each paper has one of four tags: #amal #deepak #francois #robby. These indicate which one of the four chairs is keeping track of that paper. The chair responsible will monitor external review requests, read the reviews as they come in, and guide online discussion and decision-making as needed. Each chair gets an email when you post a comment for a paper in their set.

### Format and Anonymity

All papers have been checked for format; both papers and supplementary material have been checked for anonymity. If you see something we missed, please contact the chairs as soon as you can.

Note that some authors have included an appendix with the main paper instead of submitting it as supplementary material. While the latter is preferable, the CFP does not disallow an appendix. But please treat the appendix as supplementary, i.e., you are not required to read it.

### Submit reviews as you complete them

Submit your reviews as you complete them since that allows the chairs to spot cases where we lack confidence and request additional external or PC reviews, and also gives them an opportunity to engage with reviewers and suggest improvements before author response.

### Sub-reviews by PhD students and postdocs

All requests for external reviews must go through hotcrp and be approved or sent by the chairs who will check for conflicts before a request is sent. However, you may ask one of your PhD students or postdocs -- whose conflicts you've checked are a subset of yours -- to help as a sub-reviewer. While we count on PC members to review all the papers assigned to them, having PhD students help with a review is an important part of PhD training. Please ask the student to write a draft review, then work with them to edit and expand into your own review. (Use the Comment for PC section to indicate who the sub-reviewer is.)

### How to submit a comment *before* you've submitted your review:

To indicate your expected expertise for each paper, please leave a Comment WITHOUT submitting your review -- i.e., DO NOT submit an empty review just to indicate expertise. Here's how:

When you click on a paper, you see the "Edit Review" section (where you'll eventually enter your review). But DO NOT hit Submit Review. Instead, scroll to the very bottom of the page and click on "Main". That will show you a different view where, at the bottom of the page,
you'll see an "Add Comment" button. Use that to leave a comment with you expertise.

Note that the visibility of that comment is set correctly by default to "Reviewer discussion" about "submission" -- leave those settings as they are.

A quicker way to submit comments (such as the expected expertise) before reviews, is to follow the links from https://popl23.hotcrp.com/search?q=&t=rout (after logging in).

### Please unsubscribe from ArXiv notifications and Google Scholar notifications

Authors are welcome to put their draft papers on arXiv so please either add a rule to your mailer that puts them aside for now or unsubscribe to arXiv notifications entirely.