I’m a good reviewer!

Actually, that is not true. I am an outstanding reviewer! Or so say the Association for the Advancement of Artificial Intelligence and the AAAI-21 Program Committee.

If I’m perfectly honest, I was quite insecure about reviewing for AAAI. It was my very first time as a PC member, and I was not quite sure that I was knowledgeable enough to be able to be sufficiently critical of the contents of a paper and to recognise flaws, especially those related to novelty and obscure theorems that have pages-long proofs.

My colleagues encouraged me, by reminding me that I had read and graded roughly 150 student papers/reports/theses already. For many of those, the feedback I provided led to new and improved versions, and in some cases even to scientific publications. All in all, my colleagues reckoned I was ready.

I then simply did the best I could. I did not receive any feedback from the AAAI-21 Program Committee, so I can only guess why they picked me among the thirteen outstanding PC members out of all those thousands. Just in case you are reading this looking for tips on how to be a good reviewer, here are my recommendations for reviewing anything, be it a scientific paper submitted for publication, or a student report for an (under)graduate course.

Be specific

All good feedback is actionable. For example: instead of saying “Section 4 is unclear”, say “I would find it easier to understand Section 4 if concept X was formally defined.” Note that this also reflects the fact that your feedback is your opinion, not fact. It also indicates what made Section 4 unclear to you, and what would improve Section 4’s clarity for you.

If authors have the opportunity to write a rebuttal, it is helpful if you number your points of critique and your questions. They can then easily use the numbers to refer to and respond to these critiques and questions, saving valuable rebuttal space. Conversely, you can easily find their responses and check if they have addressed all your points.

It is also helpful if you indicate, at least to your fellow reviewers, but maybe also to the authors, what you took into account in your final assessment of the work. Especially if a paper ultimately is rejected, it is good to make explicit that this was, e.g., because the quality of the experiments was lacking and not because there were too many typos.

Be supportive

I firmly believe that if I help you thrive, it makes me thrive, too. I want my students to do well, because a) their success reflects well on me and on my university, and b) it’s simply much more fun if they succeed than if they fail.

Similarly, I want the papers at the conferences where I (try to) publish my work to be of high quality. I genuinely tried to come up with feedback that would make the next version of the work I was reviewing better, regardless of whether it would be accepted or not.

Naturally, it does not hurt to also compliment the authors on anything they did particularly well. We all struggle to get our work published, so encouragement and appreciation are always welcome. Furthermore, you will also benefit if people learn what makes a paper interesting or pleasant to read for you, so you can also compliment authors for thoroughly selfish reasons!

Be curious

Reviewing papers is a great opportunity to learn, which can be a great motivator to put some serious effort into it. You get a glimpse into the latest advances in your (or a related) field, which I trust you find just as exciting as I do.

On top of that, you can also learn a lot about other people’s writing process. While reviewing, ask yourself if the language and the structure in the paper works for you. Are there helpful analogies? Are the acronyms clearly explained? Is there a particular framing that you find compelling, or quite the opposite? Take notes and learn!

Be honest

Especially if it is your first time reviewing, you may not be familiar enough with all the material to assess its quality. Try to learn as you go, but also don’t overwork yourself. If you have reached the end of your knowledge, just be open about that to your fellow reviewers and area chair. Maybe they can fill you in, or at least take into account that your knowledge is lacking.

You may also not fully understand the process or the conventions, especially because not all of them are always clearly communicated. It is absolutely okay to indicate to the other reviewers and the area chair that you are unsure if a certain paper is out of scope or if certain conditions (like anonymity) are not met, as long as you indicate how and where you have looked for the answer and what the result was of that search. You may not be the only one who is confused, but you also do not want to cause a paper to be rejected just because something was not communicated clearly.

Be confident

Out of all the people they could have asked to review this paper, they chose you. You are part of the community. You have knowledge. You have ideas. You have a vision. You are qualified to give it your best shot. You have a lot to offer, so show it to them.

You are not the only reviewer and it is okay to disagree with the others. If, during the discussion, you find that somebody makes a good argument, let yourself be pursuaded and admit that you have changed your mind. Being able to change your mind when presented with compelling new information is a strength, not a weakness. If the new information is not compelling to you, it is also perfectly fine to not change your mind. You are entitled to your opinion.

Be a benevolent critic

This is really the summary of the points above. It is a phrase that prof. Holger Hoos (my PI) likes to use, and I find it very helpful. We also use it in our group, when we give each other feedback on our work. With any feedback I give, I ask myself if I am indeed being critical enough, and not slacking off or shying away from uncomfortable truths. Additionally, I ask if the feedback I give is constructive and respectful, and indeed likely to help the other person succeed, instead of demotivating them or dragging them down.

In these days of crazy reviewing load, we are grateful to have exceptional colleagues in our community: @thserra @aldlatour @tchakra2 @kr_t https://t.co/L3mO1Re7Hn

— Kuldeep S. Meel (@ksmeel) February 4, 2021

The award came as a complete surprise to me, if not to others. As one of my colleagues put it: “You cannot do anything without winning an award for it, can you?” Clearly that is untrue, but I am not going to deny that I wasn’t a bit flattered to hear that. Ultimately, it is nice to be appreciated by your community.

Congratulations to @aldlatour for being recognized as an outstanding reviewer at #AAAI2021! Well done! https://t.co/MQOK1sYPBl

— Siegfried Nijssen (@SGRNijssen) February 4, 2021

While I was away from social media, I missed this great achievement by Anna Latour @aldlatour :
She was one of the 13 outstanding reviewers (out of thousands) in AAAI'21. https://t.co/MlpAaz4MrZ

— Behrouz Babaki (@BehrouzBabaki) August 26, 2021