anna latour

anna latour

research fellow. computer scientist. feminist.

I’m a good reviewer!

7 minute read

Published:

Actually, that is not true. I am an outstanding reviewer! Or so say the Association for the Advancement of Artificial Intelligence and the AAAI-21 Program Committee.

img

If I’m perfectly honest, I was quite insecure about reviewing for AAAI. It was my very first time as a PC member, and I was not quite sure that I was knowledgeable enough to be able to be sufficiently critical of the contents of a paper and to recognise flaws, especially those related to novelty and obscure theorems that have pages-long proofs.

My colleagues encouraged me, by reminding me that I had read and graded roughly 150 student papers/reports/theses already. For many of those, the feedback I provided led to new and improved versions, and in some cases even to scientific publications. All in all, my colleagues reckoned I was ready.

I then simply did the best I could. I did not receive any feedback from the AAAI-21 Program Committee, so I can only guess why they picked me among the thirteen outstanding PC members out of all those thousands. Just in case you are reading this looking for tips on how to be a good reviewer, here are my recommendations for reviewing anything, be it a scientific paper submitted for publication, or a student report for an (under)graduate course.

Be specific

All good feedback is actionable. For example: instead of saying “Section 4 is unclear”, say “I would find it easier to understand Section 4 if concept X was formally defined.” Note that this also reflects the fact that your feedback is your opinion, not fact. It also indicates what made Section 4 unclear to you, and what would improve Section 4’s clarity for you.

If authors have the opportunity to write a rebuttal, it is helpful if you number your points of critique and your questions. They can then easily use the numbers to refer to and respond to these critiques and questions, saving valuable rebuttal space. Conversely, you can easily find their responses and check if they have addressed all your points.

It is also helpful if you indicate, at least to your fellow reviewers, but maybe also to the authors, what you took into account in your final assessment of the work. Especially if a paper ultimately is rejected, it is good to make explicit that this was, e.g., because the quality of the experiments was lacking and not because there were too many typos.

Be supportive

I firmly believe that if I help you thrive, it makes me thrive, too. I want my students to do well, because a) their success reflects well on me and on my university, and b) it’s simply much more fun if they succeed than if they fail.

Similarly, I want the papers at the conferences where I (try to) publish my work to be of high quality. I genuinely tried to come up with feedback that would make the next version of the work I was reviewing better, regardless of whether it would be accepted or not.

Naturally, it does not hurt to also compliment the authors on anything they did particularly well. We all struggle to get our work published, so encouragement and appreciation are always welcome. Furthermore, you will also benefit if people learn what makes a paper interesting or pleasant to read for you, so you can also compliment authors for thoroughly selfish reasons!

Be curious

Reviewing papers is a great opportunity to learn, which can be a great motivator to put some serious effort into it. You get a glimpse into the latest advances in your (or a related) field, which I trust you find just as exciting as I do.

On top of that, you can also learn a lot about other people’s writing process. While reviewing, ask yourself if the language and the structure in the paper works for you. Are there helpful analogies? Are the acronyms clearly explained? Is there a particular framing that you find compelling, or quite the opposite? Take notes and learn!

Be honest

Especially if it is your first time reviewing, you may not be familiar enough with all the material to assess its quality. Try to learn as you go, but also don’t overwork yourself. If you have reached the end of your knowledge, just be open about that to your fellow reviewers and area chair. Maybe they can fill you in, or at least take into account that your knowledge is lacking.

You may also not fully understand the process or the conventions, especially because not all of them are always clearly communicated. It is absolutely okay to indicate to the other reviewers and the area chair that you are unsure if a certain paper is out of scope or if certain conditions (like anonymity) are not met, as long as you indicate how and where you have looked for the answer and what the result was of that search. You may not be the only one who is confused, but you also do not want to cause a paper to be rejected just because something was not communicated clearly.

Be confident

Out of all the people they could have asked to review this paper, they chose you. You are part of the community. You have knowledge. You have ideas. You have a vision. You are qualified to give it your best shot. You have a lot to offer, so show it to them.

You are not the only reviewer and it is okay to disagree with the others. If, during the discussion, you find that somebody makes a good argument, let yourself be pursuaded and admit that you have changed your mind. Being able to change your mind when presented with compelling new information is a strength, not a weakness. If the new information is not compelling to you, it is also perfectly fine to not change your mind. You are entitled to your opinion.

Be a benevolent critic

This is really the summary of the points above. It is a phrase that prof. Holger Hoos (my PI) likes to use, and I find it very helpful. We also use it in our group, when we give each other feedback on our work. With any feedback I give, I ask myself if I am indeed being critical enough, and not slacking off or shying away from uncomfortable truths. Additionally, I ask if the feedback I give is constructive and respectful, and indeed likely to help the other person succeed, instead of demotivating them or dragging them down.

The award came as a complete surprise to me, if not to others. As one of my colleagues put it: “You cannot do anything without winning an award for it, can you?” Clearly that is untrue, but I am not going to deny that I wasn’t a bit flattered to hear that. Ultimately, it is nice to be appreciated by your community.

You May Also Enjoy

Moon milk

12 minute read

Published:

The first conferences I ever attended as a doctorate student were IJCAI 2017 and CP 2017, which both took place in Melbourne. At the time, I had a position as a visiting researcher at ICTEAM at Université catholique de Louvain. After the conference, three fellow doctorate students from UC Louvain and I travelled to Tasmania, for a two-and-a-half-day trip. It resulted in one of my favourite travel stories of all time.

Introducing gismo: a tool for finding grouped independent supports

30 minute read

Published:

My first paper since joining Prof. Dr. Kuldeep Meel’s group at the National University of Singapore (NUS) has been accepted to IJCAI 2023! The topic of this paper was a completely new direction for me, and I learned a lot during this project. I had the great privilege to work with Kuldeep and with Prof. Dr. Arunabha Sen for Arizona State University (ASU) on this project, and learned a lot from each of them. In this blog post, I will share a bit about the problem that we solve, why it needs solving, and how we solve it, introducing our new tool: gismo.

Books that helped me through my doctorate

6 minute read

Published:

Two months ago, I defended my dissertation. I am very grateful to my advisors, my doctorate committee, my opposition committee, and my paranimphs. I thought long and hard about which thank-you presents would be appropriate and of value to them. In the end, I just decided to present them with some wisdom.

It takes a village

8 minute read

Published:

As a Master student, I once asked Siegfried if I should use ‘‘I’’ or ‘‘we’’ in my thesis. After all, the words were written by me, but we did the research together. He explained that, even if you are the sole author of a text, you should always write ‘‘we’’, because no piece of research ever is the product of just one person. I believed him then, but the experience of being a PhD candidate drove the lesson home, never to be forgotten. In this blog post, however, I am truly ‘‘I’’, and use it to thank those that make us ‘‘we’’.