To be honest, I lack experience in serving as an effective reviews, especially regarding filds outside my expertise. However, in most cases, the qualities of a good reviewer mirrior those of a good writer. Last week, my supervisor deliverd a crucial lesson for all PhD students, or at least for me. It came as a superise because I had assumed that my supervisor had been detached from the lines of academia for a long time, given that he rarely writes papers himself. Yet, he still maintains a keen instution for research.
The lesson centered on what makes a good paper, and the discussion eventually shifs to what defines a good review.
What makes a good paper ?
My supervisor provided servral points.
Among them, an important one was the PROBLEM. Whatever the problem is new or old, the problem paper addresses must be fundamental and meaningful. In my view, this implies that the paper should ideally aim to solve the real challanges in industry. Once the problem is connected to the real-word needs, the paper has taken a promising first step. As you might imagine, this relies heavily on the authors’ sense for industry. You must process a deep understanding of current and emarging industrial trends, which requires a long preiod of accumulation. However，it is regrettable to observe that an increasing number of Chinese researchers today tend to focus on niche problem, primarily to avoid the growing competition within the same research track. I believe that this phphenomenon is largely from the fact that the Chinese academic evolution system still requires further refinement.
PRESENTATION. Even if you have a fundamental idea, without excellent presentation, the paper will not be readable and impressive. Presentation is not just about the colors you use or the chart styles you design; it also emphasizes the logical rigor of the narrative and the readability of the storytelling. Some authors deliberately try to make their papers unreadable by using complex logic or convoluted sentences, even when the proposed method is simple. They may believe that this trick can elevates their work, creating an illusion of sophistication for reviewers unfamiliar with the field. Actually, this is a quite risky and controversial practice. If the reviewer is an expert in the field, this trick will be seen through，and may even be met with distaste by the reviewer.
NOVELTY. This is perhaps the most difficult goal to achieve. After all, most work is incremental. Even if a solution is simple but effective, it can still be challanged by reviewers. Sometimes the results may not be impressive, yet the work can still process novelty. I believe that at times, there is no need to be overly demanding. In certain fields, particularly those that are engineering-oriented, the protential for novelty is inherently limited. Honestly, I am too young to talk about novelty, I still have much to learn.
EXPERIMENT. Precisely, this refer to experiment excution. No matter how impressive the preceding sections of paper may be, every claim must ultimately be grounded in data, to substantiate the contributions and conclusions. In this phase, we should carefully treat every detail. For instance, one must carefully consider how to select baselines, how many to include, and how to ensure fairness in the comparison. Another crucial point is the selection of metrics. My supervisor believes that sometimes we should not overcomplicate charts by including multiple meaningless metrics. That’s because reviewers generally want to extract most impressive insight from a chart from a glance, once the the chart is overly complex, for example, crammed with numerous metrics to showcase SOTA performance, reviewers may be lose their patience even ignore the truly impotant metrics buried with data.
Once you know how to be a good writer, then you will know how to be a good reviewer.
What makes a good review ?
As RPG (Research PostGraduate) students, we find ourselves on both sides of the process: receiving reviews from conferences or journals, and writing reviews to evaluate the works of others. Ironically, sometimes we get angry because we recesive some “strange” reviews lead to the rejection of our own papers. In this lesson, my supervisor outlined three factors that constitute a good review.
When we are asked to review a paper and give some feedback, we are expected to write serval parts in the review, typically covering Summary, Strengths, Weaknesses and Final Evaluation. Whatever the part you are writing, the first rule is that the content should not be too brief. Some reviewers on OpenReview just write down a few sentences, which clearly indicates a lack of serious engagement. This is unacceptable practice.
Finally, most of the time we provide a “weak accept” or “weak reject”. We seldom strongly accept or firmly reject a paper. I believe this’s because we lack EVIDENCE to support our decisions. Therefore, evidence is of vital importance in a review, but how do we find the evidence by following some rules ? The answer is mainly from the part we talk about previously like ploblem, presentation, novelty, and experiment. Of course, we must also consider about the ranking of the venue. For instance, if the paper is summited to a CCF-C venue, reasonably I might relax certain strict critiria like the requirement of novelty.
Secondly, the review you provide should be EFFECTIVE and meaningful. This implies that the review must be constructive. When the authors receive your review, they should be able to gain valuable insights and use them to improve the paper. We should focus on the main idea and reasonably forgive some minor error. Nickpicking trivial issues does not help the paper, nor is it rhe conduct of a mature researcher.
Conclusion
We have discussed the qualities that define a good paper and a good review, as well as the connection between the two. A good paper stands on a meaningful problem, clear presentation, genuine novelty, and solid experiments. A good review, in turn, is built upon evidence drawn from these same dimensions, and should be constructive enough to genuinely help the authors improve their work. Ultimately, being a good reviewer and being a good writer are two sides of the same coin.