NeurIPS24 review results were criticized by the whole network, alarming LeCun! Reviewers gave low scores and seriously belittled the contribution of the paper
2024-08-10
한어Русский языкEnglishFrançaisIndonesianSanskrit日本語DeutschPortuguêsΕλληνικάespañolItalianoSuomalainenLatina
【New Wisdom Introduction】As soon as the review results of NeurIPS 2024 came out, it became a hot topic of complaints across the entire network.
Every year, there are always big events at the top international conferences in the field of AI.
Some time ago, the NeurIPS 2024 paper review results were finally released, and netizens started to "form groups" and argue.At this time, some netizens broke the black material:"Some reviewers seriously downplay the contribution of the paper, use unreasonable standards to judge it, and even ignore the actual improvement of the paper compared to other work."
He believes that this may be the author of a competing product, who made a dishonest review after discovering the issue.Even LeCun came to watch and immediately forwarded the post.As the area chair, I tell authors: If you encounter this situation, please contact the area chair (AC).As a writer, I ask: How do I appropriately point out this behavior and find the balance between legitimate criticism and excessive gatekeeping?
Another netizen released a classic comment from a NeurIPS reviewer: This solution is not new and has been extensively studied.This year, the 38th annual NeurIPS conference will be held in Vancouver from December 9th to 15th.According to the data statistics collected by Paper Copilot, as of now, the average score distribution is mostly between 3 and 6 points.Whose fault is it that the paper received a low score?NeurIPS papers have low scores?
An associate professor from UT Austin, who is also one of the area chairs (AC) of this year's NeurIPS conference, explained the phenomena he had seen.He said that among the batch of papers he was responsible for, 16 out of 48 had been submitted for review.So far I haven't seen a single positive review, either the reviewers have become very strict or the quality of the papers has declined?
This year, NeurIPS received a lot of "recycled" papers.Another AI researcher similarly said that either the quality of the paper has declined, or I am the hated Reviewer No. 2.Among the papers he reviewed, he basically gave them scores of 2-4.Another reviewer said, "Among the 62 papers I was responsible for, the average score was 4.3 points and the standard deviation was 1. So don't be discouraged if you get a low score at first!"It can be seen that the authors of the papers at this NeurIPS conference are having a hard time, and it is not easy to get high scores.Large model participation review
Moreover, after AI became popular, large models were also used for paper review.This time, NeurIPS 2024 is no exception.A Reddit user commented that he spent a month reviewing six papers, and was really hurt when he saw that he only got an LLM evaluation.Others pointed out that among the papers they reviewed, they found at least three review comments generated by large models, and there are probably more. Three of them were obviously copied and pasted directly from ChatGPT output without reading the paper at all.These reviews all gave a score of 6, with a Confidence score of 4, which is completely inconsistent with everyone else's evaluation.Some netizens even commented that "the quality of the paper reviews is very low."One reviewer confused the baseline of our method, and another confused the derivation of the baseline (as criticized in our work) with the derivation of our method. I suspect some of the comments were generated by LLM.What's wrong with this year's judging panel?
Some people pointed out that this was a problem of the NeurIPS official organizing committee.The description of a score of 6 for NeurIPS is completely, utterly, and bizarrely wrong.Normally, a score of 6 would mean rejection. But it is described as a medium to high impact paper with no problems?No wonder the new reviewers gave such low scores!
In the NeurIPS scoring rules, the introduction with 6 points is given as Weak Accept.Technically solid papers of medium to high impact, with no major issues with evaluation, resources, reproducibility, or ethical considerations.In June this year, a NeurIPS AC shared a lot more information about the top conference review:There are 13 submissions in the batch he is responsible for, and all papers have been automatically assigned four reviewers.Interestingly, 50% of the reviewers were PhD students, with several masters and undergraduates also serving as reviewers.For 5 of the 13 papers, none of the reviewers held a position higher than that of a doctoral student.Among all the reviewers in charge of this AC, there are 3 full professors, 1 associate professor, 4 assistant professors, 9 industry professionals, and 2 people from national laboratories. The rest are postdoctoral/research scientists, doctoral students, master students, and undergraduate students.What do you think about this year's NeurIPS review?
