Reviewing is underway for a new CSCW season! Papers accepted during the October, January, and April cycles can be presented at the CSCW conference (which will be held virtually in Fall 2021), which means that they will also be eligible for 2021 awards and recognitions.
The recent changes for CSCW, notably the removal of numeric scores, necessitated some rethinking of processes. As an effort towards transparency in process for awards and recognitions at CSCW, this post will detail how the process worked for 2020, some changes we (Casey Fiesler and Shion Guha as 2021 Awards Co-Chairs) have made so far for 2021, as well as explanation for new recognitions we are trying out for this cycle.
For 2020, during the review process, papers could be nominated for a best paper award and/or a diversity & inclusion (D&I) recognition by both reviewers and ACs. The 2020 Awards Chair (Casey Fiesler) assembled a committee from volunteers from the 2020 program committee, along with their nominees of additional excellent reviewers from outside the program committee. A full list of members of this committee can be found on the 2020 awards website.
Because of the large number of nominated papers (70+!), not all committee members evaluated every paper, but instead the chair assigned (randomly, while also taking into account conflicts) multiple reviewers to each paper. Committee members were provided both a copy of the paper without author names and copies of the original anonymized reviews. They were asked to score the papers based on both their own evaluation and the evaluations of the original reviewers, including reasons given for nominations, and to provide a narrative for the strengths of the paper that merited an award or recognition. The Papers Chairs as well as the D&I chairs assisted during this process, including in managing conflicts with the Awards Chair (who did not have access to any review information for their conflicted papers), and during both review and synchronous discussion, conflicts for committee members were handled similarly to the typical review process (i.e., leaving the room during discussion). The Awards Chair was not involved in evaluation of papers or in the decision-making process beyond organization.
Based on these scores and narratives (anonymized), a subset of the committee met synchronously to discuss and make final decisions about Best Paper and Honorable Mention awards. Final decisions for D&I recognitions were managed by the D&I chairs, based on original reviewer comments and committee members’ evaluations (which were focused on contribution to diversity and inclusion in CSCW research). The final list of papers recognized for CSCW 2020 are available on the awards page. This process is, of course, imperfect, but the papers that were selected represent examples of the excellent work published at CSCW!
A common question is: What makes a paper a “best paper”? This evaluation is, of course, as subjective as the review process in general. Reviewers making nominations and committee members themselves might have different ideas, but in looking at the narratives for papers that ultimately received this award, there were some clear themes: contribution to the field (“novel contribution to the literature”, “clear implications for future research and design”, “bridges multiple disciplines”), rigor (“solid methodology with detailed qualitative analysis”, “impressive level of rigor in the analysis”, “strong theoretical framing”), communication (“communicates how this work can be extended by others”, “well written”), novelty (“approaches the problem from a different perspective”, “novel system building”), and importance (“addresses a socially important topic”, “includes concrete ideas about what to do to address an important problem”).
One change made to the awards process for 2020 was the inclusion of the original reviews as part of the evaluation. Our hope is that, even though final decisions must be made by a subset of the community, that taking reviews into account includes more voices in the process of determining what merits of any paper are worthy of recognition. Towards this end, we also think it is important that all reviewers, not just ACs, can nominate papers.
Therefore, we highly encourage reviewers to nominate papers that you think represent the best of CSCW when you submit your review! It is also important that you explain why you are making the nomination. We want to consider whatever criteria the CSCW community might consider valuable.
Finally, though official ACM awards only include Best Papers and Honorable Mentions, CSCW has also had for a few years additional recognitions for diversity and inclusion (which represent strong examples of work that focuses on or serves minorities, otherwise excluded individuals or populations, or intervenes in systemic structures of inequality). This year, we are including two additional recognitions as part of this process, after soliciting ideas from the 2020 awards committee and consulting with 2021 Papers Chairs. These two possible recognitions are for methods (strong examples of work that includes well developed, explained, or implemented methods, and/or methodological innovation) and impact (strong examples of work that demonstrates or has clear potential to demonstrate real-world or practical impact). In our goal to recognize a breadth of contributions to CSCW scholarship, we hope that you will also nominate papers that you think are deserving of these new recognitions.
Following April 2021 paper decisions (the last that can be included for CSCW 2021 presentations), we will be convening a new awards committee and starting the process of making decisions for CSCW 2021. We welcome any feedback you have on the process! Please feel free to email awards2021@cscw.acm.org with any comments or suggestions.
