hunch_net hunch_net-2013 hunch_net-2013-485 knowledge-graph by maker-knowledge-mining
Source: html
Introduction: This post is a (near) transcript of a talk that I gave at the ICML 2013 Workshop on Peer Review and Publishing Models . Although there’s a PDF available on my website , I’ve chosen to post a slightly modified version here as well in order to better facilitate discussion. Disclaimers and Context I want to start with a couple of disclaimers and some context. First, I want to point out that although I’ve read a lot about double-blind review, this isn’t my research area and the research discussed in this post is not my own. As a result, I probably can’t answer super detailed questions about these studies. I also want to note that I’m not opposed to open peer review — I was a free and open source software developer for over ten years and I care a great deal about openness and transparency. Rather, my motivation in writing this post is simply to create awareness of and to initiate discussion about the benefits of double-blind review. Lastly, and most importantly, I think it’s e
sentIndex sentText sentNum sentScore
1 I also want to note that I’m not opposed to open peer review — I was a free and open source software developer for over ten years and I care a great deal about openness and transparency. [sent-6, score-0.657]
2 Lastly, and most importantly, I think it’s essential to acknowledge that there’s a lot of research on double-blind review out there. [sent-8, score-0.465]
3 My goal here is therefore to draw your attention to some of the key benefits of double-blind review so that we don’t lose sight of them when considering alternative reviewing models. [sent-12, score-0.56]
4 , that double-blind review isn’t really blind, so therefore there’s no point in implementing it. [sent-19, score-0.462]
5 There are several studies that directly test this assertion by asking reviewers whether authors or institutions are identifiable and, if so, to record their identities and describe the clues that led to their identification. [sent-21, score-0.865]
6 The results are pretty interesting: when asked to guess the identities of authors or institutions, reviewers are correct only 25–42% of the time [1]. [sent-22, score-0.536]
7 This indicates that journals, not just authors, bear some responsibility for the degree of identification clues present and can therefore influence the extent to which review is truly double-blind. [sent-25, score-0.734]
8 , that double-blind review isn’t needed because factors other than scientific quality do not affect reviewers’ opinions anyway. [sent-30, score-0.646]
9 There are many studies that address this assertion by testing the extent to which peer review can be biased against new ideas, women, junior researchers, and researchers from less prestigious universities or countries other than the US. [sent-32, score-0.882]
10 To quote the AAUW’s report [4] on the under-representation of women in science, “Even individuals who consciously refute gender and science stereotypes can still hold that belief at an unconscious level. [sent-36, score-0.703]
11 ” Chapters 8 and 9 of this report provide a really great overview of recent research on implicit bias and negative stereotypes in the workplace. [sent-38, score-0.431]
12 In the context of peer review, reviewers may be more likely to recommend acceptance of incomplete or inferior papers if they are authored by more prestigious researchers. [sent-47, score-0.521]
13 There’s research [6] showing that reviewers from within the United States and reviewers from outside the United States evaluate US papers more favorably, with US reviewers showing a stronger preference for US papers than non-US reviewers. [sent-49, score-0.625]
14 Gender One of the most widely discussed pieces of recent work on double-blind review and gender is that of Budden et al. [sent-51, score-0.618]
15 [1], whose research demonstrated that following the introduction of double-blind review by the journal Behavioral Ecology, there was a significant increase in papers authored by women. [sent-52, score-0.685]
16 Stereotype threat is a phenomenon in which performance in academic contexts can be harmed by the awareness that one’s behavior might be viewed through the lens of a negative stereotype about one’s social group [10]. [sent-58, score-0.514]
17 For example, studies have demonstrated that African-American students enrolled in college and female students enrolled in math and science courses score much lower on tests when they are reminded beforehand of their race or gender [10, 11]. [sent-59, score-0.727]
18 One idea that that hasn’t yet been explored in the context of peer review, but might be worth investigating, is whether requiring authors to reveal their identities during peer review induces a stereotype threat scenario. [sent-62, score-1.485]
19 “ Double-blind review favours increased representation of female authors . [sent-73, score-0.711]
20 “ Incidence and nature of unblinding by authors: our experience at two radiology journals with double-blinded peer review policies . [sent-79, score-0.587]
wordName wordTfidf (topN-words)
[('review', 0.38), ('peer', 0.207), ('stereotype', 0.201), ('identities', 0.193), ('reviewers', 0.18), ('women', 0.179), ('factors', 0.179), ('female', 0.168), ('authors', 0.163), ('studies', 0.161), ('gender', 0.134), ('threat', 0.134), ('race', 0.119), ('bias', 0.113), ('et', 0.104), ('awareness', 0.101), ('budden', 0.101), ('clues', 0.101), ('mainguy', 0.101), ('identification', 0.099), ('benefits', 0.098), ('origin', 0.089), ('stereotypes', 0.089), ('scientific', 0.087), ('research', 0.085), ('journal', 0.084), ('quote', 0.083), ('matthew', 0.083), ('therefore', 0.082), ('country', 0.078), ('phenomenon', 0.078), ('post', 0.077), ('science', 0.076), ('report', 0.075), ('influence', 0.072), ('want', 0.07), ('implicit', 0.069), ('demonstrated', 0.069), ('blind', 0.068), ('aarssen', 0.067), ('assertion', 0.067), ('authored', 0.067), ('disclaimers', 0.067), ('koricheva', 0.067), ('landscape', 0.067), ('lastly', 0.067), ('prestigious', 0.067), ('tregenza', 0.067), ('unconscious', 0.067), ('webb', 0.067)]
simIndex simValue blogId blogTitle
same-blog 1 1.0000004 485 hunch net-2013-06-29-The Benefits of Double-Blind Review
Introduction: This post is a (near) transcript of a talk that I gave at the ICML 2013 Workshop on Peer Review and Publishing Models . Although there’s a PDF available on my website , I’ve chosen to post a slightly modified version here as well in order to better facilitate discussion. Disclaimers and Context I want to start with a couple of disclaimers and some context. First, I want to point out that although I’ve read a lot about double-blind review, this isn’t my research area and the research discussed in this post is not my own. As a result, I probably can’t answer super detailed questions about these studies. I also want to note that I’m not opposed to open peer review — I was a free and open source software developer for over ten years and I care a great deal about openness and transparency. Rather, my motivation in writing this post is simply to create awareness of and to initiate discussion about the benefits of double-blind review. Lastly, and most importantly, I think it’s e
2 0.2440666 484 hunch net-2013-06-16-Representative Reviewing
Introduction: When thinking about how best to review papers, it seems helpful to have some conception of what good reviewing is. As far as I can tell, this is almost always only discussed in the specific context of a paper (i.e. your rejected paper), or at most an area (i.e. what a “good paper” looks like for that area) rather than general principles. Neither individual papers or areas are sufficiently general for a large conference—every paper differs in the details, and what if you want to build a new area and/or cross areas? An unavoidable reason for reviewing is that the community of research is too large. In particular, it is not possible for a researcher to read every paper which someone thinks might be of interest. This reason for reviewing exists independent of constraints on rooms or scheduling formats of individual conferences. Indeed, history suggests that physical constraints are relatively meaningless over the long term — growing conferences simply use more rooms and/or change fo
3 0.23498695 40 hunch net-2005-03-13-Avoiding Bad Reviewing
Introduction: If we accept that bad reviewing often occurs and want to fix it, the question is “how”? Reviewing is done by paper writers just like yourself, so a good proxy for this question is asking “How can I be a better reviewer?” Here are a few things I’ve learned by trial (and error), as a paper writer, and as a reviewer. The secret ingredient is careful thought. There is no good substitution for a deep and careful understanding. Avoid reviewing papers that you feel competitive about. You almost certainly will be asked to review papers that feel competitive if you work on subjects of common interest. But, the feeling of competition can easily lead to bad judgement. If you feel biased for some other reason, then you should avoid reviewing. For example… Feeling angry or threatened by a paper is a form of bias. See above. Double blind yourself (avoid looking at the name even in a single-blind situation). The significant effect of a name you recognize is making you pay close a
4 0.19851312 343 hunch net-2009-02-18-Decision by Vetocracy
Introduction: Few would mistake the process of academic paper review for a fair process, but sometimes the unfairness seems particularly striking. This is most easily seen by comparison: Paper Banditron Offset Tree Notes Problem Scope Multiclass problems where only the loss of one choice can be probed. Strictly greater: Cost sensitive multiclass problems where only the loss of one choice can be probed. Often generalizations don’t matter. That’s not the case here, since every plausible application I’ve thought of involves loss functions substantially different from 0/1. What’s new Analysis and Experiments Algorithm, Analysis, and Experiments As far as I know, the essence of the more general problem was first stated and analyzed with the EXP4 algorithm (page 16) (1998). It’s also the time horizon 1 simplification of the Reinforcement Learning setting for the random trajectory method (page 15) (2002). The Banditron algorithm itself is functionally identi
5 0.16517678 315 hunch net-2008-09-03-Bidding Problems
Introduction: One way that many conferences in machine learning assign reviewers to papers is via bidding, which has steps something like: Invite people to review Accept papers Reviewers look at title and abstract and state the papers they are interested in reviewing. Some massaging happens, but reviewers often get approximately the papers they bid for. At the ICML business meeting, Andrew McCallum suggested getting rid of bidding for papers. A couple reasons were given: Privacy The title and abstract of the entire set of papers is visible to every participating reviewer. Some authors might be uncomfortable about this for submitted papers. I’m not sympathetic to this reason: the point of submitting a paper to review is to publish it, so the value (if any) of not publishing a part of it a little bit earlier seems limited. Cliques A bidding system is gameable. If you have 3 buddies and you inform each other of your submissions, you can each bid for your friend’s papers a
6 0.16347207 116 hunch net-2005-09-30-Research in conferences
7 0.15475786 65 hunch net-2005-05-02-Reviewing techniques for conferences
8 0.14610972 453 hunch net-2012-01-28-Why COLT?
9 0.14554954 437 hunch net-2011-07-10-ICML 2011 and the future
10 0.14009072 320 hunch net-2008-10-14-Who is Responsible for a Bad Review?
11 0.13488765 461 hunch net-2012-04-09-ICML author feedback is open
12 0.12745358 454 hunch net-2012-01-30-ICML Posters and Scope
13 0.12556864 395 hunch net-2010-04-26-Compassionate Reviewing
14 0.11503042 207 hunch net-2006-09-12-Incentive Compatible Reviewing
15 0.11248774 318 hunch net-2008-09-26-The SODA Program Committee
16 0.11147686 288 hunch net-2008-02-10-Complexity Illness
17 0.1106113 333 hunch net-2008-12-27-Adversarial Academia
18 0.10907771 363 hunch net-2009-07-09-The Machine Learning Forum
19 0.10865261 468 hunch net-2012-06-29-ICML survey and comments
20 0.10755381 382 hunch net-2009-12-09-Future Publication Models @ NIPS
topicId topicWeight
[(0, 0.227), (1, -0.15), (2, 0.146), (3, 0.147), (4, -0.032), (5, 0.032), (6, 0.016), (7, 0.048), (8, -0.019), (9, -0.001), (10, 0.035), (11, 0.006), (12, 0.065), (13, -0.071), (14, -0.076), (15, 0.007), (16, -0.063), (17, -0.005), (18, -0.082), (19, -0.021), (20, -0.073), (21, -0.058), (22, -0.009), (23, -0.019), (24, 0.046), (25, -0.004), (26, 0.019), (27, 0.02), (28, 0.024), (29, 0.076), (30, 0.01), (31, 0.064), (32, 0.007), (33, -0.034), (34, -0.087), (35, 0.014), (36, -0.071), (37, -0.041), (38, 0.007), (39, -0.009), (40, -0.023), (41, 0.04), (42, -0.008), (43, 0.013), (44, 0.0), (45, -0.009), (46, 0.047), (47, -0.039), (48, -0.022), (49, -0.052)]
simIndex simValue blogId blogTitle
same-blog 1 0.97551084 485 hunch net-2013-06-29-The Benefits of Double-Blind Review
Introduction: This post is a (near) transcript of a talk that I gave at the ICML 2013 Workshop on Peer Review and Publishing Models . Although there’s a PDF available on my website , I’ve chosen to post a slightly modified version here as well in order to better facilitate discussion. Disclaimers and Context I want to start with a couple of disclaimers and some context. First, I want to point out that although I’ve read a lot about double-blind review, this isn’t my research area and the research discussed in this post is not my own. As a result, I probably can’t answer super detailed questions about these studies. I also want to note that I’m not opposed to open peer review — I was a free and open source software developer for over ten years and I care a great deal about openness and transparency. Rather, my motivation in writing this post is simply to create awareness of and to initiate discussion about the benefits of double-blind review. Lastly, and most importantly, I think it’s e
2 0.78766 484 hunch net-2013-06-16-Representative Reviewing
Introduction: When thinking about how best to review papers, it seems helpful to have some conception of what good reviewing is. As far as I can tell, this is almost always only discussed in the specific context of a paper (i.e. your rejected paper), or at most an area (i.e. what a “good paper” looks like for that area) rather than general principles. Neither individual papers or areas are sufficiently general for a large conference—every paper differs in the details, and what if you want to build a new area and/or cross areas? An unavoidable reason for reviewing is that the community of research is too large. In particular, it is not possible for a researcher to read every paper which someone thinks might be of interest. This reason for reviewing exists independent of constraints on rooms or scheduling formats of individual conferences. Indeed, history suggests that physical constraints are relatively meaningless over the long term — growing conferences simply use more rooms and/or change fo
3 0.7686879 40 hunch net-2005-03-13-Avoiding Bad Reviewing
Introduction: If we accept that bad reviewing often occurs and want to fix it, the question is “how”? Reviewing is done by paper writers just like yourself, so a good proxy for this question is asking “How can I be a better reviewer?” Here are a few things I’ve learned by trial (and error), as a paper writer, and as a reviewer. The secret ingredient is careful thought. There is no good substitution for a deep and careful understanding. Avoid reviewing papers that you feel competitive about. You almost certainly will be asked to review papers that feel competitive if you work on subjects of common interest. But, the feeling of competition can easily lead to bad judgement. If you feel biased for some other reason, then you should avoid reviewing. For example… Feeling angry or threatened by a paper is a form of bias. See above. Double blind yourself (avoid looking at the name even in a single-blind situation). The significant effect of a name you recognize is making you pay close a
4 0.76865906 315 hunch net-2008-09-03-Bidding Problems
Introduction: One way that many conferences in machine learning assign reviewers to papers is via bidding, which has steps something like: Invite people to review Accept papers Reviewers look at title and abstract and state the papers they are interested in reviewing. Some massaging happens, but reviewers often get approximately the papers they bid for. At the ICML business meeting, Andrew McCallum suggested getting rid of bidding for papers. A couple reasons were given: Privacy The title and abstract of the entire set of papers is visible to every participating reviewer. Some authors might be uncomfortable about this for submitted papers. I’m not sympathetic to this reason: the point of submitting a paper to review is to publish it, so the value (if any) of not publishing a part of it a little bit earlier seems limited. Cliques A bidding system is gameable. If you have 3 buddies and you inform each other of your submissions, you can each bid for your friend’s papers a
5 0.74978536 395 hunch net-2010-04-26-Compassionate Reviewing
Introduction: Most long conversations between academics seem to converge on the topic of reviewing where almost no one is happy. A basic question is: Should most people be happy? The case against is straightforward. Anyone who watches the flow of papers realizes that most papers amount to little in the longer term. By it’s nature research is brutal, where the second-best method is worthless, and the second person to discover things typically gets no credit. If you think about this for a moment, it’s very different from most other human endeavors. The second best migrant laborer, construction worker, manager, conductor, quarterback, etc… all can manage quite well. If a reviewer has even a vaguely predictive sense of what’s important in the longer term, then most people submitting papers will be unhappy. But this argument unravels, in my experience. Perhaps half of reviews are thoughtless or simply wrong with a small part being simply malicious. And yet, I’m sure that most reviewers genuine
6 0.71936178 333 hunch net-2008-12-27-Adversarial Academia
7 0.71451247 207 hunch net-2006-09-12-Incentive Compatible Reviewing
8 0.70951545 461 hunch net-2012-04-09-ICML author feedback is open
9 0.70945287 343 hunch net-2009-02-18-Decision by Vetocracy
10 0.70822358 116 hunch net-2005-09-30-Research in conferences
11 0.66229588 363 hunch net-2009-07-09-The Machine Learning Forum
12 0.65067589 320 hunch net-2008-10-14-Who is Responsible for a Bad Review?
13 0.63536125 38 hunch net-2005-03-09-Bad Reviewing
14 0.63504887 382 hunch net-2009-12-09-Future Publication Models @ NIPS
15 0.63422024 463 hunch net-2012-05-02-ICML: Behind the Scenes
16 0.60236943 318 hunch net-2008-09-26-The SODA Program Committee
17 0.5962292 437 hunch net-2011-07-10-ICML 2011 and the future
18 0.59590822 65 hunch net-2005-05-02-Reviewing techniques for conferences
19 0.58265245 221 hunch net-2006-12-04-Structural Problems in NIPS Decision Making
20 0.54063278 304 hunch net-2008-06-27-Reviewing Horror Stories
topicId topicWeight
[(3, 0.035), (16, 0.019), (21, 0.028), (27, 0.139), (38, 0.046), (42, 0.242), (48, 0.023), (53, 0.04), (55, 0.112), (60, 0.025), (61, 0.014), (73, 0.013), (84, 0.014), (94, 0.065), (95, 0.051)]
simIndex simValue blogId blogTitle
1 0.93500507 242 hunch net-2007-04-30-COLT 2007
Introduction: Registration for COLT 2007 is now open. The conference will take place on 13-15 June, 2007, in San Diego, California, as part of the 2007 Federated Computing Research Conference (FCRC), which includes STOC, Complexity, and EC. The website for COLT: http://www.learningtheory.org/colt2007/index.html The early registration deadline is May 11, and the cutoff date for discounted hotel rates is May 9. Before registering, take note that the fees are substantially lower for members of ACM and/or SIGACT than for nonmembers. If you’ve been contemplating joining either of these two societies (annual dues: $99 for ACM, $18 for SIGACT), now would be a good time!
2 0.9271242 255 hunch net-2007-07-13-The View From China
Introduction: I’m visiting Beijing for the Pao-Lu Hsu Statistics Conference on Machine Learning. I had several discussions about the state of Chinese research. Given the large population and economy, you might expect substantial research—more than has been observed at international conferences. The fundamental problem seems to be the Cultural Revolution which lobotimized higher education, and the research associated with it. There has been a process of slow recovery since then, which has begun to be felt in the research world via increased participation in international conferences and (now) conferences in China. The amount of effort going into construction in Beijing is very impressive—people are literally building a skyscraper at night outside the window of the hotel I’m staying at (and this is not unusual). If a small fraction of this effort is later focused onto supporting research, the effect could be very substantial. General growth in China’s research portfolio should be expecte
same-blog 3 0.90876895 485 hunch net-2013-06-29-The Benefits of Double-Blind Review
Introduction: This post is a (near) transcript of a talk that I gave at the ICML 2013 Workshop on Peer Review and Publishing Models . Although there’s a PDF available on my website , I’ve chosen to post a slightly modified version here as well in order to better facilitate discussion. Disclaimers and Context I want to start with a couple of disclaimers and some context. First, I want to point out that although I’ve read a lot about double-blind review, this isn’t my research area and the research discussed in this post is not my own. As a result, I probably can’t answer super detailed questions about these studies. I also want to note that I’m not opposed to open peer review — I was a free and open source software developer for over ten years and I care a great deal about openness and transparency. Rather, my motivation in writing this post is simply to create awareness of and to initiate discussion about the benefits of double-blind review. Lastly, and most importantly, I think it’s e
4 0.82395792 282 hunch net-2008-01-06-Research Political Issues
Introduction: I’ve avoided discussing politics here, although not for lack of interest. The problem with discussing politics is that it’s customary for people to say much based upon little information. Nevertheless, politics can have a substantial impact on science (and we might hope for the vice-versa). It’s primary election time in the United States, so the topic is timely, although the issues are not. There are several policy decisions which substantially effect development of science and technology in the US. Education The US has great contrasts in education. The top universities are very good places, yet the grade school education system produces mediocre results. For me, the contrast between a public education and Caltech was bracing. For many others attending Caltech, it clearly was not. Upgrading the k-12 education system in the US is a long-standing chronic problem which I know relatively little about. My own experience is that a basic attitude of “no child unrealized” i
5 0.78523952 216 hunch net-2006-11-02-2006 NIPS workshops
Introduction: I expect the NIPS 2006 workshops to be quite interesting, and recommend going for anyone interested in machine learning research. (Most or all of the workshops webpages can be found two links deep.)
6 0.77984893 89 hunch net-2005-07-04-The Health of COLT
7 0.75993133 22 hunch net-2005-02-18-What it means to do research.
8 0.6605134 40 hunch net-2005-03-13-Avoiding Bad Reviewing
9 0.64938533 452 hunch net-2012-01-04-Why ICML? and the summer conferences
10 0.63361484 146 hunch net-2006-01-06-MLTV
11 0.62793553 416 hunch net-2010-10-29-To Vidoelecture or not
12 0.62391812 36 hunch net-2005-03-05-Funding Research
13 0.62302214 406 hunch net-2010-08-22-KDD 2010
14 0.62202191 333 hunch net-2008-12-27-Adversarial Academia
15 0.62017792 437 hunch net-2011-07-10-ICML 2011 and the future
16 0.61805844 466 hunch net-2012-06-05-ICML acceptance statistics
17 0.61681139 423 hunch net-2011-02-02-User preferences for search engines
18 0.61678123 484 hunch net-2013-06-16-Representative Reviewing
19 0.61650944 464 hunch net-2012-05-03-Microsoft Research, New York City
20 0.61358517 403 hunch net-2010-07-18-ICML & COLT 2010