Does the world need one more blog about arxiv and double blind review?

By Charles Sutton on September 19, 2017

There has been a lively debate recently about the review process for research paper submissions, and how to deal with the fact that double-blind review becomes more difficult when many papers are prepublished on sites like before submission.

I’m going to start from two assumptions: double blind reviewing is good and prepublication is good. You can disagree with either assumption, or you could think that double-blind is so much more important than than prepublication that it should be preserved at all costs, or vice versa. People hold all of those views, and it would take an even longer post to pull all that apart.

Instead, I’d to think about how to reconcile these two assumptions, because I do believe them both, and how to obtain an engineering trade-off that aims at most of the advantages of both, most of the time.

A lot of people have said that allowing papers to be prepublished anonymously would be a good compromise. An appealing idea, but I worry that it may be a bad one. Instead, I’ll argue that a good compromise is this: Accept that papers will be de-blinded, but design the double-blind review process to compensate.

Perhaps the underlying point is that the conflict isn’t black and white. For double blind to work, it’s not necessary for 100% of the submissions to be unblindable, i.e., have their author identities be undiscoverable online. I might even suggest that it’s possible to have effective double blind when all author identities are available online. Just because a paper is unblindable does not mean that the reviewers are unblinded — perhaps they have not seen it, or perhaps they saw it in an email with 100 other papers and don’t remember having seen it.

What shouldn’t we do?

There are some recommendations that I’ve seen that unfortunately I don’t think will work.

Anti-Recommendation 1: I know. Let’s have an anonymous version of arxiv.

Lots of people have suggested allowing authors to prepublish papers anonymously (incidentally, there are amusing precendents for this in the history of mathematics). This could be implemented via an overlay of arxiv, or a new feature added to arxiv itself, that would allow authors to temporary hide their identity. Let’s call this AnonArxiv.

Submissions to AnonArxiv would be immediately available to all but without the author names. Then, once the paper is accepted, AnonArxiv would reveal the author names, while preserving the time stamp of the anonymous submission. The conference would then require that if submissions are prepublished, they must be prepublished anonymously; any other prepublished submissions will be rejected without review.

I used to think this was a cool idea. Now I don’t. It neglects a fundamental principle that we are sadly all familiar with, that most papers are rejected.

Let’s say I post a great paper to AnonArxiv and submit it to ACL. Like most papers it is rejected. I’m convinced that the reviewers have made a mistake, and so I want to resubmit it to EMNLP. How do we handle this?

We could (1) require resubmissions to remain anonymous. After rejection, I must choose whether to unblind the submission on AnonArxiv, in which case it cannot be resubmitted to other conferences, or whether to keep the submission anonymous, in which case the paper could spend a year-plus as anonymous, until it is finally accepted. This seems an unreasonable choice to force onto authors.

Or we could (2) allow authors, after one rejection, to unblind their AnonArxiv submissions and resubmit to a future conference. This has the benefit that papers only spend a few months as public-but-anonymous, which is not so bad. But I’m not sure it works. For one, this is difficult to enforce, because apart from the honor system, the information about whether a paper was previously submitted is confidential (keep in mind that the original submission might have been outside the NLP community). But more fundamentally, what would we do for first-time submissions that violate this rule, reject them without review? How would we justify doing that when there are many second-time submissions whose authors are already public?

This would also mean that if I submit a paper to a conference outside of NLP which allows prepublication and get rejected, I would not be able subsequently decide that an NLP conference would be a better fit, and resubmit there. It might be possible to implement a special dispensation in this case, though.

With some regret, I come to the conclusion that AnonArxiv won’t work. That said, AnonArxiv variant (2) might work if a large enough percentage of submissions were first time submits. Then we’d have the majority of the papers on AnonArxiv, and hence unblindable, which might be good enough.

Perhaps Anti-Recommendation: Require prepublication to be declared

ACL 2017 required all submissions to declare if they had been prepublished. Reviewers were notified that a paper had been prepublished. Prepublished papers that were not declared as such were summarily rejected. Unfortunately I don’t understand the rationale for how this stringent requirement was meant to help. Remember, the goal is not to prevent all papers from being unblind-able, it’s to prevent too many papers from being unblind-ed.

This could be a good idea if the hope is to warn reviewers that they should be careful about searching the web for the paper’s title during the review process. The problem with this idea though, is that it does not help if the authors very reasonable prepublish the paper just after submission. So really, all reviewers need to be careful, all the time, and the extra heads up maybe isn’t too helpful.

If the idea was to simply to gain more information about how many papers are prepublished, then I totally agree with asking the question, but I do not see why penalties for non-compliance were necessary.

If the idea was to handle prepublished papers differently in the review process than non-prepublished ones, then I am not sure why this is necessary. Instead, I’ll advocate below that we handle the review process of unblinded papers differently.

So I would argue that it might make sense for conferences to request that authors declare prepublication, but that no penalties for noncompliance be used in future years.

Recommendation-But-That’s-Actually-An-Orthogonal-Issue: Let’s use OpenReview.Net

I’ve also read the suggestion that the NLP community switch to running conferences on OpenReview. I love OpenReview, and I eagerly await the day when I can go onto a site like OpenReview and pull up any paper in computer science, from any venue, from any year past or present, and find a lively and informative discussion online.

But OpenReview is a software platform, not a reviewing process. It’s specifically designed to allow conference chairs to configure what information about the reviews and authors should be made public and when. It’s not designed to answer the policy questions about whether submissions should be anonymous and what happens after they are rejected.

All right, wise guy, so …?

One way to square the circle is to try adapting reviewing norms to adjust and compensate for the fact that it is more likely for papers to be unblinded.

Conferences are already doing these things, so I don’t claim to have any new ideas. But I think it’s useful to bring together the arguments for these ideas, rather than having programme chairs have to reconstruct these arguments for themselves every year.

Recommendation 1: Clarify Norms for Reviewers

Even if the author information for all submissions are public online, then reviewers, area chairs, and programme chairs can take steps to reduce the chances that a submission is unblinded, and to minimize the consequences when one is.

Reviewers should avoid making Web searches that would be likely to reveal the authors of the paper. It can often happen that a well-meaning search for related work inadvertently turns up an unblinded copy of the paper. I am not saying that reviewers should never search for related work, but it carries risks — it always has (ten years ago I had a reviewer of one of my papers deblinded by a tech report) — and reviewers should try to avoid it.

If the reviewer feels that a Web search is necessary, they should hold off until they have read the paper completely and formed an initial impression of it. This allows reviewers to apply the bias of cognitive dissonance to counteract the potential bias of unblinding.

If reviewers learn the author identities, then they must let the relevant person — who could variously go by the title “programme chair”, “programme committee member”, “meta-reviewer”, “senior PC member”, “area chair”, etc; I’ll use the term “area chair” (AC) — know this right away.

Area chairs should be prepared to apply their judgment to weigh the reviewers’ comments differently when some reviewers are unblinded. Consider a paper like this: it has two negative reviews and one positive review, but the positive reviewer has been unblinded, the paper comes from a famous group so there is possibility of unconscious bias, and the AC believes that the negative comments have merit. Then the AC should be prepared to give less weight to the positive reviewer. In other examples, perhaps all three reviews are positive, or the authors are lesser-known, and so unlikely to engender positive bias. Then downweighting unblinded reviewers may not be necessary.

Programme chairs should carefully write their instructions to reviewers and area chairs to make these expectations clear. They should also be prepared to assist ACs with borderline cases where there is possibility of bias.

We don’t know if these steps alone will reduce the percentage of unblind-ed submissions to an acceptable level. For example, if the percentage of unblinded reviews reaches, say, 80%, this recommendation becomes more like a band-aid that would be unlikely to preserve the benefits of double-blind review. Which brings us to the next point.

Recommendation 2: Measure and Monitor

Much of the heat around this discussion may be because we are, as it were, debating in the blind. It is not difficult to gather more evidence than we have now:

  • We should be able to measure and publicize the percentage of submissions and accepted papers which are prepublished.

  • Although a bit more delicate, we should be able to estimate the percentage of submissions which are not first time submits.

  • If we follow recommendation 1, we should also be able to monitor the percentage of reviews which are unblinded and the percentage of submissions which have had 1, 2, or 3 unblinded reviews.

  • We should also attempt to record measures of diversity in the accepted papers in terms of authors and institutions. We should keep tracking those measures, monitor for decreases, and the presumed negative correlation with percentage of unblindedness.

The not-so-great checked vs carry on debate

By Charles Sutton on September 2, 2017

This time, some advice of a lighter sort. How light? Depends how well you pack!

I hear a lot of frequent air travelers say that you should never check bags if you can avoid it, but I think it’s not always clear cut.

I do usually travel with only a carry on. This has a lot of advantages:

  • You don’t need to queue at the ticket counter to drop off bags. If you’ve checked in online, using your phone is easiest but printing your boarding pass is still good, then you can go straight to security and the gate.

  • You don’t need to wait at baggage claim on arrival. This is an especially big win if you are arriving late at night.

  • It’s easier to switch flights at the last minute. You might wonder why you’d want to do this if you’ve planned ahead. You’ll stop wondering when your first flight arrives early and you wish you could jump on an earlier connecting flight. Or when your first flight is delayed but you still have a chance to catch a different connecting flight. I’m still annoyed at the time United Airlines prevented me from doing this by checking my bag at the gate unnecessarily.

All that said, for some trips it can be more convenient to check your bags, even if you could squeeze into a smaller bag. Why?

  • You can bring a bigger bag that holds more. (duh)

  • You don’t need to lug the bag around the airport. This removes stress during a long connection.

  • You can fill up your checked bag with liquids and gels. This is especially useful if you’d like to bring gifts of food and drink back to your friends, family, or perhaps to a select group of academic bloggers who you particularly admire.

  • If you are running a few minutes late to the gate, the plane is more likely to wait for you, because if they don’t, they need to find and remove your bag from the hold.

For direct flights I think it’s clear that carry on wins most of the time, but if you have a connection then it may well depend on your itinerary.

Tags: travel tips

A first lecture on time management

By Charles Sutton on August 5, 2017

Several students have recently asked me for advice about time management. When people ask you a important and difficult question like this, usually the best thing is to think of someone else who can give a better answer than you. For time management, an obvious person to turn to is the late Randy Pausch, a noted computer scientist who became an internet sensation because of an inspirational lecture that he gave after he had been diagnosed with terminal cancer.

Today I’d like to recommend a different, more practical, and excellent lecture that he gave on time management:

You should start by watching the lecture. I found it helpful to have the slides and bullet points in separate windows on screen as I watched.

If you need further encouragement to watch the lecture, I can say that what amazed me, as poor at time management as I may be, was the number of suggestions that I actually use day-to-day. I had entirely forgotten just how much I had learned the first time I saw this material. For example:

  • Find your most creative time of day and defend it ruthlessly. The most important tip that you can learn for creative work. (I was interested to learn later that your most creative time can actually change over the years.)

  • Turn off your email notification sound. Do it now!

  • Learn ways to say “no” gently. It makes saying “no” easier.

  • Don’t ever delete emails. Search instead.

  • Keep a todo list and calendar. My today-self says, “Well, duh.” But there was a time when I didn’t. You should start now!

  • Get a speakerphone so that you can get other things done while on hold with customer serve. Now, mobile phones or even Skype has a speakerphone option that works for this.

  • Professors should keep Kleenex in their office. Yes, we all need this.

  • Keeping a clock on the wall behind where your visitors sit. Much less obtrusive than a wristwatch.

  • Write down one-minute minutes. So helpful! Just the main takeaways and actions for next time.

  • Keep a time clock. A big topic that I hope to write more about later.

There’s also an equally long list of good advice that I had forgotten, but for that, you’ll need to watch the lecture yourself!

Happy Blogoversary to me

By Charles Sutton on August 4, 2017

I don’t like to write about the blog on the blog, because that seems silly and self-indulgent, but I can’t help noticing that today is the five year anniversary of my first blog post.

It doesn’t feel like I’ve been blogging that long. Perhaps that’s because I post only once every few years. If you like the blog (and if you don’t — why would you read a post like this???), you’ll be happy to know that I’ll be posting more often in the coming year. I’ve found a posting rhythm that works for me: write all my posts at once while I’m on holiday, and queue them up for the rest of the year.

I’m happy I started doing this. I often mention my posts when my students ask me for advice, which has the benefit that my students get tired of hearing about the blog, and therefore stop wanting to meet with me. This leaves my students with more time to carry out excellent research, while leaving me with more time to play old versions of Minesweeper. Everybody wins.

Any regrets? Perhaps only that I wish I could write more silly posts like my manifesto on the political future of Scotland, but real-life politics have grown dark enough that I find myself unable to summon the spirit of fun.

Even without fun, I have lots of ideas for posts about career advice and about the academic and research enterprises. So this will probably be that kind of blog for a wee while longer. It’s fascinating how the theme and how the tone of a blog evolves. I had some ideas about what the blog would be like when I started, but the more I continued, the more I found the blog grow into something different than I expected. Much like life.

What I can guarantee (disclaimer: this is not a legal guarantee) is that the blog will continue to be reflective and introspective, and that if you’ve liked it so far, you’ll probably like what’s coming next.

Tags: metablog

Context switching

By Charles Sutton on July 1, 2017

Of the many quirks shared by computer scientists, one that has somewhat entered the popular culture is the use of computing metaphors to speak about how we think. For example, “multitasking” is actually a technical term invented by computer scientists in the 1960s to describe the way a computer pretends to execute many programs at once, even if in reality it can only do one thing at a time. You can see why the term would have been quickly co-opted to metaphorically describe humans who attempt the same trick.

For the benefit of the three or four readers of the blog who are not computer scientists, I’ll digress here in order to comment that the situation on the computing side has gotten more complicated since the 60s, as almost all modern computers can perform many operations at once, but they still use multitasking so that they can pretend to do even more. That probably sounds like some people you know, too.

Professional computer scientists have many more metaphors at their command, though. We speak of “swapping in” and “swapping out” the details of a project from our short-term memories, referring to a technique that allows computers to pretend to have infinitely large memories. We also — and now I come finally to the topic of the post — talk about “context switching”.

A context switch is a necessary result of multitasking. If a computer is pretending to run ten programs at once when it can in reality run only one, then it must switch very quickly back and forth between them, like a chef switching between stirring the sauce and browning the meat. Unlike the chef, on a computer the switching comes with a price. The price comes because computers have many different types of memory, some fast and close to the CPU, others larger and farther away. Every time the computer switches from one program to another, it has to move data about the old program out of the small fast memory, and move the new data in. The work of switching back and forth is what we call context switching. It takes a bit of time.

So it is in your mind. When you switch between different tasks, it takes a while to remember the other context: what it was that you were supposed to be doing on the new task, why you thought it was a good idea, and so on. This is why you’re not as efficient when you first start a new task as when you’ve been thinking about it for a while. This has led to the deluge of self-help articles, some of which I’m sure you’ve seen, about how mental multitasking is a bad idea. It takes time to switch contexts.

But as a professor (in the US sense), especially leading a research group, context switching cannot be avoided. The nature of the job requires us to make progress on many different types of tasks each day, from answering a student question about a problem set, to planning for the curriculum next year, to reading a paper related to one research project, to an impromptu meeting with a colleague about faculty hiring for next spring, to a scheduled meeting with a PhD student about a different research project. One of my mentors liked to joke, “Professors are stateless” (another computing metaphor). The reason that we seem stateless is that it takes some time for us to remember the previous context. Not only does it take time to context switch, but it is also a bit disorienting. A colleague once told me that the part of being a professor that they disliked the most was having to context switch many times a day.

People sometimes ask me how I handle context switching. I have a few strategies. One is a bit embarrassing, but I will tell you anyway. Perhaps I made this a long post to drive people away before I revealed the embarrassing one.

  1. First is not to be afraid to ask questions. This is useful in one-to-one student meetings about a research project. You can say something like: “Sorry, what was the vorpal sort for again?” Of course, if you do this too much, you derail the meeting. But with practice I found that much of the time, simply asking the question silently in my head was enough. Once I had articulated the question clearly, I was able remember the answer, without bothering the student to repeat what we had already discussed. I’m sure this makes me look smarter, but more importantly it saves time.

  2. I keep relevant information easy to search on my computer. This is why I am never without my laptop. I don’t try to remember all that I need to know so much as remember how to find it in my email, in my Evernote, by a Web search that I know will bring up a particular page, or whatever. Then if I’m in a large meeting, for example, where I can’t just stop and ask twenty people to remind me what they said in last week’s meeting, I can search quietly and find the context I need.

  3. When I know I’ll need to make a big context switch, I’ll take a break to clear my head. A ten-minute walk helps, or staring out a window, or something sweet. Coffee doesn’t work here for me; it’s great for energy, of course, but not for context switching.

  4. If I write something down, I know down to my unconscious that it’s OK to stop thinking about it, because now I can come back later. When I have meetings back to back, during the second meeting I might remember something I should have said in the first. If I quickly jot down a note, that helps me to stop thinking about the first meeting, and return to the task at hand. This works within meetings too. Sometimes I think of something that I need to say in a one-on-one meeting, but it’s not as important as what we’re talking about right now. I’ll write that down too. I’ll even say something as I’m doing this, so we both know there’s something to come back to later.

  5. OK, OK, I’ll tell you the embarrassing one. I have some mental imagery that I use. I imagine my focus of attention as the beam of light coming from a huge lamp, like at the top of the lighthouse. When I need to switch focus quickly, like between back to back meetings, I imagine myself grasping the lamp between my hands — it’s several feet around, and heavy, so this takes a real effort — and slowly rotating it to shine onto the new problem that I’m meant to be thinking about. Believe it or not, sometimes this works. I will never tell you when I am doing this, even if you ask.