To PhD applicants: A word about department rankings

By Charles Sutton on November 12, 2017

When I was applying for my PhD, I used rankings of computer science departments to help me decide where to apply. Rankings are never perfect, but I didn’t have access to detailed knowledge of the research landscape, and the rankings helped to steer me in the right direction. In retrospect, I can see that the rankings were not perfect, and I made one or two silly mistakes about where to apply, but I would have made even more silly mistakes without them.

I’m saying this to give you this context: I’m not anti-ranking. Very possibly rankings have had a negative effect on higher education overall, but they can be useful if done right, and if you read them in the right way.

The Computing Research Association has just released a statement urging everyone to ignore the new rankings of global computer science departments from US News and World Report. I’m sorry to see this, because I found the US News rankings helpful when I was an undergraduate. But I’ve read the new US News rankings, and I have to agree with the CRA.

These US News rankings are absurd. They are garbage. No one should read them, and I won’t even link to them. You can find them easily via a search engine. Please don’t. The ranking methodology is flawed, for a simple reason that any computer science researcher could tell them immediately. And we did. Influential researchers in computer science pointed out the flaws directly to editors at US News; they were ignored. I don’t know why the editors of US News would ignore this feedback, unless they cared a lot about creating a controversy that would generate page views, and not at all about helping students who are applying for their PhD.

I’ll repeat: Please do not read these rankings at all, not even if you intend take my advice and ignore them. If you click on them, even to laugh at them, you are spending advertisers’ money to support this magazine in misleading PhD applicants who are not as well informed as you.

My advice: If you need rankings, instead go to CSRankings.org. This is a fully open ranking from Prof Emery Berger at UMass Amherst that ranks global computer science departments directly by the amount of research they produce. You can filter the rankings by geographic area and research area. No ranking is perfect, but this is defensible and open.

I mentioned that rankings are only useful if you read them correctly. Here are some thoughts about how to do that:

  • Overall ranking is not the same as subject specific ranking. The department ranked #50 isn’t ranked that way because its research is #50 in every area of CS. Instead, it will have some research areas that are #50 — which is still pretty damn good — but a few groups that are in the top five. If you are in one of those top groups, then you are in a top group, with all the same excitement and opportunity as the top groups at a bigger name school.

  • Disregard small differences in ranking. Ranking is an ill-defined problem, so you can’t take small differences seriously. As far as overall strength goes, the school ranked #1 is exactly the same as the school ranked #5. Exactly the same. But #1 is going to be overall stronger than #18.

  • For your PhD, what matters most is your supervisor and their group, rather than the department overall. This relates to what I said above, and is probably worth a blog post of its own.

  • Rankings are not life. The distinctions that we are talking about here are small distinctions at the very top. The school ranked #100 — I haven’t looked up what it is — is a fine university with brilliant researchers where you will learn a lot. Here’s an analogy. The weakest football player in the English Premier League, who spends most of his time on the bench, is still a prodigiously talented football player who would run circles around anyone who you and I have ever met. The difference between Lionel Messi and that guy — that’s the level of difference we’re talking about here.

The key point: Use rankings as a way to discover departments you didn’t know about that are strong in your area. Don’t use them as a way to decide between departments: For that, you should be reading the work of potential supervisors that interest you. Doing a PhD is about learning to do research. What types of papers do you want to write?

How to read a research paper

By Charles Sutton on November 4, 2017

There’s lots of advice you can read about how to read a research paper. There’s some good advice in this paper:

S. Keshav. How to read a paper. ACM SIGCOMM Computer Communication Review, 37(3):83–84, 2007.

But there’s one tip that I can offer you to organise your reading of a paper that I can’t remember seeing elsewhere. Ask yourself:

What is the 5 minute summary that you would give to a Very Smart Friend?

I don’t understand a paper until I can explain the paper to a smart person who hasn’t read it. I need to be able to explain enough to the VSF so that she understands: what problem the paper is trying to solve, what sort of methods does it use, and how does it relate to the literature, i.e., what does it add.

But there’s two rules.

Rule 1: You have to use your own words, summarising the paper without looking at it. If you find yourself repeating sentences from the paper, the you haven’t internalised the paper’s message.

Rule 2: You cannot take anything the paper says at face value. You can assume that the authors won’t lie to you. But they might oversell a bit, and if you are a independent expert, you might not agree with everything they claim, or with how they interpret the new evidence that they have provided. Or you might be able to describe what’s going on a little bit better than they managed. What do you think that they have shown?

Another way of saying this: I know that my imaginary Very Smart Friend will jump on me if I say something inaccurate. So I don’t want to make a claim to my iVSF unless I can argue for it, based on what I have learned from the theory and experiments in the paper. If I just say something like “well, the authors claim X,” but X is controversial, or even dubious, then my iVSF will immediately want to know why they say that, do they really have evidence, and I had better have a answer.

It can also be good to try this exercise even before you are done. After reading the introduction, how well can you guess what the methods will be, even before you read them? Then read to see if you were right.

To sum up, I hope that I’ve convinced you that having an imaginary friend can help you in your research. You might not want to tell everyone on the internet that you have an imaginary friend, as I have just done, because it might not improve their respect for you. But hey, if it’s good for your research, then where are your priorities?

My special-est email folder

By Charles Sutton on October 7, 2017

I’ve tried a lot of different methods to organize my email. None of them work.

After many years of trying, I’ve come to the conclusion that the Gmail solution of “put all the email in one bucket and search” is the way to go. Over the years I have gotten very, very good at searching my email. But that’s a topic for another day.

Today, I wanted to mention a special email folder that I have. Maybe it’s the most important folder I have, even though I might only look at it once a year, if that.

What could be so special? Emails to pick me up when I’m feeling down. You know which ones I mean. The emails that contained job offers, promotions, grant awards, or even just kind words from trusted senior colleagues. I archive emails like this, emails with special professional successes, in their own folder. Then every once in a while, if I’ve been dealing with a difficult situation or am having a particularly bad bout of impostor syndrome, I can open up the folder and look at it just for a minute, and then I feel a little bit better.

You might ask why these are professional emails. Hopefully I’ve had happy things happen in my personal life as well? Don’t be too worried. It’s just that my personal life doesn’t usually happen over email. That’s what my wedding photos are for!

I got this idea from a blog post somewhere by some other professor, but I don’t remember who or where. If you do, please let me know in the comments!

Update: Rob Zinkov has kindly reminded me that I learned about this idea from the excellent 7 year postdoc blog post by Radhika Nagpal about life for new faculty members. Thanks!

Tags: advice, email

Making unblinding manageable: Towards reconciling prepublication and double-blind review

By Charles Sutton on September 19, 2017

There has been a lively debate recently about the review process for research paper submissions, and how to deal with the fact that double-blind review becomes more difficult when many papers are prepublished on sites like arxiv.org before submission.

This discussion is becoming increasingly important, as we have conducted a study which indicates that in 2017, 23% of top-tier CS papers were posted to arXiv. (That figure includes papers posted both during and after peer review.)

I’m going to start from two assumptions: double blind reviewing is good and prepublication is good. You can disagree with either assumption, or you could think that double-blind is so much more important than than prepublication that it should be preserved at all costs, or vice versa. People hold all of those views, and it would take an even longer post to pull all that apart.

Instead, I’d to think about how to reconcile these two assumptions, because I do believe them both, and how to obtain an engineering trade-off that aims at most of the advantages of both, most of the time.

A lot of people have said that allowing papers to be prepublished anonymously would be a good compromise. An appealing idea, but I worry that it may be a bad one. Instead, I’ll argue that a good compromise is this: Accept that papers will be de-blinded, but design the double-blind review process to compensate.

Perhaps the underlying point is that the conflict isn’t black and white. For double blind to work, it’s not necessary for 100% of the submissions to be unblindable, i.e., have their author identities be undiscoverable online. I might even suggest that it’s possible to have effective double blind when all author identities are available online. Just because a paper is unblindable does not mean that the reviewers are unblinded — perhaps they have not seen it, or perhaps they saw it in an email with 100 other papers and don’t remember having seen it.

What shouldn’t we do?

There are some recommendations that I’ve seen that unfortunately I don’t think will work.

Anti-Recommendation 1: I know. Let’s have an anonymous version of arxiv.

Lots of people have suggested allowing authors to prepublish papers anonymously (incidentally, there are amusing precendents for this in the history of mathematics). This could be implemented via an overlay of arxiv, or a new feature added to arxiv itself, that would allow authors to temporarily hide their identity. Let’s call this AnonArxiv.

Submissions to AnonArxiv would be immediately available to all but without the author names. Then, once the paper is accepted, AnonArxiv would reveal the author names, while preserving the time stamp of the anonymous submission. The conference would then require that if submissions are prepublished, they must be prepublished anonymously; any other prepublished submissions will be rejected without review.

I used to think this was a cool idea. Now I don’t. It neglects a fundamental principle that we are sadly all familiar with, that most papers are rejected.

Let’s say I post a great paper to AnonArxiv and submit it to ACL. Like most papers it is rejected. I’m convinced that the reviewers have made a mistake, and so I want to resubmit it to EMNLP. How do we handle this?

We could (1) require resubmissions to remain anonymous. After rejection, I must choose whether to unblind the submission on AnonArxiv, in which case it cannot be resubmitted to other conferences, or whether to keep the submission anonymous, in which case the paper could spend a year-plus as anonymous, until it is finally accepted. This seems an unreasonable choice to force onto authors.

Or we could (2) allow authors, after one rejection, to unblind their AnonArxiv submissions and resubmit to a future conference. This has the benefit that papers only spend a few months as public-but-anonymous, which is not so bad. But I’m not sure it works. For one, this is difficult to enforce, because apart from the honor system, the information about whether a paper was previously submitted is confidential (keep in mind that the original submission might have been outside the NLP community). But more fundamentally, what would we do for first-time submissions that violate this rule, reject them without review? How would we justify doing that when there are many second-time submissions whose authors are already public?

This would also mean that if I submit a paper to a conference outside of NLP which allows prepublication and get rejected, I would not be able subsequently decide that an NLP conference would be a better fit, and resubmit there. It might be possible to implement a special dispensation in this case, though.

With some regret, I come to the conclusion that AnonArxiv won’t work. That said, AnonArxiv variant (2) might work if a large enough percentage of submissions were first time submits. Then we’d have the majority of the papers on AnonArxiv, and hence unblindable, which might be good enough.

Perhaps Anti-Recommendation: Require prepublication to be declared

ACL 2017 required all submissions to declare if they had been prepublished. Reviewers were notified that a paper had been prepublished. Prepublished papers that were not declared as such were summarily rejected. Unfortunately I don’t understand the rationale for how this stringent requirement was meant to help. Remember, the goal is not to prevent all papers from being unblind-able, it’s to prevent too many papers from being unblind-ed.

This could be a good idea if the hope is to warn reviewers that they should be careful about searching the web for the paper’s title during the review process. The problem with this idea though, is that it does not help if the authors very reasonably prepublish the paper just after submission. So really, all reviewers need to be careful, all the time, and the extra heads up maybe isn’t too helpful.

If the idea was to simply to gain more information about how many papers are prepublished, then I totally agree with asking the question, but I do not see why penalties for non-compliance were necessary.

If the idea was to handle prepublished papers differently in the review process than non-prepublished ones, then I am not sure why this is necessary. Instead, I’ll advocate below that we handle the review process of unblinded papers differently.

So I would argue that it might make sense for conferences to request that authors declare prepublication, but that no penalties for noncompliance be used in future years.

Recommendation-But-That’s-Actually-An-Orthogonal-Issue: Let’s use OpenReview.Net

I’ve also read the suggestion that the NLP community switch to running conferences on OpenReview. I love OpenReview, and I eagerly await the day when I can go onto a site like OpenReview and pull up any paper in computer science, from any venue, from any year past or present, and find a lively and informative discussion online.

But OpenReview is a software platform, not a reviewing process. It’s specifically designed to allow conference chairs to configure what information about the reviews and authors should be made public and when. It’s not designed to answer the policy questions about whether submissions should be anonymous and what happens after they are rejected.

All right, wise guy, so …?

One way to square the circle is to try adapting reviewing norms to adjust and compensate for the fact that it is more likely for papers to be unblinded.

Conferences are already doing these things, so I don’t claim to have any new ideas. But I think it’s useful to bring together the arguments for these ideas, rather than having programme chairs have to reconstruct these arguments for themselves every year.

Recommendation 1: Clarify Norms for Reviewers

Even if the author information for all submissions are public online, then reviewers, area chairs, and programme chairs can take steps to reduce the chances that a submission is unblinded, and to minimize the consequences when one is.

Reviewers should avoid making Web searches that would be likely to reveal the authors of the paper. It can often happen that a well-meaning search for related work inadvertently turns up an unblinded copy of the paper. I am not saying that reviewers should never search for related work, but it carries risks — it always has (ten years ago I had a reviewer of one of my papers deblinded by a tech report) — and reviewers should try to avoid it.

If the reviewer feels that a Web search is necessary, they should hold off until they have read the paper completely and formed an initial impression of it. This allows reviewers to apply the bias of cognitive dissonance to counteract the potential bias of unblinding.

If reviewers learn the author identities, then they must let the relevant person — who could variously go by the title “programme chair”, “programme committee member”, “meta-reviewer”, “senior PC member”, “area chair”, etc; I’ll use the term “area chair” (AC) — know this right away.

Area chairs should be prepared to apply their judgment to weigh the reviewers’ comments differently when some reviewers are unblinded. Consider a paper like this: it has two negative reviews and one positive review, but the positive reviewer has been unblinded, the paper comes from a famous group so there is possibility of unconscious bias, and the AC believes that the negative comments have merit. Then the AC should be prepared to give less weight to the positive reviewer. In other examples, perhaps all three reviews are positive, or the authors are lesser-known, and so unlikely to engender positive bias. Then downweighting unblinded reviewers may not be necessary.

Programme chairs should carefully write their instructions to reviewers and area chairs to make these expectations clear. They should also be prepared to assist ACs with borderline cases where there is possibility of bias.

We don’t know if these steps alone will reduce the percentage of unblind-ed submissions to an acceptable level. For example, if the percentage of unblinded reviews reaches, say, 80%, this recommendation becomes more like a band-aid that would be unlikely to preserve the benefits of double-blind review. Which brings us to the next point.

Recommendation 2: Measure and Monitor

Much of the heat around this discussion may be because we are, as it were, debating in the blind. It is not difficult to gather more evidence than we have now:

  • We should be able to measure and publicize the percentage of submissions and accepted papers which are prepublished.

  • Although a bit more delicate, we should be able to estimate the percentage of submissions which are not first time submits.

  • If we follow recommendation 1, we should also be able to monitor the percentage of reviews which are unblinded and the percentage of submissions which have had 1, 2, or 3 unblinded reviews.

  • We should also attempt to record measures of diversity in the accepted papers in terms of authors and institutions. We should keep tracking those measures, monitor for decreases, and the presumed negative correlation with percentage of unblindedness.

Updated 8 May 2018: In discussion after this post was first published, a colleague pointed me to the PLDI FAQ on double blind, which has some good ideas. Another idea which I am starting to see gain traction is to have a “blackout period” in which authors are expected not to post their papers on arXiv or social media starting from a month before the conference deadline, and continuing throughout the review period. This is another interesting compromise which seems to nicely handle the resubmission problems of the AnonArxiv approach.

The not-so-great checked vs carry on debate

By Charles Sutton on September 2, 2017

This time, some advice of a lighter sort. How light? Depends how well you pack!

I hear a lot of frequent air travelers say that you should never check bags if you can avoid it, but I think it’s not always clear cut.

I do usually travel with only a carry on. This has a lot of advantages:

  • You don’t need to queue at the ticket counter to drop off bags. If you’ve checked in online, using your phone is easiest but printing your boarding pass is still good, then you can go straight to security and the gate.

  • You don’t need to wait at baggage claim on arrival. This is an especially big win if you are arriving late at night.

  • It’s easier to switch flights at the last minute. You might wonder why you’d want to do this if you’ve planned ahead. You’ll stop wondering when your first flight arrives early and you wish you could jump on an earlier connecting flight. Or when your first flight is delayed but you still have a chance to catch a different connecting flight. I’m still annoyed at the time United Airlines prevented me from doing this by checking my bag at the gate unnecessarily.

All that said, for some trips it can be more convenient to check your bags, even if you could squeeze into a smaller bag. Why?

  • You can bring a bigger bag that holds more. (duh)

  • You don’t need to lug the bag around the airport. This removes stress during a long connection.

  • You can fill up your checked bag with liquids and gels. This is especially useful if you’d like to bring gifts of food and drink back to your friends, family, or perhaps to a select group of academic bloggers who you particularly admire.

  • If you are running a few minutes late to the gate, the plane is more likely to wait for you, because if they don’t, they need to find and remove your bag from the hold.

For direct flights I think it’s clear that carry on wins most of the time, but if you have a connection then it may well depend on your itinerary.

Tags: travel tips