ICLR 2020: My first virtual conference
(or: How I learned to stop worrying and love virtual conferencing.)
Many computer science conferences are becoming virtual this year, due to travel restrictions. I enjoy physical conferences, so I was a bit skeptical about virtual conferences, despite their necessity. Since ICLR 2020 was the first major machine learning conference to be virtual, this was a good chance to test my skepticism. Here’s a personal report about the parts of the conference that I had a chance to experience, how well they worked for me, and what I would do differently the next time I attend a virtual conference.
At the top, the bottom line: ICLR was better than I expected a virtual conference could be. Both the format and the code should be a model for other computer science conferences that are going virtual. Massive congratulations are in order for all of the organizers, including the chairs Sasha Rush, Shakir Mohamed, Dawn Song, Kyunghyun Cho, and Martha White. There’s still more work to be done to make virtual conferences great, because we haven’t yet figured out serendipity, or how to recreate the fluid social dynamics of a large meeting. But now we have a great starting point to experiment from.
What is it? Who was there? What happened?
The International Conference of Learning Representations (ICLR) is one of the largest machine learning research conference. The physical ICLR 2019 conference has 2700 attendees. This year, the conference accepted 687 research papers from 2594 submissions.
The organizers of ICLR put an incredible amount of effort at short notice to build custom software to support the virtual conference (their code is open source, see links at the end). The conference format featured prerecorded talks, live Q&A sessions both for invited speakers, and text chat rooms. There were other parts of the conference where I didn’t spend much time: I only spent a little time at the workshops, and I didn’t have a chance to attend the virtual vendor booths or official conference socials. ICLR 2020 was big, even bigger than the in-person machine learning conferences: over 5600 registrations, 1300+ speakers, one million page views, and 100k+ video watches.
What I loved about Virtual ICLR
Suprisingly, I really loved the poster sessions. Every accepted research paper was presented as a virtual poster. Every paper had a web page that collected together links to the paper, the reviews (this is ICLR — they use openreview), a five-minute video by the authors, a text chat room, and links to a Zoom videoconference room specifically for the paper.
Every day of the conference had five 2-hour poster sessions, spread out to be convenient for participants from different time zones. Of course, the videos were always available, but each paper was assigned to two different poster session at which the authors could attend the Zoom room and answer questions. Assigning each paper to two sessions was designed to increase the number of time zones that were conventient for each poster.
The five-minute videos were my favorite part of the conference. They were synchronzied to slides, so if the talk was covering background you already knew, or if you missed a point, you could click back and forth in the slides. (In person, this is harder to do, and depends on the social and language skills of both participants.) I found the videos were a great way to get an overview of many papers and decide where I wanted to focus my time. They were much more polished and a little more in depth than the elevator pitch that you get at a physical poster when you ask, “Could you tell me a bit about this work?”
The Zoom rooms themselves were sparsely attended. I hopped into the rooms for probably a dozen papers, and usually I was the only one there, apart from the authors. Perhaps twice there was one person already asking a question. For me, this was great, because then I could start a ncie conversation with the authors (some tips on that later). The downside is I have learned a lot from hearing questions that my colleagues have asked at posters — that didn’t happen here.
The text chat rooms for each paper were active. People would ask detailed technical questions and the authors would respond. Some of the discussions were fairly in depth.
One way that virtual posters are arguably better than physical: you could not see who else was looking at the poster. Indeed, even though the Zoom rooms were generally quiet, the posters got a fair amount of attention. The median poster had 200 unique views, and the most popular had over 1000. Machine learning is a trend-seeking community, for as long as I’ve been a part of it. At physical conferences, posters from famous authors and research labs become very crowded, so much so that they become short talks (a poster session of repeated loops, like in Westworld) rather than scientific discussions. Hiding poster popularity prevented the bad aspects of the rich-get-richer dynamics.
Each poster session had its own page that listed the title and author of each paper (but no institutions, again to prevent rich-get-richer) along with an automatically generated thumbnail. These were randomly ordered for fairness.
The invited talks worked well also. These were prerecorded longer talks with a live Q&A. If I remember right, the video of the talks were made available the day before the Q&A. This worked well because what you get out of an invited talk, you can get just as much from a recording, and the live Q&As were very active. In some cases, speakers stayed for over an hour answering questions over both audio and text chat.
The overall paper visualization was well done. This plotted all of the papers as points in 2D space, organized by topic. You could highlight papers by keyword, authoor, and so on. These 2D visualizations are often a mess (I even have a paper discussing that), but this one seemed to locate papers together sensibly. I took some pride in that one of my papers was an outlier at the edge of the display.
Virtual vs Physical?
For me, there are two points to coming to a conference:
- Networking. Meeting new people who I can learn from, and learning new things from old colleagues.
- Broadening my view. I can attend a lot of posters and talks to get a broad snapshot of subareas that I might not have time to follow in detail when arxiv preprints come out. The curation function that a conference serves, as problematic as it is, really helps me here.
These worked differently at a virtual conference. For broadening my view, the virtual conference was actually better, because the five minute videos were overall nicer than the quick explanation that you might get at a physical poster.
For networking, as you would expect, my experience was more mixed. On the good side, it was easier to meet the poster presenters, because others were Zoom-shy. This was great for meeting PhD students. In a lot of ways, though, I found networking harder at a virtual conference. Here is what was harder to make happen:
- It was hard to connect with old colleagues, because I didn’t know who was participating in the virtual event when.
- I had fewer serendipitous meetings. I can’t count the number of research meetings that have started impromptu at local coffeeshops near the conference. Sometimes I will start talking to someone I don’t know well, only to find that there is an unexpected connection between our different research interests. This can result in collaborations, invited talks, grant proposals…
- Networking cascades. When you meet one person, they might introduce you to others. This can happen naturally, like when you are talking to someone and one of their colleagues invites them to lunch. This is harder in a virtual format.
Advice for participants: How to virtual-conference with aplomb
Attending a virtual conference is a different skill from attending a physical one. Here are some things that I will try to do better next time: For participants:
- As much as you can, clear your calendar. Treat the virtual conference as reserved time. You say, “That’s hard for me.” I know. It was hard for me too. But if you can reserve time to attend a physcial conference, you can do it for a virtual conference. Really, no one will stop you.
- Preparation helps. For physical conferences, I always have the best intentions to read lots of papers on the plane, but it doesn’t always happen. For virtual conferences, I think that preparation is more important because the face-to-face time (Zoom rooms etc) is even more limited. I wish that I had watched more of the five minute videos in advance of the poster sessions. I think it’s also good to be proactive about setting up individual meetings before the conference.
- How not to be shy during poster sessions. I have the impressions that many attendees were reluctant to participate in the poster sessions because it felt a bit socially awkward, and hard to have a good question.
I have some advice for asking virtual poster questions (this is an advice blog, after all). First, embrace the awkwardness. Yeah, it is a bit awkward at first, but once you start talking, it’s just another technical conversation like many you’ve had before. Second, yeah, you do need a question, but it doesn’t need to be brilliant, and you don’t need to have read the paper. “I watched the video, but I didn’t understand how the Thagomizer works. Could you tell me a bit more about that?” “Is your regularization scheme related to curriculum learning?” “I’m really interested in your work, because my recent work is about machine learning for baking sourdough. Do you think that your work could be applied to that?” It’s that easy to start a conversation!
What more could we do next time?
I take away two big positive lessons from virtual ICLR: poster sessions can work virtually, and text chat helps a lot. How can we do even better? How do we get more serendipity and networking in a virtual conferences?
To support social networking, there is lots of room to experiment with using videoconferencing creatively. Perhaps we could see lists of who is around the virtual conference at any one time (allowing people to explore anonymously if they prefer.) Perhaps we can have low friction ways to create a one-off videoconferencing rooms, which others can see and join. Maybe we could consider having “virtual receptions”: a one-hour block of time where many people are present to have discussions, and people can show up and self-organize videoconference discussions of even 3 or 4 people. I sometimes joke that perhaps we could virtually replicate running into someone in the hallway: 10% of the time, when you attempt to join the videoconference room for a poster, you are redirected to a videoconference with a random participant instead. That’s a bit silly, but random chat groups, perhaps in larger groups, and with safegards to prevent abuse, might feasible. Perhaps we will find creative ways to use social media during a virtual conference.
A great example of creative videoconferencing was ICLRTown: a 2D virtul world cum videoconferencing system. You moved a little 2D avatar around an 80s style graphic of a conference center. When you get close enough to someone else’s sprite, you see their video and can talk to them. We got chats of up to 9 people that way. This was fun to play with, but it still seems a work in progress.
Finally, I wonder to what extent the huge scale of ICLR actually helped it to be a virtual conference, making it easier to spread across timezones, or easier to get a critical mass for live Q&A. Most computer sciences conferences are smaller than ICLR, so different models might work well for them.
Resources about ICLR and virtual conferences
If this post wasn’t enough, here is more you can read if you find yourself in the exciting but scary position of organizing a virtual conference: