Sunday, May 01, 2016

Myth: "CS Researchers Don't Publish Code or Data"

A collaboration with Sam Tobin-Hochstadt, Assistant Professor at Indiana University.

There has been some buzz on social media about this "Extremely Angry" Twitter thread. Mike Hoye, Engineering Community Manager for Firefox at Mozilla expressed frustration about getting access to the products of research. It turns out that many other people are angry about this too.

While there are certainly legitimate aspects to these complaints, we’d like to address a specific misperception from this Twitter thread: the claim that "CS researchers don't publish code or data." The data simply shows this is not true.

First of all, while the Repeatability in Computer Science study from a few years ago highlighted some issues with reproducibility in our field, it revealed that a significant fraction of researchers (226 out of 402) in systems conferences have code available either directly linked from the paper, or on request.

Additionally, in the last few years, conferences in Programming Languages and Software Engineering have been pushing for more standardization of code-sharing and repeatability of results through Artifact Evaluation Committees. There is a comprehensive summary of Artifact Evaluation in our field here. (In fact, Jean is co-chairing the POPL 2017 AEC with Stephen Chong.) According to the site, artifacts are evaluated according to the following criteria:
  • Consistent with the paper. Does the artifact substantiate and help to reproduce the claims in the paper?
  • Complete. What is the fraction of the results that can be reproduced?
  • Well documented. Does the artifact describe and demonstrate how to apply the presented method to a new input?
  • Easy to reuse. How easy is it to reuse the provided artifact? 
The most detailed documentation is associated with the AEC for OOPSLA 2013, where 50 papers were accepted, 18 artifacts passed evaluation, and 3 artifacts were rejected. For PLDI 2014, 20 of of 50 papers submitted artifacts and 12 passed. By PLDI 2015, 27 papers (out of 52) had had approved artifacts. Even POPL, the “theoretical” PL conference, had 21 papers with approved artifacts by 2016.

For those wondering why more artifacts are not passing yet, here is a transcribed discussion by Edward Yang from PLDI 2014. The biggest takeaways are that 1) many people care about getting the community to share reproducible and reusable code and 2) it takes time to figure out the best ways to share research code. (That academia’s job is not to produce shippable products, as Sam pointed out on Twitter, is the subject of a longer conversation.)

While it’s going to take time for us to develop practices and standards that encourage reproducibility and reusability, we’ve already seen some improvements. Over the years, Artifact Evaluation has become more standardized and committees have moved towards asking researchers to package code in VMs if possible to ensure long-term reproducibility. Here are the latest instructions for authors.

Yes, we can always do better to push towards making all of our papers and code available and reusable. Yes, researchers can do better in helping bridge the communication gap between academia and industry--and this is something we've both worked at. But the evidence shows that the academic community is certainly sharing our code--and that we’ve been doing a better job of it each year.

Note: It would be really cool if someone did a survey of individual researchers. As Sam pointed out on Twitter, many of our colleagues use GitHub or other social version control and push their code even before the papers come out.


UPDATE! Here is a survey for academics to report on how we share code. Please fill it out so we can see what the numbers are like! Thanks to Emery Berger, Professor at UMass Amherst, for conducting the survey.

Related update. Some conversations with others reminded me that the times I haven't shared my code, it has been because I was collaborating with companies and corporate IP policies prevented me from sharing. (In fact, this was one of the reasons I preferred to stay in academia.) The survey above asks about this. I'm curious how the numbers come out.

Thursday, March 17, 2016

What The Bachelor Teaches Us About Choosing a PhD Advisor

This post was an experiment in simultaneous co-writing with Claire Le Goues, fellow CMU Computer Science professor, fellow undergraduate alumna of advisor Greg Morrisett from the time he was at Harvard University, and fellow reality TV enthusiast.


In academia, many PhD students are lucky enough to receive offers of admission from several different programs, and thus the opportunity to work with a handful of different advisors. During the decision-making process, the prospective advisors engage in Skype calls and often a recruitment open house filled with meetings and fun social events to compete for the favor of each PhD student. The end goal is the PhD student’s decision to work with the advisor.

 A couple of months is a short time to decide to be someone’s student for up to ten years. Especially when you only see each person briefly--and at their best. Especially when you’re trying to make a decision between so many people. Especially when you might not even be sure what you want. Especially when you’re tired from all the travel.

That’s right: the PhD student recruitment process is like The Bachelor.* On the popular reality show The Bachelor, a couple dozen eligible bachelorettes go on dates around the world over the course of twelve weeks to compete for the “final rose,” usually accompanied by a proposal of marriage, from the protagonist bachelor. In our world, the PhD student is the protagonist, and the potential advisers are the bachelorettes. (For those concerned about the gender asymmetry: there is also an inverse show, The Bachelorette.**)

It is not surprising that both on The Bachelor and in PhD-land, there are mistakes. On the 14th season of The Bachelor, Jake Pavelka famously chose Vienna Girardi despite the warnings of all other contestants. While Jake saw only the fun side of Vienna, the other contestants saw how self-centered and unpleasant she could really be. Many contestants were sent home for trying to warn Jake that Vienna “may not be here for the right reasons.” Similarly, on the 16th season, Ben Flajnik was so dazzled by Courtney Robertson’s beauty that he dismissed all hints that she was “different in the house than she is with you.” Viewers across America watched as Courtney charmed Ben while bullying the other contestants, telling them that she was “not here to make friends.”

It is interesting that both on The Bachelor and in academia, the multiple suitors often have access to information that the protagonist does not. The suitors all live together and interact with one another without the protagonist; professors are part of a wide social/professional network and often know one another from graduate school, conferences, and the like. In both cases, however, the person in a position of choice has plenty of reason to doubt the motivations of one suitor saying something negative about another.

Indeed, the smart suitors typically avoid saying negative things about competitors, as doing so typically results in a swift diminishment of their own prospects. Similarly, you will not hear much that is negative from your potential advisors during the PhD recruitment process. All other potential advisors are “brilliant researchers” who “do great work.” But if there’s anything we can learn from The Bachelor, it’s that it pays to pay attention to red flags. There is no such thing as an abusive advisor, only an “incompatible advising style.” The advisor who requires students to work weekends becomes “hands-on”, or someone who “drives students hard.” The advisor who only pays attention to students after they’ve proven themselves becomes “hands-off.” The advisor who barely pays any attention at all to students becomes someone who “gives students a lot of freedom to find themselves.” These ways of describing potential advisors can certainly be confusing. On the one hand, a potential advisor and their students may not have the potential student’s best interests in mind. On the other hand, potentially coded words may actually be the truth.

Fortunately, a key difference between PhD recruitment and The Bachelor is that PhD students are allowed to talk to whomever they want about potential advisors. This means they can talk to current students (which may be like talking to the family of potential bachelorettes in that they have a conflict of interest), graduated students, and former students who ended the advisor-advisee relationship before or without graduating. Though the tabloids do not cover prematurely terminated advisor-advisee relationships the way they covered the breakups of Jake and Vienna or Ben and Courtney, it is possible--and often very productive--to find out about these situations if you ask around. (We also strongly recommend asking a trusted mentor for what should be considered a red flag in a potential advising situation--there are more than we can list!)

When doing due diligence, it’s helpful to ask for concrete facts. How was Ben Flajnik supposed to know that “one way with you, but another way in the house” really meant “Courtney walks past us creepily when we’re having fun and berates us?” Besides, one person’s nightmare might be another person’s ideal. Maybe Ben wants a partner to do these kinds of activities with. (Though apparently not, since he broke things off with Courtney after watching footage of his season.) Similarly, when asking about potential advisors, it is helpful to know whether “hands-on” means “they give me guidance whenever I want it” or “they check in every day” or “they require me to work weekends whether I like it or not.” Daily checkins might be perfect for one student and terrible for another. Asking concretely about requirements and availability can go a long way in resolving possible confusion. Plus, concrete details are much more difficult to fabricate believably.

It’s also important to trust, but verify. On the most recent season, Leah lied to Ben Higgins by saying that obvious frontrunner Lauren H was different in the house than with him. But Leah was the only person who said this, Lauren’s shocked reaction when confronted was very believable, and all other available evidence suggested that Lauren was as warm and friendly with the other women in the competition as she appeared to Ben. Ben sent Leah home and ultimately proposed to Lauren. The lesson is that if you do hear something negative about a potential advisor, asking them--or their students--directly about it, as well as observing the other evidence at hand, are good ways to see if the rumor is true.

We can also learn from Bachelor history that the success of the whirlwind matching process is not sufficient to ensure a successful future. The handful of successful bachelor couples probably succeeded because they actually dated/got to know each other after the final rose and before getting married. Similarly, it is helpful for PhD students to take as much time as they can within the parameters of their program before confirming an advisor. This may mean Skyping with the advisor after the open house or rotating with the advisor’s group before deciding to join. It is also important to remember that the final rose doesn’t determine your fate forever. Just as Jason Mesnick ended up marrying his second choice from The Bachelor season 13 after the show ended, many successful researchers switched advisors at least once during their PhDs***.

Finally, remember that JoJo could, after a couple of months, reflect on her unexpected breakup with Ben Higgins with detached but understanding affection and is now going on to star on her own season of The Bachelorette. Similarly, your potential but ultimately unsuccessful advisors will generally understand---even if not in the moment---that the process works in mysterious and unpredictable ways. Just like the Bachelor alums who are now friends in real life, feel free to approach and talk to your rejected advisers at conferences and in other settings, and consider collaborating with them in the future. After all, you already know you have interests in common.

* We consider this analogy uncreepy only because we do not actually advocate professors developing romantic relationships with their students.
** We acknowledge the problematic nature of The Bachelor/The Bachelorette along lines of heteronormativity, race, gender roles. But so many lessons.
*** Our undergraduate advisor Greg Morrisett has said that when he first went to Carnegie Mellon University, he thought he was going to work with a professor who ended up going to Brown. Then he wanted to work with another professor, who ultimately had too many students at the time to take on another. He worked for a while with a professor who did not ultimately end up advising his Ph.D. thesis. Greg did not end up with his final advisor until late in his graduate career. He has since had an illustrious, award-winning research career and is now the Dean of CIS at Cornell.

Tuesday, March 08, 2016

Women's Voices: A Reading List

While I have mixed feelings about celebrating the fact that women are often so underappreciated we need to set aside a day for honoring women, I'll use International Women's  Day as an excuse to put together a partial list of my favorite books with a strong female voice. This list contains a mix of memoir, manifesto, and fiction and is in alphabetical order by author.
  • Kate Bolick, Spinster. Bolick examines the lives of several women, including herself, who put off the traditional path of marriage and children to pursue other interests. Many of these "spinsters" go on to marry and have more traditional lives, but most do it on their own terms, and not until they have gotten what they want out of their single years.
  • Elena Ferrante, My Brilliant Friend (and the rest of the series). The novel My Brilliant Friend, translated from Italian, is the best depiction of a friendship--and the admiration and jealousy that comes with a close friendship--that I have ever read. I especially like that the book focuses on the friendship between the two main characters and their academic competition rather than a love story, as coming-of-age books about young women tend to do. The later books are beautiful portrayals of the ups and downs of navigating career success, romances, and relationships with family.
  • Estelle Freedman, The Essential Feminist Reader. Fantastic collection of essays by authors from Mary Wollstonecraft to Audre Lorde.
  • Germaine Greer, The Female Eunuch. Possibly my favorite feminist classic. Greer writes powerfully about the repression that comes from the traditional nuclear family. From the forward of the 21st anniversary edition, by way of Wikipedia: "The freedom I pleaded for twenty years ago was freedom to be a person, with dignity, integrity, nobility, passion, pride that constitute personhood. Freedom to run, shout, talk loudly and sit with your knees apart. Freedom to know and love the earth and all that swims, lies, and crawls upon it... most of the women in the world are still afraid, still hungry, still mute and loaded by religion with all kinds of fetters, masked, muzzled, mutilated and beaten."
  • Sheila Heti, How Should a Person Be. I love the way Heti experiments with different modes of writing to capture the existential angst, friendships, of being a twenty-something artistic person in what seems like a semi-autobiographical work
  • Belle De Jour, Secret Diary of a Call Girl. Well written and surprisingly relatable, Secret Diary might change the way you view sex workers and sexual empowerment. Fun fact: during the period that this book describes, the author was concurrently pursuing a science PhD in the UK.
  • Mindy Kaling, Why Not Me? I really like what comedienne Kaling has to say about her relationship with her work, how her work gives her confidence, and how she maintains this confidence while the rest of the world pays attention to things that should matter less than the quality of her work.
  • Beryl Markham, West With the Night. Markham was not just a famous beauty who had love affairs with talented men (including Antoine de Saint-ExupĂ©ry), but also one of the first bush pilots in Africa and a fantastic writer. As she writes about her work flying in Kenya and her love of the land, she comes across as such a strong, resourceful, and brave woman.
  • Caitlin Moran, How to Be a Woman. I love Moran's voice and wisdom as she talks about growing into a woman and navigating womanhood.
  • Cheryl Strayed, Tiny Beautiful Things. Strayed opens up about her own struggles and triumphs through the "Dear Sugar" advice column she writes. Beautifully written and emotionally powerful.
  • Virginia Woolf, A Room of One's Own. Woolf is one of my favorite writers in the English language and this short book makes some of the best arguments for gender equality that I have ever read or heard. One of my favorite lines, about why women need to be allowed to earn a comfortable income: "One cannot think well, love well, sleep well, if one has not dined well. The lamp in the spine does not light on beef and prunes."
Would love to hear your recommendations!

Saturday, March 05, 2016

The Angst Overhead

In high school, our crew coach often reminded us to relax our faces. Crew is a sport based on precision and pain. Though frowning is often the natural response, holding on to tension while rowing simply wastes energy.

Recently I have been wondering when this advice applies to to creative processes, especially as many believe that tension is necessary for producing great work. One of the stories in Italo Calvino's If on a Winter's Night a Traveler features two writers, an untroubled one who produces best-sellers with ease and a tortured one pursuing an elusive truth. Each is envious of the other: the first of the second's facility of creation, and the second of the first's depth of pursuit. In the story, each writer tries to be more like the other and becomes less effective than before. This dichotomy between between productivity and depth is one many of us believe in.

There is evidence that believing creation should be difficult can slow the process of creation. In her memoir The Art of Asking, singer Amanda Palmer talks about how public perceptions of artists, as well as artists' romanticization of their own processes, can hold creation back. Amanda presents the image of the artist as solitary, locked in an attic, brooding, and probably wearing a scarf. She then describes once breaking her own rules--not going on Twitter while writing a song, because artists are solitary--and producing one of her best pieces.

Still, many of us cling to the association between creation and masochism. I've seen it in myself and among my peers in academia. We want to produce good work in the world, so we produce pain in ourselves. Perhaps subconsciously, we deny ourselves that break from our desk, that conversation with a friend, because we believe that this might somehow get in the way of the creative process. The weight of what we want to achieve overwhelms us. It is difficult not to obsess until we have brought our abstract idea into the world of the concrete. But we fail to realize that this angst, while often a product of creation, does not produce creation.

Much of my development as a researcher has involved reducing this angst. Early in my PhD, I had an internship with a researcher who, after listening to me talk about my ideas, would say, "Just go do it." I would watch in amazement as he executed on difficult tasks of uncertain outcome with little apparent angst. Similarly, I learned from my advisor the power of calmly writing things on a whiteboard to process both ideas unfamiliar to us and ideas no one has ever presented before. It is not necessary to feel the weight of the entire project with each forward step. Allowing work-angst to absorb us--or worse, believing that the angst is what makes the work good--often only wastes energy.

Computer Scientists often think in terms of "overheads," how much a process slows down the time required to achieve the desired task. Angst often incurs significant overhead with little gain. Thus, to better pursue elusive truths, I am now pursuing minimal angst overhead.

Saturday, February 27, 2016

What My PhD Was Like

What it looks like to do a PhD: the view from my desk.
Recently, I have been talking to many prospective Computer Science PhD students about choosing graduate institutions and advisors. In addition to excitement, there is fear and confusion. The confusion comes not only from not knowing what to expect, but not even knowing how to think about what to expect.

One prospective student put it well when she said that she knows how she is in other work situations, but she has never done a PhD before, so she does not know what it will be like. She had been curious about things like my relationship with my advisor, my research trajectory, and whether I had hobbies. She mentioned speaking to other people at various points during the PhD, but not having the whole picture much. After this conversation I decided it would be helpful for more people to talk about their PhD experiences in entirety. Now that I am a few months out of my Computer Science PhD at MIT and am about to begin an Assistant Professor position at Carnegie Mellon University, I can talk about my PhD experience in whole. This is my version of Philip Guo's The Ph.D. Grind.

Like many other academics, I will say that graduate school was one of the best periods of my life. I grew immensely, made many friends, and learned about the research process. During my PhD, I largely got to determine my own research agenda and day-to-day schedule. I developed my tastes as a researcher. I developed my tastes as a human being. Also important to me is that I developed some of the most intellectually and emotionally fulfilling friendships of my life. I loved graduate school so much that I decided to stay in academia.

But this does not give the entire picture. "Loving graduate school" is different from loving a good meal, a thought-provoking film, a pet, or another person. A PhD is about training you to become a researcher. The further you are from being in an experimental field, the further this growth is about concrete skills and the more it is about existential questions of how you want to be. If you want to stay in academia, you must also balance the growth with ensuring you have enough to show at the end for the intellectual coming-out party that is the academic job market. This growth is painful and anxiogenic. Even if you are in a supportive environment and have great work-life balance, a PhD is not supposed to be a comfortable ride. The flip side is that, as my friend Alex describes it, during your PhD every day is "epic."

Enough philosophizing. Here, concretely, is what my PhD was like.

Summer 2008. At the end of the summer, I move into my office at MIT, where I am to be co-advised by Mike Ernst and Saman Amarasinghe. Saman pays for me to start early. I spend a couple of weeks reading papers, mostly about parallelism, and report to him every morning about what I've read. He gives me lots of advice about what life is going to be like. Mike Ernst has been in Europe this whole time. The last week of the summer, Mike Ernst calls me and tells me he is moving to the University of Washington.

2008-2009 (Year One). I'm recovering from a repetitive stress injury in my neck, so Saman advises me to take hard theory courses that involve little computer use. I take Advanced Algorithms and Machine Learning in the fall and Advanced Complexity Theory in the spring.

I get involved with and become lead author on a PLDI submission with Saman and Martin Rinard about a programming language for trading off power and accuracy. This paper is pretty good for having come into existence over the course of 1.5 months, but not objectively a great paper--and it gets rejected.

Ras Bodik, a professor at Berkeley who tried to recruit me, emails me telling me his student Armando Solar-Lezama is starting a professorship at MIT. I meet Armando and he sends me many papers to read. Soon we are meeting and working so much I no longer have time for the other project, which was a suboptimal fit anyway for various reasons. I like how much bandwidth Armando has for technical discussions, especially since he has no other students or teaching duties at the time. I am nervous that this high meeting frequency is unsustainable, so I try hard to recruit other students to Armando's group.

Armando and I submit a paper to OOPSLA but it is not good because we don't start writing it until three days before. I also submit a paper to the Haskell Workshop about my senior undergraduate thesis, but it is not good because I don't know how to write papers. My friend Jean-Baptiste who reads it asks why my blog is so good but my paper drafts are so bad.

Since I am still figuring out what I want to do, I schedule many meetings with professors and ask them how to do research. In addition to Saman's group meeting I also attend Daniel Jackson's group meeting. I go to many PhD defense and job talks. Both Martin and Saman tell me that in order to become a good researcher, I have to flounder and fail. I have never failed before. I take it as a challenge to do work risky enough to fail.

January of this year I attend my first conference, POPL. Martin pays for me to go after I email him asking how I can get to go to POPL. I meet many people, including many people at Microsoft Research. This eventually leads to my first internship.

Outside of work, I try to avoid using my computer due to my repetitive stress injury. Because of the injury, I am also aggressive about working 8-hour days and taking breaks on weekends. I am too lazy to make many friends so I spend my free time reading, cooking, doing yoga, and running.

Summer 2009. I intern at Microsoft Research in Redmond with Chris Hawblitzel. I had seen Chris's talk at POPL about verifying garbage collectors using Boogie and thought it was one of the coolest projects I had ever seen. I had not thought I was qualified to intern with him, but some of the people I met at POPL recommended me to him. This summer, we build Verve, an operating system verified for type and memory safety. I had never verified anything or built an operating system before. I work hard, 10-12 hours a day, because there is so much to learn and because the project is so fun. I work closely with Chris and he teaches me an incredible amount about working with low-level code. Outside of work, I hang out with the other interns. I live in Redmond but on weekends I go into the city, where I have two friends I stay with.

2009-2010 (Year Two). I continue to work with Armando, on the project that eventually becomes the Jeeves programming language. My first year we started with a vague idea that dynamic constraint-solving was useful for handling corner cases that static program synthesis could not handle well. We apply this to data imputation. We learn over the course of 1.5 years of paper submissions that people don't care much about data imputation, especially since our results are not particularly good. By the end of my second year, we start thinking that security might be a better application domain. This cumulative work is my masters thesis. We still have many things to figure out at this point so I am not that excited about my thesis. I try to ask for an extension but Armando tells me to just turn it in.

In the spring, Saman and I start what becomes an annual off-site retreat of all the research groups related to programming languages at MIT. People give research talks and we have philosophical discussions about the future of research, publishing, and education. At the first one, we realize there are many more people who have common interests than previously thought. After the first meeting, the professors meet together and decide to do more activities together from now on. From this point forward there is more cohesion and activities like weekly PL-related meetings.

At the beginning of this year, I start telling people that I wish I had more female friends, since I realize that there are many fewer women around me than before. My friend Neha Narula tells me to respond to a call for people interested in starting a graduate women's group and I meet Kay Furman, a PhD student in the Health Science Technologies program. Along with a couple of other graduate women we start Graduate Women at MIT. Such a group had never existed and our proposals are met with a fair amount of skepticism, more from students than from administrators. At this point I still don't have many friends, so I spent many evenings and weekends working on building the organization: recruiting, fundraising, inviting speakers, writing materials about why we need such a group, etc.

As part of my acquisition of more female friends I join a feminist discussion group, also invited by Neha. We get together about once a month for a dinner hosted in one of our homes. These discussions and my friendships with the women in the group shape many of my views over the next few years.

During this time, I look into acquiring more hobbies. I take some break-dancing classes at the Dance Complex but decide it's not my thing. I take up acrobatics (silks, rope, and static trapeze), for which I continue weekly classes on and off for about a year until I realize I am not serious enough about it (and travel far too much) to make continuing worthwhile.

Summer 2010. I intern with Nikhil Swamy and Juan Chen at Microsoft Research Redmond. They have a dependently typed language called Fine for verifying security properties. I ask to intern with them because I want to take my work in this direction. I spend most of my time working on a tool for stuffing proofs into bytecode and taking them back out. I learn a lot, but the proofs are too big and we never quite manage to get it to work. I also participate in many meetings and calls about the F* language, which is like Fine but supports proof erasure. I learn a lot about dependent type systems and security concerns.

I live in Seattle this summer, in a studio apartment on Capitol Hill. For traffic and productivity reasons, I take the 7am bus to work and stay at work until 7pm. There is a heated vinyasa yoga studio I go to after work most days. I love having a routine and Seattle in the summer and the energy of Capitol Hill at night, so I am very happy with my life. This summer, thanks to Chris's nomination, I also participate in the Lindau Meeting of Nobel Laureates, where Nobel laureates give talks about science, the research process, and the future of the world. I make friends with scientists from around the world and find the talks inspiring.

Chris and I have a paper on Verve that appears at PLDI. We win Best Paper Award. I am too young to realize that not all projects go this smoothly and not all papers are this good, so I am confused. Since this is the first paper I ever published, this becomes my new standard for paper goodness. It takes me years to realize that it is possible to do much less and still publish a paper.

2010-2011 (Year Three). Now I am done with classes, have no more routine, and have to return to working on my project I am not that excited about, so I am less happy than before. I have this OCaml interpreter implementation for proto-Jeeves that I have been working on. To make things scale, I implement a C backend and a garbage collector. It integrates with the Yices constraints solver. It is hard to do anything with this interpreter because it requires a lot of work to experiment with language features. I do not feel much hope about the project. I shop around for other projects. I consider doing an internship in hardware synthesis with David Bacon at IBM Research, but after talking to some people decide I should not switch to this area.

In the fall, I TA a new program analysis course taught by Arvind, Martin, and Armando. The professors are excited for me to help revamp the course. There are four course staff total and six registered students. Most of the work involves doing the problem sets to make sure they work. It takes a surprisingly amount of time to grade six assignments, since each assignment is likely to make different design decisions. I hold weekly recitation where three people attend. All three students are strong, so most of the time we talk about fun PL topics.

To take a break after going straight from turning in my masters thesis to doing a internship, I treat myself by taking a two-week vacation. I take a road trip through Romania (and also parts of Serbia and Hungary) with my friend Florian who I met at some conferences. My friend Kate, who I met through the feminist discussion group and who has moved to Europe for an Erasmus masters program, meets us in Vienna, where Florian is studying.

In February I attend my first workshop at Dagstuhl, a German castle that hosts focused computer science workshops. I have involved discussions many computer scientists working in similar areas. I find it excitiyng. Afterward I meet my friend Kate for a weekend in Brussels and we have adventures as usual.

This year, I also start to make more friends. I become good friends with my "cohort," the other students interested in programming languages and software engineering who I started with. I also become friends with many female scientists. I never had many female scientist friends before and this makes me happy because I feel much more understood.

Summer 2011. I decide to stay around MIT to focus on research. My groupmate and officemate Kuat Yessenov tells me it would be simple to implement my language as an embedded domain-specific language in Scala. Kuat makes a prototype implementation and begins to collaborate with me on the project. This changes everything. All of the sudden, the language has hope of becoming practical. A nice story about Jeeves emerges: it is a convenient and practical programming model that lets you factor out information flow policies. It seems natural and like the way to go--but formalization is not entirely straightforward and there are some interesting theoretical properties too. It is great to collaborate with Kuat, who is very smart. We submit a paper to POPL. We work hard, staying long nights in the office. I briefly become nocturnal in response to receiving feedback from my advisor in the evening and feeling compelled to make the changes before I sleep.

It rains a lot this summer and we both become sad. One day, Kuat leaves a fortune cookie fortune on my desk that says "It cannot rain every day."

2011-2012 (Year Four). Armando starts putting pressure on Kuat to return to his project so Kuat stops collaborating with me. I am now collaboratorless and still haven't published anything on this language, now called Jeeves, so I become more and more hopeless. A couple of my friends, Marek Olszewski and Michal Depa, leave school to do startups. Marek invites me to join him. I'm not ready yet, but I have significant FOMO (Fear Of Missing Out). I take a class about startups and become friends with Tiffany Kosolcharoen, then a student at Sloan. Tiffany and I spend hours together every week and she teaches me an immense amount about the startup world.

My friends begin to worry I will quit graduate school. My groupmate Rishabh Singh stops teasing me about my inferiority for the first time, looks at me seriously, and says, "Miss Yang! You cannot quit. You are an intellectual." Someone leaves a fortune cookie fortune on my desk that says "It does not take guts to quit." Sasa Misailovic, another one of my cohort, has an intervention lunch with me telling me that I like graduate school the most out of all of us, so if I quit then what does it mean for everyone else. I tell him everything is meaningless.

Our paper gets into POPL and everyone is relieved. I am relieved to confirm there is life after Verve. Armando is relieved for obvious reasons. My friends are relieved because they stop thinking I will quit graduate school.

During the spring I build a small conference management system using Jeeves, not because I want to but because Armando says it is important to build things to see what breaks. This takes a lot out of me because Jeeves doesn't really work yet and I previously knew little about building web applications. We run out of memory all the time and I have to wake up in the middle of the night to restart the server. For weeks after, my friends make fun of me by sending emails telling me my server is down. For weeks, Armando and I wake up with nightmares in the middle of the night that our server is down. During this process I also realize that in order for language-based solutions to work, we have to have a better story with interfacing with the database. I feel proud that I convince Armando this is an important direction of future work. I also concede that it is important to build things.

This spring I do some traveling. I win a Facebook Fellowship and travel to the Bay Area to give talks at Facebook, Google, and Berkeley. I attend another workshop at Dagstuhl and spend the weekend in Paris beforehand and meet my friend Kate in Barcelona after.

My social life becomes fantastic this year. I find the "party circuit" orbiting a nucleus of social Harvard and MIT graduate students. I meet many interesting and brilliant people.

Summer 2012. I intern at Facebook to gain "privacy street cred." I start in July because it's after the POPL deadline. The other interns starting at the same time are all Eastern European college boys. This becomes my friend group for the summer. I live in the city and commute to Menlo Park every day, staying on campus for 10-12 hours. I work hard, building a verifier for privacy policies using Haskell. I like the work but find it isolating because I'm the only person working on my project and it's not clear what I am allowed to talk about outside of work. I take daily walks with my friend Pieter Hooimeijer, who I knew from research and went to Facebook after his PhD. I work harder than I want to because of the potential high impact of this project: it is tied to a pre-launch project and if it launches well, this could be a really nice example of verification in production use at a large company. Armando flies out to Facebook at the end of the summer to discuss my project under NDA. I am so happy to see him.

2012-2013 (Year Five). I return to school but continue contracting for Facebook. I work on my verifier one day a week and fly out to the Bay Area every few weeks for a few days at a time to continue working on my project. I maintain two social lives, one in Cambridge and one in San Francisco. The bigger project I'm a part of gets cancelled around February, for political and not technical reasons. I feel very disappointed, especially because my verifier had been very close to being put into production. I also feel relieved, because I'm exhausted from having two jobs and two social lives. I make some small efforts to try to publish on what I built, but Armando is not excited and the team at Facebook has scattered. We file a patent.

The rest of my research is not satisfying this whole year. I work on extensions to Jeeves that Armando tells me are necessary for the work, but that I am not excited about. We work on integrity extensions to Jeeves that I don't think make sense. What I really want to do is figure out how to extend the Jeeves programming model to the database, but I can't work on this until after we finish this other stuff. I submit an uninspired paper to CAV. One of the reviews says, "I cannot believe there are four authors on this paper and so many typos." I am not excited about anything by the end of this year.

In the spring, I take a class in the Media Lab with Ethan Zuckerman about new media. Half the students are technologists and half are mid-career journalists at Harvard and MIT on various fellowships. This gets me thinking much more about media, online communication, and civic participation.

Summer 2013. I stay around school to work on the integrity story for Jeeves, which I continue to not believe in. Armando tells me it is time for me to graduate and go on the academic job market. I tell him the academic job market is a big commitment because 1) I don't know that working alone forever is the best way to achieve the things I want to achieve and 2) the work/life balance doesn't seem great. He says 1) is a result of how he chose to run his group, but if I don't want to put in the work of being a professor then I should wrap up my work as quickly as possible and move on with my life. We are both sad.

2013-2014 (Year Six). I spend a month having in-person and Skype meetings with people about other career paths, for instance working at LogicBlox or Jane Street Capital. After my mother reminds me I don't have to do computer science anymore, I consider becoming a writer or an artisan specializing in linocut block prints. Rocky, who works on my floor, tells me that usually everyone on the seventh floor can hear when I laugh, but he has not heard me laugh for six months.

During this time, I also visit Andrey Rybalchenko and Byron Cook at Microsoft Research Cambridge. I enjoy myself immensely and have many stimulating meetings and become exposed to the field of executable biology. (Later I find out this was a secret interview.) I determine that nothing else is as exciting to me and even though academia is a big commitment it seems worth it. I tell Armando I'm ready to go on the job market and he says he already told everyone I wasn't anymore and plus it's better for me to wait. I am more than fine with this, since a big issue had been that I don't want to leave yet.

After I determine that one of my main issues with my project is that nobody uses Scala for prototyping web apps, I decide to switch to Python. Armando lets me take on an MEng (masters) student to convert our implementation to Python. I find Travis Hance. He finishes the conversion in the fall and we spend all spring collaborating on building a web framework based on Jeeves. I am really excited to finally be thinking about the problem of extending the programming model to the database, especially as this is something I had wanted to do for two years. I love having Travis as a collaborator. We decide to submit to OSDI. During the weeks before OSDI, we work together for 10-12 hours a day. I introduce him to many Kendall Square restaurants. During our breaks, we create Haskell Ryan Gosling.

During this time I also decide to take my interest in writing more seriously. The previous year I had come across a former problem set partner Carl Schoonover on Business Insider's list of sexiest scientists in the world and discovered he had been achieving my science writing dreams: writing a book, speaking about science, and even starting a group called NeuWrite to promote collaborations between scientists and science writers. I had brought him to MIT to speak, which led to two things: 1) me assembling an RSVP list of Harvard and MIT graduate students interested in science writing and 2) Carl deciding I was a good person to take over the NeuWrite Boston group, as group membership was shrinking and the current leaders were moving away. One of the journalists from my new media class introduces me to science writer Amanda Gefter and we take over the group. We build a new group from the event RSVP list and Amanda's writer friends. I spend much of my free time 2014-2015 building the group, thinking about writing, and writing. I start publishing not only on my own blogs but also in more public venues. A couple of my Quora answers get published on Forbes and my pieces for The Muse get chosen to run in Forbes and Newsweek.

Summer 2014. PLDI is in Edinburgh at the beginning of June and my friend is getting married in Croatia at the end of June. Since Armando is paying for the trans-Atlantic flights, I scheme to remain in Europe for the month. I set up another visit to Microsoft Research Cambridge and a visit to the Max Planck Institute in Kaiserslautern, Germany. I get good feedback on my work during both visits. I then spend a long weekend in Berlin visiting my friend Kate and a week wandering around various parts of Croatia for vacation purposes. I see Oslo for a day on the way back. I find the entire trip relaxing but find it difficult to return to normal life after.

Our OSDI submission gets rejected, so I spend the rest of the summer cleaning up the paper and doing more work to submit to PLDI. I feel an incredible amount of angst about my uncertain future as I prepare for the job market. I spend one day a week doing extreme exercise, cycling 50+ miles or running 11+ miles, to obliterate all thoughts. For various reasons I also spend a lot of weekends in New York and make more friends with people working on data and journalism.

2014-2015 (Year Seven). I spend most of the fall preparing a submission about the Jeeves-based web framework to PLDI. I also work on my job applications. I go on the job market, interview, etc. I get some offers, accept a position at Carnegie Mellon University, and defer a year to collect myself before beginning and also to explore my interest in applying programming languages and biological modeling.

To keep myself from thinking about my uncertain future all the time, I start a cybersecurity accelerator, Cybersecurity Factory, with my friend Frank Wang, with the goal of helping research-minded people start companies. After Frank and I ran out of research things to talk about the natural next step was to develop a project of mutual interest and we're both interested pushing academic research into the world.

I don't have much of a social life this year. For a couple of my interviews I stay with friends beforehand and they listen to my talks. My friends Alison and Tobi make sure to force me out of my house for non-work reasons every once in a while. Every few weeks my friend Angela comes over and cooks me dinner. I remain grateful for my friends.

Summer 2015. I spend the first part of the summer working on writing up the Jeeves-based web framework work, which has continued to be rejected from every conference. I get the framework to be much faster and work with my collaborators on a cleaner semantics. We submit to POPL and then I work on my thesis. The introduction and related work take much longer than expected, maybe about a month.

During this time I spend maybe about a third of my time working on the accelerator and doing public communication about cybersecurity. We accepted two teams for our Cybersecurity Factory pilot program and we work closely with them to make sure they are getting the support they need. I spend a good amount of time on the phone with media people getting press for our program. I write an op-ed for the MIT Tech Review.

Somewhere in there I take a 36-hour trip to Singapore to give a talk at a data privacy workshop Saman is organizing. Saman gets the people to fly me business class so I can sleep in transit. It is my first time flying business class. I am confused about what to do with such frequent deliveries of hot towels to my seat.

I don't have much of a social life during this time either. Several people text me "you have to eat sometime" after I repeatedly decline meal invitations.

2015-2016. I defend! I take two weeks off and then start a postdoc at Harvard Medical School. I slowly ease myself back into having hobbies and seeing friends. Our POPL submission gets rejected and I finally spend a lot of time thinking about how to sell the work (rather than how to do the work). I learn that despite pressures to make the work better until the last minute, I need to fight to spend a long time on the writing, especially if the proposed idea is outside of the mainstream. A nice story emerges about a new and practical web programming model that allows programmers to factor out information flow policies across the multiple runtimes of the application and database. Our paper finally gets accepted at PLDI. I feel relieved to publish the final part of my thesis. And here we are now.

For those wondering what to take away, it is important to note that every PhD experience is unique influenced by factors such as the advisor relationship, institutional requirements, the funding situation (both professional and personal), and the academic and social environments. I am privileged to have had a relatively supportive environment and few additional pressures. My experience is most likely to generalize for other computer science PhD students at top schools, where the funding situation tends to be good and advisors tend to give students a fair amount of freedom.

You may also be interested in other posts I've written about my PhD:

  • You may be interested in reading the comment thread for this post on Hacker News.
  • My friend and former groupmate Nadia pointed out that during 2014-2015, I claim to have no social life "but we went to so many cool parties." It's true. Over the course of the year I did go to some cool parties.
  • My friend Michael, who I once dated, said I should mention the existence of romantic relationships during the PhD to provide a more complete picture of my life. Chris Martens also says in her blog post about her PhD that she is frustrated when people don't discuss the interplay between romantic partnerships and work. Over the course of my PhD I was in a few committed romantic relationships, mostly with other academics, mostly in other fields. Some of the time I was in relationships coincided with periods of extreme work. This was difficult to balance, but other academics tend to understand. I was never in a relationship where someone else was routinely doing any portion of the domestic labor. I was also single for a significant portion of my PhD.
  • Berkeley PhD student Stephen Tu has begun compiling a list of PhD reflection here.

Thursday, February 25, 2016

A Game Called "Professor"

I first wrote this as a Facebook post and am reposting it here so it's easier to find.

I have been playing a game called "Professor." Still learning the rules, but here's what I have so far.

You are assigned an avatar, institution, and number of lives. You start out with some amount of money and influence. You trade the money and influence for students, which cost a fixed amount of money and variable amount of influence. Students generate discoveries, which convert to influence at variable rates partly determined by attributes of your avatar and institution. Collaborating with other professors can increase money, influence, and discoveries, but may reduce your number of lives. You have to do service work in order to retain your influence, but this reduces your number of lives. You may gain more lives through taking vacation, but this puts discovery generation on hold.
If you do not collect a cumulative total of enough students, influence, and discoveries before the "tenure point," the game is over. The game also ends when you run out of lives.

I welcome other players to share tips and tricks.

Saturday, February 20, 2016

On Wearing Clothes

The author, wearing some clothes.
A few years ago, we organized a professional wardrobe event (video here) as part of one of our Graduate Women at MIT fall conference events. We received a fair amount of criticism from people who felt that fashion was "frivolous" and such an event would tarnish the reputation of the group.

I see where this criticism is coming from, but fashion is far from frivolous--especially for women. Many people have told me that people should not care so much about clothes and only care about what's inside. Most of these people also believe that "not caring about clothes" is communicated by wearing a fairly specific uniform, for instance free t-shirts, non-form-fitting jeans, and very functional shoes. Such uniforms are often not available to women (see "fake geek girl"). In Why So Slow?, on why  psychologist Virginia Valian writes that while men often have a professional uniform available to help them blend in, women tend to be "marked" and stand out no matter what they wear. Especially since clothing serves as a powerful social signal, it's important for women to take charge of what they wear and communicate how they want to be seen. (Many others have written about this. Here's a nice piece.)

In short, it requires a lot of thought to be a woman wearing clothes. Especially since my post on dressing for academic interviews got a lot of interest and discussion, I thought it would be interesting to talk about the positions I've developed about wearing clothes, as a woman in a male-dominated field and in general.

Been building a uniform a button-down shirts and dark pants.
Embrace the uniform at work. Virginia Valian says there is no "uniform" for women, but this does not mean there cannot be a uniform. Adopting a uniform is a bit of a prisoner's dilemma for women: for any individual woman, putting in the effort to dress well and stand out provides benefits but keeps women as a whole in the clothing rat race. But if this fashion editor can wear the same outfit every day, then we should be able to embrace at least the personal uniform. (Friends have pointed out that there is a de facto "uniform" for PhD women in science: ballet flats, jeans, t-shirt, cardigan.) Life has become much easier for me once I picked five outfits to wear to work each week. Life will become even easier once I figure out the Ultimate Outfit to wear every day forever.

From my "boy" phase.
It's fine to look feminine. I see many women in STEM go through the "man" (or, more generously, "androgynous") phase. I've gone through it myself. You cut your hair. You start wearing baggy pants and t-shirts and dressing like a boy. While there's nothing wrong with this, it's also important to realize it is not necessary to do this. I started thinking about why I did this after a friend pointed out my "boy" phase coincided with a period of time in college when I was struggling with not being listened to and hypothesized my gender had something to do with it. With time, I learned that changing my communication patterns, refusing to be ignored, and establishing myself in other ways went a lot further than dressing like a man. (I wrote this fairly controversial piece on changing one's communication patterns for a male-dominated work environment.) Though I still like to dress somewhat androgynously to avoid standing out too much (and was advised to do this especially for first impressions), I'm much more comfortable looking not-like-a-man these days.

A surprisingly functional feminine not-for-work outfit.
Embrace functionality. Much of women's clothing is optimized for being looked at, rather than moving in. While it is certainly possible to wear this kind of clothing as a woman on the go, having shoes you are worried about falling off of and clothes you are worried will become not-clothes without sufficient supervision places a certain amount of cognitive load. Since I'm all about men and women having comparable amounts of cognitive load, here are the ways I've come up with for achieving more parity:
  • Take flats seriously. Many women are taught that heels are the norm and flats are just in-between shoes. I've spent a fair amount of time finding flats that are unapologetically the only shoes I will be wearing. I really like oxfords (I have this pair, this pair, and this pair) and John Fluevog's designs. (I'm wearing a pair of Fluevogs in the photo above. Women have stopped me in the street to ask about them, making references to their quests for functional shoes that look good.)  That said, I also have several pairs of heels I can run in. (I really like the brand Born for this purpose.)
  • Embrace pockets. Years ago, a close friend was dating a French billionaire male chauvinist who told her “All ladies must carry a purse.” Easily impressed by money and power, neither of us left the house without a purse for at least a year. One day I woke up and realized it was a lot of work to always have to think about what purse went with my outfit and also to then carry the purse, so I started carrying things in my pockets whenever I could. After carrying my ID, a credit card, and a couple of bills in my front pants pocket for a while, I got a thin wallet insert (by Steward/Stand) for holding my essentials. (For pants, Outlier is really good about pockets. Many standard designer brands, such as 7 for All Mankind, are as well. Also, my dress above has pockets! Check out my friend Elizabeth's Pocketist blog for more pockets.) Sometimes if I don't have pockets I will put my things in my boot or, less reliably, in this garter pocket I acquired for the purpose of holding my things. I highly recommend not being a lady and not carrying a purse.
  • Find outwear that actually serve its function. I spent years ogling the raincoats and winter coats of my male friends for how they looked good and actually worked before I found my own versions of coats that don't make me choose between form and function. I've mostly been finding functional items by non-technical brands, but I also have friends who have had success finding fashionable pieces by technical brands. I've spent a lot of time drooling over coats by Nau, designed for this niche, but I haven't yet talked myself into spending money on one.

Look ma, no purse!
Pay attention to range of motion. One time I was at a friend's for dinner when I noticed my friend Caroline behaving differently than normal. She seemed more relaxed--perhaps more confident. Towards the end of the dinner, Caroline explained that she had been trying to adopt more dominant body language. Studies have shown that dominant displays such as spreading one's legs communicate--and help establish--power. Caroline said that she had success with this new body language: at an interview, she had taken care to sit with her legs open and the interviewers kept commenting on how "confident" and "competent" she seemed. Since then, I have also become more conscious of my body language. In doing so, I have noticed how traditionally feminine clothing such as dresses and heels require women to sit and move in certain way. While the movements these clothing items encourage is often associated with femininity and feminine attractiveness, they are not associated with non-sexual forms of power. (It's a whole other long conversation why it's important to have forms of power outside of sexual power.) For these reasons it is important when choosing clothes to consider whether they will permit corporeal assertions of power.

In conclusion, solving the clothing problem is an important step towards gender equality. I once heard that gender equality is when women are allowed to be mediocre--that is, women don't have to be clear standouts to justify why they deserve to be doing something. For clothing, gender equality comes when women aren't "marked," when they can blend in whether they've spent time thinking about their clothes or not. Though we're further from this point than some might think, community thoughtfulness and good discussion can go a long way. Until then, it's important not to dismiss conversations about clothing, as they serve an important function in moving things forward.