How to do a research project

This fall, I have taken up an exciting new opportunity to work on a research project of my own. It will involve machine learning and medicine and will be exceptionally awesome. The ultimate goal is to produce a paper at the end, which would be even more awesome. In the past, I have worked on several projects of that type and, while most of them have been quite a bit of fun and I have learned a lot, I always had the impression that I could do better. This post will be dedicated to analyzing my past mistakes, looking for ways to avoid them and searching for new experimental techniques to try.

My past mistakes

These are common traps I always fall into, but will hopefully be able to avoid this time.

• Productivity drop-offs: Starting is pretty easy because there is lots of excitement and not a lot of obstacles in the way. As a project advances, more and more problems arise and it becomes less fun. I think this is a normal process and can not really be avoided, however, there might be ways to reduce its impact. I will start writing a productivity journal to chronicle every day and attempt not to get lost in the details.
• Small inefficiencies: Particularly towards the end of a project, time becomes a factor, and it is no longer possible to do everything. Stuff that pays off in the short term (e.g. writing a paper) has to contest with stuff that pays off in the long term (e.g. learning to code more efficiently). In the past, the short-term work has won most of the time and it is quite embarrassing to admit that I only recently learned to access git via command line. As a solution, I will note down possible productivity enhancements in my journal while working and set some time aside at the end of the day to investigate them. Additionally, I might read a book on effective software development.
• ML training: Training a machine learning algorithm can take a lot of time and should be planned rigorously. During my last project, I was often too lazy to do this and sometimes, after letting the computer run for a night, realized that I had made a mistake with the hyperparameter settings and the generated data was useless. Therefore, I will write a training protocol prior to each run where I record the purpose of the experiment and briefly discuss the hyperparameter values chosen.
• Literature research: Using Google to do the initial literature research is fine if you know what you are looking for, however, it can also lead you down a very narrow solution path and make you disregard superior approaches. Reading some general books on the topic beforehand will hopefully give me a better understanding of the topic and avoid these pitfalls.
• Frameworks: For some reason, in the past, I have often used direct implementations of ML algorithms. I learned a lot and it was quite fascinating, however, it took a lot of time and I didn’t always have a clear idea of which hyperparameters were preselected. Luckily, frameworks like Keras, TensorFlow and scikit-learn exist to quickly create a model and fit it to the data. I will make heavy use of that.

Research productivity tips

These are some additional tips that are immediately actionable.

• Tom Silver in Lessons from My First Two Years of AI Research recommends using visualization techniques, such as Tensorboard to interpret neural networks and ML structures
• From the same article:
  _There is, of course, no recipe for research progress, and fumbling around in the dark is part of (most of) the process. However, I now find it much easier and more fulfilling to structure my work around measurable objectives. If I have very little idea what I’m doing next, the objective can be: write down a vague idea in the greatest detail available; if, in the course of writing the idea I rule it out, write down the reason for ruling it out (rather than scrapping the whole thing and losing the measure of progress). In the absence of any ideas, progress can take the form of papers read or conversations with colleagues had. By the end of each day, I now try to have some tangible evidence of my work. Even if the ideas are never used, my morale is much improved, and I need not worry about wasting future cycles on the same ideas that I ruled out that day._I hope that I will have this covered with the journal
• Philip Guo, who is now an associate professor of Cognitive Science at UC San Diego, immortalized the struggles he faced during his six-year Ph.D. studies at Stanford in his memoir The Ph.D. Grind. (The original website seems to be deleted, but you can find the pdf scattered around the web). From this, I have taken the following excerpts:                              Tom [one of his supervisors] _would drop by my office at 5pm each afternoon before he left work to check up on my progress. Although daily check-ups could potentially be stressful, I actually found them immensely helpful since Tom wasn’t intimidating or judgmental at all. Getting immediate daily feedback made it easy for me to stay focused and motivated. The combination of a well-defined, short-term goal and continual helpful feedback made my internship workdays much more productive than those during my previous three years of grad school._And then social time was over; it was time to grind. In early November 2011, I turned into a programming beast for the final time in grad school to transform my Burrito idea [a software project] into a working prototype. I did 72 consecutive days of programming with only 5 days of breaks spread intermittently throughout the 2.5-month sprint. This period was the longest I had ever sustained an almost-painful level of nonstop intensity thus far. My initial CDE [another software project] burst during my fifth year was only 21 days of grinding, and this burst was over three times as long. I worked straight through Thanksgiving, Christmas, and New Year’s Eve, relentlessly focused on my goal of getting Burrito working well enough to submit a conference paper by the middle of January 2012. For those few months, I morphed into an antisocial grump who shunned all distraction and became deeply immersed in my craft. All I thought about was computer code; I could barely speak in coherent English sentences except during my weekly progress meetings with Margo. Even though I appeared and acted subhuman (i.e., an un__shaven disheveled mess), my emotional state was blissful. I was pro__gramming and debugging for over ten hours per day, but my mind was quite relaxed since my technical skills were well-calibrated for the challenges I faced

Further productivity methods I might try

My general approach to productivity is similar to the book Deep Work by Cal Newport: Start early, block all distractions and create a routine. This has worked very well for me in the past and there will probably be a post on that here at some time in the future. Still, I am always looking for new things to try. The following is a list of things I would like to try at some point.

• Productive meditation: This is something that Newport recommends. While doing a mindless activity like walking or cleaning, instead of thinking about what to have for dinner, try meditating on a (work-related) problem. I have tried this in the past and had problems staying focused, but a new attempt can’t hurt.
• A productivity cheat sheet: I found this tip in this thread. In order not to forget what productivity enhancements out there, write them down and keep them in sight.
• I have found a recommendation for something called “monoidealism”. The trick is to get into a trance-like state where the focus is solely on a single task. The concept seems interesting, but the comment is very short on details and I couldn’t find anything else about it. For me, the closest analogue seems to be meditation. I will try it and maybe write a post on how well it worked.
• On a similar note, there is “ultraworking”. It basically means working for a long time without breaks or distractions. This guy  programmed for 120 hours in a single week and reports that he felt happy most of the time. I still want to have a life, but something like 70 hours could be possible and still leave time for running and socializing.
• Scott Alexander proposes Applied Picoeconomics:
  The essence of picoeconomics is formally binding yourself to a rule with as few loopholes as possible. So the technique I decided to test was to write out an oath detailing exactly what I wanted to do, list in nauseating detail all of the conditions under which I could or could not be released from this oath, and then bind myself to it, with the knowledge that if I succeeded I would have a great method of self-improvement and if I failed I would be dooming myself to a life of laziness forever […].
• Working together: Alexey Guzey recommends co-working as a way to keep productivity high over long time periods. He and a friend or stranger meet in a virtual room. Everybody does their thing and they check in with each other every 30 minutes. (See also: https://guzey.com/productivity/). This is a technique I will probably try out as soon as my schedule allows it.