This is a post about managing the research process when juggling many projects and other commitments.
My idea-to-output pipeline:
I often have “a flash of potential insight.” This might be a thought about how there is a better way of solving a computational problem or a connection between unrelated concepts that might open new avenues for research exploration. These flashes happen while reading a paper, hearing a talk, walking down the street, or talking to a colleague. To say that these are half-baked ideas is an exaggeration and maybe “quarter-baked” is a better term. Let’s think of such a flash as the beginning of a pipeline. At the end of a pipeline is a research output — a paper, a lecture, a blog post, etc. It might take years to get here, e.g. a research paper.
What about the points in the middle of the pipeline? I used to think that it’s a continuous process; but with time I realized that there are discrete checkpoints, at least in the way I do it. Here is what I came up with:
1. A flash of potential insight
2. Converting the cloudy flash into a more concrete thought, often through writing it down.
3. Doing a dump of all accompanying thoughts onto paper (i.e. a brainstorm).
4. Evaluating which of this is novel and relevant — usually doing a literature review or looking at existing teaching materials.
5. Narrowing the initial brainstorm into a concrete project idea.
6-15: Execute the project idea: evaluate practical feasibility with preliminary results, maybe write a grant, match a student’s interest with the project, guide the student through the project, etc…
This post is not about steps 6 through 15, so I didn’t elaborate on them here. I’ve found that over my career, I’ve been discretizing the continuous process into these steps, starting from the end. That is, as a starting student the whole process was a continuous blur; then I recognized that there is a discrete stage 15, then stage 14, and so on. The initial 5 steps were all a continuous blur to me until somewhat recently, when I first noticed the existence of checkpoint 5, then later checkpoint 4, and so on. Only last week did I recognize the existence/importance of stage 3.
The problem:
At the moment I have a flash, I usually do not have time to follow up on it with the first five steps. I am usually busy and my mind is immersed in other projects. Maybe potential projects get lost along the way? I used to think that if a flash is truly promising, it will occur to me multiple times, and eventually I will follow up on it. But now this seems like a baseless assumption whose purpose was only to justify my status quo. Over time, I’ve applied a lot of scrutiny to the later stages of the pipeline (6-15) and tried to optimize the efficiency of those stages. Now, I want to do the same for steps 1-5. How many “flashes” eventually become outputs? I don’t know, since many flashes are not followed-up on and their existence is forgotten. How many of these flashes turn out to be nonsense when scrutinized? How many make it to at least to stage 5? I don’t know. How many flashes that could have led to outputs are lost because I forget about them? I don’t know.
Another problem is that the whole pipeline potentially stretches over many years and, since I am generally overwhelmed with the number of projects going on simultaneously, I don’t remember things well. When I don’t take good notes, I often find myself repeating steps that I have already done! For example, at the early stages, I sometimes browse through literature to get a sense of what has been done. If I don’t take good notes, then when I return to this in a month, I have to start over. Another problem is that the coolness and excitement that I have at the initial stages are sometimes lost by the time of writing the paper! Sure, sometimes it’s just that the original excitement was naive and didn’t account for things I learned later. But sometimes, I suspect that it’s just that after being down in the trenches of a project for a long time, I forget its original beauty.
What I also realized recently is that “taking good notes” is stage-dependent. For example, taking good notes at the end of stage 3 is just an unpolished, “natural flow” kind of text. Trying to do something more actually destroys the value of these notes. If I have the natural flow, I can capture the original excitement, and then a year later I can look at these notes and remember precisely why I was so excited initially. Moreover, if I require polished notes at the end of this stage then it makes it harder to find time to get through the stage and increases the chance that the flash will get forgotten. On the other hand, “taking good notes” during a literature search means being very precise, so that these notes can serve as a basis for precise published statements later.
.
The solution:
I want to take a more systematic approach in the future. I want to refine/improve/make-more-precise the pipeline steps, and make precise what kind of notes are most effective for each stage. I also don’t want ideas to unintentionally drop out of the pipeline. It should take very little time to bring an idea to the end of stage 3 (e.g. 30 minutes in a coffee shop). Now that I’ve broken down the initial stages into smaller, more manageable tasks, I think this is possible. By being systematic about this, I can iterate and refine both the stages of the pipeline and the specs for each deliverable.
Also, it is fine if ideas will get stuck in the pipeline because I can’t find time for them (e.g. I never get to stage 4). I just don’t want them to fall out. If they are stuck, then I know about it and I can always return to it. Moreover, I can analyze why they get stuck and fix the problem, e.g. I need to allocate more time to idea development.
Feedback:
This post stemmed from a “flash of potential insight” I had yesterday. Since I am on vacation, I had the opportunity to immerse myself into the pipeline of turning it into an output. I decided that a blog post would be a good output, for which the turnaround is really quick. But really these are all still half-baked thoughts in my head. I am really curious to hear about how other people approach the early stages of the pipeline.