Skip to main content
Back to blog
•2 min read

Research Is 80% Debugging

researchcodingreality

i'm six weeks into my summer research position and i have an announcement:

research is debugging. research is all debugging.

the expectation

i pictured research as: think deep thoughts, have brilliant insights, write papers, change the world.

maybe occasionally stare pensively out a window while having eureka moments.

the reality

week 1: "why won't this install"

week 2: "why does this give different results every time"

week 3: "this number should not be negative"

week 4: "found the bug. it was a typo. kill me."

week 5: "the paper didn't mention this hyperparameter. guessing."

week 6: "RESULTS. WE HAVE RESULTS. oh wait they're wrong."

the breakdown

approximate time allocation:

  • 10% reading papers
  • 10% writing/thinking
  • 10% actual experiments that work
  • 70% debugging/fixing/investigating why things don't work

this is not an exaggeration. if anything, i'm being generous.

why debugging dominates

1. academic code is often broken

original paper: "we used standard preprocessing" reality: "standard" is undefined and their code is missing half the pipeline

2. environments are fragile

"works on my machine" is the official motto of research computing. dependencies, versions, cuda drivers... everything conspires against you.

3. ml bugs are subtle

your code runs. loss goes down. model trains. but results are slightly wrong because of a shape mismatch that silently broadcasts incorrectly.

no error messages. just bad science.

4. hardware fails

gpus run out of memory. servers crash. jobs get killed. checkpoints get corrupted. normal.

what i've learned

  • print statements are debugging tools. no shame.
  • test with tiny data first. always.
  • log everything. you will forget.
  • reproducibility is a feature. seed your random numbers.
  • ask for help early. time is limited.

is it worth it?

yes.

because when something finally works—when you see a result that nobody has seen before—it's incredible.

the 80% debugging makes the 20% discovery feel earned.

also i've gotten very good at reading stack traces. transferable skill.

current debugging status: why is my learning rate doing that. it should not do that.