← Graph

Debug like a scientist

talk 22 connections

Maciej Rząsa's wroclove.rb 2024 single-speaker talk. Opens with the frustrations of bugs that 'work on your computer', 'hero-complex' colleagues dropping 1,000 lines of code, and 'hot-potato' reassignments. Uses the Michelson–Morley experiment and Einstein's special relativity (describing reality instead of yelling at it) as a parable for how physicists proposed hypotheses, designed experiments, and published results. Proposes 'hypothesis-driven development' as a mindset for debugging and illustrates it with three Chattermill war stories: (1) a cucumber integration-test suite failing because a graphql-preloading gem appeared broken — the real cause was a before_filter temporarily changing a configuration constant while Puma had threading enabled; confirmed by a Ruby script manually interleaving the before/after filters with business logic, and fixed by reverting the race-condition-introducing PR. (2) A distributed docker-compose setup intermittently failing CI with read-timeouts (not open-timeouts) — found via Google that excessive logging plus a Docker Compose bug could freeze containers; disabling health-check logging fixed the problem and gave a precise ask to the infra team. (3) Recurring production 502s every 30 minutes on single nodes: hypotheses included DB load balancers, slow clients, memory leaks, and application-level crons; by taking copious notes in Slack, correlating Grafana passenger-queue charts, and adding metrics for process spawn events, an Argentinian teammate picked up the investigation across time zones and spotted Passenger entering deployment-error mode because bootsnap's cache had grown to 10–16 GB (a six-month-old minor bootsnap upgrade changed the cache directory but the deployment script never cleared the old one) — making Passenger time out on worker startup. Closes with lessons on scaling debugging: work in parallel on multiple hypotheses, collaborate across time zones by publishing small observations, use the Socratic method with juniors, take notes during interruptions (rebutting the 'flow state' dogma), balance time in the 'lab' with time in the 'library', communicate out loud, and transfer knowledge to retain it. Q&A covers post-mortems (useful only in a healthy culture; heavy long-term fixes are not always worth it) and whether to write a paper on the method.

date
2024-03-22
type
talk
talk Debug like a scientist
about
Core methodology the talk proposes.
talk Debug like a scientist
about
Used as a parable for 'surprising behavior of reality' = a bug.
talk Debug like a scientist
about
Framed as the outcome of applying the scientific method rather than yelling at reality.
talk Debug like a scientist
about
Held up as the model: describe reality with clear assumptions instead of piling ad-hoc patches.
talk Debug like a scientist
about
Puma tool
First war story pinpoints a race condition in a cucumber suite running against a threaded Puma.
talk Debug like a scientist
about
Second war story traces CI read-timeouts to a Docker Compose + excessive-logging bug.
talk Debug like a scientist
about
Bootsnap tool
Third war story roots recurring 502s in a 10–16 GB bootsnap cache.
talk Debug like a scientist
about
Passenger tool
Passenger's deployment-error mode and queue-length metric surfaced the bootsnap problem.
talk Debug like a scientist
about
Taking notes is framed as local rubber-duck debugging.
talk Debug like a scientist
about
Grafana tool
Used at Chattermill to visualize passenger queue length and spawn events.
asked_at
Debug like a scientist talk
Audience Q&A question.
asked_at
Debug like a scientist talk
Audience Q&A question on process.
person Maciej Rząsa
authored
Debug like a scientist talk
Single-speaker author of the wroclove.rb 2024 talk.
from_talk
Debug like a scientist talk
Scaling-debugging section of the talk.
from_talk
Debug like a scientist talk
Mindset section of the talk.
from_talk
Debug like a scientist talk
Habits section of the talk.
from_talk
Debug like a scientist talk
Habits section of the talk.
from_talk
Debug like a scientist talk
Team-scaling lesson from the 30-minute-502 war story.
from_talk
Debug like a scientist talk
Repeated conclusion of each war story.
from_talk
Debug like a scientist talk
Closing recommendation of the talk.
talk Debug like a scientist
presented_at
Talk delivered at wroclove.rb 2024.
related_to
Debug like a scientist talk
Alina framed her lightning talk as a practical reference/companion to the earlier 'debug like a scientist' talk that day.

Provenance

Created
2026-04-17 16:17 seed
Read by
16 extractions