It’s 3am. PagerDuty is screaming. Checkout is failing for some customers but not others. You SSH into a box, tail the logs, and you’re greeted with a wall of INFO: processing request lines with no order ID, no user ID, no trace of which downstream service choked. Now you’re not debugging — you’re archaeology.
I’ve been on the other side of that night more times than I’d like to admit. And honestly, the difference between a 5-minute fix and a 3-hour outage is almost never the bug itself. It’s whether your logs told you anything useful. So let’s talk about what good logging actually looks like — not the textbook “log everything” advice, but the stuff that saves you when production is on fire.
The “just scale microservices” question keeps coming up whenever Spark enters the conversation. It sounds logical — you already have distributed services, just throw more at the problem. But this comparison collapses under a pretty basic question: what kind of problem are you actually solving?
It Is Not a Database. Not a Queue. People come to Spark expecting something like a faster database or a smarter Kafka. Neither is accurate.