{"id":308,"date":"2023-08-17T17:24:28","date_gmt":"2023-08-17T17:24:28","guid":{"rendered":"https:\/\/senser.tech\/?p=308"},"modified":"2023-09-28T06:38:29","modified_gmt":"2023-09-28T06:38:29","slug":"kubernetes-made-my-life-much-much-worse","status":"publish","type":"post","link":"https:\/\/senser.tech\/kubernetes-made-my-life-much-much-worse\/","title":{"rendered":"Kubernetes made my life much, much worse"},"content":{"rendered":"\n

Kubernetes was supposed to make managing distributed applications and infrastructure easy. No more worrying about servers or resilience: just deploy code and let Kubernetes handle the operational complexities like scaling, failovers, and load balancing.<\/p>\n\n\n\n

But the benefits don’t come for free. Kubernetes, alongside the distributed architecture of microservice-based applications, introduces complexity that makes debugging failures vastly more challenging.<\/p>\n\n\n\n

SRE and DevOps teams need new approaches to quickly identifying and resolving production issues. Here\u2019s why \u2013 and what to do about it.<\/p>\n\n\n\n

The curse of complexity<\/h4>\n\n\n\n

Kubernetes has become the de facto standard for container orchestration, adopted by 88% of companies in production environments according to the Cloud Native Computing Foundation<\/a>.<\/p>\n\n\n\n

The benefits are clear: it abstracts away infrastructure so you can easily deploy containerized applications and scale them on-demand. Kubernetes takes care of scheduling containers across clusters, restarting failed containers, load balancing, secrets management, and more. As a platform, it provides increased flexibility, scalability, and resilience compared to traditional monolithic applications.<\/p>\n\n\n\n

But Kubernetes comes with challenges \u2013 with the complexity of troubleshooting failures frequently cited by users as a major pain point.<\/p>\n\n\n\n

We\u2019ve seen SREs spend days trying to hunt down the root cause of issues in a Kubernetes environment. A service is timing out \u2013 is it a network problem, an overloaded pod, or a software bug? Containers keep crashing \u2013 is there a configuration error, resource constraint, or faulty application code? Bugs that would take minutes to debug in a monolithic app can take days to track down in Kubernetes.<\/p>\n\n\n\n

Common Kubernetes debugging hurdles include:<\/h4>\n\n\n\n

<\/p>\n\n\n\n

Lack of visibility<\/h5>\n\n\n\n

Kubernetes clusters run distributed workloads across nodes in the cluster. A single service can have pods scattered across multiple nodes. This distributed nature means you lack visibility into the entire system from a single vantage point. Traditional monitoring gives metrics for individual pods or services, not the correlations and interactions between them. It’s like debugging with blinders on.<\/p>\n\n\n\n

Moreover, since the application is distributed it is very dependent on the way Kubernetes decides to orchestrate it, creating a very vague line separating application issues and infrastructure issues.<\/p>\n\n\n\n

For example, say a web service dependent on a cache service is failing. Metrics show the web pods are overloaded. But traditional monitoring won’t highlight that the cache service pods on different nodes are also degraded, causing increased load on the web tier.<\/p>\n\n\n\n

Microservices-based architecture relies on small, preferably immutable units. Since the flow of the system is based on APIs \u2013 coupled with the way Kubernetes works \u2013 you’ll quickly discover that without visibility across nodes, you miss the root cause.<\/p>\n\n\n\n

Ephemeral environments<\/h5>\n\n\n\n

Pods have short lifetimes. They are constantly rescheduled across nodes based on resource needs. New pods are spun up for deployments while old ones are terminated. Issues present in one pod instance quickly disappear as the pod is killed. <\/p>\n\n\n\n

This ephemeral nature makes reproducing and tracing failures feel like whack-a-mole. For instance, a bug that causes occasional crashes will disappear into thin air as the pod restarts. Without tracing data showing the crash starting from the root request, it’s virtually impossible to reconstruct and debug.<\/p>\n\n\n\n

Complex networking<\/h5>\n\n\n\n

Pods and services communicate across a complex network fabric created by Kubernetes networking components like CoreDNS, kube-proxy, CNI plugins, etc. Connectivity issues could stem from misconfigurations in this network fabric, routing issues, DNS failures, firewall rules, and more.<\/p>\n\n\n\n

Since Kubernetes abstracts the networking layer to a degree, it\u2019s even harder to use traditional techniques to monitor it.<\/p>\n\n\n\n

Debugging networking issues requires visibility into Kubernetes network traffic flows, topology, and configurations – data that traditional monitoring lacks. Without this context, network issues turn into dead ends.<\/p>\n\n\n\n

Shared resources<\/h5>\n\n\n\n

In Kubernetes, workload containers share compute resources on nodes and storage resources on volumes. This sharing creates contention that can cause unexpected resource starvation or throttling issues.<\/p>\n\n\n\n

Slowdowns caused by resource contention are hard to differentiate from things like memory leaks or scaling problems with traditional metrics. For example, high CPU on a container could be from a memory leak or from another “noisy neighbor” container on the same node hogging a shared CPU.<\/p>\n\n\n\n

Blackbox containers<\/h5>\n\n\n\n

Traditional debugging tools centered around logs, APM agents, or JMX don’t provide full visibility in Kubernetes. You can’t easily “SSH into a container” – they are isolated and ephemeral. Agents only give metrics from inside the container without kernel or host context.<\/p>\n\n\n\n

This blackbox design means you have limited views into things like network calls between containers, host resource contention, kernel exceptions etc.<\/p>\n\n\n\n

Automation complexity<\/h5>\n\n\n\n

Kubernetes enables fully automated CI\/CD deployments. But this automation also hinders debugging by making it hard to reconstruct past failures. The automaticity restricts a human from manually recovering or diagnosing an issue postmortem.<\/p>\n\n\n\n

For example, rolling back to a previous version may be impossible if the deployment automation already terminated older pods. And the automation itself could be causing failures that are hard to detect. Without tracing showing the full deployment process, the root cause could stay hidden.<\/p>\n\n\n\n

\n

How Senser helps<\/h4>\n\n\n\n

The combination of these factors means debugging in Kubernetes requires new approaches. Traditional logging, tracing, and metrics tools designed for monoliths don’t provide the combined system-level and code-level observability needed. You end up trying to debug based on coarse alerts and surface-level symptoms.<\/p>\n\n\n\n

This is where eBPF and machine learning (ML) come in. eBPF lets you deeply inspect your environment without high overhead, giving you infrastructure and application level visibility.  ML helps you cut through the complexity, giving you insight into precisely why<\/em> things are failing and suggesting fixes.<\/p>\n\n\n\n

With Senser, we leverage these technologies to offer a purpose-built solution for cloud-native analysis. Our platform auto-discovers your microservices and correlated metrics, traces, and logs to provide context for troubleshooting. ML detects anomalies, patterns and interaction correlations pointing to the root cause. You get precise answers instead of alert floods.<\/p>\n\n\n\n

Kubernetes may have made our lives harder in some ways. But with the right solutions tailored to this new environment, teams can reap all the benefits of Kubernetes \u2013 without drowning in complexity when things go wrong.<\/p>\n<\/div><\/div>\n\n\n\n

Learn more<\/a> about how Senser helps SRE and DevOps teams go from production chaos to intelligence.<\/em><\/p>\n","protected":false},"excerpt":{"rendered":"

Kubernetes, alongside the distributed architecture of micro-service based applications, introduces complexity that makes debugging failures vastly more challenging. SRE and DevOps teams need new approaches to quickly identifying and resolving production issues.<\/p>\n","protected":false},"author":4,"featured_media":616,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_acf_changed":false,"footnotes":""},"categories":[1],"tags":[],"acf":[],"yoast_head":"\nKubernetes made my life much, much worse -<\/title>\n<meta name=\"description\" content=\"Kubernetes has made debugging failures vastly more challenging. Teams need new approaches to identify and resolve production issues.\" \/>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/senser.tech\/kubernetes-made-my-life-much-much-worse\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"Kubernetes made my life much, much worse -\" \/>\n<meta property=\"og:description\" content=\"Kubernetes has made debugging failures vastly more challenging. Teams need new approaches to identify and resolve production issues.\" \/>\n<meta property=\"og:url\" content=\"https:\/\/senser.tech\/kubernetes-made-my-life-much-much-worse\/\" \/>\n<meta property=\"article:published_time\" content=\"2023-08-17T17:24:28+00:00\" \/>\n<meta property=\"article:modified_time\" content=\"2023-09-28T06:38:29+00:00\" \/>\n<meta name=\"author\" content=\"Yuval Lev\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:image\" content=\"https:\/\/senser.tech\/wp-content\/uploads\/2023\/08\/Kubernetes-Made-My-Life-Much-Much-Worse.svg\" \/>\n<meta name=\"twitter:label1\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data1\" content=\"Yuval Lev\" \/>\n\t<meta name=\"twitter:label2\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data2\" content=\"5 minutes\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\/\/schema.org\",\"@graph\":[{\"@type\":\"WebPage\",\"@id\":\"https:\/\/senser.tech\/kubernetes-made-my-life-much-much-worse\/\",\"url\":\"https:\/\/senser.tech\/kubernetes-made-my-life-much-much-worse\/\",\"name\":\"Kubernetes made my life much, much worse -\",\"isPartOf\":{\"@id\":\"https:\/\/senser.tech\/#website\"},\"datePublished\":\"2023-08-17T17:24:28+00:00\",\"dateModified\":\"2023-09-28T06:38:29+00:00\",\"author\":{\"@id\":\"https:\/\/senser.tech\/#\/schema\/person\/437d82e941487b35320df36aaa3ae6ab\"},\"description\":\"Kubernetes has made debugging failures vastly more challenging. Teams need new approaches to identify and resolve production issues.\",\"breadcrumb\":{\"@id\":\"https:\/\/senser.tech\/kubernetes-made-my-life-much-much-worse\/#breadcrumb\"},\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\/\/senser.tech\/kubernetes-made-my-life-much-much-worse\/\"]}]},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\/\/senser.tech\/kubernetes-made-my-life-much-much-worse\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Home\",\"item\":\"https:\/\/senser.tech\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"Kubernetes made my life much, much worse\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\/\/senser.tech\/#website\",\"url\":\"https:\/\/senser.tech\/\",\"name\":\"\",\"description\":\"\",\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\/\/senser.tech\/?s={search_term_string}\"},\"query-input\":\"required name=search_term_string\"}],\"inLanguage\":\"en-US\"},{\"@type\":\"Person\",\"@id\":\"https:\/\/senser.tech\/#\/schema\/person\/437d82e941487b35320df36aaa3ae6ab\",\"name\":\"Yuval Lev\",\"image\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/senser.tech\/#\/schema\/person\/image\/\",\"url\":\"https:\/\/secure.gravatar.com\/avatar\/a985ba845187eff5799c272e9a0792f0?s=96&d=mm&r=g\",\"contentUrl\":\"https:\/\/secure.gravatar.com\/avatar\/a985ba845187eff5799c272e9a0792f0?s=96&d=mm&r=g\",\"caption\":\"Yuval Lev\"},\"url\":\"https:\/\/senser.tech\/author\/yuvalsenser-tech\/\"}]}<\/script>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"Kubernetes made my life much, much worse -","description":"Kubernetes has made debugging failures vastly more challenging. Teams need new approaches to identify and resolve production issues.","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/senser.tech\/kubernetes-made-my-life-much-much-worse\/","og_locale":"en_US","og_type":"article","og_title":"Kubernetes made my life much, much worse -","og_description":"Kubernetes has made debugging failures vastly more challenging. Teams need new approaches to identify and resolve production issues.","og_url":"https:\/\/senser.tech\/kubernetes-made-my-life-much-much-worse\/","article_published_time":"2023-08-17T17:24:28+00:00","article_modified_time":"2023-09-28T06:38:29+00:00","author":"Yuval Lev","twitter_card":"summary_large_image","twitter_image":"https:\/\/senser.tech\/wp-content\/uploads\/2023\/08\/Kubernetes-Made-My-Life-Much-Much-Worse.svg","twitter_misc":{"Written by":"Yuval Lev","Est. reading time":"5 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"WebPage","@id":"https:\/\/senser.tech\/kubernetes-made-my-life-much-much-worse\/","url":"https:\/\/senser.tech\/kubernetes-made-my-life-much-much-worse\/","name":"Kubernetes made my life much, much worse -","isPartOf":{"@id":"https:\/\/senser.tech\/#website"},"datePublished":"2023-08-17T17:24:28+00:00","dateModified":"2023-09-28T06:38:29+00:00","author":{"@id":"https:\/\/senser.tech\/#\/schema\/person\/437d82e941487b35320df36aaa3ae6ab"},"description":"Kubernetes has made debugging failures vastly more challenging. Teams need new approaches to identify and resolve production issues.","breadcrumb":{"@id":"https:\/\/senser.tech\/kubernetes-made-my-life-much-much-worse\/#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/senser.tech\/kubernetes-made-my-life-much-much-worse\/"]}]},{"@type":"BreadcrumbList","@id":"https:\/\/senser.tech\/kubernetes-made-my-life-much-much-worse\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/senser.tech\/"},{"@type":"ListItem","position":2,"name":"Kubernetes made my life much, much worse"}]},{"@type":"WebSite","@id":"https:\/\/senser.tech\/#website","url":"https:\/\/senser.tech\/","name":"","description":"","potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/senser.tech\/?s={search_term_string}"},"query-input":"required name=search_term_string"}],"inLanguage":"en-US"},{"@type":"Person","@id":"https:\/\/senser.tech\/#\/schema\/person\/437d82e941487b35320df36aaa3ae6ab","name":"Yuval Lev","image":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/senser.tech\/#\/schema\/person\/image\/","url":"https:\/\/secure.gravatar.com\/avatar\/a985ba845187eff5799c272e9a0792f0?s=96&d=mm&r=g","contentUrl":"https:\/\/secure.gravatar.com\/avatar\/a985ba845187eff5799c272e9a0792f0?s=96&d=mm&r=g","caption":"Yuval Lev"},"url":"https:\/\/senser.tech\/author\/yuvalsenser-tech\/"}]}},"_links":{"self":[{"href":"https:\/\/senser.tech\/wp-json\/wp\/v2\/posts\/308"}],"collection":[{"href":"https:\/\/senser.tech\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/senser.tech\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/senser.tech\/wp-json\/wp\/v2\/users\/4"}],"replies":[{"embeddable":true,"href":"https:\/\/senser.tech\/wp-json\/wp\/v2\/comments?post=308"}],"version-history":[{"count":24,"href":"https:\/\/senser.tech\/wp-json\/wp\/v2\/posts\/308\/revisions"}],"predecessor-version":[{"id":780,"href":"https:\/\/senser.tech\/wp-json\/wp\/v2\/posts\/308\/revisions\/780"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/senser.tech\/wp-json\/wp\/v2\/media\/616"}],"wp:attachment":[{"href":"https:\/\/senser.tech\/wp-json\/wp\/v2\/media?parent=308"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/senser.tech\/wp-json\/wp\/v2\/categories?post=308"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/senser.tech\/wp-json\/wp\/v2\/tags?post=308"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}