Last Notes
the most expensive AWS mistake isn't paying on-demand. it's locking in a multi-year commitment on a fleet you're about to halve.
the gate is simple. fleet rightsized? if no, resize first — don't commit. stable baseline? if yes, buy the compute savings plan, up to 66% off, sized to that baseline.
a smaller baseline buys a cheaper, more accurate commitment. that's why the order matters.
https://rajesh.medampudi.com/blog/audit-50k-aws-bill
https://media.medampudi.com/6a1d140d1d3f107259f3423330bc6828d61f4d2b03c40404b8ddafb0e808ce77.png
a metric that lives in a monthly finance review is not a metric anyone who moves it ever sees.
the cost of a code path is a property of that code path, same as its latency. so put cost per request on the same grafana board as latency. same screen, same on-call engineer.
an engineer who sees a new endpoint costs 4x per call fixes it in the PR, context hot, for the price of a code review. the same regression caught six weeks later in a finance reconciliation is a forensic investigation. same bug. timing decides the fix cost.
https://rajesh.medampudi.com/blog/infra-bill-as-product-metric
https://media.medampudi.com/cecf5dccb2ead8cf0001dc5a3a0fc5190dda76408d4ef6c96c7279e34ecdc281.png
worth noticing: aws now recommends savings plans over reserved instances for compute. the vendor selling you the RI is telling you to buy the other thing.
the logic is clean. the convertible RI exists for flexibility — the compute SP does that better, automatically, no manual exchanges. the standard RI exists for the deepest rate — the ec2 instance SP matches the 72% and stays flexible inside the family.
both reasons the RI existed for ec2 are now done better by a savings plan. that's why the recommendation flipped.
https://rajesh.medampudi.com/blog/ri-vs-savings-plans
https://media.medampudi.com/106caa05673e1f22154f74fea930693759dec7e634609445090de724d892e09b.png
cutting a nat bill, the order matters more than any single fix, because the cheap no-downside moves capture most of the money.
s3 + dynamodb gateway endpoints — free, one route-table edit. start here.
interface endpoints for ecr and logs — $0.01/GB, under a quarter of nat's rate. cheap, but do the per-service math.
cross-AZ cleanup, and the honest question of whether the subnet needs nat at all.
fck-nat — real savings, but no default failover and a patching burden. last resort.
most teams never get past step two.
#aws #infrastructure
https://rajesh.medampudi.com/blog/aws-nat-gateway-hidden-tax
https://media.medampudi.com/fe1fcb6aff2cf82aceb738151e99359921a3a0f88387d442d119de9c70b40d89.png
the order isn't decoration. each decision makes the next possible.
cut the waste first — free, mostly attention not architecture. then own past your break-even (datadog past my line → built it ~8x cheaper on LGTM). then observe cheaply, because you can't decide the first two without numbers. then staff light. i run all four — k8s, self-hosted observability, the services on top — alone, from a modest house in hyderabad. the load is far lower than the staffing story implies.
https://rajesh.medampudi.com/blog/what-lean-infrastructure-means
https://media.medampudi.com/c95edc690ffc1a48ebfca479d963396f3a5dd1392e01cc3dec514e40e751b38e.png
there's storage on your s3 bill you can't see in the console and you're paying for it right now.
upload a big object in parts, the upload fails partway — dropped connection, crashed job, sdk that didn't clean up — and the parts that landed stay in the bucket. you get billed for them. they don't show in the object listing. they pile up for years.
the fix is one lifecycle rule on every bucket: AbortIncompleteMultipartUpload, 7 days. applies to existing + future uploads, and per aws it doesn't trigger early-delete charges. no downside. set it today before you touch anything else — cheapest win in the whole thing.
https://rajesh.medampudi.com/blog/s3-cost-optimization-playbook
https://media.medampudi.com/fb3ce950d9a5b69530f225814c7fdd772ba75a713109dce3be771c06d9245722.png
before you buy a single savings plan, sweep the free money. none of it needs a commitment and all of it is just sitting there.
unattached EBS volumes still billing. old snapshots nobody deletes. idle public IPv4 — $0.005/hr each since feb 2024, attached or not. and untagged spend, because you can't cut what you can't see.
clean the dead weight first. then size the commitment to a real baseline.
https://rajesh.medampudi.com/blog/audit-50k-aws-bill
https://media.medampudi.com/52e881edf6c6496cf1ea5d208c555354b5c3ad0fefbee4bc74b07b3406ed5b3a.png
total dollars on a bill lie, and here's exactly how.
the signal arrives weeks after the decision that caused it. it lands in front of finance, who can read it but can't fix it. the engineer who can fix it never sees it. and total $ can't tell a bill that doubled on growth from one that doubled on waste.
four failures, all structural. divide into a per-unit number and all four fix at once — timely, actionable, owned by the right person, and it tells growth from waste at a glance.
https://rajesh.medampudi.com/blog/infra-bill-as-product-metric
https://media.medampudi.com/8413e1805c253a74e08b91eea5a034217055f472bb16d6ec664326741585818e.png
on 2 december 2025 aws finally launched database savings plans — the gap that kept most database spend stuck on reserved instances for years.
covers aurora, rds, dynamodb, elasticache, documentdb, neptune, keyspaces, timestream and dms. up to 35% on serverless, 20% on provisioned, and it follows the workload across engine, family, size and region.
two catches worth saying out loud: it's 1-year no-upfront only at launch, and it does NOT cover redshift or opensearch — i've already seen that stated wrong. check the official service list.
https://rajesh.medampudi.com/blog/ri-vs-savings-plans
https://media.medampudi.com/6290713b4b767b36083f722f9605d173a09a14665c0e548b14b3490616cb308a.png
four signs your nat gateway is doing a job a free endpoint should be doing: s3 reads from private subnets, ecr image pulls on every deploy, a nat sitting in a different AZ than your workloads, and one nat carrying multiple terabytes a month.
you don't have to guess. turn on vpc flow logs for the nat'd subnets and look at where the bytes go. a big share heading to s3, ecr, or dynamodb is the smell — and the cheapest fix you'll find this quarter.
#aws #devops
https://rajesh.medampudi.com/blog/aws-nat-gateway-hidden-tax
https://media.medampudi.com/578b5a8b735f7b0f79f976f722a28935fe0d566b4548bc929e2cc1ef20e2f53f.png
people hear "lean" and think "cheap." it's the opposite.
cheap is cutting the thing that matters to save a small number — you win a line item and lose the business. lean is spending deliberately on what actually buys you something, so you have room to do the work that moves the business. the point of cutting a $120K/mo observability bill was never the $120K. it was no longer being held hostage by it.
https://rajesh.medampudi.com/blog/what-lean-infrastructure-means
https://media.medampudi.com/48b5d0bd39ae2fd2acd61be99460b98da1c31178256118e58b2a6dbebb8ccb0b.png
how a $120K/mo observability bill quietly happens: three systems billing at once. datadog the official tool. cloudwatch still alive underneath that nobody turned off. and engineers SSHing into boxes to grep logs during incidents because paid search was too slow to trust.
none of them complete, all of them on the meter, spend climbing 15%/mo. the fix isn't clever — one stack you own: grafana LGTM backed by s3, on your own kubernetes, data stays yours. costs become compute + storage, which scale slow. no vendor between you and the 3am fix. one bill you can actually read.
#infrastructure #cloudcost #selfhosting
https://rajesh.medampudi.com/blog/observability-cost-kill
https://media.medampudi.com/4c5f790130f016a44a40e7b9b777c8e4020009ec21c33c66c606757d9f06d03a.png
the trap that quietly reverses your s3 savings: standard-IA, one zone-IA and glacier instant all bill every object as if it were at least 128KB.
so you take a bucket of 10KB thumbnails, move it to a "cheaper" class to save money, and now each object bills at 128KB. you're paying for ~12x the bytes you actually store. the per-GB label looked cheaper, the bill came back higher.
small objects stay in standard. check your average object size before tiering anything down — under 128KB and IA is just wrong, no matter how cold the data is.
https://rajesh.medampudi.com/blog/s3-cost-optimization-playbook
https://media.medampudi.com/3af0ba4769d6f20212edfc5d130ee400bcfeb98db854620cccae3ecc582644ba.png
give me read access to a $50k/mo AWS bill and i'll find the first 20-30% in a day. not clever — it's always the same four places, same order.
data transfer → rightsize → commit → storage cleanup.
the order is the whole thing. each step makes the next one cheaper. don't commit before you rightsize, don't rightsize a fleet still leaking free traffic through a paid NAT. run it bottom to top.
https://rajesh.medampudi.com/blog/audit-50k-aws-bill
https://media.medampudi.com/fbfa54a6a72776daf71bc14f3d809ab5dc315f246baf3b043324ba44be99cc48.png
the whole unlock is dividing.
total infra cost over requests = cost per request. over tenants = cost per tenant. the finops foundation calls it unit economics. the arithmetic is trivial. the shift it forces is not.
two buckets worth keeping separate: resource-efficiency (cost per GB, per vCPU, per token) tells you how the machinery is wasting. business metrics (cost per tenant, cost to serve) tell you whether it matters. once cost is per-unit, a rising bill stops being scary by default.
https://rajesh.medampudi.com/blog/infra-bill-as-product-metric
https://media.medampudi.com/f0a2a892befca3dbce052749842d659326fc732e5331af05ce9cd44110d722a0.png
savings plans are the default now, but not the answer to everything. there are exactly three corners where the reserved model is still the only lever you have.
redshift uses reserved nodes, opensearch uses reserved instances — no savings plan covers either, even after the dec 2025 launch. a zonal RI reserves capacity in a specific AZ; a savings plan reserves none. and a standard RI can be sold on the marketplace; a savings plan can't be cancelled mid-term.
everything outside those three → savings plan.
https://rajesh.medampudi.com/blog/ri-vs-savings-plans
https://media.medampudi.com/50cd1bf2b44060836666e028d1b4be3b5d505a137c898f3a384c8616042d211f.png
the most common reason a nat gateway bill creeps: s3 reads from private subnets going out through nat at $0.045/GB.
the fix is free and almost nobody does it first. gateway endpoints for s3 and dynamodb cost nothing — no hourly fee, no per-GB fee. you add a route, and that traffic goes over a private aws path instead of through the meter. it never touches nat again.
do this before anything fancier. most teams never need to go further.
#aws #cloudcost
https://rajesh.medampudi.com/blog/aws-nat-gateway-hidden-tax
https://media.medampudi.com/fb94d6dc380dd2dc3a6d29d8189aa815d8ac91e3fa6cfc351706e18bc03dd332.png
a fifth of enterprise cloud spend — ~$44.5B in 2025 — goes to resources nobody is using (harness, finops in focus 2025).
that's not carelessness. it's the default state of a bill nobody is actively cutting. idle instances, commitments bought on a guess, storage that should've aged into a cheaper tier months ago. assume you're overspending — the data says you are — and go look once a month. you'll find the leak.
https://rajesh.medampudi.com/blog/what-lean-infrastructure-means
https://media.medampudi.com/bf3b3b0a7b4ea55698a22ea374ac3976ac7a61d18f189366ae7655ffdff5f344.png
the stack that replaced the saas bill, and it's boring on purpose: loki for logs, mimir for metrics, tempo for traces, grafana for dashboards + LogQL. running on our own kubernetes, 50+ nodes, 200 services, all backed by s3 underneath.
two surprises: log search was faster than the paid tool, and with no per-seat cost 15 teams onboarded in month one instead of rationing licences. we ran it in parallel with the old tooling for a full month before decommissioning anything — you don't pull monitoring during a tournament with 100k players online and hope.
#observability #grafana #kubernetes
https://rajesh.medampudi.com/blog/observability-cost-kill
https://media.medampudi.com/c2d867877ab5ecc80c0dffc68656b2b8ad0b5fc87438d2de5d39c860c3412b46.png
the most wasteful line on a mid-size AWS bill is the one with no resource page: data transfer.
the NAT gateway is the worst of it. $0.045/GB just to process traffic — and most teams route S3 + package pulls straight through it. a VPC gateway endpoint carries that same traffic for $0. one route-table edit.
i check this on every account. it's almost always there.
https://rajesh.medampudi.com/blog/audit-50k-aws-bill
https://media.medampudi.com/71089fbc78a49523e903b0778baa8c8e5debfa9a53ce0c602429b2c996b402eb.png
s3 storage classes span a ~23x price range and the only thing separating them is access pattern. same bytes — standard is $0.023/GB-month, deep archive is about $0.00099.
most bills sit entirely in standard because that's the default and almost nobody changes it. it's the most expensive, most available class aws sells, tuned that way on purpose.
frequently read → standard. occasional + over 128KB → standard-IA. rarely read but instant → glacier instant. archive you can wait on → flexible. compliance you'll never read → deep archive. that's the whole call.
https://rajesh.medampudi.com/blog/s3-cost-optimization-playbook
https://media.medampudi.com/a885260c7f019899e98ec280b73abe27f96c9e2cecf748f7d2fb3eb9bc8d5407.png
for ec2 in 2026 the reserved instance is the legacy choice, and most people are still running the old rule in their head.
aws's own comparison: compute SP up to 66% across any family + region + fargate + lambda. ec2 instance SP up to 72% in one family. convertible RI 66% with a manual exchange. standard RI 72% but locked.
at every tier the savings plan matches the rate and beats the flexibility. same rate, less to manage. that's the whole argument.
https://rajesh.medampudi.com/blog/ri-vs-savings-plans
https://media.medampudi.com/2e9d715f05faa8867e3886d26d5502636e537a1f8614f8016fdcc360145b6966.png
the cloud bill is the only production number most teams don't own until it's already a fire.
engineering owns latency. finance owns the invoice. nobody owns the gap — and a fifth to a third of cloud spend dies in it.
stop treating the bill as an accounting artifact. treat it as a product metric: cost per request, per tenant, per feature, on the same dashboard as latency. we already run every other signal this way. cost is the one we still run like a 1990s expense report.
https://rajesh.medampudi.com/blog/infra-bill-as-product-metric
https://media.medampudi.com/17ecf773fcde095c0b149f9ba94100ba012e773c16e8be4e5e6511d4677c2ebd.png
lean infrastructure is four decisions, read top to bottom: cut the waste, own past your break-even, observe it cheaply, don't over-staff.
none of them is a technology. it's a posture — pay for what genuinely buys you something, refuse to pay for what doesn't. cut what's wasted, own what's worth owning, see everything, staff for the real load. that's the whole thing.
https://rajesh.medampudi.com/blog/what-lean-infrastructure-means
https://media.medampudi.com/1a2e07b8957c245278e5a1a757c3c4e14394700ce7f610c1f2f5d2922075c781.png
aws nat gateway charges you twice for the same packet. once at $0.045/hr just to exist (~$33/mo per gateway), and again $0.045 for every gigabyte it moves — in and out, on top of normal data transfer.
the hourly fee is visible so nobody worries about it. the per-GB fee is the one that quietly grows, because it rides on traffic you never watch — container pulls, package installs, s3 reads from private subnets.
#aws #infrastructure
https://rajesh.medampudi.com/blog/aws-nat-gateway-hidden-tax
https://media.medampudi.com/970308496c16bfa0123903f8b0215912ad2b940ef3fd47ff423aafa745b7cde1.png
at 6 TB of telemetry a day, self-hosted grafana LGTM ran our whole observability platform for ~$15-18K/mo. datadog at the same volume would've been ~$120K. that's 85% less, same telemetry, same scale.
and it gets worse over time — per-GB pricing and compute+storage scale on different curves, so the gap widens as you grow. datadog isn't a bad tool. you're just renting a meter that bills you more every time your product succeeds. model your bill at 5x volume before you sign, not after.
#observability #selfhosting #grafana
https://rajesh.medampudi.com/blog/observability-cost-kill
https://media.medampudi.com/13aa0856e976f5c57854bb63fab03f047aeda6306d98a569af0abe5f09429c53.png
aws nat gateway charges you twice for the same packet. once at $0.045/hr just to exist (~$33/mo per gateway), and again $0.045 for every gigabyte it moves.
the hourly fee is visible so nobody worries about it. the per-GB fee is the one that quietly grows, because it rides on traffic you never watch — docker pulls, apt installs, s3 reads from private subnets.
open cost explorer, group by usage type, look at NatGateway-Bytes. that's the number to chase.
#aws #infrastructure #cloudcost #devops
at 6 TB of telemetry a day, self-hosted grafana LGTM ran our whole observability platform for ~$15-18K/mo. datadog at the same volume would've been ~$120K. an 85% cut.
datadog isn't a bad tool. the problem is you're renting something you could own, on a meter that bills you more every time your product succeeds. per-GB pricing is a tax on growth. model your bill at 5x volume before you sign, not after.
#observability #selfhosting #infrastructure
5 simbotix brands. 1 company. 1 back office. 1 bill.
the brands let each conversation start where the customer actually is.
🔗 https://simbotix.com/blog/5-simbotix-brands-which-one
Final E2E test through CF→Caddy→ksvc at 2026-05-10 10:32:58 UTC. If this lands on nos.lol, the entire ClickUp→Nostr pipeline works through public infra. #pipeline #e2e
