When we first wired Finopscraft's anomaly detection to Slack, we made the same mistake every monitoring team makes: we sent too much. Every 15% week-over-week service spike. Every time a new EC2 instance type appeared in the bill. Every Lambda invocation count uptick that crossed a statistical threshold.
Within two weeks, the engineering channel had muted the bot. Nobody noticed. Another week passed before someone realized a $4,200 RDS data transfer charge had been quietly accumulating for ten days with zero response from the team.
The problem isn't the alert system. It's that cost anomaly alerting has a fundamentally different signal-to-noise profile than infrastructure alerting. A CPU spike at 95% means something is broken right now. A 30% spend increase on Cognito might be completely expected if you just shipped a new user registration flow. Context is everything, and most alert configurations ignore it.
Why Cost Alerts Fail Differently Than Infrastructure Alerts
Infrastructure alerting has decades of tooling wisdom behind it. Pagerduty, OpsGenie, and similar tools have hardened concepts around on-call rotations, escalation policies, and severity tiers. Engineers understand that a CPU alert at 3 AM demands action.
Cost alerts arrive in a completely different mental model. They're not urgent in the infrastructure sense: nothing is down. They're not actionable in the next five minutes. And the causal chain from "spend went up" to "what the engineer should do" is often opaque without additional context.
The result is that cost alerts get treated as FYI messages, which means they get muted or ignored unless they're designed specifically to force a decision. We've found that the right framing is not "here is a data point" but "here is a question you need to answer by end of day."
The Threshold Design We Use
We built Finopscraft's Slack integration around three alert tiers, each with different delivery urgency and message format.
Tier 1: Anomaly Alerts (immediate, tagged to a team)
These fire when a service or tag group exceeds 2.5 standard deviations from its rolling 28-day baseline, AND the absolute dollar delta is above $50/day. Both conditions must be true. The dollar floor eliminates noise from small services where statistical anomaly is meaningless in practice.
The message format matters as much as the threshold. We don't send: "EC2 spend up 34% this week." We send: "EC2 spend for team:backend-api is $180 above its 28-day average. Last change in that tag group: 3 new r6g.xlarge instances launched Tuesday. Is this expected?"
That final question is intentional. We want a yes or no from the tagged team lead, not a spiral of investigation. If they say yes, the alert auto-snoozes for 14 days and we recalibrate the baseline. If they say no, we create a Jira ticket for investigation. The Slack thread becomes the record.
Tier 2: Budget Threshold Alerts (daily digest, not paged)
Monthly budget thresholds trigger a daily digest at 9 AM local time, not an immediate ping. These go to a dedicated #cloud-costs channel, not the general engineering channel. The format is a short table: service, MTD spend, % of monthly budget, projected month-end.
We found that sending these to the general engineering channel was the primary driver of muting. Engineers don't want cost data intermixed with deployment notifications. A dedicated channel with a predictable daily cadence is something people actually check.
Tier 3: Waste Flags (weekly, bundled)
Idle resource detections, overprovisioning flags, and savings plan coverage gaps get bundled into a single weekly message sent Monday morning. This is not urgent. Nobody needs to know at 11 PM that their dev EC2 instance has been idle for 6 days. A Monday morning message with a clear action list ("here are 4 resources you could stop or right-size this week, estimated savings $240/month") respects the engineer's time and is genuinely useful.
What We Do Not Alert On
This is the list that reduced our noise by roughly 70% once we were honest about it.
- Savings plan coverage drops below 80%. This is a real issue, but it's a weekly planning decision, not an incident. It goes in the Monday digest, not a real-time alert.
- New service first appears in bill. Engineers spin up services all the time in dev. Unless the spend is already above $20/day on day one, this is informational noise.
- Data transfer charges spike on release day. If your deployment pipeline just pushed a new version and data transfer costs spiked 40%, that's probably normal. Our system checks whether a deployment tag update happened within 48 hours of the spike before deciding to alert.
- Reserved instance expiry warnings. These belong in a monthly FinOps review, not Slack. They have long lead times and require a procurement conversation. Sending them to Slack creates urgency theater.
A Concrete Example: The Batch Job Problem
Consider a growing data engineering team running nightly Glue jobs. Every Monday morning, weekend batch jobs inflate the AWS Glue line item significantly compared to the prior week's weekday pattern. A naive anomaly detector fires an alert every Monday.
We handle this with day-of-week segmentation in the baseline. We compare Monday spend to the prior four Mondays, not the trailing 28-day average. This cut false positive anomaly alerts for batch-heavy teams by around 60% in our testing. The underlying pattern is predictable; the alert should reflect that.
A similar issue appears with month-end: billing systems that run large reconciliation jobs on the last day of the month will show spend spikes that are calendar artifacts, not waste. Knowing the difference requires the platform to understand the shape of your spend, not just its level.
On Routing: Who Gets What
Routing matters as much as thresholds. We use tag-based routing by default: if a resource has a team: tag, anomaly alerts go to that team's channel. If it has a project: tag, waste flags go to the project channel. Untagged resources are the hardest case.
Untagged resources account for a disproportionate share of waste in teams that have been running for more than a year. We send untagged resource alerts to a designated FinOps owner (usually whoever is on the finance-engineering interface) rather than broadcasting them broadly. Broadcasting untagged alerts creates the "who is responsible for this?" problem that kills cost culture in growing teams.
We're not saying tag enforcement is the only answer to routing. Some teams genuinely cannot tag every resource, especially with third-party services or shared infrastructure. In those cases, allocation by spend proportion is an acceptable fallback, and we use it. But first-class tag-based routing is the goal.
Actionability as the Design Constraint
The rule we came back to every time we debated a new alert type: does the person receiving this message have something concrete to do in the next 48 hours? If the answer is "maybe, eventually, it depends," that's not an alert. That's a report. Put it in the weekly digest or the monthly dashboard.
Cost alerting done well looks less like a monitoring system and more like a good collaborator: it shows up at the right time, with the right context, and asks a specific question. Calibrating that requires ongoing tuning based on how your team actually responds, which is why Finopscraft tracks acknowledgment rate and snooze patterns as first-class signals for threshold adjustment.
The goal isn't zero false positives. The goal is alerts that your engineers don't mute.