Keep manual monitoring instead of adopting Azure SRE Agent

ID
ADR 9.3
Status
proposed
Date
2026-09-11
Deciders
KBP architect, KBP core team (to be confirmed in review)
Tags
operations monitoring cost tooling ai

Context and problem statement

Each sprint a member of the KBP team takes the DevOps Master role. The role monitors the Application Insights failures view several times a day, reacts to Azure Monitor alerts in the Team Intern Teams channel, and opens a bug when something needs fixing. It is recurring attention work that does not scale with the team and that nobody can hand over mid-sprint.

Azure SRE Agent became generally available and offers exactly that shape of automation: it polls fired alerts, investigates them against connected telemetry and source code, runs scheduled health checks, and reports into Teams or Azure DevOps. Should KBP adopt it to reduce this toil?

The question is a cost decision as much as a technical one, because the agent carries a fixed monthly charge that is independent of how much work it does.

Decision drivers

  • The recurring DevOps Master attention cost is real and is the thing worth removing.
  • Any tool with an ongoing licence or consumption cost has to be justified against the volume of work it actually handles, not against the volume it could handle.
  • Alert routing already works. The kbp-teams-email action group delivers every alert to the Team Intern channel, and changing it is not part of the problem.
  • fn-autobugcreate-kbp already scans Application Insights and opens Azure DevOps bugs. A second tool doing the same must either replace it or stay clearly separate.
  • Production access is constrained: the team holds Contributor on rg-kbp-prod but not User Access Administrator, so scoping any agent identity to production needs a platform-team grant.
  • Data protection: inference must stay inside the EU Data Boundary unless signed off otherwise.

Considered options

  1. Keep the manual monitoring rounds (status quo).
  2. A KBP-scoped Azure SRE Agent covering rg-kbp-test and rg-kbp-prod.
  3. A shared Azure SRE Agent covering the whole Vertrieb.Portale subscription, cost split across the teams in it.
  4. Extend what already exists: more alert rules and dashboards, and a wider scope for fn-autobugcreate-kbp.

Decision outcome

Proposed: option 1, keep the manual monitoring rounds for now, and do not create an SRE Agent for KBP.

The measurements behind this are recorded in the SRE Agent evaluation. In short: KBP’s own resource groups produced five fired alerts in thirty days, none above Sev2, while the agent costs about €276 a month in fixed charges alone. That is roughly €55 per alert investigated. The alert-triage case, which is what the agent is usually adopted for, does not hold at this volume.

Option 2 is therefore rejected on cost, not on capability. Option 4 is not pursued because more alert rules would add noise to the same channel without reducing the attention the rounds require.

Option 3 is the genuinely better economic shape and is explicitly not rejected here — the same fixed cost would cover all 637 alerts a month in the subscription instead of five. It is out of scope for this record because it is not KBP’s decision to take alone.

This record does not close the question of automating the monitoring rounds themselves. That is the part of the DevOps Master’s work with a real case behind it, and it is a separate decision that should be argued on its own terms rather than bundled into an alert-triage proposal.

Consequences

  • Good, because no ongoing cost is committed for a workload that does not justify it.
  • Good, because the reasoning and the measurements are recorded, so the question does not have to be re-researched when the agent is proposed again.
  • Good, because nothing changes operationally: alert routing, the action groups, and fn-autobugcreate-kbp all stay as they are.
  • Bad, because the DevOps Master keeps the recurring attention cost, which is the problem that prompted the evaluation and which stays unsolved.
  • Bad, because the 30-day evaluation offer that waives the fixed charge has already been used on this account, so a later pilot starts paying from the first hour.
  • Neutral, because the decision is volume-dependent rather than permanent. If the alert volume rises or the cost model changes, the arithmetic is in the evaluation note and can be redone quickly.

Confirmation

Revisit this record when any of the following becomes true:

  • the teams in Vertrieb.Portale take up a shared agent (option 3);
  • KBP’s fired-alert volume rises materially above five a month, for example after adding per-endpoint error-rate rules;
  • a renewed evaluation offer allows a pilot at consumption cost only.

Pros and cons of the options

Option 1: keep the manual monitoring rounds

  • Good, because it commits no budget and needs no new permissions.
  • Good, because the process is understood and already documented.
  • Bad, because the attention cost stays with a person every sprint.
  • Bad, because coverage depends on who holds the role and how busy they are.

Option 2: KBP-scoped SRE Agent

  • Good, because it would investigate alerts automatically and could run scheduled health checks that replace the manual rounds.
  • Good, because it needs no change to alert routing; it polls Azure Monitor directly.
  • Bad, because the fixed cost dominates at KBP’s alert volume, at roughly €55 per alert.
  • Bad, because it overlaps fn-autobugcreate-kbp without a decision on which one stays.
  • Bad, because scoping it to production needs a User Access Administrator grant the team lacks.

Option 3: shared subscription-wide SRE Agent

  • Good, because the same fixed cost amortises across every team in the subscription.
  • Good, because a response plan filtered to Sev0 and Sev1 keeps the consumption cost proportionate.
  • Bad, because it needs agreement and a cost split across teams KBP does not speak for.
  • Bad, because ownership, run-mode policy, and on-call expectations would have to be agreed centrally.

Option 4: extend alert rules, dashboards and the existing bug-creation function

  • Good, because it builds on components the team already owns and operates.
  • Bad, because more rules produce more messages in the same channel, which increases the triage load rather than reducing it.
  • Bad, because it does not address the proactive rounds at all, only the reactive path.

More information