Skip to content

feat(cloudformation): Add DevOps Agent alarm investigations template - #38

Open
LorenzoBoccaccia wants to merge 1 commit into
aws:mainfrom
LorenzoBoccaccia:add-alarm-investigations-cfn
Open

LorenzoBoccaccia wants to merge 1 commit into
aws:mainfrom
LorenzoBoccaccia:add-alarm-investigations-cfn

Conversation

@LorenzoBoccaccia

@LorenzoBoccaccia LorenzoBoccaccia commented Aug 3, 2026 •

Copy link
Copy Markdown

Description

Adds cloudformation/devops-agent-alarm-investigations/, a template that forwards
CloudWatch alarms to an AWS DevOps Agent generic webhook so each alarm opens an
investigation. Also moves the skill-policies template into its own directory with a
README, and updates the links to it.

The template has no code. One stack serves a list of alarm ARNs:

  • An EventBridge rule matches CloudWatch Alarm State Change to ALARM for those ARNs.
  • An input transformer builds the incident payload, titled with the alarm name.
  • An API destination posts it through a connection that sends the webhook API key as
    Authorization: Bearer <key>. The key is stored once per stack.
  • An IAM role grants the rule events:InvokeApiDestination on that destination only.

incidentId is the EventBridge event id, which stays the same across retries, so
redeliveries are deduplicated. The template uses an API key rather than HMAC because
EventBridge connections cannot sign each request; HMAC would need a signing Lambda.

Type of change

  • New skill
  • New custom agent
  • Update to an existing skill or agent
  • Documentation or infrastructure change

Testing

Not a skill, so the skill evaluation tool does not apply.

  • validate-template passes: 5 parameters, CAPABILITY_IAM.
  • Deployed and forced an alarm to ALARM. AWS/Events showed TriggeredRules 1,
    SuccessfulInvocationAttempts 1, FailedInvocations 0. The Agent Space received an
    INVESTIGATION task carrying the alarm ARN, with the EventBridge event id as its
    reference.referenceId.
  • Payload escaping: the same input transformer, pointed at an SQS queue, received an
    alarm named quote"back\slash'apos. The message was valid JSON and the title and ARN
    came through unchanged.
  • DevOpsAgentAlarmIntegrationTag: with a value it tags the rule and role; left empty,
    the tag value is the stack name.
  • All test resources were tagged and deleted.

Known behaviour, documented in the README: an alarm that already appears in an earlier
incident is attached to it as LINKED, even if that incident is closed, instead of
opening a new investigation. This comes from DevOps Agent correlation, not the template.

@LorenzoBoccaccia
LorenzoBoccaccia force-pushed the add-alarm-investigations-cfn branch from d636396 to 6189121 Compare August 18, 2026 09:08
@LorenzoBoccaccia
LorenzoBoccaccia force-pushed the add-alarm-investigations-cfn branch 2 times, most recently from 90b32a0 to 9239bc0 Compare August 19, 2026 07:51
@LorenzoBoccaccia
LorenzoBoccaccia force-pushed the add-alarm-investigations-cfn branch 3 times, most recently from a301d6c to c43418b Compare September 10, 2026 14:12
@LorenzoBoccaccia
LorenzoBoccaccia force-pushed the add-alarm-investigations-cfn branch from c43418b to 2a01a9c Compare September 30, 2026 12:45
@LorenzoBoccaccia
LorenzoBoccaccia force-pushed the add-alarm-investigations-cfn branch from 2a01a9c to edfa7cd Compare October 8, 2026 13:46
Forward a list of CloudWatch alarms to a DevOps Agent generic webhook so
each alarm opens an investigation. An EventBridge rule scoped to the
alarm ARNs matches ALARM state changes, an input transformer builds the
incident payload titled with the alarm name, and an API destination
POSTs it through an API key connection, so the key is stored once per
stack. No Lambda function is involved.

Move devops-agent-skill-policies.yaml into its own directory and give it
a README so each template is self-documenting, update the links that
referenced the old path, and describe both templates in llms.txt.
@LorenzoBoccaccia
LorenzoBoccaccia force-pushed the add-alarm-investigations-cfn branch from edfa7cd to 28a5728 Compare October 21, 2026 12:38
@LorenzoBoccaccia
LorenzoBoccaccia requested a review from a team as a code owner October 21, 2026 12:38

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants