PulseGuard

Lightweight health and telemetry pulse aggregation.

Concepts Overview

PulseGuard collects two kinds of data:

1. Pulse Checks (synthetic/endpoint health) – single HTTP GET requests executed per configured target.
2. Agent Checks (resource / performance metrics) – queries against external telemetry stores for one or more applications.

Each configuration instance is stored (table storage attributes visible in the code) and executed by hosted background services.

---

Pulse Check Types

PulseConfiguration fields of interest:

• Group / Name: identity / partition.
• Location: target URL (HTTP GET).
• Type: one of the below enum values.
• Timeout / DegrationTimeout: execution & optional degraded-state threshold (ms).
• IgnoreSslErrors: opt-in to bypass certificate validation (flag placed on request options).
• ComparisonValue: auxiliary value used by some check types (JSON / Contains).
• Headers: semicolon separated custom headers (Key:Value;Key:Value).

Result model: PulseReport(State, Message, Error) where State ∈ Healthy | Unhealthy | TimedOut.

1. HealthApi

Expected response body: JSON matching HealthApiResponse:

{
  "state": "Healthy | Degraded | Unhealthy | TimedOut | ...",
  "message": "optional",
  "dependencies": [ { "name": "...", "state": "..." }, ... ]
}

Processing:

• Body deserialized; extra properties stripped by re-serialization.
• If state == Healthy the raw payload is discarded (saves storage) and report marked Healthy.
• Any deserialization issue -> Fail with original (capped) body.

2. StatusCode

Success condition: HTTP status 2xx. Failure: non-success status. Body (if readable) captured into Error for diagnostics.

3. Json

Use when you want the target endpoint JSON to contain (as a structural subset) a specific JSON fragment.

• ComparisonValue must be valid JSON (subset template).
• Entire response body must be JSON.
• Implementation builds JObjects, computes set intersection and compares. Fail reasons: empty body, invalid JSON, or subset mismatch.

4. Contains

Plain substring presence check on the response body (case-sensitive string.Contains).

• ComparisonValue is the required substring. Fail reasons: empty body or substring not found.

5. HealthCheck

Simplified textual health probe.

• Expects the body to be exactly a PulseStates token (e.g. "Healthy", "Unhealthy", etc.).
• If value parses and is Healthy -> success (body dropped). Otherwise failure with derived message. Fail reasons: null body or unknown token.

6. StatusApi

Expected response body: JSON matching StatusApiResponse:

{
  "status": "Healthy | ...",
  "details": { /* optional */ },
  "entries": { /* optional */ }
}

Processing identical pattern to HealthApi: deserialize, strip extras, evaluate root status. Fail reasons: null body, deserialization error.

Failure & Timeout Semantics

• Core execution: a single GET request using shared HttpClient.
• Timeout & degraded logic (if implemented in hosting layer) will convert to TimedOut via PulseReport.TimedOut.
• Healthy responses intentionally omit large payload storage for efficiency.

---

Agent Check Types

PulseAgentConfiguration fields:

• Sqid: grouping key.
• Type: one of the AgentCheckType values (string persisted, enum used at runtime).
• Location: endpoint / workspace specific URI or identifier (see per type).
• ApplicationName: logical application (used for correlation & query filtering).
• Headers: optional custom headers (POST queries / data-plane auth if needed).

Result model: PulseAgentReport(Options, CpuPercentage, Memory, InputOutput) – null metric values indicate unavailable / failed extraction.

1. ApplicationInsights

Purpose: Pull recent (last 10 minutes) CPU, Memory %, IO metrics via the Application Insights Logs API. Mechanics:

• Issues a POST with a Kusto query against performanceCounters selecting the latest minute sample.
• CPU: % Processor Time Normalized.
• Memory: calculated as Private Bytes / Available Bytes * 100.
• IO: IO Data Bytes/sec.
• LargerThanZero helper nulls non-positive results. Failure Modes:
• Null / unreadable response.
• JSON deserialization issues (response must map to ApplicationInsightsQueryResponse).
• Empty tables / rows. On failure returns PulseAgentReport.Fail (all metrics null) per configuration.

Configuration Notes:

• Location must be the Application Insights query endpoint (e.g. https://api.applicationinsights.io/v1/apps/{appId}/query).
• Provide required API key header (e.g. x-api-key:{key}) in Headers or use an injected auth handler.

2. LogAnalyticsWorkspace

Purpose: Query a Log Analytics workspace for multiple applications at once. Mechanics:

• Builds a dynamic Kusto query over AppPerformanceCounters for all configured ApplicationName values.
• Workspace Id taken from first configuration's Location.
• Uses the app's Azure credential and LogsQueryClient.
• Produces a row per application with CPU, Memory %, IO (same calculations as above) and maps back to the correct configuration by AppRoleName. Failure Modes:
• Exception during auth/query (logged as warning).
• No tables / rows returned. Returns a collection of PulseAgentReport or Fail list when unsuccessful.

Configuration Notes:

• Location must be the Workspace Id (GUID string).
• Ensure the executing environment identity has Log Analytics Reader on the workspace.

3. WebAppDeployment

Purpose: Monitor Azure Web App deployment history and status. Mechanics:

• Queries Azure App Service deployment records via ARM API using ArmClient.
• Retrieves deployments within a 2-hour rolling window.
• Extracts deployment metadata: author, status (NotDeployed / InProgress / Succeeded / PartiallySucceeded / Failed), start/end times.
• Attempts to enrich deployment data with commit ID, build number, and deployment type from deployment message JSON.
• Returns DeploymentAgentReport per deployment containing timestamp, status, and optional enrichments. Failure Modes:
• Web App resource not found (logged as error).
• Missing or incomplete deployment data (skipped).
• Deserialization failures on deployment message (logged as warning, enrichments omitted).

Configuration Notes:

• SubscriptionId must contain the Azure subscription ID.
• Location must be the resource group name.
• ApplicationName must be the Web App name.
• Ensure the executing environment identity has Reader role on the Web App resource. See "Azure credential" below.

4. DevOpsDeployment

Purpose: Monitor Azure DevOps environment deployment records. Mechanics:

• Queries Azure DevOps Distributed Task API for environment deployment records within a 2-hour rolling window.
• Filters by BuildDefinitionId to track specific pipeline deployments.
• Enriches deployment data by calling the Builds API to retrieve author, commit ID, and build number.
• Returns DeploymentAgentReport per deployment with result status, timestamps, and metadata. Failure Modes:
• API call failures (non-success status codes logged as errors).
• Missing deployment or build data (skipped or partial data returned).
• Deserialization failures (logged as errors).

Configuration Notes:

• Location must be the Azure DevOps project name.
• ApplicationName must be the team name.
• SubscriptionId must be the environment ID (numeric).
• BuildDefinitionId must be set to filter deployments by pipeline.
• Provide Personal Access Token (PAT) with Read access to Environments, Deployments, and Builds in Headers (e.g., Authorization:Basic <base64-encoded-PAT>).

---

Webhooks

Webhooks allow external systems to be notified when pulse events occur.

Webhook entity fields:

• Id: unique identifier.
• Secret: authentication token for the webhook endpoint.
• Group / Name: filters which pulses trigger this webhook.
• Location: target URL to POST notifications.
• Enabled: activation flag.
• Type: one of WebhookType values (All / StateChange / ThresholdBreach).

Webhook Types

1. All: Triggered for any webhook type.
2. StateChange: Triggered only when pulse state transitions (Healthy ↔ Unhealthy).
3. ThresholdBreach: Triggered when consecutive failures reach the configured AlertThreshold.

Alert Threshold

The AlertThreshold (configured in PulseOptions) defines the number of consecutive unhealthy pulse results that trigger a critical alert and webhook notification for ThresholdBreach webhooks.

• When a pulse fails, its failure counter (PulseCounter.Value) increments.
• When Value equals AlertThreshold, a critical log event is emitted and webhooks of type ThresholdBreach are invoked.
• The webhook payload includes the pulse configuration, failure count, and timestamp of the first failure in the sequence.
• Use this to implement escalation policies or integrate with incident management systems.

Configuration Example:

{
  "Pulse": {
    "AlertThreshold": 3
  }
}

---

Running Multiple Instances

All instances sharing the same storage account work together as one deployment, so each environment needs its own storage account:

• At each interval, one instance claims the tick (a row in the State table) and queues one job per pulse check (and per agent group) on the checkjobs queue. That instance also runs the periodic cleanup.
  • An instance whose clock runs ahead can leave a claim in the future, which pauses scheduling until that time is reached (a warning is logged). To recover, delete the schedule row from the State table.
• Every instance executes jobs from that queue, up to SimultaneousPulses at a time, and stores the results itself.
• Jobs that fail are not retried and are skipped for that interval. Jobs interrupted by a shutdown are handed to another instance.
• Webhooks are sent by every instance from the shared webhooks queue.
• Live updates (SSE) poll the RecentPulses table while clients are connected, so they can lag a few seconds behind.

Options (pulse section): QueuePollingSeconds.

---

Headers Format

Both configuration types serialize headers as: Name:Value;Another-Header:OtherValue (no trailing semicolon). They are added with TryAddWithoutValidation.

---

Adding New Pulse Check Types

1. Extend PulseCheckType enum & fast string helper.
2. Implement a subclass of PulseCheck overriding CreateReport.
3. Register it inside PulseCheckFactory switch.
4. Handle any serialization context updates if new models are introduced.

Adding New Agent Check Types

1. Extend AgentCheckType enum & fast string helper.
2. Implement an AgentCheck subclass returning IReadOnlyList<PulseAgentReport>.
3. Add case to AgentCheckFactory.
4. Define configuration expectations (Location meaning, required headers / auth).

---

Error Handling Philosophy

• Fail fast with clear diagnostic message.
• Log deserialization and transport issues with category-specific event ids.
• Avoid storing large healthy payloads to reduce storage cost.

---

Security & Networking

• Optional SSL ignore flag for scenarios with self-signed certs (use sparingly in production).
• Agent checks authenticate to the telemetry they read (Application Insights, Log Analytics) with headers (API key) or the app's Azure credential.

Azure credential

Storage (when PulseStore holds an account name), Data Protection, the WebAppDeployment agent and the LogAnalyticsWorkspace agent share one credential:

• On App Service (WEBSITE_INSTANCE_ID set) without AZURE_CLIENT_SECRET: ManagedIdentityCredential, user-assigned when AZURE_CLIENT_ID is set, otherwise system-assigned.
• Everywhere else: DefaultAzureCredential (environment variables, workload identity, managed identity, developer logins).

---

Telemetry

PulseGuard uses OpenTelemetry for its own traces, metrics and logs. The exporter is picked at startup:

Setting	Exporter
OTEL_EXPORTER_OTLP_ENDPOINT set	OTLP (e.g. Grafana Cloud), configured with the standard OTEL_EXPORTER_OTLP_* variables such as OTEL_EXPORTER_OTLP_HEADERS and OTEL_EXPORTER_OTLP_PROTOCOL
Otherwise, APPLICATIONINSIGHTS_CONNECTION_STRING set	Azure Monitor OpenTelemetry exporter (Application Insights)
Neither	No telemetry is exported

Both exporters receive the same resource attributes and instrumentation. The Application Insights SDK is no longer used, so its settings (APPLICATIONINSIGHTS_DEPENDENCY_TRACKING, APPLICATIONINSIGHTS_DISABLE_USER_TRACKING) have no effect.

This is separate from the ApplicationInsights and LogAnalyticsWorkspace agent checks, which read telemetry of the monitored applications.

Traces cover incoming requests, outgoing HttpClient calls and Azure SDK operations; /health and /warmup are excluded. Metrics cover ASP.NET Core, HttpClient, .NET runtime and process, plus a PulseGuard meter:

Instrument	Tags	Meaning
pulseguard.check.duration (histogram, s)	pulseguard.check.type, pulseguard.check.state	Every pulse check, with its resulting state
pulseguard.check.state_changes	pulseguard.check.state	Checks whose state, message or error changed
pulseguard.check.alert_thresholds		Checks that reached the alert threshold
pulseguard.agent.duration (histogram, s)	pulseguard.agent.type, pulseguard.outcome	Agent check runs
pulseguard.jobs.scheduled	pulseguard.job.kind	Check jobs queued per interval
pulseguard.jobs.processed	pulseguard.outcome	Check jobs taken from the queue (completed, expired, failed, released)
pulseguard.webhooks.sent	pulseguard.webhook.event, pulseguard.outcome	Webhook deliveries

Logs written through ILogger are exported from Warning up, set by Logging:OpenTelemetry:LogLevel in appsettings.json. Other logging providers (e.g. the console) keep using Logging:LogLevel. PulseGuard logs several Information lines per check, so lowering the export level noticeably increases ingestion volume.

Resource attributes:

• service.name: pulseguard by default.
• service.version: the assembly version, without the commit hash.
• service.instance.id: WEBSITE_INSTANCE_ID on App Service, otherwise the machine name.
• host.*, cloud.* and other App Service attributes: resource detectors.
• Anything else, such as service.namespace or deployment.environment.name: the standard OTEL_SERVICE_NAME and OTEL_RESOURCE_ATTRIBUTES environment variables. These are applied last, so they override both the defaults above and the detected values (the App Service detector reports the slot name as deployment.environment.name).

/warmup is an anonymous health endpoint for App Service warmup (WEBSITE_WARMUP_PATH, WEBSITE_SWAP_WARMUP_PING_PATH). It loads the Data Protection key ring and queries Table storage, returning 503 when either fails. Use it for warmup only; keep the App Service health check and other monitors on /health.

---

Sample JSON Subset (Json Check)

ComparisonValue:

{"status":"ok","version":"1.0"}

Target response may contain superset:

{"status":"ok","version":"1.0","uptime":12345,"extra":"ignored"}

Check passes because comparison subset matches intersection.

---

Minimal Configuration Examples

PulseConfiguration (StatusCode):

Group=Core
Name=Homepage
Location=https://example.com/
Type=StatusCode
Timeout=5000
Enabled=true
Headers=User-Agent:PulseGuard

PulseAgentConfiguration (ApplicationInsights):

Sqid=cluster-a
Type=ApplicationInsights
Location=https://api.applicationinsights.io/v1/apps/<appId>/query
ApplicationName=api-service-a
Enabled=true
Headers=x-api-key:XXXX

PulseAgentConfiguration (WebAppDeployment):

Sqid=web-deployments
Type=WebAppDeployment
SubscriptionId=<subscription-id>
Location=my-resource-group
ApplicationName=my-web-app
Enabled=true

PulseAgentConfiguration (DevOpsDeployment):

Sqid=pipeline-deployments
Type=DevOpsDeployment
Location=MyProject
ApplicationName=MyTeam
SubscriptionId=123
BuildDefinitionId=456
Enabled=true
Headers=Authorization:Basic <base64-encoded-PAT>

---

Extensibility Notes

• Factories isolate enum → implementation mapping; missing mapping yields ArgumentOutOfRangeException ensuring misconfiguration surfaces early.
• Fast string helpers provide O(1) enum parsing without reflection; keep them updated when extending.