AlertChannel CRD

Configure notification channels for Slack, PagerDuty, email, and webhooks

Overview

The AlertChannel custom resource defines how and where to send alerts. AlertChannels are cluster-scoped resources that can be referenced by any CronJobMonitor.

// From api/v1alpha1/alertchannel_types.go:7-36
type AlertChannelSpec struct {
    Type          string               // slack, pagerduty, webhook, email
    Slack         *SlackConfig
    PagerDuty     *PagerDutyConfig
    Webhook       *WebhookConfig
    Email         *EmailConfig
    RateLimiting  *RateLimitConfig
    TestOnSave    bool
}

Channel Types

Slack

Send rich formatted alerts to Slack channels via webhooks

PagerDuty

Create incidents in PagerDuty for on-call escalation

Email

Deliver alerts via SMTP email with custom templates

Webhook

POST alerts to any HTTP endpoint with custom payloads

Slack

Send alerts to Slack channels using incoming webhooks.

apiVersion: guardian.illenium.net/v1alpha1
kind: AlertChannel
metadata:
  name: slack-alerts
spec:
  type: slack
  slack:
    webhookSecretRef:
      name: slack-webhook
      namespace: cronjob-guardian
      key: url
    defaultChannel: "#alerts"
    messageTemplate: |
      :rotating_light: *{\{ .Severity | upper }\}*: {\{ .Title }\}
      
      *CronJob:* `{\{ .CronJob.Namespace }\}/{\{ .CronJob.Name }\}`
      *Monitor:* `{\{ .MonitorRef.Namespace }\}/{\{ .MonitorRef.Name }\}`
      
      {\{ .Message }\}
      
      {\{ if .Context.SuggestedFix }\}
      :bulb: *Suggested Fix:* {\{ .Context.SuggestedFix }\}
      {\{ end }\}
  rateLimiting:
    maxAlertsPerHour: 100
    burstLimit: 10

Slack Configuration

// From api/v1alpha1/alertchannel_types.go:38-50
type SlackConfig struct {
    WebhookSecretRef  NamespacedSecretKeyRef
    DefaultChannel    string
    MessageTemplate   string
}

webhookSecretRef · object · required

Reference to a Secret containing the Slack webhook URL.

Create the secret:

kubectl create secret generic slack-webhook \
  -n cronjob-guardian \
  --from-literal=url='https://hooks.slack.com/services/YOUR/WEBHOOK/URL'

defaultChannel · string

Override the webhook’s default channel (e.g., #alerts, @username).

messageTemplate · string

Go template for custom message formatting. If not specified, uses a built-in template.

Available variables:

• {\{ .Type }\} - Alert type (JobFailed, DeadManTriggered, etc.)
• {\{ .Severity }\} - critical or warning
• {\{ .Title }\} - Alert title
• {\{ .Message }\} - Alert message
• {\{ .CronJob.Namespace }\}, {\{ .CronJob.Name }\} - CronJob reference
• {\{ .MonitorRef.Namespace }\}, {\{ .MonitorRef.Name }\} - Monitor reference
• {\{ .Context.Logs }\} - Pod logs
• {\{ .Context.Events }\} - Kubernetes events (array)
• {\{ .Context.SuggestedFix }\} - Suggested fix text
• {\{ .Context.ExitCode }\} - Container exit code
• {\{ .Context.Reason }\} - Termination reason

Template functions:

• {\{ .Severity | upper }\} - Uppercase
• {\{ .Severity | lower }\} - Lowercase
• {\{ .Message | truncate 100 }\} - Truncate to N chars
• {\{ formatTime .Timestamp "RFC3339" }\} - Format time

PagerDuty

Create incidents in PagerDuty for on-call escalation.

apiVersion: guardian.illenium.net/v1alpha1
kind: AlertChannel
metadata:
  name: pagerduty-oncall
spec:
  type: pagerduty
  pagerduty:
    routingKeySecretRef:
      name: pagerduty-key
      namespace: cronjob-guardian
      key: routing-key
    severity: error  # PagerDuty severity: critical, error, warning, info

PagerDuty Configuration

// From api/v1alpha1/alertchannel_types.go:52-61
type PagerDutyConfig struct {
    RoutingKeySecretRef  NamespacedSecretKeyRef
    Severity             string  // critical, error, warning, info
}

routingKeySecretRef · object · required

Reference to a Secret containing the PagerDuty integration/routing key.

Create the secret:

kubectl create secret generic pagerduty-key \
  -n cronjob-guardian \
  --from-literal=routing-key='YOUR_ROUTING_KEY'

severity · string · default: "error"

Default PagerDuty severity. Valid values: critical, error, warning, info.

The dispatcher maps CronJob Guardian severities to PagerDuty:

• Guardian critical → PagerDuty critical
• Guardian warning → PagerDuty configured severity (default: error)

PagerDuty incidents include dedup_key based on the alert key, so repeated alerts update the same incident instead of creating duplicates.

Email

Send alerts via SMTP email with customizable templates.

apiVersion: guardian.illenium.net/v1alpha1
kind: AlertChannel
metadata:
  name: email-team
spec:
  type: email
  email:
    smtpSecretRef:
      name: smtp-credentials
      namespace: cronjob-guardian
    from: [e-mail]
    to:
      - [e-mail]
      - [e-mail]
    subjectTemplate: "[{\{ .Severity | upper }\}] {\{ .CronJob.Name }\} - {\{ .Type }\}"
    bodyTemplate: |
      Alert Type: {\{ .Type }\}
      Severity: {\{ .Severity }\}
      
      CronJob: {\{ .CronJob.Namespace }\}/{\{ .CronJob.Name }\}
      Monitor: {\{ .MonitorRef.Namespace }\}/{\{ .MonitorRef.Name }\}
      
      {\{ .Message }\}
      
      {\{ if .Context.SuggestedFix }\}
      Suggested Fix:
      {\{ .Context.SuggestedFix }\}
      {\{ end }\}
      
      {\{ if .Context.Logs }\}
      Recent Logs:
      {\{ .Context.Logs }\}
      {\{ end }\}

Email Configuration

// From api/v1alpha1/alertchannel_types.go:82-100
type EmailConfig struct {
    SMTPSecretRef    NamespacedSecretRef
    From             string
    To               []string
    SubjectTemplate  string
    BodyTemplate     string
}

smtpSecretRef · object · required

Reference to a Secret containing SMTP configuration.

Required secret keys:

• host - SMTP server hostname
• port - SMTP server port (e.g., “587”)
• username - SMTP username
• password - SMTP password

Create the secret:

kubectl create secret generic smtp-credentials \
  -n cronjob-guardian \
  --from-literal=host='smtp.gmail.com' \
  --from-literal=port='587' \
  --from-literal=username='[e-mail]' \
  --from-literal=password='YOUR_PASSWORD'

from · string · required

Sender email address.

to · string[] · required

List of recipient email addresses.

subjectTemplate · string

Go template for email subject. Defaults to: CronJob Guardian Alert: {\{ .CronJob.Name }\}

bodyTemplate · string

Go template for email body. If not specified, uses a built-in template.

Webhook

Send alerts to custom HTTP endpoints with flexible payload templates.

apiVersion: guardian.illenium.net/v1alpha1
kind: AlertChannel
metadata:
  name: webhook-custom
spec:
  type: webhook
  webhook:
    urlSecretRef:
      name: webhook-url
      namespace: cronjob-guardian
      key: url
    method: POST
    headers:
      Content-Type: application/json
      Authorization: Bearer ${TOKEN}
    payloadTemplate: |
      {
        "alert_type": "{\{ .Type }\}",
        "severity": "{\{ .Severity }\}",
        "cronjob": "{\{ .CronJob.Namespace }\}/{\{ .CronJob.Name }\}",
        "message": {\{ .Message | jsonEscape }\},
        "timestamp": "{\{ formatTime .Timestamp "RFC3339" }\}",
        "exit_code": {\{ .Context.ExitCode }\},
        "suggested_fix": {\{ .Context.SuggestedFix | jsonEscape }\}
      }

Webhook Configuration

// From api/v1alpha1/alertchannel_types.go:63-80
type WebhookConfig struct {
    URLSecretRef     NamespacedSecretKeyRef
    Method           string  // POST, PUT
    Headers          map[string]string
    PayloadTemplate  string
}

urlSecretRef · object · required

Reference to a Secret containing the webhook URL.

Create the secret:

kubectl create secret generic webhook-url \
  -n cronjob-guardian \
  --from-literal=url='https://api.example.com/alerts'

method · string · default: "POST"

HTTP method: POST or PUT.

headers · object

Custom HTTP headers to include in requests.

Environment variable substitution is supported:

headers:
  Authorization: Bearer ${API_TOKEN}

payloadTemplate · string

Go template for the JSON payload. If not specified, sends a default structure.

Use {\{ .Field | jsonEscape }\} for string fields to ensure proper JSON escaping.

Rate Limiting

All channel types support rate limiting to prevent alert storms:

rateLimiting:
  maxAlertsPerHour: 100
  burstLimit: 10

// From api/v1alpha1/alertchannel_types.go:115-126
type RateLimitConfig struct {
    MaxAlertsPerHour  *int32  // Default: 100
    BurstLimit        *int32  // Default: 10
}

maxAlertsPerHour · int32 · default: "100"

Maximum alerts per hour for this channel.

burstLimit · int32 · default: "10"

Maximum alerts per minute (burst allowance).

Rate limits are per-channel. The global dispatcher also has rate limits (default: 50/min) that apply across all channels.

Test on Save

Enable test alert dispatch when the AlertChannel is created or updated:

spec:
  testOnSave: true

This sends a test alert to verify configuration. The test result is reflected in the status:

status:
  ready: true
  lastTestTime: "2026-03-04T10:30:00Z"
  lastTestResult: success

Status Fields

The controller updates the status with operational metrics:

status:
  ready: true
  lastTestTime: "2026-03-04T10:30:00Z"
  lastTestResult: success
  alertsSentTotal: 142
  lastAlertTime: "2026-03-04T12:45:00Z"
  alertsFailedTotal: 3
  lastFailedTime: "2026-03-03T08:20:00Z"
  lastFailedError: "connection timeout"
  consecutiveFailures: 0

ready · bool

Whether the channel is operational.

alertsSentTotal · int64

Total alerts successfully sent via this channel since creation.

alertsFailedTotal · int64

Total alerts that failed to send.

consecutiveFailures · int32

Number of consecutive failures. Resets to 0 on successful send.

Channel Lifecycle

The AlertChannel controller:

1. Registration: When an AlertChannel is created, the controller:

   • Validates the configuration
   • Fetches referenced secrets
   • Registers the channel with the alert dispatcher
   • Optionally sends a test alert (testOnSave: true)

2. Updates: When an AlertChannel is updated:

   • Re-validates configuration
   • Re-fetches secrets (in case they changed)
   • Updates the dispatcher’s channel registry
   • Optionally re-tests

3. Deletion: When an AlertChannel is deleted:

   • Removes from dispatcher registry
   • In-flight alerts may still be delivered
   • Pending (delayed) alerts referencing the channel are cancelled

Referencing Channels

CronJobMonitors reference AlertChannels by name:

apiVersion: guardian.illenium.net/v1alpha1
kind: CronJobMonitor
metadata:
  name: my-monitor
spec:
  alerting:
    channelRefs:
      - name: slack-alerts
        severities: [critical, warning]
      - name: pagerduty-oncall
        severities: [critical]

If a referenced AlertChannel doesn’t exist, the monitor will log warnings but continue operating. Ensure channels are created before monitors reference them.

Examples

Slack Example

Complete Slack configuration

PagerDuty Example

PagerDuty integration setup

Email Example

SMTP email configuration

Webhook Example

Custom webhook integration

Next Steps

CronJobMonitor

Learn how to configure monitors

Alert Templates

Customize alert message formatting