UpHeron
View as Markdown

Prometheus and Grafana

If you run Prometheus, it can scrape your workspace's monitors from UpHeron like any other target: status, response times, uptime, certificate and domain expiry, open incidents and vulnerability findings. Grafana then reads them through Prometheus, and the ready dashboard shows them all. It works on every plan, Free included, and a scrape costs nothing beyond the rate limits.

Scrape configuration

Create a token with the read scope in Settings → API tokens (see Authentication), save it in a file Prometheus can read, such as /etc/prometheus/upheron-token, and add a job to prometheus.yml:

scrape_configs:
  - job_name: upheron
    scheme: https
    metrics_path: /api/v1/metrics
    scrape_interval: 60s
    scrape_timeout: 20s
    authorization:
      type: Bearer
      credentials_file: /etc/prometheus/upheron-token
    static_configs:
      - targets: ["upheron.com"]

authorization sends the token as Authorization: Bearer; X-API-Key works too, for scrapers that set headers. A scrape every 15 to 60 seconds is fine: the numbers are built at most every 10 seconds per workspace, checks run no faster than every 10 seconds, and four scrapes a minute are far below the 300 reads a token may make. With several Prometheus servers, give each its own token.

To try it first:

curl https://upheron.com/api/v1/metrics -H "Authorization: Bearer $UPHERON_API_KEY"

The answer is the Prometheus text format, or OpenMetrics 1.0 when the scraper prefers it in its Accept header, as Prometheus does.

Metrics

Every series carries the monitor's id, name and kind. Values measured at each location also carry location, the location's code (push for heartbeat and webhook monitors). Times are Unix times in seconds and durations are in seconds, as Prometheus expects.

Metric What it is
upheron_monitor_up 1 when the monitor is up, 0 when it is down or degraded. Absent until its first check.
upheron_monitor_status 1 for the status the monitor is in, 0 for the others: status is degraded, down, pending or up.
upheron_monitor_paused 1 when the monitor is paused.
upheron_monitor_maintenance 1 while a maintenance window, or an occurrence of a repeating one, covers the monitor.
upheron_monitor_slo_target_ratio The monitor's availability objective, from 0 to 1, for monitors with one; window is its rolling window (7d, 30d or 90d).
upheron_monitor_error_budget_remaining_ratio The share of the objective's error budget left over its window, from 1 (none used) to 0 (used up).
upheron_monitor_check_interval_seconds Seconds between the monitor's checks.
upheron_monitor_location_up 1 when the newest check at the location passed, 0 when it failed.
upheron_monitor_response_time_seconds The response time of the newest check at the location.
upheron_monitor_last_check_timestamp_seconds When the newest check at the location ran.
upheron_monitor_uptime_ratio The share of checks that passed, from 0 to 1, over a window: 24h (the last 24 hours), 7d or 30d (the last 7 or 30 UTC days, today included). Absent for a window without checks.
upheron_monitor_certificate_expiry_timestamp_seconds When the TLS certificate expires, where known: certificate monitors, and website monitors that check expiry.
upheron_monitor_domain_expiry_timestamp_seconds When the domain's registration expires, where known: domain monitors, and website monitors that check expiry.
upheron_monitor_open_incidents Incidents that are open or acknowledged, including ones opened by hand that list the monitor.
upheron_monitor_vulnerabilities Open vulnerability findings by severity (critical, high, low, medium), for monitors with vulnerability alerts and code repository monitors.

Days until a certificate expires are (upheron_monitor_certificate_expiry_timestamp_seconds - time()) / 86400.

Alerting rules

UpHeron already alerts your notification contacts. If you prefer Alertmanager, rules like these work on the metrics:

groups:
  - name: upheron
    rules:
      - alert: MonitorDown
        expr: upheron_monitor_up == 0 and on (id) upheron_monitor_paused == 0 and on (id) upheron_monitor_maintenance == 0
        for: 2m
        annotations:
          summary: "{{ $labels.name }} is down"
      - alert: CertificateExpiresSoon
        expr: (upheron_monitor_certificate_expiry_timestamp_seconds - time()) / 86400 < 14
        annotations:
          summary: "The certificate of {{ $labels.name }} expires in {{ $value | humanize }} days"
      - alert: CriticalVulnerability
        expr: upheron_monitor_vulnerabilities{severity="critical"} > 0
        annotations:
          summary: "{{ $labels.name }} runs software with a critical vulnerability"

Grafana dashboard

Grafana reads UpHeron through Prometheus; there is no separate Grafana data source to install. Download the dashboard, upheron-grafana-dashboard.json, and import it in Grafana under Dashboards → New → Import, or put it in a folder Grafana provisions dashboards from. Choose your Prometheus data source at the top of the dashboard.

It shows how many monitors are up, down, paused and in maintenance, open incidents and critical and high vulnerabilities, a table of every monitor with its status, uptime over 24 hours, 7 and 30 days, response time and days until its certificate and domain expire, a status timeline, response times by location, expiry and vulnerability bars, and uptime over 30 days. Filter it by kind and monitor.

Prometheus and Grafana · UpHeron