Skip to content

Alerts

GraphPilot watches each service for a fixed set of conditions. When one holds, it opens an incident, emails the organization’s admins and members, and emails them again when the condition ends. There is nothing to set up: every rule is on for a new service.

The rules

Section titled “The rules”
RuleWhat it watches
GraphPilot returned errorsRequests that fail inside GraphPilot, before they reach your origin. Measured over five minutes.
Your origin did not respondRequests that cannot reach your origin at all. Measured over five minutes.
Your origin returned server errorsRequests your origin answers with a server error. Measured over five minutes.
GraphQL errors in your responsesSuccessful responses that carry a GraphQL error, which nothing at the HTTP level can see. Measured over fifteen minutes.
Cache hit rate droppedCache hit rate against its usual level at the same hour of day over the previous seven days. Measured over thirty minutes, and silent until the service has at least three days of history.
Traffic stoppedTraffic stopping at an hour of day that usually has some, judged against the previous seven days. Measured over fifteen minutes, and silent until the service has at least three days of history.

The first three mean something is broken, the last three that something is degraded. The thresholds are set by GraphPilot, not by you, and a rule needs a minimum number of requests in its window before it can fire, so a service with a handful of requests does not alert because one of them failed.

To see what GraphQL errors in your responses is watching, look at the GraphQL error rate on the service’s Dashboard, or GraphQL errors under Error rate in Insights. Both count the responses that carried a GraphQL error, whatever the HTTP status, and lead to the list of those responses. The Error rate beside them counts HTTP statuses of 400 or higher and does not see these.

Alerts do not depend on stored request telemetry. They are computed from every request, including requests served in lean mode and requests past your plan’s logged-request allowance. The one exception is Cache hit rate dropped: requests served in lean mode are left out of it, because lean mode does not cache.

The emails

Section titled “The emails”

When a condition starts to hold, everyone in the organization whose role is Admin or Member gets an email headed with the rule’s name, as in the table above. A condition that goes on for an hour is one email, not one per check. The condition counts as over once it has not been seen for fifteen minutes, so the second email, saying it stopped, reaches the same people about fifteen to twenty minutes after the last bad minute. If the condition comes back after that, it is a new incident with a new email.

The recipients follow the role, not its permissions. Members with the Developer or Viewer role, or any other role, get no alert email. With service:view they can follow incidents on the service’s Alerts page. To have a person emailed, give them the Admin or Member role, see members and roles. There is no per-person setting to opt in or out.

Email is the only channel today.

The incident history

Section titled “The incident history”

Open the service and choose Alerts. The page lists every incident: which rule, when it started, how long it went on, and the measurement that crossed the threshold (for example the share of requests that failed). An incident that is still open reads Ongoing, and the service’s Dashboard shows it under Open alerts. The list shows the most recent hundred incidents and says when it is capped.

The page is read-only. What you can change is which rules are evaluated, under the service’s settings. An Admin can delete the closed incidents with reset history.

Turn a rule off or on

Section titled “Turn a rule off or on”

Permission service:edit

  1. Open the service’s Settings → Alerts.

  2. Each rule is listed with what it watches, when it last fired, and who last changed it. Use the switch beside a rule to turn it off or back on.

    Last fired links to the incident history narrowed to that rule; Show every rule there lifts the filter. A rule that has never opened an incident reads Never fired.

    Last changed by names the person (point at the name for their email), or an API key by its name and redacted value. It reads Former member for someone no longer in your organization, Deleted user for a deleted account, and System for a threshold GraphPilot tuned for your service.

A switch takes effect within about two minutes. Off means off: a rule that is switched off is not evaluated at all, so no incident opens and nobody is emailed. If you switch a rule off during an incident you were already emailed about, you still get the email when that incident ends.

Members without service:edit see every rule and its state, with the switch disabled.

Example: server errors after a deployment

Section titled “Example: server errors after a deployment”

Ten minutes after a deployment of shop-api, the Admins and Members get an email headed Your origin returned server errors for shop-api, with the share of failed requests in it. Open this incident leads to the service’s Alerts page, where the incident reads Ongoing. From there:

  1. In Insights, apply the Server errors preset (under Errors) for the last 30 minutes. Each row carries the status your origin returned and the operation.
  2. Or from a terminal: gpilot requests --service shop-api --status 502 --status 503 --since 30m --wide.
  3. If the failures start with the deployment, open Deployments, choose the deployment before it and Redeploy. See roll back to a deployment that worked.

About fifteen to twenty minutes after the last failing minute, a second email says the condition stopped. The whole walk-through is the recipe triaging an alert.

Section titled “Related”