Triaging an alert
Goal: an alert email arrives. Within a few minutes you know whether the problem is your origin, your last deployment, or GraphPilot, and you have either fixed it or rolled back.
What each rule watches and who is emailed is on alerts. This recipe walks one incident from the email to the end.
The running example is shop-api. A deployment went out at 14:02. At 14:12 the Admins and
Members get an email headed Your origin returned server errors for shop-api.
What you need
Section titled “What you need”- The
gpilotCLI, logged in, or the portal. service:viewto read Insights and the Alerts page, anddeployment:editto roll back.
1. Read the email
Section titled “1. Read the email”The email names the rule, the service and the measurement, for example the share of requests your origin answered with a server error over the last five minutes. Which rule fired already narrows the search:
| Rule in the subject | Where to look first |
|---|---|
| GraphPilot returned errors | GraphPilot’s status page, linked in the email. Requests fail before they reach your origin; nothing to fix on your side. |
| Your origin did not respond | Your origin’s reachability: a firewall rule, a rate limit, an expired certificate or a DNS change. |
| Your origin returned server errors | Your application and whatever sits directly in front of it. |
| GraphQL errors in your responses | The errors your origin returned inside 200 responses. |
| Cache hit rate dropped | The skip reason the email names, and the last deployment’s caching changes. |
| Traffic stopped | Your clients and the DNS record of the service. |
Open this incident leads to the service’s Alerts page, where the incident reads Ongoing with the time it started.
2. Find the failing requests
Section titled “2. Find the failing requests”In Insights, set the time range to the last 30 minutes and apply the Server errors preset, under Errors. Each row shows the status your origin returned, the operation and the path. Open one: the Overview tab says whether the request reached your origin and with which status, and the GraphQL tab shows the document your origin received.
The same from a terminal:
gpilot requests --service shop-api --status 500 --status 502 --status 503 --since 30m --wide--wide adds the request id, the cache reason, the cache key and the surrogate keys. To follow a
request into your own logs, see correlating with your logs.
Three questions decide what to do next:
- Did it start with the deployment? Compare the first failing request with the deployment’s time on the Deployments page. Insights can also filter by deployment (the Deployment group in the Filters panel).
- Is it one operation or all of them? One operation points at a resolver or a schema change. All of them point at the origin itself.
- Did the origin answer at all? An origin status in the request detail means your application answered. No origin status means the request never got an answer, which is the territory of Your origin did not respond.
3. Roll back if the release caused it
Section titled “3. Roll back if the release caused it”If the failures start with the deployment, roll back before you debug:
- Open Deployments in the portal and the deployment before the bad one.
- Choose Redeploy. The dialog starts with that deployment’s schema and configuration.
- Confirm. The previous version keeps serving until the new deployment is live.
From the CLI, deploy the previous files again; see roll back to a deployment that worked. If the bad version stored wrong answers, purge them once the rollback is live, because the cache survives a deployment.
If your origin release, not the GraphPilot deployment, caused it, roll back the origin; nothing on the GraphPilot side needs to change.
4. Watch it close
Section titled “4. Watch it close”When the condition has not been seen for fifteen minutes, the incident closes and a second email says so, about fifteen to twenty minutes after the last failing minute. The incident then shows its duration on the Alerts page. A condition that comes back after that is a new incident with a new email.
Pitfalls
Section titled “Pitfalls”- Only Admins and Members are emailed. A Developer on call gets nothing. Give the people who should be woken the Member role; see the emails.
- Alerts see requests Insights does not. Alerts count every request, including those past
your logged-request allowance and those left out by
telemetry.record. If Insights shows fewer failures than the email, that is why. - Do not switch the rule off to stop the emails. An incident is one email at the start and one at the end. Switch Traffic stopped off for planned maintenance, not the error rules.