Connect Datadog

To connect Datadog to CoIsland you need two keys: an API key, which says which organization, and an application key, which says which user and carries that user’s permissions. CoIsland calls Datadog’s API straight from your Mac, with no CoIsland server in between, keeps both keys in your login Keychain, and only ever reads. This page covers the keys, the connector form, the three Datadog monitor kinds with queries you can paste, and the errors you may meet.

Get an API key and create an application key

Datadog runs several independent sites. Your site is in your browser’s address when you use Datadog: app.datadoghq.com is US1, us3.datadoghq.com US3, us5.datadoghq.com US5, app.datadoghq.eu EU, ap1.datadoghq.com AP1, ap2.datadoghq.com AP2 and app.ddog-gov.com US1-FED. Keys made on one site do not work on another.

API key. Open Organization Settings › API Keys. With the Standard role you can open an existing key and copy it; only an admin (the api_keys_write permission) can create a new one, so ask an admin if there is none you may use.

Application key. Open Personal Settings › Application Keys (or Organization Settings › Application Keys), select New Key, name it CoIsland, and copy it. The Standard role may create its own (the user_app_keys permission). The key acts as you, with your permissions. To narrow it, give it scopes: CoIsland needs monitors_read, incident_read and timeseries_query. On some sites Datadog shows an application key only once: copy it before you close the dialog.

An admin can remove user_app_keys from your role; Datadog then answers 403 Forbidden when you try to create a key, and you need an admin to create one for you.

Connect Datadog in CoIsland

Open Settings › Connectors, click +, and choose Datadog.

Field	What to enter
Connector name	What monitors call it: datadog. Letters, digits, _ and -.
Site	Your Datadog site. Leave it on US1 if you sign in at app.datadoghq.com.
API key	The API key you copied.
Application key	The application key you created.

Test connector saves nothing. It checks the API key alone (GET /api/v1/validate), so a wrong site or API key says so, then both keys together (GET /api/v2/current_user), and reads Connected with your name, email and site. Add Connector writes the site to ~/Library/Application Support/CoIsland/connectors.json and both keys, as one Keychain item, to your login Keychain. They are only ever sent to your site’s API host, in the DD-API-KEY and DD-APPLICATION-KEY headers. Editing the connector with both key fields blank keeps both keys.

Choose what to watch: the three Datadog monitor kinds

Open Monitors, add a monitor and choose Datadog. A new-items monitor’s first check records what is already there and raises nothing.

Monitors alerting

A Datadog monitor alerts when it enters a state you pick: Alert and Warn unless you choose, or No data. A monitor going from Warn to Alert alerts again; one that recovers and fires again alerts again. Tags narrow it. Checks every minute by default.

state:alert,warn tag:env:prod

Incidents

An incident alerts when it is declared, and again when it is raised to a higher severity. Pick severities (sev-1 to sev-5, or unknown) and states (active and stable unless you choose, or resolved).

severity:sev-1,sev-2 state:active,stable

Metric threshold

Written the way a Datadog metric monitor’s query is: an aggregation over a window, a metric query, a comparison and a number. Each series over the threshold is a row, so with by {host} each host alerts on its own, and again after it recovers and crosses again. Checks every 5 minutes by default.

avg(last_5m):avg:system.cpu.user{env:prod} by {host} > 80

The aggregation is avg, min, max, sum or last, the window from last_1m to last_1d.

What a Datadog alert shows

The notch lists a monitor as Query alert · Alert · env:prod, an incident as SEV-2 #42 · Active · Ana Silva, a series as system.cpu.user · Over > 80 · host:web-1. Clicking one opens the alert in CoIsland: a monitor fills in with the groups alerting now, its priority and its message; an incident with its commander and customer impact; a series with its value now and whether it is still over. Open in Datadog goes to the monitor or the incident.

What a Datadog monitor’s watch file looks like

-- name: Production alerts
-- kind: datadog.monitors
-- connector: datadog
-- every: 1m
-- alert: new-rows

state:alert tag:env:prod

The kinds are datadog.monitors, datadog.incidents and datadog.metric. Watch files lists every header key.

Rate limits

Datadog does not publish fixed limits for these endpoints: each answer says what is left. A monitors check is one request per state and per 100 monitors, an incidents check one per 100 incidents, a metric check one request. When Datadog answers 429, CoIsland backs off until the time Datadog gives.

Troubleshooting Datadog connector errors

• Datadog (US1) refused the API key. The API key was mistyped or revoked, or it belongs to another site: pick the site you sign in to.
• Datadog (US1) refused the keys. The application key was mistyped or revoked, or the two keys come from different sites.
• Not allowed: the application key needs the monitors_read permission. A scoped application key lacks that scope, or your role lacks the permission. The same message names incident_read or timeseries_query for the other kinds.
• Datadog refused the query: then Datadog’s reasons, for a metric query it cannot read.
• Rate limited; retry after 12s. The next check tries again.