Connect Databricks

To connect Databricks to CoIsland, you create a personal access token in your workspace, add it as a Databricks connector with the workspace URL, then pick what to watch: SQL on a SQL warehouse (Custom SQL or Validation), failed job runs, failed pipeline updates or triggered SQL alerts. You do it alone: no admin, no app registration.

CoIsland calls your workspace’s REST API straight from your Mac: there is no CoIsland server and no CoIsland account. The token stays in your login Keychain, and CoIsland only reads. The job, pipeline and alert monitors read workspace state and never wake a warehouse, so they cost nothing. The SQL monitors run on a warehouse you choose, which costs what any query costs; see what a SQL check costs.

Create a personal access token

1. Open your workspace in a browser. The address bar is the workspace URL:
   • AWS: https://<name>.cloud.databricks.com, for example https://dbc-a1b2c3d4-e5f6.cloud.databricks.com.
   • Azure: https://adb-<workspace-id>.<n>.azuredatabricks.net.
   • Google Cloud: https://<workspace-id>.<n>.gcp.databricks.com.
2. Click your username in the top bar, then Settings › Developer › Access tokens › Manage, and Generate new token.
3. Comment: one you will recognize, such as CoIsland. Lifetime: in days. A workspace admin can cap it; by default new tokens last at most 730 days. Pick a long one and set yourself a reminder: CoIsland cannot read a token’s end date.
4. Generate and copy the token. Databricks shows it once.
5. For SQL monitors, optionally: SQL Warehouses › a warehouse › Connection details, and copy its HTTP path (/sql/1.0/warehouses/<id>) or its ID. You need Can use on it.

Two things to know about these tokens:

• Unused for 90 days, a token is revoked. Databricks revokes a personal access token that has not been used for 90 days. A monitor that checks at least once every 90 days keeps it alive; a connector whose monitors are all paused that long loses it.
• An admin can turn them off. Workspace admins can switch personal access tokens off (Settings › Advanced › Personal Access Tokens) or limit who may use them. Existing tokens are not deleted, and they work again once tokens are turned back on. Databricks labels these tokens “legacy” and recommends OAuth; they remain the one way that needs neither a CLI nor an app registration. Signing in with the Databricks CLI is planned for a later version.

Connect Databricks in CoIsland

1. Open Settings › Connectors and click + (Add Connector).
2. On Add a connector, choose Databricks.
3. Fill in the fields, then click Test connector.
4. Click Add Connector.

Field	What to enter
Workspace URL	The address you open Databricks at. A trailing /, a path or ?o= pasted with it is stripped. http:// is refused
Connector name	What monitors call it: letters, digits, _ and -, starting with a letter. CoIsland suggests dbc-a1b2c3d4 for dbc-a1b2c3d4-e5f6.cloud.databricks.com
Access token	The token you copied, with no spaces or line breaks
SQL warehouse	Optional: the warehouse’s ID or HTTP path, where SQL monitors run when they name none

Test connector asks who the token belongs to (GET /api/2.0/preview/scim/v2/Me, which any user may call), then lists the SQL warehouses you can see, with their type, size, state and auto-stop. It wakes none of them and saves nothing. A regional Azure address such as westus.azuredatabricks.net works but is flagged: use your per-workspace adb- URL.

Add Connector keeps the host and the warehouse in ~/Library/Application Support/CoIsland/connectors.json, and the token in your login Keychain, never in a file. The first Databricks connector becomes the default.

Watch Databricks: the five monitor kinds

Open Settings › Monitors, click + (New Monitor) and choose Databricks. Every editor has a Test button that runs the query once, saves nothing and says what matches now. A monitor’s first check records what already matches and raises nothing; from then on, each new row or item is an alert.

Kind	What you write	What alerts	Costs
Custom SQL	Any SELECT, WITH or SHOW	A new row, or a row count crossing a number	A query on the warehouse
Validation	A table, a column and a rule	A row that breaks the rule	A query on the warehouse
Failed job runs	Jobs, results and a lookback	A run that ended failed or timed out	Nothing
Failed pipeline updates	Pipelines or a name pattern	An update that failed	Nothing
Triggered SQL alerts	Alerts and states	An alert entering TRIGGERED	Nothing

Custom SQL and Validation

The SQL runs on the monitor’s warehouse (under Advanced), else the connector’s. Advanced also takes the Catalog and Schema that unqualified table names resolve in. Only SELECT, WITH and SHOW are accepted, and one statement: a write behind a WITH or a second statement after ; is refused before it reaches Databricks. Examples in the app:

SELECT usage_date, SUM(usage_quantity) AS dbus
FROM system.billing.usage
WHERE usage_date = current_date()
GROUP BY usage_date
HAVING SUM(usage_quantity) > 50

SELECT table_catalog, table_schema, table_name, last_altered
FROM system.information_schema.tables
WHERE table_schema = 'sales' AND last_altered < current_timestamp() - INTERVAL 24 HOURS

A check keeps at most 10,000 rows. When more match, CoIsland compares nothing rather than judge from a partial list. The statement waits up to three minutes for a stopped warehouse to start; past that it is cancelled, so a runaway query stops costing. Timestamps show in your Mac’s time zone. The Statement Execution API has no session time zone, so -- timezone: only changes how times display; convert in SQL with from_utc_timestamp when you need to.

Failed job runs

The editor lists your jobs by name. None picked is every job you can view.

job:123456 job:987654 status:failed,timedout lookback:24h

• status: takes the run results Databricks reports: failed and timedout unless you pick others, such as canceled or upstream_failed.
• lookback: is from 1h to 7d, 24 hours unless given. Runs that started earlier are not looked at.
• A repaired run that fails again alerts again. Up to 480 runs are read per job; past that, narrow with job:.

The card says which job, how it failed (the termination code when it tells more than “execution error”, such as cluster error), and whether it ran on its schedule or who started it. The alert page adds each failed task’s error and stack trace.

Failed pipeline updates

pipeline:0d9c1234-5678-4abc-9def-0123456789ab status:failed
name:sales% status:failed,canceled health:unhealthy

Name pipelines with pipeline:, or match them with a name: LIKE pattern, not both. None is every pipeline you can view, from their latest updates. health:unhealthy keeps only pipelines Databricks marks unhealthy. The alert page shows the update’s error events, fatal first.

Triggered SQL alerts

state:triggered,error
alert:a1b2c3d4-0000-4000-8000-000000000001 state:triggered

An alert (Databricks SQL alerts, the current version) alerts once when it enters TRIGGERED, stays quiet while it stays triggered, and alerts again after it has gone back to OK. The card leads with its condition, such as c > 0. CoIsland never runs an alert’s query itself.

What a SQL check costs

Every SQL check runs on the warehouse, waking it if it is stopped, and the warehouse then stays on until its auto-stop. Under Advanced, the editor says what the chosen warehouse and interval come to, for example: “Serverless 2X-Small, stops 5 min after a check: at every 30 min it runs about 17% of the time.” At half the time or more, the line turns to a warning.

• Use a serverless 2X-Small warehouse, the smallest size.
• Set its auto-stop short: serverless warehouses go down to 1 minute through the API (5 in the UI); pro and classic ones to 10.
• Check every 30 minutes or more. New Databricks SQL monitors propose 30 minutes; job, pipeline and alert monitors propose 5, since they are free.
• Watch jobs and pipelines with their own kinds rather than SQL on system.lakeflow tables.
• CoIsland tags each statement with coisland in Query History, so you can see what it costs.

A Databricks monitor is a watch file

Every monitor is a .sql file in ~/.coisland/watches. This is failed-nightly-etl.sql:

-- name: Failed nightly ETL
-- kind: databricks.job-runs
-- connector: dbc-a1b2c3d4
-- every: 5m
-- alert: new-rows
-- sound: Glass

job:123456 status:failed,timedout lookback:24h

-- kind:	Monitor kind
databricks.custom-sql	Custom SQL, and Validation once saved
databricks.job-runs	Failed job runs
databricks.pipeline-updates	Failed pipeline updates
databricks.sql-alerts	Triggered SQL alerts

Keep the -- kind: line: a file without one is read as Snowflake SQL. A SQL monitor also takes -- warehouse: (an ID or HTTP path), -- database: (the catalog) and -- schema:. -- role: does not apply to Databricks and is ignored with a warning. Watch files lists every header key.

Troubleshooting Databricks connector errors

• “Databricks refused the token: it is wrong, expired, revoked, or was unused for 90 days and revoked.” Databricks answered 401. Create a new token (Settings › Developer › Access tokens) and add it as a new connector.
• “Personal access tokens are turned off in this workspace, or you are not allowed to use them.” An admin disabled tokens or did not grant you their use. Ask a workspace admin (Settings › Advanced › Personal Access Tokens).
• “Signed in, but not allowed: … You need Can use on warehouse …” The token works, but not on that warehouse. Pick another one under Advanced, or ask for Can use.
• “Not found, or not visible to this token: job 123456.” Check the ID, and that you can view the job, pipeline or alert.
• “This does not look like a Databricks workspace URL.” Paste the address you open Databricks at.
• “The workspace answered with a redirect; paste the final workspace URL.” CoIsland follows no redirect, so the token never leaves the host you typed.
• “Databricks is busy; try again in a moment.” A 429 or 503. CoIsland waits a minute, then doubles the wait each time, up to 15 minutes.
• “Pick a SQL warehouse (Advanced), or set one on the connector.” A SQL monitor needs a warehouse.

Frequently asked questions

Can CoIsland change anything in my Databricks workspace?

No. It only reads: SQL monitors must be SELECT, WITH or SHOW, and the other kinds read job runs, pipeline updates and alerts. The only other request is cancelling its own statement when it runs past three minutes.

Do the job, pipeline and alert monitors cost anything?

No. They read workspace state through the REST API and wake no warehouse.

Does it work on Azure Databricks and Google Cloud?

Yes. Paste your workspace URL; the steps are the same on AWS, Azure and Google Cloud.

Where does CoIsland keep my Databricks token?

In your login Keychain. connectors.json holds the connector’s name, host and warehouse, never the token.