Broker plugins

Broker plugins#

SkyPortal integrates with external alert brokers (e.g. BOOM, Kowalski, Fink, Lasair) through a pluggable provider interface, modeled on the follow-up facility APIs (skyportal/facility_apis/). A broker provider is a registered class; a configured connection to a broker is a Broker database record that supplies the per-instance endpoints/credentials the provider operates on.

This lets a new broker be added as a provider class (and shared helpers) instead of forking SkyPortal or re-deriving the same integration in every deployment.

Concepts#

  • BrokerAPI (skyportal/broker_apis/interface.py), the base class. Every operation is a stub that raises NotImplementedError; a provider overrides only what it supports.

  • Registry, providers are listed in the BROKERS tuple in skyportal/broker_apis/__init__.py. The broker_classnames Postgres enum (skyportal/enum_types.py) is derived from it, so provider names are validated at the database level (mirrors api_classnames). Append new providers to the end of the tuple to keep the enum stable.

  • Capabilities, BrokerAPI.implements() reports which operations a provider overrode. Handlers gate on it, and the frontend can show/hide features accordingly.

  • Broker model (skyportal/models/broker.py), one configured broker: name, broker_classname (which provider), active, and encrypted altdata (endpoints/credentials, mirroring Allocation.altdata). Only system admins may create/update/delete brokers, and altdata is redacted from non-admins.

  • Site defaults, default_alert_search and default_crossmatch name the broker the source page’s “Search alerts” button opens and the one its cross-matches (cone searches) run against. At most one broker holds each; pick them on the brokers page. The seeded data makes ALeRCE both.

Operations#

Interactive (SkyPortal → broker): query_alerts, get_alert, get_cutouts, cone_search, save_as_source, and filter management (get_filters, create_filter, update_filter, delete_filter, test_filter, filter_modules).

Ingestion (broker → SkyPortal): run_ingestion, a long-lived consumer/poller (see “Ingestion and filters” below).

Ingestion and filters (end to end)#

Once a broker is configured (a Broker record with valid altdata), ingestion, running its filters and pulling in matching alerts, is driven by a background service and gated by config.

1. Enable the ingestion service#

The broker_ingest service (services/broker_ingest/) runs one task per active broker whose provider implements run_ingestion. It is off by default; enable it in your config and restart the app:

brokers:
  ingest_enabled: true

With it disabled the service idles (it does not exit), so nothing is ever polled. Confirm it via make monitor (the broker_ingest entry should be RUNNING) and log/broker_ingest.log, which prints either broker ingestion disabled ... or starting ingestion for broker <id> (<name>).

brokers.ingest_enabled is a config value, not in the database, not in a broker’s altdata, and not exposed via the API. A broker’s run_ingestion: true capability only means the provider supports ingestion, not that it is running.

2. Attach filters#

A filter_kind: "query" broker (e.g. Lasair) runs one query per SkyPortal Filter linked to it. Create a Filter (on a Stream + your Group), then attach the query, for Lasair the SELECT / FROM / WHERE parts, from the broker page (or POST /api/brokers/<id>/filters/<filter_id>). Filters are re-read every poll cycle, so adding/editing one does not require a restart.

3. When and how often#

run_ingestion polls immediately on start, then sleeps altdata.poll_interval seconds (provider default; e.g. Lasair 86400 = daily, ALeRCE/ANTARES 3600), so the first batch appears within seconds of the service starting, not after a full interval. Changing poll_interval (or other broker altdata) is read once when a broker’s task starts, so restart the ingestion service to apply it.

4. Where results land#

Matching objects are registered as Candidates under the passing filter and appear on the Candidates / Scanning page filtered by it, not as Sources. From there you review and save the ones worth keeping.

5. Auto-save#

To save passing objects as Sources automatically (into the filter’s group) instead of only registering candidates, set the filter’s autosave flag, the “Auto-save passing objects as sources” checkbox in the Lasair filter builder, or POST /api/brokers/<id>/filters/<filter_id> with {"autosave": true}. Objects already saved to that group are skipped. Use it only when the filter is selective enough to trust everything it passes; otherwise keep it off and tighten the filter’s conditions so scanning stays manageable.

Endpoints#

  • GET/POST/PATCH/DELETE /api/brokers[/{id}], manage Broker records.

  • GET /api/brokers/{id}/alerts[/{alert_id}], query alerts (dispatched to the broker’s provider).

  • GET /api/internal/broker_apis, capabilities + config schema of every registered provider (for the frontend).

Writing a provider#

  1. Add skyportal/broker_apis/mybroker.py with a MYBROKER(BrokerAPI) class overriding the operations you support. Read per-instance config from broker.altdata. Provide form_json_schema_config (and optionally ui_json_schema, surveys, validate_config).

  2. Append it to BROKERS in skyportal/broker_apis/__init__.py.

  3. A database migration for the extended broker_apis enum is generated automatically.

See skyportal/broker_apis/generic.py (GENERICBROKER) for a working reference that talks to any REST broker via a configured base_url/token.

Contents