Data loader

Contents

Data loader#

To populate SkyPortal with initial data, use tools/data_loader.py. This is also used when doing make load_seed_data and make load_demo_data.

The data loader reads a data file in YAML format. This YAML files matches the SkyPortal API: each entry corresponds to an API endpoint, and contains a list of objects to post (see data/db_demo.yaml for an example).

References#

The attributes of a posted object can be saved as references:

groups:
  - name: Program A
    group_admins:
      - testadmin@cesium-ml.org
    =id: program_A

Here, the id field of the created group is stored as program_A. This can now be used when, e.g., posting a telescope:

telescope:
  - name: Palomar 1.5m
    nickname: P60
    lat: 33.3633675
    lon: -116.8361345
    elevation: 1870
    diameter: 1.5
    skycam_link: http://bianca.palomar.caltech.edu/images/allsky/AllSkyCurrentImage.JPG
    group_ids:
      - =program_A

Endpoints can also contain references:

groups/=program_A/users:
  - username: viewonlyuser@cesium-ml.org
    admin: false

Here, the endpoint will become, e.g., groups/6/users.

Files#

Lists of objects to post can be loaded through a top-level file entry:

telescope:
  file: telescopes.yaml

If objects reference png files, those will be read in and posted as b64-encoded bytes:

thumbnail:
  - obj_id: 14gqr
    ttype: new
    data:
      file: ../skyportal/tests/data/14gqr_new.png

CSV files are read as columns, each of which becomes an entry in the posted object:

photometry:
  - obj_id: 14gqr_unsaved_copy
    file: phot.csv
    group_ids:
      - =program_A

Here, phot.csv has columns filter, mjd, etc., so each of those would become an entry in the posted object, each with a list of values.

Contents