2. Accessing Data through the Graphical User Interface

The graphical user interface (Dashboard) for the TOAR phase 2 database is currently under development and will be described here as soon as it is available. For the time being, data from the TOAR database version 2 can only be accessed via the REST API (see Section 3).

Access to version 1 of the database (from TOAR-I) is available through the GUI at https://toar-data.fz-juelich.de/gui/v1/ which redirects to https://join.fz-juelich.de. This web interface requires registration and is described at https://join.fz-juelich.de/static/documentation/JOIN_FAQ.pdf.

3. Accessing Data through the REST Application Programming Interface

A Representational State Transfer (REST) service allows querying all metadata and data products from the TOAR database of surface ozone observations. This API can be used in a web browser or from within a program, from a Unix shell, or in a graphical web application. This section describes the URL structure and sample queries of the TOAR V2 REST interface. For general information on REST, please consult other resources 1 .

3.1. General Information

3.1.1. Base URL

https://toar-data.fz-juelich.de/api/v2/

Response: Description and documentation of the available REST services.

3.1.2. Services

The following information services are available and described individually below. Each service is invoked by appending its name and possible query arguments to the base URL.

  • stationmeta: query station ids, station names, and station location from the database

  • timeseries: query the data series id and specific metadata of a series from the database

  • data: get timeseries data from the database

  • variables: query information on variables

  • contacts: query information on contacts

  • controlled_vocabulary: query the controlled vocabulary and their description from the database

  • ontology: query the used ontology of the database

  • database_statistics: only provides number of users, stations, time series and data records, there is nothing from any kind of statistical product

  • analysis: get bulk time series data and aggregated time series data

  • geolocation_urls: query information about geolocation urls

  • stationmeta_changelog: query information on stationmeta

  • timeseries_changelog: query information on timeseries

  • search: query for stations / time series with certain metadata

  • statistics: a separate package providing various statistics on TOAR data

3.1.3. Query Arguments

In order to control the database queries and hence the response of the TOAR REST service, you can add arguments to the service URL. These arguments must adhere to the format argumentname=value. The first argument is prepended by a ? character, all other arguments are separated by & characters.

3.1.4. Response Format

The default response format is json. You can control the format with the format= option in the data and ontology queries. Currently, json and csv are supported.

3.1.5. Authentication And Authorization

Since accessing TOAR services can consume considerable resources, we are forced to introduce user registration and login to the TOAR data infrastructure. The TOAR database can be accessed in various ways, depending on the level of access desired. Users can choose to explore the database as an anonymous user, with strict limitations on the number of data sets and/or length of timeseries that can be processed. Alternatively, users can register for an account to become a registered user, which enhances download and processing abilities.

Registered users can log in via the login button on the TOAR dashboard, using the Helmholtz AAI, ORCID, or Google account. Scientists directly involved in a TOAR activity can request to become a TOAR user with even more capabilities, and individuals in charge of producing global analyses for the TOAR assessment can be elevated to power user. For help with registration, users can send an email to info@toar-data.org.

Resource limitations and user roles

To handle the very different types of database requests, we established the new TOAR Billing Unit (TBU), which combines two types of request parameters: length of timeseries and number of timeseries. A TBU is a (partial) result of the Analysis Service or the TOAR Gridding Service. It combines a daterange and the number of timeseries that are processed in one request. We count 5 years of statistical analysis of one timeseries (arbitrary number of metrics) or aggregation of data from max. 6,000 timeseries as 1 TBU. This means that, for example, a complete global map of 10,000 ozone datasets will count as 2 TBUs, or the calculation of 10 metrics on one 30-year timeseries will count as 6 TBUs. Requesting a 20-years-calculation for all ozone time series (approximately 18000 time series) will therefore result in 12 TBUs (4-times 5 years x 3-times 6,000 timeseries ids = 12 TBU). Note that API requests with no specified daterange automatically count as if the maximum available daterange (55 years = 12 TBUs per 6,000 stations) had been requested.

Anonymous user:

  • users who are not logged in

  • max download of 5 time series at a time

  • no custom gridded products, but only download of a static example file

  • access to the REST API only for 3 time series within the ID range of 1000 to 2000

registered user:

  • users who self-registered but did not get the privileged TOAR user access

  • max download of 10 time series at a time, max. 250 time series per month

  • max 1 concurrent gridded product per request, max 12 gridded products per month

TOAR user:

  • users who self-registered and were elevated to TOAR user after sending us a register request via the dashboard

  • max download of 500 time series at a time, max. 10000 time series per month

  • max 12 concurrent gridded products per request, max 120 gridded products per month

Power user:

  • special users who have a mandate to perform extensive analyses for the TOAR assessment papers or other relevant studies. These users should demonstrate that they understand the TOAR data services and can work efficiently with them.

  • “no limitations” regarding TBUs

3.2. Description of the Services

For all services the default for the number of returned entries is 10, in case you want to see more entries use the query option ?limit=<integer: count>

3.2.1. Stationmeta

Query:
https://toar-data.fz-juelich.de/api/v2/stationmeta/[id/][?QUERY-OPTIONS]

where QUERY-OPTIONS are:
limit=<integer: count> (examples: 10)
bounding_box=<min_lat>,<min_lon>,<max_lat>,<max_lon>
country=<country code>,<country code>, ...   (country code defined in ISO-3166 ALPHA-2)
htap_region_tier1_year2010=<htap region number>
...

Response:
Each query result consists of all fields of station metadata.
If no QUERY-OPTIONS are given, the complete set of stations will be returned.

Example:
https://toar-data.fz-juelich.de/api/v2/stationmeta/CPT134S00/

Further query items are:
* /stationmeta/{station_code}
* /stationmeta/id/{station_id}
* /stationmata/?<any station metadata field as defined in https://toar-data.fz-juelich.de/api/v2/#stationmeta>
*
* /stationmeta_changelog/{station_id}

3.2.2. Time Series

Query:
https://toar-data.fz-juelich.de/api/v2/timeseries/[?QUERY-OPTIONS]

where QUERY-OPTIONS are:
limit=<integer: count>
station_code=<station code1>,....
variable_id=<integer: variable identifier in TOAR BD>
format=<string> (json|csv)

Response:
Each query result consists of all fields of time series metadata.
If no QUERY-OPTIONS are given, the complete set of time series will be returned.

Example (1), query the first time series:
https://toar-data.fz-juelich.de/api/v2/timeseries/?limit=1

Example (2), query the time series with id 25:
https://toar-data.fz-juelich.de/api/v2/timeseries/25

Example (3), query the timeseries of ozone measurements at the three listed stations (variable_id 5 = ozone)
https://toar-data.fz-juelich.de/api/v2/timeseries/?station_code=MX_PED,CPT134S00,CH0001G&variable_id=5

Further query items are:
* /timeseries/{timeseries_id}
* /timeseries/id/{timeseries_id}
* /timeseries/unique/
* /timeseries/citation/{timeseries_id}
* /timeseries/?<any timeseries metadata field as defined in https://toar-data.fz-juelich.de/api/v2/#timeseries>
*
* /timeseries_changelog/{timeseries_id}

3.2.3. Data

Query:
https://toar-data.fz-juelich.de/api/v2/data/timeseries/[?QUERY-OPTIONS]

where QUERY-OPTIONS are:
format = <string> (json|csv)
flags = <string> (see controlled vocabulary for data-flags: https://esde.pages.jsc.fz-juelich.de/toar-data/toardb_fastapi/docs/toardb_fastapi.html#data-flag)

Response:
Each query result consists of the fields that are specified in the columns argument. If columns are not specified, the output of each record will consist of the fieldsseries_id, network_name, station_id, parameter_label as the series query.
If no QUERY-OPTIONS are given, the complete set of stations will be returned.

Example (1), query data of time series with id “52”:
https://toar-data.fz-juelich.de/api/v2/data/timeseries/52

Example (2), query data of time series with id “52” and return the result as comma separated list:
https://toar-data.fz-juelich.de/api/v2/data/timeseries/52/?format=csv

Further query items are:
* /data/{timeseries_id}
* /data/id/{timeseries_id}
* /data/timeseries/{timeseries_id}?flags={flag_name}
* ...

3.2.4. Variables

Query:
https://toar-data.fz-juelich.de/api/v2/variables/[id/][?QUERY-OPTIONS]

where QUERY-OPTIONS are:
limit= <integer: count> (default: 10)

Response:
Each query result consists of a list of variables  with name, longname, dispalyname, cf_standardname, units, chemical-formular, and its internal id, which can be used to directly query that specific variable.

Further query items are:
* /variables/{name}   or /variable/?name={name}
* /variables/id/{variable_id}   or /variables/?id={variable_id}

3.2.5. Contacts

Query:
https://toar-data.fz-juelich.de/api/v2/contacts/[persons/|organisations/|id/][?QUERY-OPTIONS]

where QUERY-OPTIONS are:
limit= <integer: count> (default: 10)

Response:
Each query result consists of a list of contacts, either all kinds, persons, organisations, or the information for a specific id.

Further query items are:
* /contacts/persons/id/{person_id}
* /contacts/persons/{name}
* /contacts/organisations/id/{organisation_id}
* /contacts/organisations/{name}
* /contacts/id/{contact_id}

3.2.6. Controlled Vocabulary

Query:
https://toar-data.fz-juelich.de/api/v2/controlled_vocabulary/

Response:
List of the complete vocabulary in json (raw) format.

Further query items are:
* /controlled_vocabulary/{name}

3.2.7. Database Statistics

Query:
https://toar-data.fz-juelich.de/api/v2/database_statistics/

Response:
The database statistics is given: number of users, number of stations, number of time series, and the number of data records. You can also query for only one of these numbers by using its name.

Further query items are:
* /database_statistics/{name}

3.2.8. Ontology

Query:
https://toar-data.fz-juelich.de/api/v2/ontology/[?QUERY-OPTIONS]

where QUERY-OPTIONS are:
format = <string> (xml|owl|doc)

Response:
By default, the query will return the ontology in xml format.

Example:
https://toar-data.fz-juelich.de/api/v2/ontology/?format=xml

3.2.10. Analysis

The base URL for the analysis package, a web-service for the calculation of various anaylsis on TOAR data, is https://toar-data.fz-juelich.de/api/v2/analysis . There the API is documented, especially all available analysis methods are listed with their definitions.

Query: https://toar-data.fz-juelich.de/api/v2/analysis/[ENDPOINT]/[?QUERY-OPTIONS]

where ENDPOINT is:
data/timeseries: bulk time series download
statistics: aggregated time series download

where QUERY-Options are
any metadata field = value (or comma separated list of values)
flags = (see controlled vocabulary for data-flags: https://esde.pages.jsc.fz-juelich.de/toar-data/toardb_fastapi/docs/toardb_fastapi.html#data-flag)
sampling = temporal aggregation (only for endpoint=statistics)
metrics = statistical aggregation (only for endpoint=statistics)


Response:
A zip file with the requested data in csv format.

Example:

Example (1), query data of all German time series:
https://toar-data.fz-juelich.de/api/v2/analysis/data/timeseries/?country=DE&limit=None

Example (2), query annual median values of all German time series:
https://toar-data.fz-juelich.de/api/v2/analysis/statistics/?sampling=annual&metrics=median&country=DE&limit=None

Footnotes

1

e.g. https://restfulapi.net/ or https://mlsdev.com/blog/81-a-beginner-s-tutorial-for-understanding-restful-api