.. _data-access: *************************************************** Accessing Data through the Graphical User Interface *************************************************** The graphical user interface (Dashboard) for the TOAR phase 2 database is currently under development and will be described here as soon as it is available. For the time being, data from the TOAR database version 2 can only be accessed via the REST API (see :numref:`chapter-rest-api`). Access to version 1 of the database (from TOAR-I) is available through the GUI at https://toar-data.fz-juelich.de/gui/v1/ which redirects to https://join.fz-juelich.de. This web interface requires registration and is described at https://join.fz-juelich.de/static/documentation/JOIN_FAQ.pdf. .. _chapter-rest-api: ***************************************************************** Accessing Data through the REST Application Programming Interface ***************************************************************** A Representational State Transfer (REST) service allows querying all metadata and data products from the TOAR database of surface ozone observations. This API can be used in a web browser or from within a program, from a Unix shell, or in a graphical web application. This section describes the URL structure and sample queries of the TOAR V2 REST interface. For general information on REST, please consult other resources [#f6]_ . ------------------- General Information ------------------- ~~~~~~~~ Base URL ~~~~~~~~ https://toar-data.fz-juelich.de/api/v2/ Response: Description and documentation of the available REST services. ~~~~~~~~ Services ~~~~~~~~ The following information services are available and described individually below. Each service is invoked by appending its name and possible query arguments to the base URL. * stationmeta: query station ids, station names, and station location from the database * timeseries: query the data series id and specific metadata of a series from the database * data: get timeseries data from the database * variables: query information on variables * contacts: query information on contacts * controlled_vocabulary: query the controlled vocabulary and their description from the database * ontology: query the used ontology of the database * database_statistics: only provides number of users, stations, time series and data records, there is nothing from any kind of statistical product * analysis: get bulk time series data and aggregated time series data * geolocation_urls: query information about geolocation urls * stationmeta_changelog: query information on stationmeta * timeseries_changelog: query information on timeseries * search: query for stations / time series with certain metadata * statistics: a separate package providing various statistics on TOAR data ~~~~~~~~~~~~~~~ Query Arguments ~~~~~~~~~~~~~~~ In order to control the database queries and hence the response of the TOAR REST service, you can add arguments to the service URL. These arguments must adhere to the format ``argumentname=value``. The first argument is prepended by a **?** character, all other arguments are separated by **&** characters. ~~~~~~~~~~~~~~~ Response Format ~~~~~~~~~~~~~~~ The default response format is json. You can control the format with the ``format=`` option in the data and ontology queries. Currently, `json `_ and `csv `_ are supported. ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Authentication And Authorization ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Since accessing TOAR services can consume considerable resources, we are forced to introduce user registration and login to the TOAR data infrastructure. The TOAR database can be accessed in various ways, depending on the level of access desired. Users can choose to explore the database as an anonymous user, with strict limitations on the number of data sets and/or length of timeseries that can be processed. Alternatively, users can register for an account to become a registered user, which enhances download and processing abilities. Registered users can log in via the login button on the TOAR dashboard, using the Helmholtz AAI, ORCID, or Google account. Scientists directly involved in a TOAR activity can request to become a TOAR user with even more capabilities, and individuals in charge of producing global analyses for the TOAR assessment can be elevated to power user. For help with registration, users can send an email to info@toar-data.org. **Resource limitations and user roles** To handle the very different types of database requests, we established the new TOAR Billing Unit (TBU), which combines two types of request parameters: length of timeseries and number of timeseries. A TBU is a (partial) result of the Analysis Service or the TOAR Gridding Service. It combines a daterange and the number of timeseries that are processed in one request. We count 5 years of statistical analysis of one timeseries (arbitrary number of metrics) or aggregation of data from max. 6,000 timeseries as 1 TBU. This means that, for example, a complete global map of 10,000 ozone datasets will count as 2 TBUs, or the calculation of 10 metrics on one 30-year timeseries will count as 6 TBUs. Requesting a 20-years-calculation for all ozone time series (approximately 18000 time series) will therefore result in 12 TBUs (4-times 5 years x 3-times 6,000 timeseries ids = 12 TBU). Note that API requests with no specified daterange automatically count as if the maximum available daterange (55 years = 12 TBUs per 6,000 stations) had been requested. Anonymous user: * users who are not logged in * max download of 5 time series at a time * no custom gridded products, but only download of a static example file * access to the REST API only for 3 time series within the ID range of 1000 to 2000 registered user: * users who self-registered but did not get the privileged TOAR user access * max download of 10 time series at a time, max. 250 time series per month * max 1 concurrent gridded product per request, max 12 gridded products per month TOAR user: * users who self-registered and were elevated to TOAR user after sending us a register request via the dashboard * max download of 500 time series at a time, max. 10000 time series per month * max 12 concurrent gridded products per request, max 120 gridded products per month Power user: * special users who have a mandate to perform extensive analyses for the TOAR assessment papers or other relevant studies. These users should demonstrate that they understand the TOAR data services and can work efficiently with them. * “no limitations” regarding TBUs --------------------------- Description of the Services --------------------------- For all services the default for the number of returned entries is 10, in case you want to see more entries use the query option ``?limit=`` .. _subsection-stationmeta: ~~~~~~~~~~~ Stationmeta ~~~~~~~~~~~ :: Query: https://toar-data.fz-juelich.de/api/v2/stationmeta/[id/][?QUERY-OPTIONS] where QUERY-OPTIONS are: limit= (examples: 10) bounding_box=,,, country=,, ... (country code defined in ISO-3166 ALPHA-2) htap_region_tier1_year2010= ... Response: Each query result consists of all fields of station metadata. If no QUERY-OPTIONS are given, the complete set of stations will be returned. Example: https://toar-data.fz-juelich.de/api/v2/stationmeta/CPT134S00/ Further query items are: * /stationmeta/{station_code} * /stationmeta/id/{station_id} * /stationmata/? * * /stationmeta_changelog/{station_id} ~~~~~~~~~~~ Time Series ~~~~~~~~~~~ :: Query: https://toar-data.fz-juelich.de/api/v2/timeseries/[?QUERY-OPTIONS] where QUERY-OPTIONS are: limit= station_code=,.... variable_id= format= (json|csv) Response: Each query result consists of all fields of time series metadata. If no QUERY-OPTIONS are given, the complete set of time series will be returned. Example (1), query the first time series: https://toar-data.fz-juelich.de/api/v2/timeseries/?limit=1 Example (2), query the time series with id 25: https://toar-data.fz-juelich.de/api/v2/timeseries/25 Example (3), query the timeseries of ozone measurements at the three listed stations (variable_id 5 = ozone) https://toar-data.fz-juelich.de/api/v2/timeseries/?station_code=MX_PED,CPT134S00,CH0001G&variable_id=5 Further query items are: * /timeseries/{timeseries_id} * /timeseries/id/{timeseries_id} * /timeseries/unique/ * /timeseries/citation/{timeseries_id} * /timeseries/? * * /timeseries_changelog/{timeseries_id} ~~~~ Data ~~~~ :: Query: https://toar-data.fz-juelich.de/api/v2/data/timeseries/[?QUERY-OPTIONS] where QUERY-OPTIONS are: format = (json|csv) flags = (see controlled vocabulary for data-flags: https://esde.pages.jsc.fz-juelich.de/toar-data/toardb_fastapi/docs/toardb_fastapi.html#data-flag) Response: Each query result consists of the fields that are specified in the columns argument. If columns are not specified, the output of each record will consist of the fieldsseries_id, network_name, station_id, parameter_label as the series query. If no QUERY-OPTIONS are given, the complete set of stations will be returned. Example (1), query data of time series with id “52”: https://toar-data.fz-juelich.de/api/v2/data/timeseries/52 Example (2), query data of time series with id “52” and return the result as comma separated list: https://toar-data.fz-juelich.de/api/v2/data/timeseries/52/?format=csv Further query items are: * /data/{timeseries_id} * /data/id/{timeseries_id} * /data/timeseries/{timeseries_id}?flags={flag_name} * ... .. _subsection-variables: ~~~~~~~~~ Variables ~~~~~~~~~ :: Query: https://toar-data.fz-juelich.de/api/v2/variables/[id/][?QUERY-OPTIONS] where QUERY-OPTIONS are: limit= (default: 10) Response: Each query result consists of a list of variables with name, longname, dispalyname, cf_standardname, units, chemical-formular, and its internal id, which can be used to directly query that specific variable. Further query items are: * /variables/{name} or /variable/?name={name} * /variables/id/{variable_id} or /variables/?id={variable_id} ~~~~~~~~ Contacts ~~~~~~~~ :: Query: https://toar-data.fz-juelich.de/api/v2/contacts/[persons/|organisations/|id/][?QUERY-OPTIONS] where QUERY-OPTIONS are: limit= (default: 10) Response: Each query result consists of a list of contacts, either all kinds, persons, organisations, or the information for a specific id. Further query items are: * /contacts/persons/id/{person_id} * /contacts/persons/{name} * /contacts/organisations/id/{organisation_id} * /contacts/organisations/{name} * /contacts/id/{contact_id} ~~~~~~~~~~~~~~~~~~~~~ Controlled Vocabulary ~~~~~~~~~~~~~~~~~~~~~ :: Query: https://toar-data.fz-juelich.de/api/v2/controlled_vocabulary/ Response: List of the complete vocabulary in json (raw) format. Further query items are: * /controlled_vocabulary/{name} ~~~~~~~~~~~~~~~~~~~ Database Statistics ~~~~~~~~~~~~~~~~~~~ :: Query: https://toar-data.fz-juelich.de/api/v2/database_statistics/ Response: The database statistics is given: number of users, number of stations, number of time series, and the number of data records. You can also query for only one of these numbers by using its name. Further query items are: * /database_statistics/{name} ~~~~~~~~ Ontology ~~~~~~~~ :: Query: https://toar-data.fz-juelich.de/api/v2/ontology/[?QUERY-OPTIONS] where QUERY-OPTIONS are: format = (xml|owl|doc) Response: By default, the query will return the ontology in xml format. Example: https://toar-data.fz-juelich.de/api/v2/ontology/?format=xml ~~~~~~ Search ~~~~~~ As basis for formulating searches use https://toar-data.fz-juelich.de/api/v2/#stationmeta, https://toar-data.fz-juelich.de/api/v2/#timeseries to list all metadata fields and their definitions as well as the controlled vocabulary defined for a subset of the metadata fields. All metadata fields can be used in searches and combined in one query with **&**: :: Query: https://toar-data.fz-juelich.de/api/v2/search/[?QUERY-Options] where QUERY-OPTIONS are any metadata field = value (or comma separated list of values) Response: all metadata of all stations and timeseries fullfilling the criteria. Example: https://toar-data.fz-juelich.de/api/v2/search/?bounding_box=49,7,50,8&variable_id=5,4 will get you all the time series of stations within an area between 49°N 7°E and 50°N 8°E that record ozone or pm1 https://toar-data.fz-juelich.de/api/v2/search/?name=Aachen will provide all stations located in the town of Aachen (done via similarity search) ~~~~~~~~ Analysis ~~~~~~~~ The base URL for the analysis package, a web-service for the calculation of various anaylsis on TOAR data, is https://toar-data.fz-juelich.de/api/v2/analysis . There the API is documented, especially all available analysis methods are listed with their definitions. :: Query: https://toar-data.fz-juelich.de/api/v2/analysis/[ENDPOINT]/[?QUERY-OPTIONS] where ENDPOINT is: data/timeseries: bulk time series download statistics: aggregated time series download where QUERY-Options are any metadata field = value (or comma separated list of values) flags = (see controlled vocabulary for data-flags: https://esde.pages.jsc.fz-juelich.de/toar-data/toardb_fastapi/docs/toardb_fastapi.html#data-flag) sampling = temporal aggregation (only for endpoint=statistics) metrics = statistical aggregation (only for endpoint=statistics) Response: A zip file with the requested data in csv format. Example: Example (1), query data of all German time series: https://toar-data.fz-juelich.de/api/v2/analysis/data/timeseries/?country=DE&limit=None Example (2), query annual median values of all German time series: https://toar-data.fz-juelich.de/api/v2/analysis/statistics/?sampling=annual&metrics=median&country=DE&limit=None .. rubric:: Footnotes .. [#f6] e.g. https://restfulapi.net/ or https://mlsdev.com/blog/81-a-beginner-s-tutorial-for-understanding-restful-api