JSON-SCHEMA for DLite data models

[jesper-friis]

The intension with this issue is to discuss the excellent suggestion in the README file for a JSON-SCHEMA for DLite data models: https://github.com/OO-LD/schema/blob/main/README.md#dlite

DLite datamodel representation

The suggestion in the README file is based on the soft7 json structure for DLite datamodels.
However, the soft5 format may be better suited for our purpose?

Property dimensionality

An alternative to the common dimensions used in DLite to express the dimensionality of the properties, is to define a set of coordinate systems on which the properties are defined. The benefit of this is that a property could be defined as a subclass of the class of quantities defined on a given space (or alternatively with a defined-on-a-given-space restriction). However, it also requires that we define a set of additional classes representing the needed spaces and quantity classes.

A coordinate system is defined as a set of ordered axes (or dimensions) where each axis is represented by a quantity (think EMMO quantity, which could be an ISQ dimensionless quantity).

🔰 Example
Consider a property temperature, which is the temperature measured at different sea depths at different times. Hence, it has as domain a coordinate system with two axis; depth and time.

As an extension to DLite dimensions, it would be very useful to be able to relate the axes to other properties that defines their values. Continuing the example above, it would be very useful to relate the depth axis to a property depth (who's domain is coordinate system with depth as the only axis) that defines the sea depths at which the temperature measurements are done. Similarly, can the time axis be related to a time property.

A next step, which would be very useful for interoperability, is to provide a formalisation of transformations between coordinate systems. A coordinate system spans a space. Different coordinate systems that spans the same space can be transformed into each other. It is also possible to slice a space or coordinate system.

[simontaurus]

@jesper-friis

Many thanks for providing feedback

Regarding soft5 vs soft7: is having the properties defined as dict with the name as key (soft7) vs. defined as array with the name within the property definition (soft5) the main difference? In this case soft7 is closer to the JSON-SCHEMA property notation

Regarding the example:
Let's see how it would look like in the different schema notations

dlite - I need your help here!

description: A depth-resolved temperature measurement over time - 3-dimensional data
dimensions:
  depth_samples: Number of depth values per point in time
  time_samples: Number of repeated measurements at different points in time
properties:
  data:
    type: [float32, float32, float32]
    unit: [s, m, K]
    shape: [time_samples, depth_samples, 1]
    description: 3-dimensional data

dlite schema 1

Time	Depth	Temperature
0	0	305
0	100	299
0	200	296
3600	0	303
3600	100	297
3600	200	294

dlite data 1

description: A depth-resolved temperature measurement over time - single axes?
dimensions:
  depth_samples: Number of depth values per point in time
  time_samples: Number of repeated measurements at different points in time
properties:
  time:
    type: float32
    unit: s
    shape: [time_samples]
    description: timestamp of the measurement
  depth:
    type: float32
    unit: m
    shape: [time_samples * depth_samples]
    description: depth of the measurement
  temperature:
    type: float32
    unit: K
    shape: [time_samples * depth_samples * 1]
    description: depth of the measurement

dlite schema 2

Time = [0, 3600] s
Depth = [0, 100, 200] m
Temperature = [305, 299, 296, 303, 297, 294] K

dlite data 2

in OO-LD we would create explicite classes for paired axes:

title: DepthResolvedTemperature
properties:
  depth:
    type: number
  temperature:
    type: number

title: SeriesOfDepthResolvedTemperature
properties:
  data:
    type: array
    items:
      $ref: ./DepthResolvedTemperature

title: TimeDepthResolvedTemperature
properties:
  time:
    type: number
  depth:
    type: number
  temperature:
    type: number

title: SeriesOfTimeDepthResolvedTemperature
properties:
  data:
    type: array
    items:
      $ref: ./TimeDepthResolvedTemperature

OO-LD schema 1

data:
  - {time: 0, depth: 0, temperature: 305}
  - {time: 0, depth: 100, temperature: 299}
  - {time: 0, depth: 200, temperature: 296}
  - {time: 3600, depth: 0, temperature: 303}
  - {time: 3600, depth: 100, temperature: 297}
  - {time: 3600, depth: 200, temperature: 294}

or, more accurately if there's only a single timestamp for a depth profile

title: TimeResolvedSeriesOfDepthResolvedTemperature
properties:
  time:
    type: number
  data:
    type: array
    items:
      $ref: ./DepthResolvedTemperature

OO-LD schema 2

data:
  - time: 0
    data: 
      - {depth: 0, temperature: 305}
      - {depth: 100, temperature: 299}
      - {depth: 200, temperature: 296}
  - time: 3600
     data: 
       - {depth: 0, temperature: 303}
       - {depth: 100, temperature: 297}
       - {depth: 200, temperature: 294}

OO-LD schema 2

Composition of data characteristics will also enable to extract subcharacteristics, e.g. someone who's only interested in SeriesOfDepthResolvedTemperature is aware that TimeResolvedSeriesOfDepthResolvedTemperature is a collection of those.

Note: Instead of number literals we will use quantity value objects {value: 305, unit: qudt:K, type: qudt:Temperature} with optional unit and type key (just examples here)

[jesper-friis]

Regarding soft5 vs soft7: is having the properties defined as dict with the name as key (soft7) vs. defined as array with the name within the property definition (soft5) the main difference?

Yes, that is the main difference.

In this case soft7 is closer to the JSON-SCHEMA property notation

Yes, you are right, then we use soft7 notation.

Regarding the example: Let's see how it would look like in the different schema notations

DLite datamodels for schema 1

In DLite a property can only have one unit (think of them as physical quantities). So we should split time, depth and temperature into separate properties:

uri: http://onto-ns.com/meta/oold/ex/schema1/TemperatureMeasurements
description: A sequence of temperature measurements as a function of time and depth.
dimensions:
  nmeasurements: Number of measurements.
properties:
  time: 
    type: float32
    shape: [nmeasurements]
    unit: emmo:Second
    description: The time of each measurement.
  depth:
    type: float32
    shape: [nmeasurements]
    unit: emmo:Metre
    description: The depth of each measurement.
  temperature:
    type: float32
    shape: [nmeasurements]
    unit: emmo:Kelvin
    description: The temperature at each measurement.

DLite instance:

http://onto-ns.com/data/98746634-2c26-11f0-8129-d89ef377aa2c:  # Instance IRI
  meta: http://onto-ns.com/meta/oold/ex/schema1/TemperatureMeasurements
  dimensions:
    nmeasurements: 6
  properties: 
    time: [0, 0, 0, 3600, 3600, 3600]
    depth: [0, 100, 200, 0, 100, 200]
    temperature: [305, 299, 296, 303, 297, 294]

DLite datamodel for schema 2

(essentially the same a you suggested above)

uri: http://onto-ns.com/meta/oold/ex/schema2/TemperatureMeasurements
description: Temperature measurements in a time/depth grid.
dimensions:
  time_samples: Number of times at which the temperature measurements are performed.
  depth_samples: Number of depths at which the temperature measurements are performed.
properties:
  time: 
    type: float32
    shape: [time_samples]
    unit: s
    description: The time at which each measurement is done.
  depth:
    type: float32
    shape: [depth_samples]
    unit: m
    description: The depth at which each measurement is done.
  temperature:
    type: float32
    shape: [time_samples, depth_samples]
    unit: K
    description: The measured temperatures.

DLite instance:

http://onto-ns.com/data/1f5c549c-2c25-11f0-9fac-d89ef377aa2c:
  meta: http://onto-ns.com/meta/oold/ex/schema2/TemperatureMeasurements
  dimensions:
    time_samples: 2
    depth_samples: 3
  properties: 
    time: [0, 3600]
    depth: [0, 100, 200]
    temperature:
    - [305, 299, 296]
    - [303, 297, 294]

[jesper-friis]

I think that the above shows a quite clear way to transform between OO-LD and DLite.

Is the next step to implement a conversion between OO-LD and DLite?

Since OO-LD is more based on standards, would it make sense to create a DLite storage plugin for OO-LD? Or is it smarter to implement the support for DLite in OO-LD?

[simontaurus]

For DLite datamodel for schema 2: Is the order of axes given implicitely by the order within the schema?

Is the next step to implement a conversion between OO-LD and DLite?

Would be great.

Since OO-LD is more based on standards, would it make sense to create a DLite storage plugin for OO-LD?

Makes sense. From the side of https://github.com/OO-LD/oold-python we can support with validators etc.

[jesper-friis]

For DLite datamodel for schema 2: Is the order of axes given implicitely by the order within the schema?

The dimensions and properties are unordered. The ordering is only important when defining the shape of a property.

Since OO-LD is more based on standards, would it make sense to create a DLite storage plugin for OO-LD?

Makes sense. From the side of https://github.com/OO-LD/oold-python we can support with validators etc.

Sounds great!