Data format

1 - Definitions

Dataset: An entitity that defines different data modalites collected. examples: cgm, anthropometry, human genetics.
Field: the basic unit of data. A dataset is composed of fields. Dictionary: contains metadata for each field participant
cohort
research stage

2 - Field types

The HPP data contains 5 different types of fields:
1. Tabular data Saved as parquet files with mandatory and defined indices. Each dataset wil have at least one tabular dataset named {dataset}.parquet.
2. Individual time series Saved as one parquet file per participant. File path is refeenced in a tabular data field.
3. Aggregated time series Saved as one parquet file per group of participants. File path is referenced in a tabular data field.
4. Individual bulk data (e.g. images, bulk omics files). Saved as one file per participans. File path is refeenced in a tabular data field.
4. _Group bulk data (e.g. mappings, group-wise file). File path is refeenced in a tabular data field.

3 - Mandatory inidices & columns

Tabular data contains 4 mandatory indices:

1. participant_id [integer]: A unique participant identifier
   
2. cohort [string]: A unique cohort identifier (e.g. 10k)
   
3. research_stage [string]: A value chosen from a predefined list.
   
4. array_index [integer]: Used whenever several data items are linked to the same data instance, for example in multiple choice questions or multiple measurements on same visit and time (e.g. measure liver ultrasound twice in a row).

Mandatory columns for tabular data include:

1. collection_timestamp [datetime64[ns]]
   
2. collection_date [datetime64[ns]]
   
3. timezone [string]