prodcom-data

PRODCOM data as PRObs Observations

This repository converts data from the PRODCOM database into a structure defined by the Physical Resources Observatory (PRObs) ontology.

See DEVELOPING.md for more information about using this repository.

Dataset structure

• Repository is a datalad dataset
• Input data files needing preprocessing are located in raw_data/.
• Preprocessed data files ready for conversion are located in data/.
• All custom code is located in scripts/.
• Converted data is saved to outputs/.

Installation

Getting the code

To clone the datalad dataset, in a shell/command window (e.g. git-bash) type:

datalad clone https://github.com/probs-lab/prodcom-data.git

Setting up the virtual environment and installing dependencies:

To create a virtual environment using conda/miniconda:

cd prodcom-data
conda env create

Running the code

After installation:

• Open a terminal / git-bash window
• Navigate to prodcom-data folder, e.g. cd prodcom-data
• Activate environment using conda activate prodcom-data

To download the example RDF output files (outputs/sold_production and outputs/total_production) from the server use:

datalad get outputs

To download the input csv files used to generate the output files use:

datalad get bulk_data

These files have been generated from bulk csv files downloaded from the eurostat website, which have been split into files for each country and for each year (see DEVELOPING.md).

The dodo.py script can be used to preprocess the files in raw_data and convert the files in the data and bulk_data folders:

To preprocess input data files run the script:

doit run preprocess

To convert the preprocessed data in the data folder run:

doit run convert_data

To convert all files in the bulk_data folder run:

doit convert_bulk

To run all necessary tasks (i.e. preprocessing and conversion) simply run:

doit

Individual files can be converted by running the convert_data.py script with appropriate parameters specifying the file type and the input and output filenames:

scripts/convert_data.py prodcom data/PRODCOM2016DATA.csv outputs/PRODCOM2016DATA.nt.gz

For conversion of the example PRODCOM data files in folder raw_data the type prodcom should be specified. Types prodcom_list and prodcom_correspondence are also defined, along with prodcom_bulk_sold and prodcom_bulk_total (for processing bulk files in folder bulk_data).

Converting new data

For conversion of new data files (possibly in a different format from the examples) see the DEVELOPING.md file.

Testing the code

To test the code, after installing the software and running the doit script:

cd tests
pytest