<!-- markdownlint-disable MD033 -->

# Running the PRObs system

The PRObs system can perform different operations.
Each of them is encoded in a separate RDFox `master` script.

## Modules

### Ontology conversion

Converts the Turtle ontology into Functional-Style OWL.

How to execute it:

```sh
RDFox sandbox <root> 'exec scripts/ontology-conversion/master'
```

where `<root>` is the path to the "Ontologies" folder (`.` if you are inside it).

### Data pre-processing

Converts 'raw' data into CSV files for RDFox.

How to execute it:

```sh
python Ontologies/scripts/preprocess.py
```

### Data conversion

Reads CSV files, and converts all of them into RDF (probs_original_data).

How to execute it:

```sh
RDFox sandbox <root> 'exec scripts/data-conversion/master'
```

where `<root>` is the path to the "Ontologies" folder (`.` if you are inside it).

### Data validation

Reads the RDF file (probs_original_data), and checks if some constraints are verified.

How to execute it:

```sh
RDFox sandbox <root> 'exec scripts/data-validation/master'
```

where `<root>` is the path to the "Ontologies" folder (`.` if you are inside it).

### Data enhancement

Reads the RDF file (probs_original_data), runs all the rules, and converts all of them into RDF.

How to execute it:

```sh
RDFox sandbox <root> 'exec scripts/data-enhancement/master'
```

where `<root>` is the path to the "Ontologies" folder (`.` if you are inside it).

### Test queries

Reads the RDF file with the data (probs_enhanced_data), and answers some queries.

How to execute it:

```sh
RDFox sandbox <root> 'exec scripts/test-queries/master'
```

where `<root>` is the path to the "Ontologies" folder (`.` if you are inside it).

### Reasoning

Reads the RDF file with the data (probs_enhanced_data), adds the reasoning rules, and opens the SPARQL endpoint.

How to execute it:

```sh
RDFox sandbox <root> 'exec scripts/reasoning/master'
```

where `<root>` is the path to the "Ontologies" folder (`.` if you are inside it).

Then go to [`http://localhost:12110/console/default`](http://localhost:12110/console/default) to run your SPARQL queries.

## Operations

### Get an RDFox-friendly version of the ontology

Simply run the [ontology conversion](#ontology-conversion) module.

### Convert data from CSV files (or [other data sources supported by RDFox](https://docs.oxfordsemantic.tech/data-sources.html)) to an RDF file compatible with the PRObs ontology

If you need only one `load_data` file and one `map` file (this should generally be the case):

1. Overwrite the `load_data.rdfox` and the `map.dlog` files in "data-conversion" (obviously, keeping the same names)
2. Run the [data conversion](#data-conversion) module

If you need a more complex loading (it might happen, but we do not have any example of this at the moment):

1. Overwrite the input.rdfox file in "data-conversion" (obviously, keeping the same name)
2. Run the [data conversion](#data-conversion) module

### Check if the data are "valid"

1. Run the [data validation](#data-validation) module
2. Check that all queries have 0 answers

### Add another validation check

1. Add a file "check_CUSTOM-NAME_rules" with the rules to check
2. Add a file "check_CUSTOM-NAME_queries" with the `ASK` queries to check (note that the queries must return no answer if the check is passed)
3. Add a new command `exec check CUSTOM-NAME` in the "validate" file

### Enhance an RDF file containing data compatible with the PRObs ontology

1. Save the file as `probs_original_data.nt.gz` in the `data` folder
2. Run the [data enhancement](#data-enhancement) module

### Run the test queries on an RDF file containing data compatible with the PRObs ontology

1. Save the file as `probs_enhanced_data.nt.gz` in the `data` folder
2. Run the [test queries](#test-queries) module

### Open an endpoint to run SPARQL queries on an RDF file containing data compatible with the PRObs ontology

1. Save the file as `probs_enhanced_data.nt.gz` in the `data` folder
2. Run the [reasoning](#reasoning) module

### Execute everything all at once

You should (almost) never need this, but you could achieve it by simply running:

```sh
doit
```

### Execute from one specific step onwards

We also provide `master-pipeline` scripts if you want to execute multiple commands without saving intermediate files.

To achieve this, you need to use `master-pipeline` version of the module you are interested in instead of the `master` one.

## Relationships

<details>
<summary> DOT code </summary>

```dot
digraph G {
    
  subgraph cluster_ontology_conversion {
    label = <<B>Ontology-conversion</B>>;
    colorscheme=paired10;
    color=1;
    
    original_ontology -> ontology_conversion;
    ontology_conversion -> ontology_fss;
  }
  
  subgraph cluster_data_pre_processing {
    label = <<B>Data pre-processing</B>>;
    colorscheme=paired10;
    color=2;
    
    raw_data -> data_pre_processing;
    data_pre_processing -> csv_files;
  }
  
  subgraph cluster_data_conversion {
    label = <<B>Data conversion</B>>;
    colorscheme=paired10;
    color=3;
    
    ontology_fss -> data_conversion;
    csv_files -> data_conversion;
    data_conversion -> probs_original_data;
  }
  
  subgraph cluster_data_enhancement {
    label = <<B>Data enhancement</B>>;
    colorscheme=paired10;
    color=4;
    
    ontology_fss -> data_enhancement;
    probs_original_data -> data_enhancement;
    enhancement_rules -> data_enhancement;
    data_enhancement -> probs_enhanced_data;
  }
  
  subgraph cluster_test_queries {
    label = <<B>Test queries</B>>;
    colorscheme=paired10;
    color=5;
    
    probs_enhanced_data -> test_queries;
    test_queries -> answers;
  }
  
  subgraph cluster_reasoning {
    label = <<B>Reasoning</B>>;
    colorscheme=paired10;
    color=6;
    
    probs_enhanced_data -> reasoning;
    reasoning_rules -> reasoning;
    reasoning -> endpoint;
  }

    original_ontology [label="original ontology", shape=tripleoctagon]
    ontology_conversion [label="ontology conversion", shape=rectangle]
    ontology_fss [label="Functional-Style OWL ontology", shape=tripleoctagon]

    raw_data [label="raw_data", shape=cylinder]
    data_pre_processing [label="data pre-processing", shape=rectangle]
    csv_files [label="data", shape=folder]

    data_conversion [label="data conversion", shape=rectangle]
    probs_original_data [label="probs_original_data", shape=tripleoctagon]

    enhancement_rules [label="enhancement rules", shape=hexagon]
    data_enhancement [label="data enhancement", shape=rectangle]
    probs_enhanced_data [label="probs_enhanced_data", shape=tripleoctagon]

    test_queries [label="test queries", shape=rectangle]
    answers [label="output", shape=folder]

    reasoning_rules [label="reasoning/rules", shape=hexagon]
    reasoning [label="reasoning", shape=rectangle]
    endpoint [label="endpoint", shape=component]

  subgraph cluster_legend {
    label = <<B>Legend</B>>;
    colorscheme=paired10;
    color=7;
    // rankdir=TB;
    // {rank = same; rdf_owl process }
    
    rdf_owl [label="RDF/OWL", shape=tripleoctagon]
    process [label="Process", shape=rectangle]
    datasource [label="Datasource", shape=cylinder]
    folder [label="Multiple files", shape=folder]
    datalog [label="Datalog", shape=hexagon]
    rdfox_endpoint [label="RDFox endpoint", shape=component]
    
    edge[style=invis];
    rdf_owl -> process -> datasource -> folder -> datalog -> rdfox_endpoint
  }

}
```

</details>

![Relationships diagram](relationships_diagram.svg)
