The config graph¶
maplib keeps the configuration of a model as triples in the model itself, in a named graph of its
own: the config graph. The configuration travels with the model: it is written, serialized,
served and queried like any other graph, and applied with maplib.config.apply_config.
Namespace |
|
Graph |
|
Python |
|
The namespace is separate from maplib’s vocabulary (https://datatreehouse.github.io/maplib/vocab#,
used for e.g. full text search and validation reports): the config namespace is only for
configuration.
Principles¶
Everything that configures maplib goes here. New kinds of configuration get a class in the config namespace and an individual (or several) in the config graph, read and written by
maplib.config.No secrets. A secret is configured by the name of the environment variable that holds it (
mcfg:secret), read when the configuration is applied. Passwords, tokens and client secrets never enter the model, so a model can be shared or serialized with its configuration.Replaceable items. An item (e.g. a virtualization) is its IRI and the IRIs starting with
<its IRI>/, which name its parts (parameters, resources). Writing an item replaces all of these, so an item is always written whole.Plain SPARQL to read. Anything that reads the configuration (e.g. the Model portal) can query the config graph:
SELECT * WHERE { GRAPH mcfg:ConfigGraph { ?s ?p ?o } }.
Virtualizations¶
A virtualized database: maplib (chrontext) answers the parts of SPARQL queries over its resources by querying the database, so the data is not copied into the model.
@prefix mcfg: <https://datatreehouse.github.io/maplib/config#> .
@prefix rdfs: <http://www.w3.org/2000/01/rdf-schema#> .
@prefix xsd: <http://www.w3.org/2001/XMLSchema#> .
<urn:maplib:config:virtualization:readings> a mcfg:Virtualization , mcfg:SnowflakeDatabase ;
rdfs:label "Meter readings" ;
mcfg:enabled true ;
mcfg:sqlDialect "snowflake" ; # optional, overrides the database's dialect
mcfg:parameter <urn:maplib:config:virtualization:readings/parameter/account> ;
mcfg:secret <urn:maplib:config:virtualization:readings/secret/password> ;
mcfg:resource <urn:maplib:config:virtualization:readings/resource/readings> .
<urn:maplib:config:virtualization:readings/parameter/account>
mcfg:key "account" ; mcfg:value "acme" .
<urn:maplib:config:virtualization:readings/secret/password>
mcfg:key "password" ; mcfg:environmentVariable "SNOWFLAKE_PASSWORD" .
<urn:maplib:config:virtualization:readings/resource/readings> a mcfg:Resource ;
mcfg:resourceName "readings" ;
mcfg:sql "SELECT meter_id AS id, ts AS timestamp, value FROM readings" ;
mcfg:templateDocument """@prefix maplib: <https://datatreehouse.github.io/maplib/vocab#> . ...""" ;
mcfg:property <urn:maplib:config:virtualization:readings/resource/readings/property/0> .
<urn:maplib:config:virtualization:readings/resource/readings/property/0>
mcfg:predicate <http://iec.ch/TC57/2013/CIM-schema-cim16#Measurement.unitSymbol> ;
mcfg:object <http://iec.ch/TC57/2013/CIM-schema-cim16#UnitSymbol.W> .
Classes¶
Class |
Database |
Built with |
|---|---|---|
|
every virtualized database also has this type |
|
|
any SQL database SQLAlchemy can reach (PostgreSQL, SQL Server, Oracle, MySQL, …), with its dialect in |
|
|
DuckDB |
|
|
Snowflake |
|
|
BigQuery |
|
|
Databricks SQL warehouse |
|
|
TimescaleDB (PostgreSQL), with psycopg |
|
|
Microsoft Fabric Lakehouse, its Delta tables read from OneLake (name |
|
|
Microsoft Fabric Warehouse, or a Lakehouse’s SQL analytics endpoint (name |
|
|
Azure Data Explorer, or a Microsoft Fabric Eventhouse’s KQL database (name |
|
|
InfluxDB 3: Core, Enterprise, Cloud Dedicated, Cloud Serverless, Clustered (name |
|
|
InfluxDB 2, InfluxDB Cloud (TSM) and InfluxDB 1.8 (name |
|
|
OPC UA Historical Access |
|
|
Cognite Data Fusion time series |
|
|
OSIsoft (AVEVA) PI, through the PI Web API |
|
|
OSIsoft (AVEVA) PI Data Archive (the PI historian), through the PI Web API |
|
|
AspenTech InfoPlus.21, through the aspenONE Process Data REST API |
|
|
Siemens Insights Hub (formerly MindSphere), through Asset Management, IoT Time Series and IoT Time Series Aggregates (name |
|
|
a resource of a virtualized database |
|
|
the indexes of the model |
|
|
the query result cache of the model |
|
The SQL databases, the OPC UA history database and the Cognite client are only available in maplib
Enterprise Edition (and need their client libraries, e.g. snowflake-connector-python,
cognite-sdk).
Properties¶
Property |
On |
Value |
|---|---|---|
|
|
a name for people |
|
|
|
|
SQL databases |
|
|
|
a connection parameter ( |
|
|
a secret parameter ( |
|
|
a |
|
|
the value of |
|
|
the query giving its rows, with columns named as the template’s parameters (an |
|
|
the template, as a stOTTR document |
|
|
the IRI of a template already in the model (instead of |
|
|
a setting of the resource ( |
|
|
a static property of the objects it virtualizes ( |
|
properties |
IRIs |
|
parameters, options |
strings |
|
secrets |
the name of the environment variable |
Parameters per database¶
The parameters are the keyword arguments of the database’s constructor.
Database |
Parameters |
Secrets |
|---|---|---|
General SQL |
|
|
DuckDB |
|
|
Snowflake |
|
|
BigQuery |
|
|
Databricks |
|
|
TimescaleDB |
|
|
Fabric Lakehouse |
|
|
Fabric Warehouse |
|
|
Azure Data Explorer / Fabric Eventhouse |
|
|
InfluxDB 3 |
|
|
InfluxDB 2 |
|
|
OPC UA |
|
|
CDF |
|
|
OSIsoft PI |
|
|
PI Data Archive |
|
|
AspenTech IP.21 |
|
|
Siemens Insights Hub |
|
|
CDF resources take the options of cognite-sdk’s resource map: id_type (external_id, id or
instance_id), and timestamp and value (the template parameters of the datapoints’ timestamp
and value).
InfluxDB 2 resources are measurements: the options measurement, id (the column with the
external ids of the time series: a tag, _field (the default) when each field is a time series,
or _measurement) and field (only the values of this field). A resource without them takes
the connection parameters of the same names. The templates have the parameters id,
timestamp and value; other parameters are read from the tags with the same names.
Static properties of resources (e.g. units)¶
Facts that are the same for every row of a resource, such as the unit of a measurement, are not
triples of its template: chrontext does not support constant triples in a virtualized template (a
query for them finds nothing). They are resource properties instead
(VirtualizationResource.properties, predicate IRI to object IRI). Applying the virtualization
writes each as an ordinary triple, in the default graph, on every object with maplib:hasResource the
resource name, replacing the values those objects had for the predicate:
CIM = "http://iec.ch/TC57/2013/CIM-schema-cim16#"
VirtualizationResource(name="readings", sql=..., template=..., properties={
CIM + "Measurement.unitSymbol": CIM + "UnitSymbol.W",
CIM + "Measurement.unitMultiplier": CIM + "UnitMultiplier.k",
})
So ?m cim:Measurement.unitSymbol ?u ; cim:Analog.AnalogValues ?p . ?p cim:AnalogValue.value ?v
joins the static unit with the virtualized values. The properties are on the objects, not on their
data points (the data points exist only in query results); a QUDT unit (qudt:unit) is put on the
object too. apply_properties(m, config) writes them without connecting to the database.
In a virtualized template, name the data point with a variable that is not a parameter (?point),
not a blank node (_:point): with a blank node chrontext binds none of the data point’s values.
Indexing¶
mcfg:indexing a mcfg:Indexing ;
mcfg:fts true ; # full text search
mcfg:geo false ; # GeoSPARQL (maplib EE)
mcfg:objectIndexAll false ; # object indexes for all suitable predicates
mcfg:objectIndexPredicate rdfs:label , <http://example.org/serialNumber> . # or for these only
Property |
Value |
|
|---|---|---|
|
|
|
|
|
|
|
|
|
|
a predicate IRI to index by object (any number) |
|
Subjects are always indexed. Object indexes make lookups by value fast (e.g. ?s ex:serialNumber "123").
Without mcfg:objectIndexAll or mcfg:objectIndexPredicate, rdf:type and rdfs:label have object
indexes. mcfg:objectIndexPredicate replaces these defaults, so list them too to keep them.
apply_config creates the configured indexes (Model.create_index(IndexingConfig.options())), also
for the triples already in the model, which can take some time for a large model.
Query cache¶
mcfg:queryCache a mcfg:QueryCache ;
mcfg:enabled true ;
mcfg:maxEntries 1024 ; # results kept at most
mcfg:maxBytes 268435456 ; # their total size at most (bytes)
mcfg:maxEntryBytes 16777216 ; # larger results are not kept
mcfg:ttl 300.0 ; # seconds a result is served at most (optional)
mcfg:minRunTime 0.0 ; # faster queries are not kept (seconds)
mcfg:cacheVirtualized false . # also keep results over virtualized databases
Property |
Value |
|
|---|---|---|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
Missing properties take the defaults above (no ttl). The cache serves Model.query and
Model.serve. A result is kept until a write changes the triples it can read, per named graph and
predicate, so writes elsewhere keep it. The cache does not see changes in virtualized databases:
with mcfg:cacheVirtualized true, set a mcfg:ttl. apply_config sets the cache
(Model.set_query_cache(QueryCacheConfig.options())); without a mcfg:QueryCache the model’s
setting stays.
Common questions¶
A common question is a saved SPARQL query with a name and a description: a question people often ask the model (“Which substations are there, and at which voltages?”). It is a SHACL Advanced Features SPARQL executable, named and described with Dublin Core terms, and linked to the vocabulary of the data it is about.
@prefix mcfg: <https://datatreehouse.github.io/maplib/config#> .
@prefix sh: <http://www.w3.org/ns/shacl#> .
@prefix dcterms: <http://purl.org/dc/terms/> .
@prefix rdfs: <http://www.w3.org/2000/01/rdf-schema#> .
<urn:maplib:config:question:cim:lines-lengths> a mcfg:CommonQuestion , sh:SPARQLSelectExecutable ;
dcterms:title "Lines and their lengths" ;
rdfs:label "Lines and their lengths" ;
dcterms:description "The AC line segments with their length (km), voltage and impedance, longest first." ;
sh:select """PREFIX cim16: <http://iec.ch/TC57/2013/CIM-schema-cim16#>
SELECT ?segment ?name ?length WHERE { ... }""" ;
mcfg:appliesTo <http://iec.ch/TC57/2013/CIM-schema-cim16#> ; # optional: about CIM16 data
mcfg:bundled true ; # shipped with maplib EE
sh:order 1 . # optional: listed by order, then name
Term |
On |
Value |
|---|---|---|
|
class |
a common question; also typed with its SHACL-AF executable class |
|
class |
the query form, with the query in |
|
class |
a DESCRIBE query ( |
|
|
the query, a string with its own |
|
|
the name (also written as |
|
|
what it answers (optional) |
|
|
the vocabulary (namespace IRI) of the data it is about; none for a question about any model |
|
|
|
|
|
|
The domain is the vocabulary itself rather than a class, because not every kind of data has a class
that all of it shares (OPC UA nodes have opc:NodeClass, not an rdf:type). Three have
names in maplib.config.QUESTION_DOMAINS: cim (CIM16), era (the ERA vocabulary) and
opcua (OPC UA nodes). maplib Enterprise Edition ships common questions for these, with a way
to find which the model’s data has (maplib.bundled_questions).
Prefixes¶
The prefixes of the model: short names for namespaces, used to write and show IRIs (ex:Pump for
http://example.org/Pump). They are SHACL prefix declarations
(SHACL 5.2.1), declared with sh:declare by the
owl:Ontology mcfg:prefixes:
@prefix mcfg: <https://datatreehouse.github.io/maplib/config#> .
@prefix sh: <http://www.w3.org/ns/shacl#> .
@prefix owl: <http://www.w3.org/2002/07/owl#> .
@prefix xsd: <http://www.w3.org/2001/XMLSchema#> .
mcfg:prefixes a owl:Ontology ;
sh:declare <https://datatreehouse.github.io/maplib/config#prefixes/plant> .
<https://datatreehouse.github.io/maplib/config#prefixes/plant> a sh:PrefixDeclaration ;
sh:prefix "plant" ;
sh:namespace "http://example.org/plant/"^^xsd:anyURI ;
mcfg:generated true . # optional: made by generate_prefixes
Term |
On |
Value |
|---|---|---|
|
|
a prefix declaration (any number) |
|
|
the name, e.g. |
|
|
the namespace, an |
|
|
|
A name has one namespace and a namespace one name. Since these are ordinary SHACL declarations,
anything that reads sh:declared prefixes (e.g. the Graph portal, which also reads them from shapes
graphs) finds them. apply_config makes them known to the model (Model.add_prefixes), so SPARQL,
Datalog and OTTR given to the model can use them without PREFIX declarations.
generate_prefixes(m, graph=None, max_generated=8, write=True) makes prefixes for the data in a
graph (the default graph when not given, or a list of graphs, None in it for the default graph,
whose uses are added up): the usual prefix of each well-known vocabulary it uses
(maplib.well_known_prefixes(), see below: rdf, rdfs, xsd, owl, sh, skos, dcterms,
dc, prov, geo, geof, sf, cim (CIM100), cim16 (CIM16), entsoe, md, opc, foaf,
schema, dcat, time, sosa, ssn, qudt, unit, era, maplib, …), and a short name for each of the max_generated other namespaces it uses most (the
predicates by their triples, the classes by their instances and the typed subjects;
prefix_name_for names http://example.org/plant/ plant and urn:acme:pump: pump).
Namespaces that have a prefix already, in the config graph or declared in any graph, keep it, and
taken names get a number (plant2). It returns the new prefixes, and writes them unless write=False.
Well-known prefixes¶
maplib has one table of default prefixes, in Rust (representation::prefixes::WELL_KNOWN_PREFIXES),
read in Python with maplib.well_known_prefixes() (name to namespace). Every model knows them when
parsing SPARQL, Datalog and OTTR and writes with them; generate_prefixes, template generation and
the RDF/XML writer name namespaces with them; and the Graph portal shows IRIs with
them (behind the model’s own prefixes). cim is always CIM100 (CGMES 3); CIM16 (CGMES 2.4.15) is
cim16. Applications extend this table rather than keeping their own.
from maplib.config import PrefixConfig, write_prefix, read_prefixes, remove_prefix, generate_prefixes
write_prefix(m, PrefixConfig(name="plant", namespace="http://example.org/plant/"))
m.read("substations.xml", format="cim/xml", graph="urn:graph:substations")
generate_prefixes(m, graph="urn:graph:substations") # cim, md, rdf, ...
for p in read_prefixes(m):
print(p.name, p.namespace, p.generated)
remove_prefix(m, "plant")
Python¶
from maplib.config import (
CommonQuestion, write_question, read_questions, remove_question, question_iri,
)
write_question(m, CommonQuestion(
iri=question_iri("Labels"),
name="Labels",
description="Everything with a label",
query="PREFIX rdfs: <http://www.w3.org/2000/01/rdf-schema#>\nSELECT ?s ?label WHERE { ?s rdfs:label ?label }",
))
for q in read_questions(m, "cim"): # those about CIM, and those about any model
print(q.name, q.form, m.query(q.query))
The configuration in general:
from maplib.config import (
CONFIG_GRAPH, VirtualizationConfig, VirtualizationResource, IndexingConfig, QueryCacheConfig,
write_virtualization, remove_virtualization, read_virtualizations,
write_indexing, read_indexing, write_query_cache, read_query_cache, remove_query_cache,
apply_virtualization, apply_properties, apply_query_cache, apply_config,
)
write_virtualization(m, VirtualizationConfig(
iri="urn:maplib:config:virtualization:plant",
database="opcua",
parameters={"endpoint": "opc.tcp://plant:4840/", "username": "maplib"},
secrets={"password": "OPCUA_PASSWORD"},
resources=[VirtualizationResource(name="history", template=HISTORY_TEMPLATE)],
))
write_indexing(m, IndexingConfig(fts=True))
write_query_cache(m, QueryCacheConfig(max_entries=2048, ttl=None))
m.write("model.trig", format="trig") # the configuration is in the file too
m2 = Model(); m2.read("model.trig")
apply_config(m2) # connects (secrets from the environment) and virtualizes
apply_config(m, environ=...) takes the secrets from another mapping than os.environ, and raises
KeyError naming the variable when a secret is missing.
The Model portal¶
The portal’s Graphs › Prefixes page lists, adds, edits and removes the prefixes (/api/config/prefixes),
and the portal formats IRIs and declares the prefixes of its queries with them; its Upload and Build
graph pages can generate prefixes for the data they add. The portal’s Model › Connect page edits the config graph of the portal’s model (the routes under
/api/config) and applies it. Its Query › Common questions page lists, runs, edits and removes the
common questions (/api/config/questions), and offers the bundled ones for the data the model has;
Query › SPARQL and Query › Paths save a query as a common question. Everything else reads the
config graph with SPARQL like any other graph.