Change log¶
v0.23.1¶
Bug fixes¶
maplib Enterprise Edition:
FILTER EXISTSandFILTER NOT EXISTSover virtualized time series (chrontext) failed with “Parts of the query were not evaluated” or, for an EXISTS in the condition of an OPTIONAL, gave wrong results, since v0.23.0. The EXISTS pattern is again evaluated where the virtualized parts of the query are found, and only onceThe source distribution of v0.23.0 left out the vendored
oxttlandsparesultscrates, so the Linux wheels could not be built and are missing from PyPI for v0.23.0; v0.23.1 has wheels for every platform
Performance¶
A small
DELETE DATA(or a DELETE that removes few triples) takes time in proportion to the triples it deletes instead of to the size of their tables: the rows are found through the subject and object indices and cut out in place, instead of an anti-join, a sort and a rebuild of the whole table. Deleting one triple from a model of 1.4M triples takes about 2 ms in memory, hybrid and disk storage, down from 16, 41 and 52 ms, and no longer grows with the modelWriting Turtle, RDF/XML and JSON-LD keeps a bounded number of triples (about 64k) in memory at a time instead of the dictionary strings of the whole graph, and N-Triples, N-Quads and TriG are written from slices of each table. Writing 117 MB of N-Triples as Turtle peaks at 68 MB above the model instead of 587 MB. The output is the same
Reading files with many predicates has no fixed cost per predicate: string columns are built without polars’ minimum reservation per parser thread and predicate, the builders of the threads are appended per table, and deduplication is eager. Reading a file with 2,000 predicates keeps 204 MB instead of 507 MB (139 MB for the same triples with 4 predicates), and is 2.3 to 3.4 times faster
v0.23.0¶
Breaking changes¶
Model.reason(profile, ...)(maplib Enterprise Edition) is nowModel.infer(profile=..., ...), with the same parameters (graph,target_graph,include_graphs,max_violations,max_rows), and still returns aReasoningResult.Model.infertakes eitherrulesetsorprofile, and raises if it gets both or neither, ortarget_graphormax_violationswith rulesets.Model.reasonis removed. In Rust,Model::reasonis unchangedModel.update(update, graph=G)writes toG:Gis the default graph of the update, as withWITH <G>, so triples without aGRAPHin the INSERT and DELETE templates,INSERT DATAandDELETE DATAare inserted into and deleted fromG, and CLEAR DEFAULT clears it. Before, the WHERE clause was evaluated onG, but the triples were inserted into and deleted from the default graph. Updates sent toModel.serve(graph=G)(maplib Enterprise Edition) write toGin the same wayA misspelt magic predicate, e.g.
maplib:shortestpathormaplib:Path(a different case, or for the longer names up to two typos, ofpath,shortestPath,component,bridge,articulationPoint,ftsorvectorSearch), andmaplib:ftswithout a full text index on the queried graph, are errors when no triples use them, saying what to do. They gave no solutions, silently. Other predicates in the maplib namespace (https://datatreehouse.github.io/maplib/vocab#) are ordinary data, e.g. the time series, serves, summary and config terms, and give no solutions when a model has noneSTRSTARTS,STRENDSandCONTAINSwith a second argument from the data that is not anxsd:string(an IRI, a number, an unbound value) give an unbound value, as SPARQL errors do, instead of failing the query. A constant of another type is still an error
Features¶
query(bindings=...)andquery_page(s)take Pythonbool,intandfloatvalues, bound asxsd:boolean,xsd:integerandxsd:doubleliterals. They were ignored, silently, so e.g.FILTER(?v >= ?min)withbindings={"min": 10}had no effect or no results. Values of other types raise an errorModel.update(..., bindings=...)binds variables of the WHERE clauses and of the INSERT and DELETE templates, asquery(bindings=...)does, so parameterised updates need no string templating. In Rust,Model::update_keep_triples,Model::update_with_federation_keep_triplesandTriplestore::update_keep_triplestake the bindings, andquery_processing::bindings::maybe_replace_bindings_updatereplaces them in a parsed updateDESCRIBE queries in
Model.query, which raised “Query type not supported”: the result is the triples whose subject is a described IRI (constant IRIs and the IRIs the described variables are bound to, or every variable forDESCRIBE *), as a CONSTRUCT result. Blank nodes are not described. This is what the SPARQL endpoint ofModel.servegivesREGEXwith a pattern or flags that differ per solution, e.g.REGEX(?label, ?pattern)orREGEX(?v, CONCAT("^", ?prefix)), andSTRBEFOREandSTRAFTERwith a second argument that differs per solution. They raised “The following expression should resolve to a constant literal”. An invalid pattern from the data gives an unbound value for its solutions, while an invalid constant pattern is still an errorGraphSnapshots(..., exclude_graphs=[None])(maplib Enterprise Edition) leaves graphs out of the snapshot (None or""is the default graph): they are never written, restored or loaded from other writers, and writes to them are not pending. The next snapshot drops an existing entry of such a graph from the manifest, andskipped_graphsnames the ones the lastrestore()did not load.on_refresh(callback)is called with the graphs other writers changed, under the write lock, so that an application can keep a graph it derives from them up to date. The writable Model portal (maplib Enterprise Edition) uses them with a Graph portal that rebuilds its default graph from the named graphs (run_portal(rebuild_default_graph=True)): the default graph, a copy of all uploaded data, is no longer written on every write, which rewrote a 27 MB file per upload for a default graph of 1.6M triples (274 ms locally, now 1 ms and 10 kB). Triples written only to the default graph are then not kept across restarts.PORTAL_SNAPSHOT_DEFAULT_GRAPH=1keeps it in the snapshot. In Rust,GraphSnapshotOptions::exclude,GraphSnapshots::set_on_refresh,skipped_graphsandserver::OnRefreshThe writable Model portal (maplib Enterprise Edition) answers a request that may write (POST, PUT, PATCH, DELETE, including SPARQL Update) only once its changes are published to
MODEL_URL, at mostPORTAL_DURABLE_WRITE_TIMEOUT_SECONDS(30) later, so that an acknowledged write survives a crash such as an OOM kill; theX-Maplib-Snapshotheader of the answer says whether they were (published,none,pendingorfailed). Before, writes were published 2 to 7 s after they were answered.PORTAL_DURABLE_WRITES=0answers at onceOne set of default prefixes:
representation::prefixes::WELL_KNOWN_PREFIXES(Rust) is maplib’s only table of default prefixes, read in Python withmaplib.well_known_prefixes()(name to namespace). It merges the two Rust lists there were (the parsing defaults of a model,get_default_prefixes, and the 14 prefixes of template generation) and adds what the Graph portal andmaplib.configused to define themselves: rdf, rdfs, xsd, owl, sh, skos, foaf, ottr, maplib, fx, xyz, def, dcterms, dc, prov, dcat, time, sosa, ssn, vann, void, schema, geo, geof, sf, uom, wgs, qudt, unit, quantitykind, cim, cim16, eu, entsoe, md, dm, opc, di, machinery, robotics, ia, era and mcfg.cimis CIM100 (http://iec.ch/TC57/CIM100#); CIM16 iscim16. Every model now knows all of them when parsing SPARQL, Datalog and OTTR (aPREFIXdeclaration orModel.add_prefixesstill wins), so Turtle, TriG, JSON-LD and RDF/XML written by a model abbreviate these namespaces with them; the RDF/XML and CIM/XML writers also use them for predicate namespaces without a prefix before falling back tons1,ns2, ….maplib.config.WELL_KNOWN_PREFIXES(namespace to name) is read from this table, andPREFERRED_PREFIXESis gone: generated prefixes name CIM100cimand CIM16cim16. The hint for a bare word in a SPARQL syntax error always usesex:as its example prefixPrefixes in the config graph:
maplib.config.PrefixConfig,write_prefix,write_prefixes,read_prefixes,remove_prefixandapply_prefixeskeep the prefixes of a model in its config graph as SHACL prefix declarations (sh:declare,sh:prefix,sh:namespace) of theowl:Ontologymcfg:prefixes, so they travel with the model;apply_configadds them to the model (Model.add_prefixes).generate_prefixes(m, graph=None, max_generated=8)makes prefixes for the data in a graph (or a list of graphs): the usual prefixes of the well-known vocabularies it uses (WELL_KNOWN_PREFIXES: rdf, rdfs, xsd, owl, sh, skos, dcterms, prov, geo, cim, opc, …) and short names for the other namespaces it uses most (prefix_name_for,namespace_of). See “Prefixes” in CONFIG_GRAPH.mdGraph algorithms as magic predicates:
?n maplib:component ?o(connected components, or strongly connected components withmaplib:strong true),?a maplib:bridge ?o(edges whose loss splits their component, withmaplib:objectfor the other end) and?n maplib:articulationPoint ?o(nodes whose loss does), with the outputsmaplib:componentId(numbered from 1 by decreasing size),maplib:componentSize,maplib:cutOff(the nodes cut off) andmaplib:edgePredicate. The edges are given as formaplib:shortestPath(predicate,inversePredicate,anyPredicate,excludePredicate) or, new, by the rows of other patterns of the group withmaplib:edgeSource ?a ; maplib:edgeTarget ?b, so derived edges (e.g. between the substations at the ends of a line) need no materialization.maplib:avoidleaves nodes out of the graph, which is a what-if (open switches, equipment out of service) without changing the model.maplib:countNodesmakes only some nodes count incomponentSizeandcutOff(e.g. customers cut off). A subject bound by the group restricts the rows to those nodes, and nodes among them without edges are components of their own. The algorithms are union-find, iterative Tarjan and an iterative low-link search over the cached integer graph ofmaplib:shortestPath. See “Graph algorithms” in API_USAGE.mdCutting part of a CIM grid out of a model (maplib Enterprise Edition, the
cimpackage):cim.scissors.cut_to_graph(m, interior_equipment_iris, border_equipment_iris, target_graph=..., graphs=[...], normal_topology=True, max_distance=None, context_hops=10)cuts the grid around the interior equipment, stopping at the border equipment, into a new graph and returns counts (members, non-members, context, boundary equipment, triples). It is now SPARQL that maplib evaluates instead of Datalog rules and Python loops: the reachable topology is the connected component of the interior equipment over theTerminal.ConductingEquipmentandTerminal.ConnectivityNodeedges with the border, normally open switches and equipment normally out of service avoided (maplib:componentwithmaplib:avoid), or withmax_distancethe paths of at most 4 x max_distance edges (maplib:shortestPath), and the context is added with one INSERT per step. Thegraphsare read as one dataset (USING/FROM,maplib:DefaultGraphfor the default graph), so the instance data, therdf:typetriples, the profiles and the CIM vocabulary may each be in a graph of their own.cim.cutkeeps its signature and returns the cut as a Model, with the same members, non-members and types as the Datalog rules; it no longer copies theirdt:interiorDistancebookkeeping literals into the cut, andfail_distanceandmax_iterationsare not used any more. The Model portal’s Map sends the same queries to cut a drawn area out of the modelThe serves relations of the
cimpackage (maplib Enterprise Edition):cim.add_serves(m, ..., graphs=[...], target_graph=...)reads the CIM data from several graphs as one (the profiles and therdf:typetriples may each be in a graph of their own) and addsmaplib:normallyServes*andmaplib:actuallyServes*to the target graph. Its helper predicates, which it removes when it is done, are in thecimpackage’s own namespace (https://www.data-treehouse.com/cim/servesHelper_) instead of the maplib namespaceMemory limits per query: a timeout stops a runaway query on time but does not bound its memory, since a join can produce gigabytes of rows per second.
Model.query(max_memory_mb=)andModel.update(max_memory_mb=)stop the query with the newQueryMemoryLimitError(aMaplibException) when the memory used by maplib has grown by more than that many megabytes since the query started. It is checked like the timeout, for every batch of rows a join gives, so the memory goes over the limit by at most the batches in flight.maplib.set_memory_limit(max_query_memory_mb=)sets a default limit for every query and update, also for the queries ofModel.serve(maplib Enterprise Edition), which then answer with an error instead of exhausting the memory of the server. Unlike the limit fromabort_above_mb, which stops queries when the whole process is near it, this limit is per query; the memory counted is that of the whole process, so with queries running at the same time it is approximate. In Rust,CancelToken::with_memory_limit,utils::cancel::set_default_query_memory_limitandCancelled::QueryMemoryLimitThe XPath and XQuery Functions 3.1 math functions
math:pi,math:exp,math:exp10,math:log,math:log10,math:pow,math:sqrt,math:sin,math:cos,math:tan,math:asin,math:acos,math:atanandmath:atan2(namespacehttp://www.w3.org/2005/xpath-functions/math#), as SPARQL function calls. The examples of the specification are run as compliance tests. See “XPath math functions” in API_USAGE.md
Bug fixes¶
EXISTSandNOT EXISTSwith a variable that a solution leaves unbound, e.g. from an OPTIONAL, UNION orVALUESwithUNDEF, treated the unbound value as matching nothing, so?s ex:q ?v OPTIONAL { ?s ex:p ?o } FILTER NOT EXISTS { ?o ex:r ?s }kept solutions whose?ois unbound although?o ex:r ?shas a solution. SPARQL 1.1 substitutes only the variables a solution binds, so an unbound variable stays free in the pattern and matches any value; the solutions are now split by which of the pattern’s variables they leave unbound, and each part is evaluated without them. EXISTS no longer collects its incoming solutions to number them, which kept them all in memory: it is answered for the distinct values of the variables the pattern mentions and joined back on them, within the query’s planUnary minus on an
xsd:doubleorxsd:floatis IEEE negation, so-(0.0e0)and the literal-0.0e0are negative zero, as XPathop:numeric-unary-minussaysmaplib Enterprise Edition: models with predicates of the legacy chrontext vocabulary are encoded under the maplib names they are stored as, when they are inserted and deserialized, so queries that bind the predicate (
?s ?p ?o) or use a zero-length path over them no longer panicGROUP BY a variable that the grouped pattern does not bind, e.g.
SELECT ?country ?product (AVG(?price) AS ?avg) WHERE { ?o ex:product ?product ; ex:price ?price } GROUP BY ?country ?product(BSBM business intelligence query 5), failed because the variable had no column. It now groups as one unbound key, as SPARQL 1.1 (18.5.1) specifies, and such a variable from a sub-select joins with any value of it on the other side, as an unbound variable is compatible with every valueA join, OPTIONAL, MINUS or EXISTS of two patterns that both have IRIs, blank nodes or literals that are not in the model (from VALUES or BIND, e.g.
VALUES ?x { <urn:a> } VALUES ?x { <urn:b> }) compared their codes in two local dictionaries, which may give different values the same code, so different values matched and equal values could fail to. The right side is now re-encoded into a dictionary that extends the left one. This also mademaplib:avoidwith an IRI that is only in the edges from VALUES leave out another node, or noneA FILTER in a nested group, e.g.
{ :x :p ?v . { FILTER(?v = 1) } }, and the condition of an OPTIONAL in a nested group saw the variables of the enclosing group, which SPARQL leaves unbound there (a filter applies to its own group), so they kept solutions they should remove or bound optional variables they should leave unbound. They are now evaluated without the enclosing solutions and joined with them afterwards. Inside EXISTS and LATERAL, where the outer solution is substituted into the pattern, they still see itA property path with a constant of a type its values do not have at one end, e.g.
<a> <b>+ 1.0or a literal object of a path of IRIs, panicked with a SchemaMismatch; it now has no solutionsAn empty
rdf:parseType="Literal"property element in RDF/XML, e.g.<ex:p rdf:parseType="Literal"/>, was a syntax error (“No value found for rdf:XMLLiteral value”); it is now the empty XML literal, as the RDF/XML grammar saysA FILTER over a group with a variable predicate was dropped when the filter pushdown rewrite moved it to a VALUES block, a property path,
GRAPH,SERVICEor triple patterns without a variable predicate, so the query returned the rows it should remove: inSELECT ?n WHERE { ?a ?p ?b . VALUES ?p { <urn:p> } ?a <urn:name> ?n FILTER(?n = "A") }the FILTER was ignored. A filter over the variables of two patterns of a group could also be moved next to one of them and run before the other had bound its variables, which removed every row, and a filter on a variable bound only by an OPTIONAL was moved into it. A filter is now moved only into a pattern that binds all of its variables in every solution, and is otherwise kept above it. The W3C SPARQL 1.0 tests now run every test of the suite (491), including the six that two manifests written as a blank node hid from the runnerArithmetic (
+,-,*) on a variable that holds numbers of many datatypes, e.g.?l + ?ron the objects of a predicate used with 16 numeric datatypes, used memory without bound (more than 3 GB within seconds for one row) while planning the query: the result of each pair of datatypes was combined with the ordering of COALESCE, whose expression grows with the square of the number of pairs (256). The pairs are now combined without ordering, as at most one of them has a value in a rowThe datatype of the result of
+,-and*follows the numeric type promotion of SPARQL (XPathop:numeric-addetc.):xsd:integer<xsd:decimal<xsd:float<xsd:double, where every type derived fromxsd:integercounts asxsd:integer. Soxsd:short + xsd:shortandxsd:int * xsd:longarexsd:integer(they werexsd:shortandxsd:integer, with the 16-bit sum able to overflow),xsd:double + xsd:decimalisxsd:doubleandxsd:float + xsd:decimalisxsd:float(both werexsd:decimal)The W3C SPARQL 1.0 (DAWG) tests and the RDF 1.0 Test Cases (2004) are run by the test suite (
test_w3c_sparql10.py,test_w3c_rdf10.py), with the tests maplib does not pass yet listed as known failuresA join on a variable whose values are of several kinds on both sides that share only one kind, e.g. IRIs and literals on one side and IRIs and blank nodes on the other, matched the values of the other kinds with each other:
"v"joined_:b. This gave wrong rows for OPTIONAL in SPARQL and unsound inferences in Datalog (maplib Enterprise Edition), where new triples are joined with stored ones: witheq-symandeq-transand inModel.infer(profile="owl-rl")andprofile="rdfs-plus", a literal or IRI in a group ofowl:sameAsresources was made the same as an unrelated blank node. The rows of the other kinds are now left out of such joinsmaplib:component,maplib:bridgeandmaplib:articulationPointno longer count an unbound value as a node when the edges,maplib:avoidormaplib:countNodescome from solutions where some rows leave them unbound (e.g. a UNION) and the column holds values of several types: such a row added a node without a value as a component of its own.Security advisories: the RDF/XML parser and the SPARQL XML results parser use quick-xml 0.41, which fixes quadratic run time on duplicate attribute checks (RUSTSEC-2026-0194) and unbounded memory for namespace declarations (RUSTSEC-2026-0195) on untrusted XML.
sparesultsis vendored inlib/sparesultsfor this, likeoxrdfxml. zstd is updated to 0.13.4 (RUSTSEC-2026-0391, RUSTSEC-2026-0398, RUSTSEC-2026-0403 and the unsound zstd-safe advisories). In maplib Enterprise Edition, DuckDB is updated to 1.5.6 (duckdb crate 1.10506.1), whose build no longer uses zip 6 (RUSTSEC-2026-0367, -0368, -0369 and -0429)Comparisons (
<,<=,>,>=) of string literals stored in the model, andMINandMAXof stored strings and IRIs (also in window functions), used the order in which the values were first added to the model instead of codepoint order, e.g.MIN(?label)over"urn:n/10"and"urn:n/1"gave"urn:n/10", andFILTER(?a < ?b)held for ("urn:n/10","urn:n/1").<between string literals from VALUES was false in both directions. Strings computed in the query were not affectedA variable whose name is another variable’s name followed by
l,v,I,BorN(e.g.?oand?ol) failed with “unable to find column” (and panicked inModel.serve) when?oholds values of several types, as with IRIs and literals: the internal columns of?o’s types had the other variable’s namem.add_graph(m, source_graph=..., target_graph=...), with the Model as its own source, raised an error (earlier: hung). It now copies the source graph into the target graph of the ModelA SELECT query without variables, e.g.
SELECT * WHERE { <a> <p> <b> }, gave an empty DataFrame also when the pattern matched. It now gives one row (with no columns) when it matches and none when it does notSELECT (YEAR(?d) AS ?y) ... GROUP BY (YEAR(?d))gave an unbound?y: an expression in the SELECT clause that is a GROUP BY expression is now the value of the group, as in other SPARQL enginesSTRDTwith a lexical form that is not valid for the datatype failed the whole query (“Error parsing literal to target type in STRDT function”); it now leaves that solution’s value unbound, as SPARQL errors doPolars Categorical and Enum columns, e.g. in the results of a Python virtualized database or in a DataFrame given to
map_triples, are read as String columns. They failed with “Unknown datatype Categorical” (a panic for virtualized databases)maplib Enterprise Edition: every wheel includes the
cimpackage. It was only added by the staging step of release builds, so other builds (e.g.maturin develop, the manylinux script and the benchmark image) left it outCasts of a literal of a datatype maplib stores in its dictionary other than
xsd:string, e.g.xsd:string("5389.96"^^bsbm:USD), used the value’s dictionary code instead of its lexical form: that gave"1795", andxsd:float(xsd:string(?price))the code as a number, so BSBM business intelligence query 4, which averages it, gave wrong answers. Such values are now decoded before they are cast. A language-tagged string was cast from the dictionary codes of its value and language:xsd:string("x"@en)gave"{1,0}", andxsd:dateandxsd:dateTimeof one gave a pair of dates in 1970. It now casts toxsd:stringas its lexical form, and every other cast of it is an error, which leaves the result unbound, as SPARQL says. A chain of casts of one such variable is evaluated once per distinct value and gathered back, instead of decoding and parsing every rowFILTER EXISTSandFILTER NOT EXISTSevaluated their pattern once per solution, carrying every column of the solutions through it, and then joined a boolean column back and filtered on it. When the variables the pattern shares with the solutions are bound in all of them, the pattern is now evaluated once per distinct combination of their values, and the solutions are semi or anti joined with the combinations that pass. When the filter is over two patterns that share no variables and the shared variables come from both, as in BSBM business intelligence query 4 (FILTER NOT EXISTS { ?product bsbm:productFeature ?feature }over every feature and every offer), the combinations of the two sides’ values are answered and joined with each side, so the cross product of the two is not built. With the cast fix above, that query’s peak memory on 300 products went from 112 MB to 61 MBA query that timed out during a join much larger than estimated, e.g.
?s ex:kind ?k . ?t ex:kind ?kwith few kinds, returned the timeout error, but the join went on in the background, and when it ended the process was aborted (Polars sent its result to a receiver that was gone). Such a query is now abandoned safely, and joins whose inputs could give many rows (more than 16.7 million multiplied together) run on the streaming engine when the query can be stopped, where a timeout stops the join itself. With 15 000 subjects and a 20 ms timeout the peak memory is 0.53 GB instead of 1.35 GB followed by a crashWith
maplib.set_memory_limit(abort_above_mb=...), a query raises aMemoryErrorwhen the process uses more than 90% of the limit, checked as the solutions of its joins are produced and between the steps of the query, instead of the process being aborted (unless a single step goes over the rest of the limit at once). In Rust, this isutils::cancel::Cancelled::MemoryLimit, andModel.querymakes a cancel token whenever such a limit is set. A timeout limits time, not memory: a join can produce several GB per second before a timeout of 0.5 s firesPatterns of a basic graph pattern that share two or more variables with the patterns before it took memory quadratic in the number of solutions: in
?s ex:kind ?k ; ex:region ?r . ?t ex:kind ?k ; ex:region ?rthe patterns were joined one at a time, so?t ex:kind ?kwas joined on ?k alone, and every subject met every other subject of the same kind before ?r was checked. The results were right. Such patterns are now joined on all the shared variables at once, each first restricted to the values bound so far. With 15 000 subjects, 3 kinds and 1000 regions the query needs 0.12 GB instead of 2 GBA query of
Model.query(..., timeout=...)orModel.serve(maplib Enterprise Edition) that timed out, or whose HTTP client disconnected, went on for 0.5 s to several seconds when it was evaluated with the streaming engine, e.g. a large join with a FILTER, as Polars finishes the batches it is working on. Meanwhile it held the model, so a following update waited for it. It now returns as soon as it is stopped, and the batches are finished on a thread of their ownWith
maplib.set_memory_limit(abort_above_mb=...), a query started right after one that raisedMemoryErroroften raisedMemoryErrortoo, at once: the stopped query returns before the join it was running has ended and freed its memory. A query now first waits for the work of stopped queries to free their memory, while the process uses more than 90% of the limit (until its timeout, at most a minute). The memory limit of a query of its own (max_memory_mb, or the default for every query) counts the growth from the least memory seen since the query started instead of from when it was made, which could be while a stopped query was still freeing hundreds of MB: such a query was then stopped only by the process limit, with aMemoryError, instead of by its own limit. In Rust,utils::cancel::with_tokenwaits, andutils::cancel::Leftovercounts the work that a stopped collect leaves running
Performance¶
Reading Turtle, TriG, N-Triples and N-Quads is faster. The parsers (oxttl) are vendored in
lib/oxttland recognize prefixed names made of ASCII characters without decoding them character by character, check common absolute IRIs with a fast path before the full IRI parser, and remember recently seen IRIs so that repeated predicates and classes are not checked again. Turtle and TriG gain the most; N-Triples and N-Quads gain withchecked=True, the default. The triples, the errors and their positions are unchanged. N-Quads are read in parallel by default, as N-Triples are (parallel=Falsereads them in one thread)Model.map_jsonandModel.map_xmlwith many documents (a list or iterable of strings or paths, eagerly or as a stream) add the triples of up to a thousand documents at once instead of one document at a time, which made mapping many small documents many times slower than mapping the same data as one document. The triples, their IRIs and the uuid namespace of each document are unchanged, and when a document fails, the documents before it are still mapped. In Rust,Triplestore::map_documentsandModel::map_documentsmap several documents at onceVector literals (
maplib:vector) are stored in the dictionary as the bits of their 32 bit floats, packed into seven bits per character, instead of as their lexical text, which takes about two fifths of the memory for typical embeddings. The lexical form is written only when a literal is output (query results, N-Triples, Turtle and the other serializations,STR). Mapping Polars array columns packs the floats directly instead of formatting the lexical forms and storing those, so mapping embeddings no longer holds the text of all of them at once. Only canonical lexical forms (as maplib writes them: the shortest decimal of each component, no spaces) are packed; other lexical forms, e.g. written by hand with spaces or exponents, are stored as they are, so every literal keeps its exact lexical form, and literals with the same lexical form are the same term as before. Vector search and the vector functions (maplib Enterprise Edition) read the floats of packed literals without parsing text. In exchange, outputs that contain many vector literals as text (query results, N-Triples and Turtle) format the floats when they are written, and reading canonical lexical forms (e.g. from N-Triples) packs them, so both take longer than copying text did; serializing and deserializing models takes less time, as the dictionaries are smaller. Serialized models write packed vector literals to their parquet dictionaries, which older versions of maplib read as text; models serialized by older versions are read as before. ORDER BY on vector literals orders them by their stored form, which is still a fixed order but no longer that of the lexical forms. Dictionaries on disk (maplib Enterprise Edition) keep the lexical forms. In Rust,in_memory::vectorshas the packed form and the lexical form conversions,CatMapsInMemory::new_empty_kindcreates a dictionary of vector literals,CatMaps::decode_vectors_batchgives the floats of encoded vector literals, andrepresentation::vectors::vector_columns_to_literalstakes the columns that keep their lexical formsReasoning and Datalog inference into a graph other than the one reasoned over (
Model.infer(profile=...)into the transient graph or atarget_graph, and SHACL rules, maplib Enterprise Edition) leave out the inferred triples that the graph already has while inserting them, by comparing the encoded triples, instead of running a SPARQL query for each predicate of each rule execution. The inferred triples are unchanged. In Rust,Triplestore::add_triples_vec_excludingandinsert_construct_result_excludingadd triples except those a graph already hasModel.inferwith the profiles"owl-rl"or"rdfs-plus"(maplib Enterprise Edition) handlesowl:sameAsbetween individuals with a canonical representative per group of equal resources, found with union-find, instead of copying every triple to every equal resource in each round with SPARQL queries over all triples. The rules run on the triples of the representative, and its triples are copied to the other members of the group once at the end, so large groups no longer multiply the work of every round, andeq-symandeq-transare not run as rules. Groups with blank nodes, literals, classes, properties or other resources that the schema refers to are still replaced in every round. The inferred triples and violations are unchanged; the number of rounds and rule executions in theReasoningResultcan differReading RDF/XML and CIM/XML (
Model.read(..., format="rdf/xml")or"cim/xml") is faster. The parser resolves and checks each element and attribute name once per document instead of at every use, resolves same-document references such asrdf:ID="_..."andrdf:resource="#_..."without the general IRI resolution, and skips the whitespace between elements without copying it. Large documents are split for parallel parsing by scanning for markup instead of tokenizing the whole document first, and CIM/XML reads build their datatype map once instead of for every part of a document. The triples read are unchangedSeveral virtualizations (maplib Enterprise Edition): GROUP BY and aggregates over time series in different resources, e.g. in several virtualized databases, are pushed down into each database again, instead of reading the raw data points and aggregating them in maplib. Each database computes the aggregates of its part of each group, with AVG as a SUM and a COUNT, and maplib combines them per group. COUNT, SUM, AVG, MIN, MAX and SAMPLE are pushed down this way, also with time buckets such as
FloorDateTimeToSecondsInterval. COUNT, SUM and AVG with DISTINCT and GROUP_CONCAT are still evaluated in maplib, and a HAVING or ORDER BY on such a group is evaluated after the aggregates are combinedMapping, updates, inserts and writes of small batches, as in the steps of in-place model streams (
stream=True, in_place=True) andModel.scratch_graphsloops, have a smaller fixed cost. Template expansion, decoding of dictionary codes, CONSTRUCT results that are inserted, splitting of multi-type columns and the N-Triples and N-Quads writers collected their small intermediate results with Polars’ streaming engine, which costs more per collect than the work on a few thousand rows. Results of up to 65,536 rows are now collected with the in-memory engine, as query results already were. The result of a query is also picked up sooner after it is ready: the wait between checks grows by a quarter up to 0.25 ms instead of doubling up to 1 ms. With hybrid and disk storage (maplib Enterprise Edition), writing a graph now decodes each column of each table once instead of once per chunk of the streaming enginemaplib Enterprise Edition: with hybrid and disk storage, the dictionaries on disk keep the strings and codes used recently in memory, so mapping, updates, inserts, queries and writes of small batches look up only the strings and codes they have not used recently. Each lookup otherwise ran DuckDB queries that read the whole dictionary, several times per batch, so the cost of a batch grew with the dictionary rather than with the batch. Ranking codes in string order (when triples are added) decodes them and sorts the strings, instead of sorting the dictionary in DuckDB. Strings added to a dictionary are kept with their new codes, since they are usually looked up right after. The cache holds at most about a million strings with their codes (two generations of 524,288; the older generation is dropped when the newer one is full) and is cleared when the dictionary is compacted.
MAPLIB_DISK_DICTIONARY_CACHEsets the number of strings per generation, and 0 turns the cache off
v0.22.1¶
Breaking changes¶
Model.updatereturns what the update changed instead ofNone, see Features. In Rust,UpdateResulthas achangesfield, andTriplestore::delete_construct_resultanddelete_triples_vectakekeep_triplesand return the deleted triples (triplestore::changes::Deleted) instead of nothingIRI(),URI()andfn:castToIRIgive an unbound value for a string that is not a valid absolute IRI (a relative IRI, or one with spaces or other characters IRIs do not allow), as SPARQL 1.1 specifies for errors, instead of an invalid IRI term (#60)The time series virtualization vocabulary moved from the chrontext namespace (
https://github.com/DataTreehouse/chrontext#) to the maplib vocabulary (https://datatreehouse.github.io/maplib/vocab#, prefixmaplib):maplib:hasTimeseries,maplib:hasDataPoint,maplib:hasTimestamp,maplib:hasValue,maplib:hasResource,maplib:hasExternalId, and the functionsmaplib:FloorDateTimeToSecondsInterval,maplib:dateBin,maplib:DateTimeAsMicros,maplib:MicrosAsDateTime,maplib:DateTimeAsSeconds,maplib:SecondsAsDateTime,maplib:modulus,maplib:time_bucketandmaplib:nestAggregation. None of the names clash with the existing terms of the vocabulary. The bundled templates and data (maplib.ip21, OSIsoft PI, Siemens Insights Hub, CDF, OPC UA, the config graph) use the new IRIs.maplib.config.MAPLIBis the namespace;maplib.config.CHRONTEXTis kept as a deprecated alias of it, andmaplib.config.LEGACY_CHRONTEXTis the old namespace. In Rust,representation::constants::LEGACY_CHRONTEXT_PREFIX_IRIandfrom_legacy_chrontext_irigive the old namespace and its mapping.Deprecated: the old chrontext IRIs are still accepted and read as the maplib IRIs with the same local name: in SPARQL queries and updates (when parsing), in the predicates of data that is added or deserialized (they are stored as the maplib predicates, so queries and exports show the new IRIs), and in the templates of virtualizations. The
chrontextprefix thatmap_osipiandmap_insights_hubadd still points to the old namespace, next tomaplib. IRIs of the old namespace in the object position of data (e.g.sh:pathin a shapes graph) are not rewritten. Custom Python virtualized databases that compare function IRIs, e.g. withhttps://github.com/DataTreehouse/chrontext#FloorDateTimeToSecondsInterval, now get the maplib IRI and must compare with that.AVG over values of one integer type is an
xsd:decimal, as SPARQL defines it, and AVG overxsd:floatvalues is anxsd:float. It kept the type of its input, so the average was right but expressions using it were computed in integers:MAX(?x) - AVG(?x)over 1, 2, 3 and 3 gave 1 instead of 1.25
Features¶
Reasoning profiles (maplib Enterprise Edition):
Model.reason(profile="owl-rl" | "rdfs-plus" | "rdfs", graph=None, target_graph=None, include_graphs=None, max_violations=1000)materializes the entailments of a built-in ruleset with the Datalog engine and returns aReasoningResultwith the number of inferred triples and the violations found."owl-rl"implements the OWL 2 RL/RDF rules of OWL 2 Profiles (section 4.3), includingowl:propertyChainAxiom,owl:hasKey,owl:intersectionOf,owl:unionOf,owl:oneOf, the RL forms ofowl:someValuesFrom,owl:allValuesFromandowl:hasValue, and max (qualified) cardinality 1, but not eq-ref (every termowl:sameAsitself) and the datatype rules other than dt-type1. The rules withfalsein the head (e.g. cax-dw, prp-irp, cls-nothing2, eq-diff1) are checked after inference, and their matches are returned inReasoningResult.violationsas dicts with the rule name, a message and the terms."rdfs-plus"is a cheaper preset in the style of GraphDB’s RDFS-Plus (optimized): the RDFS rules without axiomatic and reflexive triples, withowl:inverseOf,owl:SymmetricProperty,owl:TransitiveProperty,owl:equivalentClass,owl:equivalentProperty,owl:sameAs,owl:FunctionalPropertyandowl:InverseFunctionalProperty."rdfs"implements the RDFS entailment rules of RDF 1.1 Semantics (rdfD2 and rdfs1 to rdfs13 with the axiomatic triples), except rdfs4a and rdfs4b. The inferred triples go to the transient graph by default, or totarget_graph, and triples that are already in the graph are not added again. The schema is compiled into rules with constant predicates (one or two per property, and one per restriction, chain or key) that the Datalog engine evaluates semi-naively, the equality replacement rules are applied with SPARQL whenowl:sameAstriples exist, and both are repeated until the schema no longer changes. The W3C OWL 2 RL test cases, the RDF 1.1 Semantics entailment tests and the SPARQL 1.1 entailment regime tests are included in the tests, seew3c_owl2_rl_known_failures.txt,w3c_rdf_mt_known_failures.txtandw3c_sparql11_entailment_known_failures.txtfor the tests that do not pass. In Rust this isdatalog::reasoning::reasonandModel::reasonR2RML:
Model.map_r2rml(mapping, database=None, tables=None, base_iri=None, graph=None, ignore_data_errors=False)maps relational data with a W3C R2RML mapping. Each logical table (rr:tableNameorrr:sqlQuery) is queried once, from a database inmaplib.virtualization, a DuckDB connection or polars DataFrames given intables, and the term maps are evaluated column by column in parallel, so no rows are handled one by one in Python. It supports constant, column and template term maps, term types, languages, datatypes,rr:class, graph maps and referencing object maps with join conditions, and is tested with the W3C R2RML test casesMicrosoft Fabric (maplib Enterprise Edition):
Model.map_fabric(ontology, lakehouse=None, eventhouse=None, ...)maps a Fabric IQ ontology to OWL, maps the data bound to it from the lakehouse with a generated R2RML mapping (which it returns), and virtualizes its time series, queried in SQL in the lakehouse or in KQL in the eventhouse. The open-source build raisesNotImplementedErrorWindow functions in SPARQL queries (w3c-cg/sparql-dev#47), with the SQL syntax proposed there:
ROW_NUMBER(),RANK(),DENSE_RANK(), andCOUNT,SUM,AVG,MIN,MAX,SAMPLEandGROUP_CONCATfollowed byOVER (PARTITION BY ... ORDER BY ... ROWS BETWEEN ... AND ...), where each part is optional, e.g.BIND(ROW_NUMBER() OVER (PARTITION BY ?group ORDER BY DESC(?value)) AS ?rank). They can be used wherever an expression can: in a SELECT expression, BIND, FILTER and HAVING, and over aggregates in a grouped query (RANK() OVER (ORDER BY DESC(COUNT(*)))). A window function is computed over the solutions of the pattern it is evaluated on: the group of a BIND or FILTER, before it is joined with the rest of the query, or the solutions of the query (after GROUP BY) for SELECT and HAVING, soHAVING (ROW_NUMBER() OVER (PARTITION BY ?g ORDER BY DESC(?v)) <= 3)without GROUP BY selects the top three of each group. As a FILTER applies to its whole group, a window in a BIND also sees the solutions that a FILTER of the same group removes; put the FILTER in a nested group to apply it first.FILTER(?r = 1)on aRANKorDENSE_RANKwith one ordering key keeps the first rows of each partition without ranking them. The semantics are those of SQL: ORDER BY orders as the ORDER BY of SPARQL (unbound values first, values of different types as there), rows with equal keys are peers that get the same RANK and DENSE_RANK, unbound PARTITION BY values form a partition, aggregates without ORDER BY are over the whole partition, aggregates with ORDER BY are running aggregates up to and including the peers of the row, andROWSframes (n PRECEDING,CURRENT ROW,n FOLLOWING,UNBOUNDED) count rows. The aggregates follow the types and numeric promotion of the same aggregates in GROUP BY, also for values of several types. RANGE and GROUPS frames are not supportedORDER BY in GROUP_CONCAT, SAMPLE and custom aggregates (w3c-cg/sparql-dev#9):
GROUP_CONCAT(?x; SEPARATOR=", " ORDER BY ?y DESC(?z))joins the values in that order,SAMPLE(?x ORDER BY DESC(?y))takes the first value in that order. TheORDER BYmay follow the expression or the separator, with or without a;, or precede the separator (GROUP_CONCAT(?x; ORDER BY ?y; SEPARATOR="."), as in the prototype tests of the SPARQL 1.2 community group, which are included in the tests). With DISTINCT, the first occurrence of each value in the order is kept. In Rust, window calls and ordered aggregates are represented in the algebra as calls to reserved functions in the maplib namespace, seespargebra::window, so that the algebra types are unchangedLATERAL(correlated subqueries, SEP-0006): in?s a :T LATERAL { SELECT ?s ?o WHERE { ?s :p ?o } ORDER BY DESC(?o) LIMIT 3 }the group afterLATERALis evaluated for every solution of the pattern before it, with that solution’s variables substituted, so it gives e.g. the top N per entity or an aggregate per entity. A sub-select substitutes only the variables it projects; the others are variables of the sub-select. A variable that is unbound in a left solution is not substituted for it. The right side may not bind a variable of the left side again. The right side is evaluated once for all left solutions together rather than once per solution: ORDER BY with LIMIT and OFFSET slices, GROUP BY groups, and DISTINCT deduplicates for each left solution separately. To aggregate per left solution over a correlated variable that is only used in a FILTER, project it and GROUP BY it, as SPARQL does not allow projecting an ungrouped variable next to an aggregate. The LATERAL tests of Apache Jena and Oxigraph are included in the testsGRAPH blocks in CONSTRUCT templates (w3c/sparql-dev#31), with the syntax of Apache Jena and of INSERT templates:
CONSTRUCT { ?s ex:r ?o GRAPH ex:g { ?s ex:q ?o } GRAPH ?g2 { ?o ex:q ?s } } WHERE { ... }. The graph is an IRI or a variable; with a variable, each solution gives triples in the graph that the variable is bound to, and solutions where it is unbound or not an IRI give none.Model.insertandModelStream.insertadd the triples of each GRAPH block to its graph, and the other triples totarget_graph(the default graph if not given), all in one go, so the WHERE clause is evaluated once instead of once per graph.Model.queryreturns one DataFrame per template triple (and graph) as before, and the DataFrames from GRAPH blocks have an extragraphcolumn with the IRI of the graph;return_format="trig"returns the result as TriG, and"turtle"raises an error when the result has triples in named graphs. The short formCONSTRUCT WHERE { GRAPH ?g { ... } }is also supported. Blank nodes in the template are shared between its GRAPH blocks. In Rust,Query::Constructhas a newgraph_template: Vec<QuadPattern>with the triples of the GRAPH blocks,QueryResultKind::ConstructholdsConstructedTripleswith the graph of each part,template_to_solution_mappingsreturns them in template order, andTriplestore::insert_constructed_triplesinserts them. Jena ARQ’s syntax tests for CONSTRUCT quads are included in the tests, except two that use Jena’s bare{ ... }blocks in templatesRevealing update effects (w3c-cg/sparql-dev#63):
Model.updatereturns, per graph (keyed by IRI,DEFAULT_GRAPHfor the default graph), the number of triples that were really inserted and deleted: triples that were already present are not counted as inserted, and triples that were absent are not counted as deleted. This covers INSERT DATA, DELETE DATA, DELETE/INSERT WHERE, DELETE WHERE, CLEAR and DROP; the changes of the operations of an update are summed. The counts come from the deduplication and deletion that the update does anyway, so they cost nothing extra. Withreturn_triples=Truethe result also holds the inserted and deleted triples as DataFrames per predicate (or SolutionMappings withsolution_mappings=True), which costs an extra pass over the tables that triples are deleted from. The SPARQL endpoint ofModel.serve(maplib Enterprise Edition) sends the counts as JSON ({"graphs": [{"graph": ..., "inserted": ..., "deleted": ...}]}, withnullfor the default graph) to update requests that acceptapplication/json, and still answers others with no content; the Flight update action includes them in its result. In Rust,Model::update_keep_triples,Model::update_with_federation_keep_triplesandTriplestore::update_keep_tripleskeep the triplesVector literals: the datatype
maplib:vector(https://datatreehouse.github.io/maplib/vocab#vector) holds embeddings, with the components as a list of numbers in square brackets, e.g."[0.12, -0.5, 3e-4]"^^maplib:vector. InModel.mapandModel.map_triples, Polars columns of fixed size arrays of numbers (pl.Array(pl.Float32, 384)) are mapped to vector literals. In Rust,representation::vectorshas the datatype and the conversionsVector search (maplib Enterprise Edition): enable it with
IndexingOptions(vectors=True). The vectors of vector literals are indexed by predicate, and the magic predicatemaplib:vectorSearchfinds the triples whose vector literal is nearest to a query vector:?s maplib:vectorSearch [ maplib:vectorQuery "[...]"^^maplib:vector ; maplib:vectorLimit 10 ; maplib:vectorScore ?score ]. The query is a vector literal, a string with the lexical form of one, or an IRI, whose own vector is then used (“more like this”). The optional arguments aremaplib:vectorMetric("cosine", the default,"dot"or"euclidean"),maplib:vectorPredicate(an IRI to search one predicate, or a variable to bind it),maplib:vectorObject(binds the vector literal),maplib:vectorScore(the cosine similarity, dot product or Euclidean distance) andmaplib:vectorExact. A constant subject restricts the search to that subject. The search is exact by default. WithIndexingOptions(vector_ann=True)an approximate nearest neighbour index is also built and kept up to date as triples are added; cosine searches then use it unlessmaplib:vectorExact trueis given, andmaplib:vectorCandidates(default 64) trades speed for recall. The functionsmaplib:cosineSimilarity,maplib:dotProductandmaplib:euclideanDistancecompare two vectors anywhere in a query, e.g. inBIND,FILTERandORDER BY, and use the indexed vectors of stored vector literals when there is an index. The index is rebuilt from the triples on compaction and deserialization, and like the full text search index it is not updated when triples are deletedStreams can iterate on a persistent Model: with
in_place=TrueonModel.map,map_triples,map_default,map_jsonormap_xml(withstream=True), every batch is mapped into the Model itself and the steps run on it, so updates, inserts and rules join the batch with the Model’s other graphs and their results are kept. Batches run one at a time under the Model’s write lock, released between batches. The newModelStream.drop_graphs(graphs)step removes the per-batch graphs, and it still runs when an earlier step of the batch fails. When collected, an in-place stream checks that the graphs named indrop_graphssteps are empty and that write steps only write graphs that a laterdrop_graphsstep drops (or the transient graph).compact_every=Ncompacts the Model and removes unused dictionary entries every N batchesModel.drop_graphs(graphs)removes graphs with their full text and vector indexes, and in disk storage mode deletes the files of their triple tables.with m.scratch_graphs(g1, g2):drops the graphs when the block exits, also on an error. In Rust,Triplestore::drop_graphsOutbound connection policy (against server-side request forgery, SSRF):
maplib.set_outbound_policy(mode="any" | "public" | "allowlist", allow=[...], check=None)is one place where an application decides which hosts maplib may connect to. maplib checks the final scheme, host and port of every connection it makes for a configuration, a query or a server’s answer before connecting: the databases ofmaplib.config(probe,apply_virtualization,apply_config, from the host the database builds, e.g.<account>.snowflakecomputing.comfor a Snowflake account, the OneLake endpoint of a Fabric lakehouse, the cluster URI of Kusto, thebase_urlandtoken_urlof Cognite Data Fusion, the derived token endpoint of Insights Hub), the databases ofmaplib.virtualization,Model.map_osipi,map_pi_points,map_insights_hubandmap_cdf, IP.21, OPC UA,Model.query_externalandModel.query(endpoint=...). The host is resolved and every address it resolves to is checked.publicrefuses loopback, private, link-local and other non-public addresses, andallowlistallows only the hosts inallow;allowtakes host names,*.domain, addresses and CIDR ranges, which are allowed in every mode. Cloud metadata services (169.254.169.254, fd00:ec2::254, 100.100.100.200, 169.254.170.2, 192.0.0.192, metadata.google.internal) are refused in every mode, also by default and when listed. Redirects are checked by the clients maplib controls (urllib, requests and reqwest, whose DNS resolver also checks the addresses it connects to, against DNS rebinding); the absolute URLs of PI Web API links are checked too.check(scheme, host, port)is called after these checks for connections made from Python. Refusals raisemaplib.OutboundError, aPermissionError.maplib.outboundalso hascheck_url,check_host,check_connection,connection_targets,urlopenandguard_sessionfor applications and database classes of their own. In Rust, the newoutboundcrate (outbound::check,check_url,set_policy, and with thereqwestfeatureguard_client)Keyset pagination (sparql-dev#49):
Model.query_pages(query, page_size=10000)iterates over the pages of a SELECT query, andModel.query_page(query, page_size, page_token)gives one page and the token of the next. Each page starts after the last solution of the previous page, instead of skipping the solutions before it with OFFSET, so a page deep into a large result costs no more than the first. The solutions are ordered by the ORDER BY of the query and then by the projected variables, so the order is total and the pages put together are the result of the query without duplicates or gaps (a query without ORDER BY is ordered by its projected variables). Duplicate solutions are counted across pages, and the LIMIT and OFFSET of the query apply to the pages as a whole. Pages are evaluated one at a time and see the model as it is then: no solution is returned twice, solutions in the result throughout are returned exactly once, removed solutions are not returned once removed, and added solutions are returned when they sort after the last solution returned. A page token (QueryPages.token) is a string safe in URLs that holds the position, not the results, so it can be kept and used later or in another process; it is refused for another query. SELECT DISTINCT can be paged when its ORDER BY uses only projected variables. Withreturn_format="json", pages are SPARQL 1.1 JSON results. Queries of models with virtualized databases can not be paged yet. In Rust,Model::query_pageandTriplestore::query_pagetake aPageCursor(triplestore::sparql::paging)Model.serve()(maplib Enterprise Edition) takespage_sizeandpage_tokenparameters on the SPARQL query endpoints: a SELECT query gives that page of its solutions, and the token of the next page in theNext-Page-Tokenresponse header, which is absent on the last page (exposed to CORS clients).limitandoffsetparameters apply a LIMIT and an OFFSET after those of the query (sparql-dev#91). Pages are not kept in the query result cache, whose key includeslimitandoffset.Statistical aggregates from Apache Jena ARQ:
agg:stdev,agg:stdev_samp,agg:stdev_pop,agg:variance,agg:var_sampandagg:var_popinhttp://jena.apache.org/ARQ/function/aggregate#, also available inhttp://jena.apache.org/ARQ/function#as in Jena. They return anxsd:double, takeDISTINCT, and can be used in HAVING and as window functions (agg:stdev(?x) OVER (PARTITION BY ?g ORDER BY ?t ROWS BETWEEN 9 PRECEDING AND CURRENT ROW)). A group with a non-numeric value, with no values, or with a single value for the sample statistics gives an unbound result, as in Jena. Jena’s tests for them (TestStatisticsAggregates) are ported, and the results are checked against Python’sstatisticsmodule. See “Statistical aggregates” in API_USAGE.mdSHACL entailment (maplib Enterprise Edition): a shapes graph’s
sh:entailmentis honoured. For RDFS, validation first materializes the RDFS entailments withModel.reason’s"rdfs"profile, and for the OWL regimes (OWL 2 RDF-Based and Direct semantics) the"owl-rl"profile, into a temporary graph that rules and constraints read, which is removed afterwards. Simple entailment,sh:RulesEntailment,sh:Rulesandsh:RuleSetneed no reasoning, and other regimes (D, RIF, …) make validation fail instead of being ignored. Shapes graphs that declare RDFS entailment may now report more results, e.g. forrdfs:Resource
Bug fixes¶
ORDER BY puts blank nodes before IRIs, as SPARQL 1.1 specifies (unbound values, then blank nodes, then IRIs, then literals); IRIs came first. This is the order of the W3C sort tests, which are included in the tests
Model.updatereports the triples that LOAD inserts, in the graphs they went to; they were left out of the changesCONSTRUCT templates can have
{ ... }blocks of default graph triples next to GRAPH blocks, as in Apache Jena:CONSTRUCT { { ?s ?p ?o } GRAPH ex:g { ?s ?p ?o } } WHERE { ... }, also in the short formCONSTRUCT WHERE { { ... } }. INSERT and DELETE templates are unchanged, as SPARQL 1.1 does not allow these blocks thereThe SPARQL Update operations
CLEAR,DROP,COPYandMOVEwork in hybrid and disk storage (with a storage folder), where they raised “Clearing or dropping graphs is not supported with a storage folder”, and so doesModel.truncate_graph, which panicked. In disk storage the parquet files of the removed tables are deleted from the storage folder; files a model was deserialized from are never touched. The W3C SPARQL 1.1 Update tests now run in all three storage modesVector search (maplib Enterprise Edition) and the vector functions now agree with numpy and scipy on edge cases. Euclidean search is computed from the component differences, so near-duplicate vectors far from the origin are ranked correctly and a stored vector is at distance 0 from itself. Zero vectors and vectors with NaN or infinite components have no cosine similarity, so cosine searches leave them out (they were returned with a score of 0), and a zero query vector finds nothing. Vectors at an infinite distance or dot product are no longer dropped when fewer than
maplib:vectorLimitresults are found.maplib:cosineSimilarity,maplib:dotProductandmaplib:euclideanDistancecompute in 64 bit floats, so very small or large components no longer underflow or overflow. Tests now check the functions against scipy, exact search against numpy and faiss, and the recall of the approximate index against hnswlib with the same parametersThe FILTER of an OPTIONAL can use variables that only the pattern before the OPTIONAL binds, as in
?s :min ?min OPTIONAL { ?s :v ?v FILTER(?v > ?min) }. It used to be evaluated without those variables, so no optional match passed it and the variables of the OPTIONAL were always unboundA SHACL SPARQL rule (maplib Enterprise Edition) whose
sh:constructhas GRAPH blocks in its template raises an error, as Datalog rules do. The graphs were ignored and the triples went to the inferences graphGraphSnapshots.restore()(maplib Enterprise Edition, the writable Model portal) no longer adds the snapshot’s copy of a graph to a graph of the same name that the model already has, e.g. shapes loaded by the entry point before restoring: the blank nodes of such graphs were duplicated on every start and every new replica. The model’s own copy is kept, and when it differs from the snapshot’s, the next snapshot publishes itAfter
GraphSnapshots.restore(), the first snapshot no longer writes every graph again: graphs that were restored, or loaded before restoring, count as published, and a write that leaves their content unchanged (building an index, as the Model portal does at start, marks every graph as written) is not publishedThe background loop of
GraphSnapshots(and the writable Model portal) backs off when the store fails, e.g. a missing bucket or container, or auth or network errors: it waitsbackoff_min(1 s), doubling with each failure in a row up tobackoff_max(300 s), with jitter, instead of trying about once a second, and logs one warning per try on themaplib.graph_snapshotslogger. A success resets it, andstop()(the snapshot on SIGTERM) tries at onceSAMPLE skips unbound values, so it gives a value whenever the group has one, also with ORDER BY (
SAMPLE(?x ORDER BY ?y)is the first bound value in that order)AVG of no values (a group where the expression is never bound) is 0, as SPARQL 1.1 defines it, instead of unbound; also for AVG over a window
ORDER BY orders numbers of different datatypes by value, as the
<operator compares them, instead of grouping them by datatype first (1before1.5before2for an integer, a decimal and an integer). Numbers that a double cannot tell apart keep their exact orderCOUNT(?x)andCOUNT(DISTINCT ?x)skip unbound values;COUNT(DISTINCT ?x)counted them as one extra valueFILTER NOT EXISTSandFILTER EXISTSgive the right result in groups with a variable predicate pattern: the filter pushdown rewrite moved them below the patterns that bind their variables, so they were evaluated without those bindingsJSON-LD output keeps IRIs whose local part would start with
/uncompacted:ex:/awould be read back as a different, absolute IRImaplib.storagegives the same version token, the unquoted ETag, fromhead,get,listandputon every backend, so a token from any of them works withput(if_match=...)ModelStream.collect()hands its blank node counters back to the Model, also when it fails, so a later stream or mapping from the same Model no longer reuses blank node labels that an earlier stream wrote (two streams of a template with a blank node both wrote_:b_l0_p0_r0, which merged distinct nodes when the outputs were combined)A variable that an OPTIONAL or a UNION branch may leave unbound, and that other patterns of the group also use, is joined as compatible with any value, as SPARQL specifies; such solutions were lost or joined wrongly when the OPTIONAL or UNION came first in its group. The W3C tests
service4aandvalues7passmaplib:pathandmaplib:shortestPathpatterns are joined after the other patterns of their group, so they get their start and target nodes from them wherever they are writtenParsing
REGEX,SUBSTRandREPLACEtakes time linear in their nesting depth; nested calls (e.g. the about 20 nestedREPLACEcalls thatmap_csvwgenerates for some tables) took exponential time and could overflow the stack.map_csvwalso runs on a large stackA
Modelstays usable after a call panics: every Python method recovers the model’s lock instead of raising on a poisoned lockVirtualized time series queries (maplib Enterprise Edition) give the same results as the equivalent queries over materialized data in more cases: subqueries in joins (with LIMIT, DISTINCT or hidden variables), GROUP BY on a virtualized value such as the timestamp, aggregates without GROUP BY over no rows (one row, e.g. a count of 0), several data point patterns of one time series, and data points written without their time series. A differential test against materialized data now passes for all of its queries
SHACL reports (maplib Enterprise Edition) hold the whole
sh:resultPathof a result with a complex path (a sequence, alternative, inverse or zero-or-more path, i.e. a blank node of the shapes graph): its triples are copied from the shapes graph into the report graph, where the path was a dangling blank nodeGRAPH ?g { ... }on a model or dataset without named graphs keeps?gin scope, soGROUP BY ?g, ORDER BY or projecting?ggive an empty result instead of an error that?gis not defined
v0.22.0¶
Breaking changes¶
xsd:dateTimeandxsd:dateTimeStampvalues are all stored as UTC instants (PolarsDatetime("us", "UTC")), and the time zone offset of a literal stays part of its type. Naive datetime columns of mapped DataFrames used to be stored as naive datetimes, so comparing, joining or unioning them with other dateTimes failed (“could not evaluate ‘>=’ comparison between Datetime(‘μs’) and Datetime(‘μs’,’UTC’)”, “failed to determine supertype”, or a panic when such a CONSTRUCT or UPDATE wrote to a named graph). They are now read as UTC wall clock times and still have no time zone: they serialize as before ("2026-11-14T06:12:00"^^xsd:dateTime), but query results return them as UTC-aware datetimes (datetime[μs, UTC],tzinfo=timezone.utcin Python) instead of naive ones, as dateTimes read from RDF files already were. Time zone aware columns in other time zones are converted to UTC. Models serialized by earlier versions load and serialize the same, and their naive dateTimes are read as UTC wall clock times, so they also come back UTC-aware and now compare with other dateTimes. Casting a string toxsd:dateTimeacceptsZand offsets such as+02:00, which used to be an error, and gives the UTC instant without a time zone in its type (xsd:dateTime("2026-11-14T08:12:00+02:00")is"2026-11-14T06:12:00"^^xsd:dateTime), so it equals dateTimes without a time zone, not those with one. Strings that are not dateTimes give unbound instead of an error.chrontext:MicrosAsDateTime,chrontext:SecondsAsDateTimeandchrontext:FloorDateTimeToSecondsIntervalreturn UTC-aware datetimesSHACL rules (maplib Enterprise Edition) follow SHACL 1.2 Inference Rules: rules that no shape links to with
sh:rule(global rules) also run,sh:conditionrestricts the focus nodes to those conforming to the condition shapes,sh:deactivatedrules are skipped, and rules run in ascendingsh:layerandsh:order, where rules with the same order do not see each other’s new triples within an iteration. Rule types other thansh:SPARQLRule,sh:TripleRuleand SPARQL rule templates, and anysh:ruleProcessor, are errors. Therule_logof the validation report lists the layer instead of the number of targetsTransient triples are removed. Triples that should be available for querying, validation and reasoning, but not written, now go in a reserved named graph,
maplib.TRANSIENT_GRAPH(https://datatreehouse.github.io/maplib/vocab#TransientGraph). Thetransientparameter ofmap_json,map_xml,read,readsandinsertis removed: passgraph=TRANSIENT_GRAPH(ortarget_graph=TRANSIENT_GRAPHforinsert) insteadinclude_transientis replaced byinclude_graphs: Optional[List[str]]inquery,update,insert,infer,get_predicate,get_predicate_irisand the correspondingModelStreamsteps. The included graphs are merged into the queried graph, as with FROM. Forquery,update,insertandinfer,Noneincludes the transient graph (asinclude_transient=Truedid) and[]includes nothing; forget_predicateandget_predicate_iris,Noneincludes nothing.GRAPH ?granges over the transient graph as over any other named graph, andGRAPH <g>no longer sees transient triples added togRDFS inference (
infer_rdfs) and SHACL rules (without aninferences_graphequal to the data graph) put their inferred triples in the transient graph. Writing all graphs as TriG or N-Quads leaves out the transient graph; write it withgraph=TRANSIENT_GRAPHIn Rust,
QuerySettings::include_transientis replaced byinclude_graphs: Vec<NamedGraph>, and thetransientarguments of the triplestore and model methods are removed.NamedGraph::transient()gives the transient graphThe
messagecolumn ofValidationReport.results()anddetails()is a list of strings, one persh:messageof the shape or SPARQL constraint ("value"@langfor language tagged strings, the value for plain strings), and null when there are none. Use e.g..explode("message")to get one row per messageModel.serializewrites a single file by default, and the default path ofModel.serializeandModel.deserializeis./model.maplibinstead of./serialized_triples. Passsingle_file=Falseto write a folder as beforeIn Rust,
Model::expand_default,expand_default_first_batchandexpand_default_batchtake the key columns as&strand&[String], and a nested (List or Struct) key column is an error instead of a panicThe
max_resultsparameter ofModel.inferandModelStream.infer, and themax_shape_constraint_resultsparameter ofModel.validate, are removed: the memory limit (maplib.set_memory_limit) bounds inference and validation instead.max_iterationsandmax_rowsare unchanged. In Rust,Model::inferandModel::validateno longer take them
Features¶
Model.generate_template(data, primary_key_column)is a dry run ofModel.map_default: it returns the default template thatmap_defaultwould map with, as a stOTTR string, without mapping the data or adding the template to the model. Review or edit it and map withm.map(template, data).map_defaultgenerates its template with the same code (maplib::model::default::generate_default_templatein Rust), so the two agreemaplib.GraphSnapshots(model, url, ...)(maplib Enterprise Edition) keeps a model in storage as one HDT file per named graph and amanifest.json, which several writers can share: only the graphs that changed are written (found from the model’s own writes), the manifest is published with a conditional put, and a writer that loses the put merges the graphs both sides changed three-way at the triple level.restore(),snapshot(),refresh(),gc()and a background loop withstart()/stop().Model.serve(snapshot_format="graphs", refresh_interval=10)serves a model kept this way. The writable Model portal uses it, in place of its Python implementation. In Rust,VersionTable::changed_graphs_sinceandgraph_stampgive the graphs written after a stamp, andRemoteError::is_not_foundandis_precondition_failedclassify object storage errorsModel.serve(snapshot_url=..., snapshot_interval=900, storage_options=None)(maplib Enterprise Edition) snapshots the served model withModel.serialize: everysnapshot_intervalseconds if it changed since the last snapshot (through the endpoints or from Python), and when the server stops. An unchanged model is never written.python -m maplib.serveserves a serialized model configured with environment variables (MODEL_URL,ALLOW_UPDATE,SNAPSHOT_URL, …) and takes the last snapshot on SIGTERM; the cloud deploy recipe (now in maplib Enterprise Edition) runs itQuery timeouts and cancellation:
Model.query(timeout=...)andModel.update(timeout=...)(seconds) stop the query when its time is up and raisemaplib.QueryTimeoutError(aMaplibException). Queries are checked between operators, after every join while it produces rows (also on the streaming engine), while waiting for Polars collects, and in path searches and property paths, and while results are formatted (Model.query) or serialized (the server); a query with a timeout and a join that may give many rows is collected with the streaming engine, which can stop a running join. So a query that would run for hours or exhaust memory, such as a cross join of three triple patterns, stops within a fraction of a second of its timeout. Other queries running at the same time are not affected (Polars’ own abort, used for Ctrl-C, stops every query in the process). An update is checked before each operation and while its WHERE clause is evaluated, and its writes are never stopped halfway: operations of the request done before the timeout stay done.Model.serve(query_timeout=...)(maplib Enterprise Edition) limits every query and update through the endpoints, counted from when the request arrives, and requests can ask for less with thetimeoutparameter (seconds; URL or form parameter, theX-Maplib-Timeoutheader, or"timeout"in Arrow API and Flight commands and Flight update actions), or more up tomax_query_timeout. A query that times out is answered with 503 Service Unavailable (Flight:DEADLINE_EXCEEDED), and a query whose client disconnects is cancelled, so it no longer runs on after its client gave up.max_concurrent(64 by default) limits the operations on the model that run at once; waiting for a turn counts towards the timeout.Model.query(endpoint=..., timeout=...)sends the timeout to the server and raisesQueryTimeoutErrorwhen the server stops the query. In Rust,utils::cancel::CancelTokenis a deadline and a cancel flag, made current for a thread withutils::cancel::with_token;SparqlError::CancelledandInterruptableCollectError::Cancelledreport a stopped query, andServerOptionshasquery_timeout,max_query_timeoutandmax_concurrentQuery result cache:
Model.set_query_cache(QueryCacheOptions(...))keeps the results ofModel.queryand of the SPARQL protocol endpoints ofModel.serve, which share it. A result is kept until a write changes the triples it can read: every write records the named graphs and predicates it changed, and every query reads a set of (graph, predicate) pairs found from its algebra, so writes to other graphs or other predicates keep it. Variable predicates, negated property sets, zero-length paths between variables, full text search and DESCRIBE read every predicate of their graphs, andGRAPH ?greads a predicate in every graph. Changes of the prefixes, the indexing options, user defined functions and virtualizations invalidate everything. Queries calling NOW(), RAND(), UUID(), STRUUID(), BNODE() or user defined functions, withparameters, withdebugor reading virtualized databases (unlesscache_virtualized=True) are never cached. The options are on or off, the largest number of results, their total size, the largest result kept, a ttl, the shortest run time worth keeping and whether to keep results over virtualized databases, and can be changed at any time. Cached results are answered without waiting for a write that runs elsewhere in the model, and the server answers them without holding the GIL.Model.query(cache=False)skips the cache,Model.query_cache_stats()gives its counts (hits, misses, evictions, invalidations, size) andModel.clear_query_cache()empties it. The cache is off by default.Model.serve(query_cache=...)sets it for the server, which reportsX-Maplib-Cache: hit | miss | bypasson each response, acachefield on each query log entry, cache totals and hit rates in/query-logand the cache’s counts in/status. In Rust,Triplestore::query_cache(triplestore::query_cache::QueryCache),Model::cache_plan,Model::add_prefixesandServerOptions::query_cacheSHACL rules (maplib Enterprise Edition) implement SHACL Advanced Features and SHACL 1.2 Inference Rules:
sh:SPARQLRule(shape rules with$thispre-bound to each focus node, and global rules),sh:TripleRulewith node expressions (focus node, constants, lists, path values withshnex:pathValuesorsh:path,exists,if,distinct,intersection,union/concat,remove,filterShape,instancesOf,count,sum,min,maxthesparql:functions and SHACL-AFsh:SPARQLFunctioncalls), SPARQL rule templates (sh:SPARQLRuleTemplatewithsh:parameter),sh:condition,sh:deactivated,sh:order,sh:layer,sh:runOnce, rule sets (sh:RuleSet,sh:hasRule,sh:includesRuleSet) selected withsh:entailment(sh:RulesEntailmentruns all rules, as does a shapes graph withoutsh:entailment), and prefixes declared at the shapes graph. Shapes typedsh:ShapeClassare node shapes with an implicit class target. Not supported:sh:expectedPredicate, temporary triples andsh:sourceRule, which need RDF 1.2 triple terms, and the order dependent node expressions (limit,offset,orderBy,findFirst,flatMap,matchAll,nodesMatching). The W3C SHACL 1.2 inference rules tests and the TopBraid DASH SHACL-AF rules tests are included in the tests. Inferred triples that are already in the data graph are no longer reported as inferencesSPARQL-RL 1.2 (maplib Enterprise Edition):
Model.infertakes rule sets in the SPARQL-RL syntax of the W3C Data Shapes Working Group (RULE { ... } WHERE { ... },DATA { ... },NOT,NOT DATA,WHERE DATA,SET(?x := ...),GROUP BY ... { ... } [ ?n := COUNT(*) ]), checks their well-formedness and stratification, and evaluates them stratum by stratum as the specification defines.IMPORTSis not supported. The W3C SPARQL-RL syntax, well-formedness, stratification and evaluation tests are included in the testsObject storage:
Model.read,read_template,write,write_cim_xml,serializeanddeserializetake URLs of Google Cloud Storage (gs://bucket/path), Azure Blob Storage (az://container/path,abfss://container@account.dfs.core.windows.net/path,https://account.blob.core.windows.net/container/path), Amazon S3 (s3://), HTTP(S) andfile://, as well as local paths. Credentials are found as the cloud SDKs find them (environment variables, the metadata server on Google Cloud, managed and workload identities on Azure), or are given with the newstorage_optionsargument, with the option names of Polars’storage_options. Files that are read are downloaded to a temporary file first, with concurrent range requests for large files. Files that are written are uploaded as they are written, in concurrent parts, and the object only appears when the write completesmaplib.storagestores raw bytes in object storage and local files, with the URLs, credentials andstorage_optionsofModel.readandModel.write:put(url, data, if_match=None, if_none_match=False)returns a version token (the ETag, with the object generation on Google Cloud Storage),get(url)returns the bytes and their token, andhead,exists,deleteandlist(prefix_url)(names, sizes, times, tokens) are there for garbage collection.putwithif_matchreplaces the object only if it is still at that version and raisesstorage.PreconditionFailedotherwise, andif_none_match=Trueraisesstorage.AlreadyExistsif there is an object, so that several writers can publish a manifest with optimistic concurrency. S3, Google Cloud Storage (as generation preconditions; the version fromlisthas no generation there, so use the token ofgetorheadforif_match), Azure and the local file system support both; for local filesif_matchis emulated with a lock file, which serializes the writers on one machine only. Plain HTTP raisesstorage.NotSupported. A missing object raisesstorage.NotFound, aFileNotFoundError. In Rust,RemoteLocation::put_with,get_with_version,info,list_infoandparse_path_or_urlModel.serialize(url)writes a single file locally and uploads it.Model.serialize(url, single_file=False)writes a folder to a new generation under the URL (gen-<time>-<id>/) and then publishes it by replacing the objectLATESTwith its name, with a conditional put: readers see the previous or the new model whole, and if another writer published to the URL in the meantime, the serialization fails instead of replacing it. The current and the previous generation are keptTriple tables in object storage (maplib Enterprise Edition):
Model.deserialize(url, storage="disk", storage_folder=...)on a folder serialized to object storage does not download the triple tables, only the dictionaries and the sparse indices. Each table is downloaded to the storage folder the first time it is read. Withcache_size(orMAPLIB_REMOTE_CACHE_BYTES), the least recently read downloaded tables are removed when they take more space, and downloaded again when read. Tables read in the last five minutes (MAPLIB_REMOTE_CACHE_GRACE_SECONDS) are kept, as running queries may read them. So a disk mode model can be served from a VM or container with a local disk smaller than the model. Triples added afterwards stay local until the model is serialized again. Serializations record the number of triples of each table for this; folders written by earlier versions are downloaded when deserializedIn Rust,
file_io::remotehasRemoteLocation,InputandOutputfor object storage,SerializationSource,SerializationTargetandSerializedFilehaveRemotevariants,TriplestoreandModelhaveserialize_triples_remoteanddeserialize_triples_remote, andTriples::from_serializationtakes the number of triples when it is known. The metadata of a serialization folder is written after its other filesThe config graph configures the query result cache of the model (
mcfg:QueryCache, seeCONFIG_GRAPH.md):maplib.config.QueryCacheConfigwithwrite_query_cache,read_query_cache,remove_query_cacheandapply_query_cachekeeps whether it is enabled, its limits (entries, bytes, the largest result kept), a ttl, the shortest run time worth keeping and whether results over virtualized databases are kept.apply_configsets it withModel.set_query_cachemaplib.configand the Model portal’s Connect page configure InfluxDB: InfluxDB 3 (influxdb,mcfg:InfluxDBDatabase, resources are SQL over measurements) and InfluxDB 2 / 1.8 with Flux (influxdb2,mcfg:InfluxDB2Database, resources are measurements: the options or connection parametersmeasurement,idandfield). The token is a secret. Probing without SQL lists the measurementsCSV on the Web (#26):
Model.map_csvw(source, metadata=None, graph=None, minimal=False, iri_mappings=None)converts tabular data to RDF as specified by the W3C CSVW recommendations (tabular data model, metadata, csv2rdf), in standard or minimal mode. The source is a metadata document or a CSV file, for which the metadata is located as in the specification (with the site-wide configuration/.well-known/csvmif it is found, else<file>-metadata.jsonandcsv-metadata.json) or taken from the header row. Dialects, inherited properties, datatypes with formats (numbers, booleans, dates and times, regular expressions) and constraints, lists (ordered asrdf:List), virtual and suppressed columns, notes and common properties, row titles and foreign key validation are supported. The tables are added to temporary graphs using Facade-X, and a SPARQL Update generated from the metadata builds the output (URI templates are compiled to SPARQL expressions) and drops the temporary graphs. The update is returned. Warnings (invalid metadata properties, invalid cell values and values that maplib can not store with their datatype, such as dates with time zones, which are output as strings) are issued as Python warnings. Nothing is fetched over the network:iri_mappingsmaps IRI prefixes to local folders. The W3C CSVW RDF test suite is included in the tests, seew3c_csvw_known_failures.txtfor the tests that do not pass. In Rust this is the newcsvwcrate andModel::map_csvwStreaming (#87):
Model.map,map_triples,map_default,map_jsonandmap_xmltakestream=Trueand then return a lazyModelStream, e.g.m.map(template, lf, stream=True).update(...).infer(...).insert(...).write("out.nt").write("out.ttl", format="turtle").collect(). Oncollect(), each batch of the input is mapped into a fresh in-memory model with the templates, prefixes and UDFs of the model, the steps run on it in order, thewritesteps append it to their files (N-Triples, Turtle with prefixes declared once, or N-Quads), and the batch is dropped, so memory use depends on the batch size and not on the size of the input. The model’s own triples are not changed. Steps only see the triples of the current batch. Blank node labels stay unique across batchesModel.map_jsonandModel.map_xmlalso take an iterable of documents (strings or paths), e.g. a generator over the lines of a JSON Lines file, mapped one document at a time. With auuid_namespace, document number i uses the namespace<uuid_namespace>/<i>so that documents do not get the same IRIs. Withstream=True,documents_per_batchdocuments are mapped into each batch; mapping them intograph=TRANSIENT_GRAPHand inserting a CONSTRUCT query writes only the constructed triplesStorage modes:
Model(storage="memory" | "hybrid" | "disk", storage_folder=...), also forModel.deserialize."memory"keeps everything in memory,"hybrid"stores the dictionaries of IRIs and literals on disk (whatstorage_folderdid before), and the new"disk"mode also stores the triple tables on disk as parquet files in<storage_folder>/maplib_storage/triples. Files of triple tables that are replaced (by compaction or deletes) or whose model is dropped are removed. Withoutstorage, astorage_folderstill means hybrid storage, so existing code is unaffected. Hybrid and disk storage require maplib Enterprise Edition. In Rust,Model::new,Model::deserialize_triplesandTriplestore::newtake aStorageOptionsinstead of the storage folderFull text search (maplib Enterprise Edition) uses DuckDB instead of tantivy. The index stores the dictionary codes of subjects, predicates and objects, and searches the strings where the dictionaries keep them: with hybrid and disk storage it joins against the on-disk dictionary tables, so no copy of the strings is made, and with memory storage it uses an in-memory DuckDB database. Matching is unchanged: case insensitive, 1.0 for a match of the whole value and 0.75 for a substring match.
fts_path(inIndexingOptionsandModel.explore) is no longer used, as the index is stored with the dictionariesGeoSPARQL (maplib Enterprise Edition): enable it with
IndexingOptions(geo=True). Geometry literals (geo:wktLiteral,geo:wkbLiteral,geo:geoJSONLiteral) are indexed spatially, and the GeoSPARQL functions are supported: the simple features, Egenhofer and RCC8 relations, measurements with units of measure, constructions, properties and serializations, across coordinate reference systems. Spatial filters and spatial joins use the index.geof:relate, the query rewrite extension and the aggregate functions are not supported yetPretty Turtle output only declares the prefixes whose namespace starts a predicate, a datatype or an IRI in the model, sorted by name, instead of all known prefixes. Each prefix is checked with one lookup in the sorted IRI dictionary. The dictionary is shared by all graphs, so a prefix used only in another graph is still declared; with hybrid and disk storage all prefixes are declared as before
Model.serializewrites the dictionaries of IRIs and literals as parquet files (cats/<id>.parquet, columnsstrandcat, in string order) instead of DuckDB databases, so a serialized model is parquet throughout and serialization no longer needs maplib Enterprise Edition.Model.serialize(path, raw_db=True)(Enterprise Edition) writes them as DuckDB databases as before: larger and slower to write, but deserializing into hybrid or disk storage uses them in place.Model.deserializereads both, also folders written by earlier versions. In Rust,serialize_triplestakesraw_db: boolModel.map_cdf(client)(maplib Enterprise Edition) maps the data model of a Cognite Data Fusion project to RDF using a cognite-sdkCogniteClient: the data models and views (as classes, withrdfs:subClassOffor the views they implement), the container properties the views map (as predicates, shared by the views that map them), and the nodes and edges that have data in the views. Datapoints are not read: thecdf_cdm:CogniteTimeSeriesnodes are linked to chrontext resources, so?ts chrontext:hasDataPoint ?dpfetches their datapoints from CDF throughVirtualizedDatabase.cognitewhen a query needs them.data_modelsandspacesselect what to mapModel.serialize(path, single_file=True)(the default) writes the model as one file, by convention with the extension.maplib, though any name works, also withraw_db=True. An interrupted serialization never leaves a partial file atpath.Model.deserializedetects whetherpathis a file or a folder. A model deserialized from a file does not use the file afterwards. In Rust,serialize_triplestakessingle_file: bool, andCats::deserialize_catstakes aSerializationSourceArithmetic and comparisons on durations, dates, dateTimes and times, as defined by the XPath operators that SPARQL refers to:
xsd:yearMonthDurationandxsd:dayTimeDurationvalues can be added, subtracted, multiplied and divided by numbers, and divided by each other; they can be added to and subtracted fromxsd:dateTime,xsd:dateand (dayTimeDurations only)xsd:timevalues, and dateTimes, dates and times can be subtracted from each other, giving a dayTimeDuration. Durations are compared by value:=holds between durations of any duration datatype with the same months and seconds (e.g."P1D"^^xsd:dayTimeDuration = "PT24H"^^xsd:dayTimeDuration), and yearMonthDurations, dayTimeDurations and times can be ordered with<,<=,>and>=. Casting toxsd:duration,xsd:yearMonthDurationandxsd:dayTimeDurationchecks the lexical form and converts between the duration datatypes. Operations that are not defined, such as adding anxsd:durationor a number to a dateTime, are errors (unbound)Model.readalso takes bytes (bytes,bytearrayormemoryview, e.g. fromwrite_bytes) or a file-like object with areadmethod, which is read to the end into memory (text streams as UTF-8).formatis then required, except for HDT, which is detected from its header. Bytes objects are parsed without copying.reads(format="hdt")points toread. In Rust,Model::read_bytesreads from a byte sliceModel.write_bytes(format=...)writes tobytesin memory in any format, including HDT.Model.write,Model.write_cim_xmlandModelStream.writealso take a binary file-like object (anything with awritemethod accepting bytes, e.g.io.BytesIO, a file opened with"wb"or a socket file) instead of a path; writes are buffered, and the object is flushed but not closed.writes(format="hdt")points towrite_bytes.write_cim_xmlraises an exception instead of panicking when writing failsA
Modelshared between threads lets read operations run at the same time:query,query_external,write,writes,write_bytes,size,get_predicate,get_predicate_iris,get_templatesandlist_udfstake a shared lock, while operations that change the model (update,insert,read,reads,map*,validate,infer,create_index,serialize, …) take an exclusive lock, so they wait for the reads in progress and new reads wait for them. The graph explorer answers its requests concurrently. In Rust,Model::query,query_external,write_triples,write_quads,write_hdt,get_predicateandget_predicate_irisand the correspondingTriplestoremethods take&selfModel.add_graphraises an error when given the model itself, which used to deadlockMemory limit: queries that would run out of memory spill to disk instead. Over the limit, which is 80% of the system memory by default (respecting container limits), joins, aggregations, sorts and query results move intermediate data to disk.
maplib.set_memory_limit(max_memory_mb=..., spill_dir=..., max_spill_mb=..., abort_above_mb=...)sets the limit, the spill folder, the most to spill to disk, and an optional hard stop.maplib.memory_usage()andmaplib.memory_limit()report the memory used and the limit. Spilling applies to the streaming engine, which collects all but small results. In Rust,utils::memory::set_memory_limitsapplies the limits, and binaries must usepolars_ooc::Allocatoras their global allocator for spilling to workModel.serve()(maplib Enterprise Edition) serves the model over the SPARQL 1.1 Protocol and Arrow Flight at the same time, in the background, on the model itself, so changes made from Python and through the endpoints are seen by both. Queries through the endpoints run at the same time as each other and as read operations from Python, while updates wait for exclusive access. It returns aServerwith the endpoint URLs,stop(),wait()and context manager support. The SPARQL endpoints (/sparql,/query,/update) follow the W3C SPARQL 1.1 Protocol, includingdefault-graph-uri,named-graph-uri,using-graph-uriandusing-named-graph-uri, content negotiation (JSON, XML, CSV, TSV and Arrow IPC results, Turtle, N-Triples, RDF/XML, JSON-LD, N-Quads and TriG graphs) and the service description./storeimplements the SPARQL 1.1 Graph Store HTTP Protocol. DESCRIBE is supported by the endpoint. The Flight endpoint runs queries fromdo_gettickets andget_flight_infocommands, with results as the DataFrames ofModel.query(), lists the graphs as flights, maps uploaded record batches with a template throughdo_put, and runs updates as an action. Updates are refused unlessallow_update=True.auth_tokenrequires a bearer token on both endpoints (with Flight basic auth handshake support). Other web sites can not read results or send updates unless allowed withcors_origins, and on loopback addresses requests with another Host are refused, against DNS rebinding. LOAD and SERVICE are refused, so clients can not make the server fetch URLs or read files. The Flight features are also served over plain HTTP at/arrowon the SPARQL port (arrow_http=True), for networks where gRPC does not get through.Model.query(query, endpoint=server.url, endpoint_token=...)queries a served model and returns the same results asModel.query()on it, with maplib’s own types sent as Arrow over HTTP. Arrow results are zstd compressed within the Arrow format for clients on other hosts (arrow_compression="auto", or"zstd"/"none", also per request), and other responses are gzip compressed on request. HTTP/1.1 and HTTP/2 are served. Request sizes are limited bymax_request_bytes. In Rust, the API is theservercrate, the Arrow result format is inrepresentation::arrow_wire, the client is inexternal_sparql::maplib_endpoint, andTriplestore::graph_nameslists the graphs with triplesModel.write,Model.write_bytesandModel.writeswithformat="json-ld"write a compacted JSON-LD 1.1 document: a@contextwith the prefixes that may be used (as for Turtle), and a@graphwith one node object per subject. rdf:type is written as@type, xsd:string literals as JSON strings and other literals as value objects with their lexical form, blank nodes that are the object of one triple are nested, and RDF lists are written as@list. Until now, JSON-LD was written by the streaming oxjsonld serializer, which wrote one node object per triple and ignored the prefixes. A prefix whose name is the scheme of an IRI in the model (e.g.exwhen there is an IRI<ex:b>) is left out of the@context, as the IRI would be read as a compact IRI (with hybrid and disk storage this is not checked, and writing such an IRI fails). Blank nodes used in more than one of the written graphs are always written with their label. A singlegraphis written at the top level, as the document’s default graph. To write a dataset, give a list of graphs, e.g.graph=[DEFAULT_GRAPH, "http://example.com/g1"]: the default graph (if listed) is written at the top level and each other graph as a graph object ({"@id": ..., "@graph": [...]}). In Rust,Model::write_jsonldtakes the list of graphs. The writer is tested by round trips of the W3C JSON-LD 1.1 fromRdf and toRdf tests and the W3C RDF 1.1 evaluation testsgraphinModel.write,Model.write_bytesandModel.writescan be a list of graph IRIs for the dataset formats (JSON-LD, TriG and N-Quads), which writes exactly those graphs. The listed graphs must exist. The new constantmaplib.DEFAULT_GRAPHnames the default graph in such a list. In Rust,Model::write_quadsandTriplestore::write_quadstake an optional slice of graphs instead of an optional graphModel.serve(static_dir=..., static_path="/ui", static_vars={...})also serves a web application from a folder on disk, read on every request so a rebuilt application shows on refresh. Paths without a file get the folder’sindex.html(for single page applications),/redirects to the application, and the<!--#echo var='name' -->directives of nginx SSI in its HTML pages are replaced withstatic_vars.Model.explore()(maplib Enterprise Edition) runs the whole Model portal (thevalidation_portal_corepackage, which must be installed) on the model in a background server: its pages, its backend, whose SPARQL endpoint (server.sparql_url,<url>api/query) is answered by the model throughModel.serve, and its query log. The portal’s built frontend is thefrontendargument, theMODEL_PORTAL_FRONTENDenvironment variable, or the one packaged with the portal. JupyterHub’s proxy is detected. The old explorer pages and their FastAPI server are removed;page,graphandfts_pathare deprecated and ignored andpopupis deprecated (with aDeprecationWarning), andflight_portandcors_originsare gonemaplib.configconfigures the OSIsoft PI Data Archive (database"pi_points",mcfg:PIPointsDatabase): applying it reads the PI Points withModel.map_pi_points(parametersurl,data_server,name_filter,source_filter,point_attributesand the others ofmap_pi_points, secretspasswordortoken), andprobelists a few of the points. Like the Asset Framework ("osipi"), it has no resources to configureThe config graph: a model keeps its configuration as triples in a named graph of its own,
maplib.config.CONFIG_GRAPH, in the new config namespacehttps://datatreehouse.github.io/maplib/config#(seeCONFIG_GRAPH.md).maplib.configwrites and reads virtualized databases (DuckDB, Snowflake, BigQuery, Databricks and TimescaleDB over SQL, OPC UA Historical Access and Cognite Data Fusion, with their connection parameters and resources: SQL or options, and stOTTR templates) and indexes (full text search, GeoSPARQL and object indexes for all or chosen predicates), andapply_config(m)connects and virtualizes them. AspenTech IP.21 (database="ip21") is applied withmaplib.ip21.map_ip21, which adds the tags to the model. Secrets are never stored: the configuration names the environment variable holding each one. Resources have static properties (VirtualizationResource.properties, e.g. the CIM unit symbol and multiplier of a measurement), written as ordinary triples on the objects withct:hasResourcethe resource name when the virtualization is applied (apply_properties), since chrontext does not support constant triples in virtualized templatesThe config graph configures Microsoft Fabric:
fabric_lakehouse(mcfg:FabricLakehouseDatabase, a lakehouse’s Delta tables read from OneLake, with resource SQL in DuckDB SQL over the table names) andfabric_warehouse(mcfg:FabricDatabase, a warehouse or a lakehouse’s SQL analytics endpoint, with resource SQL in T-SQL), both signing in with Microsoft Entra ID (theclient_secretof a service principal is a secret).probelists the tables of a lakehouse, and runs SQL on a warehouse withTOP 5The config graph configures Azure Data Explorer and Microsoft Fabric Eventhouses:
kusto(mcfg:KustoDatabase,maplib.virtualization.KustoDatabase), with the parameterscluster(the query URI),database,tenant_id,client_idandauthentication, and theclient_secretof a service principal as a secret (else the default Azure credentials). Its resources are KQL tabular expressions, kept inmcfg:sqllike the SQL of other databases;maplib.config.QUERY_LANGUAGESgives the query language of each database.proberuns a resource’s KQL withtake 5The config graph configures Siemens Insights Hub (database
"insights_hub",mcfg:InsightsHubDatabase): applying it reads the asset model and virtualizes its time series withModel.map_insights_hub(the parameters are its arguments, the secretsclient_secretortoken), with no resources to configure.probegets a token and lists five assets of the tenantModel.servekeeps a query log:GET /query-loggives the lastquery_log_size(default 1000, 0 for none) queries and updates as JSON, each with its text, kind, status and error, rows, bytes, media type, the time spent waiting for a slot, running and serializing, and the client and page that sent it (theX-Query-Sourceheader, else the Referer), together with totals since the server started and summary statistics (percentiles, error rate, rate, most frequent queries).?since=<id>returns only newer entries;DELETE /query-logempties it./statusincludes the totalsSHACL shapes from OPC UA type definitions (maplib Enterprise Edition):
m.opcua.add_typing_shapes(shapes_graph=None, type_namespaces=None, instance_namespaces=None), afterm.map_opc_ua(folder), generates shapes that check instances against their ObjectTypes and VariableTypes: a shape per type, one per browse path of the fully inherited instance declarations, and cardinality shapes for Mandatory instance declarations, in the categoryuamapper:OPCUATypingValidation. It also adds the helper triples the shapes use (uamapper:typeNamespace,uamapper:path, …), which your own shapes can use too.maplib.opcuais now a package;maplib.opcua.OPCUAHistoryDatabaseis imported as beforeOPC UA History Access as a virtualized database (maplib Enterprise Edition):
VirtualizedDatabase(database=maplib.opcua.OPCUAHistoryDatabase("opc.tcp://...")). The external ids of the time series are OPC UA node ids such asns=2;s=ts1, and the resource templates have the parametersid,timestampandvalue. Raw values are read with theasyncuapackage, which must be installed. Conditions on the timestamp in the top level conjunction of a filter restrict the time range read from the server; all other conditions and aggregations are evaluated by maplibInfluxDB as a virtualized database (maplib Enterprise Edition). InfluxDB 3 (Core, Enterprise, Cloud Dedicated, Cloud Serverless, Clustered):
VirtualizedDatabase(maplib.virtualization.InfluxDBDatabase(host=..., database=..., token=...), resource_sql_map={"r": "SELECT sensor AS id, time AS timestamp, value FROM readings"}), queried with SQL over Arrow Flight with theinfluxdb3-pythonpackage. The new SQL dialectInfluxDBDialect("influxdb") follows PostgreSQL, with DataFusion’sdate_binforchrontext:dateBinandchrontext:FloorDateTimeToSecondsInterval, timestamp literals in UTC without a time zone andDOUBLEfor casts to doubles; timestamps without a time zone in the results are returned as UTC. InfluxDB 2, InfluxDB Cloud (TSM) and InfluxDB 1.8:VirtualizedDatabase(maplib.virtualization.InfluxDB2Database(url=..., token=..., org=..., bucket=..., resources={"r": {"measurement": "readings", "id": "sensor", "field": "value"}})), queried with Flux with theinfluxdb-clientpackage. Flux reads the time range in the top level conjunction of the filters on the timestamp and only the time series asked for; other conditions and aggregations are evaluated by maplibMicrosoft Fabric Lakehouses as virtualized databases (maplib Enterprise Edition):
maplib.virtualization.FabricLakehouseDatabase(workspace=..., lakehouse=..., tenant_id=..., client_id=..., client_secret=...)reads the Delta tables of a lakehouse directly from OneLake with DuckDB’sdeltaandazureextensions, and answers the virtualized parts of queries with DuckDB SQL. Each table is a view, so resource SQL is e.g.SELECT sensor_id AS id, ts AS timestamp, value FROM readings(dbo.readingsin lakehouses with schemas). The tables are found by listing the lakehouse’s Tables folder, or given withtables. It signs in with Microsoft Entra ID: a service principal, or one Azure credential (authentication="credential_chain"withchain="cli","managed_identity","env"or"workload_identity").rootreads the tables from another folder, such as a local folder of Delta tablesDuckDBDatabaseworks with DuckDB 2.0, where query results no longer have.pl()Microsoft Fabric Warehouses as virtualized databases (maplib Enterprise Edition):
maplib.virtualization.FabricDatabase(server=..., database=..., authentication=...)answers the virtualized parts of queries with a Fabric Warehouse or the SQL analytics endpoint of a Lakehouse, in T-SQL. It signs in with Microsoft Entra ID through azure-identity: a service principal (authentication="service_principal"withtenant_id,client_idandclient_secret),"default"(DefaultAzureCredential),"interactive"or"azure_cli", and connects with mssql-python (which includes the ODBC driver) or pyodbc (driver="pyodbc", with ODBC Driver 18 for SQL Server). Timestamps in the results, which Fabric stores without a time zone, are taken to be in UTC (timezone). The SQL is T-SQL, from the new dialectsFabricDialect("fabric") andMSSQLDialect("mssql","sqlserver","tsql", for SQL Server 2022 and later and Azure SQL), which can also be used with other databases throughsql_dialect. Limits are written asTOPandOFFSET ... FETCH, time buckets withDATEDIFF_BIG,DATETRUNCand (strides in months)DATE_BUCKET,SAMPLEasMIN, and conditions selected as values as bits. Durations can not be used as values, as T-SQL has no interval type.SQLDialecthas new methods for the parts that differ in T-SQL:ceil,round,avg_argument,as_valueandoffset_requires_order_byAspenTech InfoPlus.21 (IP.21) as a virtualized database (maplib Enterprise Edition), through the aspenONE Process Data REST API (
.../ProcessData/AtProcessDataREST.dll) with therequestspackage:maplib.ip21.map_ip21(m, url, data_source, tag_filter="*", username=..., password=...)searches for tags and adds anip21:Tag(urn:maplib_ip21:) per tag, with its name, description, engineering units and default map, and a time series (chrontext:hasTimeseries) with the tag name as external id, whose values arechrontext:hasDataPoint/hasTimestamp/hasValuewithip21:hasStatusandip21:isGood.maplib.ip21.IP21Databasecan also be used withVirtualizedDatabaseand your own templates. Time conditions in filters restrict the time range read, and grouping bychrontext:FloorDateTimeToSecondsIntervalwith MIN, MAX or AVG of the value is answered with IP.21 aggregates when the range starts and ends on the bucket grid; maplib evaluates the rest. Queries without a lower bound on the timestamp needdefault_start_time. Values that are not numbers are left outOSIsoft PI Data Archive (maplib Enterprise Edition):
Model.map_pi_points(url, data_server=None, name_filter="*", ...)reads the PI Points (tags) of the PI historian through the data servers of the PI Web API, without the Asset Framework, with their descriptor, engineering units, point type, point source and other point attributes, and virtualizes their recorded values with the same data point pattern asModel.map_osipi. Each point is a time series whose external id is its WebId. Authentication, paging, concurrency and the time range of the recorded values read are as formap_osipi. For both, aggregates over PI data points are evaluated in the virtualization, and MIN, MAX, SUM, AVG and COUNT of the values perchrontext:FloorDateTimeToSecondsIntervalbucket are read as PI summaries when the time range is aligned to the buckets and only filtered by timeVirtualizedDatabasetakespushdowns, a list of"value_conditions","group_by"and"ordering", to choose what parts of a query are pushed down into the database. When it is not given, thepushdownsattribute of the database is used if it has one, otherwise all of them, as beforeSeveral virtualizations (maplib Enterprise Edition): each call to
Model.add_virtualization(virtualized_database, resources)adds a virtualized database instead of replacing the previous one, so time series in different databases can be queried together. Each resource belongs to one virtualized database, and the virtualized queries over a resource are sent to the database that holds it; a query matching time series of resources in different databases sends a query to each and combines the results. Joins between time series (e.g. on a shared timestamp) are pushed down only when the time series are in the same database, and group by and slices are pushed down only for a single resource, as before. Adding a resource that was added before moves it to the new virtualized database, so callingadd_virtualizationagain with the same resources still replaces them. In Rust,ChrontextSettingsholds a list ofVirtualizationBackends (a virtualized database with itsVirtualization) instead of a single database, built withChrontextSettings::neworadd_virtualizationTimescaleDB as a virtualized database (maplib Enterprise Edition):
maplib.virtualization.TimescaleDatabase(url=..., host=..., port=..., database=..., user=..., password=..., sslmode=...), queried with thepsycopgpackage (version 3), which must be installed. Its dialect,TimescaleDialect("timescale"), is PostgreSQL withchrontext:dateBinandchrontext:FloorDateTimeToSecondsIntervalcomputed by TimescaleDB’stime_bucket, which also takes strides in months and years. The session time zone is UTC, and PostgreSQL numerics with a fractional part come back as doublesSiemens Insights Hub (formerly MindSphere) virtualization (maplib Enterprise Edition):
Model.map_insights_hub(host, tenant, client_id=..., client_secret=...)reads the assets of a tenant (or ofroot_assetand its descendants) from Asset Management v3 into the graph, with their hierarchy, asset types (as classes, with their inheritance), locations, asset variables, aspects, aspect types and variables (data type, unit, quality code), and virtualizes the time series of the variables of dynamic aspects, with the external idassetId/aspectName/variableName. Their data points (chrontext:hasTimestamp,chrontext:hasValuefor numbers and booleans,insights_hub:hasStringValuefor strings and timestamps,insights_hub:hasQualityCode) are read from IoT Time Series v3 when queried, limited by the bounds on the timestamp in filters and paged by time, with concurrent requests per asset and aspect; a query without a lower bound on the timestamp needsdefault_start_time(an ISO 8601 timestamp, or a duration such asP7Dbefore now). Withaggregate_pushdown=True, MIN, MAX, SUM, COUNT and AVG of the values in buckets ofchrontext:FloorDateTimeToSecondsIntervalof whole minutes dividing an hour, or whole hours dividing a day, over time ranges on the bucket grid, are read from IoT Time Series Aggregates v4 instead (which leaves out values of bad quality, and counts a value exactly on a bucket boundary in the bucket before it). Authentication is with service credentials (OAuth client credentials athttps://{tenant}.piam.{region}.{domain}/oauth/token, derived from a host likegateway.eu1.mindsphere.io, ortoken_url), app credentials through the Technical Token Manager (app_name,app_version,user_tenant), or atoken. Tokens are renewed before they expire and on a 401, and throttled requests (429) are retried. Only the Python standard library is used for HTTP. In Rust,virtualization::insights_hub::read_insights_hubreads the asset model andModel::add_insights_hubadds itAzure Data Explorer and Microsoft Fabric Eventhouses (Real-Time Intelligence KQL databases) as virtualized databases (maplib Enterprise Edition):
VirtualizedDatabase(KustoDatabase(cluster=..., database=...), resource_sql_map={"my_resource": "Telemetry | project id=DeviceId, timestamp=Timestamp, value=Value"}), frommaplib.virtualization. The resources are KQL tabular expressions with anidcolumn of type string and a column for each other template parameter, and the virtualized queries are translated to KQL (maplib.virtualization.translate_kql,KQLDialect): filters (time ranges first),summarizebybin_atforchrontext:FloorDateTimeToSecondsIntervalandchrontext:dateBin, the SPARQL aggregates (count_distinctfor COUNT(DISTINCT)), joins, ordering and slicing, with the results of the SQL translation where Kusto differs from SQL (joins do not match null keys, comparisons with null are null, SUM and AVG of no values are null). Queries use the azure-kusto-data package, with a service principal (tenant_id,client_id,client_secret) or the default Azure credentials (azure-identity), and return all rows (notruncation). The column types of each resource are looked up withgetschemaSPARQL 1.1 Federated Query:
SERVICE <endpoint> { ... },SERVICE SILENTandSERVICE ?endpoint(with the variable bound before the SERVICE) in queries and updates. The pattern is sent to the endpoint with the SPARQL 1.1 Protocol and its results (SPARQL JSON, XML or TSV, with their RDF types and unbound values) are joined with the rest of the query, also within OPTIONAL, UNION, MINUS and FILTER. SERVICE is evaluated after the local patterns of its group, and the distinct values that they bind for the shared variables are sent with the pattern as VALUES blocks (a bind join,batch_sizerows per request,concurrencyrequests at a time), so that the endpoint only returns matching solutions; the bindings of the left side of an OPTIONAL are sent to a SERVICE in the optional part. When the endpoint is a maplib server (Model.serve(), maplib Enterprise Edition), the results come in maplib’s own Arrow format, and the bindings are sent as Arrow too, in batches ofarrow_batch_sizerows.SERVICE SILENTgives a single empty solution when the endpoint fails, times out or can not be reached.Model.set_federation(allowed_endpoints=None, headers=None, timeout=300, max_rows=None, batch_size=1000, arrow_batch_size=1000000, concurrency=4, method="POST", bind_join=True, allow_load=False)sets which endpoints may be reached (by URL prefix, any http(s) URL by default), headers per endpoint (e.g. a bearer token), the timeout and a row limit per SERVICE; redirects are only followed to allowed URLs. Refused endpoints and too many rows are errors also with SILENT. In Rust, the client isexternal_sparql::federation,Triplestore::federationholds the settings,QuerySettings::federationoverrides them, andModel::query_with_federationandupdate_with_federationtake themSPARQL Update
LOAD <url> [INTO GRAPH <g>]andLOAD SILENTread an RDF file from an http(s) URL (the format from the media type of the response, else from the extension of the URL), object storage (s3://,gs://,az://) or afile://URL (only when listed explicitly inallowed_endpoints), into the default graph or the given graph. LOAD is refused unless enabled withModel.set_federation(allow_load=True), and only reaches the allowed endpointsModel.serve()(maplib Enterprise Edition) refuses SERVICE and LOAD unless allowed:serve(service_endpoints=[...])lists the endpoints that SERVICE in queries and updates sent to the server may reach (["*"]for any http(s) URL), with the other federation settings of the model, andserve(allow_load=True)allows LOAD of those URLs (withallow_update). Refused requests get 403./sparqlalso gives SELECT results in maplib’s solution mappings format (application/vnd.maplib.solution-mappings) when asked for, andPOST /arrow/querytakes PVALUES parameters as Arrow, sent after the query command (application/vnd.maplib.parameterized-query), as used by SERVICE from another maplibThe W3C SPARQL 1.1 federated query tests (
sparql11/service) are included in the tests, run against local mock endpoints with SPARQL JSON, XML and TSV resultsFunction calls in templates: template arguments can be computed with SPARQL functions, written Lisp-style as in Frog (M. Jarholt, “Frog: Functions for ontologies — An extension for the OTTR-framework”, Master’s thesis, University of Oslo, 2022, https://hdl.handle.net/10852/96925), e.g.
ottr:Triple((IRI (CONCAT "http://example.net/" ?id)), rdfs:label, (UCASE ?name)). The functions are the SPARQL built-in functions and operators, XSD casts such as(xsd:integer ?x), UDFs, Frog’s base functions that have a SPARQL counterpart, and functions defined in stOTTR documents with Frog’sdefsyntax, e.g.def ex:toIRI(xsd:string ?id) -> ottr:IRI :: (IRI (CONCAT "http://example.net/" ?id)) ., which are inlined. Function calls are evaluated by the SPARQL expression engine over all rows of an instance at once, before list expansion. A call that gives no value (an error, as in SPARQL) isnone: optional parameters are left out, defaults are used, and otherwise the instance is not expanded. Function calls can not be list expanded or take lists, and defined functions can not be recursive. In Python,FunctionCall("CONCAT", [...])is a term forArgument,InstanceandTriple. In Rust,StottrTerm::FunctionCall,Statement::Function,TemplateDataset::functionsandtemplates::functions. In the templates of virtualized resources (maplib Enterprise Edition), the function calls are evaluated on the rows the virtualized database returns, and filters on their values are not pushed downstOTTR documents can give
noneas an argument or default value, which is how templates with aNoneargument are printedShortest paths: the magic predicate
maplib:shortestPath(https://datatreehouse.github.io/maplib/vocab#shortestPath) finds the shortest paths from a source to targets, with their length, cost and nodes, in SPARQL queries. Its options are named as those of the length-constrainedmaplib:path. The sources, targets (maplib:object) and nodes to avoid (maplib:avoid) are the values the other patterns of the group give their variables, or constants, and an unbound object means every node reached. Options:predicateandinversePredicate(the edges and their direction),anyPredicate "forward"|"backward"|"both"(every predicate whose objects are IRIs or blank nodes, exceptrdf:type) withexcludePredicate,minLength(default 1),maxLength,limit(the nearest targets of each source),weightanddefaultWeight(node costs, Dijkstra),maxCost,kShortest(k shortest simple paths, Yen) andallShortest/maxPaths; outputslength,cost,pathIndex,pathNodes(a JSON array), and one row per node withpathNode,nodeIndex,stepPredicateandstepInverse. The search runs on an integer graph of the edge predicates in compressed sparse rows, cached until their triples change, with bidirectional breadth first search for single targets, early stopping, and parallel searches from different sources. See the READMEDuckDB sandbox (maplib Enterprise Edition):
DuckDBDatabase(..., sandbox=True, allowed_directories=..., allowed_paths=...)turns off DuckDB’senable_external_accessand setslock_configurationbefore any resource SQL runs, so the SQL of a resource (or given tomaplib.config.probe) can not read local files (read_csv('/etc/passwd')),ATTACHother databases,COPYto files, install extensions orSET enable_external_access = trueagain. The database file stays usable, as do the allowed directories (with subdirectories) and files.maplib.configtakessandbox,allowed_directoriesandallowed_pathsas parameters of aduckdbvirtualization, andprobeconnects in the same sandbox asapply_virtualization. The sandbox is opt-in, so existing models whose resource SQL reads files next to the database (e.g.read_parquet) keep working; it is recommended whenever users who should not read the process’s files can configure a virtualization. A deployment enforces it with the environment variableMAPLIB_DUCKDB_SANDBOX=1: everyDuckDBDatabaseis then sandboxed,sandbox=Falseis refused, the database file must already exist (no files are created throughdatabase), and the database file and the allowed directories and paths must lie inMAPLIB_DUCKDB_ALLOWED_DIRECTORIES(separated byos.pathsep; when it is unset, no further directories can be allowed).FabricLakehouseDatabaseis not sandboxed, as it needs extensions and network access
Bug fixes¶
The warning that full text search is only defined for named graphs or the default graph is no longer logged for queries that do not use full text search
maplib.add_triples(source, target, source_graph, target_graph)copies the triple tables natively in Rust instead of mapping each predicate through a temporary template from Python, and keeps the blank nodes of the source distinct from those of the target: before, blank nodes with the same label in the two models (e.g._:bread into each) were merged into one. Each copy from another model gets new blank nodes; a copy within one model (sourceandtargetthe same, e.g. to copy a graph into another graph) keeps them, as the graphs of a dataset share blank nodes. In Rust,Triplestore::copy_triplesandModel::add_triplesQueries joining on two variables, where many solutions share the value of one of them, took memory quadratic in the number of solutions once a basic graph pattern had more than about 100 000 solutions: Polars’ join reordering moved a join on only one of the variables first, e.g. the two
ua:BrowseNamepatterns of?type_child ua:BrowseName ?browse_name . ?instance_child ua:BrowseName ?browse_name, so every instance child met every other child with the same browse name. The results were right. maplib now keeps the join order it chose from the patterns (utils::polars::keep_join_order). The OPC UA browse path query of the typing shapes for 100 000 instances with a common browse name needs 0.15 GB instead of over 134 GBReading RDF/XML (also CIM/XML) bounds the expansion of
<!ENTITY>declarations, so a small file with nested entities (“billion laughs”) can no longer exhaust memory. An entity value may expand to at most 1 MiB, and all references to declared entities in a document to at most 10 MiB plus 10 bytes per input byte read, which ordinary use of entities for namespace IRIs (as in OWL files) stays far below. Exceeding a limit is a syntax error naming the entity. External entities (SYSTEM,PUBLIC) are still refused and never fetched.Model.map_xml, which also reads OPC UA NodeSet2 files, ignores the DOCTYPE and never expands declared entities, so it was not affected. In Rust,RdfXmlParser::with_max_entity_expansion,with_max_total_entity_expansionandwith_entity_amplification_factorchange the limitsModel.map_xml(and the OPC UA NodeSet2 mapping, which uses it) lost the text at entity and character references:<item>Pump & motor A</item>gave the two literals"Pump"and"motor". Character references and the five predefined entities (&,<,>,',") are now resolved, and adjacent text, CDATA and references are joined into one literal, trimmed as a whole, so the example gives"Pump & motor A". Other entity references, including entities declared in a DOCTYPE, are dropped and never expanded, as beforePrinting a
Templateparsed from stOTTR (str,repr, or the list fromModel.get_templates()) panicked with “not yet implemented”. Templates print their annotations as@@instances after the signature, and the printed text can be added again withadd_template. Templates without annotations no longer get an empty annotation list from the parserChrontext (maplib Enterprise Edition): a UNION of static patterns joined with time series, e.g.
{ ?w a A } UNION { ?w a B } ?w :hasTimeseries ..., no longer returns every solution twicemaplib.virtualization.DuckDBDatabaseworks with DuckDB 2, which has nopl()on the result ofexecuteModel.create_index(andmaplib.config.apply_config) creates the object indexes it is given for the triples already in the model, not only for triples added afterwards, and drops the object indexes of predicates that no longer have one. Sorting the stored triples by object takes about as long as adding them did. In Rust,Triples::set_object_indexingturns the object index of a triple table on or offThe
expressionof aNotexpression inmaplib.vqwasNone, so virtualized queries with a negated filter could not be translated to SQLVirtualized queries with unary minus or plus (e.g.
ORDER BY DESC(?v * -1)orFILTER(-?v < 0)) panicked with “not yet implemented” for Python databases (SQL databases, Kusto, OPC UA, Cognite). They are nowExpression.UnaryMinusandExpression.UnaryPlusinmaplib.vqand are translated to SQL and KQL; IRI constants areExpression.IRI(withiri), and other expressions that can not be passed to a Python database raise an error instead of panickingLength-constrained property paths: the
{n},{n,m},{n,}and{,m}path syntax of Apache Jena ARQ (and of the SPARQL 1.1 drafts), e.g.?s (ex:p|^ex:q){2,4} ?o, with set semantics like the other paths, and themaplib:pathmagic predicate, e.g.?s maplib:path [ maplib:predicate ex:p ; maplib:minLength 2 ; maplib:maxLength 4 ; maplib:object ?o ; maplib:length ?len ]. It takes one or more predicates and inverse predicates (or any predicate), returns the length (maplib:length), only the shortest paths (maplib:shortest true) or the nodes of each path (maplib:pathIndex,maplib:nodeIndex,maplib:pathNode), andmaplib:pathMode "walk" | "acyclic" | "simple"chooses whether paths may revisit nodes. Either end may be bound or unbound, and paths are searched from the bound end with a breadth-first search per start node over the numbered nodes, deduplicated per node and length. See themaplib:pathsection of API_USAGE.md. In Rust,SparqlError::PathErrorreports invalid optionsZero-length property paths (
ex:p*,ex:p?,ex:p{0,n}) from a variable bound by other patterns or by VALUES now match each of its values that is a node of the queried graph (a subject or object of some triple), as SPARQL 1.1 requires, also nodes withoutex:ptriples. Before,?x a ex:Node . ?x ex:p* ?yleft out the nodes withoutex:ptriplesDatalog (maplib Enterprise Edition): a rule whose BIND combines a variable of inferred triples with a variable of triples that no rule changes (e.g.
BIND(IRI(CONCAT(STR(?p), "_", STR(?n))) AS ?p2)) inferred nothing after the first iteration, whiledebug=Truegave the right resultDatalog (maplib Enterprise Edition): in rules where an OPTIONAL or a FILTER used only triples that no rule changes, the cached part could lose the variables that join it with the rest of the rule, so that an OPTIONAL matched values of unrelated nodes. Results differed from
debug=TrueSHACL: parameter declarations (
sh:parameter) are no longer read as property shapes, which failed on parameters with e.g. a list of datatypesJSON results of SELECT queries (
return_format="json") list the variables in the order of the projection, instead of an arbitrary orderModel.query_externalno longer panics (“entered unreachable code”) on SELECT results where a variable is unbound in some rows, as with OPTIONAL. Unbound values are nulls, also before the first bound value, and a variable that is never bound is a column of nullsINSERT { GRAPH ?g { ... } } WHERE { ... }andDELETE { GRAPH ?g { ... } }inModel.updateuse the graphs that?gis bound to. The graph name variable was left out of the solutions, so nothing was inserted or deleted when it appeared only as a graph name in the templateModel.insertreturned only one of the tables of a predicate whose subjects or objects have several types (e.g. both IRIs and literals as objects); it now returns all of the inserted triples of the predicate in one DataFrame. In Rust,triplestore::new_triples_by_predicatemerges new triples per predicateReading, mapping and inserting triples could hang with no CPU use. The triple tables of different predicates and types are built, compacted and deleted from in parallel, and each of these steps queried its table with the streaming engine of Polars, which can wait for threads that the parallel steps have taken. They now use the in-memory engine, which is safe there and also avoids the fixed cost of a streaming query per table
Model.query(debug=True)panicked with “not yet implemented” when a query with no results containsGRAPH ?g { ... }. The explanation now reports theGRAPH ?gpattern as a whole when it has no solutions (withGRAPH <g>, the inner pattern is explained in that graph as before). DESCRIBE queries are explained like the other query forms, and if the explanation itself fails, the query returns its results without debug output instead of failing
Performance¶
generate_templates is implemented in Rust (maplib::template_generator::generate_templates) instead of Python, with the same templates, parameter names, order and types, and runs in about 60% of the time on FIBO.
Datalog and SHACL rule inference with many small iterations is faster, about 2× for a long chain of semi-naive iterations and 15-20% for the Datalog benchmarks: CONSTRUCT results, encoding of new solutions and small solution mappings are collected with the in-memory engine instead of the streaming engine, small rank sorts no longer use the thread pool, and an in-memory query result is picked up within a millisecond of being ready instead of up to 50 ms. Evaluating a triple pattern outside a named graph no longer logs a full text search warning when there is no full text index.
Disk storage answers lookups of all properties of a subject (SELECT ?p ?o WHERE { <e> ?p ?o }) about 10× faster: at 20 M triples, 7 ms instead of 86 ms, against 2 ms with memory storage. Lookups with a known predicate are about 2× faster. Decoding a few results no longer joins them with the whole IRI and literal dictionary (maplib Enterprise Edition), a bound subject or object is encoded once per triple pattern instead of once per predicate table, and the few row groups a bound subject needs are read directly instead of through a lazy scan per range. Queries that return many rows, such as LIMIT 100000, are as slow as before with disk storage (about 90 ms at 20 M triples, 3 ms in memory).
SHACL rules run on the datalog engine with semi-naive evaluation: when a rule runs again, it only joins the triples inferred since its last run with the existing ones, parts of the rule bodies that no rule can change are computed once, and rules whose inputs did not change are not run again. Previously each iteration ran every rule on all of its focus nodes, so recursive rules (e.g. transitive closures) and long rule chains took time quadratic in the number of iterations. This also holds when the inferred predicate is used inside a property path, as in $this ex:child/ex:offspring? ?o. Rules that only add triples (no OPTIONAL, MINUS, NOT EXISTS or aggregates) and share an order see each other’s new triples in the same pass, so chains of such rules finish in fewer passes.
When inferences go to a separate graph, datalog and SHACL rules no longer compare inferred triples with the data graph for predicates that the data graph has no triples of. SHACL validation without rules no longer reads the shapes graph for rules, and finds parameter declarations with one query instead of a filter in each shape query.
In disk mode the triple tables and their deduplication index are no longer held in memory. With few triples per distinct IRI or literal (as in the CIM data), memory use is dominated by the dictionary and reading, and disk storage saves little.
Triple table segments are compacted much faster, which matters most for datalog, where every iteration adds a small batch of triples to tables that are often large:
Size-tiered compaction: only the trailing segments of comparable size are merged with the new triples, so a large table is no longer rewritten every 11 inserts. A table has O(log n) segments (at most 16).
Merging sorts packed rank keys with Polars instead of a value-by-value heap merge, and builds the object sort from the merged triples instead of also reading and merging the object sorts of the segments.
Disk mode: new triples are deduplicated against only the row groups that may hold their subjects, found through the sparse subject index, instead of reloading every stored triple of the table into an index on each insert. Small trailing segments (under 16k triples) stay in memory until merged, so small inserts don’t write and rewrite small parquet files.
Query results are collected with the Polars streaming engine, which the rest of maplib already used; until now the final step of a query ran on the in-memory engine. Whole-table scans and counts, large result sets and DISTINCT are faster, while small queries are about as fast as before. MAPLIB_POLARS_ENGINE=in-memory switches back to the in-memory engine.
IN with a list of IRIs looks the values up in a hash set instead of comparing with each IRI in turn, so long lists no longer slow a query down. The same holds for IN with a list of integers (of any integer datatype) when the left side holds integers of one integer datatype. Property paths with *, + and ? map the computed paths back to IRIs and literals by position instead of with joins. Patterns without shared variables (also inside OPTIONAL) are combined with a parallel cross join instead of a join on a constant key, which ran on one thread.
After Model.compact(), subjects are marked as sorted, so DISTINCT and grouping on subjects are faster.
ORDER BY on IRIs and strings no longer collects its input in memory for each sort column, and with memory storage it ranks the values batch by batch, so large sorts are several times faster and spill to disk under the memory limit. ORDER BY with LIMIT or OFFSET, and joins and aggregations, still keep their working state in memory.
ORDER BY with a large LIMIT (with or without OFFSET; LIMIT plus OFFSET at least 10 000 and at least a thirty-second of the estimated solutions) is collected with the in-memory engine, which sorts much faster than the streaming engine when many rows are kept. Such queries are several times faster, and many times faster for some sort keys. Smaller LIMITs are left to the streaming engine, whose top-k is faster there. Their working state is kept in memory, so they do not spill to disk under the memory limit. MAPLIB_POLARS_ENGINE=streaming keeps the streaming engine for them.
GRAPH ?g in the templates of Model.update partitions the solutions by the encoded graph IRIs in one pass, and decodes one IRI per graph, instead of decoding every row and filtering the solutions once per graph.
Deleting triples that are not in a table no longer rewrites the table, so DELETE DATA and DELETE WHERE of absent triples leave the stored triples and their indexes as they are. The count of deleted triples that decides when memory storage is compacted counts only the triples that existed.
Tantivy answered exact matches from its term dictionary, but scanned all strings for substring matches, which it ran whenever the exact matches did not fill the limit. DuckDB scans the strings once per query for both. Reading without full text search is unchanged.
Serialization, compared with the build before these changes on CIM N-Triples (2.2 M triples, IRI-heavy) and a synthetic literal-heavy file (4.5 M triples, 3 M distinct literals), release-test profile, medians of 3 runs alternating between builds, each run in a fresh process. Parquet is the new default, raw_db writes DuckDB dictionaries as before:
Category |
Workload |
CIM 2.2 M: parquet / |
Literals 4.5 M: parquet / |
|---|---|---|---|
Writing |
|
1.97 → 0.43 s (-78%) / 0.73 s (-63%) |
3.90 → 0.79 s (-80%) / 1.37 s (-65%) |
Writing |
Size on disk |
107 → 35 MB (-67%) / 107 MB |
344 → 103 MB (-70%) / 345 MB |
Reading |
|
0.12 → 0.06 s (-51%) / 0.06 s (-51%) |
0.20 → 0.10 s (-49%) / 0.15 s (-21%) |
Reading |
|
0.05 → 0.55 s (+1115%) / 0.01 s (unchanged, used in place) |
0.06 → 0.72 s (+1094%) / 0.01 s (unchanged, used in place) |
Reading |
|
0.03 → 0.56 s (+1964%) / 0.02 s (unchanged, used in place) |
0.02 → 0.74 s (+3580%) / 0.01 s (unchanged, used in place) |
Disk |
After |
0.079 → 0.043 s (-46%) |
0.081 → 0.046 s (-44%) |
Disk |
Same, insert 100 k triples (half new subjects) |
unchanged (within noise) |
0.224 → 0.171 s (-24%) / 0.193 s (-14%) |
SPARQL |
After |
unchanged (within noise) |
unchanged (within noise) |
Serialized triple tables are written with row groups of 16,384 rows, as disk storage writes its own triple tables, instead of 1,000 rows. Disk storage copies the files as they are on deserialization, and scans read far fewer row groups.
Dictionaries are built into DuckDB (hybrid and disk storage, and
raw_db) by loading the rows first and building the unique indexes afterwards, instead of inserting into an indexed table, and the dictionaries are deserialized concurrently with the triple tables.With hybrid and disk storage, parquet dictionaries are built into DuckDB databases (including their unique indexes) on deserialization, while
raw_dbdictionaries are used in place, so useraw_db=Truewhen a serialized model is loaded into hybrid or disk storage often.
Faster reading, ORDER BY, counting and deleting (memory storage):
Dictionary ranks (used to sort triples when they are added or compacted, and for ORDER BY) are looked up in arrays indexed by the dictionary code instead of hash maps, and the distinct codes of a batch are found with a bitset instead of a hash set. The deduplication index of triple tables uses a faster hash function.
The deduplication index of a triple table is built when triples are first added to an existing table, not when the table is created, so tables that are only read and queried never hold it.
Triple tables of different predicates and types are built in parallel when triples are added.
New dictionary entries are merged into the dictionary in one pass when they are many compared with its size, instead of one at a time.
COUNT(*)(or COUNT of one of the variables) over?s ?p ?oin a single graph is answered from the sizes of the triple tables.DELETE WHERE { ?s ?p ?o }andDELETE WHERE { ?s <p> ?o }(also asDELETE { .. } WHERE { .. }without INSERT or USING), matched in the graph they delete from, drop the triple tables without evaluating the pattern.Garbage collection of the dictionary after large deletes marks the used codes in a bitset instead of building range sets per table segment.
Category |
Workload |
Before |
After |
Change |
|---|---|---|---|---|
Reading |
N-Triples, 20 M triples (4 predicates, 2 M subjects) |
19.6 / 20.1 s |
10.6 / 10.9 s |
-46% |
Reading |
N-Triples, 8 M triples (400 predicates) |
4.50 s |
4.38 s |
-3% |
Reading |
RDF/XML, CIM 2.2 M triples |
1.71 / 1.90 s |
1.23 / 1.28 s |
-28% / -33% |
SPARQL |
ORDER BY over 5 M IRIs, LIMIT 10 |
0.29 s |
0.077 s |
-74% |
SPARQL |
ORDER BY over 5 M string literals, LIMIT 10 |
0.73 s |
0.077 s |
-90% |
SPARQL |
|
0.128 s |
< 1 ms |
-99.9% |
SPARQL Update |
30 CONSTRUCT inserts of ~2 k triples into a 5 M triple table |
4.13 s |
4.13 s |
unchanged |
SPARQL Update |
|
0.348 s |
0.067 s |
-81% |
SPARQL Update |
|
0.63 s |
0.57 s |
-9% |
Memory |
Peak RSS, 20 M triple workload |
6.14 GB |
6.44 GB |
+5% |
(Release-test profile, Apple M-series laptop (14 cores), medians of 3 runs alternating with the build before this change, while other jobs were running on the machine. Where two numbers are given they are from two rounds of runs. RSS after reading varied by more than ±1 GB between runs of the same build, so it is not reported; the deduplication index that is no longer built takes about 14 bytes per triple. The higher peak comes from building tables in parallel.)
In-memory dictionaries store their strings back to back in a chunked arena, with flat per-entry vectors of offsets, lengths, prefixes and codes, a hash table from strings to entries and an array from codes to entries. This replaces a BTreeMap from strings to codes, a reverse HashMap and a separate allocation per string: about 30 bytes per entry on top of the string instead of about 100. String order is no longer maintained on every insert, but computed when needed (ranks, compaction, ordered output) and cached; after inserts only the new strings are sorted and merged into the cached order. Looking up a string is a hash probe instead of a walk down a B-tree.
Category |
Workload |
Before (with the changes above) |
After |
Change |
vs. v0.21.0 |
|---|---|---|---|---|---|
Reading |
N-Triples, 20 M triples (4 predicates, 2 M subjects) |
11.7 s |
8.3 s |
-29% |
-62% |
Reading |
N-Triples, 8 M triples (400 predicates, ~8 M distinct strings) |
5.00 s |
2.96 s |
-41% |
-44% |
Reading |
RDF/XML, CIM 2.2 M triples |
1.76 s |
1.12 s |
-36% |
-43% |
Memory |
Peak RSS, 8 M triples |
3.33 GB |
2.44 GB |
-27% |
-31% |
Memory |
Peak RSS, CIM 2.2 M triples |
1.51 GB |
1.28 GB |
-15% |
-19% |
Memory |
Peak RSS, 20 M triple workload |
6.10 GB |
5.66 GB |
-7% |
-2% |
SPARQL Update |
|
0.68 s |
0.26 s |
-62% |
-75% |
SPARQL |
ORDER BY, COUNT, inserts, predicate deletes |
unchanged |
(Medians of 2 runs per variant alternating between v0.21.0, the build with the changes above and this build, under heavy memory pressure from other jobs, so RSS after reading is not reported. The 20 M triple workload peaks while parsing, before the dictionary is complete, so the dictionary saves less there.)
Faster encoding and decoding with in-memory dictionaries:
Reading, mapping and inserting encode string columns directly with global codes: all values are looked up in the dictionary in parallel, the new ones are deduplicated in parallel and added once with consecutive codes. Before, each column got its own local dictionary, which was then merged into the global dictionary and its codes re-encoded. Hybrid and disk storage keep the previous path.
Repeated values next to each other are looked up once, and constant columns (e.g. from templates) are encoded once instead of once per row.
Query results and formatted values (
<iri>,_:blank,"string") are decoded straight into string arrays in parallel, without building a string per value first.
Category |
Workload |
Before (with the changes above) |
After |
Change |
vs. v0.21.0 |
|---|---|---|---|---|---|
Reading |
N-Triples, 20 M triples (4 predicates, 2 M subjects) |
6.49 s |
3.98 s |
-39% |
-81% |
Reading |
N-Triples, 8 M triples (400 predicates) |
2.26 s |
1.53 s |
-32% |
-64% |
Reading |
RDF/XML, CIM 2.2 M triples |
0.93 s |
0.60 s |
-35% |
-68% |
Mapping |
Template with 4 triples (2 constant), 3 M rows |
1.36 s |
1.09 s |
-20% |
-89% |
SPARQL |
SELECT 5 M rows of IRIs / string literals |
0.152 / 0.090 s |
0.121 / 0.085 s |
-21% / -6% |
-32% / -36% |
Memory |
Peak RSS, 8 M triples / CIM / mapping / 20 M triples |
2.58 / 1.29 / 2.38 / 5.50 GB |
2.50 / 1.30 / 2.31 / 5.69 GB |
-3% / 0% / -3% / +3% |
-31% / -17% / -39% / -3% |
(Medians of 2 runs alternating with the build before this change. The v0.21.0 column uses v0.21.0 runs from an earlier round on the same machine.)
Faster lookups, property paths and small inserts:
Property paths (
+,*,?) with a bound end, a constant IRI or a variable whose values are known from VALUES or earlier patterns, follow the paths from those nodes only, step by step, instead of computing the transitive closure of the whole graph and filtering it.The values of a pattern’s variables are pushed down from solution mappings of up to 100 000 rows (was 500). Lookups of many values walk the sorted values and the sparse index together instead of searching the index once per value, and read an in-memory table with one gather instead of one slice per range. When the ranges cover more than half of a table the whole table is read, and at most 256 ranges of a stored table are read.
A small insert or delete into a large in-memory table without a deduplication index checks only the stored triples with the same subjects, instead of indexing every triple of the table first.
Category |
Workload |
Before (with the changes above) |
After |
Change |
|---|---|---|---|---|
SPARQL |
VALUES with 20 000 subjects joined with a 5 M triple table |
186 ms |
39 ms |
-79% |
SPARQL |
The same, then a second pattern on the ~25 000 objects found |
213 ms |
38 ms |
-82% |
SPARQL |
VALUES with 2 000 subjects (alone / with a second pattern) |
24 / 27 ms |
24 / 23 ms |
0% / -16% |
Property paths |
|
319 ms |
24 ms |
-92% |
Property paths |
|
158 ms |
22 ms |
-86% |
Property paths |
|
206 ms |
34 ms |
-84% |
Inserting |
200 triples into a 20 M triple graph (first / second insert) |
7 / 16 ms |
6 / 15 ms |
~0% |
Inserting |
30 CONSTRUCT inserts into a 20 M triple graph |
4.14 s |
4.27 s |
+3% |
Memory |
Peak RSS, lookups / 20 M triples with inserts |
5.62 / 5.70 GB |
5.39 / 5.62 GB |
-4% / -1% |
(Medians of 2 runs alternating with the build before this change, under heavy load from other processes; differences below ~5% are within the noise.)
Reading and writing HDT is much faster. Reading decodes each term of the HDT dictionary once and maps it directly into the model’s dictionaries, then adds the triples as integer IDs in parallel batches, without building HDT query indexes. Writing deduplicates and maps triples using the model’s dictionary codes, decodes only the distinct terms, and writes the triples section without building query indexes. The written files are unchanged. On a CIM model of 2.2 million triples, reading takes 0.28 s instead of 3.41 s in v0.20.32 (12x faster) and writing 0.20 s instead of 5.69 s (28x faster)
Bug fixes¶
SPARQL:
+,-,*and/were parsed as right associative, so8 - 3 - 2gave 7 and12 / 2 / 3gave 18. Operators of the same precedence are now left associative, as the SPARQL grammar specifies. Serialized queries put arithmetic in parentheses, so that8 - (3 - 2)and(1 + 2) * 3keep their meaningCtrl-C now stops a running query. Before, a long query could keep running until it finished
STRDTwithxsd:dateTimeorxsd:dateTimeStampdropped the time zone, e.g.STRDT("2015-03-15T15:02:37-05:00", xsd:dateTime)gave"2015-03-15T20:02:37". The time zone is now kept, also when the values have different time zonesA triple pattern with the same variable in two positions (e.g.
?s <p> ?s) panicked, in queries and inDELETE WHERE, when the predicate also had triples where the two positions had different types (e.g. blank node subjects with IRI objects)Hybrid storage:
Model.compact()(also run byModel.serialize()) failed with a DuckDB “Duplicate key” error once the dictionary had to be renumbered, e.g. after more than ten inserts into the same predicateSHACL rules: a rule that re-asserts a triple already in the data graph no longer makes inference loop until
max_iterations(or forever with one rule). Inserting transient triples that all already existed non-transiently dropped the earlier transient triples with the same predicate, which were then inferred again on every iteration (#76)SHACL rules: rules whose WHERE clause has no triple pattern on
$this(onlyOPTIONAL/BIND) now also run for focus nodes that become targets of their shape after the rule first ran (#77)SHACL: shapes with several
sh:messagevalues (e.g. one per language) lost the results of property shapes, so data violating them was reported as conforming, kept only one of the messages of node shapes, and gave one result per message for SPARQL constraints. Every result now has all the messages of its shape assh:resultMessagevalues. Property shapes with severalsh:nameorsh:descriptionvalues also lost their constraints (#86)HDT: writing the same model twice could give different files, as the properties of the triples section’s control information were written in random order
HDT: literals are written with their values as they are, as hdt-java and hdt-cpp do, rather than N-Triples escaped, so other HDT tools no longer read a tab as
\t. Reading takes a backslash in a literal of a file from hdt-java or hdt-cpp as part of the value, rather than as the start of an escape. Files written by earlier maplib versions are recognised and still read correctly. Writing a literal that contains U+0000 raises an error, as HDT cannot store itHDT: the header of written files describes the dataset with the same vocabulary as hdt-java and hdt-cpp: IRIs for the dataset types and the dictionary and triples formats, and
hdt:publicationInformationfor the publication informationFull text search: with a fixed subject (
ex:a maplib:fts ?fts), the limit was applied before filtering on the subject, so few or no results were returned. Triples deleted with SPARQL Update were still found, now they are no longer found afterModel.compact(). An on-disk index no longer leaves a folder behind for every index createdPretty Turtle and RDF/XML: the rdf:first and rdf:rest triples of blank nodes that do not form a well-formed list were not written, e.g. of a list without rdf:nil, a list node with two rdf:first, or a list whose later node is also referred to by another triple (shared or cyclic lists). Only lists where each node has one rdf:first and one rdf:rest, and every node after the first is only referred to by the rdf:rest before it, are written as lists; other rdf:first and rdf:rest triples are written as any other triples
Pretty RDF/XML of several graphs in one document (as for CIM/XML): a blank node used in more than one of the graphs could be nested or written as part of a list in one graph, so that the other graphs referred to a blank node that was not in the document. Such blank nodes are now written with their label
v0.21.0¶
Performance¶
Besides the changes listed below, the memory improvements earlier in this release contribute to the faster reading and SHACL validation.
RDF/XML is parsed in parallel, by splitting the document between top-level elements. The RDF/XML parser from Oxigraph (oxrdfxml) is now included in maplib to make this possible
RDF/XML is written with a new pretty serializer (see Features)
CIM/XML is written by building temporary graphs with two SPARQL queries driven by the profiles (one for the instances of the profile classes, one for their profile properties) and writing them with the pretty RDF/XML serializer, instead of running queries per class and property
Disk-backed models keep their category map databases open instead of opening a database for every lookup, which makes small queries about ten times faster
Inserting triples no longer ranks the inserted values against the whole dictionary, which made every insert cost time proportional to the number of distinct IRIs and literals in the model. This was most of the per-iteration cost of datalog inference
The sort order of the dictionary is cached between ORDER BY queries and datalog iterations, until new values are added
Removed debug output that SHACL validation printed for every sh:class constraint (it also evaluated the constraint a second time)
Breaking changes¶
The
streamingparameter is removed fromModel.query,Model.update,Model.insert,Model.validate,ValidationReport.resultsandValidationReport.details. Polars already runs lazy queries with its streaming engine, so the parameter had no effect (benchmarks withstreaming=Trueandstreaming=Falsewere identical). Remove the argument from callsRDF/XML (and CIM/XML) is read in parallel by default (
parallel=None). Passparallel=Falseto read sequentially. Line numbers in parse errors may refer to the position within a chunk of the documentRDF/XML output is now formatted differently (one element per subject, nested blank nodes), but contains the same triples
write_cim_xmlwrites CIM/XML as IEC 61970-552 / CGMES requires (see Bug fixes). Objects are identified relative to the document (rdf:ID=””, rdf:resource=”# ”), so read the files with the same base_irias the original data, as for any CGMES file
Features¶
write_cim_xmluses the pretty RDF/XML serializer. The exported triples are the same as before (the FullModel header first, then one typed element per instance with the properties its class has in the profiles, literal values as plain strings), but elements and properties are ordered differentlyPretty RDF/XML serializer, working like the pretty turtle serializer: one node element per subject with all its properties, typed node elements when the type has a prefix, blank nodes that are used once nested where they are used, RDF lists written with rdf:parseType=”Collection” when possible, and all namespaces declared on the root element
ASK queries. m.query() returns a bool, or a SPARQL JSON boolean result with return_format=”json”
isNUMERIC and RAND SPARQL functions
SHA256, SHA384 and SHA512 SPARQL functions
Read and write TriG and N-Quads (format “trig” / “nquads”, or the .trig / .nq file extensions when reading). Writing these formats outputs all graphs unless a graph is specified
Bug fixes¶
write_cim_xmlwas not compliant with IEC 61970-552 / CGMES (also before this release):Objects were written with absolute IRIs (rdf:about=”urn:…:#
”); they are now rdf:ID=” ” with rdf:resource=”#_ ” references, when all objects share one base IRI Enumeration values (e.g. cim:WindingConnection.Y) were written as objects when the vocabulary was in the data graph
Objects that also had superclass types (e.g. cim:Equipment) were written once per type; they are now written once, as their most specific class
Association ends that the profiles mark as not used (cims:AssociationUsed “No”) are never written, and the used end is derived from them when the data only has the unused end
Only the namespaces that are used are declared, and the default Model.created / Model.scenarioTime are UTC with whole seconds (e.g. 2021-02-02T22:28:41Z)
Disk-backed models wrote subjects on the boundaries between internal segments twice when writing turtle or RDF/XML (giving invalid RDF/XML with a duplicate rdf:ID)
Writing turtle could hang forever (a deadlock between the pretty turtle writer’s parallelism and Polars), e.g. for CGMES/CIM models
sh:not required every constraint of its shape to fail, instead of the shape not conforming. Combined with sh:or this gave false results, e.g. 363 false sh:or results when validating CGMES shapes against the SHACL-SHACL shapes (pyshacl agrees there are none)
sh:not with a shape that has several constraints of which some pass and some fail (e.g. two sh:class values) treated the shape as conforming, giving false sh:not results
Inserting triples could drop triples whose values had been re-encoded in the dictionary, as their values could not be ranked when sorting
Removed the pyarrow <24.0.0 pin. pyarrow only publishes Python 3.15 wheels from 26.0.0, so the pin made maplib uninstallable on Python 3.15
COUNT now returns xsd:integer (Int64) as required by the SPARQL 1.1 spec, instead of xsd:unsignedInt (UInt32)
SUM, AVG, MIN and MAX now work when the aggregated variable has multiple types. SUM and AVG promote the numeric types in each group (integer < decimal < float < double) and are unbound if a group contains a non-numeric value, MIN and MAX use the SPARQL ordering and keep the type of the selected value
Division follows SPARQL op:numeric-divide: integer / integer gives xsd:decimal (e.g. 5 / 2 = 2.5, previously truncated to 2.0 as xsd:double), mixed types are promoted (decimal < float < double), and decimal division by zero is an error (unbound)
COALESCE over arguments of different types returns the first bound argument (previously a value of a later argument could be returned nondeterministically)
Fix panic when combining lang strings with differently encoded categoricals (e.g. COALESCE(?langString, “x”@en))
Infinite xsd:double and xsd:float values are written as INF and -INF (previously inf and -inf, which are not valid lexical forms) in query results, STR() and all RDF serializations
DATATYPE() of an unbound value or an error is unbound (previously the datatype of the variable was returned)
YEAR, MONTH, DAY, HOURS and MINUTES return xsd:integer and SECONDS returns xsd:decimal including fractional seconds (previously xsd:unsignedInt, dropping fractions)
STR() of xsd:dateTime and xsd:date values gives the XSD lexical form (e.g. 2010-06-21T11:28:01+00:00 instead of 2010-06-21 11:28:01.000000+00:00)
Triple patterns without variables (e.g. { :s :p :o }) never matched. They now give one solution if the triple exists
GROUP_CONCAT concatenated internal codes instead of the string values, and used an empty default separator instead of a single space. It now concatenates the string values (as STR()) of any type, including lang strings
ABS, CEIL and FLOOR work when the argument has multiple types, and CEIL and FLOOR keep the type of the argument (previously xsd:integer)
CONCAT accepts zero or one arguments
Casting a blank node (e.g. xsd:double(?x)) gives an unbound value instead of failing the query
FILTER, IF, NOT (!), AND (&&) and OR (||) use the effective boolean value of their operands (SPARQL 1.1 17.2.2): numbers are true unless zero or NaN, strings are true unless empty, and other values are errors. Previously a non-boolean FILTER failed the query, and IF with an error condition took the else branch instead of being unbound
Casts to xsd:boolean, xsd:integer, xsd:decimal, xsd:float and xsd:double follow the XPath casting rules: strings must have a valid lexical form (e.g. xsd:boolean(“abc”) and xsd:integer(“1.5”) are unbound, previously false and a number), booleans can be cast to xsd:decimal, NaN and infinite values cast to integers or decimals are unbound, and casting other types such as dateTimes to numerics is unbound
xsd:string() of a decimal gives the canonical form (e.g. “2.5”), not the stored representation with padding zeros
“1”^^xsd:boolean and “0”^^xsd:boolean are valid booleans (previously rejected when parsing)
STRBEFORE and STRAFTER accept a language tagged second argument when it is compatible with the first argument, and give the simple literal “” when there is no match, as required by the SPARQL 1.1 spec
REPLACE keeps the language tag of its first argument
CONCAT only gives a language tagged result when all arguments have the same language tag, and is unbound when an argument is unbound or not a string
COUNT(DISTINCT ) is supported, and COUNT() works for patterns without variables
SELECT * over a pattern without variables (e.g. { :a :b :c }) no longer panics
v0.20.32¶
Features¶
Turtle output for construct queries
v0.20.30¶
Features¶
Add bindings parameter that replaces variables with ground terms (safe query parameterization without special syntax)
v0.20.29¶
Bug fixes¶
Fix query external for empty results, variable sequence and iri-columns.
v0.20.28¶
Performance¶
Pretty turtle write performance drastically improved (around 12x better)
v0.20.26¶
Features¶
The IRI maplib:DefaultGraph is the default graph
m.query_external(query, endpoint) lets you query an external SPARQL endpoint (currently only select and HTTP GET).
v0.20.25¶
Bug fixes¶
Fix issue where count over zero solution mappings returned no rows (should return one row with zero as the count)
Performance¶
Improve deserialization performance to hybrid storage and to in memory storage
v0.20.24¶
Bug fixes¶
Bug fix for serialization of lang strings
(Attempted?) Bug fix for disk based storage of categoricals on windows
v0.20.23¶
Features¶
Serialization with disk based storage of categoricals - about 40x faster than ntriples read
v0.20.22¶
Performance¶
Fix performance issue for ORDER BY when cats are on disk
Fix performance issue in queries with a lot of OPTIONAL clauses (see bugfix below)
Bug fixes¶
Fix issue where OPTIONAL clauses sometimes introduces duplicates
v0.20.21¶
Features¶
Support for OPC UA mapping to RDF using m.map_opc_ua() providing the folder with NodeSet2 XMLs as an argument
m.add_udf() allows the user to create a UDF working on Polars DataFrames where arguments form the columns. The function is callable from SPARQL.
m.serialize(“folder”) and Model.deserialize(“folder”) serializes and deserializes the triples in a Model object super quickly (20x faster than ntriples in our bench).
Breaking (but not worthy of major release IMO)¶
IndexingOptions no longer has subject_object_index argument as this is now default.
v0.20.20¶
Bug fixes¶
Avoid panic when using pyarrow >= 24.0.0
rep = m.validate() always produces rep.results() even when report_graph is specified.
Features¶
Facade X like mapping of dataframes to graph in m.map_df()
Target counts for shapes are now part of the SHACL report graph
v0.20.19¶
Bug fixes¶
Workaround for Polars issue that could sometimes produce errors in filter not exists
Move away from rdf:_{n} predicate to represent lists in Facade-X as it is does not play nice with our partitioning
Features¶
Prefix maplib:https://datatreehouse.github.io/maplib/vocab#
maplib:uuidv5 and maplib:struuidv5 functions
m.templates_to_graph() materializes templates as triples in graph so they can be queried (Thank you @thenonameguy !)
m.read(), m.reads() now eats hdt and m.write() spits it out. (Thank you @sihingkk !)
v0.20.16¶
Bug fixes¶
Work around Polars issue where double not exists can return erroneous results
Turn caching mechanism for on disk SHACL back on :-)
Fix SHACL SPARQL issue with OPTIONAL {} that could return too few results.
Escape certain characters after the prefix in prefixed IRIS.
Performance¶
Move to latest Polars
v0.20.15¶
Performance¶
Move to A* search for optimal join sequencing - we do unfortunately not take into account nesting here yet.
v0.20.14¶
Bug fixes¶
Fix issue in cross join detection / query planning.
v0.20.13¶
Bug fixes¶
Bug fix for optimization for certain types of large queries with OPTIONAL and BIND.
Improve error message when UNBOUND(?v) does not find ?v.
v0.20.12¶
Bug fixes¶
Disk based cats panicked with duplicate string (large strings)
Sometimes tried to write triples to disk (disabled functionality for the time being)
Performance¶
Caching for SHACL rules with disk based storage improves validation performance
v0.20.11¶
Bug fixes¶
Exists queries with nested subqueries could be true when they should be false
Performance¶
Improve detection of cross joins, avoiding them better
Features¶
Integrate chrontext into maplib, enabled when m.add_virtualization() has been run.
v0.20.10¶
Bugfixes¶
SHACL rules: incrementally build targets not always updated
SPARQL adding now() and a few bugfixes having to do with multiple datatypes in expressions
Performance¶
Improvements to query planning for large queries
v0.20.8¶
Bug fixes¶
xsd:decimal is now represented internally as Decimal(precision=38,scale=12)
Fix another instance of missing query parameters in debug
Features¶
max_iterations parameter for shacl rules (validate())
v0.20.7¶
Bug fixes¶
Fix bug where comparison expressions involving multiple types could fail.
v0.20.6¶
Performance¶
Subject object indexing for deduplication now default on, improves load speeds for many small files
Batch size for triples read now defaults to 10_000_000, reducing memory consumption.
v0.20.5¶
Bug fixes¶
Fix issues with IF where having multiple types for “then” and “otherwise” could cause an exception
Fix issue where THIS was not injected into SHACL SPARQL
Features¶
rules_debug in validate now prints every step so it is easier to debug recursive issues
v0.20.4¶
Bug fixes¶
Fix rounding issue
Improve error messages from SHACL rules
v0.20.3¶
Bug fixes¶
Fix issue with regex and replace where pattern was of form str(iri).
Fix issue where debug_rules could sometimes panic in validate()
Features¶
Add support for STRDT
Add missing support for SPARQL targets in SHACL Rules
v0.20.2¶
Bug fixes¶
Fix caching issue in Datalog engine
Work around issue with timestamped dates, these are now read without the timestamp, but should be fixed properly at some point.
Features¶
SHACL rules now support recursion, and has multiple optimizations.
Still missing conditions, other rule types than sparql construct, and taking ordering into account.
v0.20.1¶
Bug fixes¶
Fix nondeterminism that could sometimes cause duplicate SHACL results
SHACL could sometimes contain extra results for deeply nested shape graphs
SHACL graph no longer includes conforms bool for results, this one is just needed when include_details=True
v0.20.0¶
Breaking changes¶
map_triples, map, map_default “df”-arg renamed to “data” as it now takes alternatively a DataFrame or a SolutionMappings object, in which case types can be specified for columns. A query run with solution_mappings=True can thus be piped into these methods.
parameters for query and update for the PVALUES construction now is a dict with SolutionMappings values.
ValidationReport no longer has method graph(), instead, use the report_graph argument of validate to specify a named graph for the report. This also means that validate() argument include_shapes_graph is gone.
Features¶
Rudimentary shacl rules support (only SPARQL construct), no recursion yet. Use the inferences_graph-argument to validate to persist inferred triples to a named graph.
Bug fixes¶
Fix for #48
v0.19.25¶
Features¶
SPARQL parser returns the bad part of the query
Bug fixes¶
Fix for FTS issue when enabling for multiple named graphs
Fix lang string issue with FTS
v0.19.24¶
Bug fixes¶
Fix for missing triples in pretty turtle serialization under certain conditions involving orphaned blank nodes.
v0.19.23¶
Features¶
Implement md5, sha1 and encode_for_uri functions.
Improvements¶
General improvements in read / map performance
Improvements in error handling for PValues, missing Function implementations
Bug fixes¶
Fix bug in recursive constraints in SHACL
Issue when using Datalog with SPARQL construct syntax
Issue with lang string literal in SPARQL construct template
v0.19.22¶
Improvements¶
Graceful error handling for SPARQL queries that crash in Polars.
Bug fixes¶
Contains function in SPARQL crashed with lang strings under certain circumstances
v0.19.20¶
Improvements¶
All types with string types are now internally represented as u32 (previously it was only xsd:string, blanks and iris)
Bug fixes¶
Result path missing in SPARQL constraints
v0.19.19¶
Bug fixes¶
Bug fix for issue affecting SPARQL constraints with lang string messages with unbound vars of multiple types.
v0.19.18¶
Features¶
CIM XML import removes superfluous “#_” from iris.
v0.19.17¶
Features¶
Basic support for disk based storage of categorical maps.
Fix extra characters in CIM XML URIs by default
Bug fixes¶
Multiple bugfixes for SHACL
v0.19.14¶
Features¶
read() and reads() for json-ld is now supported. Note that contexts are not resolved against the internet but must be provided in a dict due to security concerns.
Improvements¶
Pretty turtle writing has been parallelized, improving performance.
v0.19.13¶
Features¶
CIM XML parsing is now natively supported
Improvements¶
Added possibility of setting number of triples per batch when reading serialization formats.
map_json is now compliant with Facade X.
Bugfixes¶
Fixed various cases where a panic or unwrap happens instead of a proper error.
v0.19.10¶
Features¶
m.map_json() maps a json file or string to rdf triples.
Bugfixes¶
Writing turtle could sometimes write erroneous triples.
v0.19.9¶
Improvements¶
maplib.explore()is deprecated, use the explore method on aModelinstead.Model.explore()no longer supports opening a browser window automatically. Usewebbrowser.open(s.url, new=2)instead. Wheresis returned fromexplore()Release of pretty turtle printer. It is still missing:
Anonymization of blank nodes that only have a single incoming reference.
Support for nested lists
Datalog inference now works on SPARQL construct in preparation for SHACL rules support
Improvements to prefix handling:
Default prefixes only shadowed by adding prefixes to Model
Prefixes set in pretty turtle print only shadow Model prefixes
Bugfixes¶
Fixed improper serialization of e.g. newlines in new pretty turtle printer
Fixed issue where empty triples could be added
Fixed several issues that could lead to OOM in the SHACL engine
Fixed panic in CIM xml exporter that should be error message
v0.19.6¶
Improvements¶
IRIs are now compressed in memory
SELECT * has column ordering according to when the variable is used in the query, not alphabetical as previously.
v0.19.5¶
Bugfixes¶
Inverse predicates missing from CIM exports
Improvements¶
Setting debug=True in query now prints the debugging of why the query has no solutions.
v0.19.4¶
Bugfixes¶
Fix issue where byte datatypes were incorrectly created with the polars string datatype.
Improvements¶
Template printing now uses prefixes.
Exception handling for missing functions.