Change log

v0.23.1

Bug fixes

  • maplib Enterprise Edition: FILTER EXISTS and FILTER NOT EXISTS over virtualized time series (chrontext) failed with “Parts of the query were not evaluated” or, for an EXISTS in the condition of an OPTIONAL, gave wrong results, since v0.23.0. The EXISTS pattern is again evaluated where the virtualized parts of the query are found, and only once

  • The source distribution of v0.23.0 left out the vendored oxttl and sparesults crates, so the Linux wheels could not be built and are missing from PyPI for v0.23.0; v0.23.1 has wheels for every platform

Performance

  • A small DELETE DATA (or a DELETE that removes few triples) takes time in proportion to the triples it deletes instead of to the size of their tables: the rows are found through the subject and object indices and cut out in place, instead of an anti-join, a sort and a rebuild of the whole table. Deleting one triple from a model of 1.4M triples takes about 2 ms in memory, hybrid and disk storage, down from 16, 41 and 52 ms, and no longer grows with the model

  • Writing Turtle, RDF/XML and JSON-LD keeps a bounded number of triples (about 64k) in memory at a time instead of the dictionary strings of the whole graph, and N-Triples, N-Quads and TriG are written from slices of each table. Writing 117 MB of N-Triples as Turtle peaks at 68 MB above the model instead of 587 MB. The output is the same

  • Reading files with many predicates has no fixed cost per predicate: string columns are built without polars’ minimum reservation per parser thread and predicate, the builders of the threads are appended per table, and deduplication is eager. Reading a file with 2,000 predicates keeps 204 MB instead of 507 MB (139 MB for the same triples with 4 predicates), and is 2.3 to 3.4 times faster

v0.23.0

Breaking changes

  • Model.reason(profile, ...) (maplib Enterprise Edition) is now Model.infer(profile=..., ...), with the same parameters (graph, target_graph, include_graphs, max_violations, max_rows), and still returns a ReasoningResult. Model.infer takes either rulesets or profile, and raises if it gets both or neither, or target_graph or max_violations with rulesets. Model.reason is removed. In Rust, Model::reason is unchanged

  • Model.update(update, graph=G) writes to G: G is the default graph of the update, as with WITH <G>, so triples without a GRAPH in the INSERT and DELETE templates, INSERT DATA and DELETE DATA are inserted into and deleted from G, and CLEAR DEFAULT clears it. Before, the WHERE clause was evaluated on G, but the triples were inserted into and deleted from the default graph. Updates sent to Model.serve(graph=G) (maplib Enterprise Edition) write to G in the same way

  • A misspelt magic predicate, e.g. maplib:shortestpath or maplib:Path (a different case, or for the longer names up to two typos, of path, shortestPath, component, bridge, articulationPoint, fts or vectorSearch), and maplib:fts without a full text index on the queried graph, are errors when no triples use them, saying what to do. They gave no solutions, silently. Other predicates in the maplib namespace (https://datatreehouse.github.io/maplib/vocab#) are ordinary data, e.g. the time series, serves, summary and config terms, and give no solutions when a model has none

  • STRSTARTS, STRENDS and CONTAINS with a second argument from the data that is not an xsd:string (an IRI, a number, an unbound value) give an unbound value, as SPARQL errors do, instead of failing the query. A constant of another type is still an error

Features

  • query(bindings=...) and query_page(s) take Python bool, int and float values, bound as xsd:boolean, xsd:integer and xsd:double literals. They were ignored, silently, so e.g. FILTER(?v >= ?min) with bindings={"min": 10} had no effect or no results. Values of other types raise an error

  • Model.update(..., bindings=...) binds variables of the WHERE clauses and of the INSERT and DELETE templates, as query(bindings=...) does, so parameterised updates need no string templating. In Rust, Model::update_keep_triples, Model::update_with_federation_keep_triples and Triplestore::update_keep_triples take the bindings, and query_processing::bindings::maybe_replace_bindings_update replaces them in a parsed update

  • DESCRIBE queries in Model.query, which raised “Query type not supported”: the result is the triples whose subject is a described IRI (constant IRIs and the IRIs the described variables are bound to, or every variable for DESCRIBE *), as a CONSTRUCT result. Blank nodes are not described. This is what the SPARQL endpoint of Model.serve gives

  • REGEX with a pattern or flags that differ per solution, e.g. REGEX(?label, ?pattern) or REGEX(?v, CONCAT("^", ?prefix)), and STRBEFORE and STRAFTER with a second argument that differs per solution. They raised “The following expression should resolve to a constant literal”. An invalid pattern from the data gives an unbound value for its solutions, while an invalid constant pattern is still an error

  • GraphSnapshots(..., exclude_graphs=[None]) (maplib Enterprise Edition) leaves graphs out of the snapshot (None or "" is the default graph): they are never written, restored or loaded from other writers, and writes to them are not pending. The next snapshot drops an existing entry of such a graph from the manifest, and skipped_graphs names the ones the last restore() did not load. on_refresh(callback) is called with the graphs other writers changed, under the write lock, so that an application can keep a graph it derives from them up to date. The writable Model portal (maplib Enterprise Edition) uses them with a Graph portal that rebuilds its default graph from the named graphs (run_portal(rebuild_default_graph=True)): the default graph, a copy of all uploaded data, is no longer written on every write, which rewrote a 27 MB file per upload for a default graph of 1.6M triples (274 ms locally, now 1 ms and 10 kB). Triples written only to the default graph are then not kept across restarts. PORTAL_SNAPSHOT_DEFAULT_GRAPH=1 keeps it in the snapshot. In Rust, GraphSnapshotOptions::exclude, GraphSnapshots::set_on_refresh, skipped_graphs and server::OnRefresh

  • The writable Model portal (maplib Enterprise Edition) answers a request that may write (POST, PUT, PATCH, DELETE, including SPARQL Update) only once its changes are published to MODEL_URL, at most PORTAL_DURABLE_WRITE_TIMEOUT_SECONDS (30) later, so that an acknowledged write survives a crash such as an OOM kill; the X-Maplib-Snapshot header of the answer says whether they were (published, none, pending or failed). Before, writes were published 2 to 7 s after they were answered. PORTAL_DURABLE_WRITES=0 answers at once

  • One set of default prefixes: representation::prefixes::WELL_KNOWN_PREFIXES (Rust) is maplib’s only table of default prefixes, read in Python with maplib.well_known_prefixes() (name to namespace). It merges the two Rust lists there were (the parsing defaults of a model, get_default_prefixes, and the 14 prefixes of template generation) and adds what the Graph portal and maplib.config used to define themselves: rdf, rdfs, xsd, owl, sh, skos, foaf, ottr, maplib, fx, xyz, def, dcterms, dc, prov, dcat, time, sosa, ssn, vann, void, schema, geo, geof, sf, uom, wgs, qudt, unit, quantitykind, cim, cim16, eu, entsoe, md, dm, opc, di, machinery, robotics, ia, era and mcfg. cim is CIM100 (http://iec.ch/TC57/CIM100#); CIM16 is cim16. Every model now knows all of them when parsing SPARQL, Datalog and OTTR (a PREFIX declaration or Model.add_prefixes still wins), so Turtle, TriG, JSON-LD and RDF/XML written by a model abbreviate these namespaces with them; the RDF/XML and CIM/XML writers also use them for predicate namespaces without a prefix before falling back to ns1, ns2, …. maplib.config.WELL_KNOWN_PREFIXES (namespace to name) is read from this table, and PREFERRED_PREFIXES is gone: generated prefixes name CIM100 cim and CIM16 cim16. The hint for a bare word in a SPARQL syntax error always uses ex: as its example prefix

  • Prefixes in the config graph: maplib.config.PrefixConfig, write_prefix, write_prefixes, read_prefixes, remove_prefix and apply_prefixes keep the prefixes of a model in its config graph as SHACL prefix declarations (sh:declare, sh:prefix, sh:namespace) of the owl:Ontology mcfg:prefixes, so they travel with the model; apply_config adds them to the model (Model.add_prefixes). generate_prefixes(m, graph=None, max_generated=8) makes prefixes for the data in a graph (or a list of graphs): the usual prefixes of the well-known vocabularies it uses (WELL_KNOWN_PREFIXES: rdf, rdfs, xsd, owl, sh, skos, dcterms, prov, geo, cim, opc, …) and short names for the other namespaces it uses most (prefix_name_for, namespace_of). See “Prefixes” in CONFIG_GRAPH.md

  • Graph algorithms as magic predicates: ?n maplib:component ?o (connected components, or strongly connected components with maplib:strong true), ?a maplib:bridge ?o (edges whose loss splits their component, with maplib:object for the other end) and ?n maplib:articulationPoint ?o (nodes whose loss does), with the outputs maplib:componentId (numbered from 1 by decreasing size), maplib:componentSize, maplib:cutOff (the nodes cut off) and maplib:edgePredicate. The edges are given as for maplib:shortestPath (predicate, inversePredicate, anyPredicate, excludePredicate) or, new, by the rows of other patterns of the group with maplib:edgeSource ?a ; maplib:edgeTarget ?b, so derived edges (e.g. between the substations at the ends of a line) need no materialization. maplib:avoid leaves nodes out of the graph, which is a what-if (open switches, equipment out of service) without changing the model. maplib:countNodes makes only some nodes count in componentSize and cutOff (e.g. customers cut off). A subject bound by the group restricts the rows to those nodes, and nodes among them without edges are components of their own. The algorithms are union-find, iterative Tarjan and an iterative low-link search over the cached integer graph of maplib:shortestPath. See “Graph algorithms” in API_USAGE.md

  • Cutting part of a CIM grid out of a model (maplib Enterprise Edition, the cim package): cim.scissors.cut_to_graph(m, interior_equipment_iris, border_equipment_iris, target_graph=..., graphs=[...], normal_topology=True, max_distance=None, context_hops=10) cuts the grid around the interior equipment, stopping at the border equipment, into a new graph and returns counts (members, non-members, context, boundary equipment, triples). It is now SPARQL that maplib evaluates instead of Datalog rules and Python loops: the reachable topology is the connected component of the interior equipment over the Terminal.ConductingEquipment and Terminal.ConnectivityNode edges with the border, normally open switches and equipment normally out of service avoided (maplib:component with maplib:avoid), or with max_distance the paths of at most 4 x max_distance edges (maplib:shortestPath), and the context is added with one INSERT per step. The graphs are read as one dataset (USING/FROM, maplib:DefaultGraph for the default graph), so the instance data, the rdf:type triples, the profiles and the CIM vocabulary may each be in a graph of their own. cim.cut keeps its signature and returns the cut as a Model, with the same members, non-members and types as the Datalog rules; it no longer copies their dt:interiorDistance bookkeeping literals into the cut, and fail_distance and max_iterations are not used any more. The Model portal’s Map sends the same queries to cut a drawn area out of the model

  • The serves relations of the cim package (maplib Enterprise Edition): cim.add_serves(m, ..., graphs=[...], target_graph=...) reads the CIM data from several graphs as one (the profiles and the rdf:type triples may each be in a graph of their own) and adds maplib:normallyServes* and maplib:actuallyServes* to the target graph. Its helper predicates, which it removes when it is done, are in the cim package’s own namespace (https://www.data-treehouse.com/cim/servesHelper_) instead of the maplib namespace

  • Memory limits per query: a timeout stops a runaway query on time but does not bound its memory, since a join can produce gigabytes of rows per second. Model.query(max_memory_mb=) and Model.update(max_memory_mb=) stop the query with the new QueryMemoryLimitError (a MaplibException) when the memory used by maplib has grown by more than that many megabytes since the query started. It is checked like the timeout, for every batch of rows a join gives, so the memory goes over the limit by at most the batches in flight. maplib.set_memory_limit(max_query_memory_mb=) sets a default limit for every query and update, also for the queries of Model.serve (maplib Enterprise Edition), which then answer with an error instead of exhausting the memory of the server. Unlike the limit from abort_above_mb, which stops queries when the whole process is near it, this limit is per query; the memory counted is that of the whole process, so with queries running at the same time it is approximate. In Rust, CancelToken::with_memory_limit, utils::cancel::set_default_query_memory_limit and Cancelled::QueryMemoryLimit

  • The XPath and XQuery Functions 3.1 math functions math:pi, math:exp, math:exp10, math:log, math:log10, math:pow, math:sqrt, math:sin, math:cos, math:tan, math:asin, math:acos, math:atan and math:atan2 (namespace http://www.w3.org/2005/xpath-functions/math#), as SPARQL function calls. The examples of the specification are run as compliance tests. See “XPath math functions” in API_USAGE.md

Bug fixes

  • EXISTS and NOT EXISTS with a variable that a solution leaves unbound, e.g. from an OPTIONAL, UNION or VALUES with UNDEF, treated the unbound value as matching nothing, so ?s ex:q ?v OPTIONAL { ?s ex:p ?o } FILTER NOT EXISTS { ?o ex:r ?s } kept solutions whose ?o is unbound although ?o ex:r ?s has a solution. SPARQL 1.1 substitutes only the variables a solution binds, so an unbound variable stays free in the pattern and matches any value; the solutions are now split by which of the pattern’s variables they leave unbound, and each part is evaluated without them. EXISTS no longer collects its incoming solutions to number them, which kept them all in memory: it is answered for the distinct values of the variables the pattern mentions and joined back on them, within the query’s plan

  • Unary minus on an xsd:double or xsd:float is IEEE negation, so -(0.0e0) and the literal -0.0e0 are negative zero, as XPath op:numeric-unary-minus says

  • maplib Enterprise Edition: models with predicates of the legacy chrontext vocabulary are encoded under the maplib names they are stored as, when they are inserted and deserialized, so queries that bind the predicate (?s ?p ?o) or use a zero-length path over them no longer panic

  • GROUP BY a variable that the grouped pattern does not bind, e.g. SELECT ?country ?product (AVG(?price) AS ?avg) WHERE { ?o ex:product ?product ; ex:price ?price } GROUP BY ?country ?product (BSBM business intelligence query 5), failed because the variable had no column. It now groups as one unbound key, as SPARQL 1.1 (18.5.1) specifies, and such a variable from a sub-select joins with any value of it on the other side, as an unbound variable is compatible with every value

  • A join, OPTIONAL, MINUS or EXISTS of two patterns that both have IRIs, blank nodes or literals that are not in the model (from VALUES or BIND, e.g. VALUES ?x { <urn:a> } VALUES ?x { <urn:b> }) compared their codes in two local dictionaries, which may give different values the same code, so different values matched and equal values could fail to. The right side is now re-encoded into a dictionary that extends the left one. This also made maplib:avoid with an IRI that is only in the edges from VALUES leave out another node, or none

  • A FILTER in a nested group, e.g. { :x :p ?v . { FILTER(?v = 1) } }, and the condition of an OPTIONAL in a nested group saw the variables of the enclosing group, which SPARQL leaves unbound there (a filter applies to its own group), so they kept solutions they should remove or bound optional variables they should leave unbound. They are now evaluated without the enclosing solutions and joined with them afterwards. Inside EXISTS and LATERAL, where the outer solution is substituted into the pattern, they still see it

  • A property path with a constant of a type its values do not have at one end, e.g. <a> <b>+ 1.0 or a literal object of a path of IRIs, panicked with a SchemaMismatch; it now has no solutions

  • An empty rdf:parseType="Literal" property element in RDF/XML, e.g. <ex:p rdf:parseType="Literal"/>, was a syntax error (“No value found for rdf:XMLLiteral value”); it is now the empty XML literal, as the RDF/XML grammar says

  • A FILTER over a group with a variable predicate was dropped when the filter pushdown rewrite moved it to a VALUES block, a property path, GRAPH, SERVICE or triple patterns without a variable predicate, so the query returned the rows it should remove: in SELECT ?n WHERE { ?a ?p ?b . VALUES ?p { <urn:p> } ?a <urn:name> ?n FILTER(?n = "A") } the FILTER was ignored. A filter over the variables of two patterns of a group could also be moved next to one of them and run before the other had bound its variables, which removed every row, and a filter on a variable bound only by an OPTIONAL was moved into it. A filter is now moved only into a pattern that binds all of its variables in every solution, and is otherwise kept above it. The W3C SPARQL 1.0 tests now run every test of the suite (491), including the six that two manifests written as a blank node hid from the runner

  • Arithmetic (+, -, *) on a variable that holds numbers of many datatypes, e.g. ?l + ?r on the objects of a predicate used with 16 numeric datatypes, used memory without bound (more than 3 GB within seconds for one row) while planning the query: the result of each pair of datatypes was combined with the ordering of COALESCE, whose expression grows with the square of the number of pairs (256). The pairs are now combined without ordering, as at most one of them has a value in a row

  • The datatype of the result of +, - and * follows the numeric type promotion of SPARQL (XPath op:numeric-add etc.): xsd:integer < xsd:decimal < xsd:float < xsd:double, where every type derived from xsd:integer counts as xsd:integer. So xsd:short + xsd:short and xsd:int * xsd:long are xsd:integer (they were xsd:short and xsd:integer, with the 16-bit sum able to overflow), xsd:double + xsd:decimal is xsd:double and xsd:float + xsd:decimal is xsd:float (both were xsd:decimal)

  • The W3C SPARQL 1.0 (DAWG) tests and the RDF 1.0 Test Cases (2004) are run by the test suite (test_w3c_sparql10.py, test_w3c_rdf10.py), with the tests maplib does not pass yet listed as known failures

  • A join on a variable whose values are of several kinds on both sides that share only one kind, e.g. IRIs and literals on one side and IRIs and blank nodes on the other, matched the values of the other kinds with each other: "v" joined _:b. This gave wrong rows for OPTIONAL in SPARQL and unsound inferences in Datalog (maplib Enterprise Edition), where new triples are joined with stored ones: with eq-sym and eq-trans and in Model.infer(profile="owl-rl") and profile="rdfs-plus", a literal or IRI in a group of owl:sameAs resources was made the same as an unrelated blank node. The rows of the other kinds are now left out of such joins

  • maplib:component, maplib:bridge and maplib:articulationPoint no longer count an unbound value as a node when the edges, maplib:avoid or maplib:countNodes come from solutions where some rows leave them unbound (e.g. a UNION) and the column holds values of several types: such a row added a node without a value as a component of its own.

  • Security advisories: the RDF/XML parser and the SPARQL XML results parser use quick-xml 0.41, which fixes quadratic run time on duplicate attribute checks (RUSTSEC-2026-0194) and unbounded memory for namespace declarations (RUSTSEC-2026-0195) on untrusted XML. sparesults is vendored in lib/sparesults for this, like oxrdfxml. zstd is updated to 0.13.4 (RUSTSEC-2026-0391, RUSTSEC-2026-0398, RUSTSEC-2026-0403 and the unsound zstd-safe advisories). In maplib Enterprise Edition, DuckDB is updated to 1.5.6 (duckdb crate 1.10506.1), whose build no longer uses zip 6 (RUSTSEC-2026-0367, -0368, -0369 and -0429)

  • Comparisons (<, <=, >, >=) of string literals stored in the model, and MIN and MAX of stored strings and IRIs (also in window functions), used the order in which the values were first added to the model instead of codepoint order, e.g. MIN(?label) over "urn:n/10" and "urn:n/1" gave "urn:n/10", and FILTER(?a < ?b) held for ("urn:n/10", "urn:n/1"). < between string literals from VALUES was false in both directions. Strings computed in the query were not affected

  • A variable whose name is another variable’s name followed by l, v, I, B or N (e.g. ?o and ?ol) failed with “unable to find column” (and panicked in Model.serve) when ?o holds values of several types, as with IRIs and literals: the internal columns of ?o’s types had the other variable’s name

  • m.add_graph(m, source_graph=..., target_graph=...), with the Model as its own source, raised an error (earlier: hung). It now copies the source graph into the target graph of the Model

  • A SELECT query without variables, e.g. SELECT * WHERE { <a> <p> <b> }, gave an empty DataFrame also when the pattern matched. It now gives one row (with no columns) when it matches and none when it does not

  • SELECT (YEAR(?d) AS ?y) ... GROUP BY (YEAR(?d)) gave an unbound ?y: an expression in the SELECT clause that is a GROUP BY expression is now the value of the group, as in other SPARQL engines

  • STRDT with a lexical form that is not valid for the datatype failed the whole query (“Error parsing literal to target type in STRDT function”); it now leaves that solution’s value unbound, as SPARQL errors do

  • Polars Categorical and Enum columns, e.g. in the results of a Python virtualized database or in a DataFrame given to map_triples, are read as String columns. They failed with “Unknown datatype Categorical” (a panic for virtualized databases)

  • maplib Enterprise Edition: every wheel includes the cim package. It was only added by the staging step of release builds, so other builds (e.g. maturin develop, the manylinux script and the benchmark image) left it out

  • Casts of a literal of a datatype maplib stores in its dictionary other than xsd:string, e.g. xsd:string("5389.96"^^bsbm:USD), used the value’s dictionary code instead of its lexical form: that gave "1795", and xsd:float(xsd:string(?price)) the code as a number, so BSBM business intelligence query 4, which averages it, gave wrong answers. Such values are now decoded before they are cast. A language-tagged string was cast from the dictionary codes of its value and language: xsd:string("x"@en) gave "{1,0}", and xsd:date and xsd:dateTime of one gave a pair of dates in 1970. It now casts to xsd:string as its lexical form, and every other cast of it is an error, which leaves the result unbound, as SPARQL says. A chain of casts of one such variable is evaluated once per distinct value and gathered back, instead of decoding and parsing every row

  • FILTER EXISTS and FILTER NOT EXISTS evaluated their pattern once per solution, carrying every column of the solutions through it, and then joined a boolean column back and filtered on it. When the variables the pattern shares with the solutions are bound in all of them, the pattern is now evaluated once per distinct combination of their values, and the solutions are semi or anti joined with the combinations that pass. When the filter is over two patterns that share no variables and the shared variables come from both, as in BSBM business intelligence query 4 (FILTER NOT EXISTS { ?product bsbm:productFeature ?feature } over every feature and every offer), the combinations of the two sides’ values are answered and joined with each side, so the cross product of the two is not built. With the cast fix above, that query’s peak memory on 300 products went from 112 MB to 61 MB

  • A query that timed out during a join much larger than estimated, e.g. ?s ex:kind ?k . ?t ex:kind ?k with few kinds, returned the timeout error, but the join went on in the background, and when it ended the process was aborted (Polars sent its result to a receiver that was gone). Such a query is now abandoned safely, and joins whose inputs could give many rows (more than 16.7 million multiplied together) run on the streaming engine when the query can be stopped, where a timeout stops the join itself. With 15 000 subjects and a 20 ms timeout the peak memory is 0.53 GB instead of 1.35 GB followed by a crash

  • With maplib.set_memory_limit(abort_above_mb=...), a query raises a MemoryError when the process uses more than 90% of the limit, checked as the solutions of its joins are produced and between the steps of the query, instead of the process being aborted (unless a single step goes over the rest of the limit at once). In Rust, this is utils::cancel::Cancelled::MemoryLimit, and Model.query makes a cancel token whenever such a limit is set. A timeout limits time, not memory: a join can produce several GB per second before a timeout of 0.5 s fires

  • Patterns of a basic graph pattern that share two or more variables with the patterns before it took memory quadratic in the number of solutions: in ?s ex:kind ?k ; ex:region ?r . ?t ex:kind ?k ; ex:region ?r the patterns were joined one at a time, so ?t ex:kind ?k was joined on ?k alone, and every subject met every other subject of the same kind before ?r was checked. The results were right. Such patterns are now joined on all the shared variables at once, each first restricted to the values bound so far. With 15 000 subjects, 3 kinds and 1000 regions the query needs 0.12 GB instead of 2 GB

  • A query of Model.query(..., timeout=...) or Model.serve (maplib Enterprise Edition) that timed out, or whose HTTP client disconnected, went on for 0.5 s to several seconds when it was evaluated with the streaming engine, e.g. a large join with a FILTER, as Polars finishes the batches it is working on. Meanwhile it held the model, so a following update waited for it. It now returns as soon as it is stopped, and the batches are finished on a thread of their own

  • With maplib.set_memory_limit(abort_above_mb=...), a query started right after one that raised MemoryError often raised MemoryError too, at once: the stopped query returns before the join it was running has ended and freed its memory. A query now first waits for the work of stopped queries to free their memory, while the process uses more than 90% of the limit (until its timeout, at most a minute). The memory limit of a query of its own (max_memory_mb, or the default for every query) counts the growth from the least memory seen since the query started instead of from when it was made, which could be while a stopped query was still freeing hundreds of MB: such a query was then stopped only by the process limit, with a MemoryError, instead of by its own limit. In Rust, utils::cancel::with_token waits, and utils::cancel::Leftover counts the work that a stopped collect leaves running

Performance

  • Reading Turtle, TriG, N-Triples and N-Quads is faster. The parsers (oxttl) are vendored in lib/oxttl and recognize prefixed names made of ASCII characters without decoding them character by character, check common absolute IRIs with a fast path before the full IRI parser, and remember recently seen IRIs so that repeated predicates and classes are not checked again. Turtle and TriG gain the most; N-Triples and N-Quads gain with checked=True, the default. The triples, the errors and their positions are unchanged. N-Quads are read in parallel by default, as N-Triples are (parallel=False reads them in one thread)

  • Model.map_json and Model.map_xml with many documents (a list or iterable of strings or paths, eagerly or as a stream) add the triples of up to a thousand documents at once instead of one document at a time, which made mapping many small documents many times slower than mapping the same data as one document. The triples, their IRIs and the uuid namespace of each document are unchanged, and when a document fails, the documents before it are still mapped. In Rust, Triplestore::map_documents and Model::map_documents map several documents at once

  • Vector literals (maplib:vector) are stored in the dictionary as the bits of their 32 bit floats, packed into seven bits per character, instead of as their lexical text, which takes about two fifths of the memory for typical embeddings. The lexical form is written only when a literal is output (query results, N-Triples, Turtle and the other serializations, STR). Mapping Polars array columns packs the floats directly instead of formatting the lexical forms and storing those, so mapping embeddings no longer holds the text of all of them at once. Only canonical lexical forms (as maplib writes them: the shortest decimal of each component, no spaces) are packed; other lexical forms, e.g. written by hand with spaces or exponents, are stored as they are, so every literal keeps its exact lexical form, and literals with the same lexical form are the same term as before. Vector search and the vector functions (maplib Enterprise Edition) read the floats of packed literals without parsing text. In exchange, outputs that contain many vector literals as text (query results, N-Triples and Turtle) format the floats when they are written, and reading canonical lexical forms (e.g. from N-Triples) packs them, so both take longer than copying text did; serializing and deserializing models takes less time, as the dictionaries are smaller. Serialized models write packed vector literals to their parquet dictionaries, which older versions of maplib read as text; models serialized by older versions are read as before. ORDER BY on vector literals orders them by their stored form, which is still a fixed order but no longer that of the lexical forms. Dictionaries on disk (maplib Enterprise Edition) keep the lexical forms. In Rust, in_memory::vectors has the packed form and the lexical form conversions, CatMapsInMemory::new_empty_kind creates a dictionary of vector literals, CatMaps::decode_vectors_batch gives the floats of encoded vector literals, and representation::vectors::vector_columns_to_literals takes the columns that keep their lexical forms

  • Reasoning and Datalog inference into a graph other than the one reasoned over (Model.infer(profile=...) into the transient graph or a target_graph, and SHACL rules, maplib Enterprise Edition) leave out the inferred triples that the graph already has while inserting them, by comparing the encoded triples, instead of running a SPARQL query for each predicate of each rule execution. The inferred triples are unchanged. In Rust, Triplestore::add_triples_vec_excluding and insert_construct_result_excluding add triples except those a graph already has

  • Model.infer with the profiles "owl-rl" or "rdfs-plus" (maplib Enterprise Edition) handles owl:sameAs between individuals with a canonical representative per group of equal resources, found with union-find, instead of copying every triple to every equal resource in each round with SPARQL queries over all triples. The rules run on the triples of the representative, and its triples are copied to the other members of the group once at the end, so large groups no longer multiply the work of every round, and eq-sym and eq-trans are not run as rules. Groups with blank nodes, literals, classes, properties or other resources that the schema refers to are still replaced in every round. The inferred triples and violations are unchanged; the number of rounds and rule executions in the ReasoningResult can differ

  • Reading RDF/XML and CIM/XML (Model.read(..., format="rdf/xml") or "cim/xml") is faster. The parser resolves and checks each element and attribute name once per document instead of at every use, resolves same-document references such as rdf:ID="_..." and rdf:resource="#_..." without the general IRI resolution, and skips the whitespace between elements without copying it. Large documents are split for parallel parsing by scanning for markup instead of tokenizing the whole document first, and CIM/XML reads build their datatype map once instead of for every part of a document. The triples read are unchanged

  • Several virtualizations (maplib Enterprise Edition): GROUP BY and aggregates over time series in different resources, e.g. in several virtualized databases, are pushed down into each database again, instead of reading the raw data points and aggregating them in maplib. Each database computes the aggregates of its part of each group, with AVG as a SUM and a COUNT, and maplib combines them per group. COUNT, SUM, AVG, MIN, MAX and SAMPLE are pushed down this way, also with time buckets such as FloorDateTimeToSecondsInterval. COUNT, SUM and AVG with DISTINCT and GROUP_CONCAT are still evaluated in maplib, and a HAVING or ORDER BY on such a group is evaluated after the aggregates are combined

  • Mapping, updates, inserts and writes of small batches, as in the steps of in-place model streams (stream=True, in_place=True) and Model.scratch_graphs loops, have a smaller fixed cost. Template expansion, decoding of dictionary codes, CONSTRUCT results that are inserted, splitting of multi-type columns and the N-Triples and N-Quads writers collected their small intermediate results with Polars’ streaming engine, which costs more per collect than the work on a few thousand rows. Results of up to 65,536 rows are now collected with the in-memory engine, as query results already were. The result of a query is also picked up sooner after it is ready: the wait between checks grows by a quarter up to 0.25 ms instead of doubling up to 1 ms. With hybrid and disk storage (maplib Enterprise Edition), writing a graph now decodes each column of each table once instead of once per chunk of the streaming engine

  • maplib Enterprise Edition: with hybrid and disk storage, the dictionaries on disk keep the strings and codes used recently in memory, so mapping, updates, inserts, queries and writes of small batches look up only the strings and codes they have not used recently. Each lookup otherwise ran DuckDB queries that read the whole dictionary, several times per batch, so the cost of a batch grew with the dictionary rather than with the batch. Ranking codes in string order (when triples are added) decodes them and sorts the strings, instead of sorting the dictionary in DuckDB. Strings added to a dictionary are kept with their new codes, since they are usually looked up right after. The cache holds at most about a million strings with their codes (two generations of 524,288; the older generation is dropped when the newer one is full) and is cleared when the dictionary is compacted. MAPLIB_DISK_DICTIONARY_CACHE sets the number of strings per generation, and 0 turns the cache off

v0.22.1

Breaking changes

  • Model.update returns what the update changed instead of None, see Features. In Rust, UpdateResult has a changes field, and Triplestore::delete_construct_result and delete_triples_vec take keep_triples and return the deleted triples (triplestore::changes::Deleted) instead of nothing

  • IRI(), URI() and fn:castToIRI give an unbound value for a string that is not a valid absolute IRI (a relative IRI, or one with spaces or other characters IRIs do not allow), as SPARQL 1.1 specifies for errors, instead of an invalid IRI term (#60)

  • The time series virtualization vocabulary moved from the chrontext namespace (https://github.com/DataTreehouse/chrontext#) to the maplib vocabulary (https://datatreehouse.github.io/maplib/vocab#, prefix maplib): maplib:hasTimeseries, maplib:hasDataPoint, maplib:hasTimestamp, maplib:hasValue, maplib:hasResource, maplib:hasExternalId, and the functions maplib:FloorDateTimeToSecondsInterval, maplib:dateBin, maplib:DateTimeAsMicros, maplib:MicrosAsDateTime, maplib:DateTimeAsSeconds, maplib:SecondsAsDateTime, maplib:modulus, maplib:time_bucket and maplib:nestAggregation. None of the names clash with the existing terms of the vocabulary. The bundled templates and data (maplib.ip21, OSIsoft PI, Siemens Insights Hub, CDF, OPC UA, the config graph) use the new IRIs. maplib.config.MAPLIB is the namespace; maplib.config.CHRONTEXT is kept as a deprecated alias of it, and maplib.config.LEGACY_CHRONTEXT is the old namespace. In Rust, representation::constants::LEGACY_CHRONTEXT_PREFIX_IRI and from_legacy_chrontext_iri give the old namespace and its mapping.

  • Deprecated: the old chrontext IRIs are still accepted and read as the maplib IRIs with the same local name: in SPARQL queries and updates (when parsing), in the predicates of data that is added or deserialized (they are stored as the maplib predicates, so queries and exports show the new IRIs), and in the templates of virtualizations. The chrontext prefix that map_osipi and map_insights_hub add still points to the old namespace, next to maplib. IRIs of the old namespace in the object position of data (e.g. sh:path in a shapes graph) are not rewritten. Custom Python virtualized databases that compare function IRIs, e.g. with https://github.com/DataTreehouse/chrontext#FloorDateTimeToSecondsInterval, now get the maplib IRI and must compare with that.

  • AVG over values of one integer type is an xsd:decimal, as SPARQL defines it, and AVG over xsd:float values is an xsd:float. It kept the type of its input, so the average was right but expressions using it were computed in integers: MAX(?x) - AVG(?x) over 1, 2, 3 and 3 gave 1 instead of 1.25

Features

  • Reasoning profiles (maplib Enterprise Edition): Model.reason(profile="owl-rl" | "rdfs-plus" | "rdfs", graph=None, target_graph=None, include_graphs=None, max_violations=1000) materializes the entailments of a built-in ruleset with the Datalog engine and returns a ReasoningResult with the number of inferred triples and the violations found. "owl-rl" implements the OWL 2 RL/RDF rules of OWL 2 Profiles (section 4.3), including owl:propertyChainAxiom, owl:hasKey, owl:intersectionOf, owl:unionOf, owl:oneOf, the RL forms of owl:someValuesFrom, owl:allValuesFrom and owl:hasValue, and max (qualified) cardinality 1, but not eq-ref (every term owl:sameAs itself) and the datatype rules other than dt-type1. The rules with false in the head (e.g. cax-dw, prp-irp, cls-nothing2, eq-diff1) are checked after inference, and their matches are returned in ReasoningResult.violations as dicts with the rule name, a message and the terms. "rdfs-plus" is a cheaper preset in the style of GraphDB’s RDFS-Plus (optimized): the RDFS rules without axiomatic and reflexive triples, with owl:inverseOf, owl:SymmetricProperty, owl:TransitiveProperty, owl:equivalentClass, owl:equivalentProperty, owl:sameAs, owl:FunctionalProperty and owl:InverseFunctionalProperty. "rdfs" implements the RDFS entailment rules of RDF 1.1 Semantics (rdfD2 and rdfs1 to rdfs13 with the axiomatic triples), except rdfs4a and rdfs4b. The inferred triples go to the transient graph by default, or to target_graph, and triples that are already in the graph are not added again. The schema is compiled into rules with constant predicates (one or two per property, and one per restriction, chain or key) that the Datalog engine evaluates semi-naively, the equality replacement rules are applied with SPARQL when owl:sameAs triples exist, and both are repeated until the schema no longer changes. The W3C OWL 2 RL test cases, the RDF 1.1 Semantics entailment tests and the SPARQL 1.1 entailment regime tests are included in the tests, see w3c_owl2_rl_known_failures.txt, w3c_rdf_mt_known_failures.txt and w3c_sparql11_entailment_known_failures.txt for the tests that do not pass. In Rust this is datalog::reasoning::reason and Model::reason

  • R2RML: Model.map_r2rml(mapping, database=None, tables=None, base_iri=None, graph=None, ignore_data_errors=False) maps relational data with a W3C R2RML mapping. Each logical table (rr:tableName or rr:sqlQuery) is queried once, from a database in maplib.virtualization, a DuckDB connection or polars DataFrames given in tables, and the term maps are evaluated column by column in parallel, so no rows are handled one by one in Python. It supports constant, column and template term maps, term types, languages, datatypes, rr:class, graph maps and referencing object maps with join conditions, and is tested with the W3C R2RML test cases

  • Microsoft Fabric (maplib Enterprise Edition): Model.map_fabric(ontology, lakehouse=None, eventhouse=None, ...) maps a Fabric IQ ontology to OWL, maps the data bound to it from the lakehouse with a generated R2RML mapping (which it returns), and virtualizes its time series, queried in SQL in the lakehouse or in KQL in the eventhouse. The open-source build raises NotImplementedError

  • Window functions in SPARQL queries (w3c-cg/sparql-dev#47), with the SQL syntax proposed there: ROW_NUMBER(), RANK(), DENSE_RANK(), and COUNT, SUM, AVG, MIN, MAX, SAMPLE and GROUP_CONCAT followed by OVER (PARTITION BY ... ORDER BY ... ROWS BETWEEN ... AND ...), where each part is optional, e.g. BIND(ROW_NUMBER() OVER (PARTITION BY ?group ORDER BY DESC(?value)) AS ?rank). They can be used wherever an expression can: in a SELECT expression, BIND, FILTER and HAVING, and over aggregates in a grouped query (RANK() OVER (ORDER BY DESC(COUNT(*)))). A window function is computed over the solutions of the pattern it is evaluated on: the group of a BIND or FILTER, before it is joined with the rest of the query, or the solutions of the query (after GROUP BY) for SELECT and HAVING, so HAVING (ROW_NUMBER() OVER (PARTITION BY ?g ORDER BY DESC(?v)) <= 3) without GROUP BY selects the top three of each group. As a FILTER applies to its whole group, a window in a BIND also sees the solutions that a FILTER of the same group removes; put the FILTER in a nested group to apply it first. FILTER(?r = 1) on a RANK or DENSE_RANK with one ordering key keeps the first rows of each partition without ranking them. The semantics are those of SQL: ORDER BY orders as the ORDER BY of SPARQL (unbound values first, values of different types as there), rows with equal keys are peers that get the same RANK and DENSE_RANK, unbound PARTITION BY values form a partition, aggregates without ORDER BY are over the whole partition, aggregates with ORDER BY are running aggregates up to and including the peers of the row, and ROWS frames (n PRECEDING, CURRENT ROW, n FOLLOWING, UNBOUNDED) count rows. The aggregates follow the types and numeric promotion of the same aggregates in GROUP BY, also for values of several types. RANGE and GROUPS frames are not supported

  • ORDER BY in GROUP_CONCAT, SAMPLE and custom aggregates (w3c-cg/sparql-dev#9): GROUP_CONCAT(?x; SEPARATOR=", " ORDER BY ?y DESC(?z)) joins the values in that order, SAMPLE(?x ORDER BY DESC(?y)) takes the first value in that order. The ORDER BY may follow the expression or the separator, with or without a ;, or precede the separator (GROUP_CONCAT(?x; ORDER BY ?y; SEPARATOR="."), as in the prototype tests of the SPARQL 1.2 community group, which are included in the tests). With DISTINCT, the first occurrence of each value in the order is kept. In Rust, window calls and ordered aggregates are represented in the algebra as calls to reserved functions in the maplib namespace, see spargebra::window, so that the algebra types are unchanged

  • LATERAL (correlated subqueries, SEP-0006): in ?s a :T LATERAL { SELECT ?s ?o WHERE { ?s :p ?o } ORDER BY DESC(?o) LIMIT 3 } the group after LATERAL is evaluated for every solution of the pattern before it, with that solution’s variables substituted, so it gives e.g. the top N per entity or an aggregate per entity. A sub-select substitutes only the variables it projects; the others are variables of the sub-select. A variable that is unbound in a left solution is not substituted for it. The right side may not bind a variable of the left side again. The right side is evaluated once for all left solutions together rather than once per solution: ORDER BY with LIMIT and OFFSET slices, GROUP BY groups, and DISTINCT deduplicates for each left solution separately. To aggregate per left solution over a correlated variable that is only used in a FILTER, project it and GROUP BY it, as SPARQL does not allow projecting an ungrouped variable next to an aggregate. The LATERAL tests of Apache Jena and Oxigraph are included in the tests

  • GRAPH blocks in CONSTRUCT templates (w3c/sparql-dev#31), with the syntax of Apache Jena and of INSERT templates: CONSTRUCT { ?s ex:r ?o GRAPH ex:g { ?s ex:q ?o } GRAPH ?g2 { ?o ex:q ?s } } WHERE { ... }. The graph is an IRI or a variable; with a variable, each solution gives triples in the graph that the variable is bound to, and solutions where it is unbound or not an IRI give none. Model.insert and ModelStream.insert add the triples of each GRAPH block to its graph, and the other triples to target_graph (the default graph if not given), all in one go, so the WHERE clause is evaluated once instead of once per graph. Model.query returns one DataFrame per template triple (and graph) as before, and the DataFrames from GRAPH blocks have an extra graph column with the IRI of the graph; return_format="trig" returns the result as TriG, and "turtle" raises an error when the result has triples in named graphs. The short form CONSTRUCT WHERE { GRAPH ?g { ... } } is also supported. Blank nodes in the template are shared between its GRAPH blocks. In Rust, Query::Construct has a new graph_template: Vec<QuadPattern> with the triples of the GRAPH blocks, QueryResultKind::Construct holds ConstructedTriples with the graph of each part, template_to_solution_mappings returns them in template order, and Triplestore::insert_constructed_triples inserts them. Jena ARQ’s syntax tests for CONSTRUCT quads are included in the tests, except two that use Jena’s bare { ... } blocks in templates

  • Revealing update effects (w3c-cg/sparql-dev#63): Model.update returns, per graph (keyed by IRI, DEFAULT_GRAPH for the default graph), the number of triples that were really inserted and deleted: triples that were already present are not counted as inserted, and triples that were absent are not counted as deleted. This covers INSERT DATA, DELETE DATA, DELETE/INSERT WHERE, DELETE WHERE, CLEAR and DROP; the changes of the operations of an update are summed. The counts come from the deduplication and deletion that the update does anyway, so they cost nothing extra. With return_triples=True the result also holds the inserted and deleted triples as DataFrames per predicate (or SolutionMappings with solution_mappings=True), which costs an extra pass over the tables that triples are deleted from. The SPARQL endpoint of Model.serve (maplib Enterprise Edition) sends the counts as JSON ({"graphs": [{"graph": ..., "inserted": ..., "deleted": ...}]}, with null for the default graph) to update requests that accept application/json, and still answers others with no content; the Flight update action includes them in its result. In Rust, Model::update_keep_triples, Model::update_with_federation_keep_triples and Triplestore::update_keep_triples keep the triples

  • Vector literals: the datatype maplib:vector (https://datatreehouse.github.io/maplib/vocab#vector) holds embeddings, with the components as a list of numbers in square brackets, e.g. "[0.12, -0.5, 3e-4]"^^maplib:vector. In Model.map and Model.map_triples, Polars columns of fixed size arrays of numbers (pl.Array(pl.Float32, 384)) are mapped to vector literals. In Rust, representation::vectors has the datatype and the conversions

  • Vector search (maplib Enterprise Edition): enable it with IndexingOptions(vectors=True). The vectors of vector literals are indexed by predicate, and the magic predicate maplib:vectorSearch finds the triples whose vector literal is nearest to a query vector: ?s maplib:vectorSearch [ maplib:vectorQuery "[...]"^^maplib:vector ; maplib:vectorLimit 10 ; maplib:vectorScore ?score ]. The query is a vector literal, a string with the lexical form of one, or an IRI, whose own vector is then used (“more like this”). The optional arguments are maplib:vectorMetric ("cosine", the default, "dot" or "euclidean"), maplib:vectorPredicate (an IRI to search one predicate, or a variable to bind it), maplib:vectorObject (binds the vector literal), maplib:vectorScore (the cosine similarity, dot product or Euclidean distance) and maplib:vectorExact. A constant subject restricts the search to that subject. The search is exact by default. With IndexingOptions(vector_ann=True) an approximate nearest neighbour index is also built and kept up to date as triples are added; cosine searches then use it unless maplib:vectorExact true is given, and maplib:vectorCandidates (default 64) trades speed for recall. The functions maplib:cosineSimilarity, maplib:dotProduct and maplib:euclideanDistance compare two vectors anywhere in a query, e.g. in BIND, FILTER and ORDER BY, and use the indexed vectors of stored vector literals when there is an index. The index is rebuilt from the triples on compaction and deserialization, and like the full text search index it is not updated when triples are deleted

  • Streams can iterate on a persistent Model: with in_place=True on Model.map, map_triples, map_default, map_json or map_xml (with stream=True), every batch is mapped into the Model itself and the steps run on it, so updates, inserts and rules join the batch with the Model’s other graphs and their results are kept. Batches run one at a time under the Model’s write lock, released between batches. The new ModelStream.drop_graphs(graphs) step removes the per-batch graphs, and it still runs when an earlier step of the batch fails. When collected, an in-place stream checks that the graphs named in drop_graphs steps are empty and that write steps only write graphs that a later drop_graphs step drops (or the transient graph). compact_every=N compacts the Model and removes unused dictionary entries every N batches

  • Model.drop_graphs(graphs) removes graphs with their full text and vector indexes, and in disk storage mode deletes the files of their triple tables. with m.scratch_graphs(g1, g2): drops the graphs when the block exits, also on an error. In Rust, Triplestore::drop_graphs

  • Outbound connection policy (against server-side request forgery, SSRF): maplib.set_outbound_policy(mode="any" | "public" | "allowlist", allow=[...], check=None) is one place where an application decides which hosts maplib may connect to. maplib checks the final scheme, host and port of every connection it makes for a configuration, a query or a server’s answer before connecting: the databases of maplib.config (probe, apply_virtualization, apply_config, from the host the database builds, e.g. <account>.snowflakecomputing.com for a Snowflake account, the OneLake endpoint of a Fabric lakehouse, the cluster URI of Kusto, the base_url and token_url of Cognite Data Fusion, the derived token endpoint of Insights Hub), the databases of maplib.virtualization, Model.map_osipi, map_pi_points, map_insights_hub and map_cdf, IP.21, OPC UA, Model.query_external and Model.query(endpoint=...). The host is resolved and every address it resolves to is checked. public refuses loopback, private, link-local and other non-public addresses, and allowlist allows only the hosts in allow; allow takes host names, *.domain, addresses and CIDR ranges, which are allowed in every mode. Cloud metadata services (169.254.169.254, fd00:ec2::254, 100.100.100.200, 169.254.170.2, 192.0.0.192, metadata.google.internal) are refused in every mode, also by default and when listed. Redirects are checked by the clients maplib controls (urllib, requests and reqwest, whose DNS resolver also checks the addresses it connects to, against DNS rebinding); the absolute URLs of PI Web API links are checked too. check(scheme, host, port) is called after these checks for connections made from Python. Refusals raise maplib.OutboundError, a PermissionError. maplib.outbound also has check_url, check_host, check_connection, connection_targets, urlopen and guard_session for applications and database classes of their own. In Rust, the new outbound crate (outbound::check, check_url, set_policy, and with the reqwest feature guard_client)

  • Keyset pagination (sparql-dev#49): Model.query_pages(query, page_size=10000) iterates over the pages of a SELECT query, and Model.query_page(query, page_size, page_token) gives one page and the token of the next. Each page starts after the last solution of the previous page, instead of skipping the solutions before it with OFFSET, so a page deep into a large result costs no more than the first. The solutions are ordered by the ORDER BY of the query and then by the projected variables, so the order is total and the pages put together are the result of the query without duplicates or gaps (a query without ORDER BY is ordered by its projected variables). Duplicate solutions are counted across pages, and the LIMIT and OFFSET of the query apply to the pages as a whole. Pages are evaluated one at a time and see the model as it is then: no solution is returned twice, solutions in the result throughout are returned exactly once, removed solutions are not returned once removed, and added solutions are returned when they sort after the last solution returned. A page token (QueryPages.token) is a string safe in URLs that holds the position, not the results, so it can be kept and used later or in another process; it is refused for another query. SELECT DISTINCT can be paged when its ORDER BY uses only projected variables. With return_format="json", pages are SPARQL 1.1 JSON results. Queries of models with virtualized databases can not be paged yet. In Rust, Model::query_page and Triplestore::query_page take a PageCursor (triplestore::sparql::paging)

  • Model.serve() (maplib Enterprise Edition) takes page_size and page_token parameters on the SPARQL query endpoints: a SELECT query gives that page of its solutions, and the token of the next page in the Next-Page-Token response header, which is absent on the last page (exposed to CORS clients). limit and offset parameters apply a LIMIT and an OFFSET after those of the query (sparql-dev#91). Pages are not kept in the query result cache, whose key includes limit and offset.

  • Statistical aggregates from Apache Jena ARQ: agg:stdev, agg:stdev_samp, agg:stdev_pop, agg:variance, agg:var_samp and agg:var_pop in http://jena.apache.org/ARQ/function/aggregate#, also available in http://jena.apache.org/ARQ/function# as in Jena. They return an xsd:double, take DISTINCT, and can be used in HAVING and as window functions (agg:stdev(?x) OVER (PARTITION BY ?g ORDER BY ?t ROWS BETWEEN 9 PRECEDING AND CURRENT ROW)). A group with a non-numeric value, with no values, or with a single value for the sample statistics gives an unbound result, as in Jena. Jena’s tests for them (TestStatisticsAggregates) are ported, and the results are checked against Python’s statistics module. See “Statistical aggregates” in API_USAGE.md

  • SHACL entailment (maplib Enterprise Edition): a shapes graph’s sh:entailment is honoured. For RDFS, validation first materializes the RDFS entailments with Model.reason’s "rdfs" profile, and for the OWL regimes (OWL 2 RDF-Based and Direct semantics) the "owl-rl" profile, into a temporary graph that rules and constraints read, which is removed afterwards. Simple entailment, sh:RulesEntailment, sh:Rules and sh:RuleSet need no reasoning, and other regimes (D, RIF, …) make validation fail instead of being ignored. Shapes graphs that declare RDFS entailment may now report more results, e.g. for rdfs:Resource

Bug fixes

  • ORDER BY puts blank nodes before IRIs, as SPARQL 1.1 specifies (unbound values, then blank nodes, then IRIs, then literals); IRIs came first. This is the order of the W3C sort tests, which are included in the tests

  • Model.update reports the triples that LOAD inserts, in the graphs they went to; they were left out of the changes

  • CONSTRUCT templates can have { ... } blocks of default graph triples next to GRAPH blocks, as in Apache Jena: CONSTRUCT { { ?s ?p ?o } GRAPH ex:g { ?s ?p ?o } } WHERE { ... }, also in the short form CONSTRUCT WHERE { { ... } }. INSERT and DELETE templates are unchanged, as SPARQL 1.1 does not allow these blocks there

  • The SPARQL Update operations CLEAR, DROP, COPY and MOVE work in hybrid and disk storage (with a storage folder), where they raised “Clearing or dropping graphs is not supported with a storage folder”, and so does Model.truncate_graph, which panicked. In disk storage the parquet files of the removed tables are deleted from the storage folder; files a model was deserialized from are never touched. The W3C SPARQL 1.1 Update tests now run in all three storage modes

  • Vector search (maplib Enterprise Edition) and the vector functions now agree with numpy and scipy on edge cases. Euclidean search is computed from the component differences, so near-duplicate vectors far from the origin are ranked correctly and a stored vector is at distance 0 from itself. Zero vectors and vectors with NaN or infinite components have no cosine similarity, so cosine searches leave them out (they were returned with a score of 0), and a zero query vector finds nothing. Vectors at an infinite distance or dot product are no longer dropped when fewer than maplib:vectorLimit results are found. maplib:cosineSimilarity, maplib:dotProduct and maplib:euclideanDistance compute in 64 bit floats, so very small or large components no longer underflow or overflow. Tests now check the functions against scipy, exact search against numpy and faiss, and the recall of the approximate index against hnswlib with the same parameters

  • The FILTER of an OPTIONAL can use variables that only the pattern before the OPTIONAL binds, as in ?s :min ?min OPTIONAL { ?s :v ?v FILTER(?v > ?min) }. It used to be evaluated without those variables, so no optional match passed it and the variables of the OPTIONAL were always unbound

  • A SHACL SPARQL rule (maplib Enterprise Edition) whose sh:construct has GRAPH blocks in its template raises an error, as Datalog rules do. The graphs were ignored and the triples went to the inferences graph

  • GraphSnapshots.restore() (maplib Enterprise Edition, the writable Model portal) no longer adds the snapshot’s copy of a graph to a graph of the same name that the model already has, e.g. shapes loaded by the entry point before restoring: the blank nodes of such graphs were duplicated on every start and every new replica. The model’s own copy is kept, and when it differs from the snapshot’s, the next snapshot publishes it

  • After GraphSnapshots.restore(), the first snapshot no longer writes every graph again: graphs that were restored, or loaded before restoring, count as published, and a write that leaves their content unchanged (building an index, as the Model portal does at start, marks every graph as written) is not published

  • The background loop of GraphSnapshots (and the writable Model portal) backs off when the store fails, e.g. a missing bucket or container, or auth or network errors: it waits backoff_min (1 s), doubling with each failure in a row up to backoff_max (300 s), with jitter, instead of trying about once a second, and logs one warning per try on the maplib.graph_snapshots logger. A success resets it, and stop() (the snapshot on SIGTERM) tries at once

  • SAMPLE skips unbound values, so it gives a value whenever the group has one, also with ORDER BY (SAMPLE(?x ORDER BY ?y) is the first bound value in that order)

  • AVG of no values (a group where the expression is never bound) is 0, as SPARQL 1.1 defines it, instead of unbound; also for AVG over a window

  • ORDER BY orders numbers of different datatypes by value, as the < operator compares them, instead of grouping them by datatype first (1 before 1.5 before 2 for an integer, a decimal and an integer). Numbers that a double cannot tell apart keep their exact order

  • COUNT(?x) and COUNT(DISTINCT ?x) skip unbound values; COUNT(DISTINCT ?x) counted them as one extra value

  • FILTER NOT EXISTS and FILTER EXISTS give the right result in groups with a variable predicate pattern: the filter pushdown rewrite moved them below the patterns that bind their variables, so they were evaluated without those bindings

  • JSON-LD output keeps IRIs whose local part would start with / uncompacted: ex:/a would be read back as a different, absolute IRI

  • maplib.storage gives the same version token, the unquoted ETag, from head, get, list and put on every backend, so a token from any of them works with put(if_match=...)

  • ModelStream.collect() hands its blank node counters back to the Model, also when it fails, so a later stream or mapping from the same Model no longer reuses blank node labels that an earlier stream wrote (two streams of a template with a blank node both wrote _:b_l0_p0_r0, which merged distinct nodes when the outputs were combined)

  • A variable that an OPTIONAL or a UNION branch may leave unbound, and that other patterns of the group also use, is joined as compatible with any value, as SPARQL specifies; such solutions were lost or joined wrongly when the OPTIONAL or UNION came first in its group. The W3C tests service4a and values7 pass

  • maplib:path and maplib:shortestPath patterns are joined after the other patterns of their group, so they get their start and target nodes from them wherever they are written

  • Parsing REGEX, SUBSTR and REPLACE takes time linear in their nesting depth; nested calls (e.g. the about 20 nested REPLACE calls that map_csvw generates for some tables) took exponential time and could overflow the stack. map_csvw also runs on a large stack

  • A Model stays usable after a call panics: every Python method recovers the model’s lock instead of raising on a poisoned lock

  • Virtualized time series queries (maplib Enterprise Edition) give the same results as the equivalent queries over materialized data in more cases: subqueries in joins (with LIMIT, DISTINCT or hidden variables), GROUP BY on a virtualized value such as the timestamp, aggregates without GROUP BY over no rows (one row, e.g. a count of 0), several data point patterns of one time series, and data points written without their time series. A differential test against materialized data now passes for all of its queries

  • SHACL reports (maplib Enterprise Edition) hold the whole sh:resultPath of a result with a complex path (a sequence, alternative, inverse or zero-or-more path, i.e. a blank node of the shapes graph): its triples are copied from the shapes graph into the report graph, where the path was a dangling blank node

  • GRAPH ?g { ... } on a model or dataset without named graphs keeps ?g in scope, so GROUP BY ?g, ORDER BY or projecting ?g give an empty result instead of an error that ?g is not defined

v0.22.0

Breaking changes

  • xsd:dateTime and xsd:dateTimeStamp values are all stored as UTC instants (Polars Datetime("us", "UTC")), and the time zone offset of a literal stays part of its type. Naive datetime columns of mapped DataFrames used to be stored as naive datetimes, so comparing, joining or unioning them with other dateTimes failed (“could not evaluate ‘>=’ comparison between Datetime(‘μs’) and Datetime(‘μs’,’UTC’)”, “failed to determine supertype”, or a panic when such a CONSTRUCT or UPDATE wrote to a named graph). They are now read as UTC wall clock times and still have no time zone: they serialize as before ("2026-11-14T06:12:00"^^xsd:dateTime), but query results return them as UTC-aware datetimes (datetime[μs, UTC], tzinfo=timezone.utc in Python) instead of naive ones, as dateTimes read from RDF files already were. Time zone aware columns in other time zones are converted to UTC. Models serialized by earlier versions load and serialize the same, and their naive dateTimes are read as UTC wall clock times, so they also come back UTC-aware and now compare with other dateTimes. Casting a string to xsd:dateTime accepts Z and offsets such as +02:00, which used to be an error, and gives the UTC instant without a time zone in its type (xsd:dateTime("2026-11-14T08:12:00+02:00") is "2026-11-14T06:12:00"^^xsd:dateTime), so it equals dateTimes without a time zone, not those with one. Strings that are not dateTimes give unbound instead of an error. chrontext:MicrosAsDateTime, chrontext:SecondsAsDateTime and chrontext:FloorDateTimeToSecondsInterval return UTC-aware datetimes

  • SHACL rules (maplib Enterprise Edition) follow SHACL 1.2 Inference Rules: rules that no shape links to with sh:rule (global rules) also run, sh:condition restricts the focus nodes to those conforming to the condition shapes, sh:deactivated rules are skipped, and rules run in ascending sh:layer and sh:order, where rules with the same order do not see each other’s new triples within an iteration. Rule types other than sh:SPARQLRule, sh:TripleRule and SPARQL rule templates, and any sh:ruleProcessor, are errors. The rule_log of the validation report lists the layer instead of the number of targets

  • Transient triples are removed. Triples that should be available for querying, validation and reasoning, but not written, now go in a reserved named graph, maplib.TRANSIENT_GRAPH (https://datatreehouse.github.io/maplib/vocab#TransientGraph). The transient parameter of map_json, map_xml, read, reads and insert is removed: pass graph=TRANSIENT_GRAPH (or target_graph=TRANSIENT_GRAPH for insert) instead

  • include_transient is replaced by include_graphs: Optional[List[str]] in query, update, insert, infer, get_predicate, get_predicate_iris and the corresponding ModelStream steps. The included graphs are merged into the queried graph, as with FROM. For query, update, insert and infer, None includes the transient graph (as include_transient=True did) and [] includes nothing; for get_predicate and get_predicate_iris, None includes nothing. GRAPH ?g ranges over the transient graph as over any other named graph, and GRAPH <g> no longer sees transient triples added to g

  • RDFS inference (infer_rdfs) and SHACL rules (without an inferences_graph equal to the data graph) put their inferred triples in the transient graph. Writing all graphs as TriG or N-Quads leaves out the transient graph; write it with graph=TRANSIENT_GRAPH

  • In Rust, QuerySettings::include_transient is replaced by include_graphs: Vec<NamedGraph>, and the transient arguments of the triplestore and model methods are removed. NamedGraph::transient() gives the transient graph

  • The message column of ValidationReport.results() and details() is a list of strings, one per sh:message of the shape or SPARQL constraint ("value"@lang for language tagged strings, the value for plain strings), and null when there are none. Use e.g. .explode("message") to get one row per message

  • Model.serialize writes a single file by default, and the default path of Model.serialize and Model.deserialize is ./model.maplib instead of ./serialized_triples. Pass single_file=False to write a folder as before

  • In Rust, Model::expand_default, expand_default_first_batch and expand_default_batch take the key columns as &str and &[String], and a nested (List or Struct) key column is an error instead of a panic

  • The max_results parameter of Model.infer and ModelStream.infer, and the max_shape_constraint_results parameter of Model.validate, are removed: the memory limit (maplib.set_memory_limit) bounds inference and validation instead. max_iterations and max_rows are unchanged. In Rust, Model::infer and Model::validate no longer take them

Features

  • Model.generate_template(data, primary_key_column) is a dry run of Model.map_default: it returns the default template that map_default would map with, as a stOTTR string, without mapping the data or adding the template to the model. Review or edit it and map with m.map(template, data). map_default generates its template with the same code (maplib::model::default::generate_default_template in Rust), so the two agree

  • maplib.GraphSnapshots(model, url, ...) (maplib Enterprise Edition) keeps a model in storage as one HDT file per named graph and a manifest.json, which several writers can share: only the graphs that changed are written (found from the model’s own writes), the manifest is published with a conditional put, and a writer that loses the put merges the graphs both sides changed three-way at the triple level. restore(), snapshot(), refresh(), gc() and a background loop with start() / stop(). Model.serve(snapshot_format="graphs", refresh_interval=10) serves a model kept this way. The writable Model portal uses it, in place of its Python implementation. In Rust, VersionTable::changed_graphs_since and graph_stamp give the graphs written after a stamp, and RemoteError::is_not_found and is_precondition_failed classify object storage errors

  • Model.serve(snapshot_url=..., snapshot_interval=900, storage_options=None) (maplib Enterprise Edition) snapshots the served model with Model.serialize: every snapshot_interval seconds if it changed since the last snapshot (through the endpoints or from Python), and when the server stops. An unchanged model is never written. python -m maplib.serve serves a serialized model configured with environment variables (MODEL_URL, ALLOW_UPDATE, SNAPSHOT_URL, …) and takes the last snapshot on SIGTERM; the cloud deploy recipe (now in maplib Enterprise Edition) runs it

  • Query timeouts and cancellation: Model.query(timeout=...) and Model.update(timeout=...) (seconds) stop the query when its time is up and raise maplib.QueryTimeoutError (a MaplibException). Queries are checked between operators, after every join while it produces rows (also on the streaming engine), while waiting for Polars collects, and in path searches and property paths, and while results are formatted (Model.query) or serialized (the server); a query with a timeout and a join that may give many rows is collected with the streaming engine, which can stop a running join. So a query that would run for hours or exhaust memory, such as a cross join of three triple patterns, stops within a fraction of a second of its timeout. Other queries running at the same time are not affected (Polars’ own abort, used for Ctrl-C, stops every query in the process). An update is checked before each operation and while its WHERE clause is evaluated, and its writes are never stopped halfway: operations of the request done before the timeout stay done. Model.serve(query_timeout=...) (maplib Enterprise Edition) limits every query and update through the endpoints, counted from when the request arrives, and requests can ask for less with the timeout parameter (seconds; URL or form parameter, the X-Maplib-Timeout header, or "timeout" in Arrow API and Flight commands and Flight update actions), or more up to max_query_timeout. A query that times out is answered with 503 Service Unavailable (Flight: DEADLINE_EXCEEDED), and a query whose client disconnects is cancelled, so it no longer runs on after its client gave up. max_concurrent (64 by default) limits the operations on the model that run at once; waiting for a turn counts towards the timeout. Model.query(endpoint=..., timeout=...) sends the timeout to the server and raises QueryTimeoutError when the server stops the query. In Rust, utils::cancel::CancelToken is a deadline and a cancel flag, made current for a thread with utils::cancel::with_token; SparqlError::Cancelled and InterruptableCollectError::Cancelled report a stopped query, and ServerOptions has query_timeout, max_query_timeout and max_concurrent

  • Query result cache: Model.set_query_cache(QueryCacheOptions(...)) keeps the results of Model.query and of the SPARQL protocol endpoints of Model.serve, which share it. A result is kept until a write changes the triples it can read: every write records the named graphs and predicates it changed, and every query reads a set of (graph, predicate) pairs found from its algebra, so writes to other graphs or other predicates keep it. Variable predicates, negated property sets, zero-length paths between variables, full text search and DESCRIBE read every predicate of their graphs, and GRAPH ?g reads a predicate in every graph. Changes of the prefixes, the indexing options, user defined functions and virtualizations invalidate everything. Queries calling NOW(), RAND(), UUID(), STRUUID(), BNODE() or user defined functions, with parameters, with debug or reading virtualized databases (unless cache_virtualized=True) are never cached. The options are on or off, the largest number of results, their total size, the largest result kept, a ttl, the shortest run time worth keeping and whether to keep results over virtualized databases, and can be changed at any time. Cached results are answered without waiting for a write that runs elsewhere in the model, and the server answers them without holding the GIL. Model.query(cache=False) skips the cache, Model.query_cache_stats() gives its counts (hits, misses, evictions, invalidations, size) and Model.clear_query_cache() empties it. The cache is off by default. Model.serve(query_cache=...) sets it for the server, which reports X-Maplib-Cache: hit | miss | bypass on each response, a cache field on each query log entry, cache totals and hit rates in /query-log and the cache’s counts in /status. In Rust, Triplestore::query_cache (triplestore::query_cache::QueryCache), Model::cache_plan, Model::add_prefixes and ServerOptions::query_cache

  • SHACL rules (maplib Enterprise Edition) implement SHACL Advanced Features and SHACL 1.2 Inference Rules: sh:SPARQLRule (shape rules with $this pre-bound to each focus node, and global rules), sh:TripleRule with node expressions (focus node, constants, lists, path values with shnex:pathValues or sh:path, exists, if, distinct, intersection, union/concat, remove, filterShape, instancesOf, count, sum, min, max the sparql: functions and SHACL-AF sh:SPARQLFunction calls), SPARQL rule templates (sh:SPARQLRuleTemplate with sh:parameter), sh:condition, sh:deactivated, sh:order, sh:layer, sh:runOnce, rule sets (sh:RuleSet, sh:hasRule, sh:includesRuleSet) selected with sh:entailment (sh:RulesEntailment runs all rules, as does a shapes graph without sh:entailment), and prefixes declared at the shapes graph. Shapes typed sh:ShapeClass are node shapes with an implicit class target. Not supported: sh:expectedPredicate, temporary triples and sh:sourceRule, which need RDF 1.2 triple terms, and the order dependent node expressions (limit, offset, orderBy, findFirst, flatMap, matchAll, nodesMatching). The W3C SHACL 1.2 inference rules tests and the TopBraid DASH SHACL-AF rules tests are included in the tests. Inferred triples that are already in the data graph are no longer reported as inferences

  • SPARQL-RL 1.2 (maplib Enterprise Edition): Model.infer takes rule sets in the SPARQL-RL syntax of the W3C Data Shapes Working Group (RULE { ... } WHERE { ... }, DATA { ... }, NOT, NOT DATA, WHERE DATA, SET(?x := ...), GROUP BY ... { ... } [ ?n := COUNT(*) ]), checks their well-formedness and stratification, and evaluates them stratum by stratum as the specification defines. IMPORTS is not supported. The W3C SPARQL-RL syntax, well-formedness, stratification and evaluation tests are included in the tests

  • Object storage: Model.read, read_template, write, write_cim_xml, serialize and deserialize take URLs of Google Cloud Storage (gs://bucket/path), Azure Blob Storage (az://container/path, abfss://container@account.dfs.core.windows.net/path, https://account.blob.core.windows.net/container/path), Amazon S3 (s3://), HTTP(S) and file://, as well as local paths. Credentials are found as the cloud SDKs find them (environment variables, the metadata server on Google Cloud, managed and workload identities on Azure), or are given with the new storage_options argument, with the option names of Polars’ storage_options. Files that are read are downloaded to a temporary file first, with concurrent range requests for large files. Files that are written are uploaded as they are written, in concurrent parts, and the object only appears when the write completes

  • maplib.storage stores raw bytes in object storage and local files, with the URLs, credentials and storage_options of Model.read and Model.write: put(url, data, if_match=None, if_none_match=False) returns a version token (the ETag, with the object generation on Google Cloud Storage), get(url) returns the bytes and their token, and head, exists, delete and list(prefix_url) (names, sizes, times, tokens) are there for garbage collection. put with if_match replaces the object only if it is still at that version and raises storage.PreconditionFailed otherwise, and if_none_match=True raises storage.AlreadyExists if there is an object, so that several writers can publish a manifest with optimistic concurrency. S3, Google Cloud Storage (as generation preconditions; the version from list has no generation there, so use the token of get or head for if_match), Azure and the local file system support both; for local files if_match is emulated with a lock file, which serializes the writers on one machine only. Plain HTTP raises storage.NotSupported. A missing object raises storage.NotFound, a FileNotFoundError. In Rust, RemoteLocation::put_with, get_with_version, info, list_info and parse_path_or_url

  • Model.serialize(url) writes a single file locally and uploads it. Model.serialize(url, single_file=False) writes a folder to a new generation under the URL (gen-<time>-<id>/) and then publishes it by replacing the object LATEST with its name, with a conditional put: readers see the previous or the new model whole, and if another writer published to the URL in the meantime, the serialization fails instead of replacing it. The current and the previous generation are kept

  • Triple tables in object storage (maplib Enterprise Edition): Model.deserialize(url, storage="disk", storage_folder=...) on a folder serialized to object storage does not download the triple tables, only the dictionaries and the sparse indices. Each table is downloaded to the storage folder the first time it is read. With cache_size (or MAPLIB_REMOTE_CACHE_BYTES), the least recently read downloaded tables are removed when they take more space, and downloaded again when read. Tables read in the last five minutes (MAPLIB_REMOTE_CACHE_GRACE_SECONDS) are kept, as running queries may read them. So a disk mode model can be served from a VM or container with a local disk smaller than the model. Triples added afterwards stay local until the model is serialized again. Serializations record the number of triples of each table for this; folders written by earlier versions are downloaded when deserialized

  • In Rust, file_io::remote has RemoteLocation, Input and Output for object storage, SerializationSource, SerializationTarget and SerializedFile have Remote variants, Triplestore and Model have serialize_triples_remote and deserialize_triples_remote, and Triples::from_serialization takes the number of triples when it is known. The metadata of a serialization folder is written after its other files

  • The config graph configures the query result cache of the model (mcfg:QueryCache, see CONFIG_GRAPH.md): maplib.config.QueryCacheConfig with write_query_cache, read_query_cache, remove_query_cache and apply_query_cache keeps whether it is enabled, its limits (entries, bytes, the largest result kept), a ttl, the shortest run time worth keeping and whether results over virtualized databases are kept. apply_config sets it with Model.set_query_cache

  • maplib.config and the Model portal’s Connect page configure InfluxDB: InfluxDB 3 (influxdb, mcfg:InfluxDBDatabase, resources are SQL over measurements) and InfluxDB 2 / 1.8 with Flux (influxdb2, mcfg:InfluxDB2Database, resources are measurements: the options or connection parameters measurement, id and field). The token is a secret. Probing without SQL lists the measurements

  • CSV on the Web (#26): Model.map_csvw(source, metadata=None, graph=None, minimal=False, iri_mappings=None) converts tabular data to RDF as specified by the W3C CSVW recommendations (tabular data model, metadata, csv2rdf), in standard or minimal mode. The source is a metadata document or a CSV file, for which the metadata is located as in the specification (with the site-wide configuration /.well-known/csvm if it is found, else <file>-metadata.json and csv-metadata.json) or taken from the header row. Dialects, inherited properties, datatypes with formats (numbers, booleans, dates and times, regular expressions) and constraints, lists (ordered as rdf:List), virtual and suppressed columns, notes and common properties, row titles and foreign key validation are supported. The tables are added to temporary graphs using Facade-X, and a SPARQL Update generated from the metadata builds the output (URI templates are compiled to SPARQL expressions) and drops the temporary graphs. The update is returned. Warnings (invalid metadata properties, invalid cell values and values that maplib can not store with their datatype, such as dates with time zones, which are output as strings) are issued as Python warnings. Nothing is fetched over the network: iri_mappings maps IRI prefixes to local folders. The W3C CSVW RDF test suite is included in the tests, see w3c_csvw_known_failures.txt for the tests that do not pass. In Rust this is the new csvw crate and Model::map_csvw

  • Streaming (#87): Model.map, map_triples, map_default, map_json and map_xml take stream=True and then return a lazy ModelStream, e.g. m.map(template, lf, stream=True).update(...).infer(...).insert(...).write("out.nt").write("out.ttl", format="turtle").collect(). On collect(), each batch of the input is mapped into a fresh in-memory model with the templates, prefixes and UDFs of the model, the steps run on it in order, the write steps append it to their files (N-Triples, Turtle with prefixes declared once, or N-Quads), and the batch is dropped, so memory use depends on the batch size and not on the size of the input. The model’s own triples are not changed. Steps only see the triples of the current batch. Blank node labels stay unique across batches

  • Model.map_json and Model.map_xml also take an iterable of documents (strings or paths), e.g. a generator over the lines of a JSON Lines file, mapped one document at a time. With a uuid_namespace, document number i uses the namespace <uuid_namespace>/<i> so that documents do not get the same IRIs. With stream=True, documents_per_batch documents are mapped into each batch; mapping them into graph=TRANSIENT_GRAPH and inserting a CONSTRUCT query writes only the constructed triples

  • Storage modes: Model(storage="memory" | "hybrid" | "disk", storage_folder=...), also for Model.deserialize. "memory" keeps everything in memory, "hybrid" stores the dictionaries of IRIs and literals on disk (what storage_folder did before), and the new "disk" mode also stores the triple tables on disk as parquet files in <storage_folder>/maplib_storage/triples. Files of triple tables that are replaced (by compaction or deletes) or whose model is dropped are removed. Without storage, a storage_folder still means hybrid storage, so existing code is unaffected. Hybrid and disk storage require maplib Enterprise Edition. In Rust, Model::new, Model::deserialize_triples and Triplestore::new take a StorageOptions instead of the storage folder

  • Full text search (maplib Enterprise Edition) uses DuckDB instead of tantivy. The index stores the dictionary codes of subjects, predicates and objects, and searches the strings where the dictionaries keep them: with hybrid and disk storage it joins against the on-disk dictionary tables, so no copy of the strings is made, and with memory storage it uses an in-memory DuckDB database. Matching is unchanged: case insensitive, 1.0 for a match of the whole value and 0.75 for a substring match. fts_path (in IndexingOptions and Model.explore) is no longer used, as the index is stored with the dictionaries

  • GeoSPARQL (maplib Enterprise Edition): enable it with IndexingOptions(geo=True). Geometry literals (geo:wktLiteral, geo:wkbLiteral, geo:geoJSONLiteral) are indexed spatially, and the GeoSPARQL functions are supported: the simple features, Egenhofer and RCC8 relations, measurements with units of measure, constructions, properties and serializations, across coordinate reference systems. Spatial filters and spatial joins use the index. geof:relate, the query rewrite extension and the aggregate functions are not supported yet

  • Pretty Turtle output only declares the prefixes whose namespace starts a predicate, a datatype or an IRI in the model, sorted by name, instead of all known prefixes. Each prefix is checked with one lookup in the sorted IRI dictionary. The dictionary is shared by all graphs, so a prefix used only in another graph is still declared; with hybrid and disk storage all prefixes are declared as before

  • Model.serialize writes the dictionaries of IRIs and literals as parquet files (cats/<id>.parquet, columns str and cat, in string order) instead of DuckDB databases, so a serialized model is parquet throughout and serialization no longer needs maplib Enterprise Edition. Model.serialize(path, raw_db=True) (Enterprise Edition) writes them as DuckDB databases as before: larger and slower to write, but deserializing into hybrid or disk storage uses them in place. Model.deserialize reads both, also folders written by earlier versions. In Rust, serialize_triples takes raw_db: bool

  • Model.map_cdf(client) (maplib Enterprise Edition) maps the data model of a Cognite Data Fusion project to RDF using a cognite-sdk CogniteClient: the data models and views (as classes, with rdfs:subClassOf for the views they implement), the container properties the views map (as predicates, shared by the views that map them), and the nodes and edges that have data in the views. Datapoints are not read: the cdf_cdm:CogniteTimeSeries nodes are linked to chrontext resources, so ?ts chrontext:hasDataPoint ?dp fetches their datapoints from CDF through VirtualizedDatabase.cognite when a query needs them. data_models and spaces select what to map

  • Model.serialize(path, single_file=True) (the default) writes the model as one file, by convention with the extension .maplib, though any name works, also with raw_db=True. An interrupted serialization never leaves a partial file at path. Model.deserialize detects whether path is a file or a folder. A model deserialized from a file does not use the file afterwards. In Rust, serialize_triples takes single_file: bool, and Cats::deserialize_cats takes a SerializationSource

  • Arithmetic and comparisons on durations, dates, dateTimes and times, as defined by the XPath operators that SPARQL refers to: xsd:yearMonthDuration and xsd:dayTimeDuration values can be added, subtracted, multiplied and divided by numbers, and divided by each other; they can be added to and subtracted from xsd:dateTime, xsd:date and (dayTimeDurations only) xsd:time values, and dateTimes, dates and times can be subtracted from each other, giving a dayTimeDuration. Durations are compared by value: = holds between durations of any duration datatype with the same months and seconds (e.g. "P1D"^^xsd:dayTimeDuration = "PT24H"^^xsd:dayTimeDuration), and yearMonthDurations, dayTimeDurations and times can be ordered with <, <=, > and >=. Casting to xsd:duration, xsd:yearMonthDuration and xsd:dayTimeDuration checks the lexical form and converts between the duration datatypes. Operations that are not defined, such as adding an xsd:duration or a number to a dateTime, are errors (unbound)

  • Model.read also takes bytes (bytes, bytearray or memoryview, e.g. from write_bytes) or a file-like object with a read method, which is read to the end into memory (text streams as UTF-8). format is then required, except for HDT, which is detected from its header. Bytes objects are parsed without copying. reads(format="hdt") points to read. In Rust, Model::read_bytes reads from a byte slice

  • Model.write_bytes(format=...) writes to bytes in memory in any format, including HDT. Model.write, Model.write_cim_xml and ModelStream.write also take a binary file-like object (anything with a write method accepting bytes, e.g. io.BytesIO, a file opened with "wb" or a socket file) instead of a path; writes are buffered, and the object is flushed but not closed. writes(format="hdt") points to write_bytes. write_cim_xml raises an exception instead of panicking when writing fails

  • A Model shared between threads lets read operations run at the same time: query, query_external, write, writes, write_bytes, size, get_predicate, get_predicate_iris, get_templates and list_udfs take a shared lock, while operations that change the model (update, insert, read, reads, map*, validate, infer, create_index, serialize, …) take an exclusive lock, so they wait for the reads in progress and new reads wait for them. The graph explorer answers its requests concurrently. In Rust, Model::query, query_external, write_triples, write_quads, write_hdt, get_predicate and get_predicate_iris and the corresponding Triplestore methods take &self

  • Model.add_graph raises an error when given the model itself, which used to deadlock

  • Memory limit: queries that would run out of memory spill to disk instead. Over the limit, which is 80% of the system memory by default (respecting container limits), joins, aggregations, sorts and query results move intermediate data to disk. maplib.set_memory_limit(max_memory_mb=..., spill_dir=..., max_spill_mb=..., abort_above_mb=...) sets the limit, the spill folder, the most to spill to disk, and an optional hard stop. maplib.memory_usage() and maplib.memory_limit() report the memory used and the limit. Spilling applies to the streaming engine, which collects all but small results. In Rust, utils::memory::set_memory_limits applies the limits, and binaries must use polars_ooc::Allocator as their global allocator for spilling to work

  • Model.serve() (maplib Enterprise Edition) serves the model over the SPARQL 1.1 Protocol and Arrow Flight at the same time, in the background, on the model itself, so changes made from Python and through the endpoints are seen by both. Queries through the endpoints run at the same time as each other and as read operations from Python, while updates wait for exclusive access. It returns a Server with the endpoint URLs, stop(), wait() and context manager support. The SPARQL endpoints (/sparql, /query, /update) follow the W3C SPARQL 1.1 Protocol, including default-graph-uri, named-graph-uri, using-graph-uri and using-named-graph-uri, content negotiation (JSON, XML, CSV, TSV and Arrow IPC results, Turtle, N-Triples, RDF/XML, JSON-LD, N-Quads and TriG graphs) and the service description. /store implements the SPARQL 1.1 Graph Store HTTP Protocol. DESCRIBE is supported by the endpoint. The Flight endpoint runs queries from do_get tickets and get_flight_info commands, with results as the DataFrames of Model.query(), lists the graphs as flights, maps uploaded record batches with a template through do_put, and runs updates as an action. Updates are refused unless allow_update=True. auth_token requires a bearer token on both endpoints (with Flight basic auth handshake support). Other web sites can not read results or send updates unless allowed with cors_origins, and on loopback addresses requests with another Host are refused, against DNS rebinding. LOAD and SERVICE are refused, so clients can not make the server fetch URLs or read files. The Flight features are also served over plain HTTP at /arrow on the SPARQL port (arrow_http=True), for networks where gRPC does not get through. Model.query(query, endpoint=server.url, endpoint_token=...) queries a served model and returns the same results as Model.query() on it, with maplib’s own types sent as Arrow over HTTP. Arrow results are zstd compressed within the Arrow format for clients on other hosts (arrow_compression="auto", or "zstd" / "none", also per request), and other responses are gzip compressed on request. HTTP/1.1 and HTTP/2 are served. Request sizes are limited by max_request_bytes. In Rust, the API is the server crate, the Arrow result format is in representation::arrow_wire, the client is in external_sparql::maplib_endpoint, and Triplestore::graph_names lists the graphs with triples

  • Model.write, Model.write_bytes and Model.writes with format="json-ld" write a compacted JSON-LD 1.1 document: a @context with the prefixes that may be used (as for Turtle), and a @graph with one node object per subject. rdf:type is written as @type, xsd:string literals as JSON strings and other literals as value objects with their lexical form, blank nodes that are the object of one triple are nested, and RDF lists are written as @list. Until now, JSON-LD was written by the streaming oxjsonld serializer, which wrote one node object per triple and ignored the prefixes. A prefix whose name is the scheme of an IRI in the model (e.g. ex when there is an IRI <ex:b>) is left out of the @context, as the IRI would be read as a compact IRI (with hybrid and disk storage this is not checked, and writing such an IRI fails). Blank nodes used in more than one of the written graphs are always written with their label. A single graph is written at the top level, as the document’s default graph. To write a dataset, give a list of graphs, e.g. graph=[DEFAULT_GRAPH, "http://example.com/g1"]: the default graph (if listed) is written at the top level and each other graph as a graph object ({"@id": ..., "@graph": [...]}). In Rust, Model::write_jsonld takes the list of graphs. The writer is tested by round trips of the W3C JSON-LD 1.1 fromRdf and toRdf tests and the W3C RDF 1.1 evaluation tests

  • graph in Model.write, Model.write_bytes and Model.writes can be a list of graph IRIs for the dataset formats (JSON-LD, TriG and N-Quads), which writes exactly those graphs. The listed graphs must exist. The new constant maplib.DEFAULT_GRAPH names the default graph in such a list. In Rust, Model::write_quads and Triplestore::write_quads take an optional slice of graphs instead of an optional graph

  • Model.serve(static_dir=..., static_path="/ui", static_vars={...}) also serves a web application from a folder on disk, read on every request so a rebuilt application shows on refresh. Paths without a file get the folder’s index.html (for single page applications), / redirects to the application, and the <!--#echo var='name' --> directives of nginx SSI in its HTML pages are replaced with static_vars.

  • Model.explore() (maplib Enterprise Edition) runs the whole Model portal (the validation_portal_core package, which must be installed) on the model in a background server: its pages, its backend, whose SPARQL endpoint (server.sparql_url, <url>api/query) is answered by the model through Model.serve, and its query log. The portal’s built frontend is the frontend argument, the MODEL_PORTAL_FRONTEND environment variable, or the one packaged with the portal. JupyterHub’s proxy is detected. The old explorer pages and their FastAPI server are removed; page, graph and fts_path are deprecated and ignored and popup is deprecated (with a DeprecationWarning), and flight_port and cors_origins are gone

  • maplib.config configures the OSIsoft PI Data Archive (database "pi_points", mcfg:PIPointsDatabase): applying it reads the PI Points with Model.map_pi_points (parameters url, data_server, name_filter, source_filter, point_attributes and the others of map_pi_points, secrets password or token), and probe lists a few of the points. Like the Asset Framework ("osipi"), it has no resources to configure

  • The config graph: a model keeps its configuration as triples in a named graph of its own, maplib.config.CONFIG_GRAPH, in the new config namespace https://datatreehouse.github.io/maplib/config# (see CONFIG_GRAPH.md). maplib.config writes and reads virtualized databases (DuckDB, Snowflake, BigQuery, Databricks and TimescaleDB over SQL, OPC UA Historical Access and Cognite Data Fusion, with their connection parameters and resources: SQL or options, and stOTTR templates) and indexes (full text search, GeoSPARQL and object indexes for all or chosen predicates), and apply_config(m) connects and virtualizes them. AspenTech IP.21 (database="ip21") is applied with maplib.ip21.map_ip21, which adds the tags to the model. Secrets are never stored: the configuration names the environment variable holding each one. Resources have static properties (VirtualizationResource.properties, e.g. the CIM unit symbol and multiplier of a measurement), written as ordinary triples on the objects with ct:hasResource the resource name when the virtualization is applied (apply_properties), since chrontext does not support constant triples in virtualized templates

  • The config graph configures Microsoft Fabric: fabric_lakehouse (mcfg:FabricLakehouseDatabase, a lakehouse’s Delta tables read from OneLake, with resource SQL in DuckDB SQL over the table names) and fabric_warehouse (mcfg:FabricDatabase, a warehouse or a lakehouse’s SQL analytics endpoint, with resource SQL in T-SQL), both signing in with Microsoft Entra ID (the client_secret of a service principal is a secret). probe lists the tables of a lakehouse, and runs SQL on a warehouse with TOP 5

  • The config graph configures Azure Data Explorer and Microsoft Fabric Eventhouses: kusto (mcfg:KustoDatabase, maplib.virtualization.KustoDatabase), with the parameters cluster (the query URI), database, tenant_id, client_id and authentication, and the client_secret of a service principal as a secret (else the default Azure credentials). Its resources are KQL tabular expressions, kept in mcfg:sql like the SQL of other databases; maplib.config.QUERY_LANGUAGES gives the query language of each database. probe runs a resource’s KQL with take 5

  • The config graph configures Siemens Insights Hub (database "insights_hub", mcfg:InsightsHubDatabase): applying it reads the asset model and virtualizes its time series with Model.map_insights_hub (the parameters are its arguments, the secrets client_secret or token), with no resources to configure. probe gets a token and lists five assets of the tenant

  • Model.serve keeps a query log: GET /query-log gives the last query_log_size (default 1000, 0 for none) queries and updates as JSON, each with its text, kind, status and error, rows, bytes, media type, the time spent waiting for a slot, running and serializing, and the client and page that sent it (the X-Query-Source header, else the Referer), together with totals since the server started and summary statistics (percentiles, error rate, rate, most frequent queries). ?since=<id> returns only newer entries; DELETE /query-log empties it. /status includes the totals

  • SHACL shapes from OPC UA type definitions (maplib Enterprise Edition): m.opcua.add_typing_shapes(shapes_graph=None, type_namespaces=None, instance_namespaces=None), after m.map_opc_ua(folder), generates shapes that check instances against their ObjectTypes and VariableTypes: a shape per type, one per browse path of the fully inherited instance declarations, and cardinality shapes for Mandatory instance declarations, in the category uamapper:OPCUATypingValidation. It also adds the helper triples the shapes use (uamapper:typeNamespace, uamapper:path, …), which your own shapes can use too. maplib.opcua is now a package; maplib.opcua.OPCUAHistoryDatabase is imported as before

  • OPC UA History Access as a virtualized database (maplib Enterprise Edition): VirtualizedDatabase(database=maplib.opcua.OPCUAHistoryDatabase("opc.tcp://...")). The external ids of the time series are OPC UA node ids such as ns=2;s=ts1, and the resource templates have the parameters id, timestamp and value. Raw values are read with the asyncua package, which must be installed. Conditions on the timestamp in the top level conjunction of a filter restrict the time range read from the server; all other conditions and aggregations are evaluated by maplib

  • InfluxDB as a virtualized database (maplib Enterprise Edition). InfluxDB 3 (Core, Enterprise, Cloud Dedicated, Cloud Serverless, Clustered): VirtualizedDatabase(maplib.virtualization.InfluxDBDatabase(host=..., database=..., token=...), resource_sql_map={"r": "SELECT sensor AS id, time AS timestamp, value FROM readings"}), queried with SQL over Arrow Flight with the influxdb3-python package. The new SQL dialect InfluxDBDialect ("influxdb") follows PostgreSQL, with DataFusion’s date_bin for chrontext:dateBin and chrontext:FloorDateTimeToSecondsInterval, timestamp literals in UTC without a time zone and DOUBLE for casts to doubles; timestamps without a time zone in the results are returned as UTC. InfluxDB 2, InfluxDB Cloud (TSM) and InfluxDB 1.8: VirtualizedDatabase(maplib.virtualization.InfluxDB2Database(url=..., token=..., org=..., bucket=..., resources={"r": {"measurement": "readings", "id": "sensor", "field": "value"}})), queried with Flux with the influxdb-client package. Flux reads the time range in the top level conjunction of the filters on the timestamp and only the time series asked for; other conditions and aggregations are evaluated by maplib

  • Microsoft Fabric Lakehouses as virtualized databases (maplib Enterprise Edition): maplib.virtualization.FabricLakehouseDatabase(workspace=..., lakehouse=..., tenant_id=..., client_id=..., client_secret=...) reads the Delta tables of a lakehouse directly from OneLake with DuckDB’s delta and azure extensions, and answers the virtualized parts of queries with DuckDB SQL. Each table is a view, so resource SQL is e.g. SELECT sensor_id AS id, ts AS timestamp, value FROM readings (dbo.readings in lakehouses with schemas). The tables are found by listing the lakehouse’s Tables folder, or given with tables. It signs in with Microsoft Entra ID: a service principal, or one Azure credential (authentication="credential_chain" with chain="cli", "managed_identity", "env" or "workload_identity"). root reads the tables from another folder, such as a local folder of Delta tables

  • DuckDBDatabase works with DuckDB 2.0, where query results no longer have .pl()

  • Microsoft Fabric Warehouses as virtualized databases (maplib Enterprise Edition): maplib.virtualization.FabricDatabase(server=..., database=..., authentication=...) answers the virtualized parts of queries with a Fabric Warehouse or the SQL analytics endpoint of a Lakehouse, in T-SQL. It signs in with Microsoft Entra ID through azure-identity: a service principal (authentication="service_principal" with tenant_id, client_id and client_secret), "default" (DefaultAzureCredential), "interactive" or "azure_cli", and connects with mssql-python (which includes the ODBC driver) or pyodbc (driver="pyodbc", with ODBC Driver 18 for SQL Server). Timestamps in the results, which Fabric stores without a time zone, are taken to be in UTC (timezone). The SQL is T-SQL, from the new dialects FabricDialect ("fabric") and MSSQLDialect ("mssql", "sqlserver", "tsql", for SQL Server 2022 and later and Azure SQL), which can also be used with other databases through sql_dialect. Limits are written as TOP and OFFSET ... FETCH, time buckets with DATEDIFF_BIG, DATETRUNC and (strides in months) DATE_BUCKET, SAMPLE as MIN, and conditions selected as values as bits. Durations can not be used as values, as T-SQL has no interval type. SQLDialect has new methods for the parts that differ in T-SQL: ceil, round, avg_argument, as_value and offset_requires_order_by

  • AspenTech InfoPlus.21 (IP.21) as a virtualized database (maplib Enterprise Edition), through the aspenONE Process Data REST API (.../ProcessData/AtProcessDataREST.dll) with the requests package: maplib.ip21.map_ip21(m, url, data_source, tag_filter="*", username=..., password=...) searches for tags and adds an ip21:Tag (urn:maplib_ip21:) per tag, with its name, description, engineering units and default map, and a time series (chrontext:hasTimeseries) with the tag name as external id, whose values are chrontext:hasDataPoint/hasTimestamp/hasValue with ip21:hasStatus and ip21:isGood. maplib.ip21.IP21Database can also be used with VirtualizedDatabase and your own templates. Time conditions in filters restrict the time range read, and grouping by chrontext:FloorDateTimeToSecondsInterval with MIN, MAX or AVG of the value is answered with IP.21 aggregates when the range starts and ends on the bucket grid; maplib evaluates the rest. Queries without a lower bound on the timestamp need default_start_time. Values that are not numbers are left out

  • OSIsoft PI Data Archive (maplib Enterprise Edition): Model.map_pi_points(url, data_server=None, name_filter="*", ...) reads the PI Points (tags) of the PI historian through the data servers of the PI Web API, without the Asset Framework, with their descriptor, engineering units, point type, point source and other point attributes, and virtualizes their recorded values with the same data point pattern as Model.map_osipi. Each point is a time series whose external id is its WebId. Authentication, paging, concurrency and the time range of the recorded values read are as for map_osipi. For both, aggregates over PI data points are evaluated in the virtualization, and MIN, MAX, SUM, AVG and COUNT of the values per chrontext:FloorDateTimeToSecondsInterval bucket are read as PI summaries when the time range is aligned to the buckets and only filtered by time

  • VirtualizedDatabase takes pushdowns, a list of "value_conditions", "group_by" and "ordering", to choose what parts of a query are pushed down into the database. When it is not given, the pushdowns attribute of the database is used if it has one, otherwise all of them, as before

  • Several virtualizations (maplib Enterprise Edition): each call to Model.add_virtualization(virtualized_database, resources) adds a virtualized database instead of replacing the previous one, so time series in different databases can be queried together. Each resource belongs to one virtualized database, and the virtualized queries over a resource are sent to the database that holds it; a query matching time series of resources in different databases sends a query to each and combines the results. Joins between time series (e.g. on a shared timestamp) are pushed down only when the time series are in the same database, and group by and slices are pushed down only for a single resource, as before. Adding a resource that was added before moves it to the new virtualized database, so calling add_virtualization again with the same resources still replaces them. In Rust, ChrontextSettings holds a list of VirtualizationBackends (a virtualized database with its Virtualization) instead of a single database, built with ChrontextSettings::new or add_virtualization

  • TimescaleDB as a virtualized database (maplib Enterprise Edition): maplib.virtualization.TimescaleDatabase(url=..., host=..., port=..., database=..., user=..., password=..., sslmode=...), queried with the psycopg package (version 3), which must be installed. Its dialect, TimescaleDialect ("timescale"), is PostgreSQL with chrontext:dateBin and chrontext:FloorDateTimeToSecondsInterval computed by TimescaleDB’s time_bucket, which also takes strides in months and years. The session time zone is UTC, and PostgreSQL numerics with a fractional part come back as doubles

  • Siemens Insights Hub (formerly MindSphere) virtualization (maplib Enterprise Edition): Model.map_insights_hub(host, tenant, client_id=..., client_secret=...) reads the assets of a tenant (or of root_asset and its descendants) from Asset Management v3 into the graph, with their hierarchy, asset types (as classes, with their inheritance), locations, asset variables, aspects, aspect types and variables (data type, unit, quality code), and virtualizes the time series of the variables of dynamic aspects, with the external id assetId/aspectName/variableName. Their data points (chrontext:hasTimestamp, chrontext:hasValue for numbers and booleans, insights_hub:hasStringValue for strings and timestamps, insights_hub:hasQualityCode) are read from IoT Time Series v3 when queried, limited by the bounds on the timestamp in filters and paged by time, with concurrent requests per asset and aspect; a query without a lower bound on the timestamp needs default_start_time (an ISO 8601 timestamp, or a duration such as P7D before now). With aggregate_pushdown=True, MIN, MAX, SUM, COUNT and AVG of the values in buckets of chrontext:FloorDateTimeToSecondsInterval of whole minutes dividing an hour, or whole hours dividing a day, over time ranges on the bucket grid, are read from IoT Time Series Aggregates v4 instead (which leaves out values of bad quality, and counts a value exactly on a bucket boundary in the bucket before it). Authentication is with service credentials (OAuth client credentials at https://{tenant}.piam.{region}.{domain}/oauth/token, derived from a host like gateway.eu1.mindsphere.io, or token_url), app credentials through the Technical Token Manager (app_name, app_version, user_tenant), or a token. Tokens are renewed before they expire and on a 401, and throttled requests (429) are retried. Only the Python standard library is used for HTTP. In Rust, virtualization::insights_hub::read_insights_hub reads the asset model and Model::add_insights_hub adds it

  • Azure Data Explorer and Microsoft Fabric Eventhouses (Real-Time Intelligence KQL databases) as virtualized databases (maplib Enterprise Edition): VirtualizedDatabase(KustoDatabase(cluster=..., database=...), resource_sql_map={"my_resource": "Telemetry | project id=DeviceId, timestamp=Timestamp, value=Value"}), from maplib.virtualization. The resources are KQL tabular expressions with an id column of type string and a column for each other template parameter, and the virtualized queries are translated to KQL (maplib.virtualization.translate_kql, KQLDialect): filters (time ranges first), summarize by bin_at for chrontext:FloorDateTimeToSecondsInterval and chrontext:dateBin, the SPARQL aggregates (count_distinct for COUNT(DISTINCT)), joins, ordering and slicing, with the results of the SQL translation where Kusto differs from SQL (joins do not match null keys, comparisons with null are null, SUM and AVG of no values are null). Queries use the azure-kusto-data package, with a service principal (tenant_id, client_id, client_secret) or the default Azure credentials (azure-identity), and return all rows (notruncation). The column types of each resource are looked up with getschema

  • SPARQL 1.1 Federated Query: SERVICE <endpoint> { ... }, SERVICE SILENT and SERVICE ?endpoint (with the variable bound before the SERVICE) in queries and updates. The pattern is sent to the endpoint with the SPARQL 1.1 Protocol and its results (SPARQL JSON, XML or TSV, with their RDF types and unbound values) are joined with the rest of the query, also within OPTIONAL, UNION, MINUS and FILTER. SERVICE is evaluated after the local patterns of its group, and the distinct values that they bind for the shared variables are sent with the pattern as VALUES blocks (a bind join, batch_size rows per request, concurrency requests at a time), so that the endpoint only returns matching solutions; the bindings of the left side of an OPTIONAL are sent to a SERVICE in the optional part. When the endpoint is a maplib server (Model.serve(), maplib Enterprise Edition), the results come in maplib’s own Arrow format, and the bindings are sent as Arrow too, in batches of arrow_batch_size rows. SERVICE SILENT gives a single empty solution when the endpoint fails, times out or can not be reached. Model.set_federation(allowed_endpoints=None, headers=None, timeout=300, max_rows=None, batch_size=1000, arrow_batch_size=1000000, concurrency=4, method="POST", bind_join=True, allow_load=False) sets which endpoints may be reached (by URL prefix, any http(s) URL by default), headers per endpoint (e.g. a bearer token), the timeout and a row limit per SERVICE; redirects are only followed to allowed URLs. Refused endpoints and too many rows are errors also with SILENT. In Rust, the client is external_sparql::federation, Triplestore::federation holds the settings, QuerySettings::federation overrides them, and Model::query_with_federation and update_with_federation take them

  • SPARQL Update LOAD <url> [INTO GRAPH <g>] and LOAD SILENT read an RDF file from an http(s) URL (the format from the media type of the response, else from the extension of the URL), object storage (s3://, gs://, az://) or a file:// URL (only when listed explicitly in allowed_endpoints), into the default graph or the given graph. LOAD is refused unless enabled with Model.set_federation(allow_load=True), and only reaches the allowed endpoints

  • Model.serve() (maplib Enterprise Edition) refuses SERVICE and LOAD unless allowed: serve(service_endpoints=[...]) lists the endpoints that SERVICE in queries and updates sent to the server may reach (["*"] for any http(s) URL), with the other federation settings of the model, and serve(allow_load=True) allows LOAD of those URLs (with allow_update). Refused requests get 403. /sparql also gives SELECT results in maplib’s solution mappings format (application/vnd.maplib.solution-mappings) when asked for, and POST /arrow/query takes PVALUES parameters as Arrow, sent after the query command (application/vnd.maplib.parameterized-query), as used by SERVICE from another maplib

  • The W3C SPARQL 1.1 federated query tests (sparql11/service) are included in the tests, run against local mock endpoints with SPARQL JSON, XML and TSV results

  • Function calls in templates: template arguments can be computed with SPARQL functions, written Lisp-style as in Frog (M. Jarholt, “Frog: Functions for ontologies — An extension for the OTTR-framework”, Master’s thesis, University of Oslo, 2022, https://hdl.handle.net/10852/96925), e.g. ottr:Triple((IRI (CONCAT "http://example.net/" ?id)), rdfs:label, (UCASE ?name)). The functions are the SPARQL built-in functions and operators, XSD casts such as (xsd:integer ?x), UDFs, Frog’s base functions that have a SPARQL counterpart, and functions defined in stOTTR documents with Frog’s def syntax, e.g. def ex:toIRI(xsd:string ?id) -> ottr:IRI :: (IRI (CONCAT "http://example.net/" ?id)) ., which are inlined. Function calls are evaluated by the SPARQL expression engine over all rows of an instance at once, before list expansion. A call that gives no value (an error, as in SPARQL) is none: optional parameters are left out, defaults are used, and otherwise the instance is not expanded. Function calls can not be list expanded or take lists, and defined functions can not be recursive. In Python, FunctionCall("CONCAT", [...]) is a term for Argument, Instance and Triple. In Rust, StottrTerm::FunctionCall, Statement::Function, TemplateDataset::functions and templates::functions. In the templates of virtualized resources (maplib Enterprise Edition), the function calls are evaluated on the rows the virtualized database returns, and filters on their values are not pushed down

  • stOTTR documents can give none as an argument or default value, which is how templates with a None argument are printed

  • Shortest paths: the magic predicate maplib:shortestPath (https://datatreehouse.github.io/maplib/vocab#shortestPath) finds the shortest paths from a source to targets, with their length, cost and nodes, in SPARQL queries. Its options are named as those of the length-constrained maplib:path. The sources, targets (maplib:object) and nodes to avoid (maplib:avoid) are the values the other patterns of the group give their variables, or constants, and an unbound object means every node reached. Options: predicate and inversePredicate (the edges and their direction), anyPredicate "forward"|"backward"|"both" (every predicate whose objects are IRIs or blank nodes, except rdf:type) with excludePredicate, minLength (default 1), maxLength, limit (the nearest targets of each source), weight and defaultWeight (node costs, Dijkstra), maxCost, kShortest (k shortest simple paths, Yen) and allShortest / maxPaths; outputs length, cost, pathIndex, pathNodes (a JSON array), and one row per node with pathNode, nodeIndex, stepPredicate and stepInverse. The search runs on an integer graph of the edge predicates in compressed sparse rows, cached until their triples change, with bidirectional breadth first search for single targets, early stopping, and parallel searches from different sources. See the README

  • DuckDB sandbox (maplib Enterprise Edition): DuckDBDatabase(..., sandbox=True, allowed_directories=..., allowed_paths=...) turns off DuckDB’s enable_external_access and sets lock_configuration before any resource SQL runs, so the SQL of a resource (or given to maplib.config.probe) can not read local files (read_csv('/etc/passwd')), ATTACH other databases, COPY to files, install extensions or SET enable_external_access = true again. The database file stays usable, as do the allowed directories (with subdirectories) and files. maplib.config takes sandbox, allowed_directories and allowed_paths as parameters of a duckdb virtualization, and probe connects in the same sandbox as apply_virtualization. The sandbox is opt-in, so existing models whose resource SQL reads files next to the database (e.g. read_parquet) keep working; it is recommended whenever users who should not read the process’s files can configure a virtualization. A deployment enforces it with the environment variable MAPLIB_DUCKDB_SANDBOX=1: every DuckDBDatabase is then sandboxed, sandbox=False is refused, the database file must already exist (no files are created through database), and the database file and the allowed directories and paths must lie in MAPLIB_DUCKDB_ALLOWED_DIRECTORIES (separated by os.pathsep; when it is unset, no further directories can be allowed). FabricLakehouseDatabase is not sandboxed, as it needs extensions and network access

Bug fixes

  • The warning that full text search is only defined for named graphs or the default graph is no longer logged for queries that do not use full text search

  • maplib.add_triples(source, target, source_graph, target_graph) copies the triple tables natively in Rust instead of mapping each predicate through a temporary template from Python, and keeps the blank nodes of the source distinct from those of the target: before, blank nodes with the same label in the two models (e.g. _:b read into each) were merged into one. Each copy from another model gets new blank nodes; a copy within one model (source and target the same, e.g. to copy a graph into another graph) keeps them, as the graphs of a dataset share blank nodes. In Rust, Triplestore::copy_triples and Model::add_triples

  • Queries joining on two variables, where many solutions share the value of one of them, took memory quadratic in the number of solutions once a basic graph pattern had more than about 100 000 solutions: Polars’ join reordering moved a join on only one of the variables first, e.g. the two ua:BrowseName patterns of ?type_child ua:BrowseName ?browse_name . ?instance_child ua:BrowseName ?browse_name, so every instance child met every other child with the same browse name. The results were right. maplib now keeps the join order it chose from the patterns (utils::polars::keep_join_order). The OPC UA browse path query of the typing shapes for 100 000 instances with a common browse name needs 0.15 GB instead of over 134 GB

  • Reading RDF/XML (also CIM/XML) bounds the expansion of <!ENTITY> declarations, so a small file with nested entities (“billion laughs”) can no longer exhaust memory. An entity value may expand to at most 1 MiB, and all references to declared entities in a document to at most 10 MiB plus 10 bytes per input byte read, which ordinary use of entities for namespace IRIs (as in OWL files) stays far below. Exceeding a limit is a syntax error naming the entity. External entities (SYSTEM, PUBLIC) are still refused and never fetched. Model.map_xml, which also reads OPC UA NodeSet2 files, ignores the DOCTYPE and never expands declared entities, so it was not affected. In Rust, RdfXmlParser::with_max_entity_expansion, with_max_total_entity_expansion and with_entity_amplification_factor change the limits

  • Model.map_xml (and the OPC UA NodeSet2 mapping, which uses it) lost the text at entity and character references: <item>Pump &amp; motor &#65;</item> gave the two literals "Pump" and "motor". Character references and the five predefined entities (&amp;, &lt;, &gt;, &apos;, &quot;) are now resolved, and adjacent text, CDATA and references are joined into one literal, trimmed as a whole, so the example gives "Pump & motor A". Other entity references, including entities declared in a DOCTYPE, are dropped and never expanded, as before

  • Printing a Template parsed from stOTTR (str, repr, or the list from Model.get_templates()) panicked with “not yet implemented”. Templates print their annotations as @@ instances after the signature, and the printed text can be added again with add_template. Templates without annotations no longer get an empty annotation list from the parser

  • Chrontext (maplib Enterprise Edition): a UNION of static patterns joined with time series, e.g. { ?w a A } UNION { ?w a B } ?w :hasTimeseries ..., no longer returns every solution twice

  • maplib.virtualization.DuckDBDatabase works with DuckDB 2, which has no pl() on the result of execute

  • Model.create_index (and maplib.config.apply_config) creates the object indexes it is given for the triples already in the model, not only for triples added afterwards, and drops the object indexes of predicates that no longer have one. Sorting the stored triples by object takes about as long as adding them did. In Rust, Triples::set_object_indexing turns the object index of a triple table on or off

  • The expression of a Not expression in maplib.vq was None, so virtualized queries with a negated filter could not be translated to SQL

  • Virtualized queries with unary minus or plus (e.g. ORDER BY DESC(?v * -1) or FILTER(-?v < 0)) panicked with “not yet implemented” for Python databases (SQL databases, Kusto, OPC UA, Cognite). They are now Expression.UnaryMinus and Expression.UnaryPlus in maplib.vq and are translated to SQL and KQL; IRI constants are Expression.IRI (with iri), and other expressions that can not be passed to a Python database raise an error instead of panicking

  • Length-constrained property paths: the {n}, {n,m}, {n,} and {,m} path syntax of Apache Jena ARQ (and of the SPARQL 1.1 drafts), e.g. ?s (ex:p|^ex:q){2,4} ?o, with set semantics like the other paths, and the maplib:path magic predicate, e.g. ?s maplib:path [ maplib:predicate ex:p ; maplib:minLength 2 ; maplib:maxLength 4 ; maplib:object ?o ; maplib:length ?len ]. It takes one or more predicates and inverse predicates (or any predicate), returns the length (maplib:length), only the shortest paths (maplib:shortest true) or the nodes of each path (maplib:pathIndex, maplib:nodeIndex, maplib:pathNode), and maplib:pathMode "walk" | "acyclic" | "simple" chooses whether paths may revisit nodes. Either end may be bound or unbound, and paths are searched from the bound end with a breadth-first search per start node over the numbered nodes, deduplicated per node and length. See the maplib:path section of API_USAGE.md. In Rust, SparqlError::PathError reports invalid options

  • Zero-length property paths (ex:p*, ex:p?, ex:p{0,n}) from a variable bound by other patterns or by VALUES now match each of its values that is a node of the queried graph (a subject or object of some triple), as SPARQL 1.1 requires, also nodes without ex:p triples. Before, ?x a ex:Node . ?x ex:p* ?y left out the nodes without ex:p triples

  • Datalog (maplib Enterprise Edition): a rule whose BIND combines a variable of inferred triples with a variable of triples that no rule changes (e.g. BIND(IRI(CONCAT(STR(?p), "_", STR(?n))) AS ?p2)) inferred nothing after the first iteration, while debug=True gave the right result

  • Datalog (maplib Enterprise Edition): in rules where an OPTIONAL or a FILTER used only triples that no rule changes, the cached part could lose the variables that join it with the rest of the rule, so that an OPTIONAL matched values of unrelated nodes. Results differed from debug=True

  • SHACL: parameter declarations (sh:parameter) are no longer read as property shapes, which failed on parameters with e.g. a list of datatypes

  • JSON results of SELECT queries (return_format="json") list the variables in the order of the projection, instead of an arbitrary order

  • Model.query_external no longer panics (“entered unreachable code”) on SELECT results where a variable is unbound in some rows, as with OPTIONAL. Unbound values are nulls, also before the first bound value, and a variable that is never bound is a column of nulls

  • INSERT { GRAPH ?g { ... } } WHERE { ... } and DELETE { GRAPH ?g { ... } } in Model.update use the graphs that ?g is bound to. The graph name variable was left out of the solutions, so nothing was inserted or deleted when it appeared only as a graph name in the template

  • Model.insert returned only one of the tables of a predicate whose subjects or objects have several types (e.g. both IRIs and literals as objects); it now returns all of the inserted triples of the predicate in one DataFrame. In Rust, triplestore::new_triples_by_predicate merges new triples per predicate

  • Reading, mapping and inserting triples could hang with no CPU use. The triple tables of different predicates and types are built, compacted and deleted from in parallel, and each of these steps queried its table with the streaming engine of Polars, which can wait for threads that the parallel steps have taken. They now use the in-memory engine, which is safe there and also avoids the fixed cost of a streaming query per table

  • Model.query(debug=True) panicked with “not yet implemented” when a query with no results contains GRAPH ?g { ... }. The explanation now reports the GRAPH ?g pattern as a whole when it has no solutions (with GRAPH <g>, the inner pattern is explained in that graph as before). DESCRIBE queries are explained like the other query forms, and if the explanation itself fails, the query returns its results without debug output instead of failing

Performance

generate_templates is implemented in Rust (maplib::template_generator::generate_templates) instead of Python, with the same templates, parameter names, order and types, and runs in about 60% of the time on FIBO.

Datalog and SHACL rule inference with many small iterations is faster, about 2× for a long chain of semi-naive iterations and 15-20% for the Datalog benchmarks: CONSTRUCT results, encoding of new solutions and small solution mappings are collected with the in-memory engine instead of the streaming engine, small rank sorts no longer use the thread pool, and an in-memory query result is picked up within a millisecond of being ready instead of up to 50 ms. Evaluating a triple pattern outside a named graph no longer logs a full text search warning when there is no full text index.

Disk storage answers lookups of all properties of a subject (SELECT ?p ?o WHERE { <e> ?p ?o }) about 10× faster: at 20 M triples, 7 ms instead of 86 ms, against 2 ms with memory storage. Lookups with a known predicate are about 2× faster. Decoding a few results no longer joins them with the whole IRI and literal dictionary (maplib Enterprise Edition), a bound subject or object is encoded once per triple pattern instead of once per predicate table, and the few row groups a bound subject needs are read directly instead of through a lazy scan per range. Queries that return many rows, such as LIMIT 100000, are as slow as before with disk storage (about 90 ms at 20 M triples, 3 ms in memory).

SHACL rules run on the datalog engine with semi-naive evaluation: when a rule runs again, it only joins the triples inferred since its last run with the existing ones, parts of the rule bodies that no rule can change are computed once, and rules whose inputs did not change are not run again. Previously each iteration ran every rule on all of its focus nodes, so recursive rules (e.g. transitive closures) and long rule chains took time quadratic in the number of iterations. This also holds when the inferred predicate is used inside a property path, as in $this ex:child/ex:offspring? ?o. Rules that only add triples (no OPTIONAL, MINUS, NOT EXISTS or aggregates) and share an order see each other’s new triples in the same pass, so chains of such rules finish in fewer passes.

When inferences go to a separate graph, datalog and SHACL rules no longer compare inferred triples with the data graph for predicates that the data graph has no triples of. SHACL validation without rules no longer reads the shapes graph for rules, and finds parameter declarations with one query instead of a filter in each shape query.

In disk mode the triple tables and their deduplication index are no longer held in memory. With few triples per distinct IRI or literal (as in the CIM data), memory use is dominated by the dictionary and reading, and disk storage saves little.

Triple table segments are compacted much faster, which matters most for datalog, where every iteration adds a small batch of triples to tables that are often large:

  • Size-tiered compaction: only the trailing segments of comparable size are merged with the new triples, so a large table is no longer rewritten every 11 inserts. A table has O(log n) segments (at most 16).

  • Merging sorts packed rank keys with Polars instead of a value-by-value heap merge, and builds the object sort from the merged triples instead of also reading and merging the object sorts of the segments.

  • Disk mode: new triples are deduplicated against only the row groups that may hold their subjects, found through the sparse subject index, instead of reloading every stored triple of the table into an index on each insert. Small trailing segments (under 16k triples) stay in memory until merged, so small inserts don’t write and rewrite small parquet files.

Query results are collected with the Polars streaming engine, which the rest of maplib already used; until now the final step of a query ran on the in-memory engine. Whole-table scans and counts, large result sets and DISTINCT are faster, while small queries are about as fast as before. MAPLIB_POLARS_ENGINE=in-memory switches back to the in-memory engine.

IN with a list of IRIs looks the values up in a hash set instead of comparing with each IRI in turn, so long lists no longer slow a query down. The same holds for IN with a list of integers (of any integer datatype) when the left side holds integers of one integer datatype. Property paths with *, + and ? map the computed paths back to IRIs and literals by position instead of with joins. Patterns without shared variables (also inside OPTIONAL) are combined with a parallel cross join instead of a join on a constant key, which ran on one thread.

After Model.compact(), subjects are marked as sorted, so DISTINCT and grouping on subjects are faster.

ORDER BY on IRIs and strings no longer collects its input in memory for each sort column, and with memory storage it ranks the values batch by batch, so large sorts are several times faster and spill to disk under the memory limit. ORDER BY with LIMIT or OFFSET, and joins and aggregations, still keep their working state in memory.

ORDER BY with a large LIMIT (with or without OFFSET; LIMIT plus OFFSET at least 10 000 and at least a thirty-second of the estimated solutions) is collected with the in-memory engine, which sorts much faster than the streaming engine when many rows are kept. Such queries are several times faster, and many times faster for some sort keys. Smaller LIMITs are left to the streaming engine, whose top-k is faster there. Their working state is kept in memory, so they do not spill to disk under the memory limit. MAPLIB_POLARS_ENGINE=streaming keeps the streaming engine for them.

GRAPH ?g in the templates of Model.update partitions the solutions by the encoded graph IRIs in one pass, and decodes one IRI per graph, instead of decoding every row and filtering the solutions once per graph.

Deleting triples that are not in a table no longer rewrites the table, so DELETE DATA and DELETE WHERE of absent triples leave the stored triples and their indexes as they are. The count of deleted triples that decides when memory storage is compacted counts only the triples that existed.

Tantivy answered exact matches from its term dictionary, but scanned all strings for substring matches, which it ran whenever the exact matches did not fill the limit. DuckDB scans the strings once per query for both. Reading without full text search is unchanged.

Serialization, compared with the build before these changes on CIM N-Triples (2.2 M triples, IRI-heavy) and a synthetic literal-heavy file (4.5 M triples, 3 M distinct literals), release-test profile, medians of 3 runs alternating between builds, each run in a fresh process. Parquet is the new default, raw_db writes DuckDB dictionaries as before:

Category

Workload

CIM 2.2 M: parquet / raw_db

Literals 4.5 M: parquet / raw_db

Writing

serialize (model already compacted)

1.97 → 0.43 s (-78%) / 0.73 s (-63%)

3.90 → 0.79 s (-80%) / 1.37 s (-65%)

Writing

Size on disk

107 → 35 MB (-67%) / 107 MB

344 → 103 MB (-70%) / 345 MB

Reading

deserialize, memory storage

0.12 → 0.06 s (-51%) / 0.06 s (-51%)

0.20 → 0.10 s (-49%) / 0.15 s (-21%)

Reading

deserialize, hybrid storage

0.05 → 0.55 s (+1115%) / 0.01 s (unchanged, used in place)

0.06 → 0.72 s (+1094%) / 0.01 s (unchanged, used in place)

Reading

deserialize, disk storage

0.03 → 0.56 s (+1964%) / 0.02 s (unchanged, used in place)

0.02 → 0.74 s (+3580%) / 0.01 s (unchanged, used in place)

Disk

After deserialize with disk storage: count per predicate

0.079 → 0.043 s (-46%)

0.081 → 0.046 s (-44%)

Disk

Same, insert 100 k triples (half new subjects)

unchanged (within noise)

0.224 → 0.171 s (-24%) / 0.193 s (-14%)

SPARQL

After deserialize: 50 subject lookups (memory / hybrid / disk), insert (memory / hybrid)

unchanged (within noise)

unchanged (within noise)

  • Serialized triple tables are written with row groups of 16,384 rows, as disk storage writes its own triple tables, instead of 1,000 rows. Disk storage copies the files as they are on deserialization, and scans read far fewer row groups.

  • Dictionaries are built into DuckDB (hybrid and disk storage, and raw_db) by loading the rows first and building the unique indexes afterwards, instead of inserting into an indexed table, and the dictionaries are deserialized concurrently with the triple tables.

  • With hybrid and disk storage, parquet dictionaries are built into DuckDB databases (including their unique indexes) on deserialization, while raw_db dictionaries are used in place, so use raw_db=True when a serialized model is loaded into hybrid or disk storage often.

Faster reading, ORDER BY, counting and deleting (memory storage):

  • Dictionary ranks (used to sort triples when they are added or compacted, and for ORDER BY) are looked up in arrays indexed by the dictionary code instead of hash maps, and the distinct codes of a batch are found with a bitset instead of a hash set. The deduplication index of triple tables uses a faster hash function.

  • The deduplication index of a triple table is built when triples are first added to an existing table, not when the table is created, so tables that are only read and queried never hold it.

  • Triple tables of different predicates and types are built in parallel when triples are added.

  • New dictionary entries are merged into the dictionary in one pass when they are many compared with its size, instead of one at a time.

  • COUNT(*) (or COUNT of one of the variables) over ?s ?p ?o in a single graph is answered from the sizes of the triple tables.

  • DELETE WHERE { ?s ?p ?o } and DELETE WHERE { ?s <p> ?o } (also as DELETE { .. } WHERE { .. } without INSERT or USING), matched in the graph they delete from, drop the triple tables without evaluating the pattern.

  • Garbage collection of the dictionary after large deletes marks the used codes in a bitset instead of building range sets per table segment.

Category

Workload

Before

After

Change

Reading

N-Triples, 20 M triples (4 predicates, 2 M subjects)

19.6 / 20.1 s

10.6 / 10.9 s

-46%

Reading

N-Triples, 8 M triples (400 predicates)

4.50 s

4.38 s

-3%

Reading

RDF/XML, CIM 2.2 M triples

1.71 / 1.90 s

1.23 / 1.28 s

-28% / -33%

SPARQL

ORDER BY over 5 M IRIs, LIMIT 10

0.29 s

0.077 s

-74%

SPARQL

ORDER BY over 5 M string literals, LIMIT 10

0.73 s

0.077 s

-90%

SPARQL

COUNT(*) of all 20 M triples

0.128 s

< 1 ms

-99.9%

SPARQL Update

30 CONSTRUCT inserts of ~2 k triples into a 5 M triple table

4.13 s

4.13 s

unchanged

SPARQL Update

DELETE WHERE { ?s <p> ?o }, 5 M triples (incl. garbage collection)

0.348 s

0.067 s

-81%

SPARQL Update

DELETE WHERE { ?s ?p ?o }, 15 M triples (incl. garbage collection)

0.63 s

0.57 s

-9%

Memory

Peak RSS, 20 M triple workload

6.14 GB

6.44 GB

+5%

(Release-test profile, Apple M-series laptop (14 cores), medians of 3 runs alternating with the build before this change, while other jobs were running on the machine. Where two numbers are given they are from two rounds of runs. RSS after reading varied by more than ±1 GB between runs of the same build, so it is not reported; the deduplication index that is no longer built takes about 14 bytes per triple. The higher peak comes from building tables in parallel.)

In-memory dictionaries store their strings back to back in a chunked arena, with flat per-entry vectors of offsets, lengths, prefixes and codes, a hash table from strings to entries and an array from codes to entries. This replaces a BTreeMap from strings to codes, a reverse HashMap and a separate allocation per string: about 30 bytes per entry on top of the string instead of about 100. String order is no longer maintained on every insert, but computed when needed (ranks, compaction, ordered output) and cached; after inserts only the new strings are sorted and merged into the cached order. Looking up a string is a hash probe instead of a walk down a B-tree.

Category

Workload

Before (with the changes above)

After

Change

vs. v0.21.0

Reading

N-Triples, 20 M triples (4 predicates, 2 M subjects)

11.7 s

8.3 s

-29%

-62%

Reading

N-Triples, 8 M triples (400 predicates, ~8 M distinct strings)

5.00 s

2.96 s

-41%

-44%

Reading

RDF/XML, CIM 2.2 M triples

1.76 s

1.12 s

-36%

-43%

Memory

Peak RSS, 8 M triples

3.33 GB

2.44 GB

-27%

-31%

Memory

Peak RSS, CIM 2.2 M triples

1.51 GB

1.28 GB

-15%

-19%

Memory

Peak RSS, 20 M triple workload

6.10 GB

5.66 GB

-7%

-2%

SPARQL Update

DELETE WHERE { ?s ?p ?o }, 15 M triples (incl. garbage collection)

0.68 s

0.26 s

-62%

-75%

SPARQL

ORDER BY, COUNT, inserts, predicate deletes

unchanged

(Medians of 2 runs per variant alternating between v0.21.0, the build with the changes above and this build, under heavy memory pressure from other jobs, so RSS after reading is not reported. The 20 M triple workload peaks while parsing, before the dictionary is complete, so the dictionary saves less there.)

Faster encoding and decoding with in-memory dictionaries:

  • Reading, mapping and inserting encode string columns directly with global codes: all values are looked up in the dictionary in parallel, the new ones are deduplicated in parallel and added once with consecutive codes. Before, each column got its own local dictionary, which was then merged into the global dictionary and its codes re-encoded. Hybrid and disk storage keep the previous path.

  • Repeated values next to each other are looked up once, and constant columns (e.g. from templates) are encoded once instead of once per row.

  • Query results and formatted values (<iri>, _:blank, "string") are decoded straight into string arrays in parallel, without building a string per value first.

Category

Workload

Before (with the changes above)

After

Change

vs. v0.21.0

Reading

N-Triples, 20 M triples (4 predicates, 2 M subjects)

6.49 s

3.98 s

-39%

-81%

Reading

N-Triples, 8 M triples (400 predicates)

2.26 s

1.53 s

-32%

-64%

Reading

RDF/XML, CIM 2.2 M triples

0.93 s

0.60 s

-35%

-68%

Mapping

Template with 4 triples (2 constant), 3 M rows

1.36 s

1.09 s

-20%

-89%

SPARQL

SELECT 5 M rows of IRIs / string literals

0.152 / 0.090 s

0.121 / 0.085 s

-21% / -6%

-32% / -36%

Memory

Peak RSS, 8 M triples / CIM / mapping / 20 M triples

2.58 / 1.29 / 2.38 / 5.50 GB

2.50 / 1.30 / 2.31 / 5.69 GB

-3% / 0% / -3% / +3%

-31% / -17% / -39% / -3%

(Medians of 2 runs alternating with the build before this change. The v0.21.0 column uses v0.21.0 runs from an earlier round on the same machine.)

Faster lookups, property paths and small inserts:

  • Property paths (+, *, ?) with a bound end, a constant IRI or a variable whose values are known from VALUES or earlier patterns, follow the paths from those nodes only, step by step, instead of computing the transitive closure of the whole graph and filtering it.

  • The values of a pattern’s variables are pushed down from solution mappings of up to 100 000 rows (was 500). Lookups of many values walk the sorted values and the sparse index together instead of searching the index once per value, and read an in-memory table with one gather instead of one slice per range. When the ranges cover more than half of a table the whole table is read, and at most 256 ranges of a stored table are read.

  • A small insert or delete into a large in-memory table without a deduplication index checks only the stored triples with the same subjects, instead of indexing every triple of the table first.

Category

Workload

Before (with the changes above)

After

Change

SPARQL

VALUES with 20 000 subjects joined with a 5 M triple table

186 ms

39 ms

-79%

SPARQL

The same, then a second pattern on the ~25 000 objects found

213 ms

38 ms

-82%

SPARQL

VALUES with 2 000 subjects (alone / with a second pattern)

24 / 27 ms

24 / 23 ms

0% / -16%

Property paths

ex:t199999 ex:parent+ ?a in a 200 000 node tree

319 ms

24 ms

-92%

Property paths

?d ex:parent* ex:t3 (65 535 results)

158 ms

22 ms

-86%

Property paths

ex:parent+ from 1 000 VALUES

206 ms

34 ms

-84%

Inserting

200 triples into a 20 M triple graph (first / second insert)

7 / 16 ms

6 / 15 ms

~0%

Inserting

30 CONSTRUCT inserts into a 20 M triple graph

4.14 s

4.27 s

+3%

Memory

Peak RSS, lookups / 20 M triples with inserts

5.62 / 5.70 GB

5.39 / 5.62 GB

-4% / -1%

(Medians of 2 runs alternating with the build before this change, under heavy load from other processes; differences below ~5% are within the noise.)

Reading and writing HDT is much faster. Reading decodes each term of the HDT dictionary once and maps it directly into the model’s dictionaries, then adds the triples as integer IDs in parallel batches, without building HDT query indexes. Writing deduplicates and maps triples using the model’s dictionary codes, decodes only the distinct terms, and writes the triples section without building query indexes. The written files are unchanged. On a CIM model of 2.2 million triples, reading takes 0.28 s instead of 3.41 s in v0.20.32 (12x faster) and writing 0.20 s instead of 5.69 s (28x faster)

Bug fixes

  • SPARQL: +, -, * and / were parsed as right associative, so 8 - 3 - 2 gave 7 and 12 / 2 / 3 gave 18. Operators of the same precedence are now left associative, as the SPARQL grammar specifies. Serialized queries put arithmetic in parentheses, so that 8 - (3 - 2) and (1 + 2) * 3 keep their meaning

  • Ctrl-C now stops a running query. Before, a long query could keep running until it finished

  • STRDT with xsd:dateTime or xsd:dateTimeStamp dropped the time zone, e.g. STRDT("2015-03-15T15:02:37-05:00", xsd:dateTime) gave "2015-03-15T20:02:37". The time zone is now kept, also when the values have different time zones

  • A triple pattern with the same variable in two positions (e.g. ?s <p> ?s) panicked, in queries and in DELETE WHERE, when the predicate also had triples where the two positions had different types (e.g. blank node subjects with IRI objects)

  • Hybrid storage: Model.compact() (also run by Model.serialize()) failed with a DuckDB “Duplicate key” error once the dictionary had to be renumbered, e.g. after more than ten inserts into the same predicate

  • SHACL rules: a rule that re-asserts a triple already in the data graph no longer makes inference loop until max_iterations (or forever with one rule). Inserting transient triples that all already existed non-transiently dropped the earlier transient triples with the same predicate, which were then inferred again on every iteration (#76)

  • SHACL rules: rules whose WHERE clause has no triple pattern on $this (only OPTIONAL / BIND) now also run for focus nodes that become targets of their shape after the rule first ran (#77)

  • SHACL: shapes with several sh:message values (e.g. one per language) lost the results of property shapes, so data violating them was reported as conforming, kept only one of the messages of node shapes, and gave one result per message for SPARQL constraints. Every result now has all the messages of its shape as sh:resultMessage values. Property shapes with several sh:name or sh:description values also lost their constraints (#86)

  • HDT: writing the same model twice could give different files, as the properties of the triples section’s control information were written in random order

  • HDT: literals are written with their values as they are, as hdt-java and hdt-cpp do, rather than N-Triples escaped, so other HDT tools no longer read a tab as \t. Reading takes a backslash in a literal of a file from hdt-java or hdt-cpp as part of the value, rather than as the start of an escape. Files written by earlier maplib versions are recognised and still read correctly. Writing a literal that contains U+0000 raises an error, as HDT cannot store it

  • HDT: the header of written files describes the dataset with the same vocabulary as hdt-java and hdt-cpp: IRIs for the dataset types and the dictionary and triples formats, and hdt:publicationInformation for the publication information

  • Full text search: with a fixed subject (ex:a maplib:fts ?fts), the limit was applied before filtering on the subject, so few or no results were returned. Triples deleted with SPARQL Update were still found, now they are no longer found after Model.compact(). An on-disk index no longer leaves a folder behind for every index created

  • Pretty Turtle and RDF/XML: the rdf:first and rdf:rest triples of blank nodes that do not form a well-formed list were not written, e.g. of a list without rdf:nil, a list node with two rdf:first, or a list whose later node is also referred to by another triple (shared or cyclic lists). Only lists where each node has one rdf:first and one rdf:rest, and every node after the first is only referred to by the rdf:rest before it, are written as lists; other rdf:first and rdf:rest triples are written as any other triples

  • Pretty RDF/XML of several graphs in one document (as for CIM/XML): a blank node used in more than one of the graphs could be nested or written as part of a list in one graph, so that the other graphs referred to a blank node that was not in the document. Such blank nodes are now written with their label

v0.21.0

Performance

Besides the changes listed below, the memory improvements earlier in this release contribute to the faster reading and SHACL validation.

  • RDF/XML is parsed in parallel, by splitting the document between top-level elements. The RDF/XML parser from Oxigraph (oxrdfxml) is now included in maplib to make this possible

  • RDF/XML is written with a new pretty serializer (see Features)

  • CIM/XML is written by building temporary graphs with two SPARQL queries driven by the profiles (one for the instances of the profile classes, one for their profile properties) and writing them with the pretty RDF/XML serializer, instead of running queries per class and property

  • Disk-backed models keep their category map databases open instead of opening a database for every lookup, which makes small queries about ten times faster

  • Inserting triples no longer ranks the inserted values against the whole dictionary, which made every insert cost time proportional to the number of distinct IRIs and literals in the model. This was most of the per-iteration cost of datalog inference

  • The sort order of the dictionary is cached between ORDER BY queries and datalog iterations, until new values are added

  • Removed debug output that SHACL validation printed for every sh:class constraint (it also evaluated the constraint a second time)

Breaking changes

  • The streaming parameter is removed from Model.query, Model.update, Model.insert, Model.validate, ValidationReport.results and ValidationReport.details. Polars already runs lazy queries with its streaming engine, so the parameter had no effect (benchmarks with streaming=True and streaming=False were identical). Remove the argument from calls

  • RDF/XML (and CIM/XML) is read in parallel by default (parallel=None). Pass parallel=False to read sequentially. Line numbers in parse errors may refer to the position within a chunk of the document

  • RDF/XML output is now formatted differently (one element per subject, nested blank nodes), but contains the same triples

  • write_cim_xml writes CIM/XML as IEC 61970-552 / CGMES requires (see Bug fixes). Objects are identified relative to the document (rdf:ID=””, rdf:resource=”#”), so read the files with the same base_iri as the original data, as for any CGMES file

Features

  • write_cim_xml uses the pretty RDF/XML serializer. The exported triples are the same as before (the FullModel header first, then one typed element per instance with the properties its class has in the profiles, literal values as plain strings), but elements and properties are ordered differently

  • Pretty RDF/XML serializer, working like the pretty turtle serializer: one node element per subject with all its properties, typed node elements when the type has a prefix, blank nodes that are used once nested where they are used, RDF lists written with rdf:parseType=”Collection” when possible, and all namespaces declared on the root element

  • ASK queries. m.query() returns a bool, or a SPARQL JSON boolean result with return_format=”json”

  • isNUMERIC and RAND SPARQL functions

  • SHA256, SHA384 and SHA512 SPARQL functions

  • Read and write TriG and N-Quads (format “trig” / “nquads”, or the .trig / .nq file extensions when reading). Writing these formats outputs all graphs unless a graph is specified

Bug fixes

  • write_cim_xml was not compliant with IEC 61970-552 / CGMES (also before this release):

    • Objects were written with absolute IRIs (rdf:about=”urn:…:#”); they are now rdf:ID=”” with rdf:resource=”#_” references, when all objects share one base IRI

    • Enumeration values (e.g. cim:WindingConnection.Y) were written as objects when the vocabulary was in the data graph

    • Objects that also had superclass types (e.g. cim:Equipment) were written once per type; they are now written once, as their most specific class

    • Association ends that the profiles mark as not used (cims:AssociationUsed “No”) are never written, and the used end is derived from them when the data only has the unused end

    • Only the namespaces that are used are declared, and the default Model.created / Model.scenarioTime are UTC with whole seconds (e.g. 2021-02-02T22:28:41Z)

  • Disk-backed models wrote subjects on the boundaries between internal segments twice when writing turtle or RDF/XML (giving invalid RDF/XML with a duplicate rdf:ID)

  • Writing turtle could hang forever (a deadlock between the pretty turtle writer’s parallelism and Polars), e.g. for CGMES/CIM models

  • sh:not required every constraint of its shape to fail, instead of the shape not conforming. Combined with sh:or this gave false results, e.g. 363 false sh:or results when validating CGMES shapes against the SHACL-SHACL shapes (pyshacl agrees there are none)

  • sh:not with a shape that has several constraints of which some pass and some fail (e.g. two sh:class values) treated the shape as conforming, giving false sh:not results

  • Inserting triples could drop triples whose values had been re-encoded in the dictionary, as their values could not be ranked when sorting

  • Removed the pyarrow <24.0.0 pin. pyarrow only publishes Python 3.15 wheels from 26.0.0, so the pin made maplib uninstallable on Python 3.15

  • COUNT now returns xsd:integer (Int64) as required by the SPARQL 1.1 spec, instead of xsd:unsignedInt (UInt32)

  • SUM, AVG, MIN and MAX now work when the aggregated variable has multiple types. SUM and AVG promote the numeric types in each group (integer < decimal < float < double) and are unbound if a group contains a non-numeric value, MIN and MAX use the SPARQL ordering and keep the type of the selected value

  • Division follows SPARQL op:numeric-divide: integer / integer gives xsd:decimal (e.g. 5 / 2 = 2.5, previously truncated to 2.0 as xsd:double), mixed types are promoted (decimal < float < double), and decimal division by zero is an error (unbound)

  • COALESCE over arguments of different types returns the first bound argument (previously a value of a later argument could be returned nondeterministically)

  • Fix panic when combining lang strings with differently encoded categoricals (e.g. COALESCE(?langString, “x”@en))

  • Infinite xsd:double and xsd:float values are written as INF and -INF (previously inf and -inf, which are not valid lexical forms) in query results, STR() and all RDF serializations

  • DATATYPE() of an unbound value or an error is unbound (previously the datatype of the variable was returned)

  • YEAR, MONTH, DAY, HOURS and MINUTES return xsd:integer and SECONDS returns xsd:decimal including fractional seconds (previously xsd:unsignedInt, dropping fractions)

  • STR() of xsd:dateTime and xsd:date values gives the XSD lexical form (e.g. 2010-06-21T11:28:01+00:00 instead of 2010-06-21 11:28:01.000000+00:00)

  • Triple patterns without variables (e.g. { :s :p :o }) never matched. They now give one solution if the triple exists

  • GROUP_CONCAT concatenated internal codes instead of the string values, and used an empty default separator instead of a single space. It now concatenates the string values (as STR()) of any type, including lang strings

  • ABS, CEIL and FLOOR work when the argument has multiple types, and CEIL and FLOOR keep the type of the argument (previously xsd:integer)

  • CONCAT accepts zero or one arguments

  • Casting a blank node (e.g. xsd:double(?x)) gives an unbound value instead of failing the query

  • FILTER, IF, NOT (!), AND (&&) and OR (||) use the effective boolean value of their operands (SPARQL 1.1 17.2.2): numbers are true unless zero or NaN, strings are true unless empty, and other values are errors. Previously a non-boolean FILTER failed the query, and IF with an error condition took the else branch instead of being unbound

  • Casts to xsd:boolean, xsd:integer, xsd:decimal, xsd:float and xsd:double follow the XPath casting rules: strings must have a valid lexical form (e.g. xsd:boolean(“abc”) and xsd:integer(“1.5”) are unbound, previously false and a number), booleans can be cast to xsd:decimal, NaN and infinite values cast to integers or decimals are unbound, and casting other types such as dateTimes to numerics is unbound

  • xsd:string() of a decimal gives the canonical form (e.g. “2.5”), not the stored representation with padding zeros

  • “1”^^xsd:boolean and “0”^^xsd:boolean are valid booleans (previously rejected when parsing)

  • STRBEFORE and STRAFTER accept a language tagged second argument when it is compatible with the first argument, and give the simple literal “” when there is no match, as required by the SPARQL 1.1 spec

  • REPLACE keeps the language tag of its first argument

  • CONCAT only gives a language tagged result when all arguments have the same language tag, and is unbound when an argument is unbound or not a string

  • COUNT(DISTINCT ) is supported, and COUNT() works for patterns without variables

  • SELECT * over a pattern without variables (e.g. { :a :b :c }) no longer panics

v0.20.32

Features

  • Turtle output for construct queries

v0.20.30

Features

  • Add bindings parameter that replaces variables with ground terms (safe query parameterization without special syntax)

v0.20.29

Bug fixes

  • Fix query external for empty results, variable sequence and iri-columns.

v0.20.28

Performance

  • Pretty turtle write performance drastically improved (around 12x better)

v0.20.26

Features

  • The IRI maplib:DefaultGraph is the default graph

  • m.query_external(query, endpoint) lets you query an external SPARQL endpoint (currently only select and HTTP GET).

v0.20.25

Bug fixes

  • Fix issue where count over zero solution mappings returned no rows (should return one row with zero as the count)

Performance

  • Improve deserialization performance to hybrid storage and to in memory storage

v0.20.24

Bug fixes

  • Bug fix for serialization of lang strings

  • (Attempted?) Bug fix for disk based storage of categoricals on windows

v0.20.23

Features

  • Serialization with disk based storage of categoricals - about 40x faster than ntriples read

v0.20.22

Performance

  • Fix performance issue for ORDER BY when cats are on disk

  • Fix performance issue in queries with a lot of OPTIONAL clauses (see bugfix below)

Bug fixes

  • Fix issue where OPTIONAL clauses sometimes introduces duplicates

v0.20.21

Features

  • Support for OPC UA mapping to RDF using m.map_opc_ua() providing the folder with NodeSet2 XMLs as an argument

  • m.add_udf() allows the user to create a UDF working on Polars DataFrames where arguments form the columns. The function is callable from SPARQL.

  • m.serialize(“folder”) and Model.deserialize(“folder”) serializes and deserializes the triples in a Model object super quickly (20x faster than ntriples in our bench).

Breaking (but not worthy of major release IMO)

  • IndexingOptions no longer has subject_object_index argument as this is now default.

v0.20.20

Bug fixes

  • Avoid panic when using pyarrow >= 24.0.0

  • rep = m.validate() always produces rep.results() even when report_graph is specified.

Features

  • Facade X like mapping of dataframes to graph in m.map_df()

  • Target counts for shapes are now part of the SHACL report graph

v0.20.19

Bug fixes

  • Workaround for Polars issue that could sometimes produce errors in filter not exists

  • Move away from rdf:_{n} predicate to represent lists in Facade-X as it is does not play nice with our partitioning

Features

  • Prefix maplib:https://datatreehouse.github.io/maplib/vocab#

  • maplib:uuidv5 and maplib:struuidv5 functions

  • m.templates_to_graph() materializes templates as triples in graph so they can be queried (Thank you @thenonameguy !)

  • m.read(), m.reads() now eats hdt and m.write() spits it out. (Thank you @sihingkk !)

v0.20.16

Bug fixes

  • Work around Polars issue where double not exists can return erroneous results

  • Turn caching mechanism for on disk SHACL back on :-)

  • Fix SHACL SPARQL issue with OPTIONAL {} that could return too few results.

  • Escape certain characters after the prefix in prefixed IRIS.

Performance

  • Move to latest Polars

v0.20.15

Performance

  • Move to A* search for optimal join sequencing - we do unfortunately not take into account nesting here yet.

v0.20.14

Bug fixes

  • Fix issue in cross join detection / query planning.

v0.20.13

Bug fixes

  • Bug fix for optimization for certain types of large queries with OPTIONAL and BIND.

  • Improve error message when UNBOUND(?v) does not find ?v.

v0.20.12

Bug fixes

  • Disk based cats panicked with duplicate string (large strings)

  • Sometimes tried to write triples to disk (disabled functionality for the time being)

Performance

  • Caching for SHACL rules with disk based storage improves validation performance

v0.20.11

Bug fixes

  • Exists queries with nested subqueries could be true when they should be false

Performance

  • Improve detection of cross joins, avoiding them better

Features

  • Integrate chrontext into maplib, enabled when m.add_virtualization() has been run.

v0.20.10

Bugfixes

  • SHACL rules: incrementally build targets not always updated

  • SPARQL adding now() and a few bugfixes having to do with multiple datatypes in expressions

Performance

  • Improvements to query planning for large queries

v0.20.8

Bug fixes

  • xsd:decimal is now represented internally as Decimal(precision=38,scale=12)

  • Fix another instance of missing query parameters in debug

Features

  • max_iterations parameter for shacl rules (validate())

v0.20.7

Bug fixes

  • Fix bug where comparison expressions involving multiple types could fail.

v0.20.6

Performance

  • Subject object indexing for deduplication now default on, improves load speeds for many small files

  • Batch size for triples read now defaults to 10_000_000, reducing memory consumption.

v0.20.5

Bug fixes

  • Fix issues with IF where having multiple types for “then” and “otherwise” could cause an exception

  • Fix issue where THIS was not injected into SHACL SPARQL

Features

  • rules_debug in validate now prints every step so it is easier to debug recursive issues

v0.20.4

Bug fixes

  • Fix rounding issue

  • Improve error messages from SHACL rules

v0.20.3

Bug fixes

  • Fix issue with regex and replace where pattern was of form str(iri).

  • Fix issue where debug_rules could sometimes panic in validate()

Features

  • Add support for STRDT

  • Add missing support for SPARQL targets in SHACL Rules

v0.20.2

Bug fixes

  • Fix caching issue in Datalog engine

  • Work around issue with timestamped dates, these are now read without the timestamp, but should be fixed properly at some point.

Features

  • SHACL rules now support recursion, and has multiple optimizations.

  • Still missing conditions, other rule types than sparql construct, and taking ordering into account.

v0.20.1

Bug fixes

  • Fix nondeterminism that could sometimes cause duplicate SHACL results

  • SHACL could sometimes contain extra results for deeply nested shape graphs

  • SHACL graph no longer includes conforms bool for results, this one is just needed when include_details=True

v0.20.0

Breaking changes

  • map_triples, map, map_default “df”-arg renamed to “data” as it now takes alternatively a DataFrame or a SolutionMappings object, in which case types can be specified for columns. A query run with solution_mappings=True can thus be piped into these methods.

  • parameters for query and update for the PVALUES construction now is a dict with SolutionMappings values.

  • ValidationReport no longer has method graph(), instead, use the report_graph argument of validate to specify a named graph for the report. This also means that validate() argument include_shapes_graph is gone.

Features

  • Rudimentary shacl rules support (only SPARQL construct), no recursion yet. Use the inferences_graph-argument to validate to persist inferred triples to a named graph.

Bug fixes

  • Fix for #48

v0.19.25

Features

  • SPARQL parser returns the bad part of the query

Bug fixes

  • Fix for FTS issue when enabling for multiple named graphs

  • Fix lang string issue with FTS

v0.19.24

Bug fixes

  • Fix for missing triples in pretty turtle serialization under certain conditions involving orphaned blank nodes.

v0.19.23

Features

  • Implement md5, sha1 and encode_for_uri functions.

Improvements

  • General improvements in read / map performance

  • Improvements in error handling for PValues, missing Function implementations

Bug fixes

  • Fix bug in recursive constraints in SHACL

  • Issue when using Datalog with SPARQL construct syntax

  • Issue with lang string literal in SPARQL construct template

v0.19.22

Improvements

  • Graceful error handling for SPARQL queries that crash in Polars.

Bug fixes

  • Contains function in SPARQL crashed with lang strings under certain circumstances

v0.19.20

Improvements

  • All types with string types are now internally represented as u32 (previously it was only xsd:string, blanks and iris)

Bug fixes

  • Result path missing in SPARQL constraints

v0.19.19

Bug fixes

  • Bug fix for issue affecting SPARQL constraints with lang string messages with unbound vars of multiple types.

v0.19.18

Features

  • CIM XML import removes superfluous “#_” from iris.

v0.19.17

Features

  • Basic support for disk based storage of categorical maps.

  • Fix extra characters in CIM XML URIs by default

Bug fixes

  • Multiple bugfixes for SHACL

v0.19.14

Features

  • read() and reads() for json-ld is now supported. Note that contexts are not resolved against the internet but must be provided in a dict due to security concerns.

Improvements

  • Pretty turtle writing has been parallelized, improving performance.

v0.19.13

Features

  • CIM XML parsing is now natively supported

Improvements

  • Added possibility of setting number of triples per batch when reading serialization formats.

  • map_json is now compliant with Facade X.

Bugfixes

  • Fixed various cases where a panic or unwrap happens instead of a proper error.

v0.19.10

Features

  • m.map_json() maps a json file or string to rdf triples.

Bugfixes

  • Writing turtle could sometimes write erroneous triples.

v0.19.9

Improvements

  • maplib.explore() is deprecated, use the explore method on a Model instead.

  • Model.explore() no longer supports opening a browser window automatically. Use webbrowser.open(s.url, new=2) instead. Where s is returned from explore()

  • Release of pretty turtle printer. It is still missing:

    • Anonymization of blank nodes that only have a single incoming reference.

    • Support for nested lists

  • Datalog inference now works on SPARQL construct in preparation for SHACL rules support

  • Improvements to prefix handling:

    • Default prefixes only shadowed by adding prefixes to Model

    • Prefixes set in pretty turtle print only shadow Model prefixes

Bugfixes

  • Fixed improper serialization of e.g. newlines in new pretty turtle printer

  • Fixed issue where empty triples could be added

  • Fixed several issues that could lead to OOM in the SHACL engine

  • Fixed panic in CIM xml exporter that should be error message

v0.19.6

Improvements

  • IRIs are now compressed in memory

  • SELECT * has column ordering according to when the variable is used in the query, not alphabetical as previously.

v0.19.5

Bugfixes

  • Inverse predicates missing from CIM exports

Improvements

  • Setting debug=True in query now prints the debugging of why the query has no solutions.

v0.19.4

Bugfixes

  • Fix issue where byte datatypes were incorrectly created with the polars string datatype.

Improvements

  • Template printing now uses prefixes.

  • Exception handling for missing functions.