<style type='text/css'> .reveal pre code { max-height: 100vh; } .alert { font-size: 30px; } </style> # Inequality engine and follow ups --- ## Rationale: ### expand existing Rucio semantics to allow improved (and expandable) metadata filtering ###### see [did_column_meta.py](https://github.com/rucio/rucio/blob/master/lib/rucio/core/did_meta_plugins/did_column_meta.py) --- ## Currently: > Filter arguments in form key=value,another_key>=next_value. Valid keys are type and length. Use type to specify a DID type to search for. Valid values are ALL, COLLECTION, CONTAINER, DATASET, FILE. Use length to filter for an amount of corresponding Data IDentifiers. Valid operators are =, >, >=, <, <=. ###### from [the docs](https://rucio.readthedocs.io/en/latest/man/rucio.html#list-dids) --- ## Namely: ``` $ rucio list-dids --filter 'type=all' user.jdoe:* +-------------------------------------------+--------------+ | SCOPE:NAME | [DID TYPE] | |-------------------------------------------+--------------| | user.jdoe:user.jdoe.test.container.1234.1 | CONTAINER | | user.jdoe:user.jdoe.test.container.1234.2 | CONTAINER | | user.jdoe:user.jdoe.test.cont.1234.2 | CONTAINER | | user.jdoe:user.jdoe.test.dataset.1 | DATASET | | user.jdoe:user.jdoe.test.dataset.2 | DATASET | | user.jdoe:user.jdoe.test.data.1234.1 | DATASET | | user.jdoe:test.file.1 | FILE | | user.jdoe:test.file.2 | FILE | | user.jdoe:test.file.3 | FILE | +-------------------------------------------+--------------+ ``` --- ## Inequality engine Work started in 2020 to avoid filters in the form: ``` (type=container AND length=10) OR (type=dataset AND length=256) ``` with just one semantics choice: <center> #### AND -> , #### OR -> ; </center> therefore: ``` AND_GROUP_1 -> type=container AND length=10 AND_GROUP_2 -> type=dataset AND length=256 ``` --- ### This implies support for additional columns and metadata! --- ## [First PR](https://github.com/rucio/rucio/pull/4139) opened in Nov 2020 * Originally thought as an additional CLI argument * Later used to replace the existing `filters` argument → back compatibility implemented * Got stuck in the middle of REST API engine update → some tests failing because of conflicts --- ### The code in this PR was more of a kickstarter! difficult to maintain, developed seeking for functionality over implementation and structure. --- ## [Another PR](https://github.com/rucio/rucio/pull/4746) opened by Rob Barnsley from SKAO * Based on the Inequality Engine and my preiovus findings and algorithms * Much better structured and splitted in more testable parts * still stuck on failing unittests, but we all know how that goes... --- ### We'll try to work together on this on before I leave ###### (as late as november this year)