Query DSL

This document describes the query Domain Specific Language (DSL), its JSON representation, how to evaluate queries against in-memory collections, how to compile them to Kotlin Exposed SQL operators, and how to extend the system with new visitors.

The Query DSL (from Query.kt)

The query DSL is defined by a set of Kotlin data classes and sealed interfaces, primarily in Query.kt. It allows for expressing complex data retrieval and filtering logic.

Core Components:

• Query: The top-level object representing a complete query.
  • filter: Filter: Specifies the filtering conditions. Defaults to Filter.TRUE (no filtering).
  • sort: Sort: Defines the sorting order for the results. Defaults to Sort.NO_SORT.
  • paginate: Pagination: Specifies which subset of the results to retrieve. Defaults to Pagination.FIRST_PAGE.
  • It also provides helper properties like thisPage, nextPage, and previousPage which are GZIPed and Base64 URL encoded string representations of the Query object, suitable for use as page cursors.

• Locator: A type alias for String (e.g., "fieldName", "nested.object.field"). It’s used to identify the target field/property for filtering and sorting.

• Filter: A sealed interface representing a filtering condition.
  • Filter.Literal:
    • Filter.TRUE: A filter that always evaluates to true.
    • Filter.FALSE: A filter that always evaluates to false.
  • Filter.Composite:
    • Filter.And(clauses: List<Filter>): Logical AND of multiple filter clauses.
    • Filter.Or(clauses: List<Filter>): Logical OR of multiple filter clauses.
  • Filter.Transform:
    • Filter.Not(clause: Filter): Logical NOT of a single filter clause.

  • Filter.Term: Abstract base for filters that operate on a specific field identified by a locator.

    • Filter.UnaryTerm:
      • Filter.IsNull(locator: Locator): Checks if the field’s value is null.
      • Filter.IsNotNull(locator: Locator): Checks if the field’s value is not null.
    • Filter.CompareTerm<FIELD>: Compares a field’s value with a given value.
      • Filter.Eq(locator: Locator, value: FIELD): Equal to.
      • Filter.Ne(locator: Locator, value: FIELD): Not equal to.
      • Filter.Gt(locator: Locator, value: FIELD): Greater than.
      • Filter.Ge(locator: Locator, value: FIELD): Greater than or equal to.
      • Filter.Lt(locator: Locator, value: FIELD): Less than.
      • Filter.Le(locator: Locator, value: FIELD): Less than or equal to.
      • Filter.RegEx(locator: Locator, value: String): Matches a regular expression (value must be a string pattern).
    • Filter.SetTerm<FIELD>: Checks if a field’s value is within a set of values.
      • Filter.In(locator: Locator, values: List<FIELD>): Value is in the list.
      • Filter.NotIn(locator: Locator, values: List<FIELD>): Value is not in the list.

• Sort: Defines the sorting criteria.
  • entries: List<SortEntry>: A list of fields to sort by.
  • Sort.NO_SORT: A companion object representing no specific sorting.

• SortEntry: A single sort criterion.
  • key: Locator: The field to sort by.
  • direction: SortDirection: ASC (ascending) or DESC (descending).

• Pagination: Controls result pagination.
  • index: Int: The page index (0-based).
  • size: Int: The number of items per page.
  • Pagination.FIRST_PAGE: A companion object for the default first page.
  • Helper properties: next, previous (for navigating pages), start (calculates the starting offset).

Mapping to and from JSON

The query components, especially Filter and Query, can be serialized to and deserialized from JSON. This is facilitated by custom serializers and parsers (FilterSerializer, FilterJsonVisitor, JsonFilterParser). The intended JSON structure is also described by the JSON Schema in newq/newq.yml.

Filter JSON Serialization/Deserialization:

• Serialization (FilterJsonVisitor):
  • Filter.TRUE: true
  • Filter.FALSE: false
  • Filter.And: { "and": [<clause1_json>, <clause2_json>, ...] }
  • Filter.Or: { "or": [<clause1_json>, <clause2_json>, ...] }
  • Filter.Not: { "not": <clause_json> }
  • Filter.IsNull: { "isNull": "fieldName" }
  • Filter.IsNotNull: { "isNotNull": "fieldName" }
  • Filter.Eq: { "locator": "fieldName", "eq": <value_json_primitive> }
  • Filter.Ne: { "locator": "fieldName", "ne": <value_json_primitive> }
  • Filter.Gt: { "locator": "fieldName", "gt": <value_json_primitive> } (value typically number or string)
  • Filter.Ge: { "locator": "fieldName", "ge": <value_json_primitive> }
  • Filter.Lt: { "locator": "fieldName", "lt": <value_json_primitive> }
  • Filter.Le: { "locator": "fieldName", "le": <value_json_primitive> }
  • Filter.RegEx: { "locator": "fieldName", "regex": "patternString" }
  • Filter.In: { "in": { "locator": "fieldName", "values": [<value1_json_primitive>, ...] } }
  • Filter.NotIn: { "notIn": { "locator": "fieldName", "values": [<value1_json_primitive>, ...] } }
  • Primitive values (<value_json_primitive>) are represented as JSON strings, numbers, or booleans. `
  • Instantvalues are serialized to their ISO 8601 string representation (e.g.,“2023-10-27T10:30:00Z”`).

• Deserialization (JsonFilterParser):
  • JsonFilterParser.parseFilterJson(jsonElement) attempts to convert a JsonElement back into a Filter object.
  • smartParseJsonPrimitive is used to infer the type of primitive values from JSON (e.g., "true" -> true (Boolean), "123" -> 123 (Long), "123.45" -> 123.45 (Double), "2023-01-01T00:00:00Z" -> Instant object, "text" -> "text" (String)).

See examples of JSON respresentations

Query JSON Serialization/Deserialization:

• The Query data class is @Serializable. Its JSON representation uses a specialized serializer.

Compiling Queries to Kotlin Exposed Operators (QueryCompiler.kt)

The QueryCompiler translates the Filter and Sort components of the DSL into Kotlin Exposed SQL operators, enabling their
use in database queries.

• QueryCompiler(private val tbl: EntityTable): The constructor takes an EntityTable. This EntityTable is an abstraction
  that is responsible for mapping a locator: String to an Exposed SQL Column<*> object (e.g., tbl.columnFor(“name”)).
  • This mapping is crucial for linking DSL field names to database columns.
  • It also includes bitemporal aspects, like tbl.effectiveAsOf and tbl.recordedAsOf columns.
• filter(f: Filter): Op<Boolean>: This is the primary method for converting a Filter object into an Exposed SQL
  Op<Boolean> condition.
  • It uses FilterExposedVisitor, passing the EntityTable as context data. The result is wrapped in Result,
    and getOrThrow can be used to throw an exception on failure.
• where(f: Filter, asOf: TimeCoordinates): Op: Combines the result of filter(f) with a bitemporal visibility
  condition (visibleFromCondition(asOf)), which checks effectiveAsOf and recordedAsOf against the provided TimeCoordinates.
• ordering(sort: Sort): List<Pair<Expression<*>, SortOrder>>: Converts a Sort object into a list of
  Pair<Expression<*>, SortOrder>, suitable for Exposed’s orderBy() function.
  • It also prepends a default bitemporal sorting (most recent first on recordedAsOf then effectiveAsOf).
• DecoratePaging(raw: SizedIterable<T>, p: Pagination = Pagination()): SizedIterable<T>: Applies limit and offset to
  an Exposed SizedIterable based on the Pagination object.
• FilterExposedVisitor:•Implements FilterVisitor<Op<Boolean>, EntityTable>. Each visit method for a Filter subtype
  constructs the corresponding Exposed Op<Boolean>:
  • It relies on the EntityTable (passed as data to the visitor) to resolve locator strings into actual Column objects
    and handles type conversions for comparisons.

Creating New Visitors

The system is designed to be extensible through the FilterVisitor interface. You can create new visitors to translate
the Filter DSL into different representations or to perform other operations.

• FilterVisitor<R, TDATA: Any> Interface:
  • R: The result type of the visitor’s operation (e.g., Op, String,
    a custom query object).
  • TDATA: A type for any contextual data the visitor might need during its operation (e.g., a database schema,
    a configuration object). If no context is needed, Any or Unit can be used.
  • visit(filter: Filter, data: TDATA): Result<R>: The main method to implement. It receives the Filter node to visit
    and the contextual data.
  • Filter.accept(visitor: FilterVisitor<R, TDATA>, data: TDATA): Result<R>:•Each Filter subtype implements this
    method, which simply calls visitor.visit(this, data). This is the entry point for the visitor pattern.

Steps to Create a New Visitor:

1.Define Your Target: Determine what you want to convert the Filter into (this will be your R type)
and what contextual information, if any, your visitor will need (TDATA).
2.Implement FilterVisitor: Create a new class or object that implements FilterVisitor<R, TDATA>
3.Handle All Filter Subtypes: Ensure your when statement in the visit method covers all concrete Filter types. For
composite filters like Filter.And and Filter.Or, you’ll typically recursively call accept on their child clauses.
4.Use Contextual Data: If your TDATA is not just a placeholder, use it within your visit methods to aid in the
translation (e.g., looking up field mappings, accessing configuration).
5.Return Result<R>: Wrap your output in Result.success() or Result.failure() to handle potential errors during
translation gracefully.

The cards.arda.common.lib.lang.collectAll() extension function (among others) is useful for processing lists of Result.

Comments