v6.2.0 Release Notes

v6.2.0 improves query performance and reliability, expands OpenSearch-compatible APIs, and strengthens production workflows for derived data, including materialized views, transactions, and retention.

Highlights

  • Full-text search from SQL. match, multi_match, phrase, prefix, fuzzy, regexp, and constant-score queries are available directly in SQL, so you no longer have to drop to DSL to combine relevance search with joins and aggregations.
  • A rebuilt execution engine. Aggregations, joins, sorts, and scans now run on a morsel-driven substrate that spills to disk instead of failing, respects memory budgets, and cancels promptly when you cancel a query.
  • Segment pruning everywhere. Sketches, bloom filters, and min/max statistics now skip segments across search, SQL, span queries, and match plans - large selective queries touch far less data.
  • Materialized Views v2. Incremental refresh, distributed refresh, automatic full-vs-incremental selection, target settings at create time, and per-view refresh warehouses.
  • A transaction API with UPSERT and UPDATE, including on unsharded and multi-worker runtimes.
  • Index data retention. Age out documents automatically against a timestamp field.
  • New connectors: Slack, IMAP, SMTP, Google Workspace, PostgreSQL, plus full Okta and GitHub support and Parquet ingest for S3, GCS, and Azure.
  • Auth now fails closed - see the upgrade guide before you deploy.

Upgrade Guide

Upgrade path

Refer to migration guide from v4.7.0 to v6.1.0, and migration guide to 6.2.0.

Breaking changes

1. Authorization no longer falls back to admin.

A missing X-Mach5-Authorization header previously granted full admin access. It now grants no access. Before upgrading, audit every client, script, cron job, and internal integration to ensure the header is set explicitly. This only affects the non-Keycloak, non-authentication path: standard authenticated deployments will continue to work, but automation or internal tools that connect directly to internal pods must be updated.

2. The admin role name is configurable.

It is no longer hardcoded to admin. Set it once and it is wired into every backend service as MACH5_ADMIN_ROLE_NAME:

adminRole:
    createSecretResources: true
    name: mach5-admin-role-secret
    roleName: "admin"
    secretKey: role-name

To supply your own secret, set createSecretResources: false and point name at a secret containing secretKey. If you use mach5-sdk-ts, update it - older versions relied on the removed fallback.

3. Two warehouse tuning knobs are removed and are now rejected if present:

  • mdx_admission_mpl - admission is driven by memory bytes rather than a fixed pipeline limit.
  • local_parallelism - superseded by the new execution substrate.

Previous

{
  "resource": { "num_mediators": 1 },
  "mdx_admission_mpl": 8,
  "local_parallelism": 16,
  "memory_policy": "managed",
  "enabled": true
}

6.2 snippet

{
  "resource": { "num_mediators": 1 },
  "memory_policy": "managed",
  "enabled": true
}

What You Can Use Now

Query & SQL

  • Full-text search in SQL - term, phrase, prefix, fuzzy, regexp, multi-match across fields, boolean combinations, and constant-score scoring, usable alongside joins and aggregations.
  • New functions: date_bin for time bucketing, row_to_json(), REGEXP_MATCH, span_near with in_order=false, sleep, tostring for dynamic values, and JSON extraction helpers that read straight from raw _source.
  • First-class external tables and document views, including computed mappings and decorrelation of nested constructs, so views over external and nested data plan properly instead of degrading to full scans.
  • Native Arrow timestamps. DateTime columns are stored as Arrow Timestamp rather than Int64, which fixes date display and comparison edge cases through the Postgres gateway.
  • Correctness fixes worth knowing about: EXCEPT results, IS NULL/IS NOT NULL pushdown, count-distinct over keyword and date fields, MATCH(*, ...) on cstore tables, TERMS() runtime failures, multi-value list fields in =, <=, >=, !=, bracketed ["field"] keyword syntax, and several deep-boolean-query stack overflows.

OpenSearch Compatibility

The mdx-server OpenSearch-compatible surface has expanded substantially:

  • Index admin API, document API (_doc, _create, _bulk with auto-create), _reindex, _field_caps, _count, Scroll, _tasks job tracking, index templates, index and field aliases, and highlighting.
  • Aggregations: nested aggregations, the percentile family, weighted_avg, MAD, moving_avg/moving_fn pipelines, plus composite and terms fixes for boolean and IP fields.
  • Search controls: timeout, terminate_after, track_total_hits, minimum_should_match, terms_set, and query_string support for IP and date types.
  • Per-resource authorization now applies to the OpenSearch-compatible API.
  • Correctness fixes across scaled_float round-trip, copy_to with sibling targets, stored_fields/docvalue_fields on nested and dotted fields, keyword normalizers at both query and index time, StandardTokenizer URL handling, and exact count totals.

Note on routing. By default /opensearch still serves the real OpenSearch pod, and the new DVM-backed path is exposed at /opensearch-preview. Switching the default is opt-in - see Opt-in features.

Materialized Views & Transactions

Materialized Views v2 are usable end to end:

  • Incremental evaluation with distributed refresh, and RefreshMode::Auto so the executor picks full vs. incremental instead of you guessing.
  • Target settings at create time, retention propagation to targets, and tiered compaction for MV targets.
  • A per-view refresh warehouse, selectable and editable from Dex, so refresh load doesn’t compete with interactive queries.
  • Refresh observability in Dex: query plans, per-segment progress, and terminal state tracking that no longer flaps.

The transaction API adds UPSERT and UPDATE, works on no-shard runtimes, materializes shard-aware segments for multi-worker mutations, and preserves _source on upserts. TXN_* runtime limits are configurable.

Ingestion, Connectors & Retention

  • Parquet ingest for S3, GCS, and Azure pipelines; Parquet byte arrays are accepted for keyword fields.
  • Iceberg gains predicate pruning, azdls OneLake connections, and much lower memory when analyzing incremental snapshots. Transforms are no longer silently ignored on Iceberg pipelines.
  • New connectors: Slack, IMAP, SMTP, Google Workspace, PostgreSQL warehouse connections, plus full Okta and GitHub support.
  • VRL transforms, and V8 JavaScript transforms gain a flattenDocument helper with configurable key replacements, heap caps, OOM diagnostics, and script restart intervals.
  • Bulk ingest is faster - dedup via TermInSetQuery, a pure append path, and parallel translation.
  • Pipeline run history is now durable: full run lifecycle and compaction outcomes are recorded, with accurate per-worker ingestor state and a fixed Runs tab in Dex.
  • Credentials are no longer logged by ingest pipelines.

Index data retention ages out documents against a timestamp field. Both settings are required together:

{
  "settings": {
    "index.mach5.retention.max_age": "30d",
    "index.mach5.retention.field": "event_time"
  }
}

Audit and system tables carry their own retention settings automatically.

Performance & Stability

Most of this is invisible until you hit a hard workload, at which point it matters a lot:

  • Segment pruning via sketches, bloom filters, and min/max stats now applies across cstore query paths, match plans, range execution, span queries, and SQL - and fails open rather than returning wrong results when statistics are missing.
  • A storage-level execution VM runs aggregations directly against columns, with fast paths for MIN/MAX, GROUP BY and exact count-distinct pushdown, dictionary-specialized operations, and residual LIKE filtering.
  • Better join planning through demand-aware costing and accurate cardinality estimates from a new indexstats service.
  • Top-k pushdown is generalized, including into deep UNION ALL plans and around limits.
  • Memory budgets are typed and aligned to cgroup limits, so a warehouse admits work based on real available bytes. Aggregations and sorts spill to disk instead of dying, and memory census diagnostics tell you where it went.
  • Cancellation actually cancels. Cancelling a gRPC or SQL query stops downstream aggregation and readahead work rather than letting it run to completion.
  • Reliability fixes: md-server hardening against RDS connectivity loss (with a configurable dbStatementTimeoutMs), missed index-commit and watch notifications in the index router, disk-cache eviction races, and a delta value codec skip() bug that could return corrupted values.
  • Ingestion throttling now uses a hysteresis low-watermark so compaction backpressure stops oscillating:
compactionThrottle:
    highWatermark: 20
    lowWatermark: 10
    statusIntervalSecs: 15
  • The default gRPC max message size is now 1 GiB (grpc.maxMessageSizeBytes).

Dex

  • Harmonized with the Mach5 brand design system, and the UI now uses the full width of the window.
  • Dashboards, an OpenSearch cell, logical views, a quickstart guide, and Get Started cards.
  • Notebook and editor: line-number gutter, JSON viewer with collapse, cell copy/paste, per-editor query copy, query persistence across navigation, and a default warehouse that actually persists.
  • Resource pages: name search filters, inline mapping and settings editing, index data preview wired to DSL and notebooks, copy buttons, and auto-refresh with pause/resume.
  • Jobs: sorting, filtering, duration display, persisted query plans, real DSL bodies for OpenSearch queries, and per-segment progress.

Opt-in Features

These ship in v6.2.0 but are disabled by default. Enable them deliberately.

DVM-backed OpenSearch endpoint. Controls whether /opensearch is served by the real OpenSearch pod or by mdx-server:

opensearch-legacy:
    enabled: true
    dashboards-use-legacy: true
  • enabled: true, dashboards-use-legacy: true (default) - both pods exist; /opensearch serves the real OpenSearch pod, /opensearch-preview serves mdx-server.
  • enabled: true, dashboards-use-legacy: false - /opensearch serves mdx-server (Dashboards use it); /opensearch-legacy serves the real pod.
  • enabled: false - no OpenSearch pod at all; /opensearch is served by mdx-server.

System warehouse and diagnostics. Query audit, metadata audit, and runtime diagnostics are written to indices in the _system namespace, bootstrapped by a Helm hook:

systemWarehouse:
    enabled: false
    name: default
    config:
      resource:
        num_mediators: 1
      enabled: true
      memory_policy: legacy
diagnosticsreceiver:
    enabled: false

Once enabled, you get query audit indexing, metadata audit indexing, replayable query profile bundles, gRPC and SQL execution spans, and an HTML export from the Dex diagnostics panel.

Analytics Cookies

Help us understand website usage.

Necessary storage remembers your choice. With your consent, Mach5 also uses PostHog analytics to measure website traffic and interactions.

Change this anytime from Cookie Settings in the footer. Privacy Notice.