Mach5 Search v6.2.0 - Migration Guide

This guide provides step-by-step instructions for upgrading an existing Mach5 deployment to v6.2.0. It focuses on v6.2.0-specific migration changes, including Helm chart ownership changes, PostgreSQL protection, new services, values.yaml changes and routing changes.

If you are upgrading from v4.7.0, v5.x, or any version earlier than v6.1.0, complete the v4.7.0-to-v6.1.0 migration steps before applying the v6.2.0-specific steps in this guide. You do not need to stop on or run v6.1.0 as an intermediate deployment unless your upgrade plan requires it, but the earlier migration requirements still apply. See migration guide from v4.7.0 to v6.1.0.

If Keycloak-based authentication and authorization were enabled before v5.9.0, also review the v5.9.0 authorization migration notes before upgrading to v6.2.0.

Prerequisites

  • Ensure all ingest pipelines are paused or that your upgrade window tolerates ingestion interruption.
  • Confirm the current deployment is healthy before upgrading.
  • If upgrading from v4.7.0, v5.x, or any version earlier than v6.1.0, confirm that the v4.7.0-to-v6.1.0 migration steps have been reviewed and applied as part of the upgrade plan.
  • Export current Helm values and manifests before making changes.

PostgreSQL Migration and PVC Protection

This section applies only if your deployment still uses the legacy postgresdb Deployment/PVC that was rendered by the mach5-search Helm chart. It does not apply if Mach5 metadata is already stored in Amazon RDS or another external PostgreSQL service.

v6.2.0 removes the embedded postgresdb template from the mach5-search chart. PostgreSQL must now be managed outside mach5-search, either by an external database such as RDS or by a standalone PostgreSQL Helm chart.

If your deployment still has chart-owned postgresdb resources, the upgrade must protect the existing metadata PVC before the embedded PostgreSQL deployment is removed.

v6.2.0 includes:

  • A pre-upgrade hook named <release>-protect-postgres-pvc.
  • A guard template named guard-legacy-postgres.yaml.

The guard blocks the upgrade when it detects a legacy postgresdb Deployment unless force-upgrade is explicitly set to yes-force-upgrade.

Do not set force-upgrade: yes-force-upgrade until the PostgreSQL backup and PVC protection steps are complete.

Use the following steps for migration:

1. Back up PostgreSQL

POSTGRES_POD=$(kubectl get pod -n "$NAMESPACE" -l app=postgresdb -o jsonpath='{.items[0].metadata.name}')

kubectl exec -n "$NAMESPACE" "$POSTGRES_POD" -- \
  env PGPASSWORD=<postgres-password> pg_dumpall -U postgres > mach5-postgres-before-6.2.sql

Store this dump outside the cluster.

2. Protect the existing PVC

The v6.2.0 chart hook attempts to annotate the legacy PVC automatically, but production upgrades should protect it manually first:

kubectl annotate pvc "${RELEASE}-postgresdb-pvc-ssd" \
  -n "$NAMESPACE" \
  helm.sh/resource-policy=keep \
  --overwrite

kubectl get pvc "${RELEASE}-postgresdb-pvc-ssd" -n "$NAMESPACE" \
  -o jsonpath='{.metadata.annotations.helm\.sh/resource-policy}'

Expected output:

keep

3. Upgrade mach5-search and remove embedded PostgreSQL ownership

Run the v6.2.0 mach5-search upgrade with force-upgrade only after backup and PVC protection are complete:

helm upgrade --install "$RELEASE" \
  oci://us-central1-docker.pkg.dev/mach5-dev/mach5-docker-registry/mach5-search \
  --version <target-6.2.0-version> \
  --namespace "$NAMESPACE" \
  -f values-6.2.0.yaml \
  --set force-upgrade=yes-force-upgrade

After this step, the old postgresdb Deployment from mach5-search should be removed, but the PVC should remain.

Verify:

kubectl get deploy postgresdb -n "$NAMESPACE" || true
kubectl get pvc "${RELEASE}-postgresdb-pvc-ssd" -n "$NAMESPACE"

4. Install standalone PostgreSQL using the preserved PVC

Install the standalone PostgreSQL chart and mount the preserved PVC using pvc.existingClaim:

helm upgrade --install postgresql-<env> ci/helm-charts/postgresql \
  --namespace "$NAMESPACE" \
  --set image=postgres:16 \
  --set password=<postgres-password> \
  --set pvc.existingClaim="${RELEASE}-postgresdb-pvc-ssd" \
  --wait

5. Point Mach5 metadata settings to the standalone PostgreSQL service

Update the v6.2.0 values file so metadatadb.* points to the standalone PostgreSQL service.

Example:

metadatadb:
  name: postgres
  host: postgresdb
  port: "5432"
  sslmode: disable
  user: postgres
  credentials:
    secretName: <metadata-db-secret>
    secretKey: PGPASSWORD

If the standalone PostgreSQL service name or namespace differs, update metadatadb.host accordingly.

6. Validate metadata after upgrade

For in-cluster standalone PostgreSQL:

POSTGRES_POD=$(kubectl get pod -n "$NAMESPACE" -l app=postgresdb -o jsonpath='{.items[0].metadata.name}')

kubectl exec -n "$NAMESPACE" "$POSTGRES_POD" -- \
  env PGPASSWORD=<postgres-password> psql -U postgres -d postgres -c '\dt'

Run a few deterministic checks, for example:

kubectl exec -n "$NAMESPACE" "$POSTGRES_POD" -- \
  env PGPASSWORD=<postgres-password> psql -U postgres -d postgres -c \
  "SELECT COUNT(*) FROM namespace;"

Expected result: existing Mach5 metadata is present after the upgrade.

Important notes

  • Do not run two PostgreSQL pods against the same ReadWriteOnce PVC at the same time.
  • Do not install the standalone PostgreSQL chart with pvc.existingClaim while the old embedded postgresdb pod is still running.
  • Do not use force-upgrade: yes-force-upgrade until the PostgreSQL backup is complete and the legacy PVC is protected.
  • Keep mach5-postgres-before-6.2.sql until the upgrade is fully accepted.

Internal Authorization and Admin Role

This section may be useful for your deployment if:

  • You bypass nginx and connect directly to services such as msearchserver, mdserver, weaveserver, mcpserver, warehouse OpenSearch or any other services.
  • You call Mach5 backend services directly from scripts, tests, probes, port-forwards, or operational runbooks.

If users and clients access Mach5 only through the normal nginx/auth path, no change is required.

v6.2.0 makes internal service authorization stricter. Direct calls to backend services must include the Mach5 authorization context expected by the service. Missing authorization headers no longer imply admin access; missing headers now mean no access.

v6.2.0 also adds a chart-managed admin role secret:

adminRole:
  createSecretResources: true
  name: mach5-admin-role-secret
  roleName: "admin"
  secretKey: role-name

  1. Check whether you call backend services directly.

    Search operational scripts, test code, runbooks, jobs, and probes for direct service URLs such as:

    • msearchserver
    • mdserver
    • weaveserver
    • mcpserver
    • warehouse OpenSearch service URLs

  2. Add the Mach5 authorization header to direct service calls.

    For example, if you wanted to query an index by calling the warehouse OpenSearch service directly:

INDEX=<index-name>
OPENSEARCH_SERVICE_URL=http://<warehouse-opensearch-service-host>:<opensearch-port>

curl -sS -X POST \
  "$OPENSEARCH_SERVICE_URL/$INDEX/_search" \
  -H 'Content-Type: application/json' \
  -H 'X-Mach5-Authorization: ["admin"]' \
  -d '{
    "size": 10,
    "query": {
      "match_all": {}
    }
  }'

The same requirement applies when using the Mach5 SDK against a direct MDX KQL endpoint. For example, when connecting directly to a port-forwarded MDX KQL endpoint such as http://localhost:50561, include the x-mach5-authorization metadata header:

const { createClientFactory, Metadata } = require('nice-grpc-web');
const { deadlineMiddleware } = require('nice-grpc-client-middleware-deadline');
const ms = require('ms');
const { Client } = require('@mach5-io/kql-client');

async function main() {
  const clientFactory = createClientFactory().use(deadlineMiddleware);

  const client = new Client({
    endpoint: 'http://localhost:50561',
    clientFactory,
  });

  const result = await client.query(
    'upsert_large_raw | limit 10',
    {
      deadline: ms('120s'),
      metadata: Metadata().set('x-mach5-authorization', '["admin"]'),
    }
  );

  console.log('query:', result.query);

  for await (const page of result) {
    for (const row of page) {
      console.log(row);
    }
  }
}

main().catch((err) => {
  console.error(err);
  process.exit(1);
});

No SDK change is needed when using the nginx KQL endpoint through the normal nginx/auth path.

KQL and SQL Multi-Field Projection

v6.2.0 changes how OpenSearch multi-fields are exposed through KQL and SQL projection. Multi-fields are no longer included when projecting all fields. This includes nested multi-fields.

For example, assume an index has a mapping like this:

{
  "properties": {
    "manufacturer": {
      "type": "text",
      "fields": {
        "keyword": { "type": "keyword" }
      }
    },
    "product": {
      "properties": {
        "category": {
          "type": "text",
          "fields": {
            "keyword": { "type": "keyword" }
          }
        }
      }
    }
  }
}

The following queries project all regular fields, but do not include the manufacturer.keyword or product.category.keyword multi-field columns:

SELECT * FROM products LIMIT 10;

products | limit 10

To return multi-fields, project them explicitly:

SELECT manufacturer, manufacturer.keyword, product.category, product.category.keyword
FROM products
LIMIT 10;

products
| project manufacturer, manufacturer.keyword, product.category, product.category.keyword
| limit 10

Update dashboards, saved queries, tests, or scripts that relied on SELECT * or KQL project-all output to include nested multi-fields.

System Warehouse Bootstrap

In Mach5, Observability is built on the internal _system namespace. Mach5 writes platform-owned operational records into _system and stores them in system indexes for diagnostics, query auditing, ingestion auditing, metadata auditing, and other internal runtime activity.

This section is applicable only if you want to use this feature.

If your deployment does not use a system warehouse, keep systemWarehouse.enabled=false. In that case, the bootstrap hook will not create or update the _system namespace or warehouse.

  1. Decide whether Helm should manage the system warehouse.

    Leave it disabled if you create the system warehouse manually or do not use one:

systemWarehouse:
  enabled: false

Enable it if you want Helm upgrades to create or reconcile the system warehouse:

systemWarehouse:
  enabled: true
  name: default
  config: 
    resource:
      num_mediators: 1
    enabled: true
    memory_policy: legacy
diagnosticsreceiver:
    enabled: true

  1. Provide a complete warehouse configuration when enabling the hook.

    systemWarehouse.config should contain the warehouse settings required by your environment, such as size, storage, placement, engine settings, or other fields normally sent to the Warehouse API.

Upgrade Procedure

Authenticate to the OCI registry and pull the chart:

cat reader-key.json | helm registry login https://us-central1-docker.pkg.dev -u _json_key --password-stdin

helm pull oci://us-central1-docker.pkg.dev/mach5-dev/mach5-docker-registry/mach5-search \
  --version 6.2.0

Extract the chart if you need to inspect CRDs, hooks, or rendered templates:

tar -xzvf mach5-search-6.2.0.tgz

Render first with your values:

helm template <mach5-release-name> mach5-search-6.2.0.tgz \
  -n <mach5-namespace> \
  -f values.yaml > rendered-mach5-6.2.0.yaml

Upgrade:

helm upgrade --install <mach5-release-name> mach5-search-6.2.0.tgz \
  -n <mach5-namespace> \
  -f values.yaml

If upgrading an environment with legacy chart-owned postgresdb, add force-upgrade only after the PostgreSQL backup/PVC protection steps:

helm upgrade --install <mach5-release-name> mach5-search-6.2.0.tgz \
  -n <mach5-namespace> \
  -f values.yaml \
  --set force-upgrade=yes-force-upgrade

Troubleshooting

OpenSearch Dashboards shows “Mach5 server not ready yet”

After upgrading to v6.2.0, especially from v4.7.0, OpenSearch Dashboards may fail to load with the following error:

Mach5 server not ready yet

If this happens for a warehouse after the upgrade, the OpenSearch Dashboards saved-object indexes for that warehouse may need to be recreated.

To recover:

  1. Delete the affected warehouse’s .kibana_1 and .kibana_2 indexes.
  2. Disable the warehouse that shows the error.
  3. Enable the warehouse again.
  4. Open OpenSearch Dashboards and verify that it loads successfully.

Only delete .kibana_1 and .kibana_2 for the affected warehouse. These indexes contain OpenSearch Dashboards saved objects, so confirm that recreating them is acceptable for the environment before deleting them.

Analytics Cookies

Help us understand website usage.

Necessary storage remembers your choice. With your consent, Mach5 also uses PostHog analytics to measure website traffic and interactions.

Change this anytime from Cookie Settings in the footer. Privacy Notice.