Observability in Mach5

Observability helps users understand how a system is behaving by surfacing diagnostics, operational activity, audit history, and change history.

In Mach5, Observability is built on the internal _system namespace. Mach5 writes platform-owned operational records into _system and stores them in system indexes for diagnostics, query auditing, ingestion auditing, metadata auditing, and other internal runtime activity. Once enabled, _system becomes the main place to investigate platform issues and review what happened before or during an incident.

In practical terms, Observability in Mach5 helps answer questions such as:

• What failed, and when did it fail?
• Which user or service ran a query?
• How did this object evolve over time?
• Which change explains the current state or issue?
• Is the platform healthy, degraded, or retrying in the background?

This page explains how to enable Observability, configure it in values.yaml, apply the configuration during install or upgrade, and use the _system indexes in day-to-day investigations.

How Observability works in Mach5

Mach5 stores its own operational data in _system, a reserved internal namespace.

The most common Observability indexes are:

• _system/mach5-diagnostic-events-v1
• _system/mach5-query-audit-v1
• _system/mach5-ingestion-audit-v1
• _system/mach5-metadata-audit-v1

Together, these indexes give you a system-level view of the platform:

• diagnostics show failures, degraded behavior, retries, and warnings
• query audit shows who queried what and when
• ingestion audit shows ingestion-related activity and pipeline-level operational history
• metadata audit shows what platform objects were changed and when

Enable Observability in a Mach5 cluster

Observability is enabled through the Mach5 Helm values.yaml file.

The minimum recommended configuration is:

systemWarehouse:
  enabled: true
  name: default
  config:
    enabled: true
    resource:
      num_mediators: 1
    memory_policy: legacy

diagnosticsreceiver:
  enabled: true

What each setting means

systemWarehouse.enabled

Turns on creation and management of a warehouse in the _system namespace for Observability data.

systemWarehouse.name

Sets the warehouse name inside _system. With name: default, Mach5 creates or updates the warehouse _system/default.

systemWarehouse.config

Defines the settings for the warehouse Mach5 should create or maintain inside _system.

systemWarehouse.config.enabled

Keeps the warehouse in _system enabled for writes and queries. This should normally be true.

systemWarehouse.config.resource.num_mediators

Sets warehouse size. Start with num_mediators: 1 for most clusters.

systemWarehouse.config.memory_policy

Sets the memory behavior of the warehouse in _system. legacy is the simplest starting point.

diagnosticsreceiver.enabled

Turns on durable diagnostic-event ingestion into _system.

Apply the configuration and upgrade Mach5

Use the same values.yaml pattern for both new and existing Mach5 deployments.

1. Add the Observability configuration to values.yaml.
2. Install or upgrade Mach5 with that file.
3. Verify that the warehouse exists in the _system namespace and Observability records begin to appear in the Mach5 UI. For a simple way to do this, refer to Use Observability in Mach5.
4. During deployment, a system job pod is spawned to create the warehouse in _system if it does not exist, or update it if it already exists.

Advanced configuration for the warehouse in _system

Most users should start with a small warehouse in _system and only tune it if Observability load grows.

Advanced memory settings are available under systemWarehouse.config.memory.

Common settings include:

• memory.process_budget
• memory.native_headroom
• memory.working_budget
• memory.index_access_budget
• memory.manifest_budget
• memory.runtime_system_slack

Supported memory policies include:

• legacy
• budgeted
• budgeted_spill

Example advanced configuration:

systemWarehouse:
  enabled: true
  name: default
  config:
    enabled: true
    resource:
      num_mediators: 1
    memory_policy: budgeted_spill
    memory:
      process_budget: auto
      native_headroom:
        basis_points: 1000
      working_budget: remainder
      index_access_budget:
        bytes: 268435456
      manifest_budget:
        bytes: 67108864
      runtime_system_slack:
        basis_points: 500

What these advanced settings do

memory.process_budget

Sets the process-level memory budget for the warehouse runtime.

memory.native_headroom

Keeps part of memory aside for native and non-managed runtime overhead.

memory.working_budget

Controls how much memory is available for active warehouse work.

memory.index_access_budget

Reserves memory for index access structures and related query work.

memory.manifest_budget

Caps memory used for manifest-related operations.

memory.runtime_system_slack

Leaves extra runtime safety room inside working memory.

For most clusters, start simple and only move to explicit budgets if you have a specific performance or capacity reason to do so.

If the warehouse in _system does not exist yet, Mach5 creates it as part of the deployment flow.

If the warehouse in _system already exists and you later change the systemWarehouse.config parameters, Mach5 updates that existing warehouse so the new settings are reflected.

Use Observability in Mach5

Once enabled, _system becomes the main place to investigate platform behavior.

The easiest way to start is from the Mach5 UI:

1. Open the _system namespace and click Indexes.

2. Look for these Observability indexes:

   • mach5-diagnostic-events-v1
   • mach5-query-audit-v1
   • mach5-ingestion-audit-v1
   • mach5-metadata-audit-v1

3. Open mach5-query-audit-v1 and use the Preview section to inspect data quickly.

4. If you want to investigate further, click Open in Notebook from the same index page.

For diagnostic, ingestion, and metadata investigations, use the same flow starting from mach5-diagnostic-events-v1, mach5-ingestion-audit-v1, and mach5-metadata-audit-v1

_system/mach5-diagnostic-events-v1

Use this index when you are debugging runtime problems.

It helps answer:

• what failed
• which service or component reported it
• when it happened
• whether the same issue is repeating

Simple example:

select * from mach5-diagnostic-events-v1 limit 50;

Use this when a workflow publish, controller action, or background job is failing and you want to inspect the latest operational events first.

_system/mach5-query-audit-v1

Use this index when you want to understand query activity.

It helps answer:

• who ran a query
• when it ran
• which namespace or index it touched
• whether there was unusual activity during the incident window

Simple example:

select * from mach5-query-audit-v1 limit 50;

Practical example:

If a user runs a query against an application index and you later need to review that activity, look in this index for the corresponding audit record. This is the index that helps you understand query behavior over time.

_system/mach5-ingestion-audit-v1

Use this index when you want to understand ingestion activity and ingestion-related operational history.

It helps answer:

• what ingestion activity ran
• when ingestion activity ran
• which ingestion workflow, source, or pipeline was involved
• whether ingestion behavior around an incident window explains missing, delayed, or unexpected data

Simple example:

select * from mach5-ingestion-audit-v1 limit 50;

Practical example:

If data did not arrive when expected, or if you want to understand what ingestion activity occurred before a downstream issue, start with this index to review the recent ingestion audit trail.

_system/mach5-metadata-audit-v1

Use this index when you are investigating changes to platform resources.

It helps answer:

• who changed a warehouse, notebook, index, connection, or other metadata object
• when that change happened
• what changed before a platform issue or regression

Simple example:

select * from mach5-metadata-audit-v1 limit 50;

Practical example:

If a warehouse setting, notebook, or other metadata object was modified before a reported issue, this index helps you trace that change and correlate it with later diagnostics.