Mach5 Search Helmcharts

Installation instructions for all helmcharts required for setting up Mach5 Search.

Cluster Autoscaler For AWS EKS environment

Not needed for an Azure Kubernetes Service cluster deployment, and its native autoscaler can be used.

Run the following commands to install the cluster-autoscaler helm chart:

helm repo add autoscaler https://kubernetes.github.io/autoscaler

helm install m5-cas autoscaler/cluster-autoscaler --version 9.34.1 --set 'autoDiscovery.clusterName'=<CLUSTER NAME> --set 'awsRegion=<CLUSTER_REGION>' \
   -f values_ca.yaml

Contents of values_ca.yaml:

extraArgs:
    logtostderr: true
    stderrthreshold: info
    v: 4
    ignore-daemonsets-utilization: true
    skip-nodes-with-system-pods: false
    daemonset-eviction-for-empty-nodes: true
    daemonset-eviction-for-occupied-nodes: true
    skip-nodes-with-local-storage: false
    scale-down-delay-after-add: 2m
    scale-down-unneeded-time: 2m
    unremovable-node-recheck-timeout: 2m
    expander: priority
    max-node-provision-time: 2m

Mach5 Cache Proxy

Run the following commands to install the mach5 cache proxy helm chart:
(Note: reader-key.json file has the json key for the service account for accessing Mach5 Search Artifact Registry. Contact Mach5 Search Administrator to get a copy of your reader-key.json file)

kubectl create namespace cert-manager
helm repo add jetstack https://charts.jetstack.io --force-update
helm repo update
helm install cm jetstack/cert-manager --version v1.5.3 -n cert-manager -f values_cm.yaml

kubectl create namespace cache-proxy

cat reader-key.json | helm registry login https://us-central1-docker.pkg.dev -u _json_key
 --password-stdin
 helm pull oci://us-central1-docker.pkg.dev/mach5-dev/mach5-docker-registry/mach5-cache-proxy 
 --version 1.13.3
 helm install m5-cache mach5-cache-proxy-1.13.3.tgz -n cache-proxy -f values_cp.yaml

Contents of values_cm.yaml

prometheus:
  enabled: false
installCRDs: true

Contents of values_cp.yaml

mach5ImagePullSecret:
  dockerconfigjson: "CHANGE_ME" # Contact Mach5 Search Administrator for this value (image-pull-secret file)

kube-image-keeper:
  registry:
    env:
      - name: blobdescriptorsize
        value: 1000
    persistence:
      accessModes: ReadWriteOnce
      enabled: true
      storageClass: gp2
      size: 75Gi
    nodeSelector:
      mach5-main-role: "true"
  controllers:
    nodeSelector:
      mach5-main-role: "true"
    webhook:
      ignoredImages:
        - "foundationdb.*"
  proxy:
    podAnnotations:
      "cluster-autoscaler.kubernetes.io/enable-ds-eviction": "true"
  fullnameOverride: "cache-proxy"

Configuring PostgreSQL

Mach5 can use either a managed PostgreSQL service or a PostgreSQL deployment running inside your Kubernetes cluster.

If you already have a managed PostgreSQL instance, including services such as Amazon RDS for PostgreSQL, Azure Database for PostgreSQL flexible server, Cloud SQL for PostgreSQL, or any other managed PostgreSQL offering, you can point Mach5 directly to that database using the metadatadb configuration shown below.

If you do not already have a PostgreSQL instance available, you can set one up locally in the cluster by using the Bitnami PostgreSQL Helm chart in this section.

Use the Bitnami PostgreSQL Helm chart. It directly supports TLS using the tls.enabled, and tls.autoGenerated values.

Create postgres-values.yaml

architecture: standalone

auth:
  username: appuser
  password: change-me
  database: appdb
  postgresPassword: change-postgres-password

tls:
  enabled: true
  autoGenerated: true

primary:
  persistence:
    enabled: true
    size: 20Gi

  # Require TLS for TCP connections.
  pgHbaConfiguration: |-
    local   all             all                                     trust

    hostssl all             all             0.0.0.0/0               scram-sha-256
    hostssl all             all             ::/0                    scram-sha-256

    hostnossl all           all             0.0.0.0/0               reject
    hostnossl all           all             ::/0                    reject

Install PostgreSQL

helm repo add bitnami https://charts.bitnami.com/bitnami
helm repo update

helm upgrade --install postgres bitnami/postgresql \
  --namespace postgres \
  --create-namespace \
  -f postgres-values.yaml

The chart supports custom pg_hba.conf content through primary.pgHbaConfiguration.

Validate the deployment

kubectl get pods -n postgres
kubectl get svc -n postgres

Get the application password:

export POSTGRES_PASSWORD=$(
  kubectl get secret postgres-postgresql \
    -n postgres \
    -o jsonpath='{.data.password}' |
  base64 -d
)

Port-forward PostgreSQL:

kubectl port-forward -n postgres svc/postgres-postgresql 5432:5432

Connect using SSL:

PGPASSWORD="$POSTGRES_PASSWORD" psql \
  "host=localhost port=5432 dbname=appdb user=appuser sslmode=require"

Verify TLS:

SELECT
    ssl,
    version,
    cipher,
    bits
FROM pg_stat_ssl
WHERE pid = pg_backend_pid();

You should see:

ssl | version |         cipher          | bits
-----+---------+-------------------------+------
t   | TLSv1.3 | TLS_AES_256_GCM_SHA384 |  256

Confirm that a non-SSL connection is rejected using:

PGPASSWORD="$POSTGRES_PASSWORD" psql \
  "host=localhost port=5432 dbname=appdb user=appuser sslmode=disable"

Configure Mach5 to use this PostgreSQL instance

You can configure this PostgreSQL instance to be used by Mach5 with the following configuration:

metadatadb:
  name: "appdb"
  host: "postgres-postgresql.postgres.svc.cluster.local"
  port: "5432"
  sslmode: "require"
  user: "appuser"
  password: "postgres_password_here"

Field reference:

• metadatadb.name: PostgreSQL database name that Mach5 should use. For this example, the value is appdb.
• metadatadb.host: PostgreSQL server hostname or fully qualified service name. For an in-cluster Bitnami deployment, use postgres-postgresql.postgres.svc.cluster.local. For a managed PostgreSQL service, use the DNS name provided by your database service.
• metadatadb.port: PostgreSQL listener port. The standard value is 5432.
• metadatadb.sslmode: PostgreSQL SSL mode used by Mach5 when connecting to the database. Common values are disable or require. Use require when TLS is enabled.
• metadatadb.user: PostgreSQL username that Mach5 should use. In this example, the value is appuser.
• metadatadb.password: Password value for the PostgreSQL user.

Recommended usage notes:

• When using a managed PostgreSQL service, update host, user, name, and the password.
• When using TLS-enforced PostgreSQL, keep sslmode: "require".

Mach5 Search

• A license token is required to install Mach5. Please follow the instructions in this document: License Token Setup Guide and provide the requested details to the Mach5 Administrator to obtain your license.

• Once you have the license token, create a mach5 namespace

kubectl create namespace mach5

• Run the following commands to install the mach5-search helm chart:

(Note: reader-key.json file has the json key for the service account for accessing Mach5 Search Artifact Registry. Contact Mach5 Search Administrator to get a copy of your reader-key.json file and for the version of Mach5 to be used - $version)

  cat reader-key.json | helm registry login https://us-central1-docker.pkg.dev -u _json_key --password-stdin


  helm pull oci://us-central1-docker.pkg.dev/mach5-dev/mach5-docker-registry/mach5-search --version $version


  helm upgrade --install m5s $version -nmach5 -f values.yaml

The contents of the values.yaml file, tailored to the deployment environment, are provided in the section below.
Once the installation is complete, initialize Mach5 Search by following the instructions based on cloud providers:
AKS: Initialize Mach5 Search in AKS
EKS: Initialize Mach5 Search in EKS
GKE: Initialize Mach5 Search in GKE

AKS (Azure Kubernetes Service Cluster)

Contents of values.yaml: (Edit fields marked CHANGE_ME)

mach5ImagePullSecret:
  dockerconfigjson: "CHANGE_ME" # Contact Mach5 Search Administrator for this value (image-pull-secret file)

license:
  metered: false
  token: "CHANGE_ME" # Contact Mach5 Search Administrator for this value 

metadatadb: #Azure Database for PostgreSQL flexible server details
  name: "postgres" #CHANGE_ME
  host: "postgreshost" #CHANGE_ME
  port: "5432" #CHANGE_ME
  sslmode: "disable" 
  user: "postgres" #CHANGE_ME
  password: "Password123" #CHANGE_ME
  externalPostgresdb: true
  pvc:
    storageclass: "managed"

pvc:
  storageclass: "managed"

mediator:
  replicaCount: 1
  existingClaim: false
  cstoreSegmentCache:
    enabled: true
  loglevel: info
  useGcpInstanceMetadata: false
  cstoreCachefs:
    rcachepvsize: "483183820800" #450GB
    pvsize: "485331304448" #452GB
  resourceLimit:
    enabled: false

mediatorcompactorcontroller:
  cstoreCachefs:
    rcachepvsize: "10737418240" #10GB
    pvsize: "12884901888" #12GB
  compaction_resources:
    #enabled: true
    max_compaction_memory: 
      enabled: true
      value: "536870912" # 512M

mediatoringestorcontroller:
  cstoreCachefs:
    mountroot: /cachefs
    rcachepv: /ingest
    rcachepvsize: "10737418240" #10GB
    pvsize: "12884901888" #12GB
  ingestion_resources:
    limits:
      memory:
        enabled: true
        value: "4096Mi"

mediatorwarehousecontroller:
  customnode:
    enabled: true
    workerresource:
      mediator:
        memory: "59055800320" #55GB
    headresource:
      ir:
        memory: "1073741824" #1GB
      os:
        memory: "25769803776" #24GB
      osd:
        memory: "1073741824" #1GB

nginx:
  azure:
    enabled: true
    loadBalancerInternal: "true"

argo-workflows:
  # -- Keep CRDs on chart uninstall
  crds:
    install: true
    keep: false

prometheus:
  enabled: false

EKS

Contents of values.yaml: (Edit fields marked CHANGE_ME)

mach5ImagePullSecret:
  createSecretResources: true
  dockerconfigjson: "CHANGE_ME" # Contact Mach5 Search Administrator for this value
  
metadatadb:
  name: "postgres"
  host: "postgresdb"
  port: "5432"
  sslmode: "disable"
  user: "postgres"
  password: "" #CHANGE_ME
  externalPostgresdb: false
  pvc:
    storageclass: "gp2"

mediator:
  replicaCount: 1
  existingClaim: false
  cstoreSegmentCache:
    enabled: true
  loglevel: info
  useGcpInstanceMetadata: false
  cstoreCachefs:
    rcachepvsize: "483183820800" #450GB
    pvsize: "485331304448" #452GB
  resourceLimit:
    enabled: false

pvc:
  storageclass: "gp2"

teleportcollector:
  useGcpInstanceMetadata: "false"

mediatorwarehousecontroller:
  customnode:
    enabled: true
    workerresource:
      mediator:
        memory: "59055800320" #55GB
    headresource:
      ir:
        memory: "1073741824" #1GB
      os:
        memory: "25769803776" #24GB
      osd:
        memory: "1073741824" #1GB

mediatorcompactorcontroller:
  cstoreCachefs:
    rcachepvsize: "10737418240" #10GB
    pvsize: "12884901888" #12GB
  compaction_resources:
    #enabled: true
    max_compaction_memory: 
      enabled: true
      value: "536870912" # 512M

mediatoringestorcontroller:
  cstoreCachefs:
    mountroot: /cachefs
    rcachepv: /ingest
    rcachepvsize: "10737418240" #10GB
    pvsize: "12884901888" #12GB
  ingestion_resources:
    limits:
      cpu:
        enabled: false
        value: "1000m"
      memory:
        enabled: true
        value: "4096Mi"
    requests:
      cpu:
        enabled: false
        value: "500m"
      memory:
        enabled: false
        value: "1024Mi"

nginx:
  loadBalancerInternal: "true"
  loadBalancerSourceRanges:
  - "0.0.0.0/0"
  awsElbHttps:
    enabled: false
    sslCertARN: arn:aws:iam::user:server-certificate/id # Add ACM's ARN

argo-workflows:
  # -- Keep CRDs on chart uninstall
  crds:
    install: true
    keep: false

prometheus:
  enabled: false
  server:
    persistentVolume:
      enabled: true
      size: 8Gi
    service:
      type: LoadBalancer
      loadBalancerSourceRanges: ["0.0.0.0/0"] # Change this to your CIDR
      annotations:
        service.beta.kubernetes.io/aws-load-balancer-internal: "true"

EKS Shared Cluster with Taint Configuration

• Apply the following taint to your node to mark it as a Mach5-Search node:

    kubectl taint nodes \<nodename\> mach5\=true:NoSchedule 

Contents of values.yaml: (Edit fields marked CHANGE_ME)

mach5ImagePullSecret:
  createSecretResources: true
  dockerconfigjson: "CHANGE_ME" # Contact Mach5 Search Administrator for this value
  
metadatadb:
  name: "postgres"
  host: "postgresdb"
  port: "5432"
  sslmode: "disable"
  user: "postgres"
  password: "" #CHANGE_ME
  externalPostgresdb: false
  nodeassignment:
    enabled: true
    nodeSelector:
      mach5-main-role: "true"
    tolerations:
    - key: "mach5"
      operator: "Equal"
      value: "true"
      effect: "NoSchedule"
    affinity:
  pvc:
    storageclass: "gp2"

mdserver:
  nodeassignment:
    enabled: true
    nodeSelector:
      mach5-main-role: "true"
    tolerations:
    - key: "mach5"
      operator: "Equal"
      value: "true"
      effect: "NoSchedule"
    affinity:

msearchserver:
  nodeassignment:
    enabled: true
    nodeSelector:
      mach5-main-role: "true"
    tolerations:
    - key: "mach5"
      operator: "Equal"
      value: "true"
      effect: "NoSchedule"
    affinity:

meteringserver:
  nodeassignment:
    enabled: true
    nodeSelector:
      mach5-main-role: "true"
    tolerations:
    - key: "mach5"
      operator: "Equal"
      value: "true"
      effect: "NoSchedule"
    affinity:

metricsserver:
  nodeassignment:
    enabled: true
    nodeSelector:
      mach5-main-role: "true"
    tolerations:
    - key: "mach5"
      operator: "Equal"
      value: "true"
      effect: "NoSchedule"
    affinity:

dataexplorer:
  nodeassignment:
    enabled: true
    nodeSelector:
      mach5-main-role: "true"
    tolerations:
    - key: "mach5"
      operator: "Equal"
      value: "true"
      effect: "NoSchedule"
    affinity:

mediator:
  replicaCount: 1
  existingClaim: false
  cstoreSegmentCache:
    enabled: true
  loglevel: info
  useGcpInstanceMetadata: false
  cstoreCachefs:
    rcachepvsize: "483183820800" #450GB
    pvsize: "485331304448" #452GB
  resourceLimit:
    enabled: false

pvc:
  storageclass: "gp2"

teleportcollector:
  useGcpInstanceMetadata: "false"

mediatorwarehousecontroller:
  nodeassignment:
    enabled: true
    nodeSelector: 
      mach5-main-role: "true"
    tolerations:
    - key: "mach5"
      operator: "Equal"
      value: "true"
      effect: "NoSchedule"
    affinity:
  headnodeselector:
    enabled: true
    nodeSelector:
      mach5-warehouse-head-role: "true"
    tolerations:
    - key: "mach5"
      operator: "Equal"
      value: "true"
      effect: "NoSchedule"
    affinity:
  workernodeselector:
    enabled: true
    nodeSelector:
      mach5-warehouse-worker-role: "true"
    tolerations:
    - key: "mach5"
      operator: "Equal"
      value: "true"
      effect: "NoSchedule"
    affinity:
  customnode:
    enabled: true
    workerresource:
      mediator:
        memory: "59055800320" #55GB
    headresource:
      ir:
        memory: "1073741824" #1GB
      os:
        memory: "25769803776" #24GB
      osd:
        memory: "1073741824" #1GB

mediatorcompactorcontroller:
  nodeassignment:
    enabled: true
    nodeSelector:
      mach5-main-role: "true"
    tolerations:
    - key: "mach5"
      operator: "Equal"
      value: "true"
      effect: "NoSchedule"
    affinity:
  compactornodeselector:
    enabled: true
    nodeSelector:
      mach5-compactor-role: "true"
    tolerations:
    - key: "mach5"
      operator: "Equal"
      value: "true"
      effect: "NoSchedule"
    affinity:
  cstoreCachefs:
    rcachepvsize: "10737418240" #10GB
    pvsize: "12884901888" #12GB
  compaction_resources:
    #enabled: true
    max_compaction_memory: 
      enabled: true
      value: "536870912" # 512M

mediatormaterializedviewcontroller:
  nodeassignment:
    enabled: true
    nodeSelector:
      mach5-main-role: "true"
    tolerations:
    - key: "mach5"
      operator: "Equal"
      value: "true"
      effect: "NoSchedule"
    affinity:
  ingestornodeselector:
    enabled: true
    nodeSelector:
      mach5-ingestor-role: "true"
    tolerations:
    - key: "mach5"
      operator: "Equal"
      value: "true"
      effect: "NoSchedule"
    affinity:

mediatoringestorcontroller:
  nodeassignment:
    enabled: true
    nodeSelector:
      mach5-main-role: "true"
    tolerations:
    - key: "mach5"
      operator: "Equal"
      value: "true"
      effect: "NoSchedule"
    affinity:
  ingestornodeselector:
    enabled: true
    nodeSelector:
      mach5-ingestor-role: "true"
    tolerations:
    - key: "mach5"
      operator: "Equal"
      value: "true"
      effect: "NoSchedule"
    affinity:
  cstoreCachefs:
    mountroot: /cachefs
    rcachepv: /ingest
    rcachepvsize: "10737418240" #10GB
    pvsize: "12884901888" #12GB
  ingestion_resources:
    limits:
      cpu:
        enabled: false
        value: "1000m"
      memory:
        enabled: true
        value: "4096Mi"
    requests:
      cpu:
        enabled: false
        value: "500m"
      memory:
        enabled: false
        value: "1024Mi"

nginx:
  nodeassignment:
    enabled: true
    nodeSelector:
      mach5-main-role: "true"
    tolerations:
    - key: "mach5"
      operator: "Equal"
      value: "true"
      effect: "NoSchedule"
    affinity:
  loadBalancerInternal: "true"
  loadBalancerSourceRanges:
  - "0.0.0.0/0"
  awsElbHttps:
    enabled: false
    sslCertARN: arn:aws:iam::user:server-certificate/id # Add ACM's ARN

argo-workflows:
  controller:
    tolerations:
    - key: "mach5"
      operator: "Equal"
      value: "true"
      effect: "NoSchedule"
  server:
    tolerations:
    - key: "mach5"
      operator: "Equal"
      value: "true"
      effect: "NoSchedule"
  # -- Keep CRDs on chart uninstall
  crds:
    install: true
    keep: false

otel:
  nodeassignment:
    enabled: true
    nodeSelector:
    tolerations:
    - key: "mach5"
      operator: "Equal"
      value: "true"
      effect: "NoSchedule"
    affinity:

prometheus:
  enabled: false
  server:
    persistentVolume:
      enabled: true
      size: 8Gi
    service:
      type: LoadBalancer
      loadBalancerSourceRanges: ["0.0.0.0/0"] # Change this to your CIDR
      annotations:
        service.beta.kubernetes.io/aws-load-balancer-internal: "true"

GKE

Contents of values.yaml: (Edit fields marked CHANGE_ME)

mach5ImagePullSecret:
  createSecretResources: true
  dockerconfigjson: "CHANGE_ME" # Contact Mach5 Search Administrator for this value

metadatadb:
  name: "postgres"
  host: "postgresdb"
  port: "5432"
  sslmode: "disable"
  user: "postgres"
  password: "" #CHANGE_ME
  externalPostgresdb: false
  pvc:
    storageclass: "standard"

mediator:
  replicaCount: 1
  existingClaim: false
  cstoreSegmentCache:
    enabled: true
  loglevel: info
  useGcpInstanceMetadata: false
  cstoreCachefs:
    rcachepvsize: "483183820800" #450GB
    pvsize: "485331304448" #452GB
  resourceLimit:
    enabled: false

pvc:
  storageclass: "standard"

mediatorwarehousecontroller:
  customnode:
    enabled: true
    workerresource:
      mediator:
        memory: "59055800320" 
    headresource:
      ir:
        memory: "536870912" # in bytes
      os:
        memory: "25769803776" 
      osd:
        memory: "1073741824"

mediatorcompactorcontroller:
  cstoreCachefs:
    rcachepvsize: "10737418240" #10GB
    pvsize: "12884901888" #12GB
  compaction_resources:
    #enabled: true
    max_compaction_memory: 
      enabled: true
      value: "536870912" # 512M

mediatoringestorcontroller:
  cstoreCachefs:
    mountroot: /cachefs
    rcachepv: /ingest
    rcachepvsize: "10737418240" #10GB
    pvsize: "12884901888" #12GB
  ingestion_resources:
    limits:
      cpu:
        enabled: false
        value: "1000m"
      memory:
        enabled: true
        value: "4096Mi"
    requests:
      cpu:
        enabled: false
        value: "500m"
      memory:
        enabled: false
        value: "1024Mi"

nginx:
  loadBalancerInternal: "true"
  loadBalancerSourceRanges:
  - "0.0.0.0/0"

argo-workflows:
  # -- Keep CRDs on chart uninstall
  crds:
    install: true
    keep: false

resourcecontrol:
  enabled: false

prometheus:
  enabled: false