Warning
Pre-alpha. OpenEverest v2 and this provider are under active development. CRD schemas, chart values and defaults change frequently, including in breaking ways, and there is no supported upgrade path between versions yet. Not for production use.
Run Milvus on Kubernetes through OpenEverest, backed by the Milvus Operator.
OpenEverest providers translate a single, technology-agnostic Instance custom resource into
the native custom resources of an upstream Kubernetes operator — for databases, but equally
for caches, message queues, object storage, or model-serving runtimes. This repository is the
provider for Milvus: it owns the technology-specific knowledge — topologies, versions,
parameters, backup wiring — so that users, the API server, and the UI stay technology-agnostic.
Important
This provider is not standalone. It requires an OpenEverest installation (core CRDs and controller) in the cluster. Installing this chart on its own does nothing. See Install OpenEverest.
flowchart LR
U([User / API / UI]) -->|creates| I["Instance<br/>core.openeverest.io"]
I --> P["provider-milvus<br/>(this repository)"]
P -->|reconciles into| O["Operator CR<br/>milvus.io"]
O --> W["Upstream operator"]
W --> R[("Workloads, Services,<br/>Secrets, PVCs")]
P -->|status, endpoints,<br/>credentials| I
The provider watches Instance resources whose spec.providerRef.name is
milvus, and reports workload health back onto Instance.status. It never
manages pods directly — all lifecycle work is delegated to the operator.
| provider-milvus | OpenEverest | Operator | Kubernetes |
|---|---|---|---|
0.5.x |
2.0.0-dev.5 |
1.3.13 |
1.30 – 1.36 |
0.4.x |
>= 2.0.0 |
1.3.11 |
1.30 – 1.36 |
0.3.x |
>= 2.0.0 |
1.3.11 |
1.30 – 1.34 |
0.2.x |
>= 2.0.0 |
1.3.9 |
1.30 – 1.34 |
0.1.x |
>= 2.0.0 |
1.3.9 |
1.30 – 1.34 |
What you can do to a running instance through the Instance API. Upgrading the
provider itself is covered under Installation.
| Capability | Status | Notes |
|---|---|---|
| Provisioning | ✅ | Standalone and cluster topologies |
| Horizontal scaling | ✅ | Per-component replica counts; CPU/memory autoscaling for proxy, data and query nodes; scale-in is refused while a loaded collection needs the nodes |
| Vertical scaling (CPU / memory) | ✅ | Per-component resource limits |
| Version upgrades | ✅ | spec.version; the operator performs the rolling update |
| Custom configuration | ✅ | Milvus engine config via component parameters.configuration |
| Authentication | ✅ | A root credential is generated and published to the connection Secret |
| Network exposure | ✅ | ClusterIP, NodePort or LoadBalancer via the component service |
| Pod scheduling | ✅ | Per-component schedulingPolicy (affinity, tolerations, node selector, topology spread, scheduler); not applied to bundled dependencies |
| Pod customization | ✅ | Per-component parameters.pod: annotations, env, volumes, volume mounts, container security context, init containers |
| Monitoring | ✅ | Prometheus: a PodMonitor for the Milvus components via topology.parameters.monitoring.prometheus; requires the Prometheus Operator |
| Health checks | ✅ | Ready needs Milvus and its etcd, object storage and message stream healthy; reduced redundancy is reported in status.message; per-component probe tuning via parameters.probes |
| TLS | ✅ | gRPC and REST over TLS via spec.parameters.tls, with a generated certificate, your own, or one from a cert-manager issuer; no mutual TLS or TLS between Milvus components yet |
Stateful workloads additionally report:
| Capability | Status | Notes |
|---|---|---|
| Persistent storage | ✅ | Bundled object storage (MinIO) sized via topology.parameters.dependencies.storage.persistence.size, or external object storage via storage.external; Milvus components are stateless and take no storage |
| Storage expansion | ❌ | |
| Backups (on demand) | ✅ | milvus-backup BackupClass; runs the upstream milvus-backup tool into an S3-compatible BackupStorage |
| Backups (scheduled) | ✅ | Cron schedules in spec.backup, run as CronJobs, with count- or time-based retention |
| Point-in-time recovery | ❌ | |
| Restore | ✅ | Into an existing instance or a new one (spec.dataSource); needs Milvus 2.6.12 or newer |
The provider chart is published as an OCI artifact:
helm install provider-milvus \
oci://ghcr.io/openeverest/charts/provider-milvus \
--version <chart-version> \
--namespace everest-system- The Milvus operator is bundled as a chart dependency and is installed automatically.
Upgrade and uninstall:
helm upgrade provider-milvus oci://ghcr.io/openeverest/charts/provider-milvus
helm uninstall provider-milvus --namespace everest-systemUninstalling the chart does not delete running Instance resources or their data.
Upgrading from 0.2.x moves the bundled operator to 1.3.11. Milvus pods of
existing instances restart once; their bundled MinIO is left as is, while new
instances get its successor, Silo.
Upgrading from 0.4.x moves the bundled operator to 1.3.13. Only instances that
set a container securityContext in parameters.pod restart once, as the
operator now applies it to the config init container too.
Verify that the provider registered itself:
kubectl get providers.core.openeverest.io milvusCreate an instance:
apiVersion: core.openeverest.io/v1alpha1
kind: Instance
metadata:
name: milvus-standalone
spec:
providerRef:
name: milvus
topology:
type: standalone # optional; standalone is the default
components:
standalone:
type: milvus
replicas: 1
resources:
limits:
cpu: "1"
memory: 4GiComponent names are defined by this provider — see definition/provider.yaml.
spec.version and spec.topology are optional; the provider defaults apply. More
examples (including a full cluster topology) live in examples/.
Watch it come up:
kubectl get instance milvus-standalone -wAuthentication is enabled by default. On first boot the provider generates a
random password for the built-in root user and publishes the connection
details to the Secret referenced by .status.connectionSecretRef (named
milvus-standalone-conn):
# host, port, username, password, uri and a ready-to-use root:<password> token
kubectl get secret milvus-standalone-conn \
-o go-template='{{range $k,$v := .data}}{{$k}}={{$v | base64decode}}{{"\n"}}{{end}}'From your workstation, port-forward the service and connect with pymilvus:
kubectl port-forward svc/milvus-standalone-milvus 19530:19530from pymilvus import MilvusClient
client = MilvusClient(uri="http://localhost:19530", token="root:<password>")
client.create_collection("demo", dimension=8)The root password lives in the <instance>-credentials Secret. Change its
password key and the provider updates the root user in Milvus once the
instance is healthy, then publishes the new password to the connection Secret:
kubectl patch secret milvus-standalone-credentials --type merge \
-p '{"stringData":{"password":"<new-password>"}}'Leave appliedPassword alone: it is the password Milvus currently accepts, and
the provider needs it to authenticate the change. Until the rotation lands,
the connection Secret keeps the old, working password.
Set the service type on the client-facing component (standalone for standalone,
proxy for cluster). The connection details then report the external address
automatically:
spec:
components:
standalone:
type: milvus
service:
serviceType: LoadBalancer # or NodePort- ClusterIP (default) — reachable only inside the cluster.
- LoadBalancer — the connection host is the load balancer address once assigned.
- NodePort — the connection host is a node address paired with the assigned node port.
Turn TLS on with the instance-wide tls parameter. Milvus then serves the gRPC
API (port 19530) and the REST API (moved to port 8443: Milvus cannot share a
TLS port between them) over TLS only:
spec:
parameters:
tls:
mode: requireTLS # disabled (default) | requireTLS
# Pick one certificate source, or neither to have one generated:
certificateSecretName: my-tls # tls.crt, tls.key and ca.crt
issuer: # cert-manager
name: corp-ca
kind: ClusterIssuer # Issuer (default) | ClusterIssuer
additionalNames: # extra DNS names or IPs for issued certificates
- milvus.example.com-
Certificates are, in order of precedence:
- your Secret,
certificateSecretName. Its subject alternative names must cover the names clients use, and<instance>-milvus.<namespace>.svc, which the provider and backup Jobs connect to. Leave outca.crtwhen a public CA signed it; - a cert-manager
issuer: the provider requests aCertificatefor the instance's names, written to<instance>-tls-issued; - otherwise a certificate from a CA the provider creates for the instance
(
<instance>-tls-ca, valid for 10 years), written to<instance>-tls. It is valid for a year and re-issued 30 days before it expires, under the same CA.
Issued certificates cover
localhost, the instance's Service in every form (<instance>-milvus,.<namespace>,.svc,.svc.cluster.local), the published host of a LoadBalancer or NodePort service, andadditionalNames. - your Secret,
-
Changes roll out by themselves. Milvus reads its certificate only at startup, so a new certificate (a renewal, a different source, a new published host) restarts the standalone pod or the proxies; so does turning TLS on or off.
-
Clients verify the instance and authenticate with their password, without a client certificate. The connection Secret's
uriturns tohttps://, andca.crtcarries the CA (omitted for a public CA):client = MilvusClient(uri="https://<host>:19530", token="root:<password>", server_pem_path="ca.crt")
Issued certificates cover
localhost, so a port-forward works as is; with your own certificate, passserver_name=one of its names. -
The provider's own calls (root password rotation, the scale-in check) and the
milvus-backupJobs switch to TLS with the instance.
Not supported yet (#88): mutual TLS (client certificates; milvus-backup cannot present one
to the REST API) and TLS between Milvus components or to the bundled etcd and object
storage.
parameters.pod adds pod-level settings to one component, e.g. to attach an
SR-IOV/RDMA secondary network to the query nodes:
spec:
components:
queryNode:
type: milvus
resources:
limits:
nvidia.com/gpu: "1"
rdma/hca: "1"
parameters:
pod:
annotations:
k8s.v1.cni.cncf.io/networks: sriov-rdma
securityContext:
capabilities:
add: ["IPC_LOCK"]volumes, securityContext and initContainers take Kubernetes objects and
are validated when the instance is reconciled.
parameters.groups runs proxy, dataNode, queryNode or streamingNode as
several independently placed workloads, e.g. query nodes on two GPU models.
Groups inherit the component's image, resources and pod settings; the
component's replicas, when set, must equal the groups' total:
spec:
components:
queryNode:
type: milvus
replicas: 6
parameters:
groups:
- name: l40s
replicas: 4
nodeSelector:
nvidia.com/gpu.product: NVIDIA-L40S
- name: h200
replicas: 2
nodeSelector:
nvidia.com/gpu.product: NVIDIA-H200Change a component's replicas to scale it; the standalone topology always
runs one replica. parameters.autoscaling on proxy, dataNode or
queryNode adds a HorizontalPodAutoscaler that keeps at least replicas pods
and adds up to maxReplicas under load. Targets are percentages of the
component's resource requests (its limits when no requests are set), so the
component needs CPU or memory resources; without a target it aims at 80% CPU.
Autoscaling cannot be combined with groups, and the Kubernetes metrics API
(metrics-server) must be available.
spec:
components:
queryNode:
type: milvus
replicas: 2
resources:
limits:
cpu: "2"
memory: 8Gi
parameters:
autoscaling:
maxReplicas: 6
targetCPUUtilization: 70Every replica of a loaded collection needs a query node and a streaming node
of its own. Lowering queryNode or streamingNode replicas (or the
autoscaling minimum) below the replica count of a loaded collection is
rejected; release the collection, or reload it with fewer replicas, first.
To keep Milvus data outside the cluster, point Milvus at existing services
directly. With
useIAM the pods authenticate through the ServiceAccount's cloud identity
(e.g. EKS IRSA); otherwise credentialsSecret names a Secret with the keys
accesskey and secretkey. rootPath lets several instances share one etcd
or bucket. The object store (bundled or external, endpoint and bucket) and
rootPath cannot be changed after creation:
spec:
topology:
type: cluster
parameters:
dependencies:
etcd:
external: true
endpoints: ["etcd-0.etcd:2379", "etcd-1.etcd:2379", "etcd-2.etcd:2379"]
rootPath: vectors-prod
storage:
external: true
type: S3
endpoint: s3.us-east-1.amazonaws.com:443
useSSL: true
bucket: vectors
region: us-east-1
cloudProvider: aws
useIAM: true
serviceAccountName: milvus-s3With the Prometheus Operator in the cluster,
enable Prometheus to get a PodMonitor (<instance>-metrics) that scrapes every
Milvus component on its metrics port. podMonitorLabels lets your Prometheus
podMonitorSelector pick it up; interval defaults to the Prometheus global
scrape interval. Turning it off removes the PodMonitor. The bundled etcd, MinIO
and Pulsar are not scraped.
spec:
topology:
type: standalone
parameters:
monitoring:
prometheus:
enabled: true
interval: 30s
podMonitorLabels:
release: kube-prometheus-stackThe instance is Ready only when Milvus is healthy and the operator's checks
on etcd, object storage and the message stream pass; otherwise
status.message names the failing check. A Ready instance with pods missing
(Milvus components or the bundled etcd, MinIO and Pulsar) stays Ready and
lists them, e.g. serving with reduced redundancy: etcd 2/3 ready, queryNode 1/2 ready;
once any of them has no ready pod at all the instance leaves Ready
(Milvus is unavailable: ...).
Per-component pod counts are in status.components.
Milvus containers get the operator's startup, liveness and readiness probes on
the health port. Tune their timing per component with parameters.probes,
e.g. to give query nodes loading large collections more than the default
3 minutes to start. Unset fields keep the defaults; changing probes rolls the
component's pods.
spec:
components:
queryNode:
parameters:
probes:
startup:
failureThreshold: 60 # 60 x 10s = 10 minutes
readiness:
periodSeconds: 10Backups use the milvus-backup BackupClass. For each Backup, Restore or
deletion the provider runs the upstream milvus-backup CLI as a Job in the
instance's namespace (<backup>-mb, <restore>-mr, <backup>-md); its log is
the place to look when one fails. Backups copy the collections' data and index
definitions into the BackupStorage's bucket under the instance's backup path,
<prefix>/backups/<namespace>/<instance>/<instance-uid>/<storage>, and
restores rebuild the indexes.
apiVersion: backup.openeverest.io/v1alpha1
kind: Backup
metadata:
name: before-reindex
spec:
origin:
type: Instance
instanceRef:
name: my-milvus
classRef:
name: milvus-backup
storageRef:
name: my-s3 # an s3 BackupStorage in the same namespace
parameters:
collections: "default.docs" # optional; empty backs up every collection
---
apiVersion: backup.openeverest.io/v1alpha1
kind: Restore
metadata:
name: undo-reindex
spec:
instanceRef:
name: my-milvus
dataSource:
type: Backup
backup:
backupRef:
name: before-reindex- A restore replaces each collection in the backup with its backed-up contents.
Collections that are not in the backup are left alone. Set
parameters.collectionSuffixto restore next to the existing collections instead. - Restores need Milvus 2.6.12 or newer: older 2.6 releases corrupt the restored data (milvus-io/milvus#48114). Backups work on every version.
- A new instance can start from a backup with
spec.dataSource; it must list the backup's storage inspec.backup.storages. The instance staysRestoringuntil the data is in. - Writes during a backup are not frozen: each collection is consistent with itself, but not with the others.
- Only one backup, restore or deletion runs per instance at a time; the others
wait in
Pending. - A restore from another instance's backup needs that instance to still exist: its UID is part of the backup path.
- milvus-backup verifies TLS against the system trust store only, so
BackupStorages that set
tls.insecureSkipVerifyortls.caare rejected.
Schedules live on the instance. Each enabled schedule becomes a CronJob
(<instance>-<schedule>), and every run shows up as a Backup named after its
Job, with spec.scheduleName set:
spec:
backup:
enabled: true
classRef:
name: milvus-backup
storages:
- storageRef:
name: my-s3
schedules:
- name: nightly
enabled: true
cron: "0 3 * * *" # standard 5-field cron, in UTC
retention:
type: count # or type: time with duration: 7d / 2w / 1m
count: 7
parameters:
collections: "default.*"- Retention deletes the oldest successful runs of each schedule, together
with their data unless the Backup's
deletionPolicyisRetain. Only the newest 3 failed runs per schedule are kept. Backups taken on demand, and runs of a schedule that was removed, are never deleted by retention. - Disabling or removing a schedule deletes its CronJob; its Backups stay.
- Schedules are suspended while the instance is being restored or seeded. A run that would have started then is skipped, not delayed.
- Runs that overlap are skipped. Scheduled runs do not wait for on-demand backups, restores or deletions of the same instance; on-demand ones do wait for scheduled runs.
| Topology | Default | Description |
|---|---|---|
standalone |
✅ | Single-process Milvus; smallest footprint, ideal for experimentation |
cluster |
Independently scalable components; Woodpecker (default) or Pulsar as the message stream |
| Version bundle | Default | |
|---|---|---|
2.6.27 |
✅ | |
2.6.27-gpu |
GPU build of every component | |
2.6.25 |
||
2.6.25-gpu |
GPU build of every component | |
2.6.15 |
||
2.6.15-gpu |
GPU build of every component | |
2.6.11 |
||
2.6.10 |
Source of truth: definition/versions.yaml.
GPU bundles only switch images; request GPUs per component with
resources.limits["nvidia.com/gpu"] and place the pods with schedulingPolicy.
spec.components.<name>.image overrides a single component's image (e.g. a
registry mirror). Components sharing one image are upgraded one at a time in
the operator's dependency order (imageUpdateMode: rollingUpgrade); components
on different images are updated all at once (all), because the ordered
upgrade can only finish once every component runs the same image. The Instance
reports Updating, with the operator's progress, until the rollout completes.
Versions only move forward. Milvus migrates its metadata when it upgrades, so a version older than the one deployed — for the bundle or for a single pinned component — is rejected, and so is reverting an upgrade that has already started.
Any newer bundle can be applied directly; all bundles are Milvus 2.6 patch releases. The Milvus operator ships inside the provider chart, so upgrading the chart upgrades the operator as well.
- Chart values: charts/provider-milvus/values.yaml
- Instance parameters: per-component and per-topology
parametersschemas, defined under definition/ and published on theProviderresource (kubectl get provider milvus -o yaml). The API server and the UI validate user input against these schemas.
Worth knowing:
- Object storage size —
topology.parameters.dependencies.storage.persistence.sizesizes the bundled MinIO and can only grow. The Milvus components keep their data in the object storage and take no storage of their own. - Storage class —
topology.parameters.dependencies.storageClassNamesets the StorageClass of every bundled dependency volume (etcd, MinIO, and Pulsar in cluster mode). Empty uses the cluster default. It is fixed at creation. - Message stream — cluster instances keep their write-ahead log in the object
storage (Woodpecker) unless
dependencies.messageStreamTypeispulsar. It is fixed at creation, like the topology and the etcd and storagerootPath. - Engine configuration —
parameters.configurationon a component takes Milvus YAML (e.g.log.level), merged into the Milvus configuration. - Coordinator HA — more than one
mixCoordreplica runs active-standby.
Requires Go (see go.mod), Docker, Helm, kubectl, and a Kubernetes cluster you can
reach. dev/README.md covers the environment end to end: the recommended
local k3d setup, running against a cluster you already have, and every dev/.env setting.
make dev-up # local cluster + Tilt dev environment (see dev/README.md)
make generate # RBAC, provider spec, Helm chart sync
make run # run the provider locally against the cluster
make test-unit
make test-integration # chainsaw suites under test/integration/
make dev-downmake help lists every target. make verify fails when generated files are stale — run
make generate and commit the result.
The provider contract (Validate / Sync / Status / Cleanup), RBAC markers, watches,
code generation, and the backup/restore interfaces are documented once for all providers in
PROVIDER_DEVELOPMENT.md.
| Path | Purpose |
|---|---|
cmd/provider/ |
Entry point |
internal/provider/ |
ProviderInterface implementation, backup interfaces, RBAC markers |
internal/common/ |
Component name constants |
definition/ |
Provider identity, component types, versions, topologies, backup classes |
charts/provider-milvus/ |
Helm chart (generated/ is produced by make generate) |
config/rbac/role.yaml |
Generated ClusterRole — do not edit |
test/integration/ |
Chainsaw suites (see its README.md) |
test/vars.sh |
Pinned operator and workload versions used by tests |
examples/ |
Example Instance resources |
dev/ |
Tilt dev environment, .env configuration, k3d cluster config |
.github/workflows/ |
CI: lint, build, unit and integration tests, release |
- Unit tests —
make test-unit. - Integration tests — chainsaw suites under
test/integration/, run with the operator scaled to 0. Thecore/suite covers create, authentication, dependency health, exposure, scheduling and delete;scaling/covers scale-out and autoscaling;features/covers engine configuration, pod customization, deployment groups, probes, monitoring, upgrade and downgrade rejection, external dependencies, Pulsar, a storage location and TLS (generated, user and cert-manager certificates). See test/integration/README.md. - Live end-to-end tests —
make test-e2e-liveruns real standalone and cluster instances on the current cluster with the real operator: smoke test (pymilvus), backup and restore, upgrade with data, rejected downgrade, scaling, autoscaling, and clean deletion. - Live TLS tests —
make test-e2e-tlsserves real instances over TLS: clients over gRPC and REST, plain text rejected, root password rotation, backup and restore, the user's certificate and its renewal, TLS turned off, and a cluster whose certificate a cert-manager issuer signs (when cert-manager is installed). - Conformance tests — part of
make test-unit. The OpenEverest conformance harness checks that every UI field and everysupportedFieldsdeclaration changes what the provider applies; the version matrix checks every shipped version bundle validates, deploys a single image per instance, and upgrades (never downgrades) to every other bundle. - Live version matrix —
make test-e2e-versionscreates a standalone instance at every version bundle the installed Provider offers, writes data, upgrades it to the newest bundle of the same build and verifies the data. - CI —
.github/workflows/ci.yamlruns lint, build, unit tests, generated-file verification, Helm lint, and each integration suite on every pull request..github/workflows/e2e.yamlruns the live end-to-end tests, the live TLS tests and the live version matrix on k3d (about 30 minutes each, in parallel) nightly, on demand, and on pull requests labellede2e.
kubectl logs -n everest-system deploy/provider-milvus -f| Symptom | Where to look |
|---|---|
Instance stuck in Creating |
kubectl describe instance <name> conditions, then the provider logs |
No Provider resource in the cluster |
Is the chart installed? Check the provider deployment logs |
Instance ignored entirely |
spec.providerRef.name must be milvus |
| Operator resource created but no pods | Inspect the operator's custom resource status — the failure is upstream |
| A spec change has no effect | The provider rejected it; the reason is in the provider logs |
| Restore refused | Restores need Milvus 2.6.12 or newer on the target instance |
Instance leaves Ready with a Milvus dependency is not ready |
etcd, object storage or the message stream is unreachable; the message carries the operator's check result |
Instance leaves Ready with Milvus is unavailable |
A Milvus component or bundled dependency has no ready pod; kubectl get pods in the instance namespace |
Ready with serving with reduced redundancy |
Some pods are not ready; the message lists each workload as ready/desired, e.g. etcd 2/3 ready |
| Volumes left behind after delete | The bundled etcd, MinIO and Pulsar PVCs are deleted with the instance; a StorageClass with reclaimPolicy: Retain keeps the underlying volumes. Pick a Delete class with dependencies.storageClassName when the cluster default retains |
Issues and pull requests are welcome. See PROVIDER_DEVELOPMENT.md and the OpenEverest Code of Conduct.
Report vulnerabilities per the OpenEverest security policy. Please do not open public issues for security reports.
Apache License 2.0 — see LICENSE for details.