Back to AI News
MarkTechPost

NVIDIA Open-Sources NVKE and NVKR for Managed Kubernetes on GPU Infrastructure

NVIDIA has released NVKE and NVKR, two Apache 2.0 open-source projects for running managed Kubernetes on NVIDIA GPU infrastructure. NVKE acts as the control plane for managed clusters, while NVKR builds signed node images, container images, Helm charts and related packages. The projects are in public preview and conformance-tested on Kubernetes 1.36, but the article highlights preview-grade security and lifecycle gaps, including an unauthenticated HTTP API and no management-plane upgrade or backup procedure.

AI Brief

It helps readers understand a new open-source Kubernetes stack for NVIDIA GPU clouds and the practical limits that make it unsuitable for production today.

NVIDIA has released 2 open-source projects for running managed Kubernetes on GPU infrastructure: NVIDIA Kubernetes Engine (NVKE) and NVIDIA Kubernetes Recipes (NVKR). Both were recently announced as components of the NVIDIA DSX AI Factory Platform. NVKE is the control plane for a managed Kubernetes service. NVKR builds the OS images, container images and packages those clusters depend on. Both ship under Apache 2.0 and are in public preview.

TL;DR

  • Size / footprint: 3 CPU servers for the management cluster, 3 CPU servers per workload control plane, plus a pool of GPU worker nodes. A local evaluation needs 6 CPUs, 12 GiB RAM, 100 GiB disk and no GPU.
  • Runs on: your own NVIDIA GPU bare-metal servers, provisioned through NVIDIA Infra Controller (NICo). Node images build for amd64 and arm64.
  • Performance: no throughput or provisioning-time benchmarks are published. Quality signal is certification, not speed.
  • Best: Certified Kubernetes AI Platform and Certified Kubernetes Installer, conformance tested on Kubernetes 1.36.
  • Worst: after install, the NVKE API serves plain HTTP on port 30080 with authentication off, and there is no management-plane upgrade or backup procedure.
  • Bottom line: best thing: open, attested, upstream Kubernetes for GPU clouds. Worst thing: preview-grade security and lifecycle gaps; not for production yet.

What is NVIDIA Kubernetes Engine (NVKE)?

NVKE is open-source software that an AI factory installs to run a managed Kubernetes service on its own NVIDIA GPU infrastructure. The repo calls it ‘a self-hosting factory for Kubernetes clusters,’ not a distribution. It exposes a REST API and controllers on a management cluster. Customers get standard clusters and use kubectl. The operator manages them through the NVKE API or the nvkectl CLI.

NVKE uses upstream Kubernetes, kubeadm and Cluster API. It is written in Go and has no runtime dependency on NVKR.

What does NVIDIA Kubernetes Recipes (NVKR) do?

NVKR is a build system with declarative Definitions. It produces executables, container images, Helm charts and node images for amd64 and arm64. ‘Recipes’ define validated combinations of software versions and GPU generations. Hosted workflows attach signatures, SBOMs and provenance to digest-pinned OCI references. NVKR can also build artifacts for other Kubernetes management systems.

How does the NVKE architecture work?

The NVIDIA technical blog walkthrough describes 5 stages:

  1. Bootstrap: nvkectl bootstrap site starts a temporary single-node kind cluster on a bootstrap host.
  2. Management cluster: NVKE requests 3 servers from NICo, which boots them from the node image. The temporary cluster is then deleted.
  3. Machine provisioning: NICo handles bare-metal lifecycle. CAPNICo, a Cluster API provider, connects NICo to NVKE.
  4. Release sets: operators publish a machine image, a cluster release and a node pool release. Customers pick a release by name.
  5. Workload clusters: each customer cluster has its own control plane, etcd and node pools. NVKE installs OVN-Kubernetes and NodeLocal DNSCache.

A node must present an attested SPIFFE identity before it gets join credentials. The README says SPIRE uses TPM for this. Per-cluster etcd encryption is available. An OpenTelemetry-based Helm chart exports logs, metrics and traces over OTLP.

Inside workload clusters, 3 companion projects plug in: AI Cluster Runtime (AICR) for the GPU runtime, NodeWright for node packages and NVSentinel for break-fix automation.

NVIDIA DSX AI Factory Platform · Public preview

NVKE + NVKR: managed Kubernetes on your own GPUs

NVKE runs the clusters. NVKR builds what they run. Both Apache 2.0.

1.36
Kubernetes version NVKE is conformance tested on
3
CPU servers for the management cluster
2
Architectures: amd64 and arm64
0
GPUs needed for the local quickstart (6 CPUs, 12 GiB RAM)
NVKE

Manages the clusters

REST API and controllers on a management cluster. Creates customer clusters, each with its own control plane, etcd and node pools. CLI: nvkectl.

NVKR

Builds the software

Node images, container images, Helm charts and binaries from pinned sources, with signatures, SBOMs and provenance. Works for other K8s managers too.

Certified Kubernetes AI Platform and Certified Kubernetes Installer. NVKE has no runtime dependency on NVKR.

Bootstrapkind seed
Mgmt cluster3 servers
CAPNICo + NICobare metal
Release set3 parts
Workload clusterkubectl
1
Definition
Pick outputs and pinned source versions
2
Build
Node agent and OVN-Kubernetes compiled from source; checksum-pinned K8s 1.36 + SPIRE
3
Sign
Signatures, SBOMs, provenance on digest-pinned OCI refs
4
Publish
QCOW2 node image, controller image, Helm chart to registry
Local builds are not a qualified release; no NVKR Release is qualified yet.
NVKE + NVKRGardenerKamaji
What it isManaged K8s plane + artifact builderClusters as a serviceHosted control plane manager
Control planeDedicated per clusterPods in seed clustersPods; shared Datastore possible
InfraNVIDIA GPU bare metal via NICoAWS, Azure, GCP, OpenStack, AlicloudAlongside Cluster API providers
LicenseApache 2.0Apache 2.0Apache 2.0
ConformanceTested on 1.36; AI Platform certifiedCertified up to v1.35Described as CNCF-compliant
Node identitySPIFFE/SPIRE + TPMNot disclosedNot disclosed
MaturityPublic previewSelf-described production-readyNot disclosed

!Open API by default

After install the API serves plain HTTP on port 30080 of every management node, with authentication off.

!No authorization

Any authenticated caller can create and delete any workload cluster.

!Kubeconfigs can't be revoked

A handed-over admin kubeconfig stays valid until its client certificate expires.

!No mgmt-plane lifecycle

The management cluster cannot upgrade, scale or replace its own nodes. No backup or restore procedure exists.

!Nodes not auto-replaced

Node pool config changes apply only to new nodes. Outdated nodes are flagged but not drained or deleted.

!Eval-only telemetry

The default backend stores data in memory and gives anonymous users Grafana admin access.

Click a card to expand. Source: NVKE preview status page.

Sources: NVIDIA blog · NVKE repo · preview status© Marktechpost

How does NVKE compare with Gardener and Kamaji?

NVKE + NVKR (NVIDIA)Gardener (SAP)Kamaji (Clastix)
What it isManaged Kubernetes management plane plus artifact builderKubernetes clusters as a serviceHosted control plane manager
Control plane modelDedicated per cluster: own control plane, etcd, node poolsRuns as pods in seed clustersRuns as pods; shared Datastore possible
Target infrastructureNVIDIA GPU bare metal via NICoAWS, Azure, GCP, OpenStack, Alicloud (conformance table); extensibleWorks alongside Cluster API providers
LicenseApache 2.0Apache 2.0Apache 2.0
ConformanceCertified Kubernetes AI Platform and Installer; tested on 1.36Certified up to v1.35Described as CNCF-compliant
Node identitySPIFFE/SPIRE attestation with TPMNot disclosedNot disclosed
Signed build pipelineYes (NVKR: signatures, SBOMs, provenance)Not disclosedNot disclosed
MaturityPublic preview, not for productionSelf-described production-readyNot disclosed

Who is adopting NVKE and NVKR?

NVIDIA Cloud Partners Firmus and SK Telecom are integrating both projects into AI factories. Mirantis, Rafay, Spectro Cloud, OpenNebula Systems and vCluster are adding components to managed Kubernetes offerings. Netris is enabling NVKE for its network automation.

How do you try NVKE?

A local quickstart runs on 1 Linux host or VM, amd64 or arm64. It needs at least 6 CPUs, 12 GiB free memory, 100 GiB free disk, Docker and virtual TPM support. No GPU or NVIDIA account is required. A real install needs a NICo deployment with enrolled capacity, a build host with KVM, an HTTP server and a registry.

NVKR has no qualified Release yet.

Key Takeaways

  • NVKE manages clusters; NVKR builds and signs what they run.
  • Both are Apache 2.0, in public preview, not for production.
  • Conformance tested on Kubernetes 1.36, with TPM-attested node identity.
  • Default API is HTTP on port 30080 with auth off.
  • Local evaluation needs 6 CPUs, 12 GiB RAM and no GPU.


Check out the NVKE repo, NVKR repo, docs and technical blog. You can check the deployment options and specs here through our GPULeaderboard.com chrome extension.

Also, feel free to follow us on Twitter and don’t forget to join our 150k+ML SubReddit and Subscribe to our Newsletter. Wait! are you on telegram? now you can join us on telegram as well.

Need to partner with us for promoting your GitHub Repo OR Hugging Face Page OR Product Release OR Webinar etc.? Connect with us

The post NVIDIA Open-Sources NVKE and NVKR for Managed Kubernetes on GPU Infrastructure appeared first on MarkTechPost.

Be the first to rate this article.

Comments & Questions (0)

Captcha
Please be respectful — let's keep the conversation friendly.

No comments yet

Be the first to comment!