Use Prometheus monitoring ↗

Documentation Index#

Fetch the complete documentation index at: https://docs.wandb.ai/llms.txt Use this file to discover all available pages before exploring further.

Use Prometheus with W&B Server. Prometheus installs are exposed as a kubernetes ClusterIP service.

Prometheus monitoring is only available with Self-Managed instances.

Follow the procedure below to access your Prometheus metrics endpoint (/metrics):

1. Connect to the cluster with Kubernetes CLI toolkit, kubectl. See kubernetes’ Accessing Clusters documentation for more information.

2. Find the internal address of the cluster with:

   kubectl describe svc prometheus

3. Start a shell session inside your container running in your Kubernetes cluster with kubectl exec. Hit the endpoint at <internal address>/metrics.

   Copy the command below and execute it in your terminal and replace <internal address> with your internal address:

   kubectl exec <internal address>/metrics

A test pod starts, which you can exec into just to access anything in the network:

kubectl run -it testpod --image=alpine bin/ash --restart=Never --rm

From there you can choose to keep access internal to the network or expose it yourself with a kubernetes nodeport service.