The page has been translated by Gen AI.

Installing the ServiceWatch Agent

Users can install the ServiceWatch Agent on a GPU Server to collect custom metrics and logs.

Notice
When collecting custom metrics or logs using the ServiceWatch Agent, contact us separately for the kr-south3 region by referring to Support Center > Inquiry.
Warning
Metric collection through the ServiceWatch Agent is billed separately from the default metrics collected from each service, as it is categorized as custom metrics. We recommend removing or disabling any unnecessary metric collection settings.

ServiceWatch Agent

The agents that must be installed on a GPU server to collect ServiceWatch custom metrics and logs can be broadly divided into two types. They are the Prometheus Exporter and the Open Telemetry Collector.

CategoryDetailed Description
Prometheus ExporterProvides metrics for specific applications or services in a format that Prometheus can scrape.
  • To collect OS metrics from servers, you can use Node Exporter for Linux servers and Windows Exporter for Windows servers, depending on the OS type.
Open Telemetry CollectorActs as a centralized collector that gathers telemetry data such as metrics and logs from distributed systems, processes it (e.g., filtering, sampling), and exports it to multiple backends (e.g., Prometheus, Jaeger, Elasticsearch, etc.)
  • Exports data to the ServiceWatch Gateway, enabling ServiceWatch to collect metrics and log data.
Table. Description of Prometheus Exporter and Open Telemetry Collector
Warning

If you have configured Kubernetes Engine on a GPU server, you can check GPU metrics using the metrics provided by Kubernetes Engine.

  • If you install the DCGM Exporter on a GPU server that is part of a Kubernetes Engine cluster, it may not function properly.

Preconfiguration for Using the ServiceWatch Agent

To use the ServiceWatch Agent, prepare the prerequisites by referring to Prerequisites for the ServiceWatch Agent.

Reference
The installation commands in this guide require root privileges, so they use sudo. If you are logged in as the root user, you can omit sudo.

Install Prometheus Exporter for GPU Metrics (for Ubuntu)

Install the Prometheus Exporter to collect metrics for the GPU server by following the steps below.

Step 1. NVIDIA Driver installation check
Step 2. NVSwitch Configuration and Query (NSCQ) Library Installation
Step 3. NVSwitch Device Monitoring API (NVSDM) Library Installation
Step 4. Install NVIDIA DCGM

Verify NVIDIA Driver Installation

  • Check the installed NVIDIA driver.
    Color mode
    nvidia-smi --query-gpu=driver_version --format=csv
    nvidia-smi --query-gpu=driver_version --format=csv
    Code block. Command to check the NVIDIA Driver version
    Color mode
    driver_version
535.183.06
...
535.183.06
    driver_version
535.183.06
...
535.183.06
    Code block. Example of checking the NVIDIA Driver version

Installing the NVSwitch Configuration and Query (NSCQ) Library

Reference
The NVSwitch Configuration and Query (NSCQ) library is required for Hopper or earlier generation GPUs.
Notice
The following installation commands are available in environments with internet access. If you are in an environment without internet access, you must download and upload libnvidia-nscq from https://developer.download.nvidia.com/compute/cuda/repos/.
  1. Install cuda-keyring.
    Color mode
    wget https://developer.download.nvidia.com/compute/cuda/repos/<distro>/<arch>/cuda-keyring_1.1-1_all.deb
    wget https://developer.download.nvidia.com/compute/cuda/repos/<distro>/<arch>/cuda-keyring_1.1-1_all.deb
    Code block. NSCQ library download command
Reference

<distro> and <arch> are specified to match the server environment.

  • Example: The Ubuntu 22.04 / x86_64 environment is ubuntu2204/x86_64

Color mode
sudo dpkg -i cuda-keyring_1.1-1_all.deb
sudo apt update
sudo dpkg -i cuda-keyring_1.1-1_all.deb
sudo apt update
Code block. NSCQ library installation command
Color mode
nvidia-smi --query-gpu=driver_version --format=csv
nvidia-smi --query-gpu=driver_version --format=csv
Code block. Command to check the NVIDIA Driver version
Color mode
driver_version
535.183.06
...

535.183.06
driver_version
535.183.06
...

535.183.06
Code block. Example of checking the NVIDIA Driver version

  1. Install libnvidia-nscq.
    Color mode
    apt-cache policy libnvidia-nscq-535
    apt-cache policy libnvidia-nscq-535
    Code block. NSCQ library apt-cache command
    Color mode
    libnvidia-nscq-535:
  Installed: (none)
  Candidate: 535.247.01-1
  Version table:
     535.247.01-1 600
        600 https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2204/x86_64  Packages
...
        600 https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2204/x86_64  Packages
     535.216.01-1 600
        600 https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2204/x86_64  Packages
     535.183.06-1 600  # Install the version that matches the driver
        600 https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2204/x86_64  Packages
     535.183.01-1 600
...
        600 https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2204/x86_64  Packages
     535.54.03-1 600
        600 https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2204/x86_64  Packages
    libnvidia-nscq-535:
  Installed: (none)
  Candidate: 535.247.01-1
  Version table:
     535.247.01-1 600
        600 https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2204/x86_64  Packages
...
        600 https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2204/x86_64  Packages
     535.216.01-1 600
        600 https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2204/x86_64  Packages
     535.183.06-1 600  # Install the version that matches the driver
        600 https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2204/x86_64  Packages
     535.183.01-1 600
...
        600 https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2204/x86_64  Packages
     535.54.03-1 600
        600 https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2204/x86_64  Packages
    Code block. NSCQ library apt-cache command output
    Color mode
    sudo apt install libnvidia-nscq-535=535.183.06-1
    sudo apt install libnvidia-nscq-535=535.183.06-1
    Code block. NSCQ library installation command
Notice

It must be installed with the same version as the NVIDIA Driver.

  • Example) driver version: 535.183.06, libnvidia-nscq version: 535.183.06-1

Install the NVSwitch Device Monitoring API (NVSDM) Library

Reference
The NVSDM Library must be installed for GPU Architectures after Blackwell. NVSDM Library is not available for NVIDIA Driver versions 560 and earlier.
  • Installs the NVSDM library.
    Color mode
    apt-cache policy libnvsdm
    apt-cache policy libnvsdm
    Code block. NVSDM library apt-cache command
    Color mode
    libnvsdm:
  Installed: (none)
  Candidate: 580.105.08-1
  Version table:
     580.105.08-1 600
        600 https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2204/x86_64  Packages
     580.95.05-1 600
        600 https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2204/x86_64  Packages
     580.82.07-1 600
        600 https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2204/x86_64  Packages
     580.65.06-1 600
        600 https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2204/x86_64  Packages
    libnvsdm:
  Installed: (none)
  Candidate: 580.105.08-1
  Version table:
     580.105.08-1 600
        600 https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2204/x86_64  Packages
     580.95.05-1 600
        600 https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2204/x86_64  Packages
     580.82.07-1 600
        600 https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2204/x86_64  Packages
     580.65.06-1 600
        600 https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2204/x86_64  Packages
    Code block. NVSDM library apt-cache command output
    Color mode
    sudo apt install libnvsdm=580.105.08-1
    sudo apt install libnvsdm=580.105.08-1
    Code Block. Installing the NVSDM library

Install NVIDIA DCGM (for Ubuntu)

Install the DCGM Exporter in the following order.

  1. DCGM (datacenter-gpu-manager) installation
  2. datacenter-gpu-manager-exporter installation
  3. Enabling and Starting the DCGM Service

Install DCGM (datacenter-gpu-manager)

Refers to a specific version of the NVIDIA Data Center GPU Manager (DCGM) tool, which is a package for managing and monitoring NVIDIA data center GPUs. In particular, cuda12 indicates that this management tool is installed for the CUDA 12 version, and datacenter-gpu-manager-4 refers to the 4.x version of DCGM. This tool provides a variety of features, including GPU status monitoring, diagnostics, alert systems, and power/clock management.

  1. Check the CUDA version.
    Color mode
    nvidia-smi | grep CUDA
    nvidia-smi | grep CUDA
    Code block. Check the CUDA version
    Color mode
    | NVIDIA-SMI 535.183.06             Driver Version: 535.183.06     CUDA Version: 12.2 |
    | NVIDIA-SMI 535.183.06             Driver Version: 535.183.06     CUDA Version: 12.2 |
    Code block. Example of CUDA version check results
    Color mode
    CUDA_VERSION=12
    CUDA_VERSION=12
    Code block. CUDA version setting command
  2. Install datacenter-gpu-manager-cuda.
    Color mode
    sudo apt install datacenter-gpu-manager-4-cuda${CUDA_VERSION}
    sudo apt install datacenter-gpu-manager-4-cuda${CUDA_VERSION}
    Code block. The installation command for datacenter-gpu-manager-cuda

Install datacenter-gpu-manager-exporter

A tool that collects various GPU metrics, such as GPU utilization, memory usage, temperature, and power consumption, based on the NVIDIA Data Center GPU Manager (DCGM) and exposes them for use in monitoring systems like Prometheus.

  1. Install datacenter-gpu-manager-exporter.
    Color mode
    sudo apt install datacenter-gpu-manager-exporter
    sudo apt install datacenter-gpu-manager-exporter
    Code block. datacenter-gpu-manager-exporter installation command
  2. Check the DCGM Exporter configuration file.
    Color mode
    cat /usr/lib/systemd/system/nvidia-dcgm-exporter.service | grep ExecStart
    cat /usr/lib/systemd/system/nvidia-dcgm-exporter.service | grep ExecStart
    Code block. Command to check the datacenter-gpu-manager-exporter configuration file
    Color mode
    ExecStart=/usr/bin/dcgm-exporter -f /etc/dcgm-exporter/default-counters.csv
    ExecStart=/usr/bin/dcgm-exporter -f /etc/dcgm-exporter/default-counters.csv
    Code block. Example of the datacenter-gpu-manager-exporter configuration file
  3. Check the configuration provided when installing the DCGM Exporter, remove the required metrics with #, and add the unnecessary metrics with #
    Color mode
    sudo vi /etc/dcgm-exporter/default-counters.csv
## Example ##
...
DCGM_FI_PROF_PIPE_TENSOR_ACTIVE, gauge, Ratio of cycles the tensor (HMMA) pipe is active.
DCGM_FI_PROF_DRAM_ACTIVE,        gauge, Ratio of cycles during which the device memory interface is active, sending or receiving data.
# DCGM_FI_PROF_PIPE_FP64_ACTIVE,   gauge, Ratio of cycles during which the fp64 pipes are active.
# DCGM_FI_PROF_PIPE_FP32_ACTIVE,   gauge, Ratio of cycles during which the fp32 pipes are active.
...
    sudo vi /etc/dcgm-exporter/default-counters.csv
## Example ##
...
DCGM_FI_PROF_PIPE_TENSOR_ACTIVE, gauge, Ratio of cycles the tensor (HMMA) pipe is active.
DCGM_FI_PROF_DRAM_ACTIVE,        gauge, Ratio of cycles during which the device memory interface is active, sending or receiving data.
# DCGM_FI_PROF_PIPE_FP64_ACTIVE,   gauge, Ratio of cycles during which the fp64 pipes are active.
# DCGM_FI_PROF_PIPE_FP32_ACTIVE,   gauge, Ratio of cycles during which the fp32 pipes are active.
...
    Code block. Example datacenter-gpu-manager-exporter metric configuration
Reference
For the metrics that can be collected by the GPU DCGM Exporter and how to configure them, refer to DCGM Exporter metrics.
Warning
Metric collection through the ServiceWatch Agent is billed separately from default metrics because it is categorized as custom metrics. To avoid excessive charges, remove or disable any unnecessary metric collection.

Enable and Start the DCGM Service

  1. Activate and start the nvdia-dcgm service.

    Color mode
    sudo systemctl enable --now nvidia-dcgm
    sudo systemctl enable --now nvidia-dcgm
    Code block. Commands to enable and start the nvdia-dcgm service

  2. Activate and start the nvdia-dcgm-exporter service.

    Color mode
    sudo systemctl enable --now nvidia-dcgm-exporter
    sudo systemctl enable --now nvidia-dcgm-exporter
    Code block. Commands to enable and start the nvdia-dcgm-exporter service

Notice
If you have completed the DCGM Exporter configuration, you must install the Open Telemetry Collector provided by ServiceWatch to complete the ServiceWatch Agent configuration.
For details, refer to ServiceWatch > Using ServiceWatch Agent.

Install the Prometheus Exporter for GPU Metrics (for RHEL)

Install the ServiceWatch Agent on the GPU server to collect metrics, following the steps below.

Step 1. NVIDIA Driver installation check
Step 2. Install the NVSwitch Configuration and Query (NSCQ) Library
Step 3. NVSwitch Device Monitoring API (NVSDM) Library Installation
Step 4. Install NVIDIA DCGM

Verify NVIDIA Driver Installation (for RHEL)

  1. Check the installed NVIDIA driver.
    Color mode
    nvidia-smi --query-gpu=driver_version --format=csv
    nvidia-smi --query-gpu=driver_version --format=csv
    Code block. Command to check the NVIDIA Driver version
    Color mode
    driver_version
535.183.06
...

535.183.06
    driver_version
535.183.06
...

535.183.06
    Code block. Example of checking the NVIDIA Driver version

Install the NVSwitch Configuration and Query (NSCQ) Library (for RHEL)

Reference

The NVSwitch Configuration and Query (NSCQ) library is required for Hopper or earlier generation GPUs.

  • For RHEL, check whether libnvidia-nscq is installed, and install it if it is not.
Notice
The following installation commands are available in environments with internet access. If you are in an environment without internet access, you must download and upload libnvidia-nscq from https://developer.download.nvidia.com/compute/cuda/repos/.

  1. Check the libnvidia-nscq package.

    Color mode
    rpm -qa | grep libnvidia-nscq
    rpm -qa | grep libnvidia-nscq
    Code block. Check the NSCQ library package
    Color mode
    libnvidia-nscq-535-535.183.06-1.x86_64
    libnvidia-nscq-535-535.183.06-1.x86_64
    Code block. Example of verifying the NSCQ library package

  2. Add the CUDA repository to DNF.

    Color mode
    sudo dnf config-manager --add-repo https://developer.download.nvidia.com/compute/cuda/repos/rhel8/x86_64/cuda-rhel8.repo
    sudo dnf config-manager --add-repo https://developer.download.nvidia.com/compute/cuda/repos/rhel8/x86_64/cuda-rhel8.repo
    Code Blocks. Add DNF Repository

  3. Reset NVIDIA Driver Status

    Color mode
    sudo dnf module reset nvidia-driver
    sudo dnf module reset nvidia-driver
    Code block. Reset the status of the NVIDIA Driver DNF module
    Color mode
    Updating Subscription Management repositories.
Last metadata expiration check: 0:03:15 ago on Wed 19 Nov 2025 01:23:48 AM EST.
Dependencies resolved.
=============================================
Package Architecture Version Repository Size
=============================================
Disabling module profiles:
nvidia-driver/default
nvidia-driver/fm
Resetting modules:
nvidia-driver

Transaction Summary
=============================================

Is this ok [y/N]: y
    Updating Subscription Management repositories.
Last metadata expiration check: 0:03:15 ago on Wed 19 Nov 2025 01:23:48 AM EST.
Dependencies resolved.
=============================================
Package Architecture Version Repository Size
=============================================
Disabling module profiles:
nvidia-driver/default
nvidia-driver/fm
Resetting modules:
nvidia-driver

Transaction Summary
=============================================

Is this ok [y/N]: y
    Code block. Example of the result of resetting the status of the NVIDIA Driver DNF module

  4. Enables the NVIDIA Driver module.

    Color mode
    sudo dnf module enable nvidia-driver:535-open
    sudo dnf module enable nvidia-driver:535-open
    Code block. Enable the NVIDIA Driver module
    Color mode
    Updating Subscription Management repositories.
Last metadata expiration check: 0:04:22 ago on Wed 19 Nov 2025 01:23:48 AM EST.
Dependencies resolved.
=============================================
Package Architecture Version Repository Size
=============================================
Enabling module streams:
nvidia-driver 535-open

Transaction Summary
=============================================

Is this ok [y/N]: y
    Updating Subscription Management repositories.
Last metadata expiration check: 0:04:22 ago on Wed 19 Nov 2025 01:23:48 AM EST.
Dependencies resolved.
=============================================
Package Architecture Version Repository Size
=============================================
Enabling module streams:
nvidia-driver 535-open

Transaction Summary
=============================================

Is this ok [y/N]: y
    Code block. Example of the result of enabling the NVIDIA Driver module

  5. Check the list of libnvidia-nscq modules.

    Color mode
    dnf list libnvidia-nscq-535 --showduplicates
    dnf list libnvidia-nscq-535 --showduplicates
    Code block. Check the libnvidia-nscq module list

  6. Install libnvidia-nscq.

    Color mode
    sudo dnf install libnvidia-nscq-535-535.183.06-1
    sudo dnf install libnvidia-nscq-535-535.183.06-1
    Code block. libnvidia-nscq installation command

Install the NVSwitch Device Monitoring API (NVSDM) Library (for RHEL)

Reference
The NVSDM Library must be installed for GPU architectures after Blackwell. NVSDM Library is not available for NVIDIA Driver versions 560 and earlier.

  1. Check the NVSDM library module list.

    Color mode
    dnf list libnvsdm --showduplicates
    dnf list libnvsdm --showduplicates
    Code block. Check the NVSDM library module list
    Color mode
    libnvsdm:
  Installed: (none)
  Candidate: 580.105.08-1
  Version table:
     580.105.08-1 600
        600 https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2204/x86_64  Packages
     580.95.05-1 600
        600 https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2204/x86_64  Packages
     580.82.07-1 600
        600 https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2204/x86_64  Packages
     580.65.06-1 600
        600 https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2204/x86_64  Packages
    libnvsdm:
  Installed: (none)
  Candidate: 580.105.08-1
  Version table:
     580.105.08-1 600
        600 https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2204/x86_64  Packages
     580.95.05-1 600
        600 https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2204/x86_64  Packages
     580.82.07-1 600
        600 https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2204/x86_64  Packages
     580.65.06-1 600
        600 https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2204/x86_64  Packages
    Code block. Example of checking the NVSDM library module list

  2. Install libnvsdm.

    Color mode
    sudo dnf install libnvsdm-580.105.08-1
    sudo dnf install libnvsdm-580.105.08-1
    Code Block. Installing the NVSDM library
    Color mode
    Updating Subscription Management repositories.
Last metadata expiration check: 0:08:18 ago on Wed 19 Nov 2025 01:05:28 AM EST.
Dependencies resolved.
=================================================================
Package Architecture Version Repository Size
=================================================================
Installing:
libnvsdm x86_64 580.105.08-1 cuda-rhel8-x86_64 675 k
Installing dependencies:
infiniband-diags x86_64 48.0-1.el8 rhel-8-for-x86_64-baseos-rpms 323 k
libibumad x86_64 48.0-1.el8 rhel-8-for-x86_64-baseos-rpms 34 k

Transaction Summary
=================================================================
Install 3 Packages

Total download size: 1.0 M
Installed size: 3.2 M
Is this ok [y/N]: y
    Updating Subscription Management repositories.
Last metadata expiration check: 0:08:18 ago on Wed 19 Nov 2025 01:05:28 AM EST.
Dependencies resolved.
=================================================================
Package Architecture Version Repository Size
=================================================================
Installing:
libnvsdm x86_64 580.105.08-1 cuda-rhel8-x86_64 675 k
Installing dependencies:
infiniband-diags x86_64 48.0-1.el8 rhel-8-for-x86_64-baseos-rpms 323 k
libibumad x86_64 48.0-1.el8 rhel-8-for-x86_64-baseos-rpms 34 k

Transaction Summary
=================================================================
Install 3 Packages

Total download size: 1.0 M
Installed size: 3.2 M
Is this ok [y/N]: y
    Code block. Example of the NVSDM library installation command output

Install NVIDIA DCGM (for RHEL)

Install the DCGM Exporter in the following order.

  1. DCGM (datacenter-gpu-manager) installation
  2. datacenter-gpu-manager-exporter installation
  3. Enabling and Starting the DCGM Service

Install DCGM (datacenter-gpu-manager) (for RHEL)

Refers to a specific version of the NVIDIA Data Center GPU Manager (DCGM) tool, a package for managing and monitoring NVIDIA data center GPUs. In particular, cuda12 indicates that this management tool is installed for the CUDA 12 version, and datacenter-gpu-manager-4 refers to the 4.x version of DCGM. This tool provides a variety of features, including GPU status monitoring, diagnostics, a warning system, and power/clock management.

  1. Add the CUDA repository to DNF.
    Color mode
    sudo dnf config-manager --add-repo https://developer.download.nvidia.com/compute/cuda/repos/rhel8/x86_64/cuda-rhel8.repo
    sudo dnf config-manager --add-repo https://developer.download.nvidia.com/compute/cuda/repos/rhel8/x86_64/cuda-rhel8.repo
    Code Blocks. Add DNF Repository
  2. Check the CUDA version.
    Color mode
    nvidia-smi | grep CUDA
    nvidia-smi | grep CUDA
    Code block. Check the CUDA version
    Color mode
    | NVIDIA-SMI 535.183.06             Driver Version: 535.183.06     CUDA Version: 12.2 |
    | NVIDIA-SMI 535.183.06             Driver Version: 535.183.06     CUDA Version: 12.2 |
    Code block. Example of CUDA version check results
    Color mode
    CUDA_VERSION=12
    CUDA_VERSION=12
    Code block. CUDA version setting command
  3. Check the module list for datacenter-gpu-manager-cuda.
    Color mode
    dnf list datacenter-gpu-manager-4-cuda${CUDA_VERSION} --showduplicates
    dnf list datacenter-gpu-manager-4-cuda${CUDA_VERSION} --showduplicates
    Code block. Check the datacenter-gpu-manager-cuda module list
    Color mode
    Updating Subscription Management repositories.
Unable to read consumer identity

This system is not registered with an entitlement server. You can use subscription-manager to register.

Last metadata expiration check: 0:00:34 ago on Wed 19 Nov 2025 12:26:56 AM EST.
Available Packages
datacenter-gpu-manager-4-cuda12.x86_64   1:4.0.0-1    cuda-rhel8-x86_64
datacenter-gpu-manager-4-cuda12.x86_64   1:4.1.0-1    cuda-rhel8-x86_64
datacenter-gpu-manager-4-cuda12.x86_64   1:4.1.1-1    cuda-rhel8-x86_64
datacenter-gpu-manager-4-cuda12.x86_64   1:4.2.0-1    cuda-rhel8-x86_64
datacenter-gpu-manager-4-cuda12.x86_64   1:4.2.2-1    cuda-rhel8-x86_64
datacenter-gpu-manager-4-cuda12.x86_64   1:4.2.3-1    cuda-rhel8-x86_64
datacenter-gpu-manager-4-cuda12.x86_64   1:4.2.3-2    cuda-rhel8-x86_64
datacenter-gpu-manager-4-cuda12.x86_64   1:4.3.0-1    cuda-rhel8-x86_64
datacenter-gpu-manager-4-cuda12.x86_64   1:4.3.1-1    cuda-rhel8-x86_64
datacenter-gpu-manager-4-cuda12.x86_64   1:4.4.0-1    cuda-rhel8-x86_64
datacenter-gpu-manager-4-cuda12.x86_64   1:4.4.1-1    cuda-rhel8-x86_64
datacenter-gpu-manager-4-cuda12.x86_64   1:4.4.2-1    cuda-rhel8-x86_64
    Updating Subscription Management repositories.
Unable to read consumer identity

This system is not registered with an entitlement server. You can use subscription-manager to register.

Last metadata expiration check: 0:00:34 ago on Wed 19 Nov 2025 12:26:56 AM EST.
Available Packages
datacenter-gpu-manager-4-cuda12.x86_64   1:4.0.0-1    cuda-rhel8-x86_64
datacenter-gpu-manager-4-cuda12.x86_64   1:4.1.0-1    cuda-rhel8-x86_64
datacenter-gpu-manager-4-cuda12.x86_64   1:4.1.1-1    cuda-rhel8-x86_64
datacenter-gpu-manager-4-cuda12.x86_64   1:4.2.0-1    cuda-rhel8-x86_64
datacenter-gpu-manager-4-cuda12.x86_64   1:4.2.2-1    cuda-rhel8-x86_64
datacenter-gpu-manager-4-cuda12.x86_64   1:4.2.3-1    cuda-rhel8-x86_64
datacenter-gpu-manager-4-cuda12.x86_64   1:4.2.3-2    cuda-rhel8-x86_64
datacenter-gpu-manager-4-cuda12.x86_64   1:4.3.0-1    cuda-rhel8-x86_64
datacenter-gpu-manager-4-cuda12.x86_64   1:4.3.1-1    cuda-rhel8-x86_64
datacenter-gpu-manager-4-cuda12.x86_64   1:4.4.0-1    cuda-rhel8-x86_64
datacenter-gpu-manager-4-cuda12.x86_64   1:4.4.1-1    cuda-rhel8-x86_64
datacenter-gpu-manager-4-cuda12.x86_64   1:4.4.2-1    cuda-rhel8-x86_64
    Code block. Example of the datacenter-gpu-manager-cuda module list
  4. Install datacenter-gpu-manager-cuda.
    Color mode
    sudo dnf install datacenter-gpu-manager-4-cuda${CUDA_VERSION}
    sudo dnf install datacenter-gpu-manager-4-cuda${CUDA_VERSION}
    Code block. Install datacenter-gpu-manager-cuda
    Color mode
    Updating Subscription Management repositories.
Unable to read consumer identity

This system is not registered with an entitlement server. You can use subscription-manager to register.

Last metadata expiration check: 0:07:12 ago on Wed 19 Nov 2025 12:26:56 AM EST.
Dependencies resolved.
==================================================================================================
 Package                                       Architecture   Version     Repository          Size
==================================================================================================
Installing:
 datacenter-gpu-manager-4-cuda12               x86_64         1:4.4.2-1   cuda-rhel8-x86_64   554 M
Installing dependencies:
 datacenter-gpu-manager-4-core                 x86_64         1:4.4.2-1   cuda-rhel8-x86_64   9.9 M
Installing weak dependencies:
 datacenter-gpu-manager-4-proprietary          x86_64         1:4.4.2-1   cuda-rhel8-x86_64   5.3 M
 datacenter-gpu-manager-4-proprietary-cuda12   x86_64         1:4.4.2-1   cuda-rhel8-x86_64   289 M

Transaction Summary
====================================================================================================
Install 4 Packages
...
Is this ok [y/N]: y
    Updating Subscription Management repositories.
Unable to read consumer identity

This system is not registered with an entitlement server. You can use subscription-manager to register.

Last metadata expiration check: 0:07:12 ago on Wed 19 Nov 2025 12:26:56 AM EST.
Dependencies resolved.
==================================================================================================
 Package                                       Architecture   Version     Repository          Size
==================================================================================================
Installing:
 datacenter-gpu-manager-4-cuda12               x86_64         1:4.4.2-1   cuda-rhel8-x86_64   554 M
Installing dependencies:
 datacenter-gpu-manager-4-core                 x86_64         1:4.4.2-1   cuda-rhel8-x86_64   9.9 M
Installing weak dependencies:
 datacenter-gpu-manager-4-proprietary          x86_64         1:4.4.2-1   cuda-rhel8-x86_64   5.3 M
 datacenter-gpu-manager-4-proprietary-cuda12   x86_64         1:4.4.2-1   cuda-rhel8-x86_64   289 M

Transaction Summary
====================================================================================================
Install 4 Packages
...
Is this ok [y/N]: y
    Code block. Example of the datacenter-gpu-manager-cuda installation result

Install datacenter-gpu-manager-exporter (for RHEL)

A tool that collects various GPU metrics, such as GPU utilization, memory usage, temperature, and power consumption, based on the NVIDIA Data Center GPU Manager (DCGM) and exposes them for use in monitoring systems like Prometheus.

  1. Add the CUDA repository to DNF. 1. (If you have already run this command, proceed to the next step.)

    Color mode
    sudo dnf config-manager --add-repo https://developer.download.nvidia.com/compute/cuda/repos/rhel8/x86_64/cuda-rhel8.repo
    sudo dnf config-manager --add-repo https://developer.download.nvidia.com/compute/cuda/repos/rhel8/x86_64/cuda-rhel8.repo
    Code Block. Add DNF Repository

  2. Check the CUDA version. 2. (If you have already run this command, proceed to the next step.)

    Color mode
    nvidia-smi | grep CUDA
    nvidia-smi | grep CUDA
    Code block. Check the CUDA version
    Color mode
    | NVIDIA-SMI 535.183.06             Driver Version: 535.183.06     CUDA Version: 12.2 |
    | NVIDIA-SMI 535.183.06             Driver Version: 535.183.06     CUDA Version: 12.2 |
    Code block. Example of CUDA version check results
    Color mode
    CUDA_VERSION=12
    CUDA_VERSION=12
    Code block. CUDA version setting command

  3. Check the module list for datacenter-gpu-manager-exporter.

    Color mode
    dnf list datacenter-gpu-manager-exporter --showduplicates
    dnf list datacenter-gpu-manager-exporter --showduplicates
    Code block. Check the module list for datacenter-gpu-manager-exporter
    Color mode
    Updating Subscription Management repositories.
Unable to read consumer identity

This system is not registered with an entitlement server. You can use subscription-manager to register.

Last metadata expiration check: 0:02:11 ago on Wed 19 Nov 2025 12:26:56 AM EST.
Available Packages
datacenter-gpu-manager-exporter.x86_64   4.0.1-1   cuda-rhel8-x86_64
datacenter-gpu-manager-exporter.x86_64   4.1.0-1   cuda-rhel8-x86_64
datacenter-gpu-manager-exporter.x86_64   4.1.1-1   cuda-rhel8-x86_64
datacenter-gpu-manager-exporter.x86_64   4.1.3-1   cuda-rhel8-x86_64
datacenter-gpu-manager-exporter.x86_64   4.5.0-1   cuda-rhel8-x86_64
datacenter-gpu-manager-exporter.x86_64   4.5.1-1   cuda-rhel8-x86_64
datacenter-gpu-manager-exporter.x86_64   4.5.2-1   cuda-rhel8-x86_64
datacenter-gpu-manager-exporter.x86_64   4.6.0-1   cuda-rhel8-x86_64
    Updating Subscription Management repositories.
Unable to read consumer identity

This system is not registered with an entitlement server. You can use subscription-manager to register.

Last metadata expiration check: 0:02:11 ago on Wed 19 Nov 2025 12:26:56 AM EST.
Available Packages
datacenter-gpu-manager-exporter.x86_64   4.0.1-1   cuda-rhel8-x86_64
datacenter-gpu-manager-exporter.x86_64   4.1.0-1   cuda-rhel8-x86_64
datacenter-gpu-manager-exporter.x86_64   4.1.1-1   cuda-rhel8-x86_64
datacenter-gpu-manager-exporter.x86_64   4.1.3-1   cuda-rhel8-x86_64
datacenter-gpu-manager-exporter.x86_64   4.5.0-1   cuda-rhel8-x86_64
datacenter-gpu-manager-exporter.x86_64   4.5.1-1   cuda-rhel8-x86_64
datacenter-gpu-manager-exporter.x86_64   4.5.2-1   cuda-rhel8-x86_64
datacenter-gpu-manager-exporter.x86_64   4.6.0-1   cuda-rhel8-x86_64
    Code block. Example of the datacenter-gpu-manager-exporter module list

  4. Install datacenter-gpu-manager-cuda. dcgm-exporter 4.5.X requires glibc 2.34 or later. Since RHEL9 provides glibc 2.34, install it by specifying version 4.1.3-1.

    Color mode
    sudo dnf install datacenter-gpu-manager-exporter-4.1.3-1
    sudo dnf install datacenter-gpu-manager-exporter-4.1.3-1
    Code block. Install datacenter-gpu-manager-cuda
    Color mode
    Updating Subscription Management repositories.
Unable to read consumer identity

This system is not registered with an entitlement server. You can use subscription-manager to register.

Last metadata expiration check: 0:07:12 ago on Wed 19 Nov 2025 12:26:56 AM EST.
Dependencies resolved.
====================================================================================================
 Package                                       Architecture   Version     Repository          Size
====================================================================================================
Installing:
 datacenter-gpu-manager-exporter               x86_64         4.1.3-1     cuda-rhel8-x86_64   26 M

...
Is this ok [y/N]: y
    Updating Subscription Management repositories.
Unable to read consumer identity

This system is not registered with an entitlement server. You can use subscription-manager to register.

Last metadata expiration check: 0:07:12 ago on Wed 19 Nov 2025 12:26:56 AM EST.
Dependencies resolved.
====================================================================================================
 Package                                       Architecture   Version     Repository          Size
====================================================================================================
Installing:
 datacenter-gpu-manager-exporter               x86_64         4.1.3-1     cuda-rhel8-x86_64   26 M

...
Is this ok [y/N]: y
    Code block. Example of the datacenter-gpu-manager-cuda installation result

  5. Check the DCGM Exporter configuration file.

    Color mode
    cat /usr/lib/systemd/system/nvidia-dcgm-exporter.service | grep ExecStart
    cat /usr/lib/systemd/system/nvidia-dcgm-exporter.service | grep ExecStart
    Code block. datacenter-gpu-manager-exporter configuration file
    Color mode
    ExecStart=/usr/bin/dcgm-exporter -f /etc/dcgm-exporter/default-counters.csv
    ExecStart=/usr/bin/dcgm-exporter -f /etc/dcgm-exporter/default-counters.csv
    Code block. Example of the datacenter-gpu-manager-exporter configuration file

  6. Review the configuration provided when installing the DCGM Exporter, remove the # from the required metrics, and add # to the unnecessary metrics.

    Color mode
    sudo vi /etc/dcgm-exporter/default-counters.csv
## Example ##
...
DCGM_FI_PROF_PIPE_TENSOR_ACTIVE, gauge, Ratio of cycles the tensor (HMMA) pipe is active.
DCGM_FI_PROF_DRAM_ACTIVE,        gauge, Ratio of cycles during which the device memory interface is active, sending or receiving data.
# DCGM_FI_PROF_PIPE_FP64_ACTIVE, gauge, Ratio of cycles during which the fp64 pipes are active.
# DCGM_FI_PROF_PIPE_FP32_ACTIVE,   gauge, Ratio of cycles the fp32 pipes are active.
...
    sudo vi /etc/dcgm-exporter/default-counters.csv
## Example ##
...
DCGM_FI_PROF_PIPE_TENSOR_ACTIVE, gauge, Ratio of cycles the tensor (HMMA) pipe is active.
DCGM_FI_PROF_DRAM_ACTIVE,        gauge, Ratio of cycles during which the device memory interface is active, sending or receiving data.
# DCGM_FI_PROF_PIPE_FP64_ACTIVE, gauge, Ratio of cycles during which the fp64 pipes are active.
# DCGM_FI_PROF_PIPE_FP32_ACTIVE,   gauge, Ratio of cycles the fp32 pipes are active.
...
    Code block. Example datacenter-gpu-manager-exporter metric configuration

Reference
For the metrics that can be collected by the GPU DCGM Exporter and how to configure them, refer to DCGM Exporter metrics.
Warning
Metric collection through the ServiceWatch Agent is billed separately from default metrics because it is categorized as custom metrics. To avoid excessive charges, remove or disable any unnecessary metric collection.

Enable and Start the DCGM Service (for RHEL)

  1. Activate and start the nvdia-dcgm service.

    Color mode
    sudo systemctl enable --now nvidia-dcgm
    sudo systemctl enable --now nvidia-dcgm
    Code block. Commands to enable and start the nvdia-dcgm service

  2. Activate and start the nvdia-dcgm-exporter service.

    Color mode
    sudo systemctl enable --now nvidia-dcgm-exporter
    sudo systemctl enable --now nvidia-dcgm-exporter
    Code block. Commands to enable and start the nvdia-dcgm-exporter service

Notice
If you have completed the DCGM Exporter configuration, you must install the Open Telemetry Collector provided by ServiceWatch to complete the ServiceWatch Agent configuration.
For details, see ServiceWatch > Using ServiceWatch Agent.

DCGM Exporter Metrics

Key DCGM Exporter Metrics

The following are the key GPU metrics provided by the DCGM Exporter.

CategoryDCGM FieldPrometheus Metric TypeSummary
ClocksDCGM_FI_DEV_SM_CLOCKgaugeSM clock frequency (in MHz)
ClocksDCGM_FI_DEV_MEM_CLOCKgaugeMemory clock frequency (in MHz)
TemperatureDCGM_FI_DEV_GPU_TEMPgaugeGPU temperature (in °C)
PowerDCGM_FI_DEV_POWER_USAGEgaugePower draw (in W)
UtilizationDCGM_FI_DEV_GPU_UTILgaugeGPU utilization (in %)
UtilizationDCGM_FI_DEV_MEM_COPY_UTILgaugeMemory utilization (in %)
Memory UsageDCGM_FI_DEV_FB_FREEgaugeFrame buffer memory free (in MiB)
Memory UsageDCGM_FI_DEV_FB_USEDgaugeFrame buffer memory used (in MiB)
NvlinkDCGM_FI_DEV_NVLINK_BANDWIDTH_TOTAL (8 GPUs only)counterTotal number of NVLink bandwidth counters for all lanes
Table. Key GPU metrics provided by the DCGM Exporter

DCGM Exporter Metric Collection Configuration

For the default metrics configured in DCGM Exporter, refer to DCGM Exporter > Default Metrics.

  • To configure additional metrics beyond the default settings, remove the # in default-counters.csv.
  • To exclude a metric from collection among the default metrics, add # or delete the corresponding entry.
Color mode
# Format
# If a line starts with a '#', it is considered a comment.
# DCGM FIELD, Prometheus metric type, help message

# Clocks
DCGM_FI_DEV_SM_CLOCK, gauge, SM clock frequency (in MHz).
DCGM_FI_DEV_MEM_CLOCK, gauge, Memory clock frequency (in MHz).

# Temperature
DCGM_FI_DEV_MEMORY_TEMP, gauge, Memory temperature (in C).
DCGM_FI_DEV_GPU_TEMP,    gauge, GPU temperature (in C).

# Power
DCGM_FI_DEV_POWER_USAGE,              gauge, Power draw (in W).
DCGM_FI_DEV_TOTAL_ENERGY_CONSUMPTION, counter, Total energy consumption since boot (in mJ).

# PCIE
# DCGM_FI_PROF_PCIE_TX_BYTES, counter, Total number of bytes transmitted through PCIe TX via NVML.
# DCGM_FI_PROF_PCIE_RX_BYTES, counter, Total number of bytes received through PCIe RX via NVML.
...
# Format
# If a line starts with a '#', it is considered a comment.
# DCGM FIELD, Prometheus metric type, help message

# Clocks
DCGM_FI_DEV_SM_CLOCK, gauge, SM clock frequency (in MHz).
DCGM_FI_DEV_MEM_CLOCK, gauge, Memory clock frequency (in MHz).

# Temperature
DCGM_FI_DEV_MEMORY_TEMP, gauge, Memory temperature (in C).
DCGM_FI_DEV_GPU_TEMP,    gauge, GPU temperature (in C).

# Power
DCGM_FI_DEV_POWER_USAGE,              gauge, Power draw (in W).
DCGM_FI_DEV_TOTAL_ENERGY_CONSUMPTION, counter, Total energy consumption since boot (in mJ).

# PCIE
# DCGM_FI_PROF_PCIE_TX_BYTES, counter, Total number of bytes transmitted through PCIe TX via NVML.
# DCGM_FI_PROF_PCIE_RX_BYTES, counter, Total number of bytes received through PCIe RX via NVML.
...
Code block. Example of the default-counters.csv configuration
Use NVSwitch on GPU Server
Release Note