How-to guides

• 1: Manage Image
• 2: Manage Keypair
• 3: Use Multi-instance GPU on GPU Server
• 4: Use NVSwitch on GPU Server
• 5: Install ServiceWatch Agent

1 - Manage Image

Users can create the service by entering the required information for the Image service within the GPU Server service and selecting detailed options through the Samsung Cloud Platform Console.

Create Image

You can create an image of a running GPU Server. To create an image of a GPU Server, please refer to Image Creation.

Check Image detailed information

Image service allows you to view and edit the full resource list and detailed information. The Image Details page consists of Detailed Information, Tags, Operation History tabs.

To view detailed information of the Image service, follow these steps.

1. Click the All Services > Compute > GPU Server menu. You will be taken to the Service Home page of GPU Server.
2. On the Service Home page, click the Image menu. You will be taken to the Image list page.
3. On the Image List page, click the resource to view detailed information. You will be taken to the Image Detail page.
   • Image Details page displays status information and additional feature information, and consists of Details, Tags, Activity Log tabs.

     Category	Detailed description
     Image status	Status of user-created Image Active: Available state Queued: Image has been uploaded and is waiting for processing after creation Importing: Image has been uploaded and is currently being processed after creation
     Share with another account	Image can be shared with another Account The Image’s Visibility must be set to Shared in order to be shared with another Account
     Delete image	Button to delete the Image Once the Image is deleted, it cannot be restored

     Table. GPU Server Image status information and additional features

Detailed Information

Image list page lets you view detailed information of the selected resource and modify it if necessary.

tag

On the Image List page, you can view the tag information of the selected resource and add, modify, or delete it.

Job History

You can view the operation history of the selected resource on the Image List page.

Image Resource Management

Describes the control and management functions of the generated Image.

Share to another Account

To share an Image with another Account, follow the steps below.

1. Log in to the account to be shared and click the All Services > Compute > GPU Server menu. Go to the GPU Server’s Service Home page.
2. On the Service Home page, click the Image menu. You will be taken to the Image List page.
3. On the Image List page, click the Image you want to control. You will be taken to the Image Details page.
4. Click the Share to another Account button. Navigate to the Share image to another Account page.
   • Share with another Account feature allows you to share an Image with another Account. To share an Image with another Account, the Image’s Visibility must be Shared.
5. Share image to another Account page, enter the required information, and click the Done button.

   Category	Required	Detailed description
   image name	-	Name of the image to share Input not allowed
   Image ID	-	Shareable image ID Input not allowed
   Shared Account ID	Required	Enter another Account ID to share English letters, numbers, special characters- within 64 characters

   Table. Required input fields for sharing images to another Account
6. You can view the information in the sharing status of the Image Details page.
   • When the request is first made, the status is Pending, and it changes to Accepted once approval is completed by the account receiving the share.

Receive sharing from another Account

To receive an Image shared from another Account, follow these steps.

1. Log in to the account to be shared and click the All Services > Compute > GPU Server menu. Navigate to the GPU Server’s Service Home page.
2. On the Service Home page, click the Image menu. You will be taken to the Image List page.
3. On the Image List page, click the Receive Image Share button. You will be taken to the Receive Image Share popup.
4. Receive Image Sharing In the popup window, enter the Image’s resource ID you want to receive, and click the Confirm button.
5. When image sharing is complete, you can view the shared Image in the Image list.

Delete Image

You can delete unused Images. However, since an Image cannot be recovered after deletion, you should carefully consider the impact before performing the deletion.

To delete the Image, follow these steps.

1. Click the All Services > Compute > GPU Server menu. Go to the GPU Server’s Service Home page.
2. On the Service Home page, click the Image menu. You will be taken to the Image List page.
3. Image list page, select the resource to delete, and click the Delete button.
   • On the Image List page, select multiple Image check boxes and click the Delete button at the top of the resource list.
4. After deletion is complete, verify on the Image list page that the resource has been removed.

2 - Manage Keypair

Users can create the service by entering the required Keypair information within the GPU Server service and selecting detailed options through the Samsung Cloud Platform Console.

Create a Keypair

You can create and use the Keypair service while using the GPU Server service in the Samsung Cloud Platform Console.

To create a keypair, follow these steps.

1. Click the All Services > Compute > GPU Server menu. You will be taken to the Service Home page of GPU Server.
2. On the Service Home page, click the Keypair menu. You will be taken to the Keypair List page.
3. On the Keypair List page, click the Keypair Create button. You will be taken to the Keypair Create page.
   • Enter the required information in the Service Information Input area.

     Category	Required	Detailed description
     Keypair name	Required	Enter the name of the Keypair to create using English letters, numbers, spaces, and special characters (-, _) within 255 characters
     Keypair type	Required	ssh

     Table. Keypair service information input fields
   • Additional Information Input area, please enter or select the required information.

     Category	Required status	Detailed description
     tag	Selection	Add Tag Up to 50 can be added per resource After clicking the Add Tag button, enter or select Key, Value values

     Table. Keypair additional information input fields
     Caution

     • After creation is complete, you can download the Key only once. Since reissuance is not possible, make sure it has been downloaded.
     • Store the downloaded Private Key in a safe place.
4. Check the input information and click the Create button.
   • After creation is complete, check the created resources on the Keypair List page.

View detailed information of the Keypair

The Keypair service allows you to view and edit the full resource list and detailed information. Keypair Details page consists of Details, Tags, Activity Log tabs.

To view detailed information about a keypair, follow these steps.

1. Click the All Services > Compute > GPU Server menu. You will be taken to the Service Home page of GPU Server.
2. On the Service Home page, click the Keypair menu. You will be taken to the Keypair List page.
3. On the Keypair List page, click the resource to view its details. You will be taken to the Keypair Details page.
   • Keypair Details page displays status information and additional feature information, and consists of Details, Tags, Activity Log tabs.

Detailed Information

Keypair List page allows you to view detailed information of the selected resource and edit the information if needed.

tag

On the Keypair List page, you can view the tag information of the selected resource, and add, modify, or delete it.

Job History

On the Keypair List page, you can view the operation history of the selected resource.

Keypair Resource Management

Describes the control and management functions of a keypair.

Get public key

To retrieve the public key, follow these steps.

1. Click the All Services > Compute > GPU Server menu. You will be taken to the Service Home page of GPU Server.

2. On the Service Home page, click the Keypair menu. You will be taken to the Keypair List page.

3. On the Keypair List page, click the More button at the top and then click the Import Public Key button. You will be taken to the Import Public Key page.

   • Enter or select the required information in the Required Information Input area.

     Category	Required	Detailed description
     Keypair name	Required	Name of the Keypair to create
     Keypair type	Required	ssh
     public key	Required	Enter public key Load file: Select the Attach file button to attach the public key file Only files with the following extension (.pem) can be attached Enter public key: Paste the copied public key value The public key value can be copied from the Keypair Details page

     Table. Required input fields for retrieving the public key

4. Review the entered information and click the Complete button.

   • Once creation is complete, check the created resources on the Keypair List page.

Delete Keypair

You can delete unused Keypairs. However, once a Keypair is deleted it cannot be recovered, so please review the impact thoroughly beforehand before proceeding with deletion.

To delete a keypair, follow these steps.

1. Click the All Services > Compute > GPU Server menu. Go to the Service Home page of GPU Server.
2. On the Service Home page, click the Keypair menu. You will be taken to the Keypair List page.
3. On the Keypair List page, select the resource to delete, and click the Delete button.

• On the Keypair List page, select multiple Keypair check boxes and click the Delete button at the top of the resource list.

4. After deletion is complete, check the Keypair List page to confirm that the resource has been removed.

3 - Use Multi-instance GPU on GPU Server

After creating a GPU Server, you can enable the MIG (Multi-instance GPU) feature on the GPU Server’s VM (Guest OS) and create an instance for use.

NVIDIA Multi-instance GPU Introduction

NVIDIA Multi-instance GPU (hereafter referred to as MIG) supports safely partitioning a GPU into GPU instances and running CUDA applications starting with the NVIDIA Ampere architecture. Through this, multiple users can each utilize different GPU resources to achieve optimal GPU utilization. This feature is especially useful for workloads that do not fully utilize the GPU’s computing capacity, and users can run multiple workloads in parallel to maximize utilization.

Using Multi-instance GPU feature

To use the MIG feature, create an NVIDIA GPU Server on the Samsung Cloud Platform, then enable and disable MIG. The order of applying and removing MIG is as follows.

Applying and Using MIG

After activating the MIG and creating an Instance to assign tasks, the tasks proceed in the following order.

Activate MIG

1. Check the GPU status on the VM Instance (GuestOS) before applying MIG.

   • Check whether MIG mode is Disabled.
     Sample Code Download

     Copy Code

     Color mode

     $ nvidia-smi
     Mon Sep 27 08:37:08 2021
     +-----------------------------------------------------------------------------+
     | NVIDIA-SMI 470.57.02    Driver Version: 470.57.02    CUDA Version: 11.4 |
     | -------------------------------+----------------------+---------------------- |
     | GPU  Name        Persistence-M | Bus-Id        Disp.A | Volatile Uncorr. ECC |
     | Fan  Temp  Perf  Pwr:Usage/Cap | Memory-Usage | GPU-Util  Compute M. |
     |  |  | MIG M. |
     | ===============================+======================+====================== |
     | 0  NVDIA A100-SXM...  Off | 00000000:05:00.0 Off | 0 |
     | N/A   32C   P0    59W / 400W | 0MiB / 81251MiB | 0%      Default |
     |  |  | Disabled |
     +-------------------------------+----------------------+----------------------+
     
     +-----------------------------------------------------------------------------+
     | Processes: |
     | GPU   GI   CI       PID   Type   Process name                   GPU Memory |
     | ID   ID                                                   Usage |
     | ============================================================================= |
     | No running processes found |
     +-----------------------------------------------------------------------------+

     $ nvidia-smi
     Mon Sep 27 08:37:08 2021
     +-----------------------------------------------------------------------------+
     | NVIDIA-SMI 470.57.02    Driver Version: 470.57.02    CUDA Version: 11.4 |
     | -------------------------------+----------------------+---------------------- |
     | GPU  Name        Persistence-M | Bus-Id        Disp.A | Volatile Uncorr. ECC |
     | Fan  Temp  Perf  Pwr:Usage/Cap | Memory-Usage | GPU-Util  Compute M. |
     |  |  | MIG M. |
     | ===============================+======================+====================== |
     | 0  NVDIA A100-SXM...  Off | 00000000:05:00.0 Off | 0 |
     | N/A   32C   P0    59W / 400W | 0MiB / 81251MiB | 0%      Default |
     |  |  | Disabled |
     +-------------------------------+----------------------+----------------------+
     
     +-----------------------------------------------------------------------------+
     | Processes: |
     | GPU   GI   CI       PID   Type   Process name                   GPU Memory |
     | ID   ID                                                   Usage |
     | ============================================================================= |
     | No running processes found |
     +-----------------------------------------------------------------------------+

     Sample Code Download

     Copy Code

     Color mode

     $ nvidia-smi –L
     GPU 0: NVIDIA A100-SXM-80GB (UUID: GPU-c956838f-494a-92b2-6818-56eb28fe25e0)

     $ nvidia-smi –L
     GPU 0: NVIDIA A100-SXM-80GB (UUID: GPU-c956838f-494a-92b2-6818-56eb28fe25e0)

2. Enable MIG (Enable) for each GPU on the VM Instance (GuestOS) and reboot the VM Instance.

   Sample Code Download

   Copy Code

   Color mode

   $ nvidia-smi –I 0 –mig 1
   Enabled MIG mode for GPU 00000000:05:00.0
   All done.
   
   # reboot

   $ nvidia-smi –I 0 –mig 1
   Enabled MIG mode for GPU 00000000:05:00.0
   All done.
   
   # reboot

Warning: MIG mode is in pending enable state for GPU 00000000:05:00.0: In use by another client. 00000000:05:00.0 is currently being used by one or more other processes (e.g. CUDA application or a monitoring application such as another instance of nvidia-smi).

3. Check the GPU status after applying MIG on the VM Instance(GuestOS).
   • Check whether MIG mode is Enabled.
     Sample Code Download

     Copy Code

     Color mode

     $ nvidia-smi
     Mon Sep 27 09:44:33 2021
     +-----------------------------------------------------------------------------+
     | NVIDIA-SMI 470.57.02    Driver Version: 470.57.02    CUDA Version: 11.4 |
     | -------------------------------+----------------------+---------------------- |
     | GPU  Name        Persistence-M | Bus-Id        Disp.A | Volatile Uncorr. ECC |
     | Fan  Temp  Perf  Pwr:Usage/Cap | Memory-Usage | GPU-Util  Compute M. |
     |  |  | MIG M. |
     | ===============================+======================+====================== |
     | 0  NVDIA A100-SXM...  Off | 00000000:05:00.0 Off | On |
     | N/A   32C   P0    59W / 400W | 0MiB / 81251MiB | 0%      Default |
     |  |  | Enabled |
     +-------------------------------+----------------------+----------------------+
     +-----------------------------------------------------------------------------+
     | MIG devices: |
     +-----------------------------------------------------------------------------+
     | GPU  GI  CI  MIG | Memory-Usage | Vol | Shared |
     | ID  ID  Dev | BAR1-Usage | SM     Unc | CE  ENC  DEC  OFA  JPG |
     |  |  | ECC |  |
     | ============================================================================= |
     | No MIG devices found |
     +-----------------------------------------------------------------------------+
     +-----------------------------------------------------------------------------+
     | Processes: |
     | GPU   GI   CI       PID   Type   Process name                   GPU Memory |
     | ID   ID                                                   Usage |
     | ============================================================================= |
     | No running processes found |
     +-----------------------------------------------------------------------------+

     $ nvidia-smi
     Mon Sep 27 09:44:33 2021
     +-----------------------------------------------------------------------------+
     | NVIDIA-SMI 470.57.02    Driver Version: 470.57.02    CUDA Version: 11.4 |
     | -------------------------------+----------------------+---------------------- |
     | GPU  Name        Persistence-M | Bus-Id        Disp.A | Volatile Uncorr. ECC |
     | Fan  Temp  Perf  Pwr:Usage/Cap | Memory-Usage | GPU-Util  Compute M. |
     |  |  | MIG M. |
     | ===============================+======================+====================== |
     | 0  NVDIA A100-SXM...  Off | 00000000:05:00.0 Off | On |
     | N/A   32C   P0    59W / 400W | 0MiB / 81251MiB | 0%      Default |
     |  |  | Enabled |
     +-------------------------------+----------------------+----------------------+
     +-----------------------------------------------------------------------------+
     | MIG devices: |
     +-----------------------------------------------------------------------------+
     | GPU  GI  CI  MIG | Memory-Usage | Vol | Shared |
     | ID  ID  Dev | BAR1-Usage | SM     Unc | CE  ENC  DEC  OFA  JPG |
     |  |  | ECC |  |
     | ============================================================================= |
     | No MIG devices found |
     +-----------------------------------------------------------------------------+
     +-----------------------------------------------------------------------------+
     | Processes: |
     | GPU   GI   CI       PID   Type   Process name                   GPU Memory |
     | ID   ID                                                   Usage |
     | ============================================================================= |
     | No running processes found |
     +-----------------------------------------------------------------------------+

     Sample Code Download

     Copy Code

     Color mode

     $ nvidia-smi –L
     GPU 0: NVIDIA A100-SXM-80GB (UUID: GPU-c956838f-494a-92b2-6818-56eb28fe25e0)

     $ nvidia-smi –L
     GPU 0: NVIDIA A100-SXM-80GB (UUID: GPU-c956838f-494a-92b2-6818-56eb28fe25e0)

GPU Instance creation

If you have enabled MIG and verified its status, you can create a GPU Instance.

1. Check the list of MIG GPU Instance profiles that can be created.
   Sample Code Download

   Copy Code

   Color mode

   $ nvidia-smi mig -i [GPU ID] -lgip

   $ nvidia-smi mig -i [GPU ID] -lgip

$ nvidia-smi mig -i 0 -lgip
+-----------------------------------------------------------------------------+
| GPU instance profiles: |
| GPU   Name             ID    Instances   Memory     P2P    SM    DEC   ENC |
| Free/Total   GiB              CE    JPEG  OFA |
| ============================================================================= |
| 0 MIG 1g.10gb        19    7/7         9.50       No     14     0     0 |
| 1     0     0 |
+-----------------------------------------------------------------------------+
| 0 MIG 1g.10gb+me     20    1/1         9.50       No     14     0     0 |
| 1     1     1 |
+-----------------------------------------------------------------------------+
| 0 MIG 2g.20gb        14    3/3         19.50      No     28     1     0 |
| 2     0     0 |
+-----------------------------------------------------------------------------+
| 0 MIG 3g.40gb         9    2/2         39.50      No     42     2     0 |
| 3     0     0 |
+-----------------------------------------------------------------------------+
| 0 MIG 4g.40gb         5    1/1         39.50      No     56     2     0 |
| 4     0     0 |
+-----------------------------------------------------------------------------+
| 0 MIG 7g.80gb         0    1/1         79.25      No     98     0     0 |
| 7     1     1 |
+-----------------------------------------------------------------------------+

$ nvidia-smi mig -i 0 -lgip
+-----------------------------------------------------------------------------+
| GPU instance profiles: |
| GPU   Name             ID    Instances   Memory     P2P    SM    DEC   ENC |
| Free/Total   GiB              CE    JPEG  OFA |
| ============================================================================= |
| 0 MIG 1g.10gb        19    7/7         9.50       No     14     0     0 |
| 1     0     0 |
+-----------------------------------------------------------------------------+
| 0 MIG 1g.10gb+me     20    1/1         9.50       No     14     0     0 |
| 1     1     1 |
+-----------------------------------------------------------------------------+
| 0 MIG 2g.20gb        14    3/3         19.50      No     28     1     0 |
| 2     0     0 |
+-----------------------------------------------------------------------------+
| 0 MIG 3g.40gb         9    2/2         39.50      No     42     2     0 |
| 3     0     0 |
+-----------------------------------------------------------------------------+
| 0 MIG 4g.40gb         5    1/1         39.50      No     56     2     0 |
| 4     0     0 |
+-----------------------------------------------------------------------------+
| 0 MIG 7g.80gb         0    1/1         79.25      No     98     0     0 |
| 7     1     1 |
+-----------------------------------------------------------------------------+

2. After creating a MIG GPU Instance, check it.

   • Create GPU Instance

     Sample Code Download

     Copy Code

     Color mode

     $ nvidia-smi mig -i [GPU ID] -cgi [Profile ID]

     $ nvidia-smi mig -i [GPU ID] -cgi [Profile ID]

     Sample Code Download

     Copy Code

     Color mode

     $ nvidia-smi mig -i 0 -cgi 0
     Successfully created GPU instance ID 0 on GPU 0 using profile MIG 7g.80gb (ID 0)

     $ nvidia-smi mig -i 0 -cgi 0
     Successfully created GPU instance ID 0 on GPU 0 using profile MIG 7g.80gb (ID 0)

   • Check GPU Instance

     Sample Code Download

     Copy Code

     Color mode

     $ nvidia-smi mig -i [GPU ID] -lgi

     $ nvidia-smi mig -i [GPU ID] -lgi

     Sample Code Download

     Copy Code

     Color mode

     $ nvidia-smi mig -i 0 -lgi
     +--------------------------------------------------------+
     | GPU instances: |
     | GPU   Name               Profile  Instance  Placement |
     | ID       ID      Start:Size |
     | ======================================================== |
     | 0  MIG 7g.80gb            0        0         0:8 |
     +--------------------------------------------------------+

     $ nvidia-smi mig -i 0 -lgi
     +--------------------------------------------------------+
     | GPU instances: |
     | GPU   Name               Profile  Instance  Placement |
     | ID       ID      Start:Size |
     | ======================================================== |
     | 0  MIG 7g.80gb            0        0         0:8 |
     +--------------------------------------------------------+

Compute Instance creation

If you have created a GPU Instance, you can create a Compute Instance.

1. Check the MIG Compute Instance profiles you can create.

   Sample Code Download

   Copy Code

   Color mode

   $ nvidia-smi mig -i [GPU ID] -gi [GPU Instance ID] -lcip

   $ nvidia-smi mig -i [GPU ID] -gi [GPU Instance ID] -lcip

   Sample Code Download

   Copy Code

   Color mode

   $ nvidia-smi mig -i 0 -gi 0 -lcip
   +---------------------------------------------------------------------------------+
   | Compute instance profiles: |
   | GPU     GPU     Name            Profile  Instances   Exclusive      Shared |
   | GPU   Instance                     ID    Free/Total     SM       DEC  ENC  OFA |
   | ID                                                       CE   JPEG |
   | ================================================================================= |
   | 0      0      MIG 1c.7g.80gb     0      7/5           14       5    0    1 |
   | 7    1 |
   +---------------------------------------------------------------------------------+
   | 0      0      MIG 2c.7g.80gb     1      3/3           28       5    0    1 |
   | 7    1 |
   +---------------------------------------------------------------------------------+
   | 0      0      MIG 3c.7g.80gb     2      2/2           42       5    0    1 |
   | 7    1 |
   +---------------------------------------------------------------------------------+
   | 0      0      MIG 4c.7g.80gb     3      1/1           56       5    0    1 |
   | 7    1 |
   +---------------------------------------------------------------------------------+
   | 0      0      MIG 7g.80gb        4*     1/1           98       5    0    1 |
   | 7    1 |
   +---------------------------------------------------------------------------------+

   $ nvidia-smi mig -i 0 -gi 0 -lcip
   +---------------------------------------------------------------------------------+
   | Compute instance profiles: |
   | GPU     GPU     Name            Profile  Instances   Exclusive      Shared |
   | GPU   Instance                     ID    Free/Total     SM       DEC  ENC  OFA |
   | ID                                                       CE   JPEG |
   | ================================================================================= |
   | 0      0      MIG 1c.7g.80gb     0      7/5           14       5    0    1 |
   | 7    1 |
   +---------------------------------------------------------------------------------+
   | 0      0      MIG 2c.7g.80gb     1      3/3           28       5    0    1 |
   | 7    1 |
   +---------------------------------------------------------------------------------+
   | 0      0      MIG 3c.7g.80gb     2      2/2           42       5    0    1 |
   | 7    1 |
   +---------------------------------------------------------------------------------+
   | 0      0      MIG 4c.7g.80gb     3      1/1           56       5    0    1 |
   | 7    1 |
   +---------------------------------------------------------------------------------+
   | 0      0      MIG 7g.80gb        4*     1/1           98       5    0    1 |
   | 7    1 |
   +---------------------------------------------------------------------------------+

2. Create and verify a MIG Compute Instance.

   • MIG Compute Instance creation
     Sample Code Download

     Copy Code

     Color mode

     $ nvidia-smi mig -i [GPU ID] -gi [GPU Instance ID] -cci [Compute Profile ID]

     $ nvidia-smi mig -i [GPU ID] -gi [GPU Instance ID] -cci [Compute Profile ID]

     Sample Code Download

     Copy Code

     Color mode

     $ nvidia-smi mig -i 0 -gi 0 -cci 4
     Successfully created compute instance ID 0 on GPU instance ID 0 using profile MIG 7g.80gb(ID 4)

     $ nvidia-smi mig -i 0 -gi 0 -cci 4
     Successfully created compute instance ID 0 on GPU instance ID 0 using profile MIG 7g.80gb(ID 4)

   • Check MIG Compute Instance
     Sample Code Download

     Copy Code

     Color mode

     $ nvidia-smi mig -i [GPU ID] -gi [GPU Instance ID] –lci

     $ nvidia-smi mig -i [GPU ID] -gi [GPU Instance ID] –lci

     Sample Code Download

     Copy Code

     Color mode

     $ nvidia-smi mig -i 0 -gi 0 –lci
     +-----------------------------------------------------------------+
     | Compute instance profiles: |
     | GPU     GPU     Name            Profile  Instances   Placement |
     | GPU   Instance                     ID      ID        Start:Size |
     | ID |
     | ================================================================= |
     | 0      0      MIG 7g.80gb         4       0            0:7 |
     +-----------------------------------------------------------------+

     $ nvidia-smi mig -i 0 -gi 0 –lci
     +-----------------------------------------------------------------+
     | Compute instance profiles: |
     | GPU     GPU     Name            Profile  Instances   Placement |
     | GPU   Instance                     ID      ID        Start:Size |
     | ID |
     | ================================================================= |
     | 0      0      MIG 7g.80gb         4       0            0:7 |
     +-----------------------------------------------------------------+

     Sample Code Download

     Copy Code

     Color mode

     $ nvidia-smi –L
     GPU 0: NVIDIA A100-SXM-80GB (UUID: GPU-c956838f-494a-92b2-6818-56eb28fe25e0)
       MIG 7g.80gb     Device  0: (UUID: MIG-53e20040-758b-5ecb-948e-c626d03a9a32)

     $ nvidia-smi –L
     GPU 0: NVIDIA A100-SXM-80GB (UUID: GPU-c956838f-494a-92b2-6818-56eb28fe25e0)
       MIG 7g.80gb     Device  0: (UUID: MIG-53e20040-758b-5ecb-948e-c626d03a9a32)

     Sample Code Download

     Copy Code

     Color mode

     $ nvidia-smi
     Mon Sep 27 09:52:17 2021
     +-----------------------------------------------------------------------------+
     | NVIDIA-SMI 470.57.02    Driver Version: 470.57.02    CUDA Version: 11.4 |
     | -------------------------------+----------------------+---------------------- |
     | GPU  Name        Persistence-M | Bus-Id        Disp.A | Volatile Uncorr. ECC |
     | Fan  Temp  Perf  Pwr:Usage/Cap | Memory-Usage | GPU-Util  Compute M. |
     |  |  | MIG M. |
     | ===============================+======================+====================== |
     | 0  NVDIA A100-SXM...  Off | 00000000:05:00.0 Off | On |
     | N/A   32C   P0    49W / 400W | 0MiB / 81251MiB | N/A      Default |
     |  |  | Enabled |
     +-------------------------------+----------------------+----------------------+
     
     +-----------------------------------------------------------------------------+
     | MIG devices: |
     +-----------------------------------------------------------------------------+
     | GPU  GI  CI  MIG | Memory-Usage | Vol | Shared |
     | ID  ID  Dev | BAR1-Usage | SM     Unc | CE  ENC  DEC  OFA  JPG |
     |  |  | ECC |  |
     | ============================================================================= |
     | 0    0   0    0 | 0MiB / 81251MiB | 98      0 | 7   0    5    1    1 |
     |  | 1MiB / 13107... |  |  |
     +-----------------------------------------------------------------------------+
     +-----------------------------------------------------------------------------+
     | Processes: |
     | GPU   GI   CI       PID   Type   Process name                   GPU Memory |
     | ID   ID                                                   Usage |
     | ============================================================================= |
     | No running processes found |
     +-----------------------------------------------------------------------------+

     $ nvidia-smi
     Mon Sep 27 09:52:17 2021
     +-----------------------------------------------------------------------------+
     | NVIDIA-SMI 470.57.02    Driver Version: 470.57.02    CUDA Version: 11.4 |
     | -------------------------------+----------------------+---------------------- |
     | GPU  Name        Persistence-M | Bus-Id        Disp.A | Volatile Uncorr. ECC |
     | Fan  Temp  Perf  Pwr:Usage/Cap | Memory-Usage | GPU-Util  Compute M. |
     |  |  | MIG M. |
     | ===============================+======================+====================== |
     | 0  NVDIA A100-SXM...  Off | 00000000:05:00.0 Off | On |
     | N/A   32C   P0    49W / 400W | 0MiB / 81251MiB | N/A      Default |
     |  |  | Enabled |
     +-------------------------------+----------------------+----------------------+
     
     +-----------------------------------------------------------------------------+
     | MIG devices: |
     +-----------------------------------------------------------------------------+
     | GPU  GI  CI  MIG | Memory-Usage | Vol | Shared |
     | ID  ID  Dev | BAR1-Usage | SM     Unc | CE  ENC  DEC  OFA  JPG |
     |  |  | ECC |  |
     | ============================================================================= |
     | 0    0   0    0 | 0MiB / 81251MiB | 98      0 | 7   0    5    1    1 |
     |  | 1MiB / 13107... |  |  |
     +-----------------------------------------------------------------------------+
     +-----------------------------------------------------------------------------+
     | Processes: |
     | GPU   GI   CI       PID   Type   Process name                   GPU Memory |
     | ID   ID                                                   Usage |
     | ============================================================================= |
     | No running processes found |
     +-----------------------------------------------------------------------------+

Using MIG

1. Use the MIG Instance to perform the Job.
   • Example of task execution
     Sample Code Download

     Copy Code

     Color mode

     $ docker run --gpus '"device=[GPU ID]:[MIG ID]"' -rm nvcr.io/nvidia/cuda nvidia-smi

     $ docker run --gpus '"device=[GPU ID]:[MIG ID]"' -rm nvcr.io/nvidia/cuda nvidia-smi

   • You can see an example of the work performed as follows.
     Sample Code Download

     Copy Code

     Color mode

     $ docker run --gpus '"device=0:0"' -rm -it --network=host --shm-size=1g --ipc=host -v /root/.ssh/:/root/.ssh
     
     ================
     == TensorFlow ==
     ================
     
     NVIDIA Release 21.08-tf1 (build 26012104)
     TensorFlow Version 1.15.5
     
     Container image Copyright (c) 2021, NVIDIA CORPORATION. All right reserved.
     ...
     
     # Run Python process
     root@d622a93c9281:/workspace# python /workspace/nvidia-examples/cnn/resnet.py --num_iter 100
     ...
     PY 3.8.10 (default, Jun 2 2021, 10:49:15)
     [GCC 9.4.0]
     TF 1.15.5
     ...

     $ docker run --gpus '"device=0:0"' -rm -it --network=host --shm-size=1g --ipc=host -v /root/.ssh/:/root/.ssh
     
     ================
     == TensorFlow ==
     ================
     
     NVIDIA Release 21.08-tf1 (build 26012104)
     TensorFlow Version 1.15.5
     
     Container image Copyright (c) 2021, NVIDIA CORPORATION. All right reserved.
     ...
     
     # Run Python process
     root@d622a93c9281:/workspace# python /workspace/nvidia-examples/cnn/resnet.py --num_iter 100
     ...
     PY 3.8.10 (default, Jun 2 2021, 10:49:15)
     [GCC 9.4.0]
     TF 1.15.5
     ...

2. Check the GPU usage. (Create JOB process)
   • When the job runs, you can see that a process is allocated to the MIG device and its utilization increases.
     Sample Code Download

     Copy Code

     Color mode

     $ nvidia-smi mig -i [GPU ID] -gi [GPU Instance ID] -lcip

     $ nvidia-smi mig -i [GPU ID] -gi [GPU Instance ID] -lcip

   • You can check the GPU usage as shown below.
     Sample Code Download

     Copy Code

     Color mode

     +-----------------------------------------------------------------------------+
     | MIG devices: |
     +-----------------------------------------------------------------------------+
     | GPU  GI  CI  MIG | Memory-Usage | Vol | Shared |
     | ID  ID  Dev | BAR1-Usage | SM     Unc | CE  ENC  DEC  OFA  JPG |
     |  |  | ECC |  |
     | ============================================================================= |
     | 0    0   0    0 | 66562MiB / 81251MiB | 98      0 | 7   0    5    1    1 |
     |  | 5MiB / 13107... |  |  |
     +-----------------------------------------------------------------------------+
     +-----------------------------------------------------------------------------+
     | Processes: |
     | GPU   GI   CI       PID   Type   Process name                   GPU Memory |
     | ID   ID                                                   Usage |
     | ============================================================================= |
     | 0     0    0     17483      C   python                           66559MiB |
     +-----------------------------------------------------------------------------+

     +-----------------------------------------------------------------------------+
     | MIG devices: |
     +-----------------------------------------------------------------------------+
     | GPU  GI  CI  MIG | Memory-Usage | Vol | Shared |
     | ID  ID  Dev | BAR1-Usage | SM     Unc | CE  ENC  DEC  OFA  JPG |
     |  |  | ECC |  |
     | ============================================================================= |
     | 0    0   0    0 | 66562MiB / 81251MiB | 98      0 | 7   0    5    1    1 |
     |  | 5MiB / 13107... |  |  |
     +-----------------------------------------------------------------------------+
     +-----------------------------------------------------------------------------+
     | Processes: |
     | GPU   GI   CI       PID   Type   Process name                   GPU Memory |
     | ID   ID                                                   Usage |
     | ============================================================================= |
     | 0     0    0     17483      C   python                           66559MiB |
     +-----------------------------------------------------------------------------+

Delete and release MIG Instance

Follow these steps to delete the MIG instance and detach the MIG.

Compute Instance Delete

• Delete the Compute Instance.
  Sample Code Download

  Copy Code

  Color mode

  $ nvidia-smi mig -i [GPU ID] -gi [GPU Instance ID] –dci
  $ nvidia-smi mig -i [GPU ID] -gi [GPU Instance ID] -ci [Compute Instance] –dci

  $ nvidia-smi mig -i [GPU ID] -gi [GPU Instance ID] –dci
  $ nvidia-smi mig -i [GPU ID] -gi [GPU Instance ID] -ci [Compute Instance] –dci

  Sample Code Download

  Copy Code

  Color mode

  $ nvidia-smi mig -i 0 -gi 0 –lci
  +-----------------------------------------------------------------+
  | Compute instance profiles: |
  | GPU     GPU     Name            Profile  Instances   Placement |
  | GPU   Instance                     ID      ID        Start:Size |
  | ID |
  | ================================================================= |
  | 0      0      MIG 7g.80gb         4       0            0:7 |
  +-----------------------------------------------------------------+

  $ nvidia-smi mig -i 0 -gi 0 –lci
  +-----------------------------------------------------------------+
  | Compute instance profiles: |
  | GPU     GPU     Name            Profile  Instances   Placement |
  | GPU   Instance                     ID      ID        Start:Size |
  | ID |
  | ================================================================= |
  | 0      0      MIG 7g.80gb         4       0            0:7 |
  +-----------------------------------------------------------------+

  Sample Code Download

  Copy Code

  Color mode

  $ nvidia-smi mig -i 0 -gi 0 –dci
  Successfully destroyed compute instance ID  0 from GPU instance ID  0

  $ nvidia-smi mig -i 0 -gi 0 –dci
  Successfully destroyed compute instance ID  0 from GPU instance ID  0

  Sample Code Download

  Copy Code

  Color mode

  $ nvidia-smi mig -i 0 -gi 0 –lci
  No compute instances found: Not found

  $ nvidia-smi mig -i 0 -gi 0 –lci
  No compute instances found: Not found

Delete GPU Instance

• Delete the GPU Instance.
  Sample Code Download

  Copy Code

  Color mode

  $ nvidia-smi mig -i [GPU ID] –dgi
  $ nvidia-smi mig -i [GPU ID] -gi [GPU Instance ID] –dgi

  $ nvidia-smi mig -i [GPU ID] –dgi
  $ nvidia-smi mig -i [GPU ID] -gi [GPU Instance ID] –dgi

  Sample Code Download

  Copy Code

  Color mode

  $ nvidia-smi mig -i 0 -lgi
  +--------------------------------------------------------+
  | GPU instances: |
  | GPU   Name               Profile  Instance  Placement |
  | ID       ID      Start:Size |
  | ======================================================== |
  | 0  MIG 7g.80gb            0        0         0:8 |
  +--------------------------------------------------------+

  $ nvidia-smi mig -i 0 -lgi
  +--------------------------------------------------------+
  | GPU instances: |
  | GPU   Name               Profile  Instance  Placement |
  | ID       ID      Start:Size |
  | ======================================================== |
  | 0  MIG 7g.80gb            0        0         0:8 |
  +--------------------------------------------------------+

  Sample Code Download

  Copy Code

  Color mode

  $ nvidia-smi mig -i 0 -dgi
  Successfully destroyed GPU instance ID  0 from GPU  0

  $ nvidia-smi mig -i 0 -dgi
  Successfully destroyed GPU instance ID  0 from GPU  0

  Sample Code Download

  Copy Code

  Color mode

  $ nvidia-smi mig -i 0 -lgi
  No GPU instances found: Not found

  $ nvidia-smi mig -i 0 -lgi
  No GPU instances found: Not found

Disable MIG feature (deactivation)

• After disabling MIG (Disable), reboot.
  Sample Code Download

  Copy Code

  Color mode

  $ nvidia-smi -mig 0
  Disabled MIG Mode for GPU 00000000:05:00.0
  
  All done.

  $ nvidia-smi -mig 0
  Disabled MIG Mode for GPU 00000000:05:00.0
  
  All done.

  Sample Code Download

  Copy Code

  Color mode

  $ nvidia-smi
  Mon Sep 30 05:18:28 2021
  +-----------------------------------------------------------------------------+
  | NVIDIA-SMI 470.57.02    Driver Version: 470.57.02    CUDA Version: 11.4 |
  | -------------------------------+----------------------+---------------------- |
  | GPU  Name        Persistence-M | Bus-Id        Disp.A | Volatile Uncorr. ECC |
  | Fan  Temp  Perf  Pwr:Usage/Cap | Memory-Usage | GPU-Util  Compute M. |
  |  |  | MIG M. |
  | ===============================+======================+====================== |
  | 0  NVDIA A100-SXM...  Off | 00000000:05:00.0 Off | 0 |
  | N/A   33C   P0    60W / 400W | 0MiB / 81251MiB | 0%      Default |
  |  |  | Disabled |
  +-------------------------------+----------------------+----------------------+
  +-----------------------------------------------------------------------------+
  | MIG devices: |
  +-----------------------------------------------------------------------------+
  | GPU  GI  CI  MIG | Memory-Usage | Vol | Shared |
  | ID  ID  Dev | BAR1-Usage | SM     Unc | CE  ENC  DEC  OFA  JPG |
  |  |  | ECC |  |
  | ============================================================================= |
  | No MIG devices found |
  +-----------------------------------------------------------------------------+
  +-----------------------------------------------------------------------------+
  | Processes: |
  | GPU   GI   CI       PID   Type   Process name                   GPU Memory |
  | ID   ID                                                   Usage |
  | ============================================================================= |
  | No running processes found |
  +-----------------------------------------------------------------------------+

  $ nvidia-smi
  Mon Sep 30 05:18:28 2021
  +-----------------------------------------------------------------------------+
  | NVIDIA-SMI 470.57.02    Driver Version: 470.57.02    CUDA Version: 11.4 |
  | -------------------------------+----------------------+---------------------- |
  | GPU  Name        Persistence-M | Bus-Id        Disp.A | Volatile Uncorr. ECC |
  | Fan  Temp  Perf  Pwr:Usage/Cap | Memory-Usage | GPU-Util  Compute M. |
  |  |  | MIG M. |
  | ===============================+======================+====================== |
  | 0  NVDIA A100-SXM...  Off | 00000000:05:00.0 Off | 0 |
  | N/A   33C   P0    60W / 400W | 0MiB / 81251MiB | 0%      Default |
  |  |  | Disabled |
  +-------------------------------+----------------------+----------------------+
  +-----------------------------------------------------------------------------+
  | MIG devices: |
  +-----------------------------------------------------------------------------+
  | GPU  GI  CI  MIG | Memory-Usage | Vol | Shared |
  | ID  ID  Dev | BAR1-Usage | SM     Unc | CE  ENC  DEC  OFA  JPG |
  |  |  | ECC |  |
  | ============================================================================= |
  | No MIG devices found |
  +-----------------------------------------------------------------------------+
  +-----------------------------------------------------------------------------+
  | Processes: |
  | GPU   GI   CI       PID   Type   Process name                   GPU Memory |
  | ID   ID                                                   Usage |
  | ============================================================================= |
  | No running processes found |
  +-----------------------------------------------------------------------------+

4 - Use NVSwitch on GPU Server

After creating a GPU Server, you can enable the NVSwitch feature on the GPU Server’s VM (Guest OS) and use fast GPU-to-GPU P2P communication.

Exploring NVIDIA NVSwitch for Multi GPU

NVLink expands I/O by directly connecting multiple GPUs within a server both bidirectionally and GPU-to-GPU. Using NVSwitch, you can connect all GPUs in a server with full NVLink bandwidth.

Checking NVSwitch operation

Check the NVIDIA Fabric Manager, NVIDIA NVLink topology, and NVIDIA NVLink Status on the GPU server.

NVIDIA Fabric Manager operating status

Verify that active (running) is displayed when operating normally.

~$ systemctl status nvidia-fabricmanager

nvidia-fabricmanager.service - NVIDIA fabric manager service
     Loaded: loaded (/lib/systemd/system/nvidia-fabricmanager.service; enabled; vendor preset: enabled)
     Active: active (running) since Mon 2026-02-02 16:23:27 KST; 32min ago
   Main PID: 2191 (nv-fabricmanage)
      Tasks: 18 (limit: 629145)
     Memory: 18.0M
        CPU: 33.461s
     CGroup: /system.slice/nvidia-fabricmanager.service
             └─2191 /usr/bin/nv-fabricmanager -c /usr/share/nvidia/nvswitch/fabricmanager.cfg

nvidia-fabricmanager.service - NVIDIA fabric manager service
     Loaded: loaded (/lib/systemd/system/nvidia-fabricmanager.service; enabled; vendor preset: enabled)
     Active: active (running) since Mon 2026-02-02 16:23:27 KST; 32min ago
   Main PID: 2191 (nv-fabricmanage)
      Tasks: 18 (limit: 629145)
     Memory: 18.0M
        CPU: 33.461s
     CGroup: /system.slice/nvidia-fabricmanager.service
             └─2191 /usr/bin/nv-fabricmanager -c /usr/share/nvidia/nvswitch/fabricmanager.cfg

Check NVIDIA NVLink topology

Check the NVIDIA NVLink topology.

~$ nvidia-smi topo -m

nvidia-smi topo -m
        GPU0    GPU1    GPU2    GPU3    GPU4    GPU5    GPU6    GPU7    CPU Affinity    NUMA Affinity   GPU NUMA ID
GPU0     X      NV12    NV12    NV12    NV12    NV12    NV12    NV12    0-127   0-7             N/A
GPU1    NV12     X      NV12    NV12    NV12    NV12    NV12    NV12    0-127   0-7             N/A
GPU2    NV12    NV12     X      NV12    NV12    NV12    NV12    NV12    0-127   0-7             N/A
GPU3    NV12    NV12    NV12     X      NV12    NV12    NV12    NV12    0-127   0-7             N/A
GPU4    NV12    NV12    NV12    NV12     X      NV12    NV12    NV12    0-127   0-7             N/A
GPU5    NV12    NV12    NV12    NV12    NV12     X      NV12    NV12    0-127   0-7             N/A
GPU6    NV12    NV12    NV12    NV12    NV12    NV12     X      NV12    0-127   0-7             N/A
GPU7    NV12    NV12    NV12    NV12    NV12    NV12    NV12     X      0-127   0-7             N/A

Legend:

  X    = Self
  SYS  = Connection traversing PCIe as well as the SMP interconnect between NUMA nodes (e.g., QPI/UPI)
  NODE = Connection traversing PCIe as well as the interconnect between PCIe Host Bridges within a NUMA node
  PHB  = Connection traversing PCIe as well as a PCIe Host Bridge (typically the CPU)
  PXB  = Connection traversing multiple PCIe bridges (without traversing the PCIe Host Bridge)
  PIX  = Connection traversing at most a single PCIe bridge
  NV#  = Connection traversing a bonded set of # NVLinks

nvidia-smi topo -m
        GPU0    GPU1    GPU2    GPU3    GPU4    GPU5    GPU6    GPU7    CPU Affinity    NUMA Affinity   GPU NUMA ID
GPU0     X      NV12    NV12    NV12    NV12    NV12    NV12    NV12    0-127   0-7             N/A
GPU1    NV12     X      NV12    NV12    NV12    NV12    NV12    NV12    0-127   0-7             N/A
GPU2    NV12    NV12     X      NV12    NV12    NV12    NV12    NV12    0-127   0-7             N/A
GPU3    NV12    NV12    NV12     X      NV12    NV12    NV12    NV12    0-127   0-7             N/A
GPU4    NV12    NV12    NV12    NV12     X      NV12    NV12    NV12    0-127   0-7             N/A
GPU5    NV12    NV12    NV12    NV12    NV12     X      NV12    NV12    0-127   0-7             N/A
GPU6    NV12    NV12    NV12    NV12    NV12    NV12     X      NV12    0-127   0-7             N/A
GPU7    NV12    NV12    NV12    NV12    NV12    NV12    NV12     X      0-127   0-7             N/A

Legend:

  X    = Self
  SYS  = Connection traversing PCIe as well as the SMP interconnect between NUMA nodes (e.g., QPI/UPI)
  NODE = Connection traversing PCIe as well as the interconnect between PCIe Host Bridges within a NUMA node
  PHB  = Connection traversing PCIe as well as a PCIe Host Bridge (typically the CPU)
  PXB  = Connection traversing multiple PCIe bridges (without traversing the PCIe Host Bridge)
  PIX  = Connection traversing at most a single PCIe bridge
  NV#  = Connection traversing a bonded set of # NVLinks

Check NVIDIA NVLink Status

Check the NVIDIA NVLink Status.

~$ nvidia-smi topo -m

GPU 1: NVIDIA A100-SXM4-80GB (UUID: GPU-64a2f685-bb12-c4af-105c-0726ece9c8d7)
         Link 0: 25 GB/s
         Link 1: 25 GB/s
         Link 2: 25 GB/s
         Link 3: 25 GB/s
         Link 4: 25 GB/s
         Link 5: 25 GB/s
         Link 6: 25 GB/s
         Link 7: 25 GB/s
         Link 8: 25 GB/s
         Link 9: 25 GB/s
         Link 10: 25 GB/s
         Link 11: 25 GB/s
GPU 2: NVIDIA A100-SXM4-80GB (UUID: GPU-2269851b-71cd-f6c7-50c5-ba1525cf3ce8)
         Link 0: 25 GB/s
         Link 1: 25 GB/s
         Link 2: 25 GB/s
         Link 3: 25 GB/s
         Link 4: 25 GB/s
         Link 5: 25 GB/s
         Link 6: 25 GB/s
         Link 7: 25 GB/s
         Link 8: 25 GB/s
         Link 9: 25 GB/s
         Link 10: 25 GB/s
         Link 11: 25 GB/s
GPU 3: NVIDIA A100-SXM4-80GB (UUID: GPU-4c397bbf-95fc-5c29-918a-a429cbe45a7a)
         Link 0: 25 GB/s
         Link 1: 25 GB/s
         Link 2: 25 GB/s
         Link 3: 25 GB/s
         Link 4: 25 GB/s
         Link 5: 25 GB/s
         Link 6: 25 GB/s
         Link 7: 25 GB/s
         Link 8: 25 GB/s
         Link 9: 25 GB/s
         Link 10: 25 GB/s
         Link 11: 25 GB/s
GPU 4: NVIDIA A100-SXM4-80GB (UUID: GPU-0e350204-9fb6-2cbe-538e-8f7849658eb8)
         Link 0: 25 GB/s
         Link 1: 25 GB/s
         Link 2: 25 GB/s
         Link 3: 25 GB/s
         Link 4: 25 GB/s
         Link 5: 25 GB/s
         Link 6: 25 GB/s
         Link 7: 25 GB/s
         Link 8: 25 GB/s
         Link 9: 25 GB/s
         Link 10: 25 GB/s
         Link 11: 25 GB/s
GPU 5: NVIDIA A100-SXM4-80GB (UUID: GPU-45f0c453-4760-edd4-3af9-25c5ea7473a5)
         Link 0: 25 GB/s
         Link 1: 25 GB/s
         Link 2: 25 GB/s
         Link 3: 25 GB/s
         Link 4: 25 GB/s
         Link 5: 25 GB/s
         Link 6: 25 GB/s
         Link 7: 25 GB/s
         Link 8: 25 GB/s
         Link 9: 25 GB/s
         Link 10: 25 GB/s
         Link 11: 25 GB/s
GPU 6: NVIDIA A100-SXM4-80GB (UUID: GPU-38409794-bb34-430e-3c50-90b42cb2bb72)
         Link 0: 25 GB/s
         Link 1: 25 GB/s
         Link 2: 25 GB/s
         Link 3: 25 GB/s
         Link 4: 25 GB/s
         Link 5: 25 GB/s
         Link 6: 25 GB/s
         Link 7: 25 GB/s
         Link 8: 25 GB/s
         Link 9: 25 GB/s
         Link 10: 25 GB/s
         Link 11: 25 GB/s
GPU 7: NVIDIA A100-SXM4-80GB (UUID: GPU-3fb478aa-801b-eb64-55c2-0ffc3f2ce404)
         Link 0: 25 GB/s
         Link 1: 25 GB/s
         Link 2: 25 GB/s
         Link 3: 25 GB/s
         Link 4: 25 GB/s
         Link 5: 25 GB/s
         Link 6: 25 GB/s
         Link 7: 25 GB/s
         Link 8: 25 GB/s
         Link 9: 25 GB/s
         Link 10: 25 GB/s
         Link 11: 25 GB/s

GPU 1: NVIDIA A100-SXM4-80GB (UUID: GPU-64a2f685-bb12-c4af-105c-0726ece9c8d7)
         Link 0: 25 GB/s
         Link 1: 25 GB/s
         Link 2: 25 GB/s
         Link 3: 25 GB/s
         Link 4: 25 GB/s
         Link 5: 25 GB/s
         Link 6: 25 GB/s
         Link 7: 25 GB/s
         Link 8: 25 GB/s
         Link 9: 25 GB/s
         Link 10: 25 GB/s
         Link 11: 25 GB/s
GPU 2: NVIDIA A100-SXM4-80GB (UUID: GPU-2269851b-71cd-f6c7-50c5-ba1525cf3ce8)
         Link 0: 25 GB/s
         Link 1: 25 GB/s
         Link 2: 25 GB/s
         Link 3: 25 GB/s
         Link 4: 25 GB/s
         Link 5: 25 GB/s
         Link 6: 25 GB/s
         Link 7: 25 GB/s
         Link 8: 25 GB/s
         Link 9: 25 GB/s
         Link 10: 25 GB/s
         Link 11: 25 GB/s
GPU 3: NVIDIA A100-SXM4-80GB (UUID: GPU-4c397bbf-95fc-5c29-918a-a429cbe45a7a)
         Link 0: 25 GB/s
         Link 1: 25 GB/s
         Link 2: 25 GB/s
         Link 3: 25 GB/s
         Link 4: 25 GB/s
         Link 5: 25 GB/s
         Link 6: 25 GB/s
         Link 7: 25 GB/s
         Link 8: 25 GB/s
         Link 9: 25 GB/s
         Link 10: 25 GB/s
         Link 11: 25 GB/s
GPU 4: NVIDIA A100-SXM4-80GB (UUID: GPU-0e350204-9fb6-2cbe-538e-8f7849658eb8)
         Link 0: 25 GB/s
         Link 1: 25 GB/s
         Link 2: 25 GB/s
         Link 3: 25 GB/s
         Link 4: 25 GB/s
         Link 5: 25 GB/s
         Link 6: 25 GB/s
         Link 7: 25 GB/s
         Link 8: 25 GB/s
         Link 9: 25 GB/s
         Link 10: 25 GB/s
         Link 11: 25 GB/s
GPU 5: NVIDIA A100-SXM4-80GB (UUID: GPU-45f0c453-4760-edd4-3af9-25c5ea7473a5)
         Link 0: 25 GB/s
         Link 1: 25 GB/s
         Link 2: 25 GB/s
         Link 3: 25 GB/s
         Link 4: 25 GB/s
         Link 5: 25 GB/s
         Link 6: 25 GB/s
         Link 7: 25 GB/s
         Link 8: 25 GB/s
         Link 9: 25 GB/s
         Link 10: 25 GB/s
         Link 11: 25 GB/s
GPU 6: NVIDIA A100-SXM4-80GB (UUID: GPU-38409794-bb34-430e-3c50-90b42cb2bb72)
         Link 0: 25 GB/s
         Link 1: 25 GB/s
         Link 2: 25 GB/s
         Link 3: 25 GB/s
         Link 4: 25 GB/s
         Link 5: 25 GB/s
         Link 6: 25 GB/s
         Link 7: 25 GB/s
         Link 8: 25 GB/s
         Link 9: 25 GB/s
         Link 10: 25 GB/s
         Link 11: 25 GB/s
GPU 7: NVIDIA A100-SXM4-80GB (UUID: GPU-3fb478aa-801b-eb64-55c2-0ffc3f2ce404)
         Link 0: 25 GB/s
         Link 1: 25 GB/s
         Link 2: 25 GB/s
         Link 3: 25 GB/s
         Link 4: 25 GB/s
         Link 5: 25 GB/s
         Link 6: 25 GB/s
         Link 7: 25 GB/s
         Link 8: 25 GB/s
         Link 9: 25 GB/s
         Link 10: 25 GB/s
         Link 11: 25 GB/s

5 - Install ServiceWatch Agent

Users can install the ServiceWatch Agent on a GPU server to collect custom metrics and logs.

ServiceWatch Agent

There are two main types of agents that need to be installed on a GPU server to collect custom metrics and logs for ServiceWatch. It is the Prometheus Exporter and Open Telemetry Collector.

Pre-configuration for using ServiceWatch Agent

To use the ServiceWatch Agent, please refer to ServiceWatch Agent를 위한 사전 환경 설정 and prepare the prerequisite configuration.

Install Prometheus Exporter for GPU metrics (for Ubuntu)

Install the Prometheus Exporter for collecting metrics from the GPU server according to the steps below.

Verify NVDIA Driver Installation

• Check the installed NVDIA driver.
  • Tab 0

  Sample Code Download

  Copy Code

  Color mode

  nvidia-smi --query-gpu driver_version --format csv

  nvidia-smi --query-gpu driver_version --format csv

  • Tab 0

  Sample Code Download

  Copy Code

  Color mode

  driver_version
  535.183.06
  
  535.183.06

  driver_version
  535.183.06
  
  535.183.06

NVSwitch Configuration and Query (NSCQ) Library Installation

1. Install cuda-keyring.

   • Tab 0

   Sample Code Download

   Copy Code

   Color mode

   wget https://developer.download.nvidia.com/compute/cuda/repos/<distro>/<arch>/cuda-keyring_1.1-1_all.deb

   wget https://developer.download.nvidia.com/compute/cuda/repos/<distro>/<arch>/cuda-keyring_1.1-1_all.deb

   • Tab 0

   Sample Code Download

   Copy Code

   Color mode

   sudo dpkg -i cuda-keyring_1.1-1_all.deb
   apt update

   sudo dpkg -i cuda-keyring_1.1-1_all.deb
   apt update

   • Tab 0

   Sample Code Download

   Copy Code

   Color mode

   nvidia-smi --query-gpu driver_version --format csv

   nvidia-smi --query-gpu driver_version --format csv

   • Tab 0

   Sample Code Download

   Copy Code

   Color mode

   driver_version
   535.183.06
   ...
   
   535.183.06

   driver_version
   535.183.06
   ...
   
   535.183.06

2. Install libnvidia-nscq.

   • Tab 0

   Sample Code Download

   Copy Code

   Color mode

   apt-cache policy libnvidia-nscq-535

   apt-cache policy libnvidia-nscq-535

   • Tab 0

   Sample Code Download

   Copy Code

   Color mode

   libnvidia-nscq-535:
     Installed: (none)
     Candidate: 535.247.01-1
     Version table:
        535.247.01-1 600
           600 https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2204/x86_64  Packages
   ...
           600 https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2204/x86_64  Packages
        535.216.01-1 600
           600 https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2204/x86_64  Packages
        535.183.06-1 600  # Install the version that matches the Driver
           600 https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2204/x86_64  Packages
        535.183.01-1 600
   ...
           600 https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2204/x86_64  Packages
        535.54.03-1 600
           600 https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2204/x86_64  Packages

   libnvidia-nscq-535:
     Installed: (none)
     Candidate: 535.247.01-1
     Version table:
        535.247.01-1 600
           600 https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2204/x86_64  Packages
   ...
           600 https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2204/x86_64  Packages
        535.216.01-1 600
           600 https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2204/x86_64  Packages
        535.183.06-1 600  # Install the version that matches the Driver
           600 https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2204/x86_64  Packages
        535.183.01-1 600
   ...
           600 https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2204/x86_64  Packages
        535.54.03-1 600
           600 https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2204/x86_64  Packages

   • Tab 0

   Sample Code Download

   Copy Code

   Color mode

   apt install libnvidia-nscq-535=535.183.06-1

   apt install libnvidia-nscq-535=535.183.06-1

NVSwitch Device Monitoring API(NVSDM) Library Installation

• Install the NVSDM library.
  • Tab 0

  Sample Code Download

  Copy Code

  Color mode

  apt-cache policy libnvsdm

  apt-cache policy libnvsdm

  • Tab 0

  Sample Code Download

  Copy Code

  Color mode

  libnvsdm:
    Installed: (none)
    Candidate: 580.105.08-1
    Version table:
       580.105.08-1 600
          600 https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2204/x86_64  Packages
       580.95.05-1 600
          600 https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2204/x86_64  Packages
       580.82.07-1 600
          600 https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2204/x86_64  Packages
       580.65.06-1 600
          600 https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2204/x86_64  Packages

  libnvsdm:
    Installed: (none)
    Candidate: 580.105.08-1
    Version table:
       580.105.08-1 600
          600 https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2204/x86_64  Packages
       580.95.05-1 600
          600 https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2204/x86_64  Packages
       580.82.07-1 600
          600 https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2204/x86_64  Packages
       580.65.06-1 600
          600 https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2204/x86_64  Packages

  • Tab 0

  Sample Code Download

  Copy Code

  Color mode

  apt install libnvsdm=580.105.08-1

  apt install libnvsdm=580.105.08-1

NVIDIA DCGM installation (for Ubuntu)

Install the DCGM Exporter according to the steps below.

1. DCGM(datacenter-gpu-manager) Installation
2. datacenter-gpu-manager-exporter Installation
3. DCGM Service Activation and Start

DCGM(datacenter-gpu-manager) Installation

refers to a specific version of NVIDIA’s Data Center GPU Manager (DCGM) tool, which is a package for managing and monitoring NVIDIA data center GPUs. In particular, cuda12 indicates that this management tool is installed for the CUDA 12 version, and datacenter-gpu-manager-4 refers to the 4.x version of DCGM. This tool provides various features, including GPU status monitoring, diagnostics, alert system, and power/clock management.

1. Check the CUDA version.
   • Tab 0

   Sample Code Download

   Copy Code

   Color mode

   nvidia-smi | grep CUDA

   nvidia-smi | grep CUDA

   • Tab 0

   Sample Code Download

   Copy Code

   Color mode

   | NVIDIA-SMI 535.183.06             Driver Version: 535.183.06     CUDA Version: 12.2 |

   | NVIDIA-SMI 535.183.06             Driver Version: 535.183.06     CUDA Version: 12.2 |

   • Tab 0

   Sample Code Download

   Copy Code

   Color mode

   CUDA_VERSION=12

   CUDA_VERSION=12

2. Install datacenter-gpu-manager-cuda.
   • Tab 0

   Sample Code Download

   Copy Code

   Color mode

   apt install datacenter-gpu-manager-4-cuda${CUDA_VERSION}

   apt install datacenter-gpu-manager-4-cuda${CUDA_VERSION}

datacenter-gpu-manager-exporter installation

It is a tool that, based on NVIDIA Data Center GPU Manager (DCGM), collects various GPU metrics such as GPU usage, memory usage, temperature, and power consumption, and exposes them for use in monitoring systems like Prometheus.

1. Install datacenter-gpu-manager-exporter.
   • Tab 0

   Sample Code Download

   Copy Code

   Color mode

   apt install datacenter-gpu-manager-exporter

   apt install datacenter-gpu-manager-exporter

2. Check the DCGM Exporter configuration file.
   • Tab 0

   Sample Code Download

   Copy Code

   Color mode

   cat /usr/lib/systemd/system/nvidia-dcgm-exporter.service | grep ExecStart

   cat /usr/lib/systemd/system/nvidia-dcgm-exporter.service | grep ExecStart

   • Tab 0

   Sample Code Download

   Copy Code

   Color mode

   ExecStart=/usr/bin/dcgm-exporter -f /etc/dcgm-exporter/default-counters.csv

   ExecStart=/usr/bin/dcgm-exporter -f /etc/dcgm-exporter/default-counters.csv

3. When installing the DCGM Exporter, review the provided settings and remove # for required metrics, and add # for unnecessary metrics.
   • Tab 0

   Sample Code Download

   Copy Code

   Color mode

   vi /etc/dcgm-exporter/default-counters.csv
   ## Example ##
   
   DCGM_FI_PROF_PIPE_TENSOR_ACTIVE, gauge, Ratio of cycles the tensor (HMMA) pipe is active.
   DCGM_FI_PROF_DRAM_ACTIVE,        gauge, Ratio of cycles the device memory interface is active sending or receiving data.
   # DCGM_FI_PROF_PIPE_FP64_ACTIVE,   gauge, Ratio of cycles the fp64 pipes are active.
   # DCGM_FI_PROF_PIPE_FP32_ACTIVE,   gauge, Ratio of cycles the fp32 pipes are active.

   vi /etc/dcgm-exporter/default-counters.csv
   ## Example ##
   
   DCGM_FI_PROF_PIPE_TENSOR_ACTIVE, gauge, Ratio of cycles the tensor (HMMA) pipe is active.
   DCGM_FI_PROF_DRAM_ACTIVE,        gauge, Ratio of cycles the device memory interface is active sending or receiving data.
   # DCGM_FI_PROF_PIPE_FP64_ACTIVE,   gauge, Ratio of cycles the fp64 pipes are active.
   # DCGM_FI_PROF_PIPE_FP32_ACTIVE,   gauge, Ratio of cycles the fp32 pipes are active.

Enable and start DCGM service

1. Enable and start the nvdia-dcgm service.

   • Tab 0

   Sample Code Download

   Copy Code

   Color mode

   systemctl enable --now nvidia-dcgm

   systemctl enable --now nvidia-dcgm

2. Enable and start the nvdia-dcgm-exporter service.

   • Tab 0

   Sample Code Download

   Copy Code

   Color mode

   systemctl enable --now nvidia-dcgm-exporter

   systemctl enable --now nvidia-dcgm-exporter

Installation of Prometheus Exporter for GPU metrics (for RHEL)

Install the ServiceWatch Agent according to the steps below to collect metrics from the GPU server.

NVDIA Driver Installation Check (for RHEL)

1. Check the installed NVDIA Driver.
   • Tab 0

   Sample Code Download

   Copy Code

   Color mode

   nvidia-smi --query-gpu driver_version --format csv

   nvidia-smi --query-gpu driver_version --format csv

   • Tab 0

   Sample Code Download

   Copy Code

   Color mode

   driver_version
   535.183.06
   ...
   
   535.183.06

   driver_version
   535.183.06
   ...
   
   535.183.06

NVSwitch Configuration and Query (NSCQ) Library Installation (for RHEL)

1. Checking the libnvdia-nscq package.

   • Tab 0

   Sample Code Download

   Copy Code

   Color mode

   rpm -qa | grep libnvidia-nscq libnvidia-nscq-535-535.183.06-1.x86_64

   rpm -qa | grep libnvidia-nscq libnvidia-nscq-535-535.183.06-1.x86_64

2. Add the CUDA Repository to DNF.

   • Tab 0

   Sample Code Download

   Copy Code

   Color mode

   dnf config-manager --add-repo https://developer.download.nvidia.com/compute/cuda/repos/rhel8/x86_64/cuda-rhel8.repo

   dnf config-manager --add-repo https://developer.download.nvidia.com/compute/cuda/repos/rhel8/x86_64/cuda-rhel8.repo

3. NVDIA Driver state reset

   • Tab 0

   Sample Code Download

   Copy Code

   Color mode

   dnf module reset nvidia-driver

   dnf module reset nvidia-driver

   • Tab 0

   Sample Code Download

   Copy Code

   Color mode

   Updating Subscription Management repositories.
   Last metadata expiration check: 0:03:15 ago on Wed 19 Nov 2025 01:23:48 AM EST.
   Dependencies resolved.
   =============================================
   Package Architecture Version Repository Size
   =============================================
   Disabling module profiles:
   nvidia-driver/default
   nvidia-driver/fm
   Resetting modules:
   nvidia-driver
   
   Transaction Summary
   =============================================
   
   Is this ok [y/N]: y

   Updating Subscription Management repositories.
   Last metadata expiration check: 0:03:15 ago on Wed 19 Nov 2025 01:23:48 AM EST.
   Dependencies resolved.
   =============================================
   Package Architecture Version Repository Size
   =============================================
   Disabling module profiles:
   nvidia-driver/default
   nvidia-driver/fm
   Resetting modules:
   nvidia-driver
   
   Transaction Summary
   =============================================
   
   Is this ok [y/N]: y

4. Activate the NVDIA Driver module.

   • Tab 0

   Sample Code Download

   Copy Code

   Color mode

   dnf module enable nvidia-driver:535-open

   dnf module enable nvidia-driver:535-open

   • Tab 0

   Sample Code Download

   Copy Code

   Color mode

   Updating Subscription Management repositories.
   Last metadata expiration check: 0:04:22 ago on Wed 19 Nov 2025 01:23:48 AM EST.
   Dependencies resolved.
   =============================================
   Package Architecture Version Repository Size
   =============================================
   Enabling module streams:
   nvidia-driver 535-open
   
   Transaction Summary
   =============================================
   
   Is this ok [y/N]: y

   Updating Subscription Management repositories.
   Last metadata expiration check: 0:04:22 ago on Wed 19 Nov 2025 01:23:48 AM EST.
   Dependencies resolved.
   =============================================
   Package Architecture Version Repository Size
   =============================================
   Enabling module streams:
   nvidia-driver 535-open
   
   Transaction Summary
   =============================================
   
   Is this ok [y/N]: y

5. Check the libnvdia-nscq module list.

   • Tab 0

   Sample Code Download

   Copy Code

   Color mode

   dnf list libnvidia-nscq-535 --showduplicates

   dnf list libnvidia-nscq-535 --showduplicates

6. Install libnvdia-nscq.

   • Tab 0

   Sample Code Download

   Copy Code

   Color mode

   dnf install libnvidia-nscq-535-535.183.06-1

   dnf install libnvidia-nscq-535-535.183.06-1

NVSwitch Device Monitoring API(NVSDM) Library Installation (for RHEL)

1. Check the NVSDM library module list.

   • Tab 0

   Sample Code Download

   Copy Code

   Color mode

   dnf list libnvsdm --showduplicates

   dnf list libnvsdm --showduplicates

   • Tab 0

   Sample Code Download

   Copy Code

   Color mode

   libnvsdm:
     Installed: (none)
     Candidate: 580.105.08-1
     Version table:
        580.105.08-1 600
           600 https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2204/x86_64  Packages
        580.95.05-1 600
           600 https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2204/x86_64  Packages
        580.82.07-1 600
           600 https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2204/x86_64  Packages
        580.65.06-1 600
           600 https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2204/x86_64  Packages

   libnvsdm:
     Installed: (none)
     Candidate: 580.105.08-1
     Version table:
        580.105.08-1 600
           600 https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2204/x86_64  Packages
        580.95.05-1 600
           600 https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2204/x86_64  Packages
        580.82.07-1 600
           600 https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2204/x86_64  Packages
        580.65.06-1 600
           600 https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2204/x86_64  Packages

2. Install libnvsdm.

   • Tab 0

   Sample Code Download

   Copy Code

   Color mode

   dnf install libnvsdm-580.105.08-1

   dnf install libnvsdm-580.105.08-1

   • Tab 0

   Sample Code Download

   Copy Code

   Color mode

   Updating Subscription Management repositories.
   Last metadata expiration check: 0:08:18 ago on Wed 19 Nov 2025 01:05:28 AM EST.
   Dependencies resolved.
   =========================================================================
   Package Architecture Version Repository Size
   =========================================================================
   Installing:
   libnvsdm x86_64 580.105.08-1 cuda-rhel8-x86_64 675 k
   Installing dependencies:
   infiniband-diags x86_64 48.0-1.el8 rhel-8-for-x86_64-baseos-rpms 323 k
   libibumad x86_64 48.0-1.el8 rhel-8-for-x86_64-baseos-rpms 34 k
   
   Transaction Summary
   =========================================================================
   Install 3 Packages
   
   Total download size: 1.0 M
   Installed size: 3.2 M
   Is this ok [y/N]: y

   Updating Subscription Management repositories.
   Last metadata expiration check: 0:08:18 ago on Wed 19 Nov 2025 01:05:28 AM EST.
   Dependencies resolved.
   =========================================================================
   Package Architecture Version Repository Size
   =========================================================================
   Installing:
   libnvsdm x86_64 580.105.08-1 cuda-rhel8-x86_64 675 k
   Installing dependencies:
   infiniband-diags x86_64 48.0-1.el8 rhel-8-for-x86_64-baseos-rpms 323 k
   libibumad x86_64 48.0-1.el8 rhel-8-for-x86_64-baseos-rpms 34 k
   
   Transaction Summary
   =========================================================================
   Install 3 Packages
   
   Total download size: 1.0 M
   Installed size: 3.2 M
   Is this ok [y/N]: y

NVIDIA DCGM installation (for RHEL)

Install Node Exporter according to the steps below.

1. DCGM(datacenter-gpu-manager) Installation
2. datacenter-gpu-manager-exporter installation
3. DCGM Service Activation and Start

DCGM(datacenter-gpu-manager) Installation (for RHEL)

Refers to a specific version of NVIDIA’s Data Center GPU Manager (DCGM) tool, which is a package for managing and monitoring NVIDIA data center GPUs. In particular, cuda12 indicates that this management tool is installed for the CUDA 12 version, and datacenter-gpu-manager-4 refers to the 4.x version of DCGM. This tool provides various features, including GPU status monitoring, diagnostics, alert system, and power/clock management.

1. Add the CUDA Repository to DNF.
   • Tab 0

   Sample Code Download

   Copy Code

   Color mode

   dnf config-manager --add-repo https://developer.download.nvidia.com/compute/cuda/repos/rhel8/x86_64/cuda-rhel8.repo

   dnf config-manager --add-repo https://developer.download.nvidia.com/compute/cuda/repos/rhel8/x86_64/cuda-rhel8.repo

2. Check the CUDA version.
   • Tab 0

   Sample Code Download

   Copy Code

   Color mode

   nvidia-smi | grep CUDA

   nvidia-smi | grep CUDA

   • Tab 0

   Sample Code Download

   Copy Code

   Color mode

   | NVIDIA-SMI 535.183.06             Driver Version: 535.183.06     CUDA Version: 12.2 |

   | NVIDIA-SMI 535.183.06             Driver Version: 535.183.06     CUDA Version: 12.2 |

   • Tab 0

   Sample Code Download

   Copy Code

   Color mode

   CUDA_VERSION=12

   CUDA_VERSION=12

3. Check the list of datacenter-gpu-manager-cuda modules.
   • Tab 0

   Sample Code Download

   Copy Code

   Color mode

   dnf list datacenter-gpu-manager-4-cuda${CUDA_VERSION} --showduplicates

   dnf list datacenter-gpu-manager-4-cuda${CUDA_VERSION} --showduplicates

   • Tab 0

   Sample Code Download

   Copy Code

   Color mode

   Updating Subscription Management repositories.
   Unable to read consumer identity
   
   This system is not registered with an entitlement server. You can use subscription-manager to register.
   
   Last metadata expiration check: 0:00:34 ago on Wed 19 Nov 2025 12:26:56 AM EST.
   Available Packages
   datacenter-gpu-manager-4-cuda12.x86_64   1:4.0.0-1    cuda-rhel8-x86_64
   datacenter-gpu-manager-4-cuda12.x86_64   1:4.1.0-1    cuda-rhel8-x86_64
   datacenter-gpu-manager-4-cuda12.x86_64   1:4.1.1-1    cuda-rhel8-x86_64
   datacenter-gpu-manager-4-cuda12.x86_64   1:4.2.0-1    cuda-rhel8-x86_64
   datacenter-gpu-manager-4-cuda12.x86_64   1:4.2.2-1    cuda-rhel8-x86_64
   datacenter-gpu-manager-4-cuda12.x86_64   1:4.2.3-1    cuda-rhel8-x86_64
   datacenter-gpu-manager-4-cuda12.x86_64   1:4.2.3-2    cuda-rhel8-x86_64
   datacenter-gpu-manager-4-cuda12.x86_64   1:4.3.0-1    cuda-rhel8-x86_64
   datacenter-gpu-manager-4-cuda12.x86_64   1:4.3.1-1    cuda-rhel8-x86_64
   datacenter-gpu-manager-4-cuda12.x86_64   1:4.4.0-1    cuda-rhel8-x86_64
   datacenter-gpu-manager-4-cuda12.x86_64   1:4.4.1-1    cuda-rhel8-x86_64
   datacenter-gpu-manager-4-cuda12.x86_64   1:4.4.2-1    cuda-rhel8-x86_64

   Updating Subscription Management repositories.
   Unable to read consumer identity
   
   This system is not registered with an entitlement server. You can use subscription-manager to register.
   
   Last metadata expiration check: 0:00:34 ago on Wed 19 Nov 2025 12:26:56 AM EST.
   Available Packages
   datacenter-gpu-manager-4-cuda12.x86_64   1:4.0.0-1    cuda-rhel8-x86_64
   datacenter-gpu-manager-4-cuda12.x86_64   1:4.1.0-1    cuda-rhel8-x86_64
   datacenter-gpu-manager-4-cuda12.x86_64   1:4.1.1-1    cuda-rhel8-x86_64
   datacenter-gpu-manager-4-cuda12.x86_64   1:4.2.0-1    cuda-rhel8-x86_64
   datacenter-gpu-manager-4-cuda12.x86_64   1:4.2.2-1    cuda-rhel8-x86_64
   datacenter-gpu-manager-4-cuda12.x86_64   1:4.2.3-1    cuda-rhel8-x86_64
   datacenter-gpu-manager-4-cuda12.x86_64   1:4.2.3-2    cuda-rhel8-x86_64
   datacenter-gpu-manager-4-cuda12.x86_64   1:4.3.0-1    cuda-rhel8-x86_64
   datacenter-gpu-manager-4-cuda12.x86_64   1:4.3.1-1    cuda-rhel8-x86_64
   datacenter-gpu-manager-4-cuda12.x86_64   1:4.4.0-1    cuda-rhel8-x86_64
   datacenter-gpu-manager-4-cuda12.x86_64   1:4.4.1-1    cuda-rhel8-x86_64
   datacenter-gpu-manager-4-cuda12.x86_64   1:4.4.2-1    cuda-rhel8-x86_64

4. Install datacenter-gpu-manager-cuda.
   • Tab 0

   Sample Code Download

   Copy Code

   Color mode

   dnf install datacenter-gpu-manager-4-cuda${CUDA_VERSION}

   dnf install datacenter-gpu-manager-4-cuda${CUDA_VERSION}

   • Tab 0

   Sample Code Download

   Copy Code

   Color mode

   Updating Subscription Management repositories.
   Unable to read consumer identity
   
   This system is not registered with an entitlement server. You can use subscription-manager to register.
   
   Last metadata expiration check: 0:07:12 ago on Wed 19 Nov 2025 12:26:56 AM EST.
   Dependencies resolved.
   ===================================================================================================
    Package                                       Architecture   Version     Repository          Size
   ===================================================================================================
   Installing:
    datacenter-gpu-manager-4-cuda12               x86_64         1:4.4.2-1   cuda-rhel8-x86_64   554 M
   Installing dependencies:
    datacenter-gpu-manager-4-core                 x86_64         1:4.4.2-1   cuda-rhel8-x86_64   9.9 M
   Installing weak dependencies:
    datacenter-gpu-manager-4-proprietary          x86_64         1:4.4.2-1   cuda-rhel8-x86_64   5.3 M
    datacenter-gpu-manager-4-proprietary-cuda12   x86_64         1:4.4.2-1   cuda-rhel8-x86_64   289 M
   
   Transaction Summary
   ====================================================================================================
   Install  4 Packages
   ...
   Is this ok [y/N]: y

   Updating Subscription Management repositories.
   Unable to read consumer identity
   
   This system is not registered with an entitlement server. You can use subscription-manager to register.
   
   Last metadata expiration check: 0:07:12 ago on Wed 19 Nov 2025 12:26:56 AM EST.
   Dependencies resolved.
   ===================================================================================================
    Package                                       Architecture   Version     Repository          Size
   ===================================================================================================
   Installing:
    datacenter-gpu-manager-4-cuda12               x86_64         1:4.4.2-1   cuda-rhel8-x86_64   554 M
   Installing dependencies:
    datacenter-gpu-manager-4-core                 x86_64         1:4.4.2-1   cuda-rhel8-x86_64   9.9 M
   Installing weak dependencies:
    datacenter-gpu-manager-4-proprietary          x86_64         1:4.4.2-1   cuda-rhel8-x86_64   5.3 M
    datacenter-gpu-manager-4-proprietary-cuda12   x86_64         1:4.4.2-1   cuda-rhel8-x86_64   289 M
   
   Transaction Summary
   ====================================================================================================
   Install  4 Packages
   ...
   Is this ok [y/N]: y

datacenter-gpu-manager-exporter installation (for RHEL)

It is a tool that, based on NVIDIA Data Center GPU Manager (DCGM), collects various GPU metrics such as GPU usage, memory usage, temperature, and power consumption, and exposes them for use in monitoring systems like Prometheus.

1. Add the CUDA Repository to DNF. 1. (If you have already performed this command, proceed to the next step.)

   • Tab 0

   Sample Code Download

   Copy Code

   Color mode

   dnf config-manager --add-repo https://developer.download.nvidia.com/compute/cuda/repos/rhel8/x86_64/cuda-rhel8.repo

   dnf config-manager --add-repo https://developer.download.nvidia.com/compute/cuda/repos/rhel8/x86_64/cuda-rhel8.repo

2. Check the CUDA version. 2. (If you have already performed this command, proceed to the next step.)

   • Tab 0

   Sample Code Download

   Copy Code

   Color mode

   nvidia-smi | grep CUDA

   nvidia-smi | grep CUDA

   • Tab 0

   Sample Code Download

   Copy Code

   Color mode

   | NVIDIA-SMI 535.183.06             Driver Version: 535.183.06     CUDA Version: 12.2 |

   | NVIDIA-SMI 535.183.06             Driver Version: 535.183.06     CUDA Version: 12.2 |

   • Tab 0

   Sample Code Download

   Copy Code

   Color mode

   CUDA_VERSION=12

   CUDA_VERSION=12

3. Check the list of datacenter-gpu-manager-exporter modules.

   • Tab 0

   Sample Code Download

   Copy Code

   Color mode

   dnf list datacenter-gpu-manager-exporter --showduplicates

   dnf list datacenter-gpu-manager-exporter --showduplicates

   • Tab 0

   Sample Code Download

   Copy Code

   Color mode

   Updating Subscription Management repositories.
   Unable to read consumer identity
   
   This system is not registered with an entitlement server. You can use subscription-manager to register.
   
   Last metadata expiration check: 0:02:11 ago on Wed 19 Nov 2025 12:26:56 AM EST.
   Available Packages
   datacenter-gpu-manager-exporter.x86_64   4.0.1-1   cuda-rhel8-x86_64
   datacenter-gpu-manager-exporter.x86_64   4.1.0-1   cuda-rhel8-x86_64
   datacenter-gpu-manager-exporter.x86_64   4.1.1-1   cuda-rhel8-x86_64
   datacenter-gpu-manager-exporter.x86_64   4.1.3-1   cuda-rhel8-x86_64
   datacenter-gpu-manager-exporter.x86_64   4.5.0-1   cuda-rhel8-x86_64
   datacenter-gpu-manager-exporter.x86_64   4.5.1-1   cuda-rhel8-x86_64
   datacenter-gpu-manager-exporter.x86_64   4.5.2-1   cuda-rhel8-x86_64
   datacenter-gpu-manager-exporter.x86_64   4.6.0-1   cuda-rhel8-x86_64

   Updating Subscription Management repositories.
   Unable to read consumer identity
   
   This system is not registered with an entitlement server. You can use subscription-manager to register.
   
   Last metadata expiration check: 0:02:11 ago on Wed 19 Nov 2025 12:26:56 AM EST.
   Available Packages
   datacenter-gpu-manager-exporter.x86_64   4.0.1-1   cuda-rhel8-x86_64
   datacenter-gpu-manager-exporter.x86_64   4.1.0-1   cuda-rhel8-x86_64
   datacenter-gpu-manager-exporter.x86_64   4.1.1-1   cuda-rhel8-x86_64
   datacenter-gpu-manager-exporter.x86_64   4.1.3-1   cuda-rhel8-x86_64
   datacenter-gpu-manager-exporter.x86_64   4.5.0-1   cuda-rhel8-x86_64
   datacenter-gpu-manager-exporter.x86_64   4.5.1-1   cuda-rhel8-x86_64
   datacenter-gpu-manager-exporter.x86_64   4.5.2-1   cuda-rhel8-x86_64
   datacenter-gpu-manager-exporter.x86_64   4.6.0-1   cuda-rhel8-x86_64

4. Install datacenter-gpu-manager-cuda. dcgm-exporter 4.5.X requires glibc 2.34 or newer, but since RHEL9 provides glibc 2.34, we specify the version as 4.1.3-1 for installation.

   • Tab 0

   Sample Code Download

   Copy Code

   Color mode

   dnf install datacenter-gpu-manager-exporter-4.1.3-1

   dnf install datacenter-gpu-manager-exporter-4.1.3-1

   • Tab 0

   Sample Code Download

   Copy Code

   Color mode

   Updating Subscription Management repositories.
   Unable to read consumer identity
   
   This system is not registered with an entitlement server. You can use subscription-manager to register.
   
   Last metadata expiration check: 0:07:12 ago on Wed 19 Nov 2025 12:26:56 AM EST.
   Dependencies resolved.
   ====================================================================================================
    Package                                       Architecture   Version     Repository          Size
   ====================================================================================================
   Installing:
    datacenter-gpu-manager-exporter               x86_64         4.1.3-1     cuda-rhel8-x86_64   26 M
   
   
   Is this ok [y/N]: y

   Updating Subscription Management repositories.
   Unable to read consumer identity
   
   This system is not registered with an entitlement server. You can use subscription-manager to register.
   
   Last metadata expiration check: 0:07:12 ago on Wed 19 Nov 2025 12:26:56 AM EST.
   Dependencies resolved.
   ====================================================================================================
    Package                                       Architecture   Version     Repository          Size
   ====================================================================================================
   Installing:
    datacenter-gpu-manager-exporter               x86_64         4.1.3-1     cuda-rhel8-x86_64   26 M
   
   
   Is this ok [y/N]: y

   • Tab 0

   Sample Code Download

   Copy Code

   Color mode

   cat /usr/lib/systemd/system/nvidia-dcgm-exporter.service | grep ExecStart

   cat /usr/lib/systemd/system/nvidia-dcgm-exporter.service | grep ExecStart

   • Tab 0

   Sample Code Download

   Copy Code

   Color mode

   ExecStart=/usr/bin/dcgm-exporter -f /etc/dcgm-exporter/default-counters.csv

   ExecStart=/usr/bin/dcgm-exporter -f /etc/dcgm-exporter/default-counters.csv

5. When installing the DCGM Exporter, review the provided settings and remove # for required metrics, and add # for unnecessary metrics.

   • Tab 0

   Sample Code Download

   Copy Code

   Color mode

   vi /etc/dcgm-exporter/default-counters.csv
   ## Example ##
   
   DCGM_FI_PROF_PIPE_TENSOR_ACTIVE, gauge, Ratio of cycles the tensor (HMMA) pipe is active.
   DCGM_FI_PROF_DRAM_ACTIVE,        gauge, Ratio of cycles the device memory interface is active sending or receiving data.
   # DCGM_FI_PROF_PIPE_FP64_ACTIVE,   gauge, Ratio of cycles the fp64 pipes are active.
   # DCGM_FI_PROF_PIPE_FP32_ACTIVE,   gauge, Ratio of cycles the fp32 pipes are active.

   vi /etc/dcgm-exporter/default-counters.csv
   ## Example ##
   
   DCGM_FI_PROF_PIPE_TENSOR_ACTIVE, gauge, Ratio of cycles the tensor (HMMA) pipe is active.
   DCGM_FI_PROF_DRAM_ACTIVE,        gauge, Ratio of cycles the device memory interface is active sending or receiving data.
   # DCGM_FI_PROF_PIPE_FP64_ACTIVE,   gauge, Ratio of cycles the fp64 pipes are active.
   # DCGM_FI_PROF_PIPE_FP32_ACTIVE,   gauge, Ratio of cycles the fp32 pipes are active.

Enable and start DCGM service (for RHEL)

1. Enable and start the nvdia-dcgm service.

   • Tab 0

   Sample Code Download

   Copy Code

   Color mode

   systemctl enable --now nvidia-dcgm

   systemctl enable --now nvidia-dcgm

2. Enable and start the nvdia-dcgm-exporter service.

   • Tab 0

   Sample Code Download

   Copy Code

   Color mode

   systemctl enable --now nvidia-dcgm-exporter

   systemctl enable --now nvidia-dcgm-exporter

DCGM Exporter metrics

DCGM Exporter Key Metrics

Among the metrics provided by the DCGM Exporter, the main GPU metrics are listed below.

DCGM Exporter metric collection configuration

Refer to the default metrics configured for DCGM Exporter at DCGM Exporter > 기본 지표.

• In addition to the default settings, remove # from default-counters.csv for the additional metrics to be configured.
• For default metrics you do not wish to collect, add # or delete the corresponding item.

# Format
# If line starts with a '#' it is considered a comment
# DCGM FIELD, Prometheus metric type, help message

# Clocks
DCGM_FI_DEV_SM_CLOCK, gauge, SM clock frequency (in MHz).
DCGM_FI_DEV_MEM_CLOCK, gauge, Memory clock frequency (in MHz).

# Temperature
DCGM_FI_DEV_MEMORY_TEMP, gauge, Memory temperature (in C).
DCGM_FI_DEV_GPU_TEMP,    gauge, GPU temperature (in C).

# Power
DCGM_FI_DEV_POWER_USAGE,              gauge, Power draw (in W).
DCGM_FI_DEV_TOTAL_ENERGY_CONSUMPTION, counter, Total energy consumption since boot (in mJ).

# PCIE
# DCGM_FI_PROF_PCIE_TX_BYTES,  counter, Total number of bytes transmitted through PCIe TX via NVML.
# DCGM_FI_PROF_PCIE_RX_BYTES,  counter, Total number of bytes received through PCIe RX via NVML.
...

# Format
# If line starts with a '#' it is considered a comment
# DCGM FIELD, Prometheus metric type, help message

# Clocks
DCGM_FI_DEV_SM_CLOCK, gauge, SM clock frequency (in MHz).
DCGM_FI_DEV_MEM_CLOCK, gauge, Memory clock frequency (in MHz).

# Temperature
DCGM_FI_DEV_MEMORY_TEMP, gauge, Memory temperature (in C).
DCGM_FI_DEV_GPU_TEMP,    gauge, GPU temperature (in C).

# Power
DCGM_FI_DEV_POWER_USAGE,              gauge, Power draw (in W).
DCGM_FI_DEV_TOTAL_ENERGY_CONSUMPTION, counter, Total energy consumption since boot (in mJ).

# PCIE
# DCGM_FI_PROF_PCIE_TX_BYTES,  counter, Total number of bytes transmitted through PCIe TX via NVML.
# DCGM_FI_PROF_PCIE_RX_BYTES,  counter, Total number of bytes received through PCIe RX via NVML.
...