GPU Server

• 1: Overview
• 1.1: Server type
• 1.2: Monitoring Metrics
• 1.3: ServiceWatch Metrics
• 2: How-to guides
• 2.1: Manage Image
• 2.2: Manage Keypair
• 2.3: Use Multi-instance GPU on GPU Server
• 2.4: Use NVSwitch on GPU Server
• 2.5: Install ServiceWatch Agent
• 3: API Reference
• 4: CLI Reference
• 5: Release Note

1 - Overview

Service Overview

GPU Server is a virtualized computing service that lets you freely allocate and use infrastructure resources such as CPU, GPU, and memory provided by the server, without having to purchase them individually, and allocate as much as needed at the required time. It is suitable for tasks that require fast computation speed, such as AI model experimentation, prediction, and inference in a cloud environment, and you can flexibly select and use resources with optimized performance based on the type and scale of the work. The GPU Server provides the following features.

Provided Features

• GPU Server Management: Through a web-based console, users can directly Self Service create, delete, and modify GPU Server provisioning, monitoring, and billing.
• Product offerings by GPU quantity: Depending on the project’s purpose and scale, you can freely select the number of H100/A100 GPUs to configure a virtual server.
• High‑Performance GPU Provision: We provide a high‑performance GPU server at physical‑server level using a pass‑through method.
• Storage Connection: Provides additional attached storage besides the OS disk. * You can connect and use Block Storage, File Storage, and Object Storage.
• Strong Security Application: Use the Security Group service to control inbound/outbound traffic exchanged with the external internet or other VPC(Virtual Private Cloud), securely protecting the server.
• Monitoring: You can view monitoring information such as the status of computing resources—including CPU, Memory, Disk, and GPU—through the Cloud Monitoring service.
• Network Configuration Management: The server’s subnet/IP can be easily changed from the values set at initial creation. * NAT IP provides a management feature that lets you enable or disable it as needed.
• Key Pair method: To ensure a secure OS access method, we provide a Key Pair method instead of ID/PW login.
• Image Management: You can create and manage Custom Images, and it provides sharing functionality between projects.
• ServiceWatch Service Integration Offering: You can monitor data through the ServiceWatch service.

Component

GPU Server provides GPUs, NVSwitch, and NVLink on top of virtualized computing resources.

Specifications by GPU Type

GPU (Graphic Processing Unit) performs the calculations needed to create images that compose the computer screen, and because it is specialized for parallel processing, it can handle large amounts of data quickly, processing large-scale parallel operations such as artificial intelligence (AI) and data analysis.

The following are the specifications of the GPU Types offered by the GPU Server service.

Server type

The server types offered by the GPU Server are as follows. For detailed information about the server types provided by GPU Server, refer to GPU Server 서버 타입.

OS and GPU driver version

The operating systems (OS) supported by the GPU Server are as follows. Note that GPUs of type B300 are supported only from a specific GPU version onward, so please be careful when selecting images.

Preceding Service

This is a service that must be installed in advance before creating this service. Please prepare by referring to the user guide provided in advance.

1.1 - Server type

GPU Server server type

GPU servers are categorized by the GPU type they provide, and the GPU used in a GPU server is determined by the server type selected when creating the GPU server. Please select the server type based on the specifications of the application you want to run on the GPU server.

The server types supported by the GPU Server are as follows.

GPU-H100-2 g2v12h1

g1 server type

The g1 server type is a GPU Server that uses the NVIDIA A100 Tensor Core GPU, suitable for high-performance applications.

• Up to 8 NVIDIA A100 Tensor Core GPUs provided
• Equipped with 6,912 CUDA cores and 432 Tensor cores per GPU
• Supports up to 128 vCPUs and 1,920 GB of memory
• Networking speed up to 40 Gbps
• 600 GB/s GPU and NVIDIA NVSwitch P2P communication

g2 server type

The g2 server type is a GPU Server that uses the NVIDIA H100 Tensor Core GPU, making it suitable for high-performance applications.

• Up to 8 NVIDIA H100 Tensor Core GPUs provided
• Equipped with 16,896 CUDA cores and 528 Tensor cores per GPU
• Supports up to 96 vCPUs and 1,920 GB of memory
• Networking speed up to 40 Gbps
• 900 GB/s GPU and NVIDIA NVSwitch P2P communication

g3 server type

The g3 server type is a GPU Server that uses the NVIDIA B300 Tensor Core GPU, making it suitable for high-performance applications.

• Up to 8 NVIDIA B300 Tensor Core GPUs provided
• Equipped with 20,480 CUDA cores and 640 Tensor cores per GPU
• Supports up to 128 vCPUs and 3,840 GB of memory.
• Maximum 40 Gbps networking speed
• 1.8 TB/s GPU and NVIDIA NVSwitch P2P communication

1.2 - Monitoring Metrics

GPU Server Monitoring Metrics

The table below shows the monitoring metrics of the GPU server that can be viewed through Cloud Monitoring.

Even without installing the Agent, it provides basic monitoring metrics and the table below. Please check the GPU Server monitoring metrics (provided by default). Additionally, the metrics that can be viewed by installing the Agent are in the table. Please refer to the additional monitoring metrics for GPU Server (Agent installation required).

For detailed usage of Cloud Monitoring, refer to the Cloud Monitoring guide.

1.3 - ServiceWatch Metrics

The GPU Server sends metrics to ServiceWatch. The metrics provided by default monitoring are data collected at 5‑minute intervals. If detailed monitoring is enabled, you can view data collected at 1‑minute intervals.

Refer to How-to guides > ServiceWatch Detailed Monitoring Activation for how to enable detailed monitoring of the GPU Server.

Basic Metrics

The following are the basic metrics for the Virtual Server namespace.

The indicators whose names are displayed in bold below are the key indicators selected among the basic indicators provided by Virtual Server. The key metrics are used to build service dashboards that are automatically created for each service in ServiceWatch.

Each metric indicates through the user guide which statistical value is meaningful to view for that metric, and among the meaningful statistics, the statistical values shown in bold text are the primary statistics. In the service dashboard, you can view primary metrics using the primary statistical values.

2 - How-to guides

Users can create the service by entering the required GPU Server information and selecting detailed options through the Samsung Cloud Platform Console.

Creating a GPU Server

You can create and use a GPU Server service from the Samsung Cloud Platform Console.

To create a GPU server, follow the steps below.

1. Click the All Services > Compute > GPU Server menu. You will be taken to the Service Home page of GPU Server.

2. On the Service Home page, click the GPU Server Creation button. You will be taken to the GPU Server Creation page.

3. On the GPU Server Creation page, enter the information required to create the service and select detailed options.

   • Select the required information in the Image and Version Selection area.

     Category	Required status	Detailed description
     Image	Required	Select the type of Image provided Standard: Samsung Cloud Platform standard provided Image RHEL, Ubuntu Custom: User-created Image Kubernetes: Image for Kubernetes Ubuntu
     Image version	Required	Select the version of the chosen Image Provides a list of versions for the offered server Image For detailed information about the provided server image, see OS and GPU driver versions

     Table. GPU Server image and version selection input fields
   • Service Information Input area, enter or select the required information.

     Category	Required status	Detailed description
     Number of servers	Required	Number of GPU Server servers to create concurrently Only numeric input is allowed, enter a value between 1 and 100
     Service Type > Server Type	Required	GPU Server server type Indicates the specifications of a GPU-type server, allowing selection of servers with 1, 2, 4, or 8 GPUs For detailed information about the server types provided by GPU Server, refer to GPU Server Server Types
     Service Type > Planned Compute	Selection	Resource status with Planned Compute configured In Use: Number of resources with Planned Compute that are currently in use Configured: Number of resources with Planned Compute configured Coverage Preview: Amount applied per resource by Planned Compute Apply for Planned Compute Service: Navigate to the Planned Compute service application page For more details, refer to Apply for Planned Compute
     Block Storage	Required	Configure the Block Storage used by the GPU Server according to its purpose Basic: The area where the OS is installed and used Capacity can be entered in units (the minimum capacity varies depending on the OS image type) RHEL: Values between 3 and 1,536 can be entered Ubuntu: Values between 3 and 1,536 can be entered SSD: High‑performance general volume HDD: General volume SSD/HDD_KMS: Additional encrypted volume using Samsung Cloud Platform KMS (Key Management System) encryption keys Encryption can be applied only at initial creation (cannot be changed after creation) Performance degradation occurs when using the SSD_KMS disk type SSD_Provisioned: SSD volume with configurable IOPS and throughput Additional: Used when the user needs extra space beyond the OS area After selecting Use, enter the storage type and capacity To add storage, click the + button (up to 25 can be added); to delete, click the x button Capacity can be entered in units, with values between 1 and 1,536 Since 1 Unit equals 8 GB, this creates 8 GB to 12,288 GB SSD: High‑performance general volume HDD: General volume SSD/HDD_KMS: Additional encrypted volume using Samsung Cloud Platform KMS (Key Management System) encryption keys Encryption can be applied only at initial creation (cannot be changed after creation) Performance degradation may occur when using the SSD_KMS disk type HDD/SSD_MultiAttach: Volume that can be attached to two or more servers SSD_Provisioned: SSD volume with configurable IOPS and throughput For details on each Block Storage type, refer to Create Block Storage Delete on termination: If Delete on Termination is set to Enabled, the volume is terminated together with the server Volumes with existing snapshots are not deleted even if Delete on termination is set to Enabled A multi‑attach volume can be deleted only when the server being removed is the last remaining server attached to the volume
     Max IOPS	Required	Enter a maximum IOPS value between 5,000~20,000 Can be set only when disk type is SSD_Provisioned
     Max Throughput	Required	Enter the maximum Throughput value between 250~1,000 Disk type can be set only when it is SSD_Provisioned

     Table. GPU Server Service Configuration Items
   • In the Required Information Input area, enter or select the necessary information.

     Category	required or not	Detailed description
     Server name	Required	Enter a name to distinguish the server when the selected number of servers is 1 Set the hostname using the entered server name Enter within 63 characters using letters, numbers, spaces, and special characters (- _)
     Server name Prefix	Required	Enter a Prefix to distinguish each server generated when the selected number of servers is 2 or more Automatically generated in the form of user input value (prefix) + ‘-#’ Enter within 59 characters using letters, numbers, spaces, and special characters (-, _)
     Network Settings > Create New Network Port	Required	Configure the network where the GPU Server will be installed Select a pre‑created VPC. General Subnet: Select a pre‑created general Subnet IP can be set to auto‑generate or manual entry; if manual is chosen, the user can directly input the IP NAT: Available only when there is a single server and the VPC is attached to an Internet Gateway. Checking Use enables selection of a NAT IP NAT IP: Select a NAT IP If no NAT IP is available, click the Create New button to generate a Public IP Click the Refresh button to view and select the created Public IP Creating a Public IP incurs charges according to the Public IP pricing policy Local Subnet (optional): Choose Use for a local Subnet It is not a required element for creating the service A pre‑created local Subnet must be selected IP can be set to auto‑generate or manual entry; selecting Manual allows the user to input the IP directly Security Group: Settings required to access the server Select: Choose a pre‑created Security Group Create New: If no applicable Security Group exists, you can create one separately in the Security Group service Up to 5 can be selected If no Security Group is set, all connections are blocked by default You must configure a Security Group to allow required connections
     Network Settings > Existing Network Port Assignment	Required	Set the network where the GPU server will be installed Select a pre-created VPC General Subnet: Select a pre-created general Subnet and Port NAT: Available only when there is a single server and the VPC is connected to an Internet Gateway. When you check to use it, you can select a NAT IP. NAT IP: Select a NAT IP If there is no NAT IP to select, click the Create New button to generate a Public IP Click the Refresh button to view and select the created Public IP Local Subnet (optional): Select Use for the local Subnet Select a pre-created local Subnet and Port
     Keypair	Required	User authentication methods to use when connecting to the server New creation: Create a new keypair if one is needed Refer to Keypair 생성하기 for how to create a new keypair Default login accounts by OS RHEL: cloud-user Ubuntu: ubuntu

     Table. Required input fields for GPU Server
   • Enter or select the required information in the Additional Information Input area.

     Category	Required	Detailed description
     Lock	Selection	Lock usage setting When Lock is enabled, it prevents actions such as server termination, start, and stop from being executed, thereby avoiding malfunctions caused by mistakes
     Init script	Selection	Script executed when the server starts The init script must be written as a Batch script for Windows, a Shell script for Linux, or cloud‑init, depending on the image type. Up to 45,000 bytes can be entered
     tag	Selection	Add Tag Up to 50 can be added per resource After clicking the Add Tag button, enter or select Key, Value values

     Table. GPU Server additional information input fields

4. Summary Check the detailed information and estimated billing amount generated in the panel, and click the Create button.

   • Once creation is complete, check the created resources on the GPU Server List page.

Check GPU Server detailed information

GPU Server service allows you to view and edit the full resource list and detailed information. GPU Server Details page consists of Details, Tags, Job History tabs.

To view detailed information about the GPU Server service, follow these steps.

1. Click the All Services > Compute > GPU Server menu. You will be taken to the Service Home page of GPU Server.
2. On the Service Home page, click the GPU Server menu. You will be taken to the GPU Server List page.
3. On the GPU Server List page, click the resource you want to view details for. You will be taken to the GPU Server Details page.
   • GPU Server Details page displays status information and additional feature information, and is composed of Details, Tags, Activity History tabs.
   • GPU Server Additional Features for detailed information, please refer to GPU Server Management Additional Features.

     Category	Detailed description
     GPU Server status	Status of user-created GPU Server Build: State where the Build command has been received Building: Build in progress Networking: Server creation networking process Scheduling: Server creation scheduling process Block_Device_Mapping: Connecting Block Storage during server creation Spawning: Server creation process is ongoing Active: Available state Powering_off: State when a stop request is made Deleting: Server deletion in progress Reboot_Started: Reboot in progress Error: Error state Migrating: Server migrating to another host Reboot: State where the Reboot command has been received Rebooting: Rebooting Rebuild: State where the Rebuild command has been received Rebuilding: State during Rebuild request Rebuild_Spawning: Rebuild process is ongoing Resize: State where the Resize command has been received Resizing: Resizing in progress Resize_Prep: State when a server type change is requested Resize_Migrating: Server is migrating to another host while resizing Resize_Migrated: Server has completed migration to another host during resizing Resize_Finish: Resize completed Revert_Resize: Resize or migration of the server failed for some reason. The target server is cleaned up and the original server is restarted Shutoff: State when powering off is completed Verity_Resize: After Resize_Prep following a server type change request, the server type is confirmed or can be reverted Resize_Reverting: State when a server type revert is requested Resize_Confirming: State confirming the server’s Resize request
     Server control	Buttons to change server status Start: start a stopped server Stop: stop a running server Restart: restart a running server
     Image generation	Create a custom user image from the current server image
     Console log	View console logs of the current server You can view the console logs output by the current server. For more details, see Check console logs.
     Create dump	Generate a dump of the current server The dump file is created inside the GPU Server For detailed dump creation instructions, refer to Create Dump
     Rebuild	All data and settings of the existing server are deleted, and a new server is configured For detailed information, see Execute Rebuild.
     Service termination	Cancel service button

     Table. GPU Server status information and additional features

Detailed Information

GPU Server List page lets you view detailed information of the selected resource and edit it if needed.

tag

GPU Server List page allows you to view the tag information of the selected resource, and you can add, modify, or delete it.

Job History

You can view the job history of the selected resource on the GPU Server List page.

Control GPU Server Operation

If you need to control the operation of a created GPU Server resource, you can perform the task on the GPU Server List or GPU Server Details page. You can start, stop, and restart a running server.

Getting Started with GPU Server

You can start a shutoff GPU Server. To start a GPU Server, follow the steps below.

1. Click the All Services > Compute > GPU Server menu. You will be taken to the Service Home page of the GPU Server.
2. On the Service Home page, click the GPU Server menu. You will be taken to the GPU Server List page.
3. On the GPU Server List page, click the resource you want to start among the shutoff servers, and navigate to the GPU Server Details page.
   • GPU Server List page allows you to Start each resource via the right More button.
   • After selecting multiple servers with checkboxes, you can control multiple servers simultaneously using the Start button at the top.
4. GPU Server Details page, click the Start button at the top to start the server. In the Status Display section, verify the updated server status.
   • When the GPU Server startup is complete, the server status changes from Shutoff to Active.
   • For detailed information about the GPU Server status, refer to GPU Server detailed information.

Stopping the GPU Server

You can stop a running (Active) GPU Server. To stop the GPU Server, follow the steps below.

1. Click the All Services > Compute > GPU Server menu. You will be taken to the Service Home page of GPU Server.
2. On the Service Home page, click the GPU Server menu. You will be taken to the GPU Server List page.
3. On the GPU Server List page, click the resource to stop among the servers that are running (Active), and navigate to the GPU Server Details page.
   • On the GPU Server List page, you can Stop each resource via the right More button.
   • After selecting multiple servers with checkboxes, you can control multiple servers simultaneously using the Stop button at the top.
4. GPU Server Details page, click the Stop button at the top to start the server. Check the changed server status in the Status Display item.
   • When the GPU server shutdown is complete, the server status changes from Active to Shutoff.
   • For detailed information about the GPU Server status, please refer to GPU Server Detailed Information.

Restart GPU Server

You can restart the created GPU Server. To restart the GPU Server, follow the steps below.

1. Click the All Services > Compute > GPU Server menu. You will be taken to the Service Home page of GPU Server.
2. On the Service Home page, click the GPU Server menu. You will be taken to the GPU Server List page.
3. On the GPU Server List page, click the resource to restart and go to the GPU Server Details page.
   • On the GPU Server list page, you can restart each resource using the right More button.
   • After selecting multiple servers with checkboxes, you can control multiple servers simultaneously using the Restart button at the top.
4. GPU Server Details on the page, click the Restart button at the top to start the server. Check the updated server status in the Status Display field.
   • During a GPU Server restart, the server status goes through Rebooting and finally changes to Active.
   • For detailed information about the GPU Server status, refer to GPU Server detailed information.

GPU Server resource management

If you need server control and management functions for the created GPU Server resources, you can perform the tasks on the GPU Server Resource List or GPU Server Details page.

Create Image

You can create an image of a running GPU server.

To create an image of the GPU Server, follow these steps.

1. Click the All Services > Compute > GPU Server menu. Navigate to the Service Home page of the GPU Server.

2. On the Service Home page, click the GPU Server menu. You will be taken to the GPU Server List page.

3. On the GPU Server List page, click the resource to create an Image. You will be taken to the GPU Server Details page.

4. Click the Create Image button on the GPU Server Details page. You will be taken to the Image Creation page.

   • Enter the required information in the Service Information Input area.

     Category	Required status	Detailed description
     image name	Required	Enter the name of the image to be created using English letters, numbers, spaces, and special characters (- _) within 200 characters

     Table. Image service information input fields

5. Check the input information and click the Create button.

   • When creation is complete, check the created resources on the All Services > Compute > GPU Server > Image List page.

Enabling detailed monitoring for ServiceWatch

By default, the GPU Server is linked to the basic monitoring of the ServiceWatch and Virtual Server namespaces. You can enable detailed monitoring as needed to identify operational issues more quickly and take action. For detailed information about ServiceWatch, see ServiceWatch Overview.

To enable detailed ServiceWatch monitoring on the GPU Server, follow these steps.

1. Click the All Services > Compute > GPU Server menu. You will be taken to the Service Home page of GPU Server.
2. On the Service Home page, click the GPU Server menu. You will be taken to the GPU Server List page.
3. On the GPU Server List page, click the resource to enable ServiceWatch detailed monitoring. You will be taken to the GPU Server Details page.
4. On the GPU Server Details page, click the ServiceWatch detailed monitoring Edit button. You will be taken to the ServiceWatch Detailed Monitoring Edit popup.
5. ServiceWatch Detailed Monitoring Edit In the popup window, select Enable, review the instructions, and click the Confirm button.
6. On the GPU Server Details page, check the ServiceWatch detailed monitoring items.

Disable detailed monitoring of ServiceWatch

To disable detailed monitoring of ServiceWatch on the GPU Server, follow these steps.

1. Click the All Services > Compute > GPU Server menu. You will be taken to the Service Home page of GPU Server.
2. On the Service Home page, click the GPU Server menu. You will be taken to the GPU Server List page.
3. GPU Server List page, click the resource to disable ServiceWatch detailed monitoring. Navigate to the GPU Server Details page.
4. GPU Server Details page, click the ServiceWatch detailed monitoring Edit button. You will be taken to the ServiceWatch detailed monitoring Edit popup.
5. ServiceWatch Detailed Monitoring Edit In the popup window, after deselecting Enabled, review the guidance message and click the Confirm button.
6. On the GPU Server Details page, view the ServiceWatch detailed monitoring items.

GPU Server Management Additional Features

GPU Server can view Console logs, generate Dumps, and perform Rebuilds for server management. To view Console logs, generate Dumps, or Rebuild a GPU Server, follow the steps below.

Check console log

You can view the current console log of the GPU Server.

To view the console logs of the GPU server, follow these steps.

1. Click the All Services > Compute > GPU Server menu. You will be taken to the GPU Server’s Service Home page.
2. On the Service Home page, click the GPU Server menu. You will be taken to the GPU Server List page.
3. On the GPU Server List page, click the resource to view the console log. You will be taken to the GPU Server Details page.
4. On the GPU Server Details page, click the Console Log button. You will be taken to the Console Log popup.
5. Console Log Verify the console log displayed in the popup window.

Create Dump

To create a dump file on the GPU Server, follow these steps.

1. Click the All Services > Compute > GPU Server menu. You will be taken to the Service Home page of GPU Server.
2. On the Service Home page, click the GPU Server menu. You will be taken to the GPU Server List page.
3. GPU Server List page, click the resource to view detailed information. You will be taken to the GPU Server Details page.
4. On the GPU Server Details page, click the Create Dump button.
   • The dump file is created inside the GPU server.

Perform Rebuild

You can delete all data and settings of the existing GPU Server and rebuild it on a new server.

Follow these steps to perform a Rebuild of the GPU Server.

1. Click the All Services > Compute > GPU Server menu. You will be taken to the Service Home page of GPU Server.
2. On the Service Home page, click the GPU Server menu. You will be taken to the GPU Server List page.
3. GPU Server List page, click the resource to perform Rebuild. GPU Server Details page will be displayed.
4. On the GPU Server Details page, click the Rebuild button.
   • During a GPU Server Rebuild, the server status changes to Rebuilding, and when the Rebuild is complete, it returns to its state before the Rebuild.
   • For detailed information about the GPU Server status, refer to Check GPU Server details.

Terminate GPU Server

If you terminate an unused GPU Server, you can reduce operating costs. However, terminating a GPU Server may cause the running service to stop immediately, so you should carefully consider the impact of service interruption before proceeding with the termination.

To cancel the GPU Server, follow the steps below.

1. Click the All Services > Compute > GPU Server menu. Go to the GPU Server’s Service Home page.
2. On the Service Home page, click the GPU Server menu. You will be taken to the GPU Server List page.
3. On the GPU Server List page, select the resource to cancel and click the Cancel Service button.
   • The termination of attached storage depends on the Delete on termination setting, so refer to Termination constraints.
4. After termination is complete, check on the GPU Server List page whether the resources have been terminated.

Cancellation constraints

When a GPU Server termination request cannot be processed, a popup will provide guidance. Please refer to the cases below.

Termination of attached storage depends on the Delete on termination setting.

2.1 - Manage Image

Users can create the service by entering the required information for the Image service within the GPU Server service and selecting detailed options through the Samsung Cloud Platform Console.

Create Image

You can create an image of a running GPU Server. To create an image of a GPU Server, please refer to Image Creation.

Check Image detailed information

Image service allows you to view and edit the full resource list and detailed information. The Image Details page consists of Detailed Information, Tags, Operation History tabs.

To view detailed information of the Image service, follow these steps.

1. Click the All Services > Compute > GPU Server menu. You will be taken to the Service Home page of GPU Server.
2. On the Service Home page, click the Image menu. You will be taken to the Image list page.
3. On the Image List page, click the resource to view detailed information. You will be taken to the Image Detail page.
   • Image Details page displays status information and additional feature information, and consists of Details, Tags, Activity Log tabs.

     Category	Detailed description
     Image status	Status of user-created Image Active: Available state Queued: Image has been uploaded and is waiting for processing after creation Importing: Image has been uploaded and is currently being processed after creation
     Share with another account	Image can be shared with another Account The Image’s Visibility must be set to Shared in order to be shared with another Account
     Delete image	Button to delete the Image Once the Image is deleted, it cannot be restored

     Table. GPU Server Image status information and additional features

Detailed Information

Image list page lets you view detailed information of the selected resource and modify it if necessary.

tag

On the Image List page, you can view the tag information of the selected resource and add, modify, or delete it.

Job History

You can view the operation history of the selected resource on the Image List page.

Image Resource Management

Describes the control and management functions of the generated Image.

Share to another Account

To share an Image with another Account, follow the steps below.

1. Log in to the account to be shared and click the All Services > Compute > GPU Server menu. Go to the GPU Server’s Service Home page.
2. On the Service Home page, click the Image menu. You will be taken to the Image List page.
3. On the Image List page, click the Image you want to control. You will be taken to the Image Details page.
4. Click the Share to another Account button. Navigate to the Share image to another Account page.
   • Share with another Account feature allows you to share an Image with another Account. To share an Image with another Account, the Image’s Visibility must be Shared.
5. Share image to another Account page, enter the required information, and click the Done button.

   Category	Required	Detailed description
   image name	-	Name of the image to share Input not allowed
   Image ID	-	Shareable image ID Input not allowed
   Shared Account ID	Required	Enter another Account ID to share English letters, numbers, special characters- within 64 characters

   Table. Required input fields for sharing images to another Account
6. You can view the information in the sharing status of the Image Details page.
   • When the request is first made, the status is Pending, and it changes to Accepted once approval is completed by the account receiving the share.

Receive sharing from another Account

To receive an Image shared from another Account, follow these steps.

1. Log in to the account to be shared and click the All Services > Compute > GPU Server menu. Navigate to the GPU Server’s Service Home page.
2. On the Service Home page, click the Image menu. You will be taken to the Image List page.
3. On the Image List page, click the Receive Image Share button. You will be taken to the Receive Image Share popup.
4. Receive Image Sharing In the popup window, enter the Image’s resource ID you want to receive, and click the Confirm button.
5. When image sharing is complete, you can view the shared Image in the Image list.

Delete Image

You can delete unused Images. However, since an Image cannot be recovered after deletion, you should carefully consider the impact before performing the deletion.

To delete the Image, follow these steps.

1. Click the All Services > Compute > GPU Server menu. Go to the GPU Server’s Service Home page.
2. On the Service Home page, click the Image menu. You will be taken to the Image List page.
3. Image list page, select the resource to delete, and click the Delete button.
   • On the Image List page, select multiple Image check boxes and click the Delete button at the top of the resource list.
4. After deletion is complete, verify on the Image list page that the resource has been removed.

2.2 - Manage Keypair

Users can create the service by entering the required Keypair information within the GPU Server service and selecting detailed options through the Samsung Cloud Platform Console.

Create a Keypair

You can create and use the Keypair service while using the GPU Server service in the Samsung Cloud Platform Console.

To create a keypair, follow these steps.

1. Click the All Services > Compute > GPU Server menu. You will be taken to the Service Home page of GPU Server.
2. On the Service Home page, click the Keypair menu. You will be taken to the Keypair List page.
3. On the Keypair List page, click the Keypair Create button. You will be taken to the Keypair Create page.
   • Enter the required information in the Service Information Input area.

     Category	Required	Detailed description
     Keypair name	Required	Enter the name of the Keypair to create using English letters, numbers, spaces, and special characters (-, _) within 255 characters
     Keypair type	Required	ssh

     Table. Keypair service information input fields
   • Additional Information Input area, please enter or select the required information.

     Category	Required status	Detailed description
     tag	Selection	Add Tag Up to 50 can be added per resource After clicking the Add Tag button, enter or select Key, Value values

     Table. Keypair additional information input fields
     Caution

     • After creation is complete, you can download the Key only once. Since reissuance is not possible, make sure it has been downloaded.
     • Store the downloaded Private Key in a safe place.
4. Check the input information and click the Create button.
   • After creation is complete, check the created resources on the Keypair List page.

View detailed information of the Keypair

The Keypair service allows you to view and edit the full resource list and detailed information. Keypair Details page consists of Details, Tags, Activity Log tabs.

To view detailed information about a keypair, follow these steps.

1. Click the All Services > Compute > GPU Server menu. You will be taken to the Service Home page of GPU Server.
2. On the Service Home page, click the Keypair menu. You will be taken to the Keypair List page.
3. On the Keypair List page, click the resource to view its details. You will be taken to the Keypair Details page.
   • Keypair Details page displays status information and additional feature information, and consists of Details, Tags, Activity Log tabs.

Detailed Information

Keypair List page allows you to view detailed information of the selected resource and edit the information if needed.

tag

On the Keypair List page, you can view the tag information of the selected resource, and add, modify, or delete it.

Job History

On the Keypair List page, you can view the operation history of the selected resource.

Keypair Resource Management

Describes the control and management functions of a keypair.

Get public key

To retrieve the public key, follow these steps.

1. Click the All Services > Compute > GPU Server menu. You will be taken to the Service Home page of GPU Server.

2. On the Service Home page, click the Keypair menu. You will be taken to the Keypair List page.

3. On the Keypair List page, click the More button at the top and then click the Import Public Key button. You will be taken to the Import Public Key page.

   • Enter or select the required information in the Required Information Input area.

     Category	Required	Detailed description
     Keypair name	Required	Name of the Keypair to create
     Keypair type	Required	ssh
     public key	Required	Enter public key Load file: Select the Attach file button to attach the public key file Only files with the following extension (.pem) can be attached Enter public key: Paste the copied public key value The public key value can be copied from the Keypair Details page

     Table. Required input fields for retrieving the public key

4. Review the entered information and click the Complete button.

   • Once creation is complete, check the created resources on the Keypair List page.

Delete Keypair

You can delete unused Keypairs. However, once a Keypair is deleted it cannot be recovered, so please review the impact thoroughly beforehand before proceeding with deletion.

To delete a keypair, follow these steps.

1. Click the All Services > Compute > GPU Server menu. Go to the Service Home page of GPU Server.
2. On the Service Home page, click the Keypair menu. You will be taken to the Keypair List page.
3. On the Keypair List page, select the resource to delete, and click the Delete button.

• On the Keypair List page, select multiple Keypair check boxes and click the Delete button at the top of the resource list.

4. After deletion is complete, check the Keypair List page to confirm that the resource has been removed.

2.3 - Use Multi-instance GPU on GPU Server

After creating a GPU Server, you can enable the MIG (Multi-instance GPU) feature on the GPU Server’s VM (Guest OS) and create an instance for use.

NVIDIA Multi-instance GPU Introduction

NVIDIA Multi-instance GPU (hereafter referred to as MIG) supports safely partitioning a GPU into GPU instances and running CUDA applications starting with the NVIDIA Ampere architecture. Through this, multiple users can each utilize different GPU resources to achieve optimal GPU utilization. This feature is especially useful for workloads that do not fully utilize the GPU’s computing capacity, and users can run multiple workloads in parallel to maximize utilization.

Using Multi-instance GPU feature

To use the MIG feature, create an NVIDIA GPU Server on the Samsung Cloud Platform, then enable and disable MIG. The order of applying and removing MIG is as follows.

Applying and Using MIG

After activating the MIG and creating an Instance to assign tasks, the tasks proceed in the following order.

Activate MIG

1. Check the GPU status on the VM Instance (GuestOS) before applying MIG.

   • Check whether MIG mode is Disabled.
     Sample Code Download

     Copy Code

     Color mode

     $ nvidia-smi
     Mon Sep 27 08:37:08 2021
     +-----------------------------------------------------------------------------+
     | NVIDIA-SMI 470.57.02    Driver Version: 470.57.02    CUDA Version: 11.4 |
     | -------------------------------+----------------------+---------------------- |
     | GPU  Name        Persistence-M | Bus-Id        Disp.A | Volatile Uncorr. ECC |
     | Fan  Temp  Perf  Pwr:Usage/Cap | Memory-Usage | GPU-Util  Compute M. |
     |  |  | MIG M. |
     | ===============================+======================+====================== |
     | 0  NVDIA A100-SXM...  Off | 00000000:05:00.0 Off | 0 |
     | N/A   32C   P0    59W / 400W | 0MiB / 81251MiB | 0%      Default |
     |  |  | Disabled |
     +-------------------------------+----------------------+----------------------+
     
     +-----------------------------------------------------------------------------+
     | Processes: |
     | GPU   GI   CI       PID   Type   Process name                   GPU Memory |
     | ID   ID                                                   Usage |
     | ============================================================================= |
     | No running processes found |
     +-----------------------------------------------------------------------------+

     $ nvidia-smi
     Mon Sep 27 08:37:08 2021
     +-----------------------------------------------------------------------------+
     | NVIDIA-SMI 470.57.02    Driver Version: 470.57.02    CUDA Version: 11.4 |
     | -------------------------------+----------------------+---------------------- |
     | GPU  Name        Persistence-M | Bus-Id        Disp.A | Volatile Uncorr. ECC |
     | Fan  Temp  Perf  Pwr:Usage/Cap | Memory-Usage | GPU-Util  Compute M. |
     |  |  | MIG M. |
     | ===============================+======================+====================== |
     | 0  NVDIA A100-SXM...  Off | 00000000:05:00.0 Off | 0 |
     | N/A   32C   P0    59W / 400W | 0MiB / 81251MiB | 0%      Default |
     |  |  | Disabled |
     +-------------------------------+----------------------+----------------------+
     
     +-----------------------------------------------------------------------------+
     | Processes: |
     | GPU   GI   CI       PID   Type   Process name                   GPU Memory |
     | ID   ID                                                   Usage |
     | ============================================================================= |
     | No running processes found |
     +-----------------------------------------------------------------------------+

     Sample Code Download

     Copy Code

     Color mode

     $ nvidia-smi –L
     GPU 0: NVIDIA A100-SXM-80GB (UUID: GPU-c956838f-494a-92b2-6818-56eb28fe25e0)

     $ nvidia-smi –L
     GPU 0: NVIDIA A100-SXM-80GB (UUID: GPU-c956838f-494a-92b2-6818-56eb28fe25e0)

2. Enable MIG (Enable) for each GPU on the VM Instance (GuestOS) and reboot the VM Instance.

   Sample Code Download

   Copy Code

   Color mode

   $ nvidia-smi –I 0 –mig 1
   Enabled MIG mode for GPU 00000000:05:00.0
   All done.
   
   # reboot

   $ nvidia-smi –I 0 –mig 1
   Enabled MIG mode for GPU 00000000:05:00.0
   All done.
   
   # reboot

Warning: MIG mode is in pending enable state for GPU 00000000:05:00.0: In use by another client. 00000000:05:00.0 is currently being used by one or more other processes (e.g. CUDA application or a monitoring application such as another instance of nvidia-smi).

3. Check the GPU status after applying MIG on the VM Instance(GuestOS).
   • Check whether MIG mode is Enabled.
     Sample Code Download

     Copy Code

     Color mode

     $ nvidia-smi
     Mon Sep 27 09:44:33 2021
     +-----------------------------------------------------------------------------+
     | NVIDIA-SMI 470.57.02    Driver Version: 470.57.02    CUDA Version: 11.4 |
     | -------------------------------+----------------------+---------------------- |
     | GPU  Name        Persistence-M | Bus-Id        Disp.A | Volatile Uncorr. ECC |
     | Fan  Temp  Perf  Pwr:Usage/Cap | Memory-Usage | GPU-Util  Compute M. |
     |  |  | MIG M. |
     | ===============================+======================+====================== |
     | 0  NVDIA A100-SXM...  Off | 00000000:05:00.0 Off | On |
     | N/A   32C   P0    59W / 400W | 0MiB / 81251MiB | 0%      Default |
     |  |  | Enabled |
     +-------------------------------+----------------------+----------------------+
     +-----------------------------------------------------------------------------+
     | MIG devices: |
     +-----------------------------------------------------------------------------+
     | GPU  GI  CI  MIG | Memory-Usage | Vol | Shared |
     | ID  ID  Dev | BAR1-Usage | SM     Unc | CE  ENC  DEC  OFA  JPG |
     |  |  | ECC |  |
     | ============================================================================= |
     | No MIG devices found |
     +-----------------------------------------------------------------------------+
     +-----------------------------------------------------------------------------+
     | Processes: |
     | GPU   GI   CI       PID   Type   Process name                   GPU Memory |
     | ID   ID                                                   Usage |
     | ============================================================================= |
     | No running processes found |
     +-----------------------------------------------------------------------------+

     $ nvidia-smi
     Mon Sep 27 09:44:33 2021
     +-----------------------------------------------------------------------------+
     | NVIDIA-SMI 470.57.02    Driver Version: 470.57.02    CUDA Version: 11.4 |
     | -------------------------------+----------------------+---------------------- |
     | GPU  Name        Persistence-M | Bus-Id        Disp.A | Volatile Uncorr. ECC |
     | Fan  Temp  Perf  Pwr:Usage/Cap | Memory-Usage | GPU-Util  Compute M. |
     |  |  | MIG M. |
     | ===============================+======================+====================== |
     | 0  NVDIA A100-SXM...  Off | 00000000:05:00.0 Off | On |
     | N/A   32C   P0    59W / 400W | 0MiB / 81251MiB | 0%      Default |
     |  |  | Enabled |
     +-------------------------------+----------------------+----------------------+
     +-----------------------------------------------------------------------------+
     | MIG devices: |
     +-----------------------------------------------------------------------------+
     | GPU  GI  CI  MIG | Memory-Usage | Vol | Shared |
     | ID  ID  Dev | BAR1-Usage | SM     Unc | CE  ENC  DEC  OFA  JPG |
     |  |  | ECC |  |
     | ============================================================================= |
     | No MIG devices found |
     +-----------------------------------------------------------------------------+
     +-----------------------------------------------------------------------------+
     | Processes: |
     | GPU   GI   CI       PID   Type   Process name                   GPU Memory |
     | ID   ID                                                   Usage |
     | ============================================================================= |
     | No running processes found |
     +-----------------------------------------------------------------------------+

     Sample Code Download

     Copy Code

     Color mode

     $ nvidia-smi –L
     GPU 0: NVIDIA A100-SXM-80GB (UUID: GPU-c956838f-494a-92b2-6818-56eb28fe25e0)

     $ nvidia-smi –L
     GPU 0: NVIDIA A100-SXM-80GB (UUID: GPU-c956838f-494a-92b2-6818-56eb28fe25e0)

GPU Instance creation

If you have enabled MIG and verified its status, you can create a GPU Instance.

1. Check the list of MIG GPU Instance profiles that can be created.
   Sample Code Download

   Copy Code

   Color mode

   $ nvidia-smi mig -i [GPU ID] -lgip

   $ nvidia-smi mig -i [GPU ID] -lgip

$ nvidia-smi mig -i 0 -lgip
+-----------------------------------------------------------------------------+
| GPU instance profiles: |
| GPU   Name             ID    Instances   Memory     P2P    SM    DEC   ENC |
| Free/Total   GiB              CE    JPEG  OFA |
| ============================================================================= |
| 0 MIG 1g.10gb        19    7/7         9.50       No     14     0     0 |
| 1     0     0 |
+-----------------------------------------------------------------------------+
| 0 MIG 1g.10gb+me     20    1/1         9.50       No     14     0     0 |
| 1     1     1 |
+-----------------------------------------------------------------------------+
| 0 MIG 2g.20gb        14    3/3         19.50      No     28     1     0 |
| 2     0     0 |
+-----------------------------------------------------------------------------+
| 0 MIG 3g.40gb         9    2/2         39.50      No     42     2     0 |
| 3     0     0 |
+-----------------------------------------------------------------------------+
| 0 MIG 4g.40gb         5    1/1         39.50      No     56     2     0 |
| 4     0     0 |
+-----------------------------------------------------------------------------+
| 0 MIG 7g.80gb         0    1/1         79.25      No     98     0     0 |
| 7     1     1 |
+-----------------------------------------------------------------------------+

$ nvidia-smi mig -i 0 -lgip
+-----------------------------------------------------------------------------+
| GPU instance profiles: |
| GPU   Name             ID    Instances   Memory     P2P    SM    DEC   ENC |
| Free/Total   GiB              CE    JPEG  OFA |
| ============================================================================= |
| 0 MIG 1g.10gb        19    7/7         9.50       No     14     0     0 |
| 1     0     0 |
+-----------------------------------------------------------------------------+
| 0 MIG 1g.10gb+me     20    1/1         9.50       No     14     0     0 |
| 1     1     1 |
+-----------------------------------------------------------------------------+
| 0 MIG 2g.20gb        14    3/3         19.50      No     28     1     0 |
| 2     0     0 |
+-----------------------------------------------------------------------------+
| 0 MIG 3g.40gb         9    2/2         39.50      No     42     2     0 |
| 3     0     0 |
+-----------------------------------------------------------------------------+
| 0 MIG 4g.40gb         5    1/1         39.50      No     56     2     0 |
| 4     0     0 |
+-----------------------------------------------------------------------------+
| 0 MIG 7g.80gb         0    1/1         79.25      No     98     0     0 |
| 7     1     1 |
+-----------------------------------------------------------------------------+

2. After creating a MIG GPU Instance, check it.

   • Create GPU Instance

     Sample Code Download

     Copy Code

     Color mode

     $ nvidia-smi mig -i [GPU ID] -cgi [Profile ID]

     $ nvidia-smi mig -i [GPU ID] -cgi [Profile ID]

     Sample Code Download

     Copy Code

     Color mode

     $ nvidia-smi mig -i 0 -cgi 0
     Successfully created GPU instance ID 0 on GPU 0 using profile MIG 7g.80gb (ID 0)

     $ nvidia-smi mig -i 0 -cgi 0
     Successfully created GPU instance ID 0 on GPU 0 using profile MIG 7g.80gb (ID 0)

   • Check GPU Instance

     Sample Code Download

     Copy Code

     Color mode

     $ nvidia-smi mig -i [GPU ID] -lgi

     $ nvidia-smi mig -i [GPU ID] -lgi

     Sample Code Download

     Copy Code

     Color mode

     $ nvidia-smi mig -i 0 -lgi
     +--------------------------------------------------------+
     | GPU instances: |
     | GPU   Name               Profile  Instance  Placement |
     | ID       ID      Start:Size |
     | ======================================================== |
     | 0  MIG 7g.80gb            0        0         0:8 |
     +--------------------------------------------------------+

     $ nvidia-smi mig -i 0 -lgi
     +--------------------------------------------------------+
     | GPU instances: |
     | GPU   Name               Profile  Instance  Placement |
     | ID       ID      Start:Size |
     | ======================================================== |
     | 0  MIG 7g.80gb            0        0         0:8 |
     +--------------------------------------------------------+

Compute Instance creation

If you have created a GPU Instance, you can create a Compute Instance.

1. Check the MIG Compute Instance profiles you can create.

   Sample Code Download

   Copy Code

   Color mode

   $ nvidia-smi mig -i [GPU ID] -gi [GPU Instance ID] -lcip

   $ nvidia-smi mig -i [GPU ID] -gi [GPU Instance ID] -lcip

   Sample Code Download

   Copy Code

   Color mode

   $ nvidia-smi mig -i 0 -gi 0 -lcip
   +---------------------------------------------------------------------------------+
   | Compute instance profiles: |
   | GPU     GPU     Name            Profile  Instances   Exclusive      Shared |
   | GPU   Instance                     ID    Free/Total     SM       DEC  ENC  OFA |
   | ID                                                       CE   JPEG |
   | ================================================================================= |
   | 0      0      MIG 1c.7g.80gb     0      7/5           14       5    0    1 |
   | 7    1 |
   +---------------------------------------------------------------------------------+
   | 0      0      MIG 2c.7g.80gb     1      3/3           28       5    0    1 |
   | 7    1 |
   +---------------------------------------------------------------------------------+
   | 0      0      MIG 3c.7g.80gb     2      2/2           42       5    0    1 |
   | 7    1 |
   +---------------------------------------------------------------------------------+
   | 0      0      MIG 4c.7g.80gb     3      1/1           56       5    0    1 |
   | 7    1 |
   +---------------------------------------------------------------------------------+
   | 0      0      MIG 7g.80gb        4*     1/1           98       5    0    1 |
   | 7    1 |
   +---------------------------------------------------------------------------------+

   $ nvidia-smi mig -i 0 -gi 0 -lcip
   +---------------------------------------------------------------------------------+
   | Compute instance profiles: |
   | GPU     GPU     Name            Profile  Instances   Exclusive      Shared |
   | GPU   Instance                     ID    Free/Total     SM       DEC  ENC  OFA |
   | ID                                                       CE   JPEG |
   | ================================================================================= |
   | 0      0      MIG 1c.7g.80gb     0      7/5           14       5    0    1 |
   | 7    1 |
   +---------------------------------------------------------------------------------+
   | 0      0      MIG 2c.7g.80gb     1      3/3           28       5    0    1 |
   | 7    1 |
   +---------------------------------------------------------------------------------+
   | 0      0      MIG 3c.7g.80gb     2      2/2           42       5    0    1 |
   | 7    1 |
   +---------------------------------------------------------------------------------+
   | 0      0      MIG 4c.7g.80gb     3      1/1           56       5    0    1 |
   | 7    1 |
   +---------------------------------------------------------------------------------+
   | 0      0      MIG 7g.80gb        4*     1/1           98       5    0    1 |
   | 7    1 |
   +---------------------------------------------------------------------------------+

2. Create and verify a MIG Compute Instance.

   • MIG Compute Instance creation
     Sample Code Download

     Copy Code

     Color mode

     $ nvidia-smi mig -i [GPU ID] -gi [GPU Instance ID] -cci [Compute Profile ID]

     $ nvidia-smi mig -i [GPU ID] -gi [GPU Instance ID] -cci [Compute Profile ID]

     Sample Code Download

     Copy Code

     Color mode

     $ nvidia-smi mig -i 0 -gi 0 -cci 4
     Successfully created compute instance ID 0 on GPU instance ID 0 using profile MIG 7g.80gb(ID 4)

     $ nvidia-smi mig -i 0 -gi 0 -cci 4
     Successfully created compute instance ID 0 on GPU instance ID 0 using profile MIG 7g.80gb(ID 4)

   • Check MIG Compute Instance
     Sample Code Download

     Copy Code

     Color mode

     $ nvidia-smi mig -i [GPU ID] -gi [GPU Instance ID] –lci

     $ nvidia-smi mig -i [GPU ID] -gi [GPU Instance ID] –lci

     Sample Code Download

     Copy Code

     Color mode

     $ nvidia-smi mig -i 0 -gi 0 –lci
     +-----------------------------------------------------------------+
     | Compute instance profiles: |
     | GPU     GPU     Name            Profile  Instances   Placement |
     | GPU   Instance                     ID      ID        Start:Size |
     | ID |
     | ================================================================= |
     | 0      0      MIG 7g.80gb         4       0            0:7 |
     +-----------------------------------------------------------------+

     $ nvidia-smi mig -i 0 -gi 0 –lci
     +-----------------------------------------------------------------+
     | Compute instance profiles: |
     | GPU     GPU     Name            Profile  Instances   Placement |
     | GPU   Instance                     ID      ID        Start:Size |
     | ID |
     | ================================================================= |
     | 0      0      MIG 7g.80gb         4       0            0:7 |
     +-----------------------------------------------------------------+

     Sample Code Download

     Copy Code

     Color mode

     $ nvidia-smi –L
     GPU 0: NVIDIA A100-SXM-80GB (UUID: GPU-c956838f-494a-92b2-6818-56eb28fe25e0)
       MIG 7g.80gb     Device  0: (UUID: MIG-53e20040-758b-5ecb-948e-c626d03a9a32)

     $ nvidia-smi –L
     GPU 0: NVIDIA A100-SXM-80GB (UUID: GPU-c956838f-494a-92b2-6818-56eb28fe25e0)
       MIG 7g.80gb     Device  0: (UUID: MIG-53e20040-758b-5ecb-948e-c626d03a9a32)

     Sample Code Download

     Copy Code

     Color mode

     $ nvidia-smi
     Mon Sep 27 09:52:17 2021
     +-----------------------------------------------------------------------------+
     | NVIDIA-SMI 470.57.02    Driver Version: 470.57.02    CUDA Version: 11.4 |
     | -------------------------------+----------------------+---------------------- |
     | GPU  Name        Persistence-M | Bus-Id        Disp.A | Volatile Uncorr. ECC |
     | Fan  Temp  Perf  Pwr:Usage/Cap | Memory-Usage | GPU-Util  Compute M. |
     |  |  | MIG M. |
     | ===============================+======================+====================== |
     | 0  NVDIA A100-SXM...  Off | 00000000:05:00.0 Off | On |
     | N/A   32C   P0    49W / 400W | 0MiB / 81251MiB | N/A      Default |
     |  |  | Enabled |
     +-------------------------------+----------------------+----------------------+
     
     +-----------------------------------------------------------------------------+
     | MIG devices: |
     +-----------------------------------------------------------------------------+
     | GPU  GI  CI  MIG | Memory-Usage | Vol | Shared |
     | ID  ID  Dev | BAR1-Usage | SM     Unc | CE  ENC  DEC  OFA  JPG |
     |  |  | ECC |  |
     | ============================================================================= |
     | 0    0   0    0 | 0MiB / 81251MiB | 98      0 | 7   0    5    1    1 |
     |  | 1MiB / 13107... |  |  |
     +-----------------------------------------------------------------------------+
     +-----------------------------------------------------------------------------+
     | Processes: |
     | GPU   GI   CI       PID   Type   Process name                   GPU Memory |
     | ID   ID                                                   Usage |
     | ============================================================================= |
     | No running processes found |
     +-----------------------------------------------------------------------------+

     $ nvidia-smi
     Mon Sep 27 09:52:17 2021
     +-----------------------------------------------------------------------------+
     | NVIDIA-SMI 470.57.02    Driver Version: 470.57.02    CUDA Version: 11.4 |
     | -------------------------------+----------------------+---------------------- |
     | GPU  Name        Persistence-M | Bus-Id        Disp.A | Volatile Uncorr. ECC |
     | Fan  Temp  Perf  Pwr:Usage/Cap | Memory-Usage | GPU-Util  Compute M. |
     |  |  | MIG M. |
     | ===============================+======================+====================== |
     | 0  NVDIA A100-SXM...  Off | 00000000:05:00.0 Off | On |
     | N/A   32C   P0    49W / 400W | 0MiB / 81251MiB | N/A      Default |
     |  |  | Enabled |
     +-------------------------------+----------------------+----------------------+
     
     +-----------------------------------------------------------------------------+
     | MIG devices: |
     +-----------------------------------------------------------------------------+
     | GPU  GI  CI  MIG | Memory-Usage | Vol | Shared |
     | ID  ID  Dev | BAR1-Usage | SM     Unc | CE  ENC  DEC  OFA  JPG |
     |  |  | ECC |  |
     | ============================================================================= |
     | 0    0   0    0 | 0MiB / 81251MiB | 98      0 | 7   0    5    1    1 |
     |  | 1MiB / 13107... |  |  |
     +-----------------------------------------------------------------------------+
     +-----------------------------------------------------------------------------+
     | Processes: |
     | GPU   GI   CI       PID   Type   Process name                   GPU Memory |
     | ID   ID                                                   Usage |
     | ============================================================================= |
     | No running processes found |
     +-----------------------------------------------------------------------------+

Using MIG

1. Use the MIG Instance to perform the Job.
   • Example of task execution
     Sample Code Download

     Copy Code

     Color mode

     $ docker run --gpus '"device=[GPU ID]:[MIG ID]"' -rm nvcr.io/nvidia/cuda nvidia-smi

     $ docker run --gpus '"device=[GPU ID]:[MIG ID]"' -rm nvcr.io/nvidia/cuda nvidia-smi

   • You can see an example of the work performed as follows.
     Sample Code Download

     Copy Code

     Color mode

     $ docker run --gpus '"device=0:0"' -rm -it --network=host --shm-size=1g --ipc=host -v /root/.ssh/:/root/.ssh
     
     ================
     == TensorFlow ==
     ================
     
     NVIDIA Release 21.08-tf1 (build 26012104)
     TensorFlow Version 1.15.5
     
     Container image Copyright (c) 2021, NVIDIA CORPORATION. All right reserved.
     ...
     
     # Run Python process
     root@d622a93c9281:/workspace# python /workspace/nvidia-examples/cnn/resnet.py --num_iter 100
     ...
     PY 3.8.10 (default, Jun 2 2021, 10:49:15)
     [GCC 9.4.0]
     TF 1.15.5
     ...

     $ docker run --gpus '"device=0:0"' -rm -it --network=host --shm-size=1g --ipc=host -v /root/.ssh/:/root/.ssh
     
     ================
     == TensorFlow ==
     ================
     
     NVIDIA Release 21.08-tf1 (build 26012104)
     TensorFlow Version 1.15.5
     
     Container image Copyright (c) 2021, NVIDIA CORPORATION. All right reserved.
     ...
     
     # Run Python process
     root@d622a93c9281:/workspace# python /workspace/nvidia-examples/cnn/resnet.py --num_iter 100
     ...
     PY 3.8.10 (default, Jun 2 2021, 10:49:15)
     [GCC 9.4.0]
     TF 1.15.5
     ...

2. Check the GPU usage. (Create JOB process)
   • When the job runs, you can see that a process is allocated to the MIG device and its utilization increases.
     Sample Code Download

     Copy Code

     Color mode

     $ nvidia-smi mig -i [GPU ID] -gi [GPU Instance ID] -lcip

     $ nvidia-smi mig -i [GPU ID] -gi [GPU Instance ID] -lcip

   • You can check the GPU usage as shown below.
     Sample Code Download

     Copy Code

     Color mode

     +-----------------------------------------------------------------------------+
     | MIG devices: |
     +-----------------------------------------------------------------------------+
     | GPU  GI  CI  MIG | Memory-Usage | Vol | Shared |
     | ID  ID  Dev | BAR1-Usage | SM     Unc | CE  ENC  DEC  OFA  JPG |
     |  |  | ECC |  |
     | ============================================================================= |
     | 0    0   0    0 | 66562MiB / 81251MiB | 98      0 | 7   0    5    1    1 |
     |  | 5MiB / 13107... |  |  |
     +-----------------------------------------------------------------------------+
     +-----------------------------------------------------------------------------+
     | Processes: |
     | GPU   GI   CI       PID   Type   Process name                   GPU Memory |
     | ID   ID                                                   Usage |
     | ============================================================================= |
     | 0     0    0     17483      C   python                           66559MiB |
     +-----------------------------------------------------------------------------+

     +-----------------------------------------------------------------------------+
     | MIG devices: |
     +-----------------------------------------------------------------------------+
     | GPU  GI  CI  MIG | Memory-Usage | Vol | Shared |
     | ID  ID  Dev | BAR1-Usage | SM     Unc | CE  ENC  DEC  OFA  JPG |
     |  |  | ECC |  |
     | ============================================================================= |
     | 0    0   0    0 | 66562MiB / 81251MiB | 98      0 | 7   0    5    1    1 |
     |  | 5MiB / 13107... |  |  |
     +-----------------------------------------------------------------------------+
     +-----------------------------------------------------------------------------+
     | Processes: |
     | GPU   GI   CI       PID   Type   Process name                   GPU Memory |
     | ID   ID                                                   Usage |
     | ============================================================================= |
     | 0     0    0     17483      C   python                           66559MiB |
     +-----------------------------------------------------------------------------+

Delete and release MIG Instance

Follow these steps to delete the MIG instance and detach the MIG.

Compute Instance Delete

• Delete the Compute Instance.
  Sample Code Download

  Copy Code

  Color mode

  $ nvidia-smi mig -i [GPU ID] -gi [GPU Instance ID] –dci
  $ nvidia-smi mig -i [GPU ID] -gi [GPU Instance ID] -ci [Compute Instance] –dci

  $ nvidia-smi mig -i [GPU ID] -gi [GPU Instance ID] –dci
  $ nvidia-smi mig -i [GPU ID] -gi [GPU Instance ID] -ci [Compute Instance] –dci

  Sample Code Download

  Copy Code

  Color mode

  $ nvidia-smi mig -i 0 -gi 0 –lci
  +-----------------------------------------------------------------+
  | Compute instance profiles: |
  | GPU     GPU     Name            Profile  Instances   Placement |
  | GPU   Instance                     ID      ID        Start:Size |
  | ID |
  | ================================================================= |
  | 0      0      MIG 7g.80gb         4       0            0:7 |
  +-----------------------------------------------------------------+

  $ nvidia-smi mig -i 0 -gi 0 –lci
  +-----------------------------------------------------------------+
  | Compute instance profiles: |
  | GPU     GPU     Name            Profile  Instances   Placement |
  | GPU   Instance                     ID      ID        Start:Size |
  | ID |
  | ================================================================= |
  | 0      0      MIG 7g.80gb         4       0            0:7 |
  +-----------------------------------------------------------------+

  Sample Code Download

  Copy Code

  Color mode

  $ nvidia-smi mig -i 0 -gi 0 –dci
  Successfully destroyed compute instance ID  0 from GPU instance ID  0

  $ nvidia-smi mig -i 0 -gi 0 –dci
  Successfully destroyed compute instance ID  0 from GPU instance ID  0

  Sample Code Download

  Copy Code

  Color mode

  $ nvidia-smi mig -i 0 -gi 0 –lci
  No compute instances found: Not found

  $ nvidia-smi mig -i 0 -gi 0 –lci
  No compute instances found: Not found

Delete GPU Instance

• Delete the GPU Instance.
  Sample Code Download

  Copy Code

  Color mode

  $ nvidia-smi mig -i [GPU ID] –dgi
  $ nvidia-smi mig -i [GPU ID] -gi [GPU Instance ID] –dgi

  $ nvidia-smi mig -i [GPU ID] –dgi
  $ nvidia-smi mig -i [GPU ID] -gi [GPU Instance ID] –dgi

  Sample Code Download

  Copy Code

  Color mode

  $ nvidia-smi mig -i 0 -lgi
  +--------------------------------------------------------+
  | GPU instances: |
  | GPU   Name               Profile  Instance  Placement |
  | ID       ID      Start:Size |
  | ======================================================== |
  | 0  MIG 7g.80gb            0        0         0:8 |
  +--------------------------------------------------------+

  $ nvidia-smi mig -i 0 -lgi
  +--------------------------------------------------------+
  | GPU instances: |
  | GPU   Name               Profile  Instance  Placement |
  | ID       ID      Start:Size |
  | ======================================================== |
  | 0  MIG 7g.80gb            0        0         0:8 |
  +--------------------------------------------------------+

  Sample Code Download

  Copy Code

  Color mode

  $ nvidia-smi mig -i 0 -dgi
  Successfully destroyed GPU instance ID  0 from GPU  0

  $ nvidia-smi mig -i 0 -dgi
  Successfully destroyed GPU instance ID  0 from GPU  0

  Sample Code Download

  Copy Code

  Color mode

  $ nvidia-smi mig -i 0 -lgi
  No GPU instances found: Not found

  $ nvidia-smi mig -i 0 -lgi
  No GPU instances found: Not found

Disable MIG feature (deactivation)

• After disabling MIG (Disable), reboot.
  Sample Code Download

  Copy Code

  Color mode

  $ nvidia-smi -mig 0
  Disabled MIG Mode for GPU 00000000:05:00.0
  
  All done.

  $ nvidia-smi -mig 0
  Disabled MIG Mode for GPU 00000000:05:00.0
  
  All done.

  Sample Code Download

  Copy Code

  Color mode

  $ nvidia-smi
  Mon Sep 30 05:18:28 2021
  +-----------------------------------------------------------------------------+
  | NVIDIA-SMI 470.57.02    Driver Version: 470.57.02    CUDA Version: 11.4 |
  | -------------------------------+----------------------+---------------------- |
  | GPU  Name        Persistence-M | Bus-Id        Disp.A | Volatile Uncorr. ECC |
  | Fan  Temp  Perf  Pwr:Usage/Cap | Memory-Usage | GPU-Util  Compute M. |
  |  |  | MIG M. |
  | ===============================+======================+====================== |
  | 0  NVDIA A100-SXM...  Off | 00000000:05:00.0 Off | 0 |
  | N/A   33C   P0    60W / 400W | 0MiB / 81251MiB | 0%      Default |
  |  |  | Disabled |
  +-------------------------------+----------------------+----------------------+
  +-----------------------------------------------------------------------------+
  | MIG devices: |
  +-----------------------------------------------------------------------------+
  | GPU  GI  CI  MIG | Memory-Usage | Vol | Shared |
  | ID  ID  Dev | BAR1-Usage | SM     Unc | CE  ENC  DEC  OFA  JPG |
  |  |  | ECC |  |
  | ============================================================================= |
  | No MIG devices found |
  +-----------------------------------------------------------------------------+
  +-----------------------------------------------------------------------------+
  | Processes: |
  | GPU   GI   CI       PID   Type   Process name                   GPU Memory |
  | ID   ID                                                   Usage |
  | ============================================================================= |
  | No running processes found |
  +-----------------------------------------------------------------------------+

  $ nvidia-smi
  Mon Sep 30 05:18:28 2021
  +-----------------------------------------------------------------------------+
  | NVIDIA-SMI 470.57.02    Driver Version: 470.57.02    CUDA Version: 11.4 |
  | -------------------------------+----------------------+---------------------- |
  | GPU  Name        Persistence-M | Bus-Id        Disp.A | Volatile Uncorr. ECC |
  | Fan  Temp  Perf  Pwr:Usage/Cap | Memory-Usage | GPU-Util  Compute M. |
  |  |  | MIG M. |
  | ===============================+======================+====================== |
  | 0  NVDIA A100-SXM...  Off | 00000000:05:00.0 Off | 0 |
  | N/A   33C   P0    60W / 400W | 0MiB / 81251MiB | 0%      Default |
  |  |  | Disabled |
  +-------------------------------+----------------------+----------------------+
  +-----------------------------------------------------------------------------+
  | MIG devices: |
  +-----------------------------------------------------------------------------+
  | GPU  GI  CI  MIG | Memory-Usage | Vol | Shared |
  | ID  ID  Dev | BAR1-Usage | SM     Unc | CE  ENC  DEC  OFA  JPG |
  |  |  | ECC |  |
  | ============================================================================= |
  | No MIG devices found |
  +-----------------------------------------------------------------------------+
  +-----------------------------------------------------------------------------+
  | Processes: |
  | GPU   GI   CI       PID   Type   Process name                   GPU Memory |
  | ID   ID                                                   Usage |
  | ============================================================================= |
  | No running processes found |
  +-----------------------------------------------------------------------------+

2.4 - Use NVSwitch on GPU Server

After creating a GPU Server, you can enable the NVSwitch feature on the GPU Server’s VM (Guest OS) and use fast GPU-to-GPU P2P communication.

Exploring NVIDIA NVSwitch for Multi GPU

NVLink expands I/O by directly connecting multiple GPUs within a server both bidirectionally and GPU-to-GPU. Using NVSwitch, you can connect all GPUs in a server with full NVLink bandwidth.

Checking NVSwitch operation

Check the NVIDIA Fabric Manager, NVIDIA NVLink topology, and NVIDIA NVLink Status on the GPU server.

NVIDIA Fabric Manager operating status

Verify that active (running) is displayed when operating normally.

~$ systemctl status nvidia-fabricmanager

nvidia-fabricmanager.service - NVIDIA fabric manager service
     Loaded: loaded (/lib/systemd/system/nvidia-fabricmanager.service; enabled; vendor preset: enabled)
     Active: active (running) since Mon 2026-02-02 16:23:27 KST; 32min ago
   Main PID: 2191 (nv-fabricmanage)
      Tasks: 18 (limit: 629145)
     Memory: 18.0M
        CPU: 33.461s
     CGroup: /system.slice/nvidia-fabricmanager.service
             └─2191 /usr/bin/nv-fabricmanager -c /usr/share/nvidia/nvswitch/fabricmanager.cfg

nvidia-fabricmanager.service - NVIDIA fabric manager service
     Loaded: loaded (/lib/systemd/system/nvidia-fabricmanager.service; enabled; vendor preset: enabled)
     Active: active (running) since Mon 2026-02-02 16:23:27 KST; 32min ago
   Main PID: 2191 (nv-fabricmanage)
      Tasks: 18 (limit: 629145)
     Memory: 18.0M
        CPU: 33.461s
     CGroup: /system.slice/nvidia-fabricmanager.service
             └─2191 /usr/bin/nv-fabricmanager -c /usr/share/nvidia/nvswitch/fabricmanager.cfg

Check NVIDIA NVLink topology

Check the NVIDIA NVLink topology.

~$ nvidia-smi topo -m

nvidia-smi topo -m
        GPU0    GPU1    GPU2    GPU3    GPU4    GPU5    GPU6    GPU7    CPU Affinity    NUMA Affinity   GPU NUMA ID
GPU0     X      NV12    NV12    NV12    NV12    NV12    NV12    NV12    0-127   0-7             N/A
GPU1    NV12     X      NV12    NV12    NV12    NV12    NV12    NV12    0-127   0-7             N/A
GPU2    NV12    NV12     X      NV12    NV12    NV12    NV12    NV12    0-127   0-7             N/A
GPU3    NV12    NV12    NV12     X      NV12    NV12    NV12    NV12    0-127   0-7             N/A
GPU4    NV12    NV12    NV12    NV12     X      NV12    NV12    NV12    0-127   0-7             N/A
GPU5    NV12    NV12    NV12    NV12    NV12     X      NV12    NV12    0-127   0-7             N/A
GPU6    NV12    NV12    NV12    NV12    NV12    NV12     X      NV12    0-127   0-7             N/A
GPU7    NV12    NV12    NV12    NV12    NV12    NV12    NV12     X      0-127   0-7             N/A

Legend:

  X    = Self
  SYS  = Connection traversing PCIe as well as the SMP interconnect between NUMA nodes (e.g., QPI/UPI)
  NODE = Connection traversing PCIe as well as the interconnect between PCIe Host Bridges within a NUMA node
  PHB  = Connection traversing PCIe as well as a PCIe Host Bridge (typically the CPU)
  PXB  = Connection traversing multiple PCIe bridges (without traversing the PCIe Host Bridge)
  PIX  = Connection traversing at most a single PCIe bridge
  NV#  = Connection traversing a bonded set of # NVLinks

nvidia-smi topo -m
        GPU0    GPU1    GPU2    GPU3    GPU4    GPU5    GPU6    GPU7    CPU Affinity    NUMA Affinity   GPU NUMA ID
GPU0     X      NV12    NV12    NV12    NV12    NV12    NV12    NV12    0-127   0-7             N/A
GPU1    NV12     X      NV12    NV12    NV12    NV12    NV12    NV12    0-127   0-7             N/A
GPU2    NV12    NV12     X      NV12    NV12    NV12    NV12    NV12    0-127   0-7             N/A
GPU3    NV12    NV12    NV12     X      NV12    NV12    NV12    NV12    0-127   0-7             N/A
GPU4    NV12    NV12    NV12    NV12     X      NV12    NV12    NV12    0-127   0-7             N/A
GPU5    NV12    NV12    NV12    NV12    NV12     X      NV12    NV12    0-127   0-7             N/A
GPU6    NV12    NV12    NV12    NV12    NV12    NV12     X      NV12    0-127   0-7             N/A
GPU7    NV12    NV12    NV12    NV12    NV12    NV12    NV12     X      0-127   0-7             N/A

Legend:

  X    = Self
  SYS  = Connection traversing PCIe as well as the SMP interconnect between NUMA nodes (e.g., QPI/UPI)
  NODE = Connection traversing PCIe as well as the interconnect between PCIe Host Bridges within a NUMA node
  PHB  = Connection traversing PCIe as well as a PCIe Host Bridge (typically the CPU)
  PXB  = Connection traversing multiple PCIe bridges (without traversing the PCIe Host Bridge)
  PIX  = Connection traversing at most a single PCIe bridge
  NV#  = Connection traversing a bonded set of # NVLinks

Check NVIDIA NVLink Status

Check the NVIDIA NVLink Status.

~$ nvidia-smi topo -m

GPU 1: NVIDIA A100-SXM4-80GB (UUID: GPU-64a2f685-bb12-c4af-105c-0726ece9c8d7)
         Link 0: 25 GB/s
         Link 1: 25 GB/s
         Link 2: 25 GB/s
         Link 3: 25 GB/s
         Link 4: 25 GB/s
         Link 5: 25 GB/s
         Link 6: 25 GB/s
         Link 7: 25 GB/s
         Link 8: 25 GB/s
         Link 9: 25 GB/s
         Link 10: 25 GB/s
         Link 11: 25 GB/s
GPU 2: NVIDIA A100-SXM4-80GB (UUID: GPU-2269851b-71cd-f6c7-50c5-ba1525cf3ce8)
         Link 0: 25 GB/s
         Link 1: 25 GB/s
         Link 2: 25 GB/s
         Link 3: 25 GB/s
         Link 4: 25 GB/s
         Link 5: 25 GB/s
         Link 6: 25 GB/s
         Link 7: 25 GB/s
         Link 8: 25 GB/s
         Link 9: 25 GB/s
         Link 10: 25 GB/s
         Link 11: 25 GB/s
GPU 3: NVIDIA A100-SXM4-80GB (UUID: GPU-4c397bbf-95fc-5c29-918a-a429cbe45a7a)
         Link 0: 25 GB/s
         Link 1: 25 GB/s
         Link 2: 25 GB/s
         Link 3: 25 GB/s
         Link 4: 25 GB/s
         Link 5: 25 GB/s
         Link 6: 25 GB/s
         Link 7: 25 GB/s
         Link 8: 25 GB/s
         Link 9: 25 GB/s
         Link 10: 25 GB/s
         Link 11: 25 GB/s
GPU 4: NVIDIA A100-SXM4-80GB (UUID: GPU-0e350204-9fb6-2cbe-538e-8f7849658eb8)
         Link 0: 25 GB/s
         Link 1: 25 GB/s
         Link 2: 25 GB/s
         Link 3: 25 GB/s
         Link 4: 25 GB/s
         Link 5: 25 GB/s
         Link 6: 25 GB/s
         Link 7: 25 GB/s
         Link 8: 25 GB/s
         Link 9: 25 GB/s
         Link 10: 25 GB/s
         Link 11: 25 GB/s
GPU 5: NVIDIA A100-SXM4-80GB (UUID: GPU-45f0c453-4760-edd4-3af9-25c5ea7473a5)
         Link 0: 25 GB/s
         Link 1: 25 GB/s
         Link 2: 25 GB/s
         Link 3: 25 GB/s
         Link 4: 25 GB/s
         Link 5: 25 GB/s
         Link 6: 25 GB/s
         Link 7: 25 GB/s
         Link 8: 25 GB/s
         Link 9: 25 GB/s
         Link 10: 25 GB/s
         Link 11: 25 GB/s
GPU 6: NVIDIA A100-SXM4-80GB (UUID: GPU-38409794-bb34-430e-3c50-90b42cb2bb72)
         Link 0: 25 GB/s
         Link 1: 25 GB/s
         Link 2: 25 GB/s
         Link 3: 25 GB/s
         Link 4: 25 GB/s
         Link 5: 25 GB/s
         Link 6: 25 GB/s
         Link 7: 25 GB/s
         Link 8: 25 GB/s
         Link 9: 25 GB/s
         Link 10: 25 GB/s
         Link 11: 25 GB/s
GPU 7: NVIDIA A100-SXM4-80GB (UUID: GPU-3fb478aa-801b-eb64-55c2-0ffc3f2ce404)
         Link 0: 25 GB/s
         Link 1: 25 GB/s
         Link 2: 25 GB/s
         Link 3: 25 GB/s
         Link 4: 25 GB/s
         Link 5: 25 GB/s
         Link 6: 25 GB/s
         Link 7: 25 GB/s
         Link 8: 25 GB/s
         Link 9: 25 GB/s
         Link 10: 25 GB/s
         Link 11: 25 GB/s

GPU 1: NVIDIA A100-SXM4-80GB (UUID: GPU-64a2f685-bb12-c4af-105c-0726ece9c8d7)
         Link 0: 25 GB/s
         Link 1: 25 GB/s
         Link 2: 25 GB/s
         Link 3: 25 GB/s
         Link 4: 25 GB/s
         Link 5: 25 GB/s
         Link 6: 25 GB/s
         Link 7: 25 GB/s
         Link 8: 25 GB/s
         Link 9: 25 GB/s
         Link 10: 25 GB/s
         Link 11: 25 GB/s
GPU 2: NVIDIA A100-SXM4-80GB (UUID: GPU-2269851b-71cd-f6c7-50c5-ba1525cf3ce8)
         Link 0: 25 GB/s
         Link 1: 25 GB/s
         Link 2: 25 GB/s
         Link 3: 25 GB/s
         Link 4: 25 GB/s
         Link 5: 25 GB/s
         Link 6: 25 GB/s
         Link 7: 25 GB/s
         Link 8: 25 GB/s
         Link 9: 25 GB/s
         Link 10: 25 GB/s
         Link 11: 25 GB/s
GPU 3: NVIDIA A100-SXM4-80GB (UUID: GPU-4c397bbf-95fc-5c29-918a-a429cbe45a7a)
         Link 0: 25 GB/s
         Link 1: 25 GB/s
         Link 2: 25 GB/s
         Link 3: 25 GB/s
         Link 4: 25 GB/s
         Link 5: 25 GB/s
         Link 6: 25 GB/s
         Link 7: 25 GB/s
         Link 8: 25 GB/s
         Link 9: 25 GB/s
         Link 10: 25 GB/s
         Link 11: 25 GB/s
GPU 4: NVIDIA A100-SXM4-80GB (UUID: GPU-0e350204-9fb6-2cbe-538e-8f7849658eb8)
         Link 0: 25 GB/s
         Link 1: 25 GB/s
         Link 2: 25 GB/s
         Link 3: 25 GB/s
         Link 4: 25 GB/s
         Link 5: 25 GB/s
         Link 6: 25 GB/s
         Link 7: 25 GB/s
         Link 8: 25 GB/s
         Link 9: 25 GB/s
         Link 10: 25 GB/s
         Link 11: 25 GB/s
GPU 5: NVIDIA A100-SXM4-80GB (UUID: GPU-45f0c453-4760-edd4-3af9-25c5ea7473a5)
         Link 0: 25 GB/s
         Link 1: 25 GB/s
         Link 2: 25 GB/s
         Link 3: 25 GB/s
         Link 4: 25 GB/s
         Link 5: 25 GB/s
         Link 6: 25 GB/s
         Link 7: 25 GB/s
         Link 8: 25 GB/s
         Link 9: 25 GB/s
         Link 10: 25 GB/s
         Link 11: 25 GB/s
GPU 6: NVIDIA A100-SXM4-80GB (UUID: GPU-38409794-bb34-430e-3c50-90b42cb2bb72)
         Link 0: 25 GB/s
         Link 1: 25 GB/s
         Link 2: 25 GB/s
         Link 3: 25 GB/s
         Link 4: 25 GB/s
         Link 5: 25 GB/s
         Link 6: 25 GB/s
         Link 7: 25 GB/s
         Link 8: 25 GB/s
         Link 9: 25 GB/s
         Link 10: 25 GB/s
         Link 11: 25 GB/s
GPU 7: NVIDIA A100-SXM4-80GB (UUID: GPU-3fb478aa-801b-eb64-55c2-0ffc3f2ce404)
         Link 0: 25 GB/s
         Link 1: 25 GB/s
         Link 2: 25 GB/s
         Link 3: 25 GB/s
         Link 4: 25 GB/s
         Link 5: 25 GB/s
         Link 6: 25 GB/s
         Link 7: 25 GB/s
         Link 8: 25 GB/s
         Link 9: 25 GB/s
         Link 10: 25 GB/s
         Link 11: 25 GB/s

2.5 - Install ServiceWatch Agent

Users can install the ServiceWatch Agent on a GPU server to collect custom metrics and logs.

ServiceWatch Agent

There are two main types of agents that need to be installed on a GPU server to collect custom metrics and logs for ServiceWatch. It is a Prometheus Exporter and Open Telemetry Collector.

Pre-configuration for Using ServiceWatch Agent

To utilize the ServiceWatch Agent, please refer to Prerequisite Settings for ServiceWatch Agent and prepare the prerequisite settings.

Install Prometheus Exporter for GPU metrics (for Ubuntu)

Install the Prometheus Exporter for collecting metrics from the GPU server according to the steps below.

Verify NVDIA Driver Installation

• Check the installed NVDIA Driver.
  • Tab 0

  Sample Code Download

  Copy Code

  Color mode

  nvidia-smi --query-gpu driver_version --format csv

  nvidia-smi --query-gpu driver_version --format csv

  • Tab 0

  Sample Code Download

  Copy Code

  Color mode

  driver_version
  535.183.06
  ...
  535.183.06

  driver_version
  535.183.06
  ...
  535.183.06

NVSwitch Configuration and Query (NSCQ) Library Installation

1. Install cuda-keyring.

   • Tab 0

   Sample Code Download

   Copy Code

   Color mode

   wget https://developer.download.nvidia.com/compute/cuda/repos/<distro>/<arch>/cuda-keyring_1.1-1_all.deb

   wget https://developer.download.nvidia.com/compute/cuda/repos/<distro>/<arch>/cuda-keyring_1.1-1_all.deb

   • Tab 0

   Sample Code Download

   Copy Code

   Color mode

   sudo dpkg -i cuda-keyring_1.1-1_all.deb
   apt update

   sudo dpkg -i cuda-keyring_1.1-1_all.deb
   apt update

   • Tab 0

   Sample Code Download

   Copy Code

   Color mode

   nvidia-smi --query-gpu driver_version --format csv

   nvidia-smi --query-gpu driver_version --format csv

   • Tab 0

   Sample Code Download

   Copy Code

   Color mode

   driver_version
   535.183.06
   ...
   
   535.183.06

   driver_version
   535.183.06
   ...
   
   535.183.06

2. Install libnvidia-nscq.

   • Tab 0

   Sample Code Download

   Copy Code

   Color mode

   apt-cache policy libnvidia-nscq-535

   apt-cache policy libnvidia-nscq-535

   • Tab 0

   Sample Code Download

   Copy Code

   Color mode

   libnvidia-nscq-535:
     Installed: (none)
     Candidate: 535.247.01-1
     Version table:
        535.247.01-1 600
           600 https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2204/x86_64  Packages
   ...
           600 https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2204/x86_64  Packages
        535.216.01-1 600
           600 https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2204/x86_64  Packages
        535.183.06-1 600  # Install the version that matches the driver
           600 https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2204/x86_64  Packages
        535.183.01-1 600
   
           600 https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2204/x86_64  Packages
        535.54.03-1 600
           600 https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2204/x86_64  Packages

   libnvidia-nscq-535:
     Installed: (none)
     Candidate: 535.247.01-1
     Version table:
        535.247.01-1 600
           600 https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2204/x86_64  Packages
   ...
           600 https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2204/x86_64  Packages
        535.216.01-1 600
           600 https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2204/x86_64  Packages
        535.183.06-1 600  # Install the version that matches the driver
           600 https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2204/x86_64  Packages
        535.183.01-1 600
   
           600 https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2204/x86_64  Packages
        535.54.03-1 600
           600 https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2204/x86_64  Packages

   • Tab 0

   Sample Code Download

   Copy Code

   Color mode

   apt install libnvidia-nscq-535=535.183.06-1

   apt install libnvidia-nscq-535=535.183.06-1

NVSwitch Device Monitoring API(NVSDM) Library Installation

• Install the NVSDM library.
  • Tab 0

  Sample Code Download

  Copy Code

  Color mode

  apt-cache policy libnvsdm

  apt-cache policy libnvsdm

  • Tab 0

  Sample Code Download

  Copy Code

  Color mode

  libnvsdm:
    Installed: (none)
    Candidate: 580.105.08-1
    Version table:
       580.105.08-1 600
          600 https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2204/x86_64  Packages
       580.95.05-1 600
          600 https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2204/x86_64  Packages
       580.82.07-1 600
          600 https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2204/x86_64  Packages
       580.65.06-1 600
          600 https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2204/x86_64  Packages

  libnvsdm:
    Installed: (none)
    Candidate: 580.105.08-1
    Version table:
       580.105.08-1 600
          600 https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2204/x86_64  Packages
       580.95.05-1 600
          600 https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2204/x86_64  Packages
       580.82.07-1 600
          600 https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2204/x86_64  Packages
       580.65.06-1 600
          600 https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2204/x86_64  Packages

  • Tab 0

  Sample Code Download

  Copy Code

  Color mode

  apt install libnvsdm=580.105.08-1

  apt install libnvsdm=580.105.08-1

Install NVIDIA DCGM (for Ubuntu)

Install the DCGM Exporter according to the steps below.

1. DCGM(datacenter-gpu-manager) Installation
2. datacenter-gpu-manager-exporter installation
3. DCGM Service Activation and Start

DCGM(datacenter-gpu-manager) Installation

refers to a specific version of NVIDIA’s Data Center GPU Manager (DCGM) tool, which is a package for managing and monitoring NVIDIA data center GPUs. In particular, cuda12 indicates that this management tool is installed for the CUDA 12 version, and datacenter-gpu-manager-4 refers to the 4.x version of DCGM. This tool provides a variety of features, including GPU status monitoring, diagnostics, alert system, and power/clock management.

1. Check the CUDA version.
   • Tab 0

   Sample Code Download

   Copy Code

   Color mode

   nvidia-smi | grep CUDA

   nvidia-smi | grep CUDA

   • Tab 0

   Sample Code Download

   Copy Code

   Color mode

   | NVIDIA-SMI 535.183.06             Driver Version: 535.183.06     CUDA Version: 12.2 |

   | NVIDIA-SMI 535.183.06             Driver Version: 535.183.06     CUDA Version: 12.2 |

   • Tab 0

   Sample Code Download

   Copy Code

   Color mode

   CUDA_VERSION=12

   CUDA_VERSION=12

2. Install datacenter-gpu-manager-cuda.
   • Tab 0

   Sample Code Download

   Copy Code

   Color mode

   apt install datacenter-gpu-manager-4-cuda${CUDA_VERSION}

   apt install datacenter-gpu-manager-4-cuda${CUDA_VERSION}

Install datacenter-gpu-manager-exporter

It is a tool that, based on NVIDIA Data Center GPU Manager (DCGM), collects various GPU metrics such as GPU usage, memory usage, temperature, and power consumption, and exposes them for use in monitoring systems like Prometheus.

1. Install datacenter-gpu-manager-exporter.
   • Tab 0

   Sample Code Download

   Copy Code

   Color mode

   apt install datacenter-gpu-manager-exporter

   apt install datacenter-gpu-manager-exporter

2. Check the DCGM Exporter configuration file.
   • Tab 0

   Sample Code Download

   Copy Code

   Color mode

   cat /usr/lib/systemd/system/nvidia-dcgm-exporter.service | grep ExecStart

   cat /usr/lib/systemd/system/nvidia-dcgm-exporter.service | grep ExecStart

   • Tab 0

   Sample Code Download

   Copy Code

   Color mode

   ExecStart=/usr/bin/dcgm-exporter -f /etc/dcgm-exporter/default-counters.csv

   ExecStart=/usr/bin/dcgm-exporter -f /etc/dcgm-exporter/default-counters.csv

3. When installing the DCGM Exporter, review the provided configuration and remove # for the required metrics, and add # for the unnecessary metrics.
   • Tab 0

   Sample Code Download

   Copy Code

   Color mode

   vi /etc/dcgm-exporter/default-counters.csv
   ## Example ##
   ...
   DCGM_FI_PROF_PIPE_TENSOR_ACTIVE, gauge, Ratio of cycles the tensor (HMMA) pipe is active.
   DCGM_FI_PROF_DRAM_ACTIVE,        gauge, Ratio of cycles the device memory interface is active sending or receiving data.
   # DCGM_FI_PROF_PIPE_FP64_ACTIVE,   gauge, Ratio of cycles the fp64 pipes are active.
   # DCGM_FI_PROF_PIPE_FP32_ACTIVE,   gauge, Ratio of cycles the fp32 pipes are active.
   ...

   vi /etc/dcgm-exporter/default-counters.csv
   ## Example ##
   ...
   DCGM_FI_PROF_PIPE_TENSOR_ACTIVE, gauge, Ratio of cycles the tensor (HMMA) pipe is active.
   DCGM_FI_PROF_DRAM_ACTIVE,        gauge, Ratio of cycles the device memory interface is active sending or receiving data.
   # DCGM_FI_PROF_PIPE_FP64_ACTIVE,   gauge, Ratio of cycles the fp64 pipes are active.
   # DCGM_FI_PROF_PIPE_FP32_ACTIVE,   gauge, Ratio of cycles the fp32 pipes are active.
   ...