GPU Server

• 1: Overview
• 1.1: Server type
• 1.2: Monitoring Metrics
• 1.3: ServiceWatch Metrics
• 2: How-to guides
• 2.1: Manage Image
• 2.2: Manage Keypair
• 2.3: Use Multi-instance GPU on GPU Server
• 2.4: Use NVSwitch on GPU Server
• 2.5: Install ServiceWatch Agent
• 3: API Reference
• 4: CLI Reference
• 5: Release Note

1 - Overview

Service Overview

GPU Server is a virtualized computing service that lets you freely allocate and use infrastructure resources such as CPU, GPU, and memory provided by the server, without having to purchase them individually, and allocate as much as needed at the required time. It is suitable for tasks that require fast computation speed, such as AI model experiments, predictions, and inference in a cloud environment, and you can flexibly select and use resources with optimized performance according to the type and scale of the work. The GPU Server provides the following features.

Provided features

• GPU Server Management: Through a web-based console, users can directly Self Service create, delete, and modify GPU servers from provisioning to monitoring and billing.
• Product offering by GPU quantity: Depending on the project’s purpose and scale, you can freely select the number of H100/A100 GPUs to configure a virtual server.
• High‑Performance GPU Provision: We provide high‑performance GPU servers at physical‑server level using a pass‑through method.
• Storage Connection: Provides additional attached storage besides the OS disk. * Block Storage, File Storage, Object Storage can be connected and used.
• Strong Security Enforcement: Through the Security Group service, control inbound/outbound traffic exchanged with the external Internet or other VPC (Virtual Private Cloud) to securely protect the server.
• Monitoring: You can view monitoring information such as the status of CPU, Memory, Disk, and GPU, which are computing resources, through the Cloud Monitoring service.
• Network Configuration Management: The server’s subnet/IP can be conveniently changed from the values set at initial creation. * NAT IP provides a management feature that lets you enable or disable it as needed.
• Key Pair method: To ensure a secure OS access method, we provide a Key Pair method instead of ID/PW login.
• Image Management: You can create and manage Custom Images, and it provides sharing functionality between projects.
• ServiceWatch Service Integration: You can monitor data through the ServiceWatch service.

Components

GPU Server provides GPUs, NVSwitch, and NVLink on top of virtualized computing resources.

Specifications by GPU Type

GPU (Graphic Processing Unit) performs the calculations required to generate the images that compose a computer screen, and because it is specialized for parallel processing, it can handle large amounts of data quickly, processing large‑scale parallel operations such as artificial intelligence (AI) and data analysis. The specifications of the GPU Types provided by the GPU Server service are as follows.

NPU Type Specifications

The NPU (Network Processing Unit) is a processor specialized for AI inference operations, performing generative AI and various AI inference workloads based on high throughput and power efficiency. The specifications of the NPU Type provided by the GPU Server service are as follows.

Server type

The server types offered by the GPU Server are as follows. For detailed information about the server types provided by GPU Server, see GPU Server 서버 타입.

OS and driver version

The operating systems (OS) supported by the GPU Server are as follows. Please be aware that B300-type GPUs are supported only from a certain GPU version onward, so choose your image accordingly.

Constraints

The GPU Server has the following constraints.

• We are providing Ubuntu22.04 as the current OS version.
• GPU is provided via Pass-through.
• Physical GPU cards are created in units of 1, 2, 4, or 8 cards each. * –>

Preceding Service

This service must be installed in advance before creating it. Please prepare by referring to the user guide provided in advance.

1.1 - Server type

GPU Server server type

GPU servers are categorized by the GPU type they offer, and the GPU used in a GPU server is determined by the server type selected when creating the GPU server. Please select the server type based on the specifications of the application you want to run on the GPU server.

The server types supported by the GPU Server are as follows.

GPU-H100-2 g2v12h1

g1 server type

The g1 server type is a GPU Server that uses the NVIDIA A100 Tensor Core GPU, making it suitable for high‑performance applications.

• Up to 8 NVIDIA A100 Tensor Core GPUs provided
• Equipped with 6,912 CUDA cores and 432 Tensor cores per GPU.
• Supports up to 128 vCPUs and 1,920 GB of memory
• Maximum networking speed of 40 Gbps
• 600 GB/s GPU and NVIDIA NVSwitch P2P communication
• Additional AMD CPU-based server type provided (GPU-A100-1-A)

g2 server type

The g2 server type is a GPU Server that uses the NVIDIA H100 Tensor Core GPU, making it suitable for high-performance applications.

• Up to 8 NVIDIA H100 Tensor Core GPUs provided
• Equipped with 16,896 CUDA cores and 528 Tensor cores per GPU
• Supports up to 96 vCPUs and 1,920 GB of memory
• Maximum networking speed of 40 Gbps
• 900 GB/s GPU and NVIDIA NVSwitch P2P communication

g3 server type

The g3 server type is a GPU Server that uses the NVIDIA B300 Tensor Core GPU, making it suitable for high-performance applications.

• Provides up to 8 NVIDIA B300 Tensor Core GPUs
• Equipped with 20,480 CUDA cores and 640 Tensor cores per GPU.
• Supports up to 128 vCPUs and 3,840 GB of memory.
• Up to 40 Gbps networking speed
• 1.8 TB/s GPU and NVIDIA NVSwitch P2P communication

n1 server type

The n1 server type is an NPU Server that uses the Furiosa RNGD NPU, making it suitable for AI inference workloads.

• Provides up to 8 Furiosa RNGD NPUs
• Supports 256 TFLOPS and 512 TFLOPS per NPU
• Supports up to 64 vCPUs and 848 GB of memory
• Maximum networking speed of 40 Gbps
• PCIe Gen5 x16 interface support

1.2 - Monitoring Metrics

GPU Server Monitoring Metrics

The table below shows the monitoring metrics of the GPU server that can be viewed through Cloud Monitoring.

Even without installing the Agent, it provides basic monitoring metrics and the table below. Please check the GPU Server monitoring metrics (provided by default). Additionally, the metrics that can be viewed by installing the Agent are in the table. Please refer to the additional monitoring metrics for GPU Server (Agent installation required).

For detailed usage of Cloud Monitoring, refer to the Cloud Monitoring guide.

1.3 - ServiceWatch Metrics

The GPU Server sends metrics to ServiceWatch. The metrics provided by default monitoring are data collected at 5‑minute intervals. If detailed monitoring is enabled, you can view data collected at 1‑minute intervals.

Refer to How-to guides > ServiceWatch Detailed Monitoring Activation for how to enable detailed monitoring of the GPU Server.

Basic Metrics

The following are the basic metrics for the Virtual Server namespace.

The indicators whose names are displayed in bold below are the key indicators selected among the basic indicators provided by Virtual Server. The key metrics are used to build service dashboards that are automatically created for each service in ServiceWatch.

Each metric indicates through the user guide which statistical value is meaningful to view for that metric, and among the meaningful statistics, the statistical values shown in bold text are the primary statistics. In the service dashboard, you can view primary metrics using the primary statistical values.

2 - How-to guides

Users can create the service by entering the required GPU Server information and selecting detailed options through the Samsung Cloud Platform Console.

Create GPU Server

You can create and use a GPU Server service from the Samsung Cloud Platform Console.

To create a GPU Server, follow these steps.

1. Click the All Services > Compute > GPU Server menu. 1. Navigate to the Service Home page of the GPU Server.

2. On the Service Home page, click the Create GPU Server button. 2. GPU Server Creation Go to the page.

3. GPU Server Creation page, input the information required to create the service, and select detailed options.

   • Select the required information in the Image and Version Selection area.

     Category	required status	Detailed description
     Image	Required	Select the provided Image type Standard: Samsung Cloud Platform standard Image RHEL, Ubuntu Custom: User-created Image Kubernetes: Kubernetes Image Ubuntu
     Image version	Required	Select the version of the chosen Image Provide a list of versions for the offered server Image For detailed information about the provided server image, refer to OS 및 GPU 드라이버 버전

     Table. GPU Server image and version selection input fields
   • Enter or select the required information in the Service Information Input area.

     Category	required status	Detailed description
     Number of servers	Required	Number of GPU Server instances to create concurrently only numeric input allowed, enter a value between 1 ~ 100
     Service Type > Server Type	Required	GPU Server server type Indicates the specifications of a GPU-type server, and select a server that includes 1, 2, 4, or 8 GPUs For detailed information about the server types offered by GPU Server, refer to GPU Server 서버 타입
     Service Type > Planned Compute	Selection	Status of resources with Planned Compute configured In Use: Number of resources with Planned Compute that are currently in use Configured: Number of resources with Planned Compute configured Coverage preview: Amount applied per resource by Planned Compute Apply for Planned Compute service: Navigate to the Planned Compute service application page For more details, see Planned Compute 신청하기
     Block Storage	Required	Set the Block Storage used by the GPU Server according to its purpose Default: Area where the OS is installed and used Capacity can be entered in Unit increments (minimum capacity varies by OS image type) RHEL: values between 3 and 1,536 are allowed Ubuntu: values between 3 and 1,536 are allowed SSD: high‑performance general volume HDD: general volume SSD/HDD_KMS: additional encrypted volume that uses Samsung Cloud Platform KMS (Key Management System) encryption keys Encryption can be applied only at initial creation (cannot be changed after creation) Performance degradation may occur when using the SSD_KMS disk type SSD_Provisioned: SSD volume with configurable IOPS and Throughput Additional: used when extra space beyond the OS area is needed After selecting Use, enter the storage type and capacity To add storage, click the + button (up to 25 can be added); to delete, click the x button Capacity can be entered as a value between 1 and 1,536 in Unit increments Since 1 Unit equals 8 GB, 8 ~ 12,288 GB can be created SSD: high‑performance general volume HDD: general volume SSD/HDD_KMS: additional encrypted volume that uses Samsung Cloud Platform KMS (Key Management System) encryption keys Encryption can be applied only at initial creation (cannot be changed after creation) Performance degradation may occur when using the SSD_KMS disk type HDD/SSD_MultiAttach: volume that can be attached to two or more servers SSD_Provisioned: SSD volume with configurable IOPS and Throughput For details on each Block Storage type, see Block Storage 생성하기 Delete on termination: If Delete on Termination is set to Use, the volume will be terminated together with the server Volumes that have snapshots are not deleted even when Delete on termination is set to Use A multi‑attach volume can be deleted only when the server being removed is the last remaining server attached to the volume
     Max IOPS	Required	Enter the maximum IOPS value between 5,000 and 20,000 Disk type can be set only if it is SSD_Provisioned
     Max Throughput	Required	Enter the maximum Throughput value between 250~1,000 Can be set only when the disk type is SSD_Provisioned

     Table. GPU Server Service Configuration Items
   • In the Required Information Input area, enter or select the required information.

     Category	required status	Detailed description
     Server name	Required	If the number of selected servers is 1, enter a name to distinguish the server Set the hostname to the entered server name Enter within 63 characters using English letters, numbers, spaces, and special characters (- _)
     Server name Prefix	Required	Enter a Prefix to distinguish each server generated when the selected number of servers is 2 or more Automatically generated as the user input value (prefix) + ‘-#’ format Enter within 59 characters using English letters, numbers, spaces, and special characters (-, _)
     Network Settings > Create New Network Port	Required	Set the network where the GPU Server will be installed Select a pre-created VPC. General Subnet: Select a pre-created general Subnet IP can be set to auto-generate or user input, and if input is selected, the user can directly enter the IP NAT: Available only when there is a single server and the VPC is connected to an Internet Gateway. Checking Use allows selection of a NAT IP NAT IP: Select a NAT IP If there is no NAT IP to select, click the Create New button to generate a Public IP Refresh button to view and select the created Public IP Creating a Public IP incurs charges according to the Public IP pricing policy Local Subnet (optional): Select Use for the local Subnet It is not a required element for creating the service A pre-created local Subnet must be selected IP can be set to auto-generate or user input, and selecting Input allows the user to directly enter the IP
     Network Settings > Assign Existing Network Port	Required	Set the network where the GPU Server will be installed Select a pre‑created VPC General Subnet: Select a pre‑created general Subnet and Port NAT: Available only when there is a single server and the VPC is connected to an Internet Gateway; checking the option allows you to select a NAT IP. NAT IP: Select a NAT IP If no NAT IP is available to select, click the Create New button to generate a Public IP Refresh button to view and select the created Public IP Local Subnet (Optional): Select Use for the local Subnet Select a pre‑created local Subnet and Port
     Security Group	Selection	Settings required to connect to the server Selection: You can select up to 5 pre‑created Security Groups Create New: If there is no Security Group to apply, create one separately in the Security Group service If you do not configure a Security Group, all connections will be blocked, so you must set it to allow required access
     Keypair	Required	User authentication method to use when connecting to the server New creation: Create a new one when a new Keypair is required Refer to Keypair 생성하기 Default login account list by OS RHEL: cloud-user Ubuntu: ubuntu

     Table. GPU Server required information input items
   • Additional Information Input area, enter or select the required information.

     Category	required status	Detailed description
     Lock	Selection	Lock usage setting When using Lock, it prevents actions such as server termination, start, and stop from being executed, thereby avoiding malfunctions caused by mistakes
     Init script	Selection	Script executed when the server starts The init script must be written as a Batch script for Windows, a Shell script for Linux, or cloud‑init, depending on the image type. Up to 45,000 bytes can be entered
     tag	Selection	Add Tag Up to 50 per resource can be added After clicking the Add Tag button, enter or select Key, Value values

     Table. GPU Server Additional Information Input Items

4. Summary Check the detailed information and estimated charges generated in the panel, and click the Create button.

5. When the popup notifying creation opens, click the Confirm button.

   • Once creation is complete, check the created resources on the GPU Server List page.

Check GPU Server detailed information

The GPU Server service allows you to view and edit the full resource list and detailed information. The GPU Server Details page consists of Details, Tags, Activity Log tabs. To view detailed information about the GPU Server service, follow these steps.

1. All Services > Compute > GPU Server Click the menu. 1. Navigate to the Service Home page of the GPU Server.
2. Click the GPU Server menu on the Service Home page. 2. GPU Server List Go to the page.
3. GPU Server List page, click the resource to view detailed information. 3. Navigate to the GPU Server Details page.
   • GPU Server Details page displays status information and additional feature information, and consists of Details, Tags, Activity History tabs.
   • For detailed information about GPU Server Additional Features, please refer to GPU Server 관리 부가 기능.

     Category	Detailed description
     GPU Server status	Status of the GPU Server created by the user Build: State where the Build command has been received Building: Build in progress Networking: Server creation process in progress Scheduling: Server creation process in progress Block_Device_Mapping: Connecting Block Storage during server creation Spawning: State where the server creation process is ongoing Active: Available state Powering_off: State when a stop request is made Deleting: Server deletion in progress Reboot_Started: Reboot in progress state Error: Error state Migrating: State where the server is migrating to another host Reboot: State where the Reboot command has been received Rebooting: Reboot in progress Rebuild: State where the Rebuild command has been received Rebuilding: State when a Rebuild request is made Rebuild_Spawning: State where the Rebuild process is ongoing Shutoff: State when Powering off is completed
     Server control	Buttons to change server status Start: Start a stopped server Stop: Stop a running server Restart: Restart a running server
     Image generation	Create a user custom image from the current server’s image
     Console log	View the console log of the current server You can view the console logs output by the current server. For more details, see 콘솔 로그 확인하기.
     Create dump	Create a dump of the current server The dump file is created inside the GPU Server For detailed dump creation instructions, see Dump 생성하기
     Rebuild	All data and settings of the existing server are deleted, and a new server is set up For details, see Rebuild 수행하기.
     Service cancellation	Cancel service button

     Table. GPU Server status information and additional features

Detailed Information

GPU Server List page allows you to view detailed information of the selected resource and, if needed, modify the information.

Tag

GPU Server list page allows you to view the tag information of the selected resource, and to add, modify, or delete it.

Job History

GPU Server List page allows you to view the operation history of the selected resource.

Control GPU Server Operation

If you need to control the operation of the created GPU Server resources, you can perform the task from the GPU Server List or GPU Server Details page. You can start, stop, or restart a running server.

Getting Started with GPU Server

You can start a GPU server that is shut off. To start the GPU Server, follow these steps.

1. All Services > Compute > GPU Server Click the menu. 1. Navigate to the Service Home page of the GPU Server.
2. Click the GPU Server menu on the Service Home page. 2. GPU Server List Go to the page.
3. GPU Server List page, click the resource to start among the shutoff servers, and navigate to the GPU Server Details page.
   • On the GPU Server list page, you can start each resource via the right More button.
   • After selecting multiple servers with the checkboxes, you can control multiple servers simultaneously using the Start button at the top.
4. On the GPU Server Details page, click the Start button at the top to start the server. 4. In the Status Indicator item, check the status of the changed server.
   • When the GPU Server startup is complete, the server status changes from Shutoff to Active.
   • For detailed information about the GPU Server status, please refer to GPU Server 상세 정보 확인하기.

Stop GPU Server

You can stop a GPU Server that is running (Active). To stop the GPU Server, follow the steps below.

1. All Services > Compute > GPU Server Click the menu. 1. Navigate to the Service Home page of the GPU Server.
2. Click the GPU Server menu on the Service Home page. 2. GPU Server List Go to the page.
3. On the GPU Server List page, click the resource to stop among the servers that are active (Active), and go to the GPU Server Details page.
   • On the GPU Server List page, you can stop each resource using the right More button.
   • After selecting multiple servers with the checkboxes, you can control multiple servers simultaneously using the Stop button at the top.
4. On the GPU Server Details page, click the Stop button at the top to start the server. 4. Check the status of the changed server in the Status Indicator item.
   • When the GPU Server shutdown is complete, the server status changes from Active to Shutoff.
   • For detailed information about the GPU Server status, refer to GPU Server 상세 정보 확인하기.

Restart GPU Server

You can restart the created GPU Server. To restart the GPU server, follow these steps.

1. All Services > Compute > GPU Server Click the menu. 1. Navigate to the Service Home page of the GPU Server.
2. Click the GPU Server menu on the Service Home page. 2. GPU Server List Go to the page.
3. GPU Server List page, click the resource to restart, and navigate to the GPU Server Details page.
   • On the GPU Server List page, you can restart each resource through the right More button.
   • After selecting multiple servers with the checkboxes, you can control multiple servers simultaneously using the Restart button at the top.
4. On the GPU Server Details page, click the Restart button at the top to start the server. 4. Check the status of the changed server in the Status Indicator item.
   • During a GPU Server restart, the server status goes through Rebooting and finally changes to Active.
   • For detailed information about the GPU Server status, please refer to GPU Server 상세 정보 확인하기.

GPU Server Resource Management

If you need server control and management functions for the created GPU Server resources, you can perform tasks on the GPU Server Resource List or GPU Server Details page.

Create Image

You can create an image of a running GPU server.

To create an Image of the GPU Server, follow the steps below.

1. All Services > Compute > GPU Server Click the menu. 1. Navigate to the Service Home page of the GPU Server.

2. Click the GPU Server menu on the Service Home page. 2. GPU Server List Go to the page.

3. On the GPU Server List page, click the resource to create an Image. 3. Navigate to the GPU Server Details page.

4. On the GPU Server Details page, click the Create Image button. 4. Navigate to the Image creation page.

   • Enter the required information in the Service Information Input area.

     Category	required status	Detailed description
     Image name	Required	Name of the image to be created English letters, numbers, spaces, and special characters(- _) using them, enter within 200 characters

     Table. Image service information input items

5. Check the input information and click the Generate button.

   • Once creation is complete, check the created resources on the All Services > Compute > GPU Server > Image List page.

Enable detailed monitoring for ServiceWatch

By default, the GPU Server is linked to the basic monitoring of the ServiceWatch and Virtual Server namespaces. You can enable detailed monitoring as needed to more quickly identify operational issues and take action. For detailed information about ServiceWatch, refer to the ServiceWatch 개요.

To enable detailed ServiceWatch monitoring on the GPU Server, follow these steps.

1. All Services > Compute > GPU Server menu, click it. 1. Navigate to the Service Home page of the GPU Server.
2. Click the GPU Server menu on the Service Home page. 2. GPU Server List Go to the page.
3. On the GPU Server list page, click the resource to enable detailed ServiceWatch monitoring. 3. Navigate to the GPU Server Details page.
4. On the GPU Server Details page, click the Edit button for ServiceWatch detailed monitoring. 4. ServiceWatch Detailed Monitoring Edit Navigate to the popup window.
5. ServiceWatch Detailed Monitoring Edit In the popup window, after selecting Enable, review the instructions and click the Confirm button.
6. Check the ServiceWatch detailed monitoring items on the GPU Server Details page.

Disable detailed monitoring of ServiceWatch

To disable detailed monitoring of ServiceWatch on the GPU Server, follow these steps.

1. All Services > Compute > GPU Server Click the menu. 1. Go to the Service Home page of the GPU Server.
2. Click the GPU Server menu on the Service Home page. 2. GPU Server List Go to the page.
3. On the GPU Server List page, click the resource to disable ServiceWatch detailed monitoring. 3. Navigate to the GPU Server Details page.
4. GPU Server Details page, click the Edit button for ServiceWatch detailed monitoring. 4. ServiceWatch Detailed Monitoring Edit Navigate to the popup window.
5. ServiceWatch Detailed Monitoring Edit In the popup window, after deselecting Enable, review the guidance message and click the Confirm button.
6. Check the detailed ServiceWatch monitoring items on the GPU Server Details page.

GPU Server management additional features

For managing the GPU Server, you can view Console logs, generate Dumps, and perform Rebuilds. To view the Console logs, create a Dump, and perform a Rebuild on the GPU Server, follow these steps.

Check console log

You can view the current console log of the GPU Server.

To check the console log of the GPU Server, follow these steps.

1. All Services > Compute > GPU Server Click the menu. 1. Navigate to the Service Home page of the GPU Server.
2. Click the GPU Server menu on the Service Home page. 2. GPU Server List Go to the page.
3. On the GPU Server List page, click the resource to view the console log. 3. Navigate to the GPU Server Details page.
4. On the GPU Server Details page, click the Console Log button. 4. Console Log navigates to the popup window.
5. Console Log Check the console log displayed in the popup window.

Create Dump

To create a dump file on the GPU Server, follow these steps.

1. All Services > Compute > GPU Server Click the menu. 1. Navigate to the Service Home page of the GPU Server.
2. Click the GPU Server menu on the Service Home page. 2. GPU Server List Go to the page.
3. GPU Server List page, click the resource to view detailed information. 3. Navigate to the GPU Server Details page.
4. On the GPU Server Details page, click the Generate Dump button.
   • The dump file is created inside the GPU server.

Execute Rebuild

You can delete all data and settings of the existing GPU Server and rebuild it on a new server.

To perform a Rebuild of the GPU Server, follow these steps.

1. All Services > Compute > GPU Server Click the menu. 1. Navigate to the Service Home page of the GPU Server.
2. Click the GPU Server menu on the Service Home page. 2. GPU Server List Go to the page.
3. On the GPU Server List page, click the resource to perform Rebuild. 3. Navigate to the GPU Server Details page.
4. On the GPU Server Details page, click the Rebuild button.
   • During a GPU Server Rebuild, the server status changes to Rebuilding, and when the Rebuild is complete, it returns to the state it was in before the Rebuild.
   • For detailed information about the GPU Server status, please refer to GPU Server 상세 정보 확인하기.

Terminate GPU Server

If you terminate an unused GPU server, you can reduce operating costs. However, if you terminate the GPU server, the running service may be stopped immediately, so you should proceed with the termination only after fully considering the impact of service interruption.

To cancel the GPU Server, follow the steps below.

1. All Services > Compute > GPU Server Click the menu. 1. Navigate to the Service Home page of the GPU Server.
2. Click the GPU Server menu on the Service Home page. 2. GPU Server List Go to the page.
3. On the GPU Server List page, select the resource to cancel, and click the Cancel Service button.
   • The termination of attached storage depends on the Delete on termination setting, so refer to 해지 제약 사항.
4. When termination is complete, check on the GPU Server List page whether the resources have been terminated.

Termination constraints

When a GPU Server termination request cannot be processed, a popup will provide guidance. Please refer to the case below.

The termination of attached storage depends on the Delete on termination setting.

2.1 - Manage Image

Users can create the service by entering the required information for the Image service within the GPU Server service and selecting detailed options through the Samsung Cloud Platform Console.

Create Image

You can create an image of a running GPU Server. To create an image of a GPU Server, please refer to Image Creation.

Check Image detailed information

Image service allows you to view and edit the full resource list and detailed information. The Image Details page consists of Detailed Information, Tags, Operation History tabs.

To view detailed information of the Image service, follow these steps.

1. Click the All Services > Compute > GPU Server menu. You will be taken to the Service Home page of GPU Server.
2. On the Service Home page, click the Image menu. You will be taken to the Image list page.
3. On the Image List page, click the resource to view detailed information. You will be taken to the Image Detail page.
   • Image Details page displays status information and additional feature information, and consists of Details, Tags, Activity Log tabs.

     Category	Detailed description
     Image status	Status of user-created Image Active: Available state Queued: Image has been uploaded and is waiting for processing after creation Importing: Image has been uploaded and is currently being processed after creation
     Share with another account	Image can be shared with another Account The Image’s Visibility must be set to Shared in order to be shared with another Account
     Delete image	Button to delete the Image Once the Image is deleted, it cannot be restored

     Table. GPU Server Image status information and additional features

Detailed Information

Image list page lets you view detailed information of the selected resource and modify it if necessary.

tag

On the Image List page, you can view the tag information of the selected resource and add, modify, or delete it.

Job History

You can view the operation history of the selected resource on the Image List page.

Image Resource Management

Describes the control and management functions of the generated Image.

Share to another Account

To share an Image with another Account, follow the steps below.

1. Log in to the account to be shared and click the All Services > Compute > GPU Server menu. Go to the GPU Server’s Service Home page.
2. On the Service Home page, click the Image menu. You will be taken to the Image List page.
3. On the Image List page, click the Image you want to control. You will be taken to the Image Details page.
4. Click the Share to another Account button. Navigate to the Share image to another Account page.
   • Share with another Account feature allows you to share an Image with another Account. To share an Image with another Account, the Image’s Visibility must be Shared.
5. Share image to another Account page, enter the required information, and click the Done button.

   Category	Required	Detailed description
   image name	-	Name of the image to share Input not allowed
   Image ID	-	Shareable image ID Input not allowed
   Shared Account ID	Required	Enter another Account ID to share English letters, numbers, special characters- within 64 characters

   Table. Required input fields for sharing images to another Account
6. You can view the information in the sharing status of the Image Details page.
   • When the request is first made, the status is Pending, and it changes to Accepted once approval is completed by the account receiving the share.

Receive sharing from another Account

To receive an Image shared from another Account, follow these steps.

1. Log in to the account to be shared and click the All Services > Compute > GPU Server menu. Navigate to the GPU Server’s Service Home page.
2. On the Service Home page, click the Image menu. You will be taken to the Image List page.
3. On the Image List page, click the Receive Image Share button. You will be taken to the Receive Image Share popup.
4. Receive Image Sharing In the popup window, enter the Image’s resource ID you want to receive, and click the Confirm button.
5. When image sharing is complete, you can view the shared Image in the Image list.

Delete Image

You can delete unused Images. However, since an Image cannot be recovered after deletion, you should carefully consider the impact before performing the deletion.

To delete the Image, follow these steps.

1. Click the All Services > Compute > GPU Server menu. Go to the GPU Server’s Service Home page.
2. On the Service Home page, click the Image menu. You will be taken to the Image List page.
3. Image list page, select the resource to delete, and click the Delete button.
   • On the Image List page, select multiple Image check boxes and click the Delete button at the top of the resource list.
4. After deletion is complete, verify on the Image list page that the resource has been removed.

2.2 - Manage Keypair

Users can create the service by entering the required Keypair information within the GPU Server service and selecting detailed options through the Samsung Cloud Platform Console.

Create a Keypair

You can create and use the Keypair service while using the GPU Server service in the Samsung Cloud Platform Console.

To create a keypair, follow these steps.

1. Click the All Services > Compute > GPU Server menu. You will be taken to the Service Home page of GPU Server.
2. On the Service Home page, click the Keypair menu. You will be taken to the Keypair List page.
3. On the Keypair List page, click the Keypair Create button. You will be taken to the Keypair Create page.
   • Enter the required information in the Service Information Input area.

     Category	Required	Detailed description
     Keypair name	Required	Enter the name of the Keypair to create using English letters, numbers, spaces, and special characters (-, _) within 255 characters
     Keypair type	Required	ssh

     Table. Keypair service information input fields
   • Additional Information Input area, please enter or select the required information.

     Category	Required status	Detailed description
     tag	Selection	Add Tag Up to 50 can be added per resource After clicking the Add Tag button, enter or select Key, Value values

     Table. Keypair additional information input fields
     Caution

     • After creation is complete, you can download the Key only once. Since reissuance is not possible, make sure it has been downloaded.
     • Store the downloaded Private Key in a safe place.
4. Check the input information and click the Create button.
   • After creation is complete, check the created resources on the Keypair List page.

View detailed information of the Keypair

The Keypair service allows you to view and edit the full resource list and detailed information. Keypair Details page consists of Details, Tags, Activity Log tabs.

To view detailed information about a keypair, follow these steps.

1. Click the All Services > Compute > GPU Server menu. You will be taken to the Service Home page of GPU Server.
2. On the Service Home page, click the Keypair menu. You will be taken to the Keypair List page.
3. On the Keypair List page, click the resource to view its details. You will be taken to the Keypair Details page.
   • Keypair Details page displays status information and additional feature information, and consists of Details, Tags, Activity Log tabs.

Detailed Information

Keypair List page allows you to view detailed information of the selected resource and edit the information if needed.

tag

On the Keypair List page, you can view the tag information of the selected resource, and add, modify, or delete it.

Job History

On the Keypair List page, you can view the operation history of the selected resource.

Keypair Resource Management

Describes the control and management functions of a keypair.

Get public key

To retrieve the public key, follow these steps.

1. Click the All Services > Compute > GPU Server menu. You will be taken to the Service Home page of GPU Server.

2. On the Service Home page, click the Keypair menu. You will be taken to the Keypair List page.

3. On the Keypair List page, click the More button at the top and then click the Import Public Key button. You will be taken to the Import Public Key page.

   • Enter or select the required information in the Required Information Input area.

     Category	Required	Detailed description
     Keypair name	Required	Name of the Keypair to create
     Keypair type	Required	ssh
     public key	Required	Enter public key Load file: Select the Attach file button to attach the public key file Only files with the following extension (.pem) can be attached Enter public key: Paste the copied public key value The public key value can be copied from the Keypair Details page

     Table. Required input fields for retrieving the public key

4. Review the entered information and click the Complete button.

   • Once creation is complete, check the created resources on the Keypair List page.

Delete Keypair

You can delete unused Keypairs. However, once a Keypair is deleted it cannot be recovered, so please review the impact thoroughly beforehand before proceeding with deletion.

To delete a keypair, follow these steps.

1. Click the All Services > Compute > GPU Server menu. Go to the Service Home page of GPU Server.
2. On the Service Home page, click the Keypair menu. You will be taken to the Keypair List page.
3. On the Keypair List page, select the resource to delete, and click the Delete button.

• On the Keypair List page, select multiple Keypair check boxes and click the Delete button at the top of the resource list.

4. After deletion is complete, check the Keypair List page to confirm that the resource has been removed.

2.3 - Use Multi-instance GPU on GPU Server

After creating a GPU Server, you can enable the MIG (Multi-instance GPU) feature on the GPU Server’s VM (Guest OS) and create an instance for use.

NVIDIA Multi-instance GPU Introduction

NVIDIA Multi-instance GPU (hereafter referred to as MIG) supports safely partitioning a GPU into GPU instances and running CUDA applications starting with the NVIDIA Ampere architecture. Through this, multiple users can each utilize different GPU resources to achieve optimal GPU utilization. This feature is especially useful for workloads that do not fully utilize the GPU’s computing capacity, and users can run multiple workloads in parallel to maximize utilization.

Using Multi-instance GPU feature

To use the MIG feature, create an NVIDIA GPU Server on the Samsung Cloud Platform, then enable and disable MIG. The order of applying and removing MIG is as follows.

Applying and Using MIG

After activating the MIG and creating an Instance to assign tasks, the tasks proceed in the following order.

Activate MIG

1. Check the GPU status on the VM Instance (GuestOS) before applying MIG.

   • Check whether MIG mode is Disabled.
     Sample Code Download

     Copy Code

     Color mode

     $ nvidia-smi
     Mon Sep 27 08:37:08 2021
     +-----------------------------------------------------------------------------+
     | NVIDIA-SMI 470.57.02    Driver Version: 470.57.02    CUDA Version: 11.4 |
     | -------------------------------+----------------------+---------------------- |
     | GPU  Name        Persistence-M | Bus-Id        Disp.A | Volatile Uncorr. ECC |
     | Fan  Temp  Perf  Pwr:Usage/Cap | Memory-Usage | GPU-Util  Compute M. |
     |  |  | MIG M. |
     | ===============================+======================+====================== |
     | 0  NVDIA A100-SXM...  Off | 00000000:05:00.0 Off | 0 |
     | N/A   32C   P0    59W / 400W | 0MiB / 81251MiB | 0%      Default |
     |  |  | Disabled |
     +-------------------------------+----------------------+----------------------+
     
     +-----------------------------------------------------------------------------+
     | Processes: |
     | GPU   GI   CI       PID   Type   Process name                   GPU Memory |
     | ID   ID                                                   Usage |
     | ============================================================================= |
     | No running processes found |
     +-----------------------------------------------------------------------------+

     $ nvidia-smi
     Mon Sep 27 08:37:08 2021
     +-----------------------------------------------------------------------------+
     | NVIDIA-SMI 470.57.02    Driver Version: 470.57.02    CUDA Version: 11.4 |
     | -------------------------------+----------------------+---------------------- |
     | GPU  Name        Persistence-M | Bus-Id        Disp.A | Volatile Uncorr. ECC |
     | Fan  Temp  Perf  Pwr:Usage/Cap | Memory-Usage | GPU-Util  Compute M. |
     |  |  | MIG M. |
     | ===============================+======================+====================== |
     | 0  NVDIA A100-SXM...  Off | 00000000:05:00.0 Off | 0 |
     | N/A   32C   P0    59W / 400W | 0MiB / 81251MiB | 0%      Default |
     |  |  | Disabled |
     +-------------------------------+----------------------+----------------------+
     
     +-----------------------------------------------------------------------------+
     | Processes: |
     | GPU   GI   CI       PID   Type   Process name                   GPU Memory |
     | ID   ID                                                   Usage |
     | ============================================================================= |
     | No running processes found |
     +-----------------------------------------------------------------------------+

     Sample Code Download

     Copy Code

     Color mode

     $ nvidia-smi –L
     GPU 0: NVIDIA A100-SXM-80GB (UUID: GPU-c956838f-494a-92b2-6818-56eb28fe25e0)

     $ nvidia-smi –L
     GPU 0: NVIDIA A100-SXM-80GB (UUID: GPU-c956838f-494a-92b2-6818-56eb28fe25e0)

2. Enable MIG (Enable) for each GPU on the VM Instance (GuestOS) and reboot the VM Instance.

   Sample Code Download

   Copy Code

   Color mode

   $ nvidia-smi –I 0 –mig 1
   Enabled MIG mode for GPU 00000000:05:00.0
   All done.
   
   # reboot

   $ nvidia-smi –I 0 –mig 1
   Enabled MIG mode for GPU 00000000:05:00.0
   All done.
   
   # reboot

Warning: MIG mode is in pending enable state for GPU 00000000:05:00.0: In use by another client. 00000000:05:00.0 is currently being used by one or more other processes (e.g. CUDA application or a monitoring application such as another instance of nvidia-smi).

3. Check the GPU status after applying MIG on the VM Instance(GuestOS).
   • Check whether MIG mode is Enabled.
     Sample Code Download

     Copy Code

     Color mode

     $ nvidia-smi
     Mon Sep 27 09:44:33 2021
     +-----------------------------------------------------------------------------+
     | NVIDIA-SMI 470.57.02    Driver Version: 470.57.02    CUDA Version: 11.4 |
     | -------------------------------+----------------------+---------------------- |
     | GPU  Name        Persistence-M | Bus-Id        Disp.A | Volatile Uncorr. ECC |
     | Fan  Temp  Perf  Pwr:Usage/Cap | Memory-Usage | GPU-Util  Compute M. |
     |  |  | MIG M. |
     | ===============================+======================+====================== |
     | 0  NVDIA A100-SXM...  Off | 00000000:05:00.0 Off | On |
     | N/A   32C   P0    59W / 400W | 0MiB / 81251MiB | 0%      Default |
     |  |  | Enabled |
     +-------------------------------+----------------------+----------------------+
     +-----------------------------------------------------------------------------+
     | MIG devices: |
     +-----------------------------------------------------------------------------+
     | GPU  GI  CI  MIG | Memory-Usage | Vol | Shared |
     | ID  ID  Dev | BAR1-Usage | SM     Unc | CE  ENC  DEC  OFA  JPG |
     |  |  | ECC |  |
     | ============================================================================= |
     | No MIG devices found |
     +-----------------------------------------------------------------------------+
     +-----------------------------------------------------------------------------+
     | Processes: |
     | GPU   GI   CI       PID   Type   Process name                   GPU Memory |
     | ID   ID                                                   Usage |
     | ============================================================================= |
     | No running processes found |
     +-----------------------------------------------------------------------------+

     $ nvidia-smi
     Mon Sep 27 09:44:33 2021
     +-----------------------------------------------------------------------------+
     | NVIDIA-SMI 470.57.02    Driver Version: 470.57.02    CUDA Version: 11.4 |
     | -------------------------------+----------------------+---------------------- |
     | GPU  Name        Persistence-M | Bus-Id        Disp.A | Volatile Uncorr. ECC |
     | Fan  Temp  Perf  Pwr:Usage/Cap | Memory-Usage | GPU-Util  Compute M. |
     |  |  | MIG M. |
     | ===============================+======================+====================== |
     | 0  NVDIA A100-SXM...  Off | 00000000:05:00.0 Off | On |
     | N/A   32C   P0    59W / 400W | 0MiB / 81251MiB | 0%      Default |
     |  |  | Enabled |
     +-------------------------------+----------------------+----------------------+
     +-----------------------------------------------------------------------------+
     | MIG devices: |
     +-----------------------------------------------------------------------------+
     | GPU  GI  CI  MIG | Memory-Usage | Vol | Shared |
     | ID  ID  Dev | BAR1-Usage | SM     Unc | CE  ENC  DEC  OFA  JPG |
     |  |  | ECC |  |
     | ============================================================================= |
     | No MIG devices found |
     +-----------------------------------------------------------------------------+
     +-----------------------------------------------------------------------------+
     | Processes: |
     | GPU   GI   CI       PID   Type   Process name                   GPU Memory |
     | ID   ID                                                   Usage |
     | ============================================================================= |
     | No running processes found |
     +-----------------------------------------------------------------------------+

     Sample Code Download

     Copy Code

     Color mode

     $ nvidia-smi –L
     GPU 0: NVIDIA A100-SXM-80GB (UUID: GPU-c956838f-494a-92b2-6818-56eb28fe25e0)

     $ nvidia-smi –L
     GPU 0: NVIDIA A100-SXM-80GB (UUID: GPU-c956838f-494a-92b2-6818-56eb28fe25e0)

GPU Instance creation

If you have enabled MIG and verified its status, you can create a GPU Instance.

1. Check the list of MIG GPU Instance profiles that can be created.
   Sample Code Download

   Copy Code

   Color mode

   $ nvidia-smi mig -i [GPU ID] -lgip

   $ nvidia-smi mig -i [GPU ID] -lgip

$ nvidia-smi mig -i 0 -lgip
+-----------------------------------------------------------------------------+
| GPU instance profiles: |
| GPU   Name             ID    Instances   Memory     P2P    SM    DEC   ENC |
| Free/Total   GiB              CE    JPEG  OFA |
| ============================================================================= |
| 0 MIG 1g.10gb        19    7/7         9.50       No     14     0     0 |
| 1     0     0 |
+-----------------------------------------------------------------------------+
| 0 MIG 1g.10gb+me     20    1/1         9.50       No     14     0     0 |
| 1     1     1 |
+-----------------------------------------------------------------------------+
| 0 MIG 2g.20gb        14    3/3         19.50      No     28     1     0 |
| 2     0     0 |
+-----------------------------------------------------------------------------+
| 0 MIG 3g.40gb         9    2/2         39.50      No     42     2     0 |
| 3     0     0 |
+-----------------------------------------------------------------------------+
| 0 MIG 4g.40gb         5    1/1         39.50      No     56     2     0 |
| 4     0     0 |
+-----------------------------------------------------------------------------+
| 0 MIG 7g.80gb         0    1/1         79.25      No     98     0     0 |
| 7     1     1 |
+-----------------------------------------------------------------------------+

$ nvidia-smi mig -i 0 -lgip
+-----------------------------------------------------------------------------+
| GPU instance profiles: |
| GPU   Name             ID    Instances   Memory     P2P    SM    DEC   ENC |
| Free/Total   GiB              CE    JPEG  OFA |
| ============================================================================= |
| 0 MIG 1g.10gb        19    7/7         9.50       No     14     0     0 |
| 1     0     0 |
+-----------------------------------------------------------------------------+
| 0 MIG 1g.10gb+me     20    1/1         9.50       No     14     0     0 |
| 1     1     1 |
+-----------------------------------------------------------------------------+
| 0 MIG 2g.20gb        14    3/3         19.50      No     28     1     0 |
| 2     0     0 |
+-----------------------------------------------------------------------------+
| 0 MIG 3g.40gb         9    2/2         39.50      No     42     2     0 |
| 3     0     0 |
+-----------------------------------------------------------------------------+
| 0 MIG 4g.40gb         5    1/1         39.50      No     56     2     0 |
| 4     0     0 |
+-----------------------------------------------------------------------------+
| 0 MIG 7g.80gb         0    1/1         79.25      No     98     0     0 |
| 7     1     1 |
+-----------------------------------------------------------------------------+

2. After creating a MIG GPU Instance, check it.

   • Create GPU Instance

     Sample Code Download

     Copy Code

     Color mode

     $ nvidia-smi mig -i [GPU ID] -cgi [Profile ID]

     $ nvidia-smi mig -i [GPU ID] -cgi [Profile ID]

     Sample Code Download

     Copy Code

     Color mode

     $ nvidia-smi mig -i 0 -cgi 0
     Successfully created GPU instance ID 0 on GPU 0 using profile MIG 7g.80gb (ID 0)

     $ nvidia-smi mig -i 0 -cgi 0
     Successfully created GPU instance ID 0 on GPU 0 using profile MIG 7g.80gb (ID 0)

   • Check GPU Instance

     Sample Code Download

     Copy Code

     Color mode

     $ nvidia-smi mig -i [GPU ID] -lgi

     $ nvidia-smi mig -i [GPU ID] -lgi

     Sample Code Download

     Copy Code

     Color mode

     $ nvidia-smi mig -i 0 -lgi
     +--------------------------------------------------------+
     | GPU instances: |
     | GPU   Name               Profile  Instance  Placement |
     | ID       ID      Start:Size |
     | ======================================================== |
     | 0  MIG 7g.80gb            0        0         0:8 |
     +--------------------------------------------------------+

     $ nvidia-smi mig -i 0 -lgi
     +--------------------------------------------------------+
     | GPU instances: |
     | GPU   Name               Profile  Instance  Placement |
     | ID       ID      Start:Size |
     | ======================================================== |
     | 0  MIG 7g.80gb            0        0         0:8 |
     +--------------------------------------------------------+

Compute Instance creation

If you have created a GPU Instance, you can create a Compute Instance.

1. Check the MIG Compute Instance profiles you can create.

   Sample Code Download

   Copy Code

   Color mode

   $ nvidia-smi mig -i [GPU ID] -gi [GPU Instance ID] -lcip

   $ nvidia-smi mig -i [GPU ID] -gi [GPU Instance ID] -lcip

   Sample Code Download

   Copy Code

   Color mode

   $ nvidia-smi mig -i 0 -gi 0 -lcip
   +---------------------------------------------------------------------------------+
   | Compute instance profiles: |
   | GPU     GPU     Name            Profile  Instances   Exclusive      Shared |
   | GPU   Instance                     ID    Free/Total     SM       DEC  ENC  OFA |
   | ID                                                       CE   JPEG |
   | ================================================================================= |
   | 0      0      MIG 1c.7g.80gb     0      7/5           14       5    0    1 |
   | 7    1 |
   +---------------------------------------------------------------------------------+
   | 0      0      MIG 2c.7g.80gb     1      3/3           28       5    0    1 |
   | 7    1 |
   +---------------------------------------------------------------------------------+
   | 0      0      MIG 3c.7g.80gb     2      2/2           42       5    0    1 |
   | 7    1 |
   +---------------------------------------------------------------------------------+
   | 0      0      MIG 4c.7g.80gb     3      1/1           56       5    0    1 |
   | 7    1 |
   +---------------------------------------------------------------------------------+
   | 0      0      MIG 7g.80gb        4*     1/1           98       5    0    1 |
   | 7    1 |
   +---------------------------------------------------------------------------------+

   $ nvidia-smi mig -i 0 -gi 0 -lcip
   +---------------------------------------------------------------------------------+
   | Compute instance profiles: |
   | GPU     GPU     Name            Profile  Instances   Exclusive      Shared |
   | GPU   Instance                     ID    Free/Total     SM       DEC  ENC  OFA |
   | ID                                                       CE   JPEG |
   | ================================================================================= |
   | 0      0      MIG 1c.7g.80gb     0      7/5           14       5    0    1 |
   | 7    1 |
   +---------------------------------------------------------------------------------+
   | 0      0      MIG 2c.7g.80gb     1      3/3           28       5    0    1 |
   | 7    1 |
   +---------------------------------------------------------------------------------+
   | 0      0      MIG 3c.7g.80gb     2      2/2           42       5    0    1 |
   | 7    1 |
   +---------------------------------------------------------------------------------+
   | 0      0      MIG 4c.7g.80gb     3      1/1           56       5    0    1 |
   | 7    1 |
   +---------------------------------------------------------------------------------+
   | 0      0      MIG 7g.80gb        4*     1/1           98       5    0    1 |
   | 7    1 |
   +---------------------------------------------------------------------------------+

2. Create and verify a MIG Compute Instance.

   • MIG Compute Instance creation
     Sample Code Download

     Copy Code

     Color mode

     $ nvidia-smi mig -i [GPU ID] -gi [GPU Instance ID] -cci [Compute Profile ID]

     $ nvidia-smi mig -i [GPU ID] -gi [GPU Instance ID] -cci [Compute Profile ID]

     Sample Code Download

     Copy Code

     Color mode

     $ nvidia-smi mig -i 0 -gi 0 -cci 4
     Successfully created compute instance ID 0 on GPU instance ID 0 using profile MIG 7g.80gb(ID 4)

     $ nvidia-smi mig -i 0 -gi 0 -cci 4
     Successfully created compute instance ID 0 on GPU instance ID 0 using profile MIG 7g.80gb(ID 4)

   • Check MIG Compute Instance
     Sample Code Download

     Copy Code

     Color mode

     $ nvidia-smi mig -i [GPU ID] -gi [GPU Instance ID] –lci

     $ nvidia-smi mig -i [GPU ID] -gi [GPU Instance ID] –lci

     Sample Code Download

     Copy Code

     Color mode

     $ nvidia-smi mig -i 0 -gi 0 –lci
     +-----------------------------------------------------------------+
     | Compute instance profiles: |
     | GPU     GPU     Name            Profile  Instances   Placement |
     | GPU   Instance                     ID      ID        Start:Size |
     | ID |
     | ================================================================= |
     | 0      0      MIG 7g.80gb         4       0            0:7 |
     +-----------------------------------------------------------------+

     $ nvidia-smi mig -i 0 -gi 0 –lci
     +-----------------------------------------------------------------+
     | Compute instance profiles: |
     | GPU     GPU     Name            Profile  Instances   Placement |
     | GPU   Instance                     ID      ID        Start:Size |
     | ID |
     | ================================================================= |
     | 0      0      MIG 7g.80gb         4       0            0:7 |
     +-----------------------------------------------------------------+

     Sample Code Download

     Copy Code

     Color mode

     $ nvidia-smi –L
     GPU 0: NVIDIA A100-SXM-80GB (UUID: GPU-c956838f-494a-92b2-6818-56eb28fe25e0)
       MIG 7g.80gb     Device  0: (UUID: MIG-53e20040-758b-5ecb-948e-c626d03a9a32)

     $ nvidia-smi –L
     GPU 0: NVIDIA A100-SXM-80GB (UUID: GPU-c956838f-494a-92b2-6818-56eb28fe25e0)
       MIG 7g.80gb     Device  0: (UUID: MIG-53e20040-758b-5ecb-948e-c626d03a9a32)

     Sample Code Download

     Copy Code

     Color mode

     $ nvidia-smi
     Mon Sep 27 09:52:17 2021
     +-----------------------------------------------------------------------------+
     | NVIDIA-SMI 470.57.02    Driver Version: 470.57.02    CUDA Version: 11.4 |
     | -------------------------------+----------------------+---------------------- |
     | GPU  Name        Persistence-M | Bus-Id        Disp.A | Volatile Uncorr. ECC |
     | Fan  Temp  Perf  Pwr:Usage/Cap | Memory-Usage | GPU-Util  Compute M. |
     |  |  | MIG M. |
     | ===============================+======================+====================== |
     | 0  NVDIA A100-SXM...  Off | 00000000:05:00.0 Off | On |
     | N/A   32C   P0    49W / 400W | 0MiB / 81251MiB | N/A      Default |
     |  |  | Enabled |
     +-------------------------------+----------------------+----------------------+
     
     +-----------------------------------------------------------------------------+
     | MIG devices: |
     +-----------------------------------------------------------------------------+
     | GPU  GI  CI  MIG | Memory-Usage | Vol | Shared |
     | ID  ID  Dev | BAR1-Usage | SM     Unc | CE  ENC  DEC  OFA  JPG |
     |  |  | ECC |  |
     | ============================================================================= |
     | 0    0   0    0 | 0MiB / 81251MiB | 98      0 | 7   0    5    1    1 |
     |  | 1MiB / 13107... |  |  |
     +-----------------------------------------------------------------------------+
     +-----------------------------------------------------------------------------+
     | Processes: |
     | GPU   GI   CI       PID   Type   Process name                   GPU Memory |
     | ID   ID                                                   Usage |
     | ============================================================================= |
     | No running processes found |
     +-----------------------------------------------------------------------------+

     $ nvidia-smi
     Mon Sep 27 09:52:17 2021
     +-----------------------------------------------------------------------------+
     | NVIDIA-SMI 470.57.02    Driver Version: 470.57.02    CUDA Version: 11.4 |
     | -------------------------------+----------------------+---------------------- |
     | GPU  Name        Persistence-M | Bus-Id        Disp.A | Volatile Uncorr. ECC |
     | Fan  Temp  Perf  Pwr:Usage/Cap | Memory-Usage | GPU-Util  Compute M. |
     |  |  | MIG M. |
     | ===============================+======================+====================== |
     | 0  NVDIA A100-SXM...  Off | 00000000:05:00.0 Off | On |
     | N/A   32C   P0    49W / 400W | 0MiB / 81251MiB | N/A      Default |
     |  |  | Enabled |
     +-------------------------------+----------------------+----------------------+
     
     +-----------------------------------------------------------------------------+
     | MIG devices: |
     +-----------------------------------------------------------------------------+
     | GPU  GI  CI  MIG | Memory-Usage | Vol | Shared |
     | ID  ID  Dev | BAR1-Usage | SM     Unc | CE  ENC  DEC  OFA  JPG |
     |  |  | ECC |  |
     | ============================================================================= |
     | 0    0   0    0 | 0MiB / 81251MiB | 98      0 | 7   0    5    1    1 |
     |  | 1MiB / 13107... |  |  |
     +-----------------------------------------------------------------------------+
     +-----------------------------------------------------------------------------+
     | Processes: |
     | GPU   GI   CI       PID   Type   Process name                   GPU Memory |
     | ID   ID                                                   Usage |
     | ============================================================================= |
     | No running processes found |
     +-----------------------------------------------------------------------------+

Using MIG

1. Use the MIG Instance to perform the Job.
   • Example of task execution
     Sample Code Download

     Copy Code

     Color mode

     $ docker run --gpus '"device=[GPU ID]:[MIG ID]"' -rm nvcr.io/nvidia/cuda nvidia-smi

     $ docker run --gpus '"device=[GPU ID]:[MIG ID]"' -rm nvcr.io/nvidia/cuda nvidia-smi

   • You can see an example of the work performed as follows.
     Sample Code Download

     Copy Code

     Color mode

     $ docker run --gpus '"device=0:0"' -rm -it --network=host --shm-size=1g --ipc=host -v /root/.ssh/:/root/.ssh
     
     ================
     == TensorFlow ==
     ================
     
     NVIDIA Release 21.08-tf1 (build 26012104)
     TensorFlow Version 1.15.5
     
     Container image Copyright (c) 2021, NVIDIA CORPORATION. All right reserved.
     ...
     
     # Run Python process
     root@d622a93c9281:/workspace# python /workspace/nvidia-examples/cnn/resnet.py --num_iter 100
     ...
     PY 3.8.10 (default, Jun 2 2021, 10:49:15)
     [GCC 9.4.0]
     TF 1.15.5
     ...

     $ docker run --gpus '"device=0:0"' -rm -it --network=host --shm-size=1g --ipc=host -v /root/.ssh/:/root/.ssh
     
     ================
     == TensorFlow ==
     ================
     
     NVIDIA Release 21.08-tf1 (build 26012104)
     TensorFlow Version 1.15.5
     
     Container image Copyright (c) 2021, NVIDIA CORPORATION. All right reserved.
     ...
     
     # Run Python process
     root@d622a93c9281:/workspace# python /workspace/nvidia-examples/cnn/resnet.py --num_iter 100
     ...
     PY 3.8.10 (default, Jun 2 2021, 10:49:15)
     [GCC 9.4.0]
     TF 1.15.5
     ...

2. Check the GPU usage. (Create JOB process)
   • When the job runs, you can see that a process is allocated to the MIG device and its utilization increases.
     Sample Code Download

     Copy Code

     Color mode

     $ nvidia-smi mig -i [GPU ID] -gi [GPU Instance ID] -lcip

     $ nvidia-smi mig -i [GPU ID] -gi [GPU Instance ID] -lcip

   • You can check the GPU usage as shown below.
     Sample Code Download

     Copy Code

     Color mode

     +-----------------------------------------------------------------------------+
     | MIG devices: |
     +-----------------------------------------------------------------------------+
     | GPU  GI  CI  MIG | Memory-Usage | Vol | Shared |
     | ID  ID  Dev | BAR1-Usage | SM     Unc | CE  ENC  DEC  OFA  JPG |
     |  |  | ECC |  |
     | ============================================================================= |
     | 0    0   0    0 | 66562MiB / 81251MiB | 98      0 | 7   0    5    1    1 |
     |  | 5MiB / 13107... |  |  |
     +-----------------------------------------------------------------------------+
     +-----------------------------------------------------------------------------+
     | Processes: |
     | GPU   GI   CI       PID   Type   Process name                   GPU Memory |
     | ID   ID                                                   Usage |
     | ============================================================================= |
     | 0     0    0     17483      C   python                           66559MiB |
     +-----------------------------------------------------------------------------+

     +-----------------------------------------------------------------------------+
     | MIG devices: |
     +-----------------------------------------------------------------------------+
     | GPU  GI  CI  MIG | Memory-Usage | Vol | Shared |
     | ID  ID  Dev | BAR1-Usage | SM     Unc | CE  ENC  DEC  OFA  JPG |
     |  |  | ECC |  |
     | ============================================================================= |
     | 0    0   0    0 | 66562MiB / 81251MiB | 98      0 | 7   0    5    1    1 |
     |  | 5MiB / 13107... |  |  |
     +-----------------------------------------------------------------------------+
     +-----------------------------------------------------------------------------+
     | Processes: |
     | GPU   GI   CI       PID   Type   Process name                   GPU Memory |
     | ID   ID                                                   Usage |
     | ============================================================================= |
     | 0     0    0     17483      C   python                           66559MiB |
     +-----------------------------------------------------------------------------+

Delete and release MIG Instance

Follow these steps to delete the MIG instance and detach the MIG.

Compute Instance Delete

• Delete the Compute Instance.
  Sample Code Download

  Copy Code

  Color mode

  $ nvidia-smi mig -i [GPU ID] -gi [GPU Instance ID] –dci
  $ nvidia-smi mig -i [GPU ID] -gi [GPU Instance ID] -ci [Compute Instance] –dci

  $ nvidia-smi mig -i [GPU ID] -gi [GPU Instance ID] –dci
  $ nvidia-smi mig -i [GPU ID] -gi [GPU Instance ID] -ci [Compute Instance] –dci

  Sample Code Download

  Copy Code

  Color mode

  $ nvidia-smi mig -i 0 -gi 0 –lci
  +-----------------------------------------------------------------+
  | Compute instance profiles: |
  | GPU     GPU     Name            Profile  Instances   Placement |
  | GPU   Instance                     ID      ID        Start:Size |
  | ID |
  | ================================================================= |
  | 0      0      MIG 7g.80gb         4       0            0:7 |
  +-----------------------------------------------------------------+

  $ nvidia-smi mig -i 0 -gi 0 –lci
  +-----------------------------------------------------------------+
  | Compute instance profiles: |
  | GPU     GPU     Name            Profile  Instances   Placement |
  | GPU   Instance                     ID      ID        Start:Size |
  | ID |
  | ================================================================= |
  | 0      0      MIG 7g.80gb         4       0            0:7 |
  +-----------------------------------------------------------------+

  Sample Code Download

  Copy Code

  Color mode

  $ nvidia-smi mig -i 0 -gi 0 –dci
  Successfully destroyed compute instance ID  0 from GPU instance ID  0

  $ nvidia-smi mig -i 0 -gi 0 –dci
  Successfully destroyed compute instance ID  0 from GPU instance ID  0

  Sample Code Download

  Copy Code

  Color mode

  $ nvidia-smi mig -i 0 -gi 0 –lci
  No compute instances found: Not found

  $ nvidia-smi mig -i 0 -gi 0 –lci
  No compute instances found: Not found

Delete GPU Instance

• Delete the GPU Instance.
  Sample Code Download

  Copy Code

  Color mode

  $ nvidia-smi mig -i [GPU ID] –dgi
  $ nvidia-smi mig -i [GPU ID] -gi [GPU Instance ID] –dgi

  $ nvidia-smi mig -i [GPU ID] –dgi
  $ nvidia-smi mig -i [GPU ID] -gi [GPU Instance ID] –dgi

  Sample Code Download

  Copy Code

  Color mode

  $ nvidia-smi mig -i 0 -lgi
  +--------------------------------------------------------+
  | GPU instances: |
  | GPU   Name               Profile  Instance  Placement |
  | ID       ID      Start:Size |
  | ======================================================== |
  | 0  MIG 7g.80gb            0        0         0:8 |
  +--------------------------------------------------------+

  $ nvidia-smi mig -i 0 -lgi
  +--------------------------------------------------------+
  | GPU instances: |
  | GPU   Name               Profile  Instance  Placement |
  | ID       ID      Start:Size |
  | ======================================================== |
  | 0  MIG 7g.80gb            0        0         0:8 |
  +--------------------------------------------------------+

  Sample Code Download

  Copy Code

  Color mode

  $ nvidia-smi mig -i 0 -dgi
  Successfully destroyed GPU instance ID  0 from GPU  0

  $ nvidia-smi mig -i 0 -dgi
  Successfully destroyed GPU instance ID  0 from GPU  0

  Sample Code Download

  Copy Code

  Color mode

  $ nvidia-smi mig -i 0 -lgi
  No GPU instances found: Not found

  $ nvidia-smi mig -i 0 -lgi
  No GPU instances found: Not found

Disable MIG feature (deactivation)

• After disabling MIG (Disable), reboot.
  Sample Code Download

  Copy Code

  Color mode

  $ nvidia-smi -mig 0
  Disabled MIG Mode for GPU 00000000:05:00.0
  
  All done.

  $ nvidia-smi -mig 0
  Disabled MIG Mode for GPU 00000000:05:00.0
  
  All done.

  Sample Code Download

  Copy Code

  Color mode

  $ nvidia-smi
  Mon Sep 30 05:18:28 2021
  +-----------------------------------------------------------------------------+
  | NVIDIA-SMI 470.57.02    Driver Version: 470.57.02    CUDA Version: 11.4 |
  | -------------------------------+----------------------+---------------------- |
  | GPU  Name        Persistence-M | Bus-Id        Disp.A | Volatile Uncorr. ECC |
  | Fan  Temp  Perf  Pwr:Usage/Cap | Memory-Usage | GPU-Util  Compute M. |
  |  |  | MIG M. |
  | ===============================+======================+====================== |
  | 0  NVDIA A100-SXM...  Off | 00000000:05:00.0 Off | 0 |
  | N/A   33C   P0    60W / 400W | 0MiB / 81251MiB | 0%      Default |
  |  |  | Disabled |
  +-------------------------------+----------------------+----------------------+
  +-----------------------------------------------------------------------------+
  | MIG devices: |
  +-----------------------------------------------------------------------------+
  | GPU  GI  CI  MIG | Memory-Usage | Vol | Shared |
  | ID  ID  Dev | BAR1-Usage | SM     Unc | CE  ENC  DEC  OFA  JPG |
  |  |  | ECC |  |
  | ============================================================================= |
  | No MIG devices found |
  +-----------------------------------------------------------------------------+
  +-----------------------------------------------------------------------------+
  | Processes: |
  | GPU   GI   CI       PID   Type   Process name                   GPU Memory |
  | ID   ID                                                   Usage |
  | ============================================================================= |
  | No running processes found |
  +-----------------------------------------------------------------------------+

  $ nvidia-smi
  Mon Sep 30 05:18:28 2021
  +-----------------------------------------------------------------------------+
  | NVIDIA-SMI 470.57.02    Driver Version: 470.57.02    CUDA Version: 11.4 |
  | -------------------------------+----------------------+---------------------- |
  | GPU  Name        Persistence-M | Bus-Id        Disp.A | Volatile Uncorr. ECC |
  | Fan  Temp  Perf  Pwr:Usage/Cap | Memory-Usage | GPU-Util  Compute M. |
  |  |  | MIG M. |
  | ===============================+======================+====================== |
  | 0  NVDIA A100-SXM...  Off | 00000000:05:00.0 Off | 0 |
  | N/A   33C   P0    60W / 400W | 0MiB / 81251MiB | 0%      Default |
  |  |  | Disabled |
  +-------------------------------+----------------------+----------------------+
  +-----------------------------------------------------------------------------+
  | MIG devices: |
  +-----------------------------------------------------------------------------+
  | GPU  GI  CI  MIG | Memory-Usage | Vol | Shared |
  | ID  ID  Dev | BAR1-Usage | SM     Unc | CE  ENC  DEC  OFA  JPG |
  |  |  | ECC |  |
  | ============================================================================= |
  | No MIG devices found |
  +-----------------------------------------------------------------------------+
  +-----------------------------------------------------------------------------+
  | Processes: |
  | GPU   GI   CI       PID   Type   Process name                   GPU Memory |
  | ID   ID                                                   Usage |
  | ============================================================================= |
  | No running processes found |
  +-----------------------------------------------------------------------------+

2.4 - Use NVSwitch on GPU Server

After creating a GPU Server, you can enable the NVSwitch feature on the GPU Server’s VM (Guest OS) and use fast GPU-to-GPU P2P communication.

Exploring NVIDIA NVSwitch for Multi GPU

NVLink expands I/O by directly connecting multiple GPUs within a server both bidirectionally and GPU-to-GPU. Using NVSwitch, you can connect all GPUs in a server with full NVLink bandwidth.

Checking NVSwitch operation

Check the NVIDIA Fabric Manager, NVIDIA NVLink topology, and NVIDIA NVLink Status on the GPU server.

NVIDIA Fabric Manager operating status

Verify that active (running) is displayed when operating normally.

~$ systemctl status nvidia-fabricmanager

nvidia-fabricmanager.service - NVIDIA fabric manager service
     Loaded: loaded (/lib/systemd/system/nvidia-fabricmanager.service; enabled; vendor preset: enabled)
     Active: active (running) since Mon 2026-02-02 16:23:27 KST; 32min ago
   Main PID: 2191 (nv-fabricmanage)
      Tasks: 18 (limit: 629145)
     Memory: 18.0M
        CPU: 33.461s
     CGroup: /system.slice/nvidia-fabricmanager.service
             └─2191 /usr/bin/nv-fabricmanager -c /usr/share/nvidia/nvswitch/fabricmanager.cfg

nvidia-fabricmanager.service - NVIDIA fabric manager service
     Loaded: loaded (/lib/systemd/system/nvidia-fabricmanager.service; enabled; vendor preset: enabled)
     Active: active (running) since Mon 2026-02-02 16:23:27 KST; 32min ago
   Main PID: 2191 (nv-fabricmanage)
      Tasks: 18 (limit: 629145)
     Memory: 18.0M
        CPU: 33.461s
     CGroup: /system.slice/nvidia-fabricmanager.service
             └─2191 /usr/bin/nv-fabricmanager -c /usr/share/nvidia/nvswitch/fabricmanager.cfg

Check NVIDIA NVLink topology

Check the NVIDIA NVLink topology.

~$ nvidia-smi topo -m

nvidia-smi topo -m
        GPU0    GPU1    GPU2    GPU3    GPU4    GPU5    GPU6    GPU7    CPU Affinity    NUMA Affinity   GPU NUMA ID
GPU0     X      NV12    NV12    NV12    NV12    NV12    NV12    NV12    0-127   0-7             N/A
GPU1    NV12     X      NV12    NV12    NV12    NV12    NV12    NV12    0-127   0-7             N/A
GPU2    NV12    NV12     X      NV12    NV12    NV12    NV12    NV12    0-127   0-7             N/A
GPU3    NV12    NV12    NV12     X      NV12    NV12    NV12    NV12    0-127   0-7             N/A
GPU4    NV12    NV12    NV12    NV12     X      NV12    NV12    NV12    0-127   0-7             N/A
GPU5    NV12    NV12    NV12    NV12    NV12     X      NV12    NV12    0-127   0-7             N/A
GPU6    NV12    NV12    NV12    NV12    NV12    NV12     X      NV12    0-127   0-7             N/A
GPU7    NV12    NV12    NV12    NV12    NV12    NV12    NV12     X      0-127   0-7             N/A

Legend:

  X    = Self
  SYS  = Connection traversing PCIe as well as the SMP interconnect between NUMA nodes (e.g., QPI/UPI)
  NODE = Connection traversing PCIe as well as the interconnect between PCIe Host Bridges within a NUMA node
  PHB  = Connection traversing PCIe as well as a PCIe Host Bridge (typically the CPU)
  PXB  = Connection traversing multiple PCIe bridges (without traversing the PCIe Host Bridge)
  PIX  = Connection traversing at most a single PCIe bridge
  NV#  = Connection traversing a bonded set of # NVLinks

nvidia-smi topo -m
        GPU0    GPU1    GPU2    GPU3    GPU4    GPU5    GPU6    GPU7    CPU Affinity    NUMA Affinity   GPU NUMA ID
GPU0     X      NV12    NV12    NV12    NV12    NV12    NV12    NV12    0-127   0-7             N/A
GPU1    NV12     X      NV12    NV12    NV12    NV12    NV12    NV12    0-127   0-7             N/A
GPU2    NV12    NV12     X      NV12    NV12    NV12    NV12    NV12    0-127   0-7             N/A
GPU3    NV12    NV12    NV12     X      NV12    NV12    NV12    NV12    0-127   0-7             N/A
GPU4    NV12    NV12    NV12    NV12     X      NV12    NV12    NV12    0-127   0-7             N/A
GPU5    NV12    NV12    NV12    NV12    NV12     X      NV12    NV12    0-127   0-7             N/A
GPU6    NV12    NV12    NV12    NV12    NV12    NV12     X      NV12    0-127   0-7             N/A
GPU7    NV12    NV12    NV12    NV12    NV12    NV12    NV12     X      0-127   0-7             N/A

Legend:

  X    = Self
  SYS  = Connection traversing PCIe as well as the SMP interconnect between NUMA nodes (e.g., QPI/UPI)
  NODE = Connection traversing PCIe as well as the interconnect between PCIe Host Bridges within a NUMA node
  PHB  = Connection traversing PCIe as well as a PCIe Host Bridge (typically the CPU)
  PXB  = Connection traversing multiple PCIe bridges (without traversing the PCIe Host Bridge)
  PIX  = Connection traversing at most a single PCIe bridge
  NV#  = Connection traversing a bonded set of # NVLinks

Check NVIDIA NVLink Status

Check the NVIDIA NVLink Status.

~$ nvidia-smi topo -m

GPU 1: NVIDIA A100-SXM4-80GB (UUID: GPU-64a2f685-bb12-c4af-105c-0726ece9c8d7)
         Link 0: 25 GB/s
         Link 1: 25 GB/s
         Link 2: 25 GB/s
         Link 3: 25 GB/s
         Link 4: 25 GB/s
         Link 5: 25 GB/s
         Link 6: 25 GB/s
         Link 7: 25 GB/s
         Link 8: 25 GB/s
         Link 9: 25 GB/s
         Link 10: 25 GB/s
         Link 11: 25 GB/s
GPU 2: NVIDIA A100-SXM4-80GB (UUID: GPU-2269851b-71cd-f6c7-50c5-ba1525cf3ce8)
         Link 0: 25 GB/s
         Link 1: 25 GB/s
         Link 2: 25 GB/s
         Link 3: 25 GB/s
         Link 4: 25 GB/s
         Link 5: 25 GB/s
         Link 6: 25 GB/s
         Link 7: 25 GB/s
         Link 8: 25 GB/s
         Link 9: 25 GB/s
         Link 10: 25 GB/s
         Link 11: 25 GB/s
GPU 3: NVIDIA A100-SXM4-80GB (UUID: GPU-4c397bbf-95fc-5c29-918a-a429cbe45a7a)
         Link 0: 25 GB/s
         Link 1: 25 GB/s
         Link 2: 25 GB/s
         Link 3: 25 GB/s
         Link 4: 25 GB/s
         Link 5: 25 GB/s
         Link 6: 25 GB/s
         Link 7: 25 GB/s
         Link 8: 25 GB/s
         Link 9: 25 GB/s
         Link 10: 25 GB/s
         Link 11: 25 GB/s
GPU 4: NVIDIA A100-SXM4-80GB (UUID: GPU-0e350204-9fb6-2cbe-538e-8f7849658eb8)
         Link 0: 25 GB/s
         Link 1: 25 GB/s
         Link 2: 25 GB/s
         Link 3: 25 GB/s
         Link 4: 25 GB/s
         Link 5: 25 GB/s
         Link 6: 25 GB/s
         Link 7: 25 GB/s
         Link 8: 25 GB/s
         Link 9: 25 GB/s
         Link 10: 25 GB/s
         Link 11: 25 GB/s
GPU 5: NVIDIA A100-SXM4-80GB (UUID: GPU-45f0c453-4760-edd4-3af9-25c5ea7473a5)
         Link 0: 25 GB/s
         Link 1: 25 GB/s
         Link 2: 25 GB/s
         Link 3: 25 GB/s
         Link 4: 25 GB/s
         Link 5: 25 GB/s
         Link 6: 25 GB/s
         Link 7: 25 GB/s
         Link 8: 25 GB/s
         Link 9: 25 GB/s
         Link 10: 25 GB/s
         Link 11: 25 GB/s
GPU 6: NVIDIA A100-SXM4-80GB (UUID: GPU-38409794-bb34-430e-3c50-90b42cb2bb72)
         Link 0: 25 GB/s
         Link 1: 25 GB/s
         Link 2: 25 GB/s
         Link 3: 25 GB/s
         Link 4: 25 GB/s
         Link 5: 25 GB/s
         Link 6: 25 GB/s
         Link 7: 25 GB/s
         Link 8: 25 GB/s
         Link 9: 25 GB/s
         Link 10: 25 GB/s
         Link 11: 25 GB/s
GPU 7: NVIDIA A100-SXM4-80GB (UUID: GPU-3fb478aa-801b-eb64-55c2-0ffc3f2ce404)
         Link 0: 25 GB/s
         Link 1: 25 GB/s
         Link 2: 25 GB/s
         Link 3: 25 GB/s
         Link 4: 25 GB/s
         Link 5: 25 GB/s
         Link 6: 25 GB/s
         Link 7: 25 GB/s
         Link 8: 25 GB/s
         Link 9: 25 GB/s
         Link 10: 25 GB/s
         Link 11: 25 GB/s

GPU 1: NVIDIA A100-SXM4-80GB (UUID: GPU-64a2f685-bb12-c4af-105c-0726ece9c8d7)
         Link 0: 25 GB/s
         Link 1: 25 GB/s
         Link 2: 25 GB/s
         Link 3: 25 GB/s
         Link 4: 25 GB/s
         Link 5: 25 GB/s
         Link 6: 25 GB/s
         Link 7: 25 GB/s
         Link 8: 25 GB/s
         Link 9: 25 GB/s
         Link 10: 25 GB/s
         Link 11: 25 GB/s
GPU 2: NVIDIA A100-SXM4-80GB (UUID: GPU-2269851b-71cd-f6c7-50c5-ba1525cf3ce8)
         Link 0: 25 GB/s
         Link 1: 25 GB/s
         Link 2: 25 GB/s
         Link 3: 25 GB/s
         Link 4: 25 GB/s
         Link 5: 25 GB/s
         Link 6: 25 GB/s
         Link 7: 25 GB/s
         Link 8: 25 GB/s
         Link 9: 25 GB/s
         Link 10: 25 GB/s
         Link 11: 25 GB/s
GPU 3: NVIDIA A100-SXM4-80GB (UUID: GPU-4c397bbf-95fc-5c29-918a-a429cbe45a7a)
         Link 0: 25 GB/s
         Link 1: 25 GB/s
         Link 2: 25 GB/s
         Link 3: 25 GB/s
         Link 4: 25 GB/s
         Link 5: 25 GB/s
         Link 6: 25 GB/s
         Link 7: 25 GB/s
         Link 8: 25 GB/s
         Link 9: 25 GB/s
         Link 10: 25 GB/s
         Link 11: 25 GB/s
GPU 4: NVIDIA A100-SXM4-80GB (UUID: GPU-0e350204-9fb6-2cbe-538e-8f7849658eb8)
         Link 0: 25 GB/s
         Link 1: 25 GB/s
         Link 2: 25 GB/s
         Link 3: 25 GB/s
         Link 4: 25 GB/s
         Link 5: 25 GB/s
         Link 6: 25 GB/s
         Link 7: 25 GB/s
         Link 8: 25 GB/s
         Link 9: 25 GB/s
         Link 10: 25 GB/s
         Link 11: 25 GB/s
GPU 5: NVIDIA A100-SXM4-80GB (UUID: GPU-45f0c453-4760-edd4-3af9-25c5ea7473a5)
         Link 0: 25 GB/s
         Link 1: 25 GB/s
         Link 2: 25 GB/s
         Link 3: 25 GB/s
         Link 4: 25 GB/s
         Link 5: 25 GB/s
         Link 6: 25 GB/s
         Link 7: 25 GB/s
         Link 8: 25 GB/s
         Link 9: 25 GB/s
         Link 10: 25 GB/s
         Link 11: 25 GB/s
GPU 6: NVIDIA A100-SXM4-80GB (UUID: GPU-38409794-bb34-430e-3c50-90b42cb2bb72)
         Link 0: 25 GB/s
         Link 1: 25 GB/s
         Link 2: 25 GB/s
         Link 3: 25 GB/s
         Link 4: 25 GB/s
         Link 5: 25 GB/s
         Link 6: 25 GB/s
         Link 7: 25 GB/s
         Link 8: 25 GB/s
         Link 9: 25 GB/s
         Link 10: 25 GB/s
         Link 11: 25 GB/s
GPU 7: NVIDIA A100-SXM4-80GB (UUID: GPU-3fb478aa-801b-eb64-55c2-0ffc3f2ce404)
         Link 0: 25 GB/s
         Link 1: 25 GB/s
         Link 2: 25 GB/s
         Link 3: 25 GB/s
         Link 4: 25 GB/s
         Link 5: 25 GB/s
         Link 6: 25 GB/s
         Link 7: 25 GB/s
         Link 8: 25 GB/s
         Link 9: 25 GB/s
         Link 10: 25 GB/s
         Link 11: 25 GB/s

2.5 - Install ServiceWatch Agent

Users can install the ServiceWatch Agent on a GPU server to collect custom metrics and logs.

ServiceWatch Agent

There are two main types of agents that need to be installed on a GPU server to collect custom metrics and logs for ServiceWatch. It is the Prometheus Exporter and Open Telemetry Collector.

Pre-configuration for using ServiceWatch Agent

To use the ServiceWatch Agent, please refer to ServiceWatch Agent를 위한 사전 환경 설정 and prepare the prerequisite configuration.

Install Prometheus Exporter for GPU metrics (for Ubuntu)

Install the Prometheus Exporter for collecting metrics from the GPU server according to the steps below.

Verify NVDIA Driver Installation

• Check the installed NVDIA driver.
  • Tab 0

  Sample Code Download

  Copy Code

  Color mode

  nvidia-smi --query-gpu driver_version --format csv

  nvidia-smi --query-gpu driver_version --format csv

  • Tab 0

  Sample Code Download

  Copy Code

  Color mode

  driver_version
  535.183.06
  
  535.183.06

  driver_version
  535.183.06
  
  535.183.06

NVSwitch Configuration and Query (NSCQ) Library Installation

1. Install cuda-keyring.

   • Tab 0

   Sample Code Download

   Copy Code

   Color mode

   wget https://developer.download.nvidia.com/compute/cuda/repos/<distro>/<arch>/cuda-keyring_1.1-1_all.deb

   wget https://developer.download.nvidia.com/compute/cuda/repos/<distro>/<arch>/cuda-keyring_1.1-1_all.deb

   • Tab 0

   Sample Code Download

   Copy Code

   Color mode

   sudo dpkg -i cuda-keyring_1.1-1_all.deb
   apt update

   sudo dpkg -i cuda-keyring_1.1-1_all.deb
   apt update

   • Tab 0

   Sample Code Download

   Copy Code

   Color mode

   nvidia-smi --query-gpu driver_version --format csv

   nvidia-smi --query-gpu driver_version --format csv

   • Tab 0

   Sample Code Download

   Copy Code

   Color mode

   driver_version
   535.183.06
   ...
   
   535.183.06

   driver_version
   535.183.06
   ...
   
   535.183.06

2. Install libnvidia-nscq.

   • Tab 0

   Sample Code Download

   Copy Code

   Color mode

   apt-cache policy libnvidia-nscq-535

   apt-cache policy libnvidia-nscq-535

   • Tab 0

   Sample Code Download

   Copy Code

   Color mode

   libnvidia-nscq-535:
     Installed: (none)
     Candidate: 535.247.01-1
     Version table:
        535.247.01-1 600
           600 https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2204/x86_64  Packages
   ...
           600 https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2204/x86_64  Packages
        535.216.01-1 600
           600 https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2204/x86_64  Packages
        535.183.06-1 600  # Install the version that matches the Driver
           600 https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2204/x86_64  Packages
        535.183.01-1 600
   ...
           600 https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2204/x86_64  Packages
        535.54.03-1 600
           600 https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2204/x86_64  Packages

   libnvidia-nscq-535:
     Installed: (none)
     Candidate: 535.247.01-1
     Version table:
        535.247.01-1 600
           600 https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2204/x86_64  Packages
   ...
           600 https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2204/x86_64  Packages
        535.216.01-1 600
           600 https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2204/x86_64  Packages
        535.183.06-1 600  # Install the version that matches the Driver
           600 https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2204/x86_64  Packages
        535.183.01-1 600
   ...
           600 https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2204/x86_64  Packages
        535.54.03-1 600
           600 https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2204/x86_64  Packages

   • Tab 0

   Sample Code Download

   Copy Code

   Color mode

   apt install libnvidia-nscq-535=535.183.06-1

   apt install libnvidia-nscq-535=535.183.06-1

NVSwitch Device Monitoring API(NVSDM) Library Installation

• Install the NVSDM library.
  • Tab 0

  Sample Code Download

  Copy Code

  Color mode

  apt-cache policy libnvsdm

  apt-cache policy libnvsdm

  • Tab 0

  Sample Code Download

  Copy Code

  Color mode

  libnvsdm:
    Installed: (none)
    Candidate: 580.105.08-1
    Version table:
       580.105.08-1 600
          600 https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2204/x86_64  Packages
       580.95.05-1 600
          600 https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2204/x86_64  Packages
       580.82.07-1 600
          600 https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2204/x86_64  Packages
       580.65.06-1 600
          600 https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2204/x86_64  Packages

  libnvsdm:
    Installed: (none)
    Candidate: 580.105.08-1
    Version table:
       580.105.08-1 600
          600 https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2204/x86_64  Packages
       580.95.05-1 600
          600 https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2204/x86_64  Packages
       580.82.07-1 600
          600 https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2204/x86_64  Packages
       580.65.06-1 600
          600 https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2204/x86_64  Packages

  • Tab 0

  Sample Code Download

  Copy Code

  Color mode

  apt install libnvsdm=580.105.08-1

  apt install libnvsdm=580.105.08-1

NVIDIA DCGM installation (for Ubuntu)

Install the DCGM Exporter according to the steps below.

1. DCGM(datacenter-gpu-manager) Installation
2. datacenter-gpu-manager-exporter Installation
3. DCGM Service Activation and Start

DCGM(datacenter-gpu-manager) Installation

refers to a specific version of NVIDIA’s Data Center GPU Manager (DCGM) tool, which is a package for managing and monitoring NVIDIA data center GPUs. In particular, cuda12 indicates that this management tool is installed for the CUDA 12 version, and datacenter-gpu-manager-4 refers to the 4.x version of DCGM. This tool provides various features, including GPU status monitoring, diagnostics, alert system, and power/clock management.

1. Check the CUDA version.
   • Tab 0

   Sample Code Download

   Copy Code

   Color mode

   nvidia-smi | grep CUDA

   nvidia-smi | grep CUDA

   • Tab 0

   Sample Code Download

   Copy Code

   Color mode

   | NVIDIA-SMI 535.183.06             Driver Version: 535.183.06     CUDA Version: 12.2 |

   | NVIDIA-SMI 535.183.06             Driver Version: 535.183.06     CUDA Version: 12.2 |

   • Tab 0

   Sample Code Download

   Copy Code

   Color mode

   CUDA_VERSION=12

   CUDA_VERSION=12

2. Install datacenter-gpu-manager-cuda.
   • Tab 0

   Sample Code Download

   Copy Code

   Color mode

   apt install datacenter-gpu-manager-4-cuda${CUDA_VERSION}

   apt install datacenter-gpu-manager-4-cuda${CUDA_VERSION}

datacenter-gpu-manager-exporter installation

It is a tool that, based on NVIDIA Data Center GPU Manager (DCGM), collects various GPU metrics such as GPU usage, memory usage, temperature, and power consumption, and exposes them for use in monitoring systems like Prometheus.

1. Install datacenter-gpu-manager-exporter.
   • Tab 0

   Sample Code Download

   Copy Code

   Color mode

   apt install datacenter-gpu-manager-exporter

   apt install datacenter-gpu-manager-exporter

2. Check the DCGM Exporter configuration file.
   • Tab 0

   Sample Code Download

   Copy Code

   Color mode

   cat /usr/lib/systemd/system/nvidia-dcgm-exporter.service | grep ExecStart

   cat /usr/lib/systemd/system/nvidia-dcgm-exporter.service | grep ExecStart

   • Tab 0

   Sample Code Download

   Copy Code

   Color mode

   ExecStart=/usr/bin/dcgm-exporter -f /etc/dcgm-exporter/default-counters.csv

   ExecStart=/usr/bin/dcgm-exporter -f /etc/dcgm-exporter/default-counters.csv

3. When installing the DCGM Exporter, review the provided settings and remove # for required metrics, and add # for unnecessary metrics.
   • Tab 0

   Sample Code Download

   Copy Code

   Color mode

   vi /etc/dcgm-exporter/default-counters.csv
   ## Example ##
   
   DCGM_FI_PROF_PIPE_TENSOR_ACTIVE, gauge, Ratio of cycles the tensor (HMMA) pipe is active.
   DCGM_FI_PROF_DRAM_ACTIVE,        gauge, Ratio of cycles the device memory interface is active sending or receiving data.
   # DCGM_FI_PROF_PIPE_FP64_ACTIVE,   gauge, Ratio of cycles the fp64 pipes are active.
   # DCGM_FI_PROF_PIPE_FP32_ACTIVE,   gauge, Ratio of cycles the fp32 pipes are active.

   vi /etc/dcgm-exporter/default-counters.csv
   ## Example ##
   
   DCGM_FI_PROF_PIPE_TENSOR_ACTIVE, gauge, Ratio of cycles the tensor (HMMA) pipe is active.
   DCGM_FI_PROF_DRAM_ACTIVE,        gauge, Ratio of cycles the device memory interface is active sending or receiving data.
   # DCGM_FI_PROF_PIPE_FP64_ACTIVE,   gauge, Ratio of cycles the fp64 pipes are active.
   # DCGM_FI_PROF_PIPE_FP32_ACTIVE,   gauge, Ratio of cycles the fp32 pipes are active.