Multi-node GPU Cluster

• 1: Overview
• 1.1: Server Type
• 1.2: Monitoring Metrics
• 2: How-to guides
• 2.1: Cluster Fabric Management
• 2.2: ServiceWatch Agent Install
• 2.3: Multi-node GPU Cluster Service Scope and Inspection Guide
• 3: Release Note

1 - Overview

Service Overview

Multi-node GPU Cluster is a service that provides physical GPU servers without virtualization for large-scale high-performance AI calculations. It can cluster multiple GPUs using two or more bare metal servers with GPUs, and can be used conveniently with Samsung Cloud Platform’s high-performance storage and networking services.

Provided Features

Multi-node GPU Cluster provides the following functions.

• Auto Provisioning and Management: Through the web-based Console, you can easily use the standard GPU Bare Metal model server with 8 GPU cards, from provisioning to resource and cost management.
• Network Connection: Two or more Bare Metal Servers can be clustered through high-speed interconnects to process multiple GPUs, and by configuring the GPU Direct RDMA (Remote Direct Memory Access) environment, direct data IO between GPU memories is possible, enabling high-speed AI/Machine Learning calculations.
• Storage Connection: It provides various additional connection storages other than OS disks. High-speed network and high-performance SSD NAS File Storage, Block Storage, and Object Storage that are directly linked can also be used in conjunction.
• Network Setting Management: The server’s subnet/IP can be easily changed with the initially set value. NAT IP provides a management function that can be used or cancelled according to needs.
• Monitoring: You can check the monitoring information of computing resources such as CPU, GPU, Memory, Disk, etc. through Cloud Monitoring. To use the Cloud Monitoring service for Multi-node GPU Cluster, you need to install the Agent. Please install the Agent for stable service use. For more information, please refer to Multi-node GPU Cluster Monitoring Metrics.

Component

Multi-node GPU Cluster provides GPU as a Bare Metal Server type with standard images and server types, and NVSwitch and NVLink are provided.

GPU(H100)

GPU (Graphic Processing Unit) is specialized in parallel calculations that can process a large amount of data quickly, enabling large-scale parallel calculation processing in fields such as artificial intelligence (AI) and data analysis.

The following are the specifications of the GPU Type provided by the Multi-node GPU Cluster service.

OS and GPU Driver Version

The operating systems (OS) supported by Multi-node GPU Cluster are as follows.

Server Type

The server types provided by Multi-node GPU Cluster are as follows. For a detailed description of the server types provided by Multi-node GPU Cluster, please refer to Multi-node GPU Cluster server type.

g2c96h8_metal

Preceding Service

This is a list of services that must be pre-configured before creating this service. Please refer to the guide provided for each service and prepare in advance for more details.

1.1 - Server Type

Multi-node GPU Cluster Server Type

Multi-node GPU Cluster is divided based on the provided GPU Type, and the GPU used in the Multi-node GPU Cluster is determined by the server type selected when creating a GPU Node. Please select the server type according to the specifications of the application you want to run in the Multi-node GPU Cluster.

The server types supported by Multi-node GPU Cluster are in the following format:

g2c96h8_metal

g2 Server Type

The g2 server type is a GPU Bare Metal Server using NVIDIA H100 Tensor Core GPU, suitable for large-scale high-performance AI computing.

• Provides up to 8 NVIDIA H100 Tensor Core GPUs
• Each GPU has 16,896 CUDA cores and 528 Tensor cores
• Supports up to 96 vCPUs and 1,920 GB of memory
• Supports up to 100 Gbps networking speed
• 900GB/s GPU and NVIDIA NVSwitch P2P communication

1.2 - Monitoring Metrics

Multi-node GPU Cluster monitoring metrics

The following table shows the monitoring metrics of Multi-node GPU Cluster that can be checked through Cloud Monitoring.

Multi-node GPU Cluster [Cluster]

Multi-node GPU Cluster [Node]

2 - How-to guides

The user can enter the required information for the Multi-node GPU Cluster service through the Samsung Cloud Platform Console, select detailed options, and create the service.

Multi-node GPU Cluster Getting Started

You can create and use a Multi-node GPU Cluster service in the Samsung Cloud Platform Console.

This service consists of GPU Node and Cluster Fabric services.

GPU Node Creation

To create a Multi-node GPU Cluster, follow the steps below.

1. All Services > Compute > Multi-node GPU Cluster Click the menu. Navigate to the Service Home page of Multi-node GPU Cluster.
2. Click the GPU Node creation button on the Service Home page. You will be taken to the GPU Node creation page.
3. GPU Node creation on the page, enter the information required to create the service, and select detailed options.
   • Image and Version Selection Select the required information in the area.

     Category	Required	Detailed description
     Image	Required	Select provided image type Ubuntu
     Image Version	Required	Select version of the chosen image Provides a list of versions of the provided server images

     Table. GPU Node image and version selection items

• Enter service information area, input or select the required information.

  Category	Required	Detailed description
  Number of servers	Required	Number of GPU Node servers to create simultaneously Only numbers can be entered, and the minimum number of servers to create is 2. Only during the initial setup can you create 2 or more, and expansion is possible one at a time.
  Service Type > Server Type	Required	GPU Node Server Type Select desired CPU, Memory, GPU, Disk specifications For detailed information about the server types provided by GPU Node, refer to Multi-node GPU Cluster Server Type
  Service Type > Planned Compute	Required	Status of resources with Planned Compute set In Use: Number of resources with Planned Compute that are currently in use Configured: Number of resources with Planned Compute set Coverage Preview: Amount applied per resource by Planned Compute Planned Compute Service Application: Go to the Planned Compute service application page For details, refer to Apply for Planned Compute

  Table. GPU Node Service Information Input Items
  • Required Information Input area, enter or select the required information.

    Category	Required or not	Detailed description
    Administrator Account	Required	Set the administrator account and password to be used when connecting to the server Ubuntu OS is provided fixed as root
    Server Name Prefix	Required	Enter a Prefix to distinguish each GPU Node generated when the number of selected servers is 2 or more Automatically generated as user input value (prefix) + ‘-###’ format Start with a lowercase English letter, and use lowercase letters, numbers, and special characters (-) within 3 to 11 characters Must not end with a special character (-)
    Network Settings	Required	Set the network where the GPU Node will be installed VPC Name:Select a pre-created VPC General Subnet Name: Select a pre-created general Subnet IP can be set to auto-generate or user input, and if input is selected, the user enters the IP directly NAT: Can be used only when there is 1 server and the VPC has an Internet Gateway attached. Checking ‘use’ allows selection of a NAT IP. (When first created, it is generated only with 2 or more servers, so modify on the resource detail page) NAT IP: Select NAT IP If there is no NAT IP to select, click the Create New button to generate a Public IP Click the Refresh button to view and select the created Public IP Creating a Public IP incurs charges according to the Public IP pricing policy

    Table. GPU Node required information entry items
• Cluster selection area, create or select a Cluster Fabric.

  Category	Required	Detailed description
  Cluster Fabric	Required	Setting of a group of GPU Node servers that can apply GPU Direct RDMA together Optimal GPU performance and speed can be secured only within the same Cluster Fabric When creating a new Cluster Fabric, select *New Input > Node pool, then enter the name of the Cluster Fabric to be created To add to an existing Cluster Fabric, select Existing Input > Node pool, then select the already created Cluster Fabric

  Table. GPU Node Cluster Fabric selection items
  • Additional Information Input Enter or select the required information in the area.

    Category	Required or not	Detailed description
    Lock	Select	Using Lock prevents accidental actions that could terminate/start/stop the server
    Init Script	Select	Script to run when the server starts Init Script must be selected differently depending on the image type For Linux: Select Shell Script or cloud-init
    Tag	Select	Add Tag Up to 50 can be added per resource After clicking the Add Tag button, enter or select Key, Value values

    Table. GPU Node additional information input items

4. Summary Check the detailed information and estimated billing amount generated in the panel, and click the Complete button.
   • Once creation is complete, check the created resources on the GPU Node List page.

GPU Node Check Detailed Information

The Multi-node GPU Cluster service allows you to view and edit the full list of GPU Node resources and detailed information.

GPU Node Details page consists of Details, Tags, Job History tabs.

To view detailed information of the GPU Node, follow the steps below.

1. All Services > Compute > Multi-node GPU Cluster > GPU Node Click the menu. Navigate to the Service Home page of Multi-node GPU Cluster.

2. Click the GPU Node menu on the Service Home page. Navigate to the GPU Node List page.

   • Resource items other than required columns can be added via the Settings button.

     Category	Required or not	Detailed description
     Resource ID	Select	User-created GPU Node ID
     Cluster Fabric name	required	Cluster Fabric name created by the user
     Server name	Required	User-created GPU Node name
     Server Type	Required	Server type of GPU Node User can check the number of cores, memory capacity, GPU type and count of the created resources
     Image	Required	User-generated GPU Node image version
     IP	Required	IP of the GPU Node created by the user
     Status	Required	Status of the GPU Node created by the user
     Creation Time	Select	GPU Node creation time

     Table. GPU Node Resource List Items

3. GPU Node List Click the resource to view detailed information. GPU Node Details You will be taken to the page.

   • GPU Server Details At the top of the page, status information and descriptions of additional features are displayed.

     Category	Detailed description
     GPU Node status	Status of GPU Node created by the user Creating: State where the server is being created Running:: State where creation is complete and usable Editing:: State where IP is being changed Unknown: Error state Starting: State where the server is starting Stopping: State where the server is stopping Stopped: State where the server has stopped Terminating: State where termination is in progress Terminated: State where termination is complete
     Server Control	Button to change server status Start: Start a stopped server Stop: Stop a running server
     Service cancellation	Button to cancel the service

     Table. GPU Node status information and additional features

Detailed Information

GPU Node List page’s Details Tab you can view the detailed information of the selected resource, and if necessary, edit the information.

Tag

GPU Node List page’s Tag Tab you can view the tag information of the selected resource, and add, modify, or delete it.

Work History

GPU Node List page’s Job History tab allows you to view the job history of the selected resource.

GPU Node Operation Control

If you need server control and management functions for the generated GPU Node resources, you can perform tasks on the GPU Node List or GPU Node Details page. You can start and stop the running GPU Node resources.

GPU Node Getting Started

You can start a stopped GPU Node. To start the GPU Node, follow the steps below.

1. All Services > Compute > Multi-node GPU Cluster Click the menu. Navigate to the Service Home page of Multi-node GPU Cluster.
2. Click the GPU Node menu on the Service Home page. You will be taken to the GPU Node List page.
   • On the GPU Node List page, after selecting individual or multiple servers with the checkbox, you can Start via the More button at the top.
3. GPU Node List page, click the resource. GPU Node Details page will be opened.
   • GPU Node Details on the page, click the Start button at the top to start the server.
4. Check the server status and complete the status change.

Stop GPU Node

You can stop a GPU Node that is active. To stop the GPU Node, follow the steps below.

1. All Services > Compute > Multi-node GPU Cluster Click the menu. Move to Multi-node GPU Cluster’s Service Home page.
2. Click the GPU Node menu on the Service Home page. You will be taken to the GPU Node List page.
   • GPU Node List page, you can control individual or multiple servers by selecting the checkboxes and then using the Stop button at the top.
3. GPU Node List page, click the resource. GPU Node Details page, navigate.
   • GPU Node Details on the page, click the Stop button at the top to stop the server.
4. Check the server status and complete the status change.

GPU Node Cancel

You can cancel unused GPU nodes to reduce operating costs. However, if you cancel the service, the running service may be stopped immediately, so you should consider the impact of service interruption sufficiently before proceeding with the cancellation.

To cancel the GPU Node, follow the steps below.

1. All Services > Compute > Multi-node GPU Server Click the menu. Navigate to the Service Home page of the Multi-node GPU Cluster.
2. Click the Cluster Fabric menu on the Service Home page. You will be taken to the Cluster Fabric List page.
3. Cluster Fabric List page, select the resource to cancel, and click the Cancel Service button.
   • Resources using the same Cluster Fabric can be terminated simultaneously.
4. Once the termination is complete, check on the GPU Node List page whether the resources have been terminated.

2.1 - Cluster Fabric Management

Cluster Fabric is a service that helps manage servers (GPU Nodes) included in a GPU Cluster. Using Cluster Fabric, you can move servers between GPU Clusters in the same Node pool and optimize the performance and speed of GPUs within the same GPU Cluster.

Creating Cluster Fabric

Cluster Fabric can be created together with a GPU Node, and it cannot be created or deleted separately. When all GPU Nodes within a Cluster Fabric are terminated, the Cluster Fabric is automatically deleted. If you haven’t created a GPU Node, please create one first. For more information, refer to Creating a GPU Node.

Checking Cluster Fabric Details

You can check the created Cluster Fabric list and details, and move servers on the Cluster Fabric List page and Cluster Fabric Details page.

1. Click on All Services > Compute > Multi-node GPU Server menu. It will move to the Service Home page of the Multi-node GPU Cluster.

2. Click on the Cluster Fabric menu on the Service Home page. It will move to the Cluster Fabric List page.

   • On the Cluster Fabric List page, you can view the list of resources of the GPU Cluster created by the user.
   • Resource items other than required columns can be added through the Settings button.

     Category	Required	Description
     Resource ID	Optional	Cluster Fabric ID created by the user
     Cluster Fabric Name	Required	Cluster Fabric name created by the user
     Node Pool	Optional	A collection of nodes that can be bundled into the same Cluster Fabric
     Number of Servers	Optional	Number of GPU Nodes
     Server Type	Optional	Server type of the GPU Node The user can check the number of cores, memory capacity, and GPU type and number of the created resource
     Status	Optional	Status of the Cluster Fabric created by the user
     Creation Time	Optional	Time when the Cluster Fabric was created

     Table. Cluster Fabric resource list items

3. Click on the resource to check the details on the Cluster Fabric List page. It will move to the Cluster Fabric Details page.

   • At the top of the Cluster Fabric Details page, status information and additional feature descriptions are displayed.

     Category	Description
     Cluster Fabric Status	Status of the Cluster Fabric created by the user Creating: Cluster creation in progress Active: Creation completed and available Editing: IP change in progress Deleting: Termination in progress Deleted: Termination completed
     Add Target Server	Function to move a server from another cluster to this cluster

     Table. Cluster Fabric status information and additional features

Details

On the Details tab of the Cluster Fabric List page, you can check the details of the selected resource and bring in servers from other clusters.

Bringing in Cluster Fabric Servers

Using the Add Target Server feature on the Cluster Fabric Details page, you can bring in servers from other clusters and add them to the selected cluster.

1. Click on All Services > Compute > Multi-node GPU Server menu. It will move to the Service Home page of the Multi-node GPU Cluster.
2. Click on the Cluster Fabric menu on the Service Home page. It will move to the Cluster Fabric List page.
3. Click on the resource to check the details on the Cluster Fabric List page. It will move to the Cluster Fabric Details page.
4. Click the Add button on the right side of the target server on the details tab.
   • The target server addition popup window opens.
     • Cluster Fabric Select a cluster.
     • The GPU Node bound to the selected cluster is retrieved, and you can select the GPU Node to bring in.
     • The selected GPU Node is listed at the bottom with the GPU Node name.
     • Click the Confirm button to complete.
     • Click the Cancel button to cancel the task.
   • Check if the added GPU Node is retrieved in the target server.

Terminating Cluster Fabric

When all GPU Nodes within a Cluster Fabric are terminated, the Cluster Fabric is automatically deleted. For more information, refer to Terminating a GPU Node.

2.2 - ServiceWatch Agent Install

Users can install the ServiceWatch Agent on the GPU node of a Multi-node GPU Cluster to collect custom metrics and logs.

ServiceWatch Agent

The agents that need to be installed on the GPU nodes of a multi-node GPU cluster for collecting ServiceWatch custom metrics and logs can be broadly divided into two types. This is Prometheus Exporter and Open Telemetry Collector.

2.3 - Multi-node GPU Cluster Service Scope and Inspection Guide

Multi-node GPU Cluster service scope

In the event of an IaaS HW level issue with the Multi-node GPU Cluster service, technical support can be received through the Support Center’s Contact Us. However, risks due to changes such as OS Kernel updates or application installation are the responsibility of the user, so technical support may be difficult, please be cautious when performing system updates or other tasks.

IaaS HW level problem

• IPMI(iLO) HW monitoring console where the server’s internal HW fault event occurrence message occurs
• GPU HW operation error confirmed in nvdia-smi command
• HW error messages occurring from InfiniBand HCA card or InfiniBand Switch inspection

IaaS HW Inspection Guide

After applying for the Multi-node GPU Cluster service, it is recommended to check the IaaS HW level according to the inspection guide.

OS Kernel and Package Holding

To proceed with OS Kernel and Package holding, follow the procedure below.

1. Use the following command to check the version of the kernel and IB-related packages.
   Sample Code Download

   Copy Code

   Color mode

   root@bm-dev-001:~# dpkg -l | egrep -i "kernel | mlnx"
   root@bm-dev-001:~# dpkg -l | egrep -i "kernel | nvidia"
   root@bm-dev-001:~# dpkg -l | egrep -i "kernel | linux-image"
   ii  crash                                 7.2.8-1ubuntu1.20.04.1                  amd64        kernel debugging utility, allowing gdb like syntax
   ii  dkms                                  2.8.1-5ubuntu2                          all          Dynamic Kernel Module Support Framework
   ii  dmeventd                              2:1.02.167-1ubuntu1                     amd64        Linux Kernel Device Mapper event daemon
   ii  dmsetup                               2:1.02.167-1ubuntu1                     amd64        Linux Kernel Device Mapper userspace library
   ii  iser-dkms                             5.4-OFED.5.4.3.0.1.1                    all          DKMS support fo iser kernel modules
   ii  isert-dkms                            5.4-OFED.5.4.3.0.1.1                    all          DKMS support fo isert kernel modules
   ii  kernel-mft-dkms                       4.17.2-12                               all          DKMS support for kernel-mft kernel modules
   ii  kmod                                  27-1ubuntu2                             amd64        tools for managing Linux kernel modules
   ii  knem                                  1.1.4.90mlnx1-OFED.5.1.2.5.0.1          amd64        userspace tools for the KNEM kernel module
   ii  knem-dkms                             1.1.4.90mlnx1-OFED.5.1.2.5.0.1          all          DKMS support for mlnx-ofed kernel modules
   ii  libaio1:amd64                         0.3.112-5                               amd64        Linux kernel AIO access library - shared library
   ii  libdevmapper-event1.02.1:amd64        2:1.02.167-1ubuntu1                     amd64        Linux Kernel Device Mapper event support library
   ii  libdevmapper1.02.1:amd64              2:1.02.167-1ubuntu1                     amd64        Linux Kernel Device Mapper userspace library
   ii  libdrm-amdgpu1:amd64                  2.4.107-8ubuntu1~20.04.2                amd64        Userspace interface to amdgpu-specific kernel DRM services -- runtime
   ii  libdrm-common                         2.4.107-8ubuntu1~20.04.2                all          Userspace interface to kernel DRM services -- common files
   ii  libdrm-intel1:amd64                   2.4.107-8ubuntu1~20.04.2                amd64        Userspace interface to intel-specific kernel DRM services -- runtime
   ii  libdrm-nouveau2:amd64                 2.4.107-8ubuntu1~20.04.2                amd64        Userspace interface to nouveau-specific kernel DRM services -- runtime
   ii  libdrm-radeon1:amd64                  2.4.107-8ubuntu1~20.04.2                amd64        Userspace interface to radeon-specific kernel DRM services -- runtime
   ii  libdrm2:amd64                         2.4.107-8ubuntu1~20.04.2                amd64        Userspace interface to kernel DRM services -- runtime
   ii  linux-firmware                        1.187.29                                all          Firmware for Linux kernel drivers
   hi  linux-generic                         5.4.0.105.109                           amd64        Complete Generic Linux kernel and headers
   ii  linux-headers-5.4.0-104               5.4.0-104.118                           all          Header files related to Linux kernel version 5.4.0
   ii  linux-headers-5.4.0-104-generic       5.4.0-104.118                           amd64        Linux kernel headers for version 5.4.0 on 64 bit x86 SMP
   ii  linux-headers-5.4.0-105               5.4.0-105.119                           all          Header files related to Linux kernel version 5.4.0
   ii  linux-headers-5.4.0-105-generic       5.4.0-105.119                           amd64        Linux kernel headers for version 5.4.0 on 64 bit x86 SMP
   hi  linux-headers-generic                 5.4.0.105.109                           amd64        Generic Linux kernel headers
   ii  linux-image-5.4.0-104-generic         5.4.0-104.118                           amd64        Signed kernel image generic
   ii  linux-image-5.4.0-105-generic         5.4.0-105.119                           amd64        Signed kernel image generic
   hi  linux-image-generic                   5.4.0.105.109                           amd64        Generic Linux kernel image
   ii  linux-libc-dev:amd64                  5.4.0-105.119                           amd64        Linux Kernel Headers for development
   ii  linux-modules-5.4.0-104-generic       5.4.0-104.118                           amd64        Linux kernel extra modules for version 5.4.0 on 64 bit x86 SMP
   ii  linux-modules-5.4.0-105-generic       5.4.0-105.119                           amd64        Linux kernel extra modules for version 5.4.0 on 64 bit x86 SMP
   ii  linux-modules-extra-5.4.0-104-generic 5.4.0-104.118                           amd64        Linux kernel extra modules for version 5.4.0 on 64 bit x86 SMP
   ii  linux-modules-extra-5.4.0-105-generic 5.4.0-105.119                           amd64        Linux kernel extra modules for version 5.4.0 on 64 bit x86 SMP
   ii  mlnx-ofed-kernel-dkms                 5.4-OFED.5.4.3.0.3.1                    all          DKMS support for mlnx-ofed kernel modules
   ii  mlnx-ofed-kernel-utils                5.4-OFED.5.4.3.0.3.1                    amd64        Userspace tools to restart and tune mlnx-ofed kernel modules
   ii  mlnx-tools                            5.2.0-0.54303                           amd64        Userspace tools to restart and tune MLNX_OFED kernel modules
   ii  nvidia-kernel-common-470              470.103.01-0ubuntu0.20.04.1             amd64        Shared files used with the kernel module
   ii  nvidia-kernel-source-470              470.103.01-0ubuntu0.20.04.1             amd64        NVIDIA kernel source package
   ii  nvidia-peer-memory                    1.2-0                                   all          nvidia peer memory kernel module.
   ii  nvidia-peer-memory-dkms               1.2-0                                   all          DKMS support for nvidia-peer-memory kernel modules
   ii  rsyslog                               8.2001.0-1ubuntu1.1                     amd64        reliable system and kernel logging daemon
   ii  srp-dkms                              5.4-OFED.5.4.3.0.1.1                    all          DKMS support fo srp kernel modules

   root@bm-dev-001:~# dpkg -l | egrep -i "kernel | mlnx"
   root@bm-dev-001:~# dpkg -l | egrep -i "kernel | nvidia"
   root@bm-dev-001:~# dpkg -l | egrep -i "kernel | linux-image"
   ii  crash                                 7.2.8-1ubuntu1.20.04.1                  amd64        kernel debugging utility, allowing gdb like syntax
   ii  dkms                                  2.8.1-5ubuntu2                          all          Dynamic Kernel Module Support Framework
   ii  dmeventd                              2:1.02.167-1ubuntu1                     amd64        Linux Kernel Device Mapper event daemon
   ii  dmsetup                               2:1.02.167-1ubuntu1                     amd64        Linux Kernel Device Mapper userspace library
   ii  iser-dkms                             5.4-OFED.5.4.3.0.1.1                    all          DKMS support fo iser kernel modules
   ii  isert-dkms                            5.4-OFED.5.4.3.0.1.1                    all          DKMS support fo isert kernel modules
   ii  kernel-mft-dkms                       4.17.2-12                               all          DKMS support for kernel-mft kernel modules
   ii  kmod                                  27-1ubuntu2                             amd64        tools for managing Linux kernel modules
   ii  knem                                  1.1.4.90mlnx1-OFED.5.1.2.5.0.1          amd64        userspace tools for the KNEM kernel module
   ii  knem-dkms                             1.1.4.90mlnx1-OFED.5.1.2.5.0.1          all          DKMS support for mlnx-ofed kernel modules
   ii  libaio1:amd64                         0.3.112-5                               amd64        Linux kernel AIO access library - shared library
   ii  libdevmapper-event1.02.1:amd64        2:1.02.167-1ubuntu1                     amd64        Linux Kernel Device Mapper event support library
   ii  libdevmapper1.02.1:amd64              2:1.02.167-1ubuntu1                     amd64        Linux Kernel Device Mapper userspace library
   ii  libdrm-amdgpu1:amd64                  2.4.107-8ubuntu1~20.04.2                amd64        Userspace interface to amdgpu-specific kernel DRM services -- runtime
   ii  libdrm-common                         2.4.107-8ubuntu1~20.04.2                all          Userspace interface to kernel DRM services -- common files
   ii  libdrm-intel1:amd64                   2.4.107-8ubuntu1~20.04.2                amd64        Userspace interface to intel-specific kernel DRM services -- runtime
   ii  libdrm-nouveau2:amd64                 2.4.107-8ubuntu1~20.04.2                amd64        Userspace interface to nouveau-specific kernel DRM services -- runtime
   ii  libdrm-radeon1:amd64                  2.4.107-8ubuntu1~20.04.2                amd64        Userspace interface to radeon-specific kernel DRM services -- runtime
   ii  libdrm2:amd64                         2.4.107-8ubuntu1~20.04.2                amd64        Userspace interface to kernel DRM services -- runtime
   ii  linux-firmware                        1.187.29                                all          Firmware for Linux kernel drivers
   hi  linux-generic                         5.4.0.105.109                           amd64        Complete Generic Linux kernel and headers
   ii  linux-headers-5.4.0-104               5.4.0-104.118                           all          Header files related to Linux kernel version 5.4.0
   ii  linux-headers-5.4.0-104-generic       5.4.0-104.118                           amd64        Linux kernel headers for version 5.4.0 on 64 bit x86 SMP
   ii  linux-headers-5.4.0-105               5.4.0-105.119                           all          Header files related to Linux kernel version 5.4.0
   ii  linux-headers-5.4.0-105-generic       5.4.0-105.119                           amd64        Linux kernel headers for version 5.4.0 on 64 bit x86 SMP
   hi  linux-headers-generic                 5.4.0.105.109                           amd64        Generic Linux kernel headers
   ii  linux-image-5.4.0-104-generic         5.4.0-104.118                           amd64        Signed kernel image generic
   ii  linux-image-5.4.0-105-generic         5.4.0-105.119                           amd64        Signed kernel image generic
   hi  linux-image-generic                   5.4.0.105.109                           amd64        Generic Linux kernel image
   ii  linux-libc-dev:amd64                  5.4.0-105.119                           amd64        Linux Kernel Headers for development
   ii  linux-modules-5.4.0-104-generic       5.4.0-104.118                           amd64        Linux kernel extra modules for version 5.4.0 on 64 bit x86 SMP
   ii  linux-modules-5.4.0-105-generic       5.4.0-105.119                           amd64        Linux kernel extra modules for version 5.4.0 on 64 bit x86 SMP
   ii  linux-modules-extra-5.4.0-104-generic 5.4.0-104.118                           amd64        Linux kernel extra modules for version 5.4.0 on 64 bit x86 SMP
   ii  linux-modules-extra-5.4.0-105-generic 5.4.0-105.119                           amd64        Linux kernel extra modules for version 5.4.0 on 64 bit x86 SMP
   ii  mlnx-ofed-kernel-dkms                 5.4-OFED.5.4.3.0.3.1                    all          DKMS support for mlnx-ofed kernel modules
   ii  mlnx-ofed-kernel-utils                5.4-OFED.5.4.3.0.3.1                    amd64        Userspace tools to restart and tune mlnx-ofed kernel modules
   ii  mlnx-tools                            5.2.0-0.54303                           amd64        Userspace tools to restart and tune MLNX_OFED kernel modules
   ii  nvidia-kernel-common-470              470.103.01-0ubuntu0.20.04.1             amd64        Shared files used with the kernel module
   ii  nvidia-kernel-source-470              470.103.01-0ubuntu0.20.04.1             amd64        NVIDIA kernel source package
   ii  nvidia-peer-memory                    1.2-0                                   all          nvidia peer memory kernel module.
   ii  nvidia-peer-memory-dkms               1.2-0                                   all          DKMS support for nvidia-peer-memory kernel modules
   ii  rsyslog                               8.2001.0-1ubuntu1.1                     amd64        reliable system and kernel logging daemon
   ii  srp-dkms                              5.4-OFED.5.4.3.0.1.1                    all          DKMS support fo srp kernel modules

2. Use the apt-mark command to hold the package update.
   Sample Code Download

   Copy Code

   Color mode

   # apt-mark hold <package name>

   # apt-mark hold <package name>

Intel E810 Driver Update

Check the version of the Intel E810 driver and update it to the recommended version.

The driver update method is as follows.

1. Move the basic driver tar file to the desired directory.

Example: /home/username/ice or /usr/local/src/ice

2. Untar / unzip the Archiver file.

   • x.x.x is the version number of the driver tar file.
     Sample Code Download

     Copy Code

     Color mode

     tar zxf ice-x.x.x.tar.gz

     tar zxf ice-x.x.x.tar.gz

3. Change to the driver src directory.

   • x.x.x is the version number of the driver tar file.
     Sample Code Download

     Copy Code

     Color mode

     cd ice-x.x.x/src/

     cd ice-x.x.x/src/

4. Compile the driver module.

   Sample Code Download

   Copy Code

   Color mode

   make install

   make install

5. After the update is complete, check the version.

   Sample Code Download

   Copy Code

   Color mode

   lsmod | grep ice
   modinfo ice | grep version

   lsmod | grep ice
   modinfo ice | grep version

NVIDIA driver check

To check the NVIDIA driver (nvidia-smi topo, IB nv_peer_mem status) and inspect the IaaS HW level, follow the next procedure.

1. Check the GPU driver and HW status.

   Sample Code Download

   Copy Code

   Color mode

   user@bm-dev-001:~$ nvidia-smi topo -m
           GPU0    GPU1    GPU2    GPU3    GPU4    GPU5    GPU6    GPU7    mlx5_0  mlx5_1  mlx5_2  mlx5_3  CPU Affinity    NUMA Affinity
   GPU0     X      NV12    NV12    NV12    NV12    NV12    NV12    NV12    SYS     PXB     SYS     SYS     48-63   3
   GPU1    NV12     X      NV12    NV12    NV12    NV12    NV12    NV12    SYS     PXB     SYS     SYS     48-63   3
   GPU2    NV12    NV12     X      NV12    NV12    NV12    NV12    NV12    PXB     SYS     SYS     SYS     16-31   1
   GPU3    NV12    NV12    NV12     X      NV12    NV12    NV12    NV12    PXB     SYS     SYS     SYS     16-31   1
   GPU4    NV12    NV12    NV12    NV12     X      NV12    NV12    NV12    SYS     SYS     SYS     PXB     112-127 7
   GPU5    NV12    NV12    NV12    NV12    NV12     X      NV12    NV12    SYS     SYS     SYS     PXB     112-127 7
   GPU6    NV12    NV12    NV12    NV12    NV12    NV12     X      NV12    SYS     SYS     PXB     SYS     80-95   5
   GPU7    NV12    NV12    NV12    NV12    NV12    NV12    NV12     X      SYS     SYS     PXB     SYS     80-95   5
   mlx5_0  SYS     SYS     PXB     PXB     SYS     SYS     SYS     SYS      X      SYS     SYS     SYS
   mlx5_1  PXB     PXB     SYS     SYS     SYS     SYS     SYS     SYS     SYS      X      SYS     SYS
   mlx5_2  SYS     SYS     SYS     SYS     SYS     SYS     PXB     PXB     SYS     SYS      X      SYS
   mlx5_3  SYS     SYS     SYS     SYS     PXB     PXB     SYS     SYS     SYS     SYS     SYS      X
   
   Legend:
   
     X    = Self
     SYS  = Connection traversing PCIe as well as the SMP interconnect between NUMA nodes (e.g., QPI/UPI)
     NODE = Connection traversing PCIe as well as the interconnect between PCIe Host Bridges within a NUMA node
     PHB  = Connection traversing PCIe as well as a PCIe Host Bridge (typically the CPU)
     PXB  = Connection traversing multiple PCIe bridges (without traversing the PCIe Host Bridge)
     PIX  = Connection traversing at most a single PCIe bridge
     NV#  = Connection traversing a bonded set of # NVLinks

   user@bm-dev-001:~$ nvidia-smi topo -m
           GPU0    GPU1    GPU2    GPU3    GPU4    GPU5    GPU6    GPU7    mlx5_0  mlx5_1  mlx5_2  mlx5_3  CPU Affinity    NUMA Affinity
   GPU0     X      NV12    NV12    NV12    NV12    NV12    NV12    NV12    SYS     PXB     SYS     SYS     48-63   3
   GPU1    NV12     X      NV12    NV12    NV12    NV12    NV12    NV12    SYS     PXB     SYS     SYS     48-63   3
   GPU2    NV12    NV12     X      NV12    NV12    NV12    NV12    NV12    PXB     SYS     SYS     SYS     16-31   1
   GPU3    NV12    NV12    NV12     X      NV12    NV12    NV12    NV12    PXB     SYS     SYS     SYS     16-31   1
   GPU4    NV12    NV12    NV12    NV12     X      NV12    NV12    NV12    SYS     SYS     SYS     PXB     112-127 7
   GPU5    NV12    NV12    NV12    NV12    NV12     X      NV12    NV12    SYS     SYS     SYS     PXB     112-127 7
   GPU6    NV12    NV12    NV12    NV12    NV12    NV12     X      NV12    SYS     SYS     PXB     SYS     80-95   5
   GPU7    NV12    NV12    NV12    NV12    NV12    NV12    NV12     X      SYS     SYS     PXB     SYS     80-95   5
   mlx5_0  SYS     SYS     PXB     PXB     SYS     SYS     SYS     SYS      X      SYS     SYS     SYS
   mlx5_1  PXB     PXB     SYS     SYS     SYS     SYS     SYS     SYS     SYS      X      SYS     SYS
   mlx5_2  SYS     SYS     SYS     SYS     SYS     SYS     PXB     PXB     SYS     SYS      X      SYS
   mlx5_3  SYS     SYS     SYS     SYS     PXB     PXB     SYS     SYS     SYS     SYS     SYS      X
   
   Legend:
   
     X    = Self
     SYS  = Connection traversing PCIe as well as the SMP interconnect between NUMA nodes (e.g., QPI/UPI)
     NODE = Connection traversing PCIe as well as the interconnect between PCIe Host Bridges within a NUMA node
     PHB  = Connection traversing PCIe as well as a PCIe Host Bridge (typically the CPU)
     PXB  = Connection traversing multiple PCIe bridges (without traversing the PCIe Host Bridge)
     PIX  = Connection traversing at most a single PCIe bridge
     NV#  = Connection traversing a bonded set of # NVLinks

2. Check the NVSwitch HW status.

   Sample Code Download

   Copy Code

   Color mode

   user@bm-dev-001:~$ nvidia-smi nvlink --status
   GPU 0: NVIDIA A100-SXM4-80GB (UUID: GPU-2c0d1d6b-e348-55fc-44cf-cd65a954b36c)
            Link 0: 25 GB/s
            Link 1: 25 GB/s
            Link 2: 25 GB/s
            Link 3: 25 GB/s
            Link 4: 25 GB/s
            Link 5: 25 GB/s
            Link 6: 25 GB/s
            Link 7: 25 GB/s
            Link 8: 25 GB/s
            Link 9: 25 GB/s
            Link 10: 25 GB/s
            Link 11: 25 GB/s
   GPU 1: NVIDIA A100-SXM4-80GB (UUID: GPU-96f429d8-893a-a9ea-deca-feffd90669e9)
            Link 0: 25 GB/s
            Link 1: 25 GB/s
            Link 2: 25 GB/s
            Link 3: 25 GB/s
            Link 4: 25 GB/s
            Link 5: 25 GB/s
            Link 6: 25 GB/s
            Link 7: 25 GB/s
            Link 8: 25 GB/s
            Link 9: 25 GB/s
            Link 10: 25 GB/s
            Link 11: 25 GB/s
   GPU 2: NVIDIA A100-SXM4-80GB (UUID: GPU-2e601952-b442-b757-a035-725cd320f589)
            Link 0: 25 GB/s
            Link 1: 25 GB/s
            Link 2: 25 GB/s
            Link 3: 25 GB/s
            Link 4: 25 GB/s
            Link 5: 25 GB/s
            Link 6: 25 GB/s
            Link 7: 25 GB/s
            Link 8: 25 GB/s
            Link 9: 25 GB/s
            Link 10: 25 GB/s
            Link 11: 25 GB/s
   GPU 3: NVIDIA A100-SXM4-80GB (UUID: GPU-bcbfd885-a9f8-ec8c-045b-c521472b4fed)
            Link 0: 25 GB/s
            Link 1: 25 GB/s
            Link 2: 25 GB/s
            Link 3: 25 GB/s
            Link 4: 25 GB/s
            Link 5: 25 GB/s
            Link 6: 25 GB/s
            Link 7: 25 GB/s
            Link 8: 25 GB/s
            Link 9: 25 GB/s
            Link 10: 25 GB/s
            Link 11: 25 GB/s
   GPU 4: NVIDIA A100-SXM4-80GB (UUID: GPU-30273090-2d78-fc7a-a360-ec5f871dd488)
            Link 0: 25 GB/s
            Link 1: 25 GB/s
            Link 2: 25 GB/s
            Link 3: 25 GB/s
            Link 4: 25 GB/s
            Link 5: 25 GB/s
            Link 6: 25 GB/s
            Link 7: 25 GB/s
            Link 8: 25 GB/s
            Link 9: 25 GB/s
            Link 10: 25 GB/s
            Link 11: 25 GB/s
   GPU 5: NVIDIA A100-SXM4-80GB (UUID: GPU-5ce7ef61-56dd-fb18-aa7c-be610c8d51c3)
            Link 0: 25 GB/s
            Link 1: 25 GB/s
            Link 2: 25 GB/s
            Link 3: 25 GB/s
            Link 4: 25 GB/s
            Link 5: 25 GB/s
            Link 6: 25 GB/s
            Link 7: 25 GB/s
            Link 8: 25 GB/s
            Link 9: 25 GB/s
            Link 10: 25 GB/s
            Link 11: 25 GB/s
   GPU 6: NVIDIA A100-SXM4-80GB (UUID: GPU-740a527b-b286-8b85-35eb-b6b41c0bb6d7)
            Link 0: 25 GB/s
            Link 1: 25 GB/s
            Link 2: 25 GB/s
            Link 3: 25 GB/s
            Link 4: 25 GB/s
            Link 5: 25 GB/s
            Link 6: 25 GB/s
            Link 7: 25 GB/s
            Link 8: 25 GB/s
            Link 9: 25 GB/s
            Link 10: 25 GB/s
            Link 11: 25 GB/s
   GPU 7: NVIDIA A100-SXM4-80GB (UUID: GPU-1fb6de95-60f6-dbf2-ffca-f7680577e37c)
            Link 0: 25 GB/s
            Link 1: 25 GB/s
            Link 2: 25 GB/s
            Link 3: 25 GB/s
            Link 4: 25 GB/s
            Link 5: 25 GB/s
            Link 6: 25 GB/s
            Link 7: 25 GB/s
            Link 8: 25 GB/s
            Link 9: 25 GB/s
            Link 10: 25 GB/s
            Link 11: 25 GB/s

   user@bm-dev-001:~$ nvidia-smi nvlink --status
   GPU 0: NVIDIA A100-SXM4-80GB (UUID: GPU-2c0d1d6b-e348-55fc-44cf-cd65a954b36c)
            Link 0: 25 GB/s
            Link 1: 25 GB/s
            Link 2: 25 GB/s
            Link 3: 25 GB/s
            Link 4: 25 GB/s
            Link 5: 25 GB/s
            Link 6: 25 GB/s
            Link 7: 25 GB/s
            Link 8: 25 GB/s
            Link 9: 25 GB/s
            Link 10: 25 GB/s
            Link 11: 25 GB/s
   GPU 1: NVIDIA A100-SXM4-80GB (UUID: GPU-96f429d8-893a-a9ea-deca-feffd90669e9)
            Link 0: 25 GB/s
            Link 1: 25 GB/s
            Link 2: 25 GB/s
            Link 3: 25 GB/s
            Link 4: 25 GB/s
            Link 5: 25 GB/s
            Link 6: 25 GB/s
            Link 7: 25 GB/s
            Link 8: 25 GB/s
            Link 9: 25 GB/s
            Link 10: 25 GB/s
            Link 11: 25 GB/s
   GPU 2: NVIDIA A100-SXM4-80GB (UUID: GPU-2e601952-b442-b757-a035-725cd320f589)
            Link 0: 25 GB/s
            Link 1: 25 GB/s
            Link 2: 25 GB/s
            Link 3: 25 GB/s
            Link 4: 25 GB/s
            Link 5: 25 GB/s
            Link 6: 25 GB/s
            Link 7: 25 GB/s
            Link 8: 25 GB/s
            Link 9: 25 GB/s
            Link 10: 25 GB/s
            Link 11: 25 GB/s
   GPU 3: NVIDIA A100-SXM4-80GB (UUID: GPU-bcbfd885-a9f8-ec8c-045b-c521472b4fed)
            Link 0: 25 GB/s
            Link 1: 25 GB/s
            Link 2: 25 GB/s
            Link 3: 25 GB/s
            Link 4: 25 GB/s
            Link 5: 25 GB/s
            Link 6: 25 GB/s
            Link 7: 25 GB/s
            Link 8: 25 GB/s
            Link 9: 25 GB/s
            Link 10: 25 GB/s
            Link 11: 25 GB/s
   GPU 4: NVIDIA A100-SXM4-80GB (UUID: GPU-30273090-2d78-fc7a-a360-ec5f871dd488)
            Link 0: 25 GB/s
            Link 1: 25 GB/s
            Link 2: 25 GB/s
            Link 3: 25 GB/s
            Link 4: 25 GB/s
            Link 5: 25 GB/s
            Link 6: 25 GB/s
            Link 7: 25 GB/s
            Link 8: 25 GB/s
            Link 9: 25 GB/s
            Link 10: 25 GB/s
            Link 11: 25 GB/s
   GPU 5: NVIDIA A100-SXM4-80GB (UUID: GPU-5ce7ef61-56dd-fb18-aa7c-be610c8d51c3)
            Link 0: 25 GB/s
            Link 1: 25 GB/s
            Link 2: 25 GB/s
            Link 3: 25 GB/s
            Link 4: 25 GB/s
            Link 5: 25 GB/s
            Link 6: 25 GB/s
            Link 7: 25 GB/s
            Link 8: 25 GB/s
            Link 9: 25 GB/s
            Link 10: 25 GB/s
            Link 11: 25 GB/s
   GPU 6: NVIDIA A100-SXM4-80GB (UUID: GPU-740a527b-b286-8b85-35eb-b6b41c0bb6d7)
            Link 0: 25 GB/s
            Link 1: 25 GB/s
            Link 2: 25 GB/s
            Link 3: 25 GB/s
            Link 4: 25 GB/s
            Link 5: 25 GB/s
            Link 6: 25 GB/s
            Link 7: 25 GB/s
            Link 8: 25 GB/s
            Link 9: 25 GB/s
            Link 10: 25 GB/s
            Link 11: 25 GB/s
   GPU 7: NVIDIA A100-SXM4-80GB (UUID: GPU-1fb6de95-60f6-dbf2-ffca-f7680577e37c)
            Link 0: 25 GB/s
            Link 1: 25 GB/s
            Link 2: 25 GB/s
            Link 3: 25 GB/s
            Link 4: 25 GB/s
            Link 5: 25 GB/s
            Link 6: 25 GB/s
            Link 7: 25 GB/s
            Link 8: 25 GB/s
            Link 9: 25 GB/s
            Link 10: 25 GB/s
            Link 11: 25 GB/s

3. Check the InfiniBand(IB) HCA card HW status and Link.

   Sample Code Download

   Copy Code

   Color mode

   user@bm-dev-001:~$ ibdev2netdev -v
   cat: /sys/class/infiniband/mlx5_0/device/vpd: Permission denied
   0000:45:00.0 mlx5_0 (MT4123 -            )                 fw 20.29.1016 port 1 (ACTIVE) ==> ibs18 (Down)
   cat: /sys/class/infiniband/mlx5_1/device/vpd: Permission denied
   0000:0e:00.0 mlx5_1 (MT4123 -            )                 fw 20.29.1016 port 1 (ACTIVE) ==> ibs17 (Down)
   cat: /sys/class/infiniband/mlx5_2/device/vpd: Permission denied
   0000:c5:00.0 mlx5_2 (MT4123 -            )                 fw 20.29.1016 port 1 (ACTIVE) ==> ibs20 (Down)
   cat: /sys/class/infiniband/mlx5_3/device/vpd: Permission denied
   0000:85:00.0 mlx5_3 (MT4123 -            )                 fw 20.29.1016 port 1 (ACTIVE) ==> ibs19 (Down)
   user@bm-dev-001:~$
   
   
   root@bm-dev-001:~# ibstat
   CA 'mlx5_0'
           CA type: MT4123
           Number of ports: 1
           Firmware version: 20.29.1016
           Hardware version: 0
           Node GUID: 0x88e9a4ffff5060ac
           System image GUID: 0x88e9a4ffff5060ac
           Port 1:
                   State: Active
                   Physical state: LinkUp
                   Rate: 200
                   Base lid: 8
                   LMC: 0
                   SM lid: 1
                   Capability mask: 0x2651e848
                   Port GUID: 0x88e9a4ffff5060ac
                   Link layer: InfiniBand
   CA 'mlx5_1'
           CA type: MT4123
           Number of ports: 1
           Firmware version: 20.29.1016
           Hardware version: 0
           Node GUID: 0x88e9a4ffff504080
           System image GUID: 0x88e9a4ffff504080
           Port 1:
                   State: Active
                   Physical state: LinkUp
                   Rate: 200
                   Base lid: 5
                   LMC: 0
                   SM lid: 1
                   Capability mask: 0x2651e848
                   Port GUID: 0x88e9a4ffff504080
                   Link layer: InfiniBand
   CA 'mlx5_2'
           CA type: MT4123
           Number of ports: 1
           Firmware version: 20.29.1016
           Hardware version: 0
           Node GUID: 0x88e9a4ffff505038
           System image GUID: 0x88e9a4ffff505038
           Port 1:
                   State: Active
                   Physical state: LinkUp
                   Rate: 200
                   Base lid: 2
                   LMC: 0
                   SM lid: 1
                   Capability mask: 0x2651e848
                   Port GUID: 0x88e9a4ffff505038
                   Link layer: InfiniBand
   CA 'mlx5_3'
           CA type: MT4123
           Number of ports: 1
           Firmware version: 20.29.1016
           Hardware version: 0
           Node GUID: 0x88e9a4ffff504094
           System image GUID: 0x88e9a4ffff504094
           Port 1:
                   State: Active
                   Physical state: LinkUp
                   Rate: 200
                   Base lid: 7
                   LMC: 0
                   SM lid: 1
                   Capability mask: 0x2651e848
                   Port GUID: 0x88e9a4ffff504094
                   Link layer: InfiniBand

   user@bm-dev-001:~$ ibdev2netdev -v
   cat: /sys/class/infiniband/mlx5_0/device/vpd: Permission denied
   0000:45:00.0 mlx5_0 (MT4123 -            )                 fw 20.29.1016 port 1 (ACTIVE) ==> ibs18 (Down)
   cat: /sys/class/infiniband/mlx5_1/device/vpd: Permission denied
   0000:0e:00.0 mlx5_1 (MT4123 -            )                 fw 20.29.1016 port 1 (ACTIVE) ==> ibs17 (Down)
   cat: /sys/class/infiniband/mlx5_2/device/vpd: Permission denied
   0000:c5:00.0 mlx5_2 (MT4123 -            )                 fw 20.29.1016 port 1 (ACTIVE) ==> ibs20 (Down)
   cat: /sys/class/infiniband/mlx5_3/device/vpd: Permission denied
   0000:85:00.0 mlx5_3 (MT4123 -            )                 fw 20.29.1016 port 1 (ACTIVE) ==> ibs19 (Down)
   user@bm-dev-001:~$
   
   
   root@bm-dev-001:~# ibstat
   CA 'mlx5_0'
           CA type: MT4123
           Number of ports: 1
           Firmware version: 20.29.1016
           Hardware version: 0
           Node GUID: 0x88e9a4ffff5060ac
           System image GUID: 0x88e9a4ffff5060ac
           Port 1:
                   State: Active
                   Physical state: LinkUp
                   Rate: 200
                   Base lid: 8
                   LMC: 0
                   SM lid: 1
                   Capability mask: 0x2651e848
                   Port GUID: 0x88e9a4ffff5060ac
                   Link layer: InfiniBand
   CA 'mlx5_1'
           CA type: MT4123
           Number of ports: 1
           Firmware version: 20.29.1016
           Hardware version: 0
           Node GUID: 0x88e9a4ffff504080
           System image GUID: 0x88e9a4ffff504080
           Port 1:
                   State: Active
                   Physical state: LinkUp
                   Rate: 200
                   Base lid: 5
                   LMC: 0
                   SM lid: 1
                   Capability mask: 0x2651e848
                   Port GUID: 0x88e9a4ffff504080
                   Link layer: InfiniBand
   CA 'mlx5_2'
           CA type: MT4123
           Number of ports: 1
           Firmware version: 20.29.1016
           Hardware version: 0
           Node GUID: 0x88e9a4ffff505038
           System image GUID: 0x88e9a4ffff505038
           Port 1:
                   State: Active
                   Physical state: LinkUp
                   Rate: 200
                   Base lid: 2
                   LMC: 0
                   SM lid: 1
                   Capability mask: 0x2651e848
                   Port GUID: 0x88e9a4ffff505038
                   Link layer: InfiniBand
   CA 'mlx5_3'
           CA type: MT4123
           Number of ports: 1
           Firmware version: 20.29.1016
           Hardware version: 0
           Node GUID: 0x88e9a4ffff504094
           System image GUID: 0x88e9a4ffff504094
           Port 1:
                   State: Active
                   Physical state: LinkUp
                   Rate: 200
                   Base lid: 7
                   LMC: 0
                   SM lid: 1
                   Capability mask: 0x2651e848
                   Port GUID: 0x88e9a4ffff504094
                   Link layer: InfiniBand

IB bandwidth communication check

To check the IB bandwidth communication status (ib_send_bw) and inspect the IaaS HW level, follow these steps.

1. Check the name of the IB HCA interface.

   Sample Code Download

   Copy Code

   Color mode

   user@bm-dev-001:~$ ibdev2netdev
   mlx5_0 port 1 ==> ibs18 (Down)
   mlx5_1 port 1 ==> ibs17 (Down)
   mlx5_2 port 1 ==> ibs20 (Down)
   mlx5_3 port 1 ==> ibs19 (Down)

   user@bm-dev-001:~$ ibdev2netdev
   mlx5_0 port 1 ==> ibs18 (Down)
   mlx5_1 port 1 ==> ibs17 (Down)
   mlx5_2 port 1 ==> ibs20 (Down)
   mlx5_3 port 1 ==> ibs19 (Down)

2. Check the HCA interface that can communicate with IB Switch#1.

   Sample Code Download

   Copy Code

   Color mode

   mlx5_0 port 1 ==> ibs18 (Down)
   mlx5_2 port 1 ==> ibs20 (Down)

   mlx5_0 port 1 ==> ibs18 (Down)
   mlx5_2 port 1 ==> ibs20 (Down)

3. Check the HCA interface that can communicate with IB Switch#2.

   Sample Code Download

   Copy Code

   Color mode

   mlx5_1 port 1 ==> ibs17 (Down)
   mlx5_3 port 1 ==> ibs19 (Down)

   mlx5_1 port 1 ==> ibs17 (Down)
   mlx5_3 port 1 ==> ibs19 (Down)

4. Use SERVER Side commands to check the communication status.

   • Client Side command is entered secondarily for mutual communication
     Sample Code Download

     Copy Code

     Color mode

     user@bm-dev-001:~$ ib_send_bw -d mlx5_3 -i 1 –F
     ************************************
     * Waiting for client to connect... *
     ************************************
     ---------------------------------------------------------------------------------------
                         Send BW Test
      Dual-port       : OFF          Device         : mlx5_3
      Number of qps   : 1            Transport type : IB
      Connection type : RC           Using SRQ      : OFF
      PCIe relax order: ON
      ibv_wr* API     : ON
      RX depth        : 512
      CQ Moderation   : 1
      Mtu             : 4096[B]
      Link type       : IB
      Max inline data : 0[B]
      rdma_cm QPs     : OFF
      Data ex. method : Ethernet
     ---------------------------------------------------------------------------------------
      local address: LID 0x07 QPN 0x002e PSN 0xa86622
      remote address: LID 0x0a QPN 0x002d PSN 0xfc58dd
     ---------------------------------------------------------------------------------------
      #bytes     #iterations    BW peak[MB/sec]    BW average[MB/sec]   MsgRate[Mpps]
      65536      1000             0.00               19827.40                   0.317238
     ---------------------------------------------------------------------------------------

     user@bm-dev-001:~$ ib_send_bw -d mlx5_3 -i 1 –F
     ************************************
     * Waiting for client to connect... *
     ************************************
     ---------------------------------------------------------------------------------------
                         Send BW Test
      Dual-port       : OFF          Device         : mlx5_3
      Number of qps   : 1            Transport type : IB
      Connection type : RC           Using SRQ      : OFF
      PCIe relax order: ON
      ibv_wr* API     : ON
      RX depth        : 512
      CQ Moderation   : 1
      Mtu             : 4096[B]
      Link type       : IB
      Max inline data : 0[B]
      rdma_cm QPs     : OFF
      Data ex. method : Ethernet
     ---------------------------------------------------------------------------------------
      local address: LID 0x07 QPN 0x002e PSN 0xa86622
      remote address: LID 0x0a QPN 0x002d PSN 0xfc58dd
     ---------------------------------------------------------------------------------------
      #bytes     #iterations    BW peak[MB/sec]    BW average[MB/sec]   MsgRate[Mpps]
      65536      1000             0.00               19827.40                   0.317238
     ---------------------------------------------------------------------------------------

5. Use the CLIENT Side command to check the communication status.

   • SERVER Side command is entered first for mutual communication
     Sample Code Download

     Copy Code

     Color mode

     root@bm-dev-003:~# ib_send_bw -d mlx5_3 -i 1 -F <SERVER Side IP>
     ---------------------------------------------------------------------------------------
                         Send BW Test
      Dual-port       : OFF          Device         : mlx5_3
      Number of qps   : 1            Transport type : IB
      Connection type : RC           Using SRQ      : OFF
      PCIe relax order: ON
      ibv_wr* API     : ON
      TX depth        : 128
      CQ Moderation   : 1
      Mtu             : 4096[B]
      Link type       : IB
      Max inline data : 0[B]
      rdma_cm QPs     : OFF
      Data ex. method : Ethernet
     ---------------------------------------------------------------------------------------
      local address: LID 0x0a QPN 0x002a PSN 0x98a48e
      remote address: LID 0x07 QPN 0x002c PSN 0xe68304
     ---------------------------------------------------------------------------------------
      #bytes     #iterations    BW peak[MB/sec]    BW average[MB/sec]   MsgRate[Mpps]
      65536      1000             19008.49            19006.37                  0.304102
     ---------------------------------------------------------------------------------------

     root@bm-dev-003:~# ib_send_bw -d mlx5_3 -i 1 -F <SERVER Side IP>
     ---------------------------------------------------------------------------------------
                         Send BW Test
      Dual-port       : OFF          Device         : mlx5_3
      Number of qps   : 1            Transport type : IB
      Connection type : RC           Using SRQ      : OFF
      PCIe relax order: ON
      ibv_wr* API     : ON
      TX depth        : 128
      CQ Moderation   : 1
      Mtu             : 4096[B]
      Link type       : IB
      Max inline data : 0[B]
      rdma_cm QPs     : OFF
      Data ex. method : Ethernet
     ---------------------------------------------------------------------------------------
      local address: LID 0x0a QPN 0x002a PSN 0x98a48e
      remote address: LID 0x07 QPN 0x002c PSN 0xe68304
     ---------------------------------------------------------------------------------------
      #bytes     #iterations    BW peak[MB/sec]    BW average[MB/sec]   MsgRate[Mpps]
      65536      1000             19008.49            19006.37                  0.304102
     ---------------------------------------------------------------------------------------

Check IB Service Related Kernel Modules

Check the IB service-related kernel modules (lsmod) to inspect the IaaS HW level.

user@bm-dev-001:~$ lsmod | grep nv_peer_mem
nv_peer_mem            16384  0
ib_core               315392  9 rdma_cm,ib_ipoib,nv_peer_mem,iw_cm,ib_umad,rdma_ucm,ib_uverbs,mlx5_ib,ib_cm
nvidia              35315712  156 nvidia_uvm,nv_peer_mem,nvidia_modeset

user@bm-dev-001:~$ lsmod | grep nv_peer_mem
nv_peer_mem            16384  0
ib_core               315392  9 rdma_cm,ib_ipoib,nv_peer_mem,iw_cm,ib_umad,rdma_ucm,ib_uverbs,mlx5_ib,ib_cm
nvidia              35315712  156 nvidia_uvm,nv_peer_mem,nvidia_modeset

user@bm-dev-001:~$ service nv_peer_mem status
  nv_peer_mem.service - LSB: Activates/Deactivates nv_peer_mem to \ start at boot time.
     Loaded: loaded (/etc/init.d/nv_peer_mem; generated)
     Active: active (exited) since Mon 2023-03-13 16:21:33 KST; 2 days ago
       Docs: man:systemd-sysv-generator(8)
    Process: 4913 ExecStart=/etc/init.d/nv_peer_mem start (code=exited, status=0/SUCCESS)

user@bm-dev-001:~$ service nv_peer_mem status
  nv_peer_mem.service - LSB: Activates/Deactivates nv_peer_mem to \ start at boot time.
     Loaded: loaded (/etc/init.d/nv_peer_mem; generated)
     Active: active (exited) since Mon 2023-03-13 16:21:33 KST; 2 days ago
       Docs: man:systemd-sysv-generator(8)
    Process: 4913 ExecStart=/etc/init.d/nv_peer_mem start (code=exited, status=0/SUCCESS)

user@bm-dev-001:~$ lsmod | grep ib
libiscsi_tcp           32768  1 iscsi_tcp
libiscsi               57344  2 libiscsi_tcp,iscsi_tcp
scsi_transport_iscsi   110592  4 libiscsi_tcp,iscsi_tcp,libiscsi
ib_ipoib              131072  0
ib_cm                  57344  2 rdma_cm,ib_ipoib
ib_umad                24576  8
mlx5_ib               380928  0
ib_uverbs             135168  18 rdma_ucm,mlx5_ib
ib_core               315392  9 rdma_cm,ib_ipoib,nv_peer_mem,iw_cm,ib_umad,rdma_ucm,ib_uverbs,mlx5_ib,ib_cm
libcrc32c              16384  2 btrfs,raid456
mlx5_core            1458176  1 mlx5_ib
auxiliary              16384  2 mlx5_ib,mlx5_core
mlx_compat             65536  12 rdma_cm,ib_ipoib,mlxdevm,iw_cm,auxiliary,ib_umad,ib_core,rdma_ucm,ib_uverbs,mlx5_ib,ib_cm,mlx5_core

user@bm-dev-001:~$ lsmod | grep ib
libiscsi_tcp           32768  1 iscsi_tcp
libiscsi               57344  2 libiscsi_tcp,iscsi_tcp
scsi_transport_iscsi   110592  4 libiscsi_tcp,iscsi_tcp,libiscsi
ib_ipoib              131072  0
ib_cm                  57344  2 rdma_cm,ib_ipoib
ib_umad                24576  8
mlx5_ib               380928  0
ib_uverbs             135168  18 rdma_ucm,mlx5_ib
ib_core               315392  9 rdma_cm,ib_ipoib,nv_peer_mem,iw_cm,ib_umad,rdma_ucm,ib_uverbs,mlx5_ib,ib_cm
libcrc32c              16384  2 btrfs,raid456
mlx5_core            1458176  1 mlx5_ib
auxiliary              16384  2 mlx5_ib,mlx5_core
mlx_compat             65536  12 rdma_cm,ib_ipoib,mlxdevm,iw_cm,auxiliary,ib_umad,ib_core,rdma_ucm,ib_uverbs,mlx5_ib,ib_cm,mlx5_core

Storage Physical Disk Resources and Multi-Path Check

Check the IaaS HW level by checking the storage physical disk resources and Multi-Path.

root@bm-dev-002:/tmp# fdisk –l

root@bm-dev-002:/tmp# fdisk –l

root@bm-dev-002:/tmp# multipath –ll

root@bm-dev-002:/tmp# multipath –ll

Multi-node GPU Cluster new deployment after checking Service Network

Use the following command to check if the MII Status of Bonding and Slave Interface is up.

• command

  Sample Code Download

  Copy Code

  Color mode

  root@mngc-001:~# cat /proc/net/bonding/bond-srv
  Ethernet Channel Bonding Driver: v5.15.0-25-generic

  root@mngc-001:~# cat /proc/net/bonding/bond-srv
  Ethernet Channel Bonding Driver: v5.15.0-25-generic

• confirmation result

  Sample Code Download

  Copy Code

  Color mode

  Bonding Mode: fault-tolerance (active-backup)
  Primary Slave: None
  Currently Active Slave: ens9f0
  MII Status: up
  MII Polling Interval (ms): 100
  Up Delay (ms): 0
  Down Delay (ms): 0
  Peer Notification Delay (ms): 0
  
  Slave Interface: ens9f0
  MII Status: up
  Speed: 100000 Mbps
  Duplex: full
  Link Failure Count: 0
  Permanent HW addr: 30:3e:a7:02:35:70
  Slave queue ID: 0
  
  Slave Interface: ens11f0
  MII Status: up
  Speed: 100000 Mbps
  Duplex: full
  Link Failure Count: 0
  Permanent HW addr: 30:3e:a7:02:2f:e8
  Slave queue ID: 0

  Bonding Mode: fault-tolerance (active-backup)
  Primary Slave: None
  Currently Active Slave: ens9f0
  MII Status: up
  MII Polling Interval (ms): 100
  Up Delay (ms): 0
  Down Delay (ms): 0
  Peer Notification Delay (ms): 0
  
  Slave Interface: ens9f0
  MII Status: up
  Speed: 100000 Mbps
  Duplex: full
  Link Failure Count: 0
  Permanent HW addr: 30:3e:a7:02:35:70
  Slave queue ID: 0
  
  Slave Interface: ens11f0
  MII Status: up
  Speed: 100000 Mbps
  Duplex: full
  Link Failure Count: 0
  Permanent HW addr: 30:3e:a7:02:2f:e8
  Slave queue ID: 0

Multi-node GPU Cluster new deployment after checking Time Server and time synchronization

The OS image has the chrony daemon installed and set to synchronize with the SCP NTP server. Use the following command to check if there are any lines marked with ^* in the MS Name column.

• command

  Sample Code Download

  Copy Code

  Color mode

  root@mngc-001:~# chronyc sources -V

  root@mngc-001:~# chronyc sources -V

• confirmation result

  Sample Code Download

  Copy Code

  Color mode

  MS Name/IP address         Stratum Poll Reach LastRx Last sample
  ===============================================================================
  ^+ 198.19.1.53                   4  10   377  1040    -16us[  -37us] +/- 9982us
  ^* 198.19.1.54                   4  10   377   312   -367us[ -388us] +/-   13ms

  MS Name/IP address         Stratum Poll Reach LastRx Last sample
  ===============================================================================
  ^+ 198.19.1.53                   4  10   377  1040    -16us[  -37us] +/- 9982us
  ^* 198.19.1.54                   4  10   377   312   -367us[ -388us] +/-   13ms

GPU MIG/ECC Setting Initialization Check Guide

When applying for a multi-node GPU cluster product, the GPU MIG/ECC setting is initialized. However, to apply the exact setting value, please restart it once at the beginning, and then check and use it according to the inspection guide to see if the setting value is applied.

MIG Setup Initialization

Refer to the following for how to check and initialize MIG settings.

Use the following command to check if the status value of MIG M is Disabled.

• command

  Sample Code Download

  Copy Code

  Color mode

  root@bm-dev-001:~#nvidia-smi

  root@bm-dev-001:~#nvidia-smi

• confirmation result

  Sample Code Download

  Copy Code

  Color mode

  +-----------------------------------------------------------------------------------------+
  |  NVIDIA-SMI 470.129.06        Driver version: 470.129.06        CUDA Version: 11.4      |
  |----------------------------------+-----------------------------+------------------------|
  |  GPU  Name        Persistence-M  |  Bus-Id             Disp.A  |  Volatile Uncorr. ECC  |
  |  Fan  Temp  Perf  Pwr:Usage/Cap  |               Memory-Usage  |  GPU-Util  Compute M.  |
  |                                  |                             |                MIG M.  |
  |==================================+=============================+========================|
  |    0  NVIDIA A100-SXM...    Off  |  00000000:03:00.0      Off  |                   Off  |
  |  N/A  29C     P0    57W  /  400W |          0MiB  /  81251MiB  |    0%         Default  |
  |                                  |                             |              Disabled  |
  +----------------------------------+-----------------------------+------------------------+
  |    0  NVIDIA A100-SXM...    Off  |  00000000:0C:00.0      Off  |                   Off  |
  |  N/A  30C     P0    58W  /  400W |          0MiB  /  81251MiB  |    18%        Default  |
  |                                  |                             |              Disabled  |
  +-----------------------------------------------------------------------------------------+

  +-----------------------------------------------------------------------------------------+
  |  NVIDIA-SMI 470.129.06        Driver version: 470.129.06        CUDA Version: 11.4      |
  |----------------------------------+-----------------------------+------------------------|
  |  GPU  Name        Persistence-M  |  Bus-Id             Disp.A  |  Volatile Uncorr. ECC  |
  |  Fan  Temp  Perf  Pwr:Usage/Cap  |               Memory-Usage  |  GPU-Util  Compute M.  |
  |                                  |                             |                MIG M.  |
  |==================================+=============================+========================|
  |    0  NVIDIA A100-SXM...    Off  |  00000000:03:00.0      Off  |                   Off  |
  |  N/A  29C     P0    57W  /  400W |          0MiB  /  81251MiB  |    0%         Default  |
  |                                  |                             |              Disabled  |
  +----------------------------------+-----------------------------+------------------------+
  |    0  NVIDIA A100-SXM...    Off  |  00000000:0C:00.0      Off  |                   Off  |
  |  N/A  30C     P0    58W  /  400W |          0MiB  /  81251MiB  |    18%        Default  |
  |                                  |                             |              Disabled  |
  +-----------------------------------------------------------------------------------------+

• If MIG M.’s status value is not Disabled, use the following command to initialize MIG.

  Sample Code Download

  Copy Code

  Color mode

  root@bm-dev-001:~# nvidia-smi -mig 0
  root@bm-dev-001:~# nvidia-smi --gpu-reset

  root@bm-dev-001:~# nvidia-smi -mig 0
  root@bm-dev-001:~# nvidia-smi --gpu-reset

ECC Setting Initialization

Refer to the following for how to check and initialize the ECC settings.

Use the following command to check if the status value of Volatile Uncorr. ECC is Off.

• command

  Sample Code Download

  Copy Code

  Color mode

  root@bm-dev-001:~#nvidia-smi

  root@bm-dev-001:~#nvidia-smi

• confirmation result

  Sample Code Download

  Copy Code

  Color mode

  +-----------------------------------------------------------------------------------------+
  |  NVIDIA-SMI 470.129.06        Driver version: 470.129.06        CUDA Version: 11.4      |
  |----------------------------------+-----------------------------+------------------------|
  |  GPU  Name        Persistence-M  |  Bus-Id             Disp.A  |  Volatile Uncorr. ECC  |
  |  Fan  Temp  Perf  Pwr:Usage/Cap  |               Memory-Usage  |  GPU-Util  Compute M.  |
  |                                  |                             |                MIG M.  |
  |==================================+=============================+========================|
  |    0  NVIDIA A100-SXM...    Off  |  00000000:03:00.0      Off  |                   Off  |
  |  N/A  29C     P0    57W  /  400W |          0MiB  /  81251MiB  |    0%         Default  |
  |                                  |                             |              Disabled  |
  +----------------------------------+-----------------------------+------------------------+
  |    0  NVIDIA A100-SXM...    Off  |  00000000:0C:00.0      Off  |                   Off  |
  |  N/A  30C     P0    61W  /  400W |          0MiB  /  81251MiB  |    18%        Default  |
  |                                  |                             |              Disabled  |
  +-----------------------------------------------------------------------------------------+

  +-----------------------------------------------------------------------------------------+
  |  NVIDIA-SMI 470.129.06        Driver version: 470.129.06        CUDA Version: 11.4      |
  |----------------------------------+-----------------------------+------------------------|
  |  GPU  Name        Persistence-M  |  Bus-Id             Disp.A  |  Volatile Uncorr. ECC  |
  |  Fan  Temp  Perf  Pwr:Usage/Cap  |               Memory-Usage  |  GPU-Util  Compute M.  |
  |                                  |                             |                MIG M.  |
  |==================================+=============================+========================|
  |    0  NVIDIA A100-SXM...    Off  |  00000000:03:00.0      Off  |                   Off  |
  |  N/A  29C     P0    57W  /  400W |          0MiB  /  81251MiB  |    0%         Default  |
  |                                  |                             |              Disabled  |
  +----------------------------------+-----------------------------+------------------------+
  |    0  NVIDIA A100-SXM...    Off  |  00000000:0C:00.0      Off  |                   Off  |
  |  N/A  30C     P0    61W  /  400W |          0MiB  /  81251MiB  |    18%        Default  |
  |                                  |                             |              Disabled  |
  +-----------------------------------------------------------------------------------------+

• Volatile Uncorr. ECC’s status value is On*, please proceed with rebooting.

• Volatile Uncorr. ECC status value is not On* or Off, use the following command to initialize ECC. After initialization, reboot and check if the status value is Off.

root@bm-dev-001:~# nvidia-smi --ecc-config=0

root@bm-dev-001:~# nvidia-smi --ecc-config=0

3 - Release Note

Multi-node GPU Cluster