The page has been translated by Gen AI.

Using Job Failover

Job Failover is a feature that automatically reallocates to normal resources to continue training, ensuring that training is not interrupted even if hardware failures such as GPUs occur while training with On-Demand Training. By using the Job Failover feature, you can ensure training continuity without the user having to manually detect failures or recreate the job when system errors or hardware faults are detected.

Job Failover Overview

During a Training Job, various hardware issues such as GPU errors or node failures can occur on the node where the training is deployed. When you use the Job Failover feature, the system automatically avoids the failed node when such hardware failures are detected, reassigns the Training Job to healthy resources, and resumes training. The main operation of Job Failover is as follows.

Fault Detection: Continuously monitors the GPU and hardware status of nodes running training to detect anomalies.
Automatic Reallocation: If a hardware failure is detected, the system excludes the failed node and reallocates the Training Job to healthy nodes, restarting the training.
Unnecessary relocation prevention: User code errors, configuration errors, and similar issues that are not hardware failures are excluded from the Failover target.
Retry Count Limit: Failover is performed only within the specified maximum number of attempts (3), and if this is exceeded, training ends in failure.

Job Failover Usage Conditions

Job Failover can be used only when the training type is selected as On-Demand Training when creating a Training Job. * When Spot Training is used, Job Failover cannot be used.
You can select whether to use the Job Failover feature in the Service Information input area’s Job Failover item when creating a Training Job. * After creation, you can check its usage on the Training Job Details page’s Details tab.

Information

If training is reallocated to another resource due to failover, the memory state at the point where training was interrupted is not retained. * To continue training, we recommend configuring the training script to save checkpoints to shared storage (File Storage, Object Storage).
Failover does not always guarantee immediate execution.
- Jobs that are reassigned by failover are set with a high priority, so they receive resources and run before other jobs that are waiting.
- However, depending on the priority and queue order of Jobs already in the queue at the time of reallocation, a reallocated Job may be executed later than those Jobs.
While checking the node status to decide on Job relocation (up to 5 minutes), the Job may remain in Pending - failoverinprogress state.

Check error cause

When a training interruption occurs, the system checks the node’s GPU and hardware status to determine whether the cause is a hardware failure or a non‑hardware issue such as a user application or configuration. Whether to perform failover is determined based on this judgment result.

If it is judged to be a hardware failure

If a physical or hardware-level error that prevents normal GPU usage is detected, it is considered a hardware failure and a failover is performed. This error prevents further training on the affected node, so the failed node is excluded and the workload is reallocated to healthy resources to resume training. Examples of errors that are considered hardware failures are as follows.

XID code	Error	Explanation
48	GPU memory (HBM) uncorrectable error (Uncorrectable Double Bit ECC)	If an uncorrectable error occurs in the GPU’s large memory The GPU’s computation results cannot be trusted, so it cannot be used further
79	GPU has been detached from the system (GPU fallen off the bus)	The system can no longer recognize the GPU itself The GPU is unresponsive, so training cannot proceed
94, 95	GPU internal memory (SRAM) unrecoverable error	An uncorrectable error occurred in the high-speed memory inside the GPU chip Hardware damage occurred
-	GPU overheating, power supply unit (PSU), PCIe, and other hardware component failures	If the temperature exceeds the allowed limit or there is a problem with the power or connected components GPU cannot be used normally

Table: Examples of errors considered hardware failures

Reference

XID code: a number assigned by the NVIDIA GPU driver to differentiate error types, which can be referenced to identify the cause of errors in GPU logs.

If it is judged to be an internal error

If the cause of the training interruption is determined to be user application code or configuration issues rather than a hardware failure, there is a high likelihood that the same problem will recur even after redeployment, so we reject the Failover and terminate in an internalerror state. Examples of errors that are considered internal errors are as follows.

XID code	Error	Explanation
-	Forced termination due to insufficient memory (Out Of Memory)	When the training program attempts to use more memory than the allocated GPU or system memory, causing it to be forcibly terminated Resource shortage and configuration issues, not a GPU memory hardware error (XID 48)Since the same issue occurs even after reallocation, it is necessary to adjust training settings such as reducing the batch size or increasing the memory allocation
31	GPU memory page fault	Error that occurs when a learning program attempts to access a prohibited memory region Mostly caused by bugs in the code
43	GPU processing halted	If an error occurs in the running program and the GPU task is interrupted It is likely an application issue
13	Graphics and Compute Engine Exception	When it occurs because the command sent to the GPU is incorrect Usually caused by issues in the training code or library
-	GPU configuration warnings, software errors, etc.	The hardware itself is fine, but the issue is at the configuration/software level GPU replacement or node relocation is unnecessary

Table. Example of errors considered internal errors

Reference

XID code: a number assigned by the NVIDIA GPU driver to differentiate error types, which can be referenced to identify the cause of errors in GPU logs.
In this case, check the training script, execution Command, input data, resource settings, etc. * For detailed information on how to check logs, see Job Failover 로그 확인하기.

Check Job Failover status

Click the All Services > AI/ML > Simple AI Training menu. 1. Go to the Service Home page of Simple AI Training.
On the Service Home page, click the Training Job menu. 2. Navigate to the Training Job List page.

On the Training Job List page, check the Job’s status.

The state flow based on whether failover is used and the hardware fault assessment result is as follows.

Category	Detailed description
Failover success	When a hardware failure is detected and the resources have been successfully reallocated to normal resources Running → Pending - failoverinprogress → Running If the status changes in this order, the failover is performed correctly and training resumes
Failover Rejection	The training was halted, but since no hardware failure signal was detected, it was determined not to be a candidate for reallocation Running → Pending - failoverinprogress → Pending - internalerror If the status changes in this order, the failover is rejected Since it may be caused by non-hardware reasons such as user code errors, checking the logs is necessary

Failover not configured	When Job Failover is disabled If a training interruption occurs, do not attempt reallocation and terminate the training in the Running → Failed order

Table. State flow according to Job Failover

The main status values displayed during failover are as follows.

status	Detailed description
Pending - failoverinprogress	Detecting hardware failures and assessing failover feasibility, or currently reallocating to normal resources.
Pending - maxretriesexceeded	State in which failover exceeds the maximum number of attempts and no further reallocation is performed.
Pending - imagepullbackoff	Unable to load the container image, preventing training from starting (image URL and authentication information need to be verified)
Pending - internalerror	The failover was denied because it was determined that the system is not a failover candidate, such as when no hardware fault signal is detected.

Table. Main status values during Job Failover

Check Job Failover logs

When a failover occurs, you can view the logs of the previous training that was interrupted by the failure and the logs of the training that resumed after reallocation together in a single log stream.

Reference

The logs of the Training Job are provided in conjunction with the ServiceWatch service. How to check the Training Job logs, please refer to Training Job 로그 확인하기.

To check the logs of Job Failover, follow these steps.

Click the All Services > AI/ML > Simple AI Training menu. 1. Go to the Service Home page of Simple AI Training.
On the Service Home page, click the Training Job menu. 2. Navigate to the Training Job List page.
On the Training Job List page, select the resource whose logs you want to view. 3. Navigate to the Training Job Details page.
On the Training Job Details page, click the Log tab. 4. The log information for this job is displayed.
Click the name below the Training Job name in the log information. 5. Go to the ServiceWatch Log Group Details page.
On the Log Group Details page, click the Log Stream tab. 6. The list of log streams is displayed.
Click the log stream name to check (e.g., master). 7. The logs for this stream are displayed in chronological order.
- When a failover occurs, the logs of the training that was interrupted due to the failure and the logs of the training that was restarted after reallocation are recorded together in a single stream.
- If the initialization log displayed at the start of training (e.g., Starting dataset initialization) appears more than once, you can confirm that the training was rescheduled and restarted.

How-to guides

Concurrent Checkpointing