The page has been translated by Gen AI.

Using Job Failover

Job Failover is a feature that automatically reallocates to normal resources to continue training, ensuring that training is not interrupted even if hardware failures such as GPU occur during On-Demand Training. By using the Job Failover feature, you can ensure training continuity without the user having to manually detect failures or recreate the Job when system errors or hardware failures are detected.

Job Failover Overview

During the execution of a Training Job, various hardware issues such as GPU errors or node failures can occur on the node where training is deployed. When you use the Job Failover feature, if such hardware failures are detected, the system avoids the failed node, automatically reallocates the Training Job to healthy resources, and resumes training. The main operation of Job Failover is as follows.

Fault detection: Continuously monitors the GPU and hardware status of nodes running training to detect anomalies.
Automatic Reallocation: If a hardware failure is detected, the Training Job is reallocated to healthy nodes, excluding the failed node, and training restarts.
Prevent unnecessary failover: User code errors, configuration errors, and similar issues that are not hardware failures are excluded from failover targets.
Retry Count Limit: Failover is performed only within the specified maximum number of attempts (3), and if exceeded, training terminates as a failure.

Job Failover usage conditions

Job Failover can only be used when the training type is selected as On-Demand Training when creating a Training Job. * When using Spot Training, Job Failover cannot be used.
Whether to use the Job Failover feature can be selected in the Job Failover item of the service information input area when creating a Training Job. * After creation, you can verify usage on the Training Job Details page’s Details tab.

Notice

If training is reallocated to another resource due to failover, the memory state at the point where training was interrupted is not retained. * It is recommended to configure the training script to save checkpoints to shared storage (File Storage, Object Storage) in order to continue training.
Failover does not always guarantee immediate execution.
- Jobs that are reassigned due to failover are set with a high priority, so they receive resources and run before other jobs that are waiting.
- However, depending on the priority and waiting order of Jobs already in the queue at the time of reallocation, the reallocated Job may be executed later than those Jobs.
While checking the node status to determine whether to relocate the Job (up to 5 minutes), the Job may remain in the Pending - failoverinprogress state.

Error cause inspection

When a training interruption occurs, the system checks the node’s GPU and hardware status to determine whether the cause is a hardware failure or a non‑hardware issue such as a user application or configuration. Whether to execute a failover is determined by this assessment result.

When determined to be a hardware fault

If a physical or hardware-level error that prevents normal GPU usage is detected, it is considered a hardware failure and a failover is performed. Since this error prevents continued training on the affected node, we exclude the faulty node and reallocate to healthy resources to resume training. Examples of errors that are considered hardware failures are as follows.

XID code	Error	Explanation
48	GPU memory (HBM) uncorrectable error (Uncorrectable Double Bit ECC)	If uncorrectable damage occurs in the GPU’s high‑capacity memory The computational results of the GPU cannot be trusted, so it cannot be used further
79	GPU has been detached from the system (GPU fallen off the bus)	The system no longer recognizes the GPU itself GPU is unresponsive, preventing training progress
94, 95	GPU internal memory (SRAM) unrecoverable error	An uncorrectable error occurred in the high-speed memory inside the GPU chip Hardware damage occurred
-	GPU overheating, power supply unit (PSU), PCIe, and other hardware component failures	If the temperature exceeds the allowed limit or there is a problem with the power or connected components GPU cannot be used normally

Table. Example errors considered hardware failures

Reference

XID code: a number assigned by the NVIDIA GPU driver to differentiate error types, which can be referenced to identify the cause of errors in GPU logs.

If it is judged to be an internal error

If the cause of the training interruption is determined to be user application code or configuration issues rather than a hardware failure, there is a high likelihood that the same problem will recur even after redeployment, so we reject the Failover and terminate in an internalerror state. Examples of errors considered internal errors are as follows.

XID code	Error	Explanation
-	Forced termination due to insufficient memory (Out Of Memory)	When the training program attempts to use more memory than the allocated GPU or system memory, causing it to be forcibly terminated. Resource shortage and configuration issues, not a GPU memory hardware error (XID 48).Since the same issue occurs even after relocating, it is necessary to adjust training settings such as reducing the batch size and increasing the memory allocation.
31	GPU memory page fault	Error that occurs when a learning program attempts to access a prohibited memory region Mostly caused by bugs in the code
43	GPU processing stopped	If an error occurs in the running program causing GPU tasks to stop It is likely an application issue
13	Graphics·Compute Engine Exception	When a command sent to the GPU is incorrect, causing the issue Typically due to training code or library problems
-	GPU configuration warnings, software errors, etc.	The hardware itself is fine, but the issue is at the configuration/software level GPU replacement or node relocation is unnecessary

Table. Example of errors considered internal errors

Reference

XID code: a number assigned by the NVIDIA GPU driver to differentiate error types, which can be referenced to identify the cause of errors in GPU logs.
In this case, check the training script, execution command, input data, resource settings, and so on. For detailed information on how to check logs, refer to Job Failover 로그 확인하기.

Check Job Failover status

All Services > AI/ML > Simple AI Training Click the menu. 1. Navigate to the Service Home page of Simple AI Training.
On the Service Home page, click the Training Job menu. 2. Navigate to the Training Job List page.

On the Training Job List page, check the Job’s status.

The state flow based on whether Failover is used and the hardware failure assessment result is as follows.

Category	Detailed description
Failover success	When a hardware failure is detected and the resource has been successfully reallocated to a normal state Running → Pending - failoverinprogress → Running if the status changes in that order, the Failover is performed correctly and training resumes successfully
Failover denied	The training was halted, but no hardware failure signal was detected, so it is considered not a candidate for relocation Running → Pending - failoverinprogress → Pending - internalerror If the status changes in this order, the Failover is denied Since it may be due to non-hardware causes such as user code errors, checking the logs is necessary

Failover not configured	When Job Failover is disabled If a training interruption occurs, it does not attempt to reallocate and ends the training in the Running → Failed order

Table. State flow according to Job Failover

The main status values displayed during failover are as follows.

status	Detailed description
Pending - failoverinprogress	Detecting hardware failures and assessing failover feasibility, or reallocating to healthy resources.
Pending - maxretriesexceeded	State where the maximum number of failover attempts has been exceeded and no further reallocation is performed.
Pending - imagepullbackoff	Unable to load the container image, preventing training from starting (image URL and authentication information need to be verified)
Pending - internalerror	The failover was denied because it was determined that the target is not eligible for failover, such as when no hardware failure signal is detected.

Table. Main status values during Job Failover

Checking Job Failover Log

If a failover occurs, you can view the logs of the previous training that was interrupted due to a fault and the logs of the training that resumed after reallocation together in a single log stream.

Reference

The logs of the Training Job are provided in conjunction with the ServiceWatch service. To check the Training Job logs, please refer to Training Job 로그 확인하기.

To check the logs of Job Failover, follow these steps.

All Services > AI/ML > Simple AI Training Click the menu. 1. Navigate to the Service Home page of Simple AI Training.
On the Service Home page, click the Training Job menu. 2. Training Job List Navigate to the page.
On the Training Job List page, select the resource for which you want to view logs. 3. Navigate to the Training Job Details page.
On the Training Job Details page, click the Log tab. 4. The log information for this job is displayed.
Click the name at the bottom of the Training Job name in the log information. 5. Go to the ServiceWatch Log Group Details page.
On the Log Group Details page, click the Log Stream tab. 6. The list of log streams is displayed.
Click the log stream name to check (e.g., master). 7. The stream’s logs are displayed in chronological order.
- When a failover occurs, the logs of the training that was interrupted due to the failure and the logs of the training that was restarted after reallocation are recorded together in a single stream.
- If the initialization log displayed at the start of training (e.g., Starting dataset initialization) appears more than once, you can confirm that the training was reallocated and restarted.

Using Training Lab

Concurrent Checkpointing