Data Ops

• 1: Overview
• 1.1: ServiceWatch metric
• 2: How-to guides
• 2.1: Data Ops Services
• 3: API Reference
• 4: CLI Reference
• 5: Release Note

1 - Overview

Service Overview

Data Ops is a managed workflow orchestration service based on Apache Airflow that creates workflows for data processing tasks that occur periodically or repeatedly and automates job scheduling. Users can automate the process of delivering useful data to the right place at the required time and monitor the configuration and progress of data pipelines.

Provided features

Data Ops provides the following features.

• Convenient Installation and Management: Data Ops can be easily installed through a web-based Console in a standard Kubernetes cluster environment. * Apache Airflow and the management module are installed automatically, and integrated monitoring of the web server and scheduler execution status is available through the unified dashboard.
• Dynamic pipeline configuration: Pipeline configuration for data tasks is possible based on Python code. * Since tasks are generated dynamically in conjunction with data job scheduling, you can freely configure the desired workflow structure and scheduling.
• Convenient workflow management: DAG (Direct Acyclic Graph: directed acyclic graph) configuration is visualized and managed through a web-based UI, allowing easy understanding of the sequence and parallel relationships of data flow. * You can also easily manage each task’s timeout, retry count, and priority definitions.

Component

Data Ops is composed of Manager and Service modules, and provides Apache Airflow as a packaged solution.

Data Ops Manager

Data Ops Manager provides various managing features to enable more efficient use of Airflow.

• You can upload Plugin Files, Shared Files, and Python Library Files for use in Ops Service through Ops Manager.
• You can easily provision configuration information for Airflow components within the cluster.
• You can manage other service configuration information within the Airflow cluster and provision it easily.

Data Ops Service

• We provide a managed workflow orchestration service based on Apache Airflow.
• When providing Airflow, you can set Description, required resource size, DAGs GitSync, and Host Alias.
• After creating the service, you can modify the Description, resource size used, DAGs GitSync, and Host Alias to reflect changes to the service.

Server spec type

When creating a Data Ops service, check the following.

• Recommended Service Installation Specifications: CPU KubernetesExecutor 43 cores, CPU CeleryExecutor 25 cores, Memory 50 GB, storage at least 100 GB

Provision status by region

Data Ops is available in the environments below.

Preliminary Service

This is a list of services that must be pre‑configured before creating the service. Please refer to the guide provided for each service and prepare in advance.

1.1 - ServiceWatch metric

In ServiceWatch, you can view Kubernetes Engine metrics for the Kubernetes Engine created by Data Ops. As with Kubernetes Engine, the metrics provided by default monitoring are data collected at one‑minute intervals.

Basic Metrics

The following are the default metrics for the Kubernetes Engine namespace.

The metrics whose names are displayed in bold below are the key metrics selected from the default metrics provided by Kubernetes Engine. Key metrics are used to build service dashboards that are automatically created for each service in ServiceWatch.

Each metric provides guidance in the user guide on which statistical values are meaningful when querying that metric, and among the meaningful statistics, the values shown in bold are the primary statistics. In the service dashboard, you can view key metrics using primary statistical values.

2 - How-to guides

Users can create the service by entering the required Data Ops information and selecting detailed options through the Samsung Cloud Platform Console.

Create Data Ops

You can create and use the Data Ops service in the Samsung Cloud Platform Console.

To create a Data Ops, follow these steps.

1. All Services > Data Analytics > Data Ops Click the menu. 1. Navigate to the Service Home page of Data Ops.

2. On the Service Home page, click the Create Data Ops button. 2. Navigate to the Create Data Ops page.

3. On the Data Ops creation page, enter the information required to create the service and select detailed options.

   • Select the required information in the Version selection area.

     Category	Required	Detailed description
     Data Ops version	Required	Select version of the selected image Provide version list of the provided server image

     Table. Data Ops version selection item
   • In the Cluster Selection area, enter or select the required information. * To install Data Ops, you first need to create nodes for the Kubernetes cluster and the work environment.

     Category	Required	Detailed description
     Cluster name	Required	Select cluster to use

     Table. Data Ops cluster selection item
   • Service Information Input area: enter or select the required information.

     Category	Required	Detailed description
     Data Ops name	Required	Enter Data Ops name must start with a lowercase English letter and must not end with a special character (HTML_TAG_PROTECTED_0-HTML_TAG_PROTECTED_1), use lowercase letters, numbers, and special characters (HTML_TAG_PROTECTED_0-HTML_TAG_PROTECTED_1) to enter 3 to 30 characters
     Storage Class	Required	Select the storage class used by the chosen cluster
     Explanation	Select	Enter additional information or description about Data Ops within 150 characters.
     Domain Settings	Required	Enter Data Ops domain must start with a lowercase English letter and must not end with a special character (-), use lowercase letters, numbers, and special characters (-) to enter 3 to 50 characters {Data Ops name}.{configured domain} becomes the Data Ops access URL.
     Node Selector	Required	To install on a specific node, enter a distinguishable label among the node’s labels If you enter the node label incorrectly, an installation error may occur, so verify the node label in advance The node label can be checked in the node’s yaml file
     Account	Required	Enter Data Ops Manager account ID: Enter a value between 6 and 30 characters, starting with a lowercase English letter and using lowercase letters and numbers Password: Enter 8 to 50 characters, including uppercase letters, lowercase letters, numbers, and special characters (!@#$%^&*) Confirm Password: Enter the same password again
     Host Alias	Select	Add host information to be connected with Data Ops (up to 20 total, including defaults) After selecting Use, click the + button Hostname: Enter 3~63 characters in host name or domain format, using lowercase letters, numbers, and special characters (-) IP: Enter in IP format To delete, click the X button The added host information can be used only if the firewall between the cluster and the server is open

     Table. Data Ops Service Information Input Items
   • In the Additional Information Input area, enter or select the required information.

     Category	Required status	Detailed description
     tag	Selection	Tag addition Tag addition Click the button to create and add tags, or add existing tags Up to 50 tags can be added Newly added tags are applied after the service creation is completed

     Table. Data Ops additional information input fields

4. Summary Check the detailed information and estimated charges generated in the panel, and click the Complete button.

   • When creation is complete, verify the created resources on the Data Ops List page.

View detailed information of Data Ops

You can view and edit the complete list of resources and detailed information in Data Ops. Data Ops Details page consists of Details, Tags, Job History tabs.

To view detailed information of Data Ops, follow the steps below.

1. All Services > Data Analytics > Data Ops Click the menu. 1. Navigate to the Service Home page of Data Ops.
2. On the Service Home page, click the Data Ops menu. 2. Go to the Data Ops List page.
3. On the Data Ops List page, click the resource to view detailed information. 3. Navigate to the Data Ops Details page.
   • Data Ops Details At the top of the page, status information and details about additional features are displayed.

     Category	Detailed description
     Status display	Data Ops status Creating: Creating Running: Running, Data Ops Services can be created Updating: Updating settings Terminating: Service termination Error: Error occurred during creation or service abnormal state
     Hosts file setting information	Button to view and copy host file information for accessing Data Ops
     Service termination	Button to cancel the service

     Table. Data Ops status information and additional features

Detailed Information

Data Ops List page lets you view detailed information of the selected resource and edit it if needed.

tag

Data Ops List page lets you view the tag information of the selected resource, and add, modify, or delete it.

Job History

On the Data Ops List page, you can view the operation history of the selected resource.

Data Ops Terminate

You can cancel unused Data Ops to reduce operating costs. However, if you terminate the service, the currently operating service may be discontinued immediately, so you should proceed with the termination only after fully considering the impact that may arise from the service interruption.

To cancel Data Ops, follow these steps.

1. All Services > Data Analytics > Data Ops Click the menu. 1. Navigate to the Service Home page of Data Ops.
2. On the Service Home page, click the Data Ops menu. Data Ops 목록 페이지로 이동합니다. Data Ops 목록 페이지에서 해지할 자원을 선택하고, 서비스 해지 버튼을 클릭하세요. 해지가 완료되면, Data Ops 목록 페이지에서 자원이 해지되었는지 확인하세요.

2.1 - Data Ops Services

Users can create the service by entering the required information for Data Ops Services within the Data Ops service through the Samsung Cloud Platform Console and selecting detailed options.

Create Data Ops Services

Users can add a service by selecting detailed options of Data Ops or by entering configuration values.

To create Data Ops Services, follow these steps.

1. Click the All Services > Data Analytics > Data Ops menu. Navigate to the Service Home page of Data Ops.

2. On the Service Home page, click Data Ops Servies. You will be taken to the Data Ops Services list page.

3. On the Data Ops Services List page, click the Create Data Ops Services button. You will be taken to the Create Data Ops Services page.

4. On the Data Ops Services Creation page, enter the information required to create the service and select detailed options.

   • Enter or select the required information in the Service Information Input area.

     Category	Required status	Detailed description
     Data Ops name	Required	Select Data Ops
     Ops Service name	Required	Enter the Data Ops Services name starting with a lowercase English letter and not ending with a special character (-), using lowercase letters, numbers, and special characters (-) for 3 to 30 characters
     Storage Class	Required	Select the storage class used by the selected cluster
     Explanation	Selection	Enter additional information or description about Data Ops Services within 150 characters.
     Domain Settings	Required	Data Ops Services domain entry Enter 3 to 50 characters, starting with a lowercase English letter, not ending with special characters (-), using lowercase letters, numbers, and special characters (-) {Data Ops Services name}.{configured domain} becomes the Data Ops Services access address.
     Node Selector	Required	To install on a specific node, enter a distinguishable label among the node’s Labels If you enter an incorrect node label, installation errors may occur, so verify the node label in advance The node label can be checked in the node’s yaml file
     Service Workload	Required	Web Server: Provides visualization of DAG components and status, and serves as the Airflow configuration management module Scheculer: Manages DAG scheduling and execution to orchestrate various DAGs and their tasks Worker: Executes actual orchestration and data processing tasks Worker(Kubernetes): When the Worker operates, pods are dynamically created, allowing efficient resource usage. When Kubernetes is selected, the Replica text box is disabled. Worker(Celery): When the Worker operates, a static pod is created and maintained, enabling faster handling of a large number of requests. When Celery is selected, the Replica text box is enabled and user input is allowed. The selected executor type cannot be changed once chosen
     account	Required	Airflow account input ID: Start with a lowercase English letter and use lowercase letters and numbers, entering a value between 6 and 30 characters Password: Include uppercase (English), lowercase (English), numbers, and special characters (!@#$%^&*) and enter 8 to 50 characters Confirm Password: Re-enter the password identically

     Table. Data Ops Services service information input items
   • Additional Information Input area, enter or select the required information.

     Category	required or not	Detailed description
     Host Alias	Select	Add host information to be linked with Data Ops (up to 20 total, including defaults) Select Use, then click the + button Hostname: Enter the host name or domain format using lowercase letters, numbers, and special characters (-) with a length of 3 to 63 characters IP: Enter in IP format Click the X button to delete The firewall between the cluster and the server must be open to use the added host information
     tag	Selection	Add Tag Add Tag Click the button to create and add a tag, or add an existing tag. You can add up to 50 tags. The newly added tags will be applied after the service creation is completed.

     Table. Data Ops additional information input fields

5. Summary Check the detailed information and estimated billing amount generated in the panel, and click the Complete button.

   • Once creation is complete, check the created resources on the Data Ops Services List page.

Data Ops Services View detailed information

You can view and edit the full list of resources and detailed information for Data Ops Services. The Data Ops Services Details page consists of Details, Tags, Activity History tabs.

Follow these steps to view detailed information about Data Ops Services.

1. Click the All Services > Data Analytics > Data Ops menu. You will be taken to the Service Home page of Data Ops.
2. On the Service Home page, click the Data Ops Services menu. You will be taken to the Data Ops Services list page.
3. On the Data Ops Services List page, click the resource to view detailed information. You will be taken to the Data Ops Services Details page.
   • Data Ops Services Details At the top of the page, status information and additional feature information are displayed.

     Category	Detailed description
     Status display	Data Ops Services status Creating: Creating Running: Running Updating: Updating configuration Terminating: Terminating service Error: Creation failed or service unavailable
     Hosts file configuration information	Button to view and copy host file information for accessing Data Ops Services
     Delete Data Ops Services	Cancel service button

     Table. Data Ops Services status information and additional features

Detailed Information

Data Ops Services List page lets you view detailed information of the selected resource and modify the information if necessary.

tag

Data Ops Services List page allows you to view the tag information of the selected resource, and to add, modify, or delete it.

Job History

You can view the operation history of the selected resource on the Data Ops Services List page.

Data Ops Services Terminate

You can cancel unused Data Ops Services to reduce operating costs. However, canceling a service may cause the running service to stop immediately, so you should fully consider the impact of any service interruption before proceeding with the cancellation.

To cancel Data Ops Services, follow the steps below.

1. Click the All Services > Data Analytics > Data Ops menu. Go to the Service Home page of Data Ops.
2. From the Service Home page, click the Data Ops Services menu. You will be taken to the Data Ops Services list page.
3. On the Data Ops Services List page, select the resource to cancel and click the Data Ops Services Delete button.
4. When termination is complete, verify on the Data Ops Services List page whether the resource has been terminated.

3 - API Reference

4 - CLI Reference

5 - Release Note

Data Ops