> For the complete documentation index, see [llms.txt](https://seeknal.gitbook.io/seeknal/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://seeknal.gitbook.io/seeknal/readme.md).

# Introduction

## Introduction

Seeknal is a platform that abstracts away the complexity of data transformation and AI/ML engineering. It is a collection of tools that help you transform data, store it, and use it for machine learning and data analytics.

Seeknal lets you:

* **Define** data and feature transformations from raw data sources using Pythonic APIs and YAML.
* **Register** transformations and feature groups by names and get transformed data and features for various use cases including AI/ML modeling, data engineering, business metrics calculation and more.
* **Share** transformations and feature groups across teams and company.

Seeknal is useful in multiple use cases including:

* AI/ML modeling: computes your feature transformations and incorporates them into your training data, using point-in-time joins to prevent data leakage while supporting the materialization and deployment of your features for online use in production.
* Data analytics: build data pipelines to extract features and metrics from raw data for Analytics and AI/ML modeling.

## High Level Design

The Seeknal architecture is designed as an **end-to-end data transformation platform** that enables teams to build, operate, and maintain large-scale ETL pipelines. Below is a conceptual overview of how Seeknal’s components work together, from data ingestion to feature storage and orchestration.

### **Overview**

<figure><img src="/files/HY1yc7kzEwVNulUjvy0M" alt=""><figcaption></figcaption></figure>

Open transformer architecture.drawio.png![](blob:https://mtatech.atlassian.net/ab17e870-2d0b-48d3-b611-19f879f53494#media-blob-url=true\&id=f5a0cc66-57c5-49ff-920f-489aaaad3ae1\&collection=contentId-289603738\&contextId=289603738\&mimeType=image%2Fpng\&name=transformer%20architecture.drawio.png\&size=44346\&width=801\&height=511\&alt=)

1. **Data Sources**\
   Seeknal accepts data from diverse sources (databases, APIs, files, etc.) to power its transformation workflows. These sources can be defined and configured within each project’s workspace.
2. **Transformer**
   * **Common Artifacts**: Contains shared configurations and variables that standardize the way data transformations are defined and executed.
   * **Rule**: Houses any validation or transformation rules that can be reused across multiple pipelines.
   * **DXL (Data Transformation Language)**: Encapsulates transformation logic in YAML, making it easy to collaborate, audit, and modify.
   * **Project and Workspaces**: Provide isolation and organization, allowing different teams to manage separate pipelines under a unified system.
3. **Feature Store & Transformed Data**\
   After processing, data can be published to the **Feature Store** for machine learning applications or stored as **transformed data** for downstream analytics. This ensures a centralized and consistent location for both operational and ML-driven use cases.
4. **Orchestration (Prefect)**\
   Seeknal integrates with **Prefect** to automate and schedule transformations, enabling hands-free operation of daily or real-time pipelines.
5. **Python SDK**\
   The Python SDK offers a user-friendly interface for defining, executing, and debugging transformations. It allows teams to incorporate Seeknal’s functionalities into existing Python workflows seamlessly.