Skerter

Case study / 02

DCP

One workflow. Three runtimes.

Role
End-to-end engineering
Context
ML / MLOps system
Status
Live demo
Core
Python / Dask / Kubernetes

Context

Distributed Churn Prediction is an engineering ML/MLOps project that carries the same workflow from one machine to a distributed Kubernetes environment.

The pipeline validates and loads data, creates features, writes Parquet, trains an XGBoost model, evaluates it and stores the resulting model, metrics, plots and logs. The challenge was to change execution scale without creating three unrelated applications.

Run flow

  1. Enter through one of four interfaces

    CLI, Web API, dashboard and Telegram bot invoke the same application workflow instead of owning separate pipeline logic.

  2. Create a run, not a long request

    The application returns a run_id immediately. Status remains addressable while the long-running pipeline continues separately.

  3. Select the execution profile

    Configuration chooses pandas, a local Dask cluster or Dask on Kubernetes while the use cases and pipeline stages stay consistent.

  4. Persist status and artifacts

    The run store exposes lifecycle state while models, metrics, plots, logs and MLflow experiments preserve the output of each execution.

Decisions

  1. One core, multiple surfaces

    Presentation adapters translate HTTP, CLI and chat interactions. Dataset, feature, training and evaluation behavior remains in the application layer.

  2. Profiles over forks

    Runtime-specific orchestration changes how work is executed, not what the workflow means. A local run can grow into a Dask cluster without rewriting the product boundary.

  3. Explicit lifecycle for long work

    queued, running, succeeded and failed are durable concepts. After a process restart, stale active runs are marked failed instead of remaining indefinitely ambiguous.

What shipped

The project exposes the same system through local tooling and a deployed demonstration environment.

  • pandas, local Dask and Kubernetes execution profiles.
  • CLI, FastAPI, browser dashboard and Telegram interfaces over one core.
  • Run cancellation, timeout handling and stale-run recovery after a process restart.
  • MLflow tracking that degrades to a warning instead of failing the pipeline when tracking is unavailable.
  • Smoke, unit and integration tests plus container builds in GitHub Actions.
  • Dask Operator manifests, Kustomize overlays and shared persistent storage for scheduler, workers and pipeline jobs.