Skip to content

Changelog and Centralization

Purpose

This page refers to the history of centralization of pipeline configuration and documentation: moving from per-script or per-repo config to a single pipelines_config.yml and a unified docs structure (including this Engineering section).

What “Centralization” Means Here

  • One config file for pipeline parameters: each pipeline has a top-level section; scripts and scheduler read from the same file.
  • Unified path and storage: Same keys (source_folder, output_folder, input_path, storage) and the same resolution logic across pipelines.
  • Unified docs: User-facing docs (Quick Start, Pipelines, Prefect) plus Engineering (philosophy, context, validation, overfitting, ETL design) in one place.

Changelog

Specific version-by-version changes (e.g. “added S3 support”, “added idempotency”, “added Engineering section”) may be recorded in:

  • A CHANGELOG file in the repo root or under docs/.
  • Release notes or commit history.

This Engineering page anchors the changelog and centralization theme; for current behavior always refer to the main config and the Pipelines/Engineering docs.