Enterprise data platforms from the major cloud vendors are extraordinarily capable — and extraordinarily expensive. For growth-stage companies and public sector organisations with real data needs but constrained budgets, a well-designed open-source stack can deliver 80% of the value at 15% of the cost.

The Stack We Recommend

Ingestion: Apache Kafka for real-time event streaming, or Airbyte for batch ELT from SaaS sources. Both are open-source and cloud-managed options exist if you want to reduce operational overhead.

Storage: A cloud object store (S3, GCS, or Azure Blob) as your data lake, with Apache Iceberg as the table format. This gives you ACID transactions, schema evolution, and time-travel queries on top of cheap object storage.

Transformation: dbt (data build tool) for SQL-based transformations. dbt’s combination of version-controlled SQL, automated testing, and documentation generation has made it the de facto standard for analytics engineering, and the open-source version is free.

Orchestration: Apache Airflow (or Prefect for a more modern developer experience) to schedule and monitor your pipeline runs.

Serving: DuckDB for ad hoc analytical queries on your object store, and Apache Superset for self-service dashboards. Both are open-source and production-ready.

What You Will Pay For

The hidden costs are compute (you still need cloud VMs or managed services to run your jobs), data egress, and engineering time. Budget honestly for these. A well-run open-source data platform for a mid-size organisation typically costs $3,000–$8,000 per month in cloud infrastructure — a fraction of a managed Snowflake or Databricks contract at equivalent scale.