Architecture
AetherLake is a decoupled, microservices-oriented data lakehouse running entirely on Kubernetes. It is split into two Helm charts:
security-stack— Keycloak (OIDC/SSO) + its PostgreSQL.core-data-stack— MinIO, Trino, Apache Polaris, Apache Spark, Apache Airflow, Apache Superset, Apache Kafka (Strimzi), Apache Flink (operator), the oauth2-proxy SSO gate, Milvus, and the shared PostgreSQL/Redis.
System overview
Layers
| Layer | Component(s) | Responsibility |
|---|---|---|
| Identity | Keycloak + oauth2-proxy | Single sign-on, OIDC clients, realm roles; SSO gate for UIs without native OIDC |
| Storage | MinIO | S3-compatible object storage (Iceberg data, vectors, raw files) |
| Catalog | Apache Polaris | Iceberg REST catalog + S3 credential vending |
| Query | Trino | Federated SQL over the Iceberg catalog, Kafka topics, and other sources |
| Streaming | Apache Kafka (Strimzi) | Durable event streaming, in-cluster + authenticated external access |
| Stream processing | Apache Flink | SQL jobs that read/write Kafka topics (per-job mini-clusters) |
| Processing | Apache Spark | Distributed batch processing |
| Orchestration | Apache Airflow | DAG-based pipeline scheduling |
| Analytics / BI | Apache Superset | Dashboards and SQL exploration over Trino |
| Vector search | Milvus | Similarity search for AI/ML workloads |
| Control | Control Panel, MCP Server | Management UI + agent tooling |
SSO / OIDC flow
Every service authenticates against the single aetherlake Keycloak realm. The token issuer is http://keycloak.aetherlake.local/realms/aetherlake.
In-cluster DNS
keycloak.aetherlake.local is an ingress host and does not resolve via cluster DNS by default, so server-side OIDC discovery (MinIO, Superset, Airflow, Polaris, oauth2-proxy) would fail. install.sh adds a CoreDNS rewrite mapping that hostname to the Keycloak Service, keeping in-cluster discovery and browser redirects consistent. See Keycloak / SSO.
UIs without native OIDC: the oauth2-proxy gate
Some UIs have no Keycloak integration of their own — the Trino web UI and Milvus Attu. They are protected by an oauth2-proxy deployment sitting behind nginx external-auth annotations:
One login covers every gated host: the session cookie is scoped to .aetherlake.local. Trino runs the web UI with a fixed service user (web-ui.authentication.type=fixed), since humans already passed the Keycloak gate. The in-cluster Trino service (used by the Control Panel and MCP server) is unaffected by the gate. See Keycloak — SSO gate.
Streaming data path (Flink → Kafka → Trino)
External producers/consumers connect through the external listener (nodeport, TLS + SCRAM-SHA-512, KafkaUser credentials) — see Kafka — Producing from outside the cluster.
Streaming lakehouse bridge (Kafka → Flink → Iceberg → Trino)
See Data Pipelines — Kafka-to-Iceberg Bridge.
Lakehouse write path (Trino → Polaris → MinIO)
This credential vending (subscoping) path gives each query short-lived, table-scoped S3 credentials instead of long-lived root keys. See Apache Polaris.
Next
- Components overview — one-line summary + status of each service.
- Kafka — Streaming and Flink — Stream Processing — the streaming layer in detail.
- Per-component reference pages with every setting live under Component Reference in the sidebar.