[docs] Add HTTP Gateway lakehouse quickstart by pranavshuklaa · Pull Request #4231 · apache/fluss

pranavshuklaa · 2026-09-03T21:38:28Z

Purpose

Linked issue: close #4221

Add an end-to-end quickstart showing how to build a real-time lakehouse using the Fluss HTTP Gateway, Fluss, Flink, Paimon, and RustFS.

Brief change log

Add a Docker Compose setup including the Fluss Gateway.
Document building the Gateway image locally.
Show how to create a datalake-enabled Fluss table through the Gateway REST API.
Show how to ingest records through the Gateway HTTP API.
Show how to start the Lakehouse Tiering Service and tier data to Paimon.
Demonstrate querying tiered data through $lake and $lake$snapshots.
Demonstrate Union Read for the current unified Fluss + Paimon view.
Document Gateway preview limitations, write-result handling, readiness, and cleanup.

Tests

Manually ran the full quickstart end-to-end locally, following the commands
in the guide exactly:

Started the Compose stack (Fluss coordinator/tablet servers, Fluss Gateway,
Flink job/task managers, Paimon via RustFS).
Created the lake-enabled orders table through the Gateway REST API.
Ingested the batch of JSON order records over HTTP and confirmed the
write response reported zero failures.
Queried the rows immediately through Flink SQL Union Read, before any
tiering had occurred, and confirmed all rows were visible.
Started the Lakehouse Tiering Service and confirmed orders$lake$snapshots
and orders$lake reflected the tiered rows once the tiering cycle ran.
Wrote an additional record after tiering began and confirmed it was
visible immediately via Union Read, then later visible in orders$lake
once tiered.
Ran the documented cleanup steps (table/database deletion via the
Gateway, docker compose down -v) and confirmed they completed cleanly.

API and Format

No API or storage format changes.

Documentation

Adds a new HTTP Gateway quickstart for a real-time lakehouse and documents the existing Gateway REST API, Fluss datalake integration, Flink Union Read, and Paimon tiering workflow.

pranavshuklaa · 2026-09-03T21:44:31Z

Could someone please review this PR when you get a chance? It adds the HTTP Gateway lakehouse quickstart for #4221. Thanks!!

pbanakar

Hi @pranavshuklaa, thanks for the PR! left some comments

pbanakar · 2026-09-05T08:31:27Z

@@ -0,0 +1,490 @@
+---
+title: Real-Time Lakehouse via the HTTP Gateway
+sidebar_position: 3


check if this conflicts with an existing quickstart. Our page_user_profile.md is at 4 and lakehouse.md and security.md both using sidebar_position 2 could you correct it? currently no file at 3?

Okay so in my latest commit I am fixing the underlying collision rather than just slotting around it. Renumbered so every quickstart page has a unique, sequential position: flink.md=1, lakehouse.md=2, security.md=3 (was 2), this page gateway-lakehouse.md=4 (was 3), page_user_profile.mdx=5 (was 4). Pushed as part of this PR.

pbanakar · 2026-09-05T08:33:33Z

+(however, Compose v1 might work with a few adaptions).
+:::
+
+### Build the Gateway image


every other quickstart uses a pre-built Docker Hub image. This one requires the user to have the Fluss source repo checked out locally and run docker/fluss-gateway/build.sh. That's not a quickstart experience that's a dev setup. The guide should either wait for a published Gateway image or clearly state this is a developer preview requiring a source build upfront, and this feature is for 1.0v that is not released yet

Yeah agreed. Added a statement that this is a developer preview requiring a source build. I think this solves it:

This guide covers a feature shipping in the upcoming fluss 1.0 release.
A pre-built Docker Hub image will be available at GA - until then,
building it locally takes one extra step: run the script below from the
root of your ...

pbanakar · 2026-09-05T08:34:21Z

+### Quitting SQL Client
+
+```sql title="Flink SQL"
+quit;


all other Fluss quickstarts use exit;

Fixed this. switched to exit; to match the convention in the other quickstarts.

…tart

beryllw

Could we also link to this quickstart from the Gateway overview so readers can easily find the end-to-end example?

beryllw · 2026-09-08T09:25:58Z

+REST API instead of Flink SQL. You'll create a datalake-enabled table with
+`curl`, ingest JSON records with `curl`, then use Flink SQL only to run the
+Lakehouse Tiering Service and query the unified real-time + historical data
+(Union Read).


Could we introduce a concrete application scenario and clarify that the application writes through HTTP without separately writing to Paimon?

Yes this would help the new users to understand better and also clarify the aim. the intro now leads with the order management scenario and explicitly states the application only speaks HTTP, without separately writing to Paimon.

beryllw · 2026-09-08T09:29:11Z

+```
+
+```sql title="Flink SQL"
+SELECT order_id, customer, amount_cents, status FROM orders$lake;


Could we query orders before orders$lake to highlight that querying does not need to wait for tiering?

Yes changed the order and added the info to query immediately, no need to wait for tiering.

beryllw · 2026-09-08T09:29:31Z

+performance, but reflecting only what's been tiered so far.
+
+```sql title="Flink SQL"
+SELECT order_id, customer, amount_cents, status FROM orders;


Could we include expected SQL results, lake row counts, and a small comparison of the two query forms?

Added the expected output block (text not ss) for orders and added some comments on what can be expected and added some query statements as well. Should i add order$lake outputs as well?

beryllw · 2026-09-08T09:29:52Z

+starts, not once it's actually accepting connections, so the first
+`/ready` call (or the first database-creation call below) can briefly
+return an error or HTTP 503 right after `docker compose up`. Retry after a
+few seconds if that happens.


Could we wait for a successful metadata request with a timeout? /ready alone does not verify Fluss connectivity.

beryllw · 2026-09-08T09:32:39Z

+Enter the Flink SQL CLI container:
+
+```shell
+docker compose run sql-client


Could we use flink run -d and docker compose run --rm sql-client, and explain that the non-interactive SQL client container may exit normally?

Yes thanks for the suggestion, added it with a note

beryllw · 2026-09-08T09:33:55Z

+After finishing the tutorial, run `exit` to exit the Flink SQL CLI
+container.
+
+## Preview limitations


Could we add a short takeaway explaining which application-side components are no longer needed, while clarifying that the shared Tiering Service still runs on Flink?

The "What this quickstart demonstrates" lists the components the appln no longer needs while explicitly noting the tiering service still runs as shared Flink infrastructure

Also linked this from the gateway overiew as suggested. I checked your reference for the overall structure and it helped in modifying this file . Thanks for sharing that!

beryllw · 2026-09-08T09:43:05Z

I put together a similar end-to-end demo earlier: HTTP Gateway lakehouse quickstart. Feel free to use it as a reference.

beryllw

+1 on the overall direction—thanks for adding this quickstart! Just two small, non-blocking suggestions.

beryllw · 2026-09-11T09:25:40Z

+```bash
+curl -sS --fail-with-body -X POST \
+  -H 'Content-Type: application/json' \
+  "$GATEWAY_URL/v1/clusters/$CLUSTER/databases/$DATABASE/tables/orders/records" \


This step asks readers to use another terminal, but GATEWAY_URL, CLUSTER, and DATABASE were only initialized in the original shell. In a fresh terminal, the URL expands without a hostname and curl exits with error 3, so the fourth order is never written. Please repeat the variable assignments here or provide a shared environment file to source.

beryllw · 2026-09-11T09:32:17Z

+- create a lake-enabled table through the Gateway REST API;
+- write JSON order records over HTTP;
+- read the records back immediately through Flink SQL before any tiering occurs;
+- confirm that the rows are continuously tiered into Paimon.


Could we add an architecture diagram, similar to the Page User Profile quickstart, to help readers understand the overall setup?

beryllw · 2026-09-11T09:32:43Z

 ---
 title: Secure Your Fluss Cluster
-sidebar_position: 2
+sidebar_position: 3


Would it make sense to place this quickstart at the end of the sidebar, keeping the existing guides’ order unchanged?

beryllw · 2026-09-11T10:03:07Z

@leekeiabstraction @fresh-borzoni cc

pranavshuklaa added 12 commits September 4, 2026 01:42

docs: add HTTP Gateway lakehouse quickstart

c745acd

test: validate gateway quickstart environment

3fcea59

test: validate gateway quickstart environment

1af7868

test: validate gateway quickstart environment

255b313

test: validate gateway quickstart environment

38ab71b

test: validate gateway quickstart environment

3207589

test: validate gateway quickstart environment

aed0d67

test: validate gateway quickstart environment

551ede5

test: validate gateway quickstart environment

69f64ad

test: validate gateway quickstart environment

ff3ad52

test: validate gateway quickstart environment

7d9d7f5

[docs] Add HTTP Gateway lakehouse quickstart

4377ce7

pranavshuklaa changed the title ~~Issue 4221 http gateway quickstart~~ [docs] Add HTTP Gateway lakehouse quickstart Sep 3, 2026

pranavshuklaa mentioned this pull request Sep 3, 2026

[docs] Add an HTTP Gateway quickstart for a real-time lakehouse #4221

Open

2 tasks

pbanakar reviewed Sep 5, 2026

View reviewed changes

pranavshuklaa added 2 commits September 6, 2026 18:30

changes to incorporate minor fixes for issue-4221-http-gateway-quicks…

c7aee0d

…tart

changes to incorporate minor fixes for issue-4221-http-gateway-quicks…

c018359

…tart

beryllw reviewed Sep 8, 2026

View reviewed changes

pranavshuklaa added 3 commits September 10, 2026 01:49

[docs] Update HTTP Gateway lakehouse quickstart

e2401dd

Merge branch 'main' into issue-4221-http-gateway-quickstart

219d797

Updated gateway-lakehouse.md

fe84786

beryllw approved these changes Sep 11, 2026

View reviewed changes

Uh oh!

Conversation

pranavshuklaa commented Sep 3, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Purpose

Brief change log

Tests

API and Format

Documentation

Uh oh!

pranavshuklaa commented Sep 3, 2026

Uh oh!

pbanakar left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

beryllw left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

beryllw commented Sep 8, 2026

Uh oh!

beryllw left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

beryllw commented Sep 11, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

pranavshuklaa commented Sep 3, 2026 •

edited

Loading