Real-Time Data Analytics with Apache Flink: A Hands-On SQL Tutorial

Some questions cannot wait for tonight’s batch job: is traffic spiking right now, is this payment suspicious, did the checkout page just break? Apache Flink answers them by processing events as they arrive. This guide is a hands-on introduction: you will run a real Flink cluster on your machine, write a streaming SQL query that counts page views in 10-second windows, and learn the handful of ideas (event time, watermarks, windows, checkpoints) that decide whether such a job gives correct answers.

Updated October 2026: I rewrote this February 2025 post. Its title promised a “hands-on tutorial”, but it contained no commands or code, only descriptions, and it opened with a statistic (“used by 2,117 companies”) that I could not verify against a primary source, so I removed it. This version is a working tutorial that I ran on Apache Flink 2.3.0, and it states what I tested and what I could not.

What Flink Is and When to Use It

Apache Flink is an open-source framework for stateful computations over data streams. A stream is data that keeps arriving (clicks, payments, sensor readings, log lines) and may never end. Flink keeps running programs that read such streams, remember what they need (the “state”, such as a running count), and emit results continuously. It also runs batch jobs, treating a finite dataset as a stream that ends.

• Good fits: real-time monitoring and alerting, fraud and anomaly detection, live dashboards, streaming ETL (cleaning and enriching events on the way into a warehouse), and event-driven features such as recommendations, where the answer must reflect what happened seconds ago.
• Poor fits: a nightly report where nobody needs results sooner (a simple batch job is cheaper to run and operate), and tiny workloads where a database query or a small script is enough. Flink is a distributed system, and running it well is real operational work unless you use a managed service.

How Flink Compares With Other Tools

These tools overlap, and each has changed over the years, so use this as orientation and check current documentation before you decide:

The practical difference is operational: Kafka Streams has no cluster to run, while Flink and Spark are clusters (or managed services) that can read from many systems, not only Kafka.

The Concepts That Matter

Streams, state and parallelism

A Flink job is a graph of operators that read, transform and write streams. It runs in parallel across TaskManager workers, coordinated by a JobManager. Operators that need memory (counts, joins, windows) keep state, which Flink manages for you and protects against failures.

Event time versus processing time

Processing time is the clock of the machine running the job. Event time is when the event actually happened, as recorded in the data. They differ whenever data is delayed: a mobile app that was offline sends yesterday’s clicks today. If you count by processing time, those clicks land in the wrong minute. Flink’s documentation on time explains the model; for analytics that must be correct, use event time.

Watermarks

Event-time results need an answer to “when can I close this window?” A watermark is Flink’s running claim that no event older than time T is still expected. In the example below, event_time - INTERVAL '2' SECOND tells Flink to tolerate events up to two seconds out of order. A longer delay gives more correct results and higher latency; a shorter one gives faster, riskier results. Records that arrive after the watermark has passed their window are treated as late, and in window aggregations they are normally dropped, so choose the delay from how late your data really is.

Windows

Streams never end, so you aggregate over windows: tumbling windows are fixed, non-overlapping slices (every 10 seconds), hopping (sliding) windows overlap (a 1-minute window that moves every 10 seconds), and session windows group bursts of activity separated by gaps of inactivity. The SQL documentation for window table-valued functions and window aggregation covers each kind.

Checkpoints and exactly-once

Flink periodically takes checkpoints, consistent snapshots of all state and stream positions. After a failure, the job restores the last checkpoint and continues, so counts are not lost or double-counted inside Flink. End-to-end exactly-once results also need a source that can replay and a sink that supports transactions or idempotent writes. See the checkpoint documentation. Checkpointing is not on by default; you enable it with, for example, SET 'execution.checkpointing.interval' = '10s'; in the SQL client.

Hands-On: Run Flink and Count Page Views

Step 1: Install and start a local cluster

You need a Java runtime that your Flink version supports (check the requirements for your release; I used OpenJDK 21). The simplest routes are the binary download from the Flink downloads page, or on macOS with Homebrew:

brew install apache-flink      # installs Flink 2.3.0 and OpenJDK 21
export JAVA_HOME=$(/usr/libexec/java_home -v 21)   # if your default Java is older

FLINK_HOME=/opt/homebrew/opt/apache-flink/libexec   # Homebrew on Apple silicon
$FLINK_HOME/bin/start-cluster.sh

With a downloaded archive, unpack it and run ./bin/start-cluster.sh from the folder instead. Then open http://localhost:8081 for the web dashboard, or check it from the terminal:

curl -s localhost:8081/overview
# {"taskmanagers":1,"slots-total":1,...,"flink-version":"2.3.0",...}

I installed it through Homebrew and the cluster started with one TaskManager and one task slot, as shown above.

Step 2: Write the SQL

Flink SQL lets you describe streaming analytics without writing Java. Save this as clicks.sql. It defines a source table that generates 60 click events (one “second” of event time apart, rotating over three pages) with the built-in datagen connector, declares event_time as the time attribute with a 2-second watermark delay, and counts views per page in 10-second tumbling windows:

SET 'sql-client.execution.result-mode' = 'tableau';
SET 'execution.runtime-mode' = 'streaming';
SET 'parallelism.default' = '1';
SET 'table.local-time-zone' = 'UTC';

CREATE TABLE clicks (
  seq BIGINT,
  page AS CASE MOD(seq, 3) WHEN 0 THEN '/home' WHEN 1 THEN '/pricing' ELSE '/docs' END,
  event_time AS TO_TIMESTAMP_LTZ(seq * 1000, 3),
  WATERMARK FOR event_time AS event_time - INTERVAL '2' SECOND
) WITH (
  'connector' = 'datagen',
  'fields.seq.kind' = 'sequence',
  'fields.seq.start' = '0',
  'fields.seq.end' = '59',
  'rows-per-second' = '1000'
);

SELECT
  window_start,
  window_end,
  page,
  COUNT(*) AS page_views
FROM TABLE(TUMBLE(TABLE clicks, DESCRIPTOR(event_time), INTERVAL '10' SECOND))
GROUP BY window_start, window_end, page;

• page AS CASE MOD(seq, 3) ... and event_time AS TO_TIMESTAMP_LTZ(seq * 1000, 3) are computed columns: they derive values from the generated seq number, which makes the output predictable for a tutorial.
• WATERMARK FOR event_time AS event_time - INTERVAL '2' SECOND is what turns event_time into an event-time attribute with the tolerance discussed above.
• TABLE(TUMBLE(TABLE clicks, DESCRIPTOR(event_time), INTERVAL '10' SECOND)) assigns each row to its 10-second window and adds window_start and window_end columns, which you group by.
• 'table.local-time-zone' = 'UTC' makes the window times print in UTC instead of your machine’s time zone.
• One gotcha I hit: I first named the count column views, and the parser rejected it, because VIEWS is a reserved word in Flink SQL. Call it page_views, or put reserved words in backticks.

Step 3: Run it in the SQL client

$FLINK_HOME/bin/sql-client.sh -f clicks.sql

The SQL client submits the query as a streaming job and prints the results. This is the output I got from Flink 2.3.0 (first rows shown, with the column spacing tidied up):

+----+-------------------------+-------------------------+----------+------------+
| op |            window_start |              window_end |     page | page_views |
+----+-------------------------+-------------------------+----------+------------+
| +I | 1970-01-01 00:00:00.000 | 1970-01-01 00:00:10.000 |    /home |          4 |
| +I | 1970-01-01 00:00:00.000 | 1970-01-01 00:00:10.000 | /pricing |          3 |
| +I | 1970-01-01 00:00:00.000 | 1970-01-01 00:00:10.000 |    /docs |          3 |
| +I | 1970-01-01 00:00:10.000 | 1970-01-01 00:00:20.000 | /pricing |          4 |
| +I | 1970-01-01 00:00:10.000 | 1970-01-01 00:00:20.000 |    /home |          3 |
| +I | 1970-01-01 00:00:10.000 | 1970-01-01 00:00:20.000 |    /docs |          3 |
...
Received a total of 18 rows

Check it by hand: events 0 to 9 fall into the first window, and ten events spread over three pages give 4, 3 and 3 views. Eighteen rows is six windows times three pages. The +I marks an insert; windows emit one final row each when the watermark passes their end. The timestamps start in 1970 because the generated events use the sequence number as their time; with real data they would be real timestamps.

Notice that the job finished: the source was bounded (60 rows), so when it ended Flink advanced the watermark to the end of time and closed the last windows. With an unbounded source such as Kafka, the job keeps running and each window closes as the watermark passes it.

Stop the cluster when you are done:

$FLINK_HOME/bin/stop-cluster.sh

Run It With Docker Instead

If you prefer containers, this Compose file runs a JobManager and a TaskManager from the official image. It follows the layout of the Flink Docker setup guide but omits its obsolete version: line, publishes the dashboard on 127.0.0.1 only and mounts the SQL file:

services:
  jobmanager:
    image: flink:2.3.0-java21
    command: jobmanager
    ports:
      - "127.0.0.1:8081:8081"
    volumes:
      - ./clicks.sql:/opt/flink/clicks.sql:ro
    environment:
      - |
        FLINK_PROPERTIES=
        jobmanager.rpc.address: jobmanager

  taskmanager:
    image: flink:2.3.0-java21
    command: taskmanager
    depends_on:
      - jobmanager
    environment:
      - |
        FLINK_PROPERTIES=
        jobmanager.rpc.address: jobmanager
        taskmanager.numberOfTaskSlots: 2

docker compose up -d
docker compose exec jobmanager ./bin/sql-client.sh -f /opt/flink/clicks.sql
docker compose down

I confirmed the YAML parses and that the image tag exists on Docker Hub, but I could not run Docker on my machine, so this variant is untested. The SQL itself is the one I ran above, and the image carries the same Flink distribution.

From Demo to Real Pipeline

Replace the datagen source with real events. A common setup reads JSON events from Kafka (see the Kafka connector documentation, which also tells you which connector JAR to add to Flink’s lib folder):

CREATE TABLE clicks (
  user_id STRING,
  page STRING,
  event_time TIMESTAMP_LTZ(3),
  WATERMARK FOR event_time AS event_time - INTERVAL '5' SECOND
) WITH (
  'connector' = 'kafka',
  'topic' = 'clicks',
  'properties.bootstrap.servers' = 'localhost:9092',
  'properties.group.id' = 'click-analytics',
  'scan.startup.mode' = 'latest-offset',
  'format' = 'json'
);

This DDL follows the documented option names, but I did not run it, since it needs a Kafka broker and the connector JAR. Then write results to a sink table instead of printing them: an upsert Kafka topic, a database through JDBC, files, or a lakehouse table, using CREATE TABLE and INSERT INTO. A realistic first project is an alert query:

SELECT window_start, page, COUNT(*) AS page_views
FROM TABLE(TUMBLE(TABLE clicks, DESCRIPTOR(event_time), INTERVAL '1' MINUTE))
GROUP BY window_start, window_end, page
HAVING COUNT(*) > 1000;   -- pages with a traffic spike

Common Problems and How to Handle Them

• No output, ever. Event-time windows only close when the watermark advances. If one source partition is idle, the watermark can stall; Flink’s docs on watermarks describe idle-source handling. Check the dashboard’s watermark values per operator.
• Results are missing late data. Increase the watermark delay to match how late your data really arrives, or route late events to a separate path to inspect them.
• Data skew. If one key (one popular page or customer) gets most of the traffic, one task is overloaded while the others idle. Pre-aggregate, split hot keys, or rethink the key.
• State keeps growing. Unbounded state (for example a distinct count over all history without a time limit) eventually exhausts memory or disk. Use windows, or set a state time-to-live where the SQL semantics allow it.
• Slow recovery or checkpoints. Large state means slow checkpoints. Tune the interval, use incremental checkpoints with the RocksDB state backend for big state, and watch checkpoint duration in the dashboard.
• Backpressure. When a sink or operator is slower than the input, upstream operators slow down. The dashboard marks busy and backpressured tasks; fix the bottleneck, or add parallelism (more TaskManagers or slots).

Running Flink in Production

• Do not use the local cluster for production. Run Flink on Kubernetes, YARN or a managed service, with high availability for the JobManager and durable storage for checkpoints.
• Enable checkpoints and store them somewhere durable, such as object storage. Practise restoring from one.
• Monitor checkpoint duration and failures, backpressure, watermark lag and consumer lag at the source. Alert on stalls, not only on errors.
• Version and test your SQL. Keep queries in Git, test them with small bounded inputs like the one above, and plan how state is migrated when you change a query.
• Pin versions of Flink and connectors; they must match each other.

Conclusion

Flink’s value is correct answers about a stream while it is still flowing: event time and watermarks decide when a result is final, windows decide what is aggregated, and checkpoints decide whether a failure can corrupt it. You now have a cluster on your machine and a verified windowed query to modify. Change the window size, add a second aggregate, or point the query at a Kafka topic, and see how the watermark delay affects your results. If you are choosing streaming infrastructure for a business problem, start from the latency you actually need before you accept the operational cost of a streaming cluster.