Skip to content

[FLINK-40282][docs] Expand and correct the concepts Glossary - #28863

Open
nicusX wants to merge 5 commits into
apache:masterfrom
nicusX:FLINK-40282-docs-expand_glossary
Open

[FLINK-40282][docs] Expand and correct the concepts Glossary#28863
nicusX wants to merge 5 commits into
apache:masterfrom
nicusX:FLINK-40282-docs-expand_glossary

Conversation

@nicusX

@nicusX nicusX commented Aug 1, 2026

Copy link
Copy Markdown
Contributor

What is the purpose of the change

The current Flink Glossary doc page has gaps, some inconsistencies, and ambiguities which may be confusing for new Flink users. It also focus on DataStream API exclusively.
This PR expands and fixes the Glossary page.

Brief change log

New entries (10)

  • Checkpoint, Savepoint: referenced by other entries but never defined
  • Watermark, Parallelism, Channel, Source Split: absent
  • Flink SQL Statement, Statement Set, StreamExchange Operator: first Table API / SQL
    coverage in the glossary
  • Job Graph: promoted from an alias to a full entry
  • Added 2 pointer entries: StreamGraphs --> Logical Graph, Execution Graph --> Physical Graph

Corrected definitions

  • Job Graph no longer documented as a synonym of Logical Graph
  • Task and Sub-Task no longer conflated
  • Physical Graph fixed definition
  • Partition split into Logical Partitioning and Physical Partitioning sub-entries; also added note about for external system such as Kafka topic partitions
  • State Backend reframed as the storage itself rather than a configuration
  • Checkpoint Storage adds the durable-storage framing and a pointer to disaggregated state
  • (Runtime) Execution Mode was a bare link to another page; now describes STREAMING and
    BATCH modes
  • Flink Application: Java or Python, DataStream or Table API (was "a Java Application").
  • Fixed 3 broken cross-links that used capitalized anchors which Hugo doesn't generate (#Record and #Function).

Expanded entries

Flink Job, Logical Graph, Operator (adds Source/Sink Operators and a note that the docs use
"Operator" loosely for Task/Sub-Task), Operator Chain, Function (Python, UDF, PTF), Flink Cluster,
Flink JobManager, Flink JobMaster, Flink TaskManager, Sub-Task

Consistency

  • Applied one capitalization rule: a defined term is capitalized when it refers to a glossary entry
    and lowercase in generic English use.
  • Restored alphabetical order, disregarding the Flink prefix (5 entries were misplaced). The page
    sets bookToc: false, so ordering is the only way to locate a term.
  • Retitled for consistency with sibling entries: ExecutionGraph --> Execution Graph,
    JobGraph --> Job Graph, UID hash --> UID Hash
  • Anchors unchanged except #jobgraph --> #job-graph, which is not referenced outside of this page
  • Added "Also " lines recording the spellings readers meet in code, logs and the Web UI
  • Removed trailing whitespace.

Verifying this change

  • Docs changes only

Does this pull request potentially affect one of the following parts:

  • Dependencies (does it add or upgrade a dependency): No
  • The public API, i.e., is any changed class annotated with @Public(Evolving): No
  • The serializers: No
  • The runtime per-record code paths (performance sensitive): No
  • Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn, ZooKeeper: No
  • The S3 file system connector: No

Documentation

  • Does this pull request introduce a new feature? No
  • If yes, how is the feature documented? not applicable

Was generative AI tooling used to co-author this PR?
  • Yes (please specify the tool below)

@flinkbot

flinkbot commented Aug 1, 2026

Copy link
Copy Markdown
Collaborator

CI report:

Bot commands The @flinkbot bot supports the following commands:
  • @flinkbot run azure re-run the last Azure build

@alpinegizmo alpinegizmo left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There are a few points where I either disagree, or think some additional clarification is needed.

Comment thread docs/content/docs/concepts/glossary.md Outdated
Comment thread docs/content/docs/concepts/glossary.md Outdated
Comment thread docs/content/docs/concepts/glossary.md Outdated
Comment thread docs/content/docs/concepts/glossary.md Outdated
Comment thread docs/content/docs/concepts/glossary.md Outdated
Comment thread docs/content/docs/concepts/glossary.md Outdated
Comment thread docs/content/docs/concepts/glossary.md Outdated
Comment thread docs/content/docs/concepts/glossary.md Outdated
Comment thread docs/content/docs/concepts/glossary.md Outdated
Comment thread docs/content/docs/concepts/glossary.md Outdated

@nicusX nicusX left a comment

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for reviewing this, @alpinegizmo
I should have addressed all the comments

Comment thread docs/content/docs/concepts/glossary.md Outdated
Comment thread docs/content/docs/concepts/glossary.md Outdated
Comment thread docs/content/docs/concepts/glossary.md Outdated
Comment thread docs/content/docs/concepts/glossary.md Outdated
Comment thread docs/content/docs/concepts/glossary.md Outdated
Comment thread docs/content/docs/concepts/glossary.md Outdated
Comment thread docs/content/docs/concepts/glossary.md Outdated
@github-actions github-actions Bot added the community-reviewed PR has been reviewed by the community. label Aug 5, 2026
… Fixed stale inbound link from Flink Architecture page.
@nicusX

nicusX commented Aug 13, 2026

Copy link
Copy Markdown
Contributor Author

I split entry reordering into a separate FLINK-40383 and PR #28969. To simplify reviewing this PR once the other is merged

@alpinegizmo alpinegizmo left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this is solid. If you wanted to go further, I think we could find more terms to add -- the one that comes to mind immediately is key group -- but I think this is would be good place to stop.

Comment thread docs/content/docs/concepts/glossary.md Outdated
per-record basis, but might also only change its Partitioning or perform an aggregation. While
[Operators](#operator) and [Functions](#function) are the "physical" parts of Flink's API,
Transformations are only an API concept. Specifically, most transformations are
Transformations are only an API concept. Specifically, most Transformations are

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I am not sure what " most Transformations are implemented by certain Operators" is telling me. I would either remove this sentence or be explicit about the certain operators you are referring to or point to how to find the details.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I agree the sentence is unclear.
I think it's important to keep it because users think Tranformations == Operators, which is not true.
I expanded the wording and added examples from DataStream API (Table API/SQL would bring us down a rabbithole).
@davidradl please let me know if it's clear now

timestamp *t*. A `Watermark(t)` declares that event time has reached *t* in that stream, and
therefore that no further records with a timestamp *t' <= t* are expected. This is what allows an
[Operator](#operator) to decide that an event-time window can be closed, or that an event-time timer
must fire. A record that arrives after the Watermark has already passed its timestamp is a *late* record.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

should we mention allowed lateness here?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@davidradl I think this would be too much for a Glossary. The Watermark section is already very long.
I deliberately decided to stop at this point because Watermarks would deserve a page (a book?) by themselves :)
In particular, mentioning allowed lateness would require: 1) explaining why some operators but not others drop late events, 2) how this behaviour is modified by allowed lateness, 3) how these operators behave on receiving late events within allowed lateness, 4) how this affect state retention, and that 5) allowed lateness is only available in some operators and DataStream API only....

…y Group; 2) Reworded and expanded the statement about Transformations mapping to Operators. Also added newlines to break accidentally long lines (to keep consistency across the doc)
#### Key Group

#### History Server
A Key Group is the atomic unit of key distribution and state assignment across parallel

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@alpinegizmo I added Key Group, as you suggested.
I deliberately omitted saying that this why changing maxParallelism breaks the ability to restore from a snapshot. It looked too much for a Glossary

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

community-reviewed PR has been reviewed by the community.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants