[FLINK-40282][docs] Expand and correct the concepts Glossary - #28863
[FLINK-40282][docs] Expand and correct the concepts Glossary#28863nicusX wants to merge 5 commits into
Conversation
alpinegizmo
left a comment
There was a problem hiding this comment.
There are a few points where I either disagree, or think some additional clarification is needed.
nicusX
left a comment
There was a problem hiding this comment.
Thanks for reviewing this, @alpinegizmo
I should have addressed all the comments
… Fixed stale inbound link from Flink Architecture page.
|
I split entry reordering into a separate FLINK-40383 and PR #28969. To simplify reviewing this PR once the other is merged |
| per-record basis, but might also only change its Partitioning or perform an aggregation. While | ||
| [Operators](#operator) and [Functions](#function) are the "physical" parts of Flink's API, | ||
| Transformations are only an API concept. Specifically, most transformations are | ||
| Transformations are only an API concept. Specifically, most Transformations are |
There was a problem hiding this comment.
I am not sure what " most Transformations are implemented by certain Operators" is telling me. I would either remove this sentence or be explicit about the certain operators you are referring to or point to how to find the details.
There was a problem hiding this comment.
I agree the sentence is unclear.
I think it's important to keep it because users think Tranformations == Operators, which is not true.
I expanded the wording and added examples from DataStream API (Table API/SQL would bring us down a rabbithole).
@davidradl please let me know if it's clear now
| timestamp *t*. A `Watermark(t)` declares that event time has reached *t* in that stream, and | ||
| therefore that no further records with a timestamp *t' <= t* are expected. This is what allows an | ||
| [Operator](#operator) to decide that an event-time window can be closed, or that an event-time timer | ||
| must fire. A record that arrives after the Watermark has already passed its timestamp is a *late* record. |
There was a problem hiding this comment.
should we mention allowed lateness here?
There was a problem hiding this comment.
@davidradl I think this would be too much for a Glossary. The Watermark section is already very long.
I deliberately decided to stop at this point because Watermarks would deserve a page (a book?) by themselves :)
In particular, mentioning allowed lateness would require: 1) explaining why some operators but not others drop late events, 2) how this behaviour is modified by allowed lateness, 3) how these operators behave on receiving late events within allowed lateness, 4) how this affect state retention, and that 5) allowed lateness is only available in some operators and DataStream API only....
…y Group; 2) Reworded and expanded the statement about Transformations mapping to Operators. Also added newlines to break accidentally long lines (to keep consistency across the doc)
| #### Key Group | ||
|
|
||
| #### History Server | ||
| A Key Group is the atomic unit of key distribution and state assignment across parallel |
There was a problem hiding this comment.
@alpinegizmo I added Key Group, as you suggested.
I deliberately omitted saying that this why changing maxParallelism breaks the ability to restore from a snapshot. It looked too much for a Glossary
What is the purpose of the change
The current Flink Glossary doc page has gaps, some inconsistencies, and ambiguities which may be confusing for new Flink users. It also focus on DataStream API exclusively.
This PR expands and fixes the Glossary page.
Brief change log
New entries (10)
coverage in the glossary
Corrected definitions
STREAMINGandBATCHmodes#Recordand#Function).Expanded entries
Flink Job, Logical Graph, Operator (adds Source/Sink Operators and a note that the docs use
"Operator" loosely for Task/Sub-Task), Operator Chain, Function (Python, UDF, PTF), Flink Cluster,
Flink JobManager, Flink JobMaster, Flink TaskManager, Sub-Task
Consistency
and lowercase in generic English use.
Flinkprefix (5 entries were misplaced). The pagesets
bookToc: false, so ordering is the only way to locate a term.JobGraph --> Job Graph, UID hash --> UID Hash
#jobgraph-->#job-graph, which is not referenced outside of this pageVerifying this change
Does this pull request potentially affect one of the following parts:
@Public(Evolving): NoDocumentation
Was generative AI tooling used to co-author this PR?