Glossary

Flink Application Cluster

A Flink Application Cluster is a dedicated Flink Cluster that onlyexecutes a single Flink Job. The lifetime of theFlink Cluster is bound to the lifetime of the Flink Job. FormerlyFlink Application Clusters were also known as Flink Clusters in job mode. Compare toFlink Session Cluster.

Flink Cluster

A distributed system consisting of (typically) one Flink Master and one or moreFlink TaskManager processes.

Event

An event is a statement about a change of the state of the domain modelled by theapplication. Events can be input and/or output of a stream or batch processing application.Events are special types of records.

ExecutionGraph

see Physical Graph

Function

Functions are implemented by the user and encapsulate theapplication logic of a Flink program. Most Functions are wrapped by a correspondingOperator.

Instance

The term instance is used to describe a specific instance of a specific type (usuallyOperator or Function) during runtime. As Apache Flink is mostly written inJava, this corresponds to the definition of Instance or Object in Java. In the context of ApacheFlink, the term parallel instance is also frequently used to emphasize that multiple instances ofthe same Operator or Function type are running in parallel.

Flink Job

A Flink Job is the runtime representation of a Flink program. A Flink Job can either be submittedto a long running Flink Session Cluster or it can be started as aself-contained Flink Application Cluster.

JobGraph

see Logical Graph

Flink JobManager

JobManagers are one of the components running in the Flink Master. A JobManager isresponsible for supervising the execution of the Tasks of a single job. Historically, thewhole Flink Master was called JobManager.

Logical Graph

A logical graph is a directed graph describing the high-level logic of a stream processing program.The nodes are Operators and the edges indicate input/output-relationships ordata streams or data sets.

Managed State

Managed State describes application state which has been registered with the framework. ForManaged State, Apache Flink will take care about persistence and rescaling among other things.

Flink Master

The Flink Master is the master of a Flink Cluster. It contains three distinctcomponents: Flink Resource Manager, Flink Dispatcher and one Flink JobManagerper running Flink Job.

Operator

Node of a Logical Graph. An Operator performs a certain operation, which isusually executed by a Function. Sources and Sinks are special Operators for dataingestion and data egress.

Operator Chain

An Operator Chain consists of two or more consecutive Operators without anyrepartitioning in between. Operators within the same Operator Chain forward records to each otherdirectly without going through serialization or Flink’s network stack.

Partition

A partition is an independent subset of the overall data stream or data set. A data stream ordata set is divided into partitions by assigning each record to one or more partitions.Partitions of data streams or data sets are consumed by Tasks during runtime. Atransformation which changes the way a data stream or data set is partitioned is often calledrepartitioning.

Physical Graph

A physical graph is the result of translating a Logical Graph for execution in adistributed runtime. The nodes are Tasks and the edges indicate input/output-relationshipsor partitions of data streams or data sets.

Record

Records are the constituent elements of a data set or data stream. Operators andFunctions receive records as input and emit records as output.

Flink Session Cluster

A long-running Flink Cluster which accepts multiple Flink Jobs forexecution. The lifetime of this Flink Cluster is not bound to the lifetime of any Flink Job.Formerly, a Flink Session Cluster was also known as a Flink Cluster in session mode. Compare toFlink Application Cluster.

State Backend

For stream processing programs, the State Backend of a Flink Job determines how itsstate is stored on each TaskManager (Java Heap of TaskManager or (embedded)RocksDB) as well as where it is written upon a checkpoint (Java Heap ofFlink Master or Filesystem).

Sub-Task

A Sub-Task is a Task responsible for processing a partition ofthe data stream. The term “Sub-Task” emphasizes that there are multiple parallel Tasks for the sameOperator or Operator Chain.

Task

Node of a Physical Graph. A task is the basic unit of work, which is executed byFlink’s runtime. Tasks encapsulate exactly one parallel instance of anOperator or Operator Chain.

Flink TaskManager

TaskManagers are the worker processes of a Flink Cluster. Tasks arescheduled to TaskManagers for execution. They communicate with each other to exchange data betweensubsequent Tasks.

Transformation

A Transformation is applied on one or more data streams or data sets and results in one or moreoutput data streams or data sets. A transformation might change a data stream or data set on aper-record basis, but might also only change its partitioning or perform an aggregation. WhileOperators and Functions) are the “physical” parts of Flink’s API,Transformations are only an API concept. Specifically, most - but not all - transformations areimplemented by certain Operators.