Data Source 和 Sink 的容错保证

Data Source 和 Sink 的容错保证

Flink’s fault tolerance mechanism recovers programs in the presence of failures andcontinues to execute them. Such failures include machine hardware failures, network failures,transient program failures, etc.

Flink can guarantee exactly-once state updates to user-defined state only when the source participates in thesnapshotting mechanism. The following table lists the state update guarantees of Flink coupled with the bundled connectors.

Please read the documentation of each connector to understand the details of the fault tolerance guarantees.

Source	Guarantees	Notes
Apache Kafka	exactly once	Use the appropriate Kafka connector for your version
AWS Kinesis Streams	exactly once
RabbitMQ	at most once (v 0.10) / exactly once (v 1.0)
Twitter Streaming API	at most once
Collections	exactly once
Files	exactly once
Sockets	at most once

To guarantee end-to-end exactly-once record delivery (in addition to exactly-once state semantics), the data sink needsto take part in the checkpointing mechanism. The following table lists the delivery guarantees (assuming exactly-oncestate updates) of Flink coupled with bundled sinks:

Sink	Guarantees	Notes
HDFS BucketingSink	exactly once	Implementation depends on Hadoop version
Elasticsearch	at least once
Kafka producer	at least once/ exactly once	exactly once with transactional producers (v 0.11+)
Cassandra sink	at least once / exactly once	exactly once only for idempotent updates
AWS Kinesis Streams	at least once
File sinks	at least once
Socket sinks	at least once
Standard output	at least once
Redis sink	at least once