Flink Checkpoint expired before completing解决方法

在Flink消费Kafka日志的时候出现了这样的一则报错，

JobManager报错如下：

2024-03-0715:21:12,500[CheckpointTimer]WARNorg.apache.flink.runtime.checkpoint.CheckpointFailureManager[]-Failedtotrigger or complete checkpoint 181for job 835243b848a64f2fae918faf23c5392c.(0 consecutive failed attempts so far)org.apache.flink.runtime.checkpoint.CheckpointException:Checkpoint expired before completing.
    at org.apache.flink.runtime.checkpoint.CheckpointCoordinator$CheckpointCanceller.run(CheckpointCoordinator.java:2438)[flink-dist-1.17-vvr-8.0.7-2-SNAPSHOT.jar:1.17-vvr-8.0.7-2-SNAPSHOT]
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)[?:1.8.0_372]
    at java.util.concurrent.FutureTask.run(FutureTask.java:266)[?:1.8.0_372]
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)[?:1.8.0_372]
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)[?:1.8.0_372]
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)[?:1.8.0_372]
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadP

TaskManager报错如下：

2024-03-0715:22:50,882WARNorg.apache.flink.runtime.checkpoint.CheckpointFailureManager[]-Failedtotrigger or complete checkpoint 4673for job 894b4dd298704c6b91bfaa2c06aa40b4.(2 consecutive failed attempts so far)org.apache.flink.runtime.checkpoint.CheckpointException:Checkpoint expired before completing.
    at org.apache.flink.runtime.checkpoint.CheckpointCoordinator$CheckpointCanceller.run(CheckpointCoordinator.java:2259)[flink-dist-1.15-vvr-6.0.6-SNAPSHOT.jar:1.15-vvr-6.0.6-SNAPSHOT]
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)[?:1.8.0_102]
    at java.util.concurrent.FutureTask.run(FutureTask.java:266)[?:1.8.0_102]
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:186)[?:1.8.0_102]
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:299)[?:1.8.0_102]
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1147)[?:1.8.0_102]
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:622)[?:1.8.0_102]
    at java.lang.Thread.run(Thread.java:834)[?:1.8.0_102]2024-03-0715:22:50,886INFOorg.apache.flink.runtime.jobmaster.JobMaster[]-Tryingtorecover from a global failure.
org.apache.flink.util.FlinkRuntimeException:Exceeded checkpoint tolerable failure threshold.

这个出错的原因主要是：CheckPoint超时导致的。
要解决这个问题：首先要从CheckPoint的原理说起：

1. JobManager在定时做CheckPoint操作的时候，CheckpointCoordinator会周期性的向所有source算子发送barrier
2. 当Source算子接收到Barrier的时候，就会启动CheckPoint处理。有同步和异步两个过程，把当前的状态写入到持久化存储中，最后向CheckPointCoordinator报告快照制作情况，同事向下游算子广播该barrier，恢复数据处理
3. 每个算子按照步骤3不断制作快照并且向下广播，直到最后的barrier传递到sink算子，Sink算子向CheckpointCoordinator报告快照完成，交给CheckpointCoordinator处理
4. 当CheckpointCoordinator收到所有的算子报告之后，认为该周期的快照制作成功，否则，如果在规定的时间内没有收到算子的报告，超时了，则认为本周期快照制作失败。

现在回到我们的报错，这个明显是算子的Checkpoint时间超时了，点开CheckPoint节点一看，Checkpoint一直报错

刚好我们设置的Checkpoint超时是6秒，如下:

env.getCheckpointConfig().setCheckpointTimeout(6000L)

为此本次报错主要解决方法有几种：
1.关闭Checkpoint，但是这个有风险
主要有2种方法：
第一种：直接关闭

env.getCheckpointConfig().disableCheckpointing()

第二种：加大CheckPoint出错容忍告警次数

env.getCheckpointConfig().setTolerableCheckpointFailureNumber(Integer.MAX_VALUE);

2.延长Checkpoint的超时时间并且把Checkpoint的间隔时间调长一点

env.getCheckpointConfig().setCheckpointTimeout(60*1000L);//CK 1分钟延迟
env.enableCheckpointing(60*60*1000L);// CK间隔1小时做一次

3.这种方法如果不行，说明是CK过于复杂，需要较多的资源和时间，这个时候，可以考虑修改CK逻辑，使其尽量简便，也可以使用RockDB加快CK的速度