【大数据】Flink 内存管理（四）：TaskManager 内存分配（实战篇）

《Flink 内存管理》系列（已完结），共包含以下 4 篇文章：

• Flink 内存管理（一）：设置 Flink 进程内存
• Flink 内存管理（二）：JobManager 内存分配（含实际计算案例）
• Flink 内存管理（三）：TaskManager 内存分配（理论篇）
• Flink 内存管理（四）：TaskManager 内存分配（实战篇）

😊 如果您觉得这篇文章有用 ✔️ 的话，请给博主一个一键三连 🚀🚀🚀 吧 （点赞 🧡、关注 💛、收藏 💚）！！！您的支持 💖💖💖 将激励 🔥 博主输出更多优质内容！！！

Flink 内存管理（四）：TaskManager 内存分配（实战篇） 

在 《Flink 内存管理（一）：设置 Flink 进程内存》中我们提到，必须使用下述三种方法之一配置 Flink 的内存（本地执行除外），否则 Flink 启动将失败。这意味着必须明确配置以下选项子集之一，这些子集没有默认值。
序号for TaskManagerfor JobManager1️⃣

taskmanager.memory.flink.size

jobmanager.memory.flink.size

2️⃣

taskmanager.memory.process.size

jobmanager.memory.process.size

3️⃣

taskmanager.memory.task.heap.size

和

taskmanager.memory.managed.size

jobmanager.memory.heap.size

1.单独分配 Total Process Size

单独分配

Total Process Size

，其它的组件都会自动分配。

taskmanager.memory.process.size: 2000m

内存分配步骤如下：

• 首先 Total Process Size = 2000 M = 2000M =2000M
• 因为没有显示分配组件中的任何参数，所以 JVM Overhead = 2000 M × 0.1 = 200 M = 2000M × 0.1 = 200M =2000M×0.1=200M
• JVM Metaspace = 256 M = 256M =256M
• ⭐ 所以 Native Memory = JVM Overhead + JVM Metaspace = 456 M = 456M =456M
• Total Flink Size = 2000 M − 200 M − 256 M = 1544 M B = 1.508 G B = 2000M - 200M - 256M = 1544MB = 1.508GB =2000M−200M−256M=1544MB=1.508GB
• Network Memory = 1544 × 0.1 = 154.4 M = 1544 × 0.1 = 154.4M =1544×0.1=154.4M
• Task Off-Heap = = = 0 M B 0MB 0MB（默认）
• Framework Off-Heap = = = 128 M 128M 128M（默认）
• ⭐ 所以 Total Direct Memory = 154.4 M + 0 + 128 M = 282.4 M = 154.4M + 0 + 128M = 282.4M =154.4M+0+128M=282.4M
• Managed Memory = 1544 M B × 0.4 = 617.6 M = 1544MB × 0.4 = 617.6M =1544MB×0.4=617.6M
• Total JVM Heap Memory = 1544 M − 282.4 M − 617.6 M = 644 M B = 1544M - 282.4M - 617.6M = 644MB =1544M−282.4M−617.6M=644MB
• Framework Heap = 128 M = 128M =128M
• Task Heap = 644 M − 128 M = 516 M = 644M - 128M = 516M =644M−128M=516M

可以与以下的日志进行对比，完全能对上，😁😁😁！

2.单独分配 Total Flink Size

taskmanager.memory.flink.size: 2000m

假如直接只分配

taskmanager.memory.flink.size: 2000m

• Total Flink Size = 2000 M = 2000M =2000M
• Managed Memory = 2000 M × 0.4 = 800 M = 2000M × 0.4 = 800M =2000M×0.4=800M
• NetWork Memory = 2000 M × 0.1 = 200 M = 2000M × 0.1 = 200M =2000M×0.1=200M
• Framework Off-Heap = 128 M = 128M =128M
• Task Off-Heap = 0 B y t e = 0 M = 0Byte = 0M =0Byte=0M
• ⭐ 所以 Total Direct Memory = 200 M + 128 M + 0 M = 328 M = 200M + 128M + 0M= 328M =200M+128M+0M=328M
• Total Off-Heap Memory = 800 M + 328 M = 1128 M = 800M + 328M = 1128M =800M+328M=1128M
• Total JVM Heap = 2000 M − 800 M − 328 M = 872 M = 2000M - 800M - 328M = 872M =2000M−800M−328M=872M
• Framework Heap = 128 M = 128M =128M
• Task Heap = 872 M − 128 M = 744 M = 872M - 128M = 744M =872M−128M=744M
• JVM MetaSpace = 256 M = 256M =256M（默认）
• JVM Overhead = ( = ( =(JVM Overhead + 256 M +\ 256M + 256MMetaspace + 2000 M +\ 2000M + 2000MTotal Flink Size ) × 0.1 ) × 0.1 )×0.1，求解 JVM Overhead = 250.667 M = 250.667M =250.667M 在 192 M B ～ 1 G B 192MB ～ 1GB 192MB～1GB，生效
• Total Process Size = 2000 M + 256 M + 250.667 M = 2506.667 M = 2.448 G B = 2000M + 256M + 250.667M = 2506.667M = 2.448GB =2000M+256M+250.667M=2506.667M=2.448GB

3.单独分配 Heap Size && Managed Memory

taskmanager.memory.task.heap.size: 1000m
taskmanager.memory.managed.size: 1000m

• Framework Heap = 128 M = 128M =128M（默认）
• Task Heap = 1000 M = 1000M =1000M（配置）
• Total JVM Heap = 1000 M + 128 M = 1128 M = 1.102 G B = 1000M + 128M = 1128M = 1.102GB =1000M+128M=1128M=1.102GB
• Managed Memory = 1000 M = 1000M =1000M（配置）
• Framework Off-Heap = 128 M = 128M =128M
• Task Off-Heap = 0 M = 0M =0M
• NetWork = = =Total Flink Size × 0.1 ×\ 0.1 × 0.1 = ( = ( =(NetWork + 1128 M + 1000 M + 128 M + 0 M ) × 0.1 +\ 1128M + 1000M + 128M + 0M) × 0.1 + 1128M+1000M+128M+0M)×0.1，计算得到 Network = 250.667 M B = 250.667MB =250.667MB，处于 64 M B ～ 1 G B 64MB ～ 1GB 64MB～1GB，有效
• ⭐ 所以 Total Direct Memory = 128 M + 250.667 M = 378.667 M = 128M + 250.667M = 378.667M =128M+250.667M=378.667M
• Total Flink Size = 1128 M + 1378.667 M = 2506.667 M = 2.448 G B = 1128M + 1378.667M = 2506.667M = 2.448GB =1128M+1378.667M=2506.667M=2.448GB
• JVM Metaspace = 256 M = 256M =256M（默认）
• JVM Overhead = ( = ( =(JVM Overhead + 1128 M + 1000 M + 378.667 M + 256 M ) × 0.1 = 306.963 M +\ 1128M + 1000M + 378.667M + 256M) × 0.1 = 306.963M + 1128M+1000M+378.667M+256M)×0.1=306.963M，处于 192 M ～ 1 G B 192M ～ 1GB 192M～1GB，有效
• Total Process Size = 2506.667 M + 256 M + 306.963 M = 3069.63 M = 2.998 G = 2506.667M + 256M + 306.963M = 3069.63M = 2.998G =2506.667M+256M+306.963M=3069.63M=2.998G

4.分配 Total Process Size 和 Heap Size && Managed Memory

指定

Total Process Size

，同时显式分配组件

JVM Heap

和

Mamaged Memory

。

taskmanager.memory.process.size: 3000m
taskmanager.memory.task.heap.size: 1000m
taskmanager.memory.managed.size: 1000m

• Total Process Size = 3000 M = 3000M =3000M
• Framework Heap = 128 M = 128M =128M（默认）
• Task Heap = 1000 M = 1000M =1000M（配置）
• Total JVM Heap = = =Framework Heap + + +Task Heap = 128 M + 1000 M = 1128 M = 1.102 G = 128M + 1000M = 1128M = 1.102G =128M+1000M=1128M=1.102G
• Managed Memory = 1000 M = 1000M =1000M（配置）
• Framework Off-Heap = 128 M = 128M =128M（默认）
• Task Off-Heap = 0 M = 0M =0M（默认）
• Network Memory = ( = ( =(Network Memory + 1128 M + 1128 M ) × 0.1 = 250.667 M +\ 1128M + 1128M) × 0.1 = 250.667M + 1128M+1128M)×0.1=250.667M，在 64 M ～ 1 G B 64M ～ 1GB 64M～1GB 之间，满足要求
• Total Off-Heap = 1000 M + 128 M + 250.667 M + 0 M = 1378.667 M = 1.346 G B = 1000M + 128M + 250.667M + 0M = 1378.667M = 1.346GB =1000M+128M+250.667M+0M=1378.667M=1.346GB
• Total Flink Size = 1128 M + 1378.667 M = 2506.667 M = 2.448 G B = 1128M + 1378.667M = 2506.667M = 2.448GB =1128M+1378.667M=2506.667M=2.448GB
• JVM Metaspace = 256 M = 256M =256M
• JVM Overhead = 3000 M − 2506.667 M − 256 M = 237.333 M = 3000M - 2506.667M - 256M = 237.333M =3000M−2506.667M−256M=237.333M，在 192 M ～ 1 G B 192M ～ 1GB 192M～1GB 之间，满足要求

5.分配 Total Flink Size 和 Heap Size && Managed Memory

指定

Total Flink Size

，同时显式分配组件

JVM Heap

和

Mamaged Memory

。

taskmanager.memory.flink.size: 3000m
taskmanager.memory.task.heap.size: 1000m
taskmanager.memory.managed.size: 1000m

• Total Flink Size = 3000 M = 2.93 G B = 3000M = 2.93GB =3000M=2.93GB（配置）
• Managed Memory = 1000 M = 1000M =1000M（配置）
• Task Heap = 1000 M = 1000M =1000M（配置）
• Framework Heap = 128 M = 128M =128M（默认）
• Total JVM Heap = = =Framework Heap + Task Heap = 128 M + 1000 M = 1128 M = 128M + 1000M =1128M =128M+1000M=1128M
• Total Off-Heap Memory = 3000 M − 1128 M = 1872 M = 1.828 G B = 3000M - 1128M = 1872M = 1.828GB =3000M−1128M=1872M=1.828GB
• Direct Memory = = =Total Off-Heap Memory - Managed Memory = 1872 M − 1000 M = 872 M = 1872M - 1000M = 872M =1872M−1000M=872M
• Task Off-Heap = 0 M = 0M =0M（默认）
• Framework Off-Heap = 128 M = 128M =128M（默认）
• Network Memory = = =Direct Memory − - −Task Off-Heap - Framework Off-Heap = 872 M − 0 M − 128 M = 744 M = 872M - 0M - 128M = 744M =872M−0M−128M=744M
• JVM Metaspace = 256 M = 256M =256M（默认）
• JVM Overhead = ( = ( =(JVM Overhead + 3000 M + 256 M ) × 0.1 +\ 3000M + 256M) × 0.1 + 3000M+256M)×0.1，计算得到 JVM Overhead = 361.778 M = 361.778M =361.778M，处于 192 M ～ 1 G 192M～1G 192M～1G 之间，符合条件
• Total Process Size = 3000 M + 256 M + 361.778 M = 3617.778 M = 3.533 G B = 3000M + 256M + 361.778M = 3617.778M = 3.533GB =3000M+256M+361.778M=3617.778M=3.533GB

6.内存分配小结

在 Flink 的集群内存分配的过程中，我们大致可以通过

     3 
    
   
  
    3 
   
  
3 种方式进行分配。

• 指定 Total Process Size 或 Total Flink Size，取决于你用什么方式部署。
• 单独指定某个组件，比如 Task-Heap 的大小，其它的组件都会被推导出来。
• 指定 Total Process / Flink Size && Heap or Off-Heap 其中之一，其它的组件通过默认值进行填充或者进推导，如： - Total Flink Size = Total Heap Size + Total Off-Heap Size- Total Heap Size = Task Heap + Framework Heap- Total Off-Heap = Task Off-Heap + Framework Off-Heap + Network Memory + Managed Memory- Network = Total Flink Size × 0.1 ×\ 0.1 × 0.1（没有指定其它组件情况下）- JVM Overhead = Total Process Size × 0.1 ×\ 0.1 × 0.1（没有指定其它组件情况下）- … …