🤵 Author ：Horizon Max
✨ 编程技巧篇：各种操作小结
🎇 机器视觉篇：会变魔术 OpenCV
💥 深度学习篇：简单入门 PyTorch
🏆 神经网络篇：经典网络模型
💻 算法篇：再忙也别忘了 LeetCode
[ 注意力机制 ] 经典网络模型1——SENet 详解与复现
🚀 Squeeze-and-Excitation Networks
Squeeze
：挤压
Excitation
：激励 ；
Squeeze-and-Excitation Networks 简称
SENet
，由 Momenta 和 牛津大学 的Jie Hu等人 提出的一种新的网络结构；
目标是通过建模 卷积特征通道之间的相互依赖关系 来提高网络的表示能力；
在2017年最后一届 ImageNet 挑战赛(ILSVRC) classification 任务中获得 冠军，将错误率降低到 2.251% ；
🔗 论文地址：Squeeze-and-Excitation Networks
🚀 SENet 详解
🎨 Squeeze-and-Excitation block
Squeeze-and-Excitation block
对于任意给定的变换： Ftr ：X → U ，其中 X ∈ R H’xW’xC’ ， U ∈ R HxWxC ，Ftr 用作一个卷积算子 ；
🚩 Squeeze: Global Information Embedding
挤压：全局信息嵌入
（1）
Squeeze
：特征U通过 squeeze 压缩操作，将跨空间维度H × W的特征映射进行聚合，生成一个通道描述符，
HxWxC → 1x1xC
；
将 全局空间信息 压缩到上述 通道描述符 中，使来这些 通道描述符 可以被 其输入的层 利用，这里采用的是
global average pooling
；
🚩 Excitation: Adaptive Recalibration
激励：自适应调整
（2）
Excitation
：每个通道通过一个 基于通道依赖 的自选门机制 来学习特定样本的激活，使其学会使用全局信息，有选择地强调信息特征，并抑制不太有用的特征，这里采用的是
sigmoid
，并在中间嵌入了
ReLU
函数用于限制模型的复杂性和帮助训练 ；
通过
两个全连接层(FC)
构成的瓶颈来参数化门控机制，即
W1
用于降低维度，
W2
用于维度递增 ；
（3）
Reweight
：将 Excitation 输出的权重通过乘法逐通道加权到输入特征上；
总的来说
SE Block
就是在 Layer 的输入和输出之间添加结构：
global average pooling
-
FC
-
ReLU
-
FC
-
sigmoid
；
SE block
的灵活性意味着它可以直接应用于标准卷积以外的转换，通过将 SE block 集成到任何复杂模型当中来开发SENet；
🚩 在非残差网络中的应用
应用于 非残差网络 Inception network 当中，形成
SE-Inception module
；
非残差网络结构框图（Inception block）
Scale
： 改变（文字、图片）的尺寸大小
🚩 在残差网络中的应用
应用于 残差网络 Residual network 当中，形成
SE-ResNet module
；
残差网络结构框图（Residual Block）
论文中对 SE block 的应用用于实验对比：
SE-ResNet-50 网络的准确性优于 ResNet-50 和模型深化版的 ResNet101 网络 ；
对于224 × 224像素的输入图像，ResNet-50 需要164 ms，而 SE-ResNet-50 需要167 ms ；
🚀 SENet 复现
这里实现的是
SE-ResNet
系列网络 ：
# Here is the code :import torch
import torch.nn as nn
import torch.nn.functional as F
from torchinfo import summary
classSE_Block(nn.Module):# Squeeze-and-Excitation blockdef__init__(self, in_planes):super(SE_Block, self).__init__()
self.avgpool = nn.AdaptiveAvgPool2d((1,1))
self.conv1 = nn.Conv2d(in_planes, in_planes //16, kernel_size=1)
self.relu = nn.ReLU()
self.conv2 = nn.Conv2d(in_planes //16, in_planes, kernel_size=1)
self.sigmoid = nn.Sigmoid()defforward(self, x):
x = self.avgpool(x)
x = self.conv1(x)
x = self.relu(x)
x = self.conv2(x)
out = self.sigmoid(x)return out
classBasicBlock(nn.Module):# 左侧的 residual block 结构（18-layer、34-layer）
expansion =1def__init__(self, in_planes, planes, stride=1):# 两层卷积 Conv2d + Shutcutssuper(BasicBlock, self).__init__()
self.conv1 = nn.Conv2d(in_planes, planes, kernel_size=3,
stride=stride, padding=1, bias=False)
self.bn1 = nn.BatchNorm2d(planes)
self.conv2 = nn.Conv2d(planes, planes, kernel_size=3,
stride=1, padding=1, bias=False)
self.bn2 = nn.BatchNorm2d(planes)
self.SE = SE_Block(planes)# Squeeze-and-Excitation block
self.shortcut = nn.Sequential()if stride !=1or in_planes != self.expansion*planes:# Shutcuts用于构建 Conv Block 和 Identity Block
self.shortcut = nn.Sequential(
nn.Conv2d(in_planes, self.expansion*planes,
kernel_size=1, stride=stride, bias=False),
nn.BatchNorm2d(self.expansion*planes))defforward(self, x):
out = F.relu(self.bn1(self.conv1(x)))
out = self.bn2(self.conv2(out))
SE_out = self.SE(out)
out = out * SE_out
out += self.shortcut(x)
out = F.relu(out)return out
classBottleneck(nn.Module):# 右侧的 residual block 结构（50-layer、101-layer、152-layer）
expansion =4def__init__(self, in_planes, planes, stride=1):# 三层卷积 Conv2d + Shutcutssuper(Bottleneck, self).__init__()
self.conv1 = nn.Conv2d(in_planes, planes, kernel_size=1, bias=False)
self.bn1 = nn.BatchNorm2d(planes)
self.conv2 = nn.Conv2d(planes, planes, kernel_size=3,
stride=stride, padding=1, bias=False)
self.bn2 = nn.BatchNorm2d(planes)
self.conv3 = nn.Conv2d(planes, self.expansion*planes,
kernel_size=1, bias=False)
self.bn3 = nn.BatchNorm2d(self.expansion*planes)
self.SE = SE_Block(self.expansion*planes)# Squeeze-and-Excitation block
self.shortcut = nn.Sequential()if stride !=1or in_planes != self.expansion*planes:# Shutcuts用于构建 Conv Block 和 Identity Block
self.shortcut = nn.Sequential(
nn.Conv2d(in_planes, self.expansion*planes,
kernel_size=1, stride=stride, bias=False),
nn.BatchNorm2d(self.expansion*planes))defforward(self, x):
out = F.relu(self.bn1(self.conv1(x)))
out = F.relu(self.bn2(self.conv2(out)))
out = self.bn3(self.conv3(out))
SE_out = self.SE(out)
out = out * SE_out
out += self.shortcut(x)
out = F.relu(out)return out
classSE_ResNet(nn.Module):def__init__(self, block, num_blocks, num_classes=1000):super(SE_ResNet, self).__init__()
self.in_planes =64
self.conv1 = nn.Conv2d(3,64, kernel_size=3,
stride=1, padding=1, bias=False)# conv1
self.bn1 = nn.BatchNorm2d(64)
self.layer1 = self._make_layer(block,64, num_blocks[0], stride=1)# conv2_x
self.layer2 = self._make_layer(block,128, num_blocks[1], stride=2)# conv3_x
self.layer3 = self._make_layer(block,256, num_blocks[2], stride=2)# conv4_x
self.layer4 = self._make_layer(block,512, num_blocks[3], stride=2)# conv5_x
self.avgpool = nn.AdaptiveAvgPool2d((1,1))
self.linear = nn.Linear(512* block.expansion, num_classes)def_make_layer(self, block, planes, num_blocks, stride):
strides =[stride]+[1]*(num_blocks-1)
layers =[]for stride in strides:
layers.append(block(self.in_planes, planes, stride))
self.in_planes = planes * block.expansion
return nn.Sequential(*layers)defforward(self, x):
x = F.relu(self.bn1(self.conv1(x)))
x = self.layer1(x)
x = self.layer2(x)
x = self.layer3(x)
x = self.layer4(x)
x = self.avgpool(x)
x = torch.flatten(x,1)
out = self.linear(x)return out
defSE_ResNet18():return SE_ResNet(BasicBlock,[2,2,2,2])defSE_ResNet34():return SE_ResNet(BasicBlock,[3,4,6,3])defSE_ResNet50():return SE_ResNet(Bottleneck,[3,4,6,3])defSE_ResNet101():return SE_ResNet(Bottleneck,[3,4,23,3])defSE_ResNet152():return SE_ResNet(Bottleneck,[3,8,36,3])deftest():
net = SE_ResNet50()
y = net(torch.randn(1,3,224,224))print(y.size())
summary(net,(1,3,224,224))if __name__ =='__main__':
test()
输出结果：
torch.Size([1,1000])===============================================================================================
Layer (type:depth-idx) Output Shape Param #===============================================================================================
SE_ResNet ----
├─Conv2d:1-1[1,64,224,224]1,728
├─BatchNorm2d:1-2[1,64,224,224]128
├─Sequential:1-3[1,256,224,224]--
│ └─Bottleneck:2-1[1,256,224,224]--
│ │ └─Conv2d:3-1[1,64,224,224]4,096
│ │ └─BatchNorm2d:3-2[1,64,224,224]128
│ │ └─Conv2d:3-3[1,64,224,224]36,864
│ │ └─BatchNorm2d:3-4[1,64,224,224]128
│ │ └─Conv2d:3-5[1,256,224,224]16,384
│ │ └─BatchNorm2d:3-6[1,256,224,224]512
│ │ └─SE_Block:3-7[1,256,1,1]8,464
│ │ └─Sequential:3-8[1,256,224,224]16,896
│ └─Bottleneck:2-2[1,256,224,224]--
│ │ └─Conv2d:3-9[1,64,224,224]16,384
│ │ └─BatchNorm2d:3-10[1,64,224,224]128
│ │ └─Conv2d:3-11[1,64,224,224]36,864
│ │ └─BatchNorm2d:3-12[1,64,224,224]128
│ │ └─Conv2d:3-13[1,256,224,224]16,384
│ │ └─BatchNorm2d:3-14[1,256,224,224]512
│ │ └─SE_Block:3-15[1,256,1,1]8,464
│ │ └─Sequential:3-16[1,256,224,224]--
│ └─Bottleneck:2-3[1,256,224,224]--
│ │ └─Conv2d:3-17[1,64,224,224]16,384
│ │ └─BatchNorm2d:3-18[1,64,224,224]128
│ │ └─Conv2d:3-19[1,64,224,224]36,864
│ │ └─BatchNorm2d:3-20[1,64,224,224]128
│ │ └─Conv2d:3-21[1,256,224,224]16,384
│ │ └─BatchNorm2d:3-22[1,256,224,224]512
│ │ └─SE_Block:3-23[1,256,1,1]8,464
│ │ └─Sequential:3-24[1,256,224,224]--
├─Sequential:1-4[1,512,112,112]--
│ └─Bottleneck:2-4[1,512,112,112]--
│ │ └─Conv2d:3-25[1,128,224,224]32,768
│ │ └─BatchNorm2d:3-26[1,128,224,224]256
│ │ └─Conv2d:3-27[1,128,112,112]147,456
│ │ └─BatchNorm2d:3-28[1,128,112,112]256
│ │ └─Conv2d:3-29[1,512,112,112]65,536
│ │ └─BatchNorm2d:3-30[1,512,112,112]1,024
│ │ └─SE_Block:3-31[1,512,1,1]33,312
│ │ └─Sequential:3-32[1,512,112,112]132,096
│ └─Bottleneck:2-5[1,512,112,112]--
│ │ └─Conv2d:3-33[1,128,112,112]65,536
│ │ └─BatchNorm2d:3-34[1,128,112,112]256
│ │ └─Conv2d:3-35[1,128,112,112]147,456
│ │ └─BatchNorm2d:3-36[1,128,112,112]256
│ │ └─Conv2d:3-37[1,512,112,112]65,536
│ │ └─BatchNorm2d:3-38[1,512,112,112]1,024
│ │ └─SE_Block:3-39[1,512,1,1]33,312
│ │ └─Sequential:3-40[1,512,112,112]--
│ └─Bottleneck:2-6[1,512,112,112]--
│ │ └─Conv2d:3-41[1,128,112,112]65,536
│ │ └─BatchNorm2d:3-42[1,128,112,112]256
│ │ └─Conv2d:3-43[1,128,112,112]147,456
│ │ └─BatchNorm2d:3-44[1,128,112,112]256
│ │ └─Conv2d:3-45[1,512,112,112]65,536
│ │ └─BatchNorm2d:3-46[1,512,112,112]1,024
│ │ └─SE_Block:3-47[1,512,1,1]33,312
│ │ └─Sequential:3-48[1,512,112,112]--
│ └─Bottleneck:2-7[1,512,112,112]--
│ │ └─Conv2d:3-49[1,128,112,112]65,536
│ │ └─BatchNorm2d:3-50[1,128,112,112]256
│ │ └─Conv2d:3-51[1,128,112,112]147,456
│ │ └─BatchNorm2d:3-52[1,128,112,112]256
│ │ └─Conv2d:3-53[1,512,112,112]65,536
│ │ └─BatchNorm2d:3-54[1,512,112,112]1,024
│ │ └─SE_Block:3-55[1,512,1,1]33,312
│ │ └─Sequential:3-56[1,512,112,112]--
├─Sequential:1-5[1,1024,56,56]--
│ └─Bottleneck:2-8[1,1024,56,56]--
│ │ └─Conv2d:3-57[1,256,112,112]131,072
│ │ └─BatchNorm2d:3-58[1,256,112,112]512
│ │ └─Conv2d:3-59[1,256,56,56]589,824
│ │ └─BatchNorm2d:3-60[1,256,56,56]512
│ │ └─Conv2d:3-61[1,1024,56,56]262,144
│ │ └─BatchNorm2d:3-62[1,1024,56,56]2,048
│ │ └─SE_Block:3-63[1,1024,1,1]132,160
│ │ └─Sequential:3-64[1,1024,56,56]526,336
│ └─Bottleneck:2-9[1,1024,56,56]--
│ │ └─Conv2d:3-65[1,256,56,56]262,144
│ │ └─BatchNorm2d:3-66[1,256,56,56]512
│ │ └─Conv2d:3-67[1,256,56,56]589,824
│ │ └─BatchNorm2d:3-68[1,256,56,56]512
│ │ └─Conv2d:3-69[1,1024,56,56]262,144
│ │ └─BatchNorm2d:3-70[1,1024,56,56]2,048
│ │ └─SE_Block:3-71[1,1024,1,1]132,160
│ │ └─Sequential:3-72[1,1024,56,56]--
│ └─Bottleneck:2-10[1,1024,56,56]--
│ │ └─Conv2d:3-73[1,256,56,56]262,144
│ │ └─BatchNorm2d:3-74[1,256,56,56]512
│ │ └─Conv2d:3-75[1,256,56,56]589,824
│ │ └─BatchNorm2d:3-76[1,256,56,56]512
│ │ └─Conv2d:3-77[1,1024,56,56]262,144
│ │ └─BatchNorm2d:3-78[1,1024,56,56]2,048
│ │ └─SE_Block:3-79[1,1024,1,1]132,160
│ │ └─Sequential:3-80[1,1024,56,56]--
│ └─Bottleneck:2-11[1,1024,56,56]--
│ │ └─Conv2d:3-81[1,256,56,56]262,144
│ │ └─BatchNorm2d:3-82[1,256,56,56]512
│ │ └─Conv2d:3-83[1,256,56,56]589,824
│ │ └─BatchNorm2d:3-84[1,256,56,56]512
│ │ └─Conv2d:3-85[1,1024,56,56]262,144
│ │ └─BatchNorm2d:3-86[1,1024,56,56]2,048
│ │ └─SE_Block:3-87[1,1024,1,1]132,160
│ │ └─Sequential:3-88[1,1024,56,56]--
│ └─Bottleneck:2-12[1,1024,56,56]--
│ │ └─Conv2d:3-89[1,256,56,56]262,144
│ │ └─BatchNorm2d:3-90[1,256,56,56]512
│ │ └─Conv2d:3-91[1,256,56,56]589,824
│ │ └─BatchNorm2d:3-92[1,256,56,56]512
│ │ └─Conv2d:3-93[1,1024,56,56]262,144
│ │ └─BatchNorm2d:3-94[1,1024,56,56]2,048
│ │ └─SE_Block:3-95[1,1024,1,1]132,160
│ │ └─Sequential:3-96[1,1024,56,56]--
│ └─Bottleneck:2-13[1,1024,56,56]--
│ │ └─Conv2d:3-97[1,256,56,56]262,144
│ │ └─BatchNorm2d:3-98[1,256,56,56]512
│ │ └─Conv2d:3-99[1,256,56,56]589,824
│ │ └─BatchNorm2d:3-100[1,256,56,56]512
│ │ └─Conv2d:3-101[1,1024,56,56]262,144
│ │ └─BatchNorm2d:3-102[1,1024,56,56]2,048
│ │ └─SE_Block:3-103[1,1024,1,1]132,160
│ │ └─Sequential:3-104[1,1024,56,56]--
├─Sequential:1-6[1,2048,28,28]--
│ └─Bottleneck:2-14[1,2048,28,28]--
│ │ └─Conv2d:3-105[1,512,56,56]524,288
│ │ └─BatchNorm2d:3-106[1,512,56,56]1,024
│ │ └─Conv2d:3-107[1,512,28,28]2,359,296
│ │ └─BatchNorm2d:3-108[1,512,28,28]1,024
│ │ └─Conv2d:3-109[1,2048,28,28]1,048,576
│ │ └─BatchNorm2d:3-110[1,2048,28,28]4,096
│ │ └─SE_Block:3-111[1,2048,1,1]526,464
│ │ └─Sequential:3-112[1,2048,28,28]2,101,248
│ └─Bottleneck:2-15[1,2048,28,28]--
│ │ └─Conv2d:3-113[1,512,28,28]1,048,576
│ │ └─BatchNorm2d:3-114[1,512,28,28]1,024
│ │ └─Conv2d:3-115[1,512,28,28]2,359,296
│ │ └─BatchNorm2d:3-116[1,512,28,28]1,024
│ │ └─Conv2d:3-117[1,2048,28,28]1,048,576
│ │ └─BatchNorm2d:3-118[1,2048,28,28]4,096
│ │ └─SE_Block:3-119[1,2048,1,1]526,464
│ │ └─Sequential:3-120[1,2048,28,28]--
│ └─Bottleneck:2-16[1,2048,28,28]--
│ │ └─Conv2d:3-121[1,512,28,28]1,048,576
│ │ └─BatchNorm2d:3-122[1,512,28,28]1,024
│ │ └─Conv2d:3-123[1,512,28,28]2,359,296
│ │ └─BatchNorm2d:3-124[1,512,28,28]1,024
│ │ └─Conv2d:3-125[1,2048,28,28]1,048,576
│ │ └─BatchNorm2d:3-126[1,2048,28,28]4,096
│ │ └─SE_Block:3-127[1,2048,1,1]526,464
│ │ └─Sequential:3-128[1,2048,28,28]--
├─AdaptiveAvgPool2d:1-7[1,2048,1,1]--
├─Linear:1-8[1,1000]2,049,000===============================================================================================
Total params:28,080,344
Trainable params:28,080,344
Non-trainable params:0
Total mult-adds (G):63.60===============================================================================================
Input size (MB):0.60
Forward/backward pass size (MB):2691.18
Params size (MB):112.32
Estimated Total Size (MB):2804.10===============================================================================================
本文转载自: https://blog.csdn.net/weixin_45084253/article/details/124234120
版权归原作者 Horizon Max 所有， 如有侵权，请联系我们删除。
版权归原作者 Horizon Max 所有， 如有侵权，请联系我们删除。