0
0

更简单的掩码图像建模框架SimMIM介绍和PyTorch代码实现

MAE发布以来，各种使用掩码技术的自监督掩码模型在其基础之上有了更进一步的研究。在本文中我们将探索一篇和MAE同期的工作：SimMIM: A Simple Framework for Masked Image Modeling，研究团队是微软亚研院，并在PyTorch中编写它，最后我们也会提供相关的代码。

SimMIM的骨干网络是VIT，熟悉自监督学习的基础知识也非常有帮助，最后我们还要精通PyTorch，因为我们使用它来实现我们的模型。

图像中的掩码技术

在过去的几年中，对比学习和非对比学习方法一直是计算机视觉(CV)的自监督学习(SSL)的主要形式，他们中的最先进的(SOTA)模型与监督学习处于同等地位。从根本上说，对比学习的目的是教会神经网络将相似的数据点(正对)放在一起，并将不同的数据点(负对)分开，这是一项需要学习视觉模式的任务。非对比学习克服了与对比学习相关的障碍(例如，需要大量的标注数据)。

而自然语言处理(NLP)为SSL使用掩码建模，其中输入的一个随机片段被掩码，模型的目标是根据剩余的信息恢复它，这样做的印象是将教会模型语法。像BERT这样的神经网络就属于这一类，这种方式已经取得了惊人的性能。

NLP 和视觉之间存在一定的差异，图像中的局部性非常强，即附近的像素高度相关，因此即使一个像素被屏蔽，通过分析其邻居也可以相对容易地推断出它的值。并且照片是连续的不像 NLP 中的标记是离散的，像素是低级原始特征而单词是人类构建的高级概念。

随着ViT的出现，蒙面建模最近已进入计算机视觉领域，在 ImageNet 分类等下游任务上取得了具有竞争力的分数。但是这些方法很棘手，并且依赖于精细的组件，如像 iGPT 一样像素聚类，以及通过额外的离散变分自动编码器 (dVAE) 进行标记化，这是 BEiT 使用的一种技术。

SimMIM 是一个简单的掩码图像建模框架并且超越了以前的 SOTA 基线，在没有复杂的元素的同时保持了效率。具体来说在提取图像的标记后，SimMIM 通过用可学习的掩码标记替换它们来随机屏蔽一些标记，并用 ViT 对数据进行编码。接下来通过将掩码标记的编码表示传递给线性层来重建缺失部分，损失是预测像素和实际像素之间的 L1 损失除以掩码标记的数量。

Pytorch实现

SimMIM 很简单而且没有特别复杂的操作。我们假设从一组维度为 batch_size X n_tokens X token_dim 的令牌开始。

  1. fromtorchimport (
  2. randn,
  3. )
  5. # tokens is currently a dummy tensor.
  6. # Later, it will be replaced by the actual tokens
  7. tokens=randn(batch_size, n_tokens, token_dim)

首先必须确定要屏蔽哪些标记。一种策略是在每个样本的 [0, n_tokens-1] （从零开始的索引）范围内生成一组索引，这些索引将是为该行屏蔽的标记的索引。

  1. fromtorchimport (
  2. randn,
  3. )
  5. tokens=randn(batch_size, n_tokens, token_dim)
  7. indices_to_mask=randn(batch_size, n_tokens)
  9. # Number of tokens to mask
  10. # 50% of the total number of tokens performs well on average.
  11. # However, for smaller patch sizes, a higher masking ratio is generally better.
  12. # For example, for a patch size of 32, 0.5 performs well but for 
  13. # a patch size of 16, it would be worthwhile to increase it to 0.8.
  14. n_masked_tokens=int(0.5*n_tokens)
  16. # topk returns the k largest elements as well as their indices
  17. # dim=1 tells it to find the maximum values and their indices
  18. # on a per-row basis
  19. # The indices of the tokens that are to be masked is going
  20. # to be the indices of the n_masked_tokens largest values
  21. indices_to_mask=indices_to_mask.topk(
  22. k=n_masked_tokens,
  23. dim=1,
  24. )
  26. # The largest values can be accesses via indices_to_mask.values,
  27. # and their indices can be accessed via indices_to_mask.indices
  28. indices_to_mask=indices_to_mask.indices

indices_to_mask 的形状为 batch_size X n_masked_tokens，每一行都包含要为该特定数据点屏蔽标记的索引。使用 indices_to_mask 索引标记会稍微复杂一些，所以更好的方法是构建大小为 batch_size * n_tokens 的位掩码，其中如果标记[i][j] 被屏蔽，则 bitmask[i][j] 为 True，否则为 False .

  1. fromtorchimport (
  2. randn,
  3. zeros,
  4. )
  6. tokens=randn(batch_size, n_tokens, token_dim)
  8. indices_to_mask=randn(batch_size, n_tokens)
  10. n_masked_tokens=int(0.5*n_tokens)
  12. indices_to_mask=indices_to_mask.topk(
  13. k=n_masked_tokens,
  14. dim=1,
  15. )
  17. indices_to_mask=indices_to_mask.indices
  19. # Initially, bitmask is simply full of zeros (i.e., False)
  20. bitmask=zeros(batch_size, n_tokens)
  22. # What this line does is as follows:
  23. # For every row i, bitmask[i][j] is replaced
  24. # by the value argument (in this case 1), where j takes every value
  25. # in indices_to_mask[i].
  26. # For example, if indices_to_mask[3] is
  27. # [2, 4, 7], then bitmask[3][2], bitmask[3][4], and bitmask[3][7]
  28. # are all set to 1.
  29. bitmask=bitmask.scatter(
  30. dim=1,
  31. index=indices_to_mask,
  32. value=1,
  33. )
  34. bitmask=bitmask.bool()

要使用位掩码首先要通过VIT从输入产生令牌。我们这里使用的ViT来自timm包，但是它可以很容易地为转换为其他实现。

  1. fromtorchimport (
  2. randn,
  3. zeros,
  4. )
  6. # vit is assumed to be a vision transformer from timm
  7. # To get tokens from a timm ViT, one must call its patch_embed method
  8. # tokens is now of shape batch_size X n_tokens X token_dim
  9. # Keep in mind that input is image data and of size 
  10. # batch_size X n_channels X height X width
  11. tokens=vit.patch_embed(input)
  13. indices_to_mask=randn(batch_size, n_tokens)
  15. n_masked_tokens=int(0.5*n_tokens)
  17. indices_to_mask=indices_to_mask.topk(
  18. k=n_masked_tokens,
  19. dim=1,
  20. )
  22. indices_to_mask=indices_to_mask.indices
  24. bitmask=zeros(batch_size, n_tokens)
  26. bitmask=bitmask.scatter(
  27. dim=1,
  28. index=indices_to_mask,
  29. value=1,
  30. )
  31. bitmask=bitmask.bool()

下一步使用掩码令牌替换切片。PyTorch 不允许以Inplace的方式修改变量，不能直接将掩码标记赋值给令牌[bitmask]；所以必须用掩码标记填充形状为 batch_size * n_tokens * token_dim 的张量（维度相同），

  1. fromtorchimport (
  2. randn,
  3. zeros,
  4. )
  5. fromtorch.nnimport (
  6. Parameter,
  7. )
  9. tokens=vit.patch_embed(input)
  11. # The mask token itself is simply a vector of dimension token_dim
  12. mask_token=Parameter(randn(token_dim))
  14. # mask_token is repeated to make it the same shape as tokens
  15. # mask_tokens is now of size batch_size X n_tokens X token_dim
  16. mask_tokens=mask_token.repeat(batch_size, n_tokens, 1)
  18. indices_to_mask=randn(batch_size, n_tokens)
  20. n_masked_tokens=int(0.5*n_tokens)
  22. indices_to_mask=indices_to_mask.topk(
  23. k=n_masked_tokens,
  24. dim=1,
  25. )
  27. indices_to_mask=indices_to_mask.indices
  29. bitmask=zeros(batch_size, n_tokens)
  31. bitmask=bitmask.scatter(
  32. dim=1,
  33. index=indices_to_mask,
  34. value=1,
  35. )
  36. bitmask=bitmask.bool()

这样就完成了掩码的过程

  1. fromtorchimport (
  2. randn,
  3. zeros,
  4. )
  5. fromtorch.nnimport (
  6. Parameter,
  7. )
  9. tokens=vit.patch_embed(input)
  11. mask_token=Parameter(randn(token_dim))
  13. mask_tokens=mask_token.repeat(batch_size, n_tokens, 1)
  15. indices_to_mask=randn(batch_size, n_tokens)
  17. n_masked_tokens=int(0.5*n_tokens)
  19. indices_to_mask=indices_to_mask.topk(
  20. k=n_masked_tokens,
  21. dim=1,
  22. )
  24. indices_to_mask=indices_to_mask.indices
  26. bitmask=zeros(batch_size, n_tokens)
  28. bitmask=bitmask.scatter(
  29. dim=1,
  30. index=indices_to_mask,
  31. value=1,
  32. )
  33. bitmask=bitmask.bool()
  35. # bitmask must have the same number of axes as tokens and mask_tokens
  36. # Therefore, unsqueeze(2) adds an axis to it and it is now of shape batch_size X n_tokens X 1
  37. bitmask=bitmask.unsqueeze(2)
  39. # ~bitmask turns True to False and False to True
  40. # Here, all that is taking place is (~bitmask) is multiplied by tokens
  41. # to zero out every token that is supposed to be masked, and the result is added
  42. # to bitmask*mask_tokens, in which everything is 0 except the tokens that are
  43. # supposed to mask.
  44. tokens= (~bitmask)*tokens+bitmask*mask_tokens

然后就是位置嵌入

  1. fromtorchimport (
  2. randn,
  3. zeros,
  4. )
  5. fromtorch.nnimport (
  6. Parameter,
  7. )
  9. tokens=vit.patch_embed(input)
  11. mask_token=Parameter(randn(token_dim))
  13. mask_tokens=mask_token.repeat(batch_size, n_tokens, 1)
  15. indices_to_mask=randn(batch_size, n_tokens)
  17. n_masked_tokens=int(0.5*n_tokens)
  19. indices_to_mask=indices_to_mask.topk(
  20. k=n_masked_tokens,
  21. dim=1,
  22. )
  24. indices_to_mask=indices_to_mask.indices
  26. bitmask=zeros(batch_size, n_tokens)
  28. bitmask=bitmask.scatter(
  29. dim=1,
  30. index=indices_to_mask,
  31. value=1,
  32. )
  33. bitmask=bitmask.bool()
  35. bitmask=bitmask.unsqueeze(2)
  37. tokens= (~bitmask)*tokens+bitmask*mask_tokens
  39. # In timm, a ViT's position embedding is accessible via vit.pos_embed
  40. # The reason for vit.pos_embed[:, 1:] in place of simply vit.pos_embed
  41. # is that the first position embedding vector is for the class token,
  42. # which is not used for self-supervised learning.
  43. tokens=tokens+vit.pos_embed[:, 1:]

令牌可以被输入到 ViT获得它的编码表示。

  1. fromtorchimport (
  2. randn,
  3. zeros,
  4. )
  5. fromtorch.nnimport (
  6. Parameter,
  7. )
  9. tokens=vit.patch_embed(input)
  11. mask_token=Parameter(randn(token_dim))
  13. mask_tokens=mask_token.repeat(batch_size, n_tokens, 1)
  15. indices_to_mask=randn(batch_size, n_tokens)
  17. n_masked_tokens=int(0.5*n_tokens)
  19. indices_to_mask=indices_to_mask.topk(
  20. k=n_masked_tokens,
  21. dim=1,
  22. )
  24. indices_to_mask=indices_to_mask.indices
  26. bitmask=zeros(batch_size, n_tokens)
  28. bitmask=bitmask.scatter(
  29. dim=1,
  30. index=indices_to_mask,
  31. value=1,
  32. )
  33. bitmask=bitmask.bool()
  35. bitmask=bitmask.unsqueeze(2)
  37. tokens= (~bitmask)*tokens+bitmask*mask_tokens
  39. tokens=tokens+vit.pos_embed[:, 1:]
  41. # The encoded representation of tokens
  42. encoded=vit.blocks(tokens)

被屏蔽的令牌将从编码中获取，然后它们通过线性层来重建像素值。

  1. fromtorchimport (
  2. randn,
  3. zeros,
  4. )
  5. fromtorch.nnimport (
  6. Linear,
  7. Parameter,
  8. )
  10. tokens=vit.patch_embed(input)
  12. mask_token=Parameter(randn(token_dim))
  14. mask_tokens=mask_token.repeat(batch_size, n_tokens, 1)
  16. indices_to_mask=randn(batch_size, n_tokens)
  18. n_masked_tokens=int(0.5*n_tokens)
  20. indices_to_mask=indices_to_mask.topk(
  21. k=n_masked_tokens,
  22. dim=1,
  23. )
  25. indices_to_mask=indices_to_mask.indices
  27. bitmask=zeros(batch_size, n_tokens)
  29. bitmask=bitmask.scatter(
  30. dim=1,
  31. index=indices_to_mask,
  32. value=1,
  33. )
  34. bitmask=bitmask.bool()
  36. bitmask=bitmask.unsqueeze(2)
  38. tokens= (~bitmask)*tokens+bitmask*mask_tokens
  40. tokens=tokens+vit.pos_embed[:, 1:]
  42. encoded=vit.blocks(tokens)
  44. # To index input and encoded with bitmask,
  45. # the axis that was added must be removed.
  46. # This reverts bit_mask to a size of batch_size X n_tokens
  47. bitmask=bitmask.squeeze(2)
  49. # The encoded mask tokens, of shape batch_size X n_masked_tokens X token_dim
  50. masked_tokens_encoded=encoded[bitmask]
  52. # In timm, A ViT's patch height and width are vit.patch_embed.patch_size
  53. patch_height=patch_width=vit.patch_embed.patch_size
  55. # The input is the tokens,
  56. # the output is the reconstructed raw pixel values.
  57. # Therefore, the output shape is 3 (for 3 channels)
  58. # multiplied by patch_height*patch_width, which is the original shape
  59. # of the patches before they were tokenized
  60. decoder_out_dim=3*patch_height*patch_width
  61. decoder=Linear(
  62. in_features=token_dim,
  63. out_features=decoder_out_dim,
  64. )
  66. # The reconstructed pixels, of shape batch_size X n_masked_tokens X 3*patch_height*patch_width
  67. masked_patches_reconstructed=decoder(masked_tokens_encoded)

最后masked_patchesde的重构与初始数据中的原始像素进行比较。因为输入的patches不可用，因此必须对输入进行patche处理。PyTorch的reshap函数有一些限制，用torch进行拼接。重塑将产生不正确的输出。所以一个简单的解决方案是einops（它是一个方便用于操作张量的库，并且与框架无关）。

需要注意的是，patches和令牌（Token）是不同的。patches是从 batch_size * 3 * height * width 重塑为 batch_size * n_tokens * 3 * patch_height * patch_width 的数据，而令牌是通过沿最终轴线性变换patches创建的。

  1. fromeinopsimport (
  2. rearrange,
  3. )
  4. fromtorchimport (
  5. randn,
  6. zeros,
  7. )
  8. fromtorch.nnimport (
  9. Linear,
  10. Parameter,
  11. )
  13. tokens=vit.patch_embed(input)
  15. mask_token=torch.nn.Parameter(torch.randn(token_dim))
  17. mask_tokens=self.mask_token.repeat(batch_size, n_tokens, 1)
  19. indices_to_mask=randn(batch_size, n_tokens)
  21. n_masked_tokens=int(0.5*n_tokens)
  23. indices_to_mask=indices_to_mask.topk(
  24. k=n_masked_tokens,
  25. dim=1,
  26. )
  28. indices_to_mask=indices_to_mask.indices
  30. bitmask=zeros(batch_size, n_tokens)
  32. bitmask=bitmask.scatter(
  33. dim=1,
  34. index=indices_to_mask,
  35. value=1,
  36. )
  37. bitmask=bitmask.bool()
  39. bitmask=bitmask.unsqueeze(2)
  41. tokens= (~bitmask)*tokens+bitmask*mask_tokens
  43. tokens=tokens+vit.pos_embed[:, 1:]
  45. encoded=vit.blocks(tokens)
  47. bitmask=bitmask.squeeze(2)
  49. masked_tokens_encoded=encoded[bitmask]
  51. patch_height=patch_width=vit.patch_embed.patch_size
  53. decoder_out_dim=3*patch_height*patch_width
  54. decoder=Linear(
  55. in_features=token_dim,
  56. out_features=decoder_out_dim,
  57. )
  59. masked_patches_reconstructed=decoder(masked_tokens_encoded)
  61. # patterns tells einops how to rearrange the tensor
  62. # Its layout is as follows: 'shape_before -> shape_after'
  63. # In this case, the shape before would be batch_size X n_channels X height X width,
  64. # and the shape after would be batch_size X n_tokens X n_channels*patch_height*patch_width
  65. # However, in einops, variables that are in shape_before must be in shape_after as well and vice versa
  66. # For example, in this case, height is in shape_before but not shape_after.
  67. # Therefore, shape_before and shape_after must be restructured.
  68. # Particularly, two new variables can be introduced, n_patches_height and n_patches_width,
  69. # that say how many patches are along the height and width axes respectively.
  70. # Thus, height = n_patches_height * patch_height,
  71. # width = n_patches_width * patch_width, and 
  72. # n_tokens = n_patches_height * n_patches width
  73. # Multiplying two variables in einops is denoted by (x y).
  74. pattern= (
  75. 'batch_size n_channels (n_patches_height patch_height) (n_patches_width patch_width) -> '
  76. 'batch_size (n_patches_height n_patches_width) (n_channels patch_height patch_width)'
  77.   )
  79. # einops.rearrange is like torch.reshape
  80. # einops cannot infer patch_height and patch_width,
  81. # so they must be passed manually
  82. # patches is now of shape batch_size X n_tokens X 3*patch_height*patch_width
  83. patches=rearrange(
  84. tensor=input,
  85. pattern=pattern,
  86. patch_height=patch_height,
  87. patch_width=patch_width,
  88. )

得对应于 masked_patches_reconstructed 的patche部分，

  1. fromeinopsimport (
  2. rearrange,
  3. )
  4. fromtorchimport (
  5. randn,
  6. zeros,
  7. )
  8. fromtorch.nnimport (
  9. Linear,
  10. Parameter,
  11. )
  13. tokens=vit.patch_embed(input)
  15. mask_token=torch.nn.Parameter(torch.randn(token_dim))
  17. mask_tokens=self.mask_token.repeat(batch_size, n_tokens, 1)
  19. indices_to_mask=randn(batch_size, n_tokens)
  21. n_masked_tokens=int(0.5*n_tokens)
  23. indices_to_mask=indices_to_mask.topk(
  24. k=n_masked_tokens,
  25. dim=1,
  26. )
  28. indices_to_mask=indices_to_mask.indices
  30. bitmask=zeros(batch_size, n_tokens)
  32. bitmask=bitmask.scatter(
  33. dim=1,
  34. index=indices_to_mask,
  35. value=1,
  36. )
  37. bitmask=bitmask.bool()
  39. bitmask=bitmask.unsqueeze(2)
  41. tokens= (~bitmask)*tokens+bitmask*mask_tokens
  43. tokens=tokens+vit.pos_embed[:, 1:]
  45. encoded=vit.blocks(tokens)
  47. bitmask=bitmask.squeeze(2)
  49. masked_tokens_encoded=encoded[bitmask]
  51. patch_height=patch_width=vit.patch_embed.patch_size
  53. decoder_out_dim=3*patch_height*patch_width
  54. decoder=Linear(
  55. in_features=token_dim,
  56. out_features=decoder_out_dim,
  57. )
  59. masked_patches_reconstructed=decoder(masked_tokens_encoded)
  61. pattern= (
  62. 'batch_size n_channels (n_patches_height patch_height) (n_patches_width patch_width) -> '
  63. 'batch_size (n_patches_height n_patches_width) (n_channels patch_height patch_width)'
  64.   )
  66. patches=einops.rearrange(
  67. tensor=input,
  68. pattern=pattern,
  69. patch_height=patch_height,
  70. patch_width=patch_width,
  71. )
  73. # Similar to how masked_tokens_encoded was computed
  74. maskes_patches_original=patches[bitmask]

评估损失。

  1. fromeinopsimport (
  2. rearrange,
  3. )
  4. fromtorchimport (
  5. randn,
  6. zeros,
  7. )
  8. fromtorch.nnimport (
  9. Linear,
  10. Parameter,
  11. )
  12. fromtorch.nn.functionalimport (
  13. l1_loss,
  14. )
  16. tokens=vit.patch_embed(input)
  18. mask_token=torch.nn.Parameter(torch.randn(token_dim))
  20. mask_tokens=self.mask_token.repeat(batch_size, n_tokens, 1)
  22. indices_to_mask=randn(batch_size, n_tokens)
  24. n_masked_tokens=int(0.5*n_tokens)
  26. indices_to_mask=indices_to_mask.topk(
  27. k=n_masked_tokens,
  28. dim=1,
  29. )
  31. indices_to_mask=indices_to_mask.indices
  32. bitmask=zeros(batch_size, n_tokens)
  34. bitmask=bitmask.scatter(
  35. dim=1,
  36. index=indices_to_mask,
  37. value=1,
  38. )
  39. bitmask=bitmask.bool()
  41. bitmask=bitmask.unsqueeze(2)
  43. tokens= (~bitmask)*tokens+bitmask*mask_tokens
  45. tokens=tokens+vit.pos_embed[:, 1:]
  47. encoded=vit.blocks(tokens)
  49. bitmask=bitmask.squeeze(2)
  51. masked_tokens_encoded=encoded[bitmask]
  53. patch_height=patch_width=vit.patch_embed.patch_size
  55. decoder_out_dim=3*patch_height*patch_width
  56. decoder=Linear(
  57. in_features=token_dim,
  58. out_features=decoder_out_dim,
  59. )
  61. masked_patches_reconstructed=decoder(masked_tokens_encoded)
  63. pattern= (
  64. 'batch_size n_channels (n_patches_height patch_height) (n_patches_width patch_width) -> '
  65. 'batch_size (n_patches_height n_patches_width) (n_channels patch_height patch_width)'
  66.   )
  68. patches=einops.rearrange(
  69. tensor=input,
  70. pattern=pattern,
  71. patch_height=patch_height,
  72. patch_width=patch_width,
  73. )
  75. maskes_patches_original=patches[bitmask]
  77. # The loss is the L1 difference between 
  78. # the predicted pixel values and the ground truth,
  79. # divided by the number of masked patches
  80. loss=l1_loss(
  81. input=masked_patches_reconstructed,
  82. target=maskes_patches_original,
  83. )/n_masked_tokens

把上面的代码封装成类并增加一些辅助函数，这里就不贴了有兴趣的看下最后的源代码。然后使用的时候如下：

  1. fromtimmimport (
  2. create_model,
  3. )
  4. fromtorch.nn.functionalimport (
  5. l1_loss,
  6. )
  7. fromtorch.optimimport (
  8. AdamW,
  9. )
  11. vit=create_model(
  12. 'vit_small_patch32_224',
  13. num_classes=0,
  14. )
  15. simmim=SimMIM(
  16. vit=vit,
  17. masking_ratio=0.5,
  18. )
  19. optimizer=AdamW(
  20. params=simmim.parameters(),
  21. lr=1e-4,
  22. weight_decay=5e-2,
  23. )
  25. forepochinrange(n_epochs):
  26. forinputindataloader:
  27. n_masked_tokens, masked_patches_reconstructed, masked_patches_original=simmim(input)
  29. loss=l1_loss(
  30. input=masked_patches_reconstructed,
  31. target=maskes_patches_original,
  32. )
  33. loss/=n_masked_tokens
  34. loss.backward()
  36. optimizer.backward()
  37. optimizer.zero_grad()

上面的代码可以配置各种超参数（例如学习率，使用余弦退火，但为简单起见，此处省略）。我们的dataloader仅返回随机调整大小的裁剪、随机水平翻转和标准化的图像。

使用上面的代码，任何VIT都可以在大量未注释的数据上进行训练，并且可以很好地学习下游任务。看起来很简单吧，这也就是论文的名字sample的来源。

总结

在本文中，我们介绍 SimMIM，这是一种受掩码建模启发的强大 SSL 算法，其中一部分输入数据被掩码，模型的目标是最小化重建损失。为了更熟悉模型的运行方式我们还是用Pytorch对其进行了实现 ，这样可以帮助我们了解模型的细节。

引用：

A Simple Framework for Contrastive Learning of Visual Representations

https://arxiv.org/abs/2002.05709

Exploring Simple Siamese Representation Learning

https://arxiv.org/abs/2011.10566

SimMIM: A Simple Framework for Masked Image Modeling

https://arxiv.org/abs/2111.09886

本文代码：

https://github.com/BobMcDear/PyTorch-SimMIM

作者：Borna Ahmadzadeh

标签： 深度学习 自监督学习 神经网络
登录后发布评论

“更简单的掩码图像建模框架SimMIM介绍和PyTorch代码实现”的评论:

还没有评论
关于作者
...
Deephub
更多文章请关注公众号：Deephub-IMBA
相关阅读

FlashTokenizer: 基于C++的高性能分词引擎，速度可以提升8-15倍

计算加速技术比较分析：GPU、FPGA、ASIC、TPU与NPU的技术特性、应用场景及产业生态

标签噪声下的模型评估：如何准确评估AI模型的真实性能，提高模型性能测量的可信度

VideoMind：Chain-of-LoRA突破时间盲区让AI真正看懂长视频

9个主流GAN损失函数的数学原理和Pytorch代码实现：从经典模型到现代变体

DAPO: 面向开源大语言模型的解耦裁剪与动态采样策略优化系统

多模态AI核心技术：CLIP与SigLIP技术原理与应用进展