0
0

基于kaggle数据集的猫狗识别(超详细版本)

目录

下载kaggle数据集

kaggle猫狗识别数据集共包含25000张JPEG数据集照片，其中猫和狗的照片各占12500张。数据集大小经过压缩打包后占543MB。
数据集可以从kaggle官方网站下载，链接如下：

  1. https://www.kaggle.com/c/dogs-vs-cats/data

如果嫌官网下载麻烦，也可以从博主之前分享的百度网盘链接中直接获取：

网盘分享—博客链接，点击>>>

在下载的kaggle数据集基础上，创建一个新的小数据集，其中包含三个子集。

猫和狗的数据集：各 1000 个样本的训练集、各 500 个样本的验证集、各 500 个样本的测试集。

创建新的小数据集

下面的网络所使用的数据集不是从kaggle网站中直接下载下来的完整数据集，而是基于kaggle完整数据集的部分小数据集。

  1. import os, shutil
  2. # 下载的kaggle数据集路径
  3. original_dataset_dir = '/Users/Downloads/kaggle_original_data' 
  4. # 新的小数据集放置路径
  5. base_dir = '/Users/cats_and_dogs_small' 
  6. os.mkdir(base_dir)
  7. train_dir = os.path.join(base_dir, 'train')
  8. os.mkdir(train_dir)
  9. validation_dir = os.path.join(base_dir, 'validation')
  10. os.mkdir(validation_dir)
  11. test_dir = os.path.join(base_dir, 'test')
  12. os.mkdir(test_dir)
  14. train_cats_dir = os.path.join(train_dir, 'cats')
  15. os.mkdir(train_cats_dir)
  16. train_dogs_dir = os.path.join(train_dir, 'dogs')
  17. os.mkdir(train_dogs_dir)
  18. validation_cats_dir = os.path.join(validation_dir, 'cats')
  19. os.mkdir(validation_cats_dir)
  20. validation_dogs_dir = os.path.join(validation_dir, 'dogs')
  21. os.mkdir(validation_dogs_dir)
  22. test_cats_dir = os.path.join(test_dir, 'cats')
  23. os.mkdir(test_cats_dir)
  24. test_dogs_dir = os.path.join(test_dir, 'dogs')
  25. os.mkdir(test_dogs_dir)
  27. fnames = ['cat.{}.jpg'.format(i) for i in range(1000)]
  28. for fname in fnames:
  29.     src = os.path.join(original_dataset_dir, fname)
  30.     dst = os.path.join(train_cats_dir, fname)
  31.     shutil.copyfile(src, dst)
  33. fnames = ['cat.{}.jpg'.format(i) for i in range(1000, 1500)]
  34. for fname in fnames:
  35.     src = os.path.join(original_dataset_dir, fname)
  36.     dst = os.path.join(validation_cats_dir, fname)
  37.     shutil.copyfile(src, dst)
  39. fnames = ['cat.{}.jpg'.format(i) for i in range(1500, 2000)]
  40. for fname in fnames:
  41.      src = os.path.join(original_dataset_dir, fname)
  42.      dst = os.path.join(test_cats_dir, fname)
  43.      shutil.copyfile(src, dst)
  45. fnames = ['dog.{}.jpg'.format(i) for i in range(1000)]
  46. for fname in fnames:
  47.     src = os.path.join(original_dataset_dir, fname)
  48.     dst = os.path.join(train_dogs_dir, fname)
  49.     shutil.copyfile(src, dst)
  51. fnames = ['dog.{}.jpg'.format(i) for i in range(1000, 1500)]
  52. for fname in fnames:
  53.     src = os.path.join(original_dataset_dir, fname)
  54.     dst = os.path.join(validation_dogs_dir, fname)
  55.     shutil.copyfile(src, dst)
  57. fnames = ['dog.{}.jpg'.format(i) for i in range(1500, 2000)]
  58. for fname in fnames:
  59.     src = os.path.join(original_dataset_dir, fname)
  60.     dst = os.path.join(test_dogs_dir, fname)
  61.     shutil.copyfile(src, dst)
  63. print('total training cat images:', len(os.listdir(train_cats_dir)))
  64. print('total training dog images:', len(os.listdir(train_dogs_dir)))
  65. print('total validation cat images:', len(os.listdir(validation_cats_dir)))
  66. print('total validation dog images:', len(os.listdir(validation_dogs_dir)))
  67. print('total test cat images:', len(os.listdir(test_cats_dir)))
  68. print('total test dog images:', len(os.listdir(test_dogs_dir)))

以上程序会生成各个文件夹路径，并将对应的训练集、验证集、测试集复制进去生成新的小数据集。

以上程序输出结果如下：

  1. total training cat images: 1000
  2. total training dog images: 1000
  3. total validation cat images: 500
  4. total validation dog images: 500
  5. total test cat images: 500
  6. total test dog images: 500

构建猫狗分类的小型卷积神经网络

猫狗分类的网络架构

  1. # 网络架构
  2. from keras import layers
  3. from keras import models
  5. model = models.Sequential()
  6. model.add(layers.Conv2D(32, (3, 3), activation='relu',input_shape=(150, 150, 3)))
  7. model.add(layers.MaxPooling2D((2, 2)))
  8. model.add(layers.Conv2D(64, (3, 3), activation='relu'))
  9. model.add(layers.MaxPooling2D((2, 2)))
  10. model.add(layers.Conv2D(128, (3, 3), activation='relu'))
  11. model.add(layers.MaxPooling2D((2, 2)))
  12. model.add(layers.Conv2D(128, (3, 3), activation='relu'))
  13. model.add(layers.MaxPooling2D((2, 2)))
  14. model.add(layers.Flatten())
  15. model.add(layers.Dense(512, activation='relu'))
  16. model.add(layers.Dense(1, activation='sigmoid'))
  17. print(model.summary())

输出的特征图的维度随层变化的情况如下：

  1. Model: "sequential"
  2. _________________________________________________________________
  3. Layer (type)                 Output Shape              Param #   
  4. =================================================================
  5. conv2d (Conv2D)              (None, 148, 148, 32)      896       
  6. _________________________________________________________________
  7. max_pooling2d (MaxPooling2D) (None, 74, 74, 32)        0         
  8. _________________________________________________________________
  9. conv2d_1 (Conv2D)            (None, 72, 72, 64)        18496     
  10. _________________________________________________________________
  11. max_pooling2d_1 (MaxPooling2 (None, 36, 36, 64)        0         
  12. _________________________________________________________________
  13. conv2d_2 (Conv2D)            (None, 34, 34, 128)       73856     
  14. _________________________________________________________________
  15. max_pooling2d_2 (MaxPooling2 (None, 17, 17, 128)       0         
  16. _________________________________________________________________
  17. conv2d_3 (Conv2D)            (None, 15, 15, 128)       147584    
  18. _________________________________________________________________
  19. max_pooling2d_3 (MaxPooling2 (None, 7, 7, 128)         0         
  20. _________________________________________________________________
  21. flatten (Flatten)            (None, 6272)              0         
  22. _________________________________________________________________
  23. dense (Dense)                (None, 512)               3211776   
  24. _________________________________________________________________
  25. dense_1 (Dense)              (None, 1)                 513       
  26. =================================================================
  27. Total params: 3,453,121
  28. Trainable params: 3,453,121
  29. Non-trainable params: 0
  30. _________________________________________________________________

模型的配置

from tensorflow.keras import optimizers
model.compile(loss=‘binary_crossentropy’,
optimizer=optimizers.RMSprop(lr=1e-4),
metrics=[‘acc’])

图像的预处理

  1. # 图像预处理
  2. from keras.preprocessing.image import ImageDataGenerator
  3. train_datagen = ImageDataGenerator(rescale=1./255)
  4. test_datagen = ImageDataGenerator(rescale=1./255)
  6. # 此处改成自己的路径
  7. train_dir='D:\\0 keras shujuji\\kaggle\\modle_date\\train'
  8. validation_dir='D:\\0 keras shujuji\\kaggle\\modle_date\\validation'
  10. train_generator = train_datagen.flow_from_directory(train_dir,
  11.                                                     target_size=(150, 150),
  12.                                                     batch_size=20,
  13.                                                     class_mode='binary')
  14. validation_generator = test_datagen.flow_from_directory(validation_dir,
  15.                                                         target_size=(150, 150),
  16.                                                         batch_size=20,
  17.                                                         class_mode='binary')

利用批量生成器拟合模型

  1. history = model.fit_generator(train_generator,
  2.                               steps_per_epoch=100,
  3.                               epochs=30,
  4.                               validation_data=validation_generator,
  5.                               validation_steps=50)

##保存模型

  1. model.save('cats_and_dogs_small_1.h5')

绘制精度和损失

  1. import matplotlib.pyplot as plt
  2. acc = history.history['acc']
  3. val_acc = history.history['val_acc']
  4. loss = history.history['loss']
  5. val_loss = history.history['val_loss']
  6. epochs = range(1, len(acc) + 1)
  7. plt.plot(epochs, acc, 'bo', label='Training acc')
  8. plt.plot(epochs, val_acc, 'b', label='Validation acc')
  9. plt.title('Training and validation accuracy')
  10. plt.legend()
  11. plt.figure()
  12. plt.plot(epochs, loss, 'bo', label='Training loss')
  13. plt.plot(epochs, val_loss, 'b', label='Validation loss')
  14. plt.title('Training and validation loss')
  15. plt.legend()
  16. plt.show()

结果显示

在这里插入图片描述

把本节(构建猫狗分类的小型卷积神经网络)各个子程序结合在一起就可以显示结果了，需要修改 模型的预处理 一节中的数据集放置路径

随机增强后的训练图像显示

  1. from keras.preprocessing import image
  2. import os
  3. import matplotlib.pyplot as plt
  4. from keras.preprocessing.image import ImageDataGenerator
  6. datagen = ImageDataGenerator(
  7.  rotation_range=40,
  8.  width_shift_range=0.2,
  9.  height_shift_range=0.2,
  10.  shear_range=0.2,
  11.  zoom_range=0.2,
  12.  horizontal_flip=True,
  13.  fill_mode='nearest')
  15. # 自己的train_cats_dir数据集路径
  16. train_cats_dir='D:\\modle_date\\train\\cats'
  18. fnames = [os.path.join(train_cats_dir, fname) for fname in os.listdir(train_cats_dir)]
  19. img_path = fnames[3]
  20. img = image.load_img(img_path, target_size=(150, 150))
  21. x = image.img_to_array(img)
  22. x = x.reshape((1,) + x.shape)
  23. i = 0
  24. for batch in datagen.flow(x, batch_size=1):
  25.     plt.figure(i)
  26.     imgplot = plt.imshow(image.array_to_img(batch[0]))
  27.     i += 1
  28.     if i % 4 == 0:
  29.         break
  30. plt.show()

结果显示

在这里插入图片描述

这一节(随机增强后的训练图像显示)的代码也可以单独出结果

使用数据增强的卷积神经网络

网络架构

  1. # 网络架构
  2. from keras import layers
  3. from keras import models
  4. from tensorflow.keras import optimizers
  5. from keras.preprocessing.image import ImageDataGenerator
  7. model = models.Sequential()
  8. model.add(layers.Conv2D(32, (3, 3), activation='relu',input_shape=(150, 150, 3)))
  9. model.add(layers.MaxPooling2D((2, 2)))
  10. model.add(layers.Conv2D(64, (3, 3), activation='relu'))
  11. model.add(layers.MaxPooling2D((2, 2)))
  12. model.add(layers.Conv2D(128, (3, 3), activation='relu'))
  13. model.add(layers.MaxPooling2D((2, 2)))
  14. model.add(layers.Conv2D(128, (3, 3), activation='relu'))
  15. model.add(layers.MaxPooling2D((2, 2)))
  16. model.add(layers.Flatten())
  17. model.add(layers.Dropout(0.5))
  18. model.add(layers.Dense(512, activation='relu'))
  19. model.add(layers.Dense(1, activation='sigmoid'))

模型的编译

  1. model.compile(loss='binary_crossentropy',
  2.               optimizer=optimizers.RMSprop(lr=1e-4),
  3.               metrics=['acc'])

利用数据增强生成器重新训练网络

  1. # 自己的数据集路径
  2. train_dir='D:\\kaggle\\modle_date\\train'
  3. validation_dir='D:\\kaggle\\modle_date\\validation'
  5. train_datagen = ImageDataGenerator(rescale=1./255,
  6.                                    rotation_range=40,
  7.                                    width_shift_range=0.2,
  8.                                    height_shift_range=0.2,
  9.                                    shear_range=0.2,
  10.                                    zoom_range=0.2,
  11.                                    horizontal_flip=True,)
  12. test_datagen = ImageDataGenerator(rescale=1./255)
  14. train_generator = train_datagen.flow_from_directory(train_dir,
  15.                                                     target_size=(150, 150),
  16.                                                     batch_size=32,
  17.                                                     class_mode='binary')
  18. validation_generator = test_datagen.flow_from_directory(validation_dir,
  19.                                                         target_size=(150, 150),
  20.                                                         batch_size=32,
  21.                                                         class_mode='binary')

修改后的拟合函数

拟合函数这里改动了下，原来的steps_per_epoch=100，运行时会出错，原因是数据集量变小，结合运行错误提示，上限可以到63，因此这里改为steps_per_epoch=63；同理， validation_steps也应该随着改变，改为 validation_steps=32，以下代码已做更正。

  1. history = model.fit_generator(train_generator,
  2.                               # steps_per_epoch=100,
  3.                               steps_per_epoch=63,  #  取上限63
  4.                               epochs=100,
  5.                               validation_data=validation_generator,
  6.                               validation_steps=32) # 改为32

模型的保存

  1. model.save('cats_and_dogs_small_2.h5')

结果输出

  1. import matplotlib.pyplot as plt
  2. acc = history.history['acc']
  3. val_acc = history.history['val_acc']
  4. loss = history.history['loss']
  5. val_loss = history.history['val_loss']
  7. epochs = range(1, len(acc) + 1)
  8. plt.plot(epochs, acc, 'bo', label='Training acc')
  9. plt.plot(epochs, val_acc, 'b', label='Validation acc')
  10. plt.title('Training and validation accuracy')
  11. plt.legend()
  12. plt.figure()
  13. plt.plot(epochs, loss, 'bo', label='Training loss')
  14. plt.plot(epochs, val_loss, 'b', label='Validation loss')
  15. plt.title('Training and validation loss')
  16. plt.legend()
  17. plt.show()

结果展示

经过一百个轮次，模型训练完毕
在这里插入图片描述
模型的精度和损失图像：
在这里插入图片描述

将本节(使用数据增强的卷积神经网络)子代码汇总后就可以编译出以上结果。

代码参考：deep learning with python

标签： 人工智能 算法 深度学习
本文转载自: https://blog.csdn.net/ximu__l/article/details/125885974
版权归原作者 Dream_Bri 所有， 如有侵权，请联系我们删除。
登录后发布评论

“基于kaggle数据集的猫狗识别(超详细版本)”的评论:

还没有评论
关于作者
...
overfit同步小助手
文章同步
相关阅读

网络安全法-网络运行安全

使用selenium/drissionpage时如何阻止chrome自动跳转http到https

docker desktop 里部署的Open WebUI 管理员密码忘记了的处理方法

在ubuntu20.04中搭建onsite比赛运行环境

利用开源的低代码表单设计器FcDesigner高效管理和渲染复杂表单结构

Kafka学习笔记

【前端】浏览器输入url到页面呈现发生了什么？