8. Neural Networks and Deep Learning

8. Neural networks and deep learning¶

This week, we’ll learn about neural nets and build a model for classifying images of clothes

8.1 Fashion classification
8.1b Setting up the Environment on Saturn Cloud
8.2 TensorFlow and Keras
8.3 Pre-trained convolutional neural networks
8.4 Convolutional neural networks
8.5 Transfer learning
8.6 Adjusting the learning rate
8.7 Checkpointing
8.8 Adding more layers
8.9 Regularization and dropout
8.10 Data augmentation
8.11 Training a larger model
8.12 Using the model
8.13 Summary
8.14 Explore more
8.15 Homework

8.1 Fashion classification¶

Dataset:

Links:

https://cs231n.github.io/

We will be learning how to build a model using image recognition to classify clothing items into a choice of 10 different categories. This will be a multi-classification model. When complete this model will accept an image and then predict the class. Here are the 10 classes:

`{‘dress’: 0,

‘hat’: 1, ‘longsleeve’: 2, ‘outwear’: 3, ‘pants’: 4, ‘shirt’: 5, ‘shoes’: 6, ‘shorts’: 7, ‘skirt’: 8, ‘t-shirt’: 9}`

This will be a practical learning session, not a theoretical discussion of how neural networks operate.

8.2 TensorFlow and Keras¶

Installing TensorFlow
Loading images

Tensorflow is a library used for training deep learning models. Keras has a python frontend is used as a high level abstraction of Tensorflow. In other words it allows us to interface with Tensorflow using python. You will have to install Tensorflow on your platform. I used conda install tensorflow-gpu in order to work with my RTX3080 GPU. I first ran the code on my CPU and each epoch took approximately 50 seconds. After I setup Tensorflow to work my GPU that time was signifcantly reduced to each epoch taking approximately 10 seconds.

In [1]:

import numpy as np
import matplotlib.pyplot as plt
%matplotlib inline

In [2]:

#x = np.linspace(0, 10, 100)
#plt.plot(x, np.sin(x))
#plt.plot(x, np.cos(x))
#plt.show()

In [24]:

import tensorflow as tf 
from tensorflow import keras

Let’s use load_img to load one of our images.

In [4]:

from tensorflow.keras.preprocessing.image import load_img

In [5]:

path = './clothing-dataset-small/train/t-shirt'
name = '5f0a3fa0-6a3d-4b68-b213-72766a643de7.jpg'
fullname = f'{path}/{name}'
load_img(fullname)

Out[5]:

We will import these images at 299 x 299. Tensorflow and Keras requires specific sizes. Below, using shape we can see the image is 299 x 299 with the three color matricies of red, green and blue (RGB).

In [6]:

img = load_img(fullname, target_size=(299, 299))

In [7]:

x = np.array(img)
x.shape

Out[7]:

(299, 299, 3)

8.3 Pre-trained convolutional neural networks¶

Imagenet dataset: https://www.image-net.org/
Pre-trained models: https://keras.io/api/applications/

Important: If you rent a GPU from a cloud provider (such as AWS), don’t forget to turn it off after you finish. It’s not free and you might get a large bill at the end of the month.

Links

Renting a GPU with AWS SageMaker

Here we are going to use a pre-trained model here. The Xception model due to it’s small size, high accuracy and speed. The Xception model is a convolutional neural network.

We will import the applications.

In [25]:

from tensorflow.keras.applications.xception import Xception
from tensorflow.keras.applications.xception import preprocess_input
from tensorflow.keras.applications.xception import decode_predictions

Now we will create the model. The arguments we will use are weights='imagenet' so that our model will be pre-trained on the imagenet dataset and our input_shape of 299, 299, 3.

In [26]:

model = Xception(weights='imagenet', input_shape=(299, 299, 3))

In [27]:

X = np.array([x])

In [28]:

X.shape

Out[28]:

(1, 299, 299, 3)

We have to preprocess the data prior to pushing it to the model.

In [29]:

X = preprocess_input(X)

Now we can perform a prediction after the preprocessing.

In [30]:

pred = model.predict(X)

Now we can decode the predictions. We can see that our model has classified our image as a ‘jersey’. Imagenet doesn’t have a t-shirt category.

In [31]:

decode_predictions(pred)

Out[31]:

[[('n03595614', 'jersey', 0.67922413),
  ('n02916936', 'bulletproof_vest', 0.039596543),
  ('n04370456', 'sweatshirt', 0.035304554),
  ('n03710637', 'maillot', 0.010884605),
  ('n04525038', 'velvet', 0.0018057536)]]

8.4 Convolutional neural networks¶

Types of layers: convolutional and dense
Convolutional layers and filters
Dense layers

There are more layers. Read here: https://cs231n.github.io/

There are two main types of layers for convolultional neural networks, convolutional layers and dense layers.

Convolutional layers contain filters, which are small images containing shapes used to compare the input image with these small images. The filter images are ‘slid’ across the input image and records similarity. This creates a filter array called a feature map.

This feature map is thought of as a new image that may or may not go through a number of other convolutional layers to create greater complexity of shapes. The model will learn these shapes and create the feature maps. The output of the convolutional layers is a vector.

This vector will be the input to the dense layers which, simply explained, are matrix multiplications. There could be many dense layers and the output of the dense layers in a mutli-class model, for instance, would be a prediction of of which class our input image might belong.

8.5 Tranfser learning¶

Reading data with ImageDataGenerator
Train Xception on smaller images (150×150)

(Better to run it with a GPU)

In [32]:

from tensorflow.keras.preprocessing.image import ImageDataGenerator

We will setup a generator to train our model using the ImageDataGenerator from keras. We will use target_size of 150 X 150 to train the model faster while we adjust settings and optimize. The batch_size will be the number of images we read in at a time.

In [33]:

print(tf.__version__)

2.6.0

In [34]:

train_gen = ImageDataGenerator(preprocessing_function=preprocess_input)
train_ds = train_gen.flow_from_directory(
    './clothing-dataset-small/train',
    target_size=(150, 150),
    batch_size=32
)

Found 3068 images belonging to 10 classes.

We have 3068 images with 10 classes. Below are the classes in our data set.

In [35]:

train_ds.class_indices

Out[35]:

{'dress': 0,
 'hat': 1,
 'longsleeve': 2,
 'outwear': 3,
 'pants': 4,
 'shirt': 5,
 'shoes': 6,
 'shorts': 7,
 'skirt': 8,
 't-shirt': 9}

We can assign the train_ds to X and y using next which will show our next batch set and then we can slice into the data set to see what is contained in the first 5 rows. The one-hot encoding shows us which target class the prediction has been assigned. The first record with the 1 in the 3rd column is predicting ‘longsleeve’.

In [36]:

X, y = next(train_ds)

In [37]:

y[:5] # This shows our one-hot encoding of our `class_indices` or 'target value'.

Out[37]:

array([[0., 0., 0., 0., 1., 0., 0., 0., 0., 0.],
       [0., 1., 0., 0., 0., 0., 0., 0., 0., 0.],
       [0., 0., 0., 1., 0., 0., 0., 0., 0., 0.],
       [0., 0., 0., 0., 0., 0., 0., 1., 0., 0.],
       [0., 0., 1., 0., 0., 0., 0., 0., 0., 0.]], dtype=float32)

We need to do the same thing with our validation set

In [38]:

val_gen = ImageDataGenerator(preprocessing_function=preprocess_input)
val_ds = val_gen.flow_from_directory(
    './clothing-dataset-small/validation/',
    target_size=(150, 150),
    batch_size=32,
    shuffle=False
)

Found 341 images belonging to 10 classes.

Now we will train a model. We want to keep the convolutional layers of our pretrained Xception model but we want to drop the dense layers that are included, we do this with the include_top attribute. In a Keras diagram of CCN’s the dense layer will be the top, hence the include_top. We will create our own dense layers.

In [39]:

base_model = Xception(
    weights='imagenet',
    include_top=False, # disregard the dense layers, we will create our own
    input_shape=(150,150,3)
)
base_model.trainable = False # Freezes convolutional layers

Let’s create the new ‘top’ of our model. Our input will will go to our base model. Our inputs goes into our base_model. The output of the base_model is base, which is the input to our keras.layers. The keras.layers function will vectorize the outputs of our base_model. The vectors will be sent to the dense layer where the output is used as the input of our prediction function.

In [40]:

inputs = keras.Input(shape=(150, 150, 3))
base = base_model(inputs, training=False)
vectors = keras.layers.GlobalAveragePooling2D()(base)
outputs = keras.layers.Dense(10)(vectors)
model = keras.Model(inputs, outputs)

We can now make some predictions on our images using model.predict.

In [42]:

preds = model.predict(X)

Our prediction shape is 32 x 10. 32 is our batch size and there are 10 prediction classes.

In [43]:

preds.shape

Out[43]:

(32, 10)

To allow our model to improve we will implement an optimzier. There are many optimizers but we will use Adam, which is recommended as the default algorithm to use. We will use a loss function to validate any training improvements, specifically with CategoricalCrossentropy which is a good selection for multi-class predictions. The from_logits=True prevents us from using the activation layer and will actually use the row scores. Then we will compile the model and use accuracy as our grading metric.

In [44]:

learning_rate = 0.01
optimizer = keras.optimizers.Adam(learning_rate=learning_rate)
loss = keras.losses.CategoricalCrossentropy(from_logits=True)
model.compile(optimizer=optimizer, loss=loss, metrics=['accuracy'])

Fitting the model is the next step. The epochs number is how many times the model will cycle through the data.

In [45]:

history = model.fit(train_ds, epochs=10, validation_data=val_ds)

Epoch 1/10
96/96 [==============================] - 37s 349ms/step - loss: 1.1854 - accuracy: 0.6822 - val_loss: 0.7201 - val_accuracy: 0.7918
Epoch 2/10
96/96 [==============================] - 10s 101ms/step - loss: 0.5453 - accuracy: 0.8321 - val_loss: 1.1662 - val_accuracy: 0.7390
Epoch 3/10
96/96 [==============================] - 10s 103ms/step - loss: 0.3815 - accuracy: 0.8791 - val_loss: 0.9704 - val_accuracy: 0.7977
Epoch 4/10
96/96 [==============================] - 10s 105ms/step - loss: 0.2447 - accuracy: 0.9130 - val_loss: 0.8653 - val_accuracy: 0.8006
Epoch 5/10
96/96 [==============================] - 10s 101ms/step - loss: 0.1646 - accuracy: 0.9404 - val_loss: 0.9065 - val_accuracy: 0.7830
Epoch 6/10
96/96 [==============================] - 10s 100ms/step - loss: 0.1302 - accuracy: 0.9553 - val_loss: 0.8500 - val_accuracy: 0.8094
Epoch 7/10
96/96 [==============================] - 10s 102ms/step - loss: 0.0977 - accuracy: 0.9628 - val_loss: 1.0498 - val_accuracy: 0.7801
Epoch 8/10
96/96 [==============================] - 10s 101ms/step - loss: 0.0890 - accuracy: 0.9697 - val_loss: 1.1913 - val_accuracy: 0.7478
Epoch 9/10
96/96 [==============================] - 11s 110ms/step - loss: 0.0928 - accuracy: 0.9723 - val_loss: 1.0409 - val_accuracy: 0.8006
Epoch 10/10
96/96 [==============================] - 10s 102ms/step - loss: 0.0904 - accuracy: 0.9651 - val_loss: 1.0781 - val_accuracy: 0.7830

In [46]:

history_dict = history.history
print(history_dict.keys())

dict_keys(['loss', 'accuracy', 'val_loss', 'val_accuracy'])

In [47]:

print(history.history['val_accuracy'])

[0.7917888760566711, 0.7390029430389404, 0.7976539731025696, 0.8005865216255188, 0.7829912304878235, 0.8093841671943665, 0.7800586223602295, 0.7478005886077881, 0.8005865216255188, 0.7829912304878235]

In [48]:

# plt.plot(history.history['accuracy'], label='train')
plt.plot(history.history['val_accuracy'], label='val')
plt.xticks(np.arange(10))
plt.legend()

Out[48]:

<matplotlib.legend.Legend at 0x155f3f9ad60>

8.6 Adjusting the learning rate¶

What’s the learning rate
Trying different values

In [49]:

def make_model(learning_rate=0.01):
    base_model = Xception(
        weights='imagenet',
        include_top=False,
        input_shape=(150, 150, 3)
    )
    base_model.trainable = False
    #########################################
    inputs = keras.Input(shape=(150, 150, 3))
    base = base_model(inputs, training=False)
    vectors = keras.layers.GlobalAveragePooling2D()(base)
    outputs = keras.layers.Dense(10)(vectors)
    model = keras.Model(inputs, outputs)
    
    #########################################
    optimizer = keras.optimizers.Adam(learning_rate=learning_rate)
    loss = keras.losses.CategoricalCrossentropy(from_logits=True)
    model.compile(
        optimizer=optimizer,
        loss=loss,
        metrics=['accuracy']
    )
    
    return model

We want to iterate over different learning values.

In [50]:

scores = {}
for lr in [0.0001, 0.001, 0.01, 0.1]:
    print(lr)
    model = make_model(learning_rate=lr)
    history = model.fit(train_ds, epochs=10, validation_data=val_ds)
    scores[lr] = history.history
    print()
    print()

0.0001
Epoch 1/10
96/96 [==============================] - 14s 109ms/step - loss: 1.8433 - accuracy: 0.3921 - val_loss: 1.5253 - val_accuracy: 0.5220
Epoch 2/10
96/96 [==============================] - 10s 101ms/step - loss: 1.3516 - accuracy: 0.5799 - val_loss: 1.2166 - val_accuracy: 0.6100
Epoch 3/10
96/96 [==============================] - 10s 100ms/step - loss: 1.1255 - accuracy: 0.6477 - val_loss: 1.0468 - val_accuracy: 0.6657
Epoch 4/10
96/96 [==============================] - 10s 103ms/step - loss: 0.9951 - accuracy: 0.6969 - val_loss: 0.9410 - val_accuracy: 0.7038
Epoch 5/10
96/96 [==============================] - 10s 102ms/step - loss: 0.9043 - accuracy: 0.7203 - val_loss: 0.8692 - val_accuracy: 0.7331
Epoch 6/10
96/96 [==============================] - 10s 103ms/step - loss: 0.8393 - accuracy: 0.7396 - val_loss: 0.8210 - val_accuracy: 0.7243
Epoch 7/10
96/96 [==============================] - 10s 102ms/step - loss: 0.7871 - accuracy: 0.7552 - val_loss: 0.7857 - val_accuracy: 0.7507
Epoch 8/10
96/96 [==============================] - 10s 102ms/step - loss: 0.7461 - accuracy: 0.7630 - val_loss: 0.7512 - val_accuracy: 0.7419
Epoch 9/10
96/96 [==============================] - 10s 101ms/step - loss: 0.7107 - accuracy: 0.7722 - val_loss: 0.7316 - val_accuracy: 0.7595
Epoch 10/10
96/96 [==============================] - 10s 102ms/step - loss: 0.6804 - accuracy: 0.7823 - val_loss: 0.7041 - val_accuracy: 0.7683
0.001
Epoch 1/10
96/96 [==============================] - 14s 111ms/step - loss: 1.0661 - accuracy: 0.6366 - val_loss: 0.7051 - val_accuracy: 0.7889
Epoch 2/10
96/96 [==============================] - 10s 102ms/step - loss: 0.6282 - accuracy: 0.7846 - val_loss: 0.6064 - val_accuracy: 0.8035
Epoch 3/10
96/96 [==============================] - 10s 102ms/step - loss: 0.5053 - accuracy: 0.8331 - val_loss: 0.5763 - val_accuracy: 0.8065
Epoch 4/10
96/96 [==============================] - 10s 101ms/step - loss: 0.4292 - accuracy: 0.8618 - val_loss: 0.5667 - val_accuracy: 0.7947
Epoch 5/10
96/96 [==============================] - 10s 102ms/step - loss: 0.3730 - accuracy: 0.8885 - val_loss: 0.5253 - val_accuracy: 0.8328
Epoch 6/10
96/96 [==============================] - 10s 103ms/step - loss: 0.3263 - accuracy: 0.9035 - val_loss: 0.5388 - val_accuracy: 0.8182
Epoch 7/10
96/96 [==============================] - 10s 102ms/step - loss: 0.2914 - accuracy: 0.9179 - val_loss: 0.5218 - val_accuracy: 0.8358
Epoch 8/10
96/96 [==============================] - 10s 102ms/step - loss: 0.2592 - accuracy: 0.9345 - val_loss: 0.5179 - val_accuracy: 0.8446
Epoch 9/10
96/96 [==============================] - 10s 103ms/step - loss: 0.2363 - accuracy: 0.9364 - val_loss: 0.5096 - val_accuracy: 0.8328
Epoch 10/10
96/96 [==============================] - 10s 104ms/step - loss: 0.2093 - accuracy: 0.9537 - val_loss: 0.5283 - val_accuracy: 0.8270
0.01
Epoch 1/10
96/96 [==============================] - 14s 112ms/step - loss: 1.3478 - accuracy: 0.6473 - val_loss: 1.0464 - val_accuracy: 0.7537
Epoch 2/10
96/96 [==============================] - 10s 101ms/step - loss: 0.5010 - accuracy: 0.8364 - val_loss: 0.9191 - val_accuracy: 0.7507
Epoch 3/10
96/96 [==============================] - 10s 101ms/step - loss: 0.3317 - accuracy: 0.8823 - val_loss: 0.7713 - val_accuracy: 0.8211
Epoch 4/10
96/96 [==============================] - 10s 102ms/step - loss: 0.2124 - accuracy: 0.9257 - val_loss: 0.8813 - val_accuracy: 0.8065
Epoch 5/10
96/96 [==============================] - 10s 101ms/step - loss: 0.1611 - accuracy: 0.9456 - val_loss: 0.8175 - val_accuracy: 0.8240
Epoch 6/10
96/96 [==============================] - 10s 101ms/step - loss: 0.1194 - accuracy: 0.9553 - val_loss: 0.9177 - val_accuracy: 0.7977
Epoch 7/10
96/96 [==============================] - 10s 103ms/step - loss: 0.1169 - accuracy: 0.9615 - val_loss: 0.8941 - val_accuracy: 0.8152
Epoch 8/10
96/96 [==============================] - 10s 101ms/step - loss: 0.0928 - accuracy: 0.9707 - val_loss: 0.9890 - val_accuracy: 0.7947
Epoch 9/10
96/96 [==============================] - 10s 102ms/step - loss: 0.1256 - accuracy: 0.9589 - val_loss: 1.0005 - val_accuracy: 0.8065
Epoch 10/10
96/96 [==============================] - 10s 101ms/step - loss: 0.1101 - accuracy: 0.9658 - val_loss: 0.9735 - val_accuracy: 0.8152
0.1
Epoch 1/10
96/96 [==============================] - 14s 115ms/step - loss: 9.8948 - accuracy: 0.6392 - val_loss: 8.2893 - val_accuracy: 0.7067
Epoch 2/10
96/96 [==============================] - 10s 102ms/step - loss: 4.5691 - accuracy: 0.7806 - val_loss: 6.8975 - val_accuracy: 0.7683
Epoch 3/10
96/96 [==============================] - 10s 100ms/step - loss: 2.9672 - accuracy: 0.8462 - val_loss: 6.9654 - val_accuracy: 0.7449
Epoch 4/10
96/96 [==============================] - 10s 102ms/step - loss: 3.2524 - accuracy: 0.8494 - val_loss: 9.0911 - val_accuracy: 0.7595
Epoch 5/10
96/96 [==============================] - 9s 98ms/step - loss: 2.5422 - accuracy: 0.8768 - val_loss: 8.2455 - val_accuracy: 0.7625
Epoch 6/10
96/96 [==============================] - 9s 98ms/step - loss: 2.2939 - accuracy: 0.8875 - val_loss: 10.1055 - val_accuracy: 0.7859
Epoch 7/10
96/96 [==============================] - 10s 99ms/step - loss: 2.2133 - accuracy: 0.8986 - val_loss: 9.7912 - val_accuracy: 0.8006
Epoch 8/10
96/96 [==============================] - 10s 101ms/step - loss: 1.6303 - accuracy: 0.9156 - val_loss: 14.0856 - val_accuracy: 0.7537
Epoch 9/10
96/96 [==============================] - 9s 99ms/step - loss: 1.0831 - accuracy: 0.9420 - val_loss: 9.1315 - val_accuracy: 0.8123
Epoch 10/10
96/96 [==============================] - 9s 98ms/step - loss: 1.1607 - accuracy: 0.9371 - val_loss: 10.0731 - val_accuracy: 0.7801

In [51]:

for lr, hist in scores.items():
    #plt.plot(hist['accuracy'], label=('train=%s' % lr))
    plt.plot(hist['val_accuracy'], label=('val=%s' % lr))
plt.xticks(np.arange(10))
plt.legend()

Out[51]:

<matplotlib.legend.Legend at 0x156004a5e50>

As we can see above the learning rates for 0.0001 is very slow and the rate of 0.1 doesn’t have an accuracy as high as the other two. Let’s drop both of those and see what our chart looks like.

In [52]:

del scores[0.1]
del scores[0.0001]

In [53]:

for lr, hist in scores.items():
    plt.plot(hist['accuracy'], label=('train=%s' % lr))
    plt.plot(hist['val_accuracy'], label=('val=%s' % lr))
plt.xticks(np.arange(10))
plt.legend()

Out[53]:

<matplotlib.legend.Legend at 0x156004c1fa0>

In the graph above we can see that the learning rate of 0.01 on the training data is overfitting reacing ~ 99% accuracy. It also shows the gap between the train and the val is greater than the train and val on the learning rate of 0.001. We can see below that the 0.01 learning rate is not a very smooth line. These all indicate a bit of a flawed model on that the learning rate of 0.01 so we will choose 0.001 as our optimized learning rate.

In [54]:

for lr, hist in scores.items():
    #plt.plot(hist['accuracy'], label=('train=%s' % lr))
    plt.plot(hist['val_accuracy'], label=('val=%s' % lr))
plt.xticks(np.arange(10))
plt.legend()

Out[54]:

<matplotlib.legend.Legend at 0x156006d55e0>

0.001 will be our optimal learning rate going forward.

In [55]:

learning_rate = 0.001

8.7 Checkpointing¶

Saving the best model only
Training a model with callbacks

In [56]:

model.save_weights('model_v1.h5', save_format='h5')

In [57]:

checkpoint = keras.callbacks.ModelCheckpoint(
    'xception_v1_{epoch:02d}_{val_accuracy:.3f}.h5',
    save_best_only=True,
    monitor='val_accuracy',
    mode='max'
)

In [58]:

learning_rate = 0.001
model = make_model(learning_rate=learning_rate)
history = model.fit(
    train_ds,
    epochs=10,
    validation_data=val_ds,
    callbacks=[checkpoint]
)

Epoch 1/10
96/96 [==============================] - 13s 109ms/step - loss: 1.0942 - accuracy: 0.6186 - val_loss: 0.7296 - val_accuracy: 0.7595

C:\Users\daver\miniconda3\envs\tf_gpu\lib\site-packages\tensorflow\python\keras\utils\generic_utils.py:494: CustomMaskWarning: Custom mask layers require a config and must override get_config. When loading, the custom mask layer must be passed to the custom_objects argument.
  warnings.warn('Custom mask layers require a config and must override '

Epoch 2/10
96/96 [==============================] - 9s 98ms/step - loss: 0.6302 - accuracy: 0.7787 - val_loss: 0.6511 - val_accuracy: 0.7830
Epoch 3/10
96/96 [==============================] - 9s 98ms/step - loss: 0.5096 - accuracy: 0.8321 - val_loss: 0.5710 - val_accuracy: 0.8123
Epoch 4/10
96/96 [==============================] - 10s 99ms/step - loss: 0.4261 - accuracy: 0.8673 - val_loss: 0.5854 - val_accuracy: 0.7918
Epoch 5/10
96/96 [==============================] - 9s 98ms/step - loss: 0.3675 - accuracy: 0.8882 - val_loss: 0.5393 - val_accuracy: 0.8240
Epoch 6/10
96/96 [==============================] - 10s 102ms/step - loss: 0.3268 - accuracy: 0.9058 - val_loss: 0.5503 - val_accuracy: 0.8123
Epoch 7/10
96/96 [==============================] - 9s 98ms/step - loss: 0.2883 - accuracy: 0.9166 - val_loss: 0.5591 - val_accuracy: 0.8152
Epoch 8/10
96/96 [==============================] - 10s 100ms/step - loss: 0.2573 - accuracy: 0.9377 - val_loss: 0.5391 - val_accuracy: 0.8123
Epoch 9/10
96/96 [==============================] - 9s 97ms/step - loss: 0.2355 - accuracy: 0.9400 - val_loss: 0.5183 - val_accuracy: 0.8211
Epoch 10/10
96/96 [==============================] - 10s 105ms/step - loss: 0.2114 - accuracy: 0.9488 - val_loss: 0.5227 - val_accuracy: 0.8328

8.8 Adding more layers¶

Adding one inner dense layer
Experimenting with different sizes of inner layer

Now we will add an inner layer after the vectors after our vector representation. We will also use the relu as the activation layer on our inner layer. This will convert the outputs of the inner layer. If the values are negative then those values are set to zero, if they are positive then they are set to a linear line value.

In [59]:

def make_model(learning_rate=0.01, size_inner=100):
    base_model = Xception(
        weights='imagenet',
        include_top=False,
        input_shape=(150, 150, 3)
    )
    base_model.trainable = False
    #########################################
    inputs = keras.Input(shape=(150, 150, 3))
    base = base_model(inputs, training=False)
    vectors = keras.layers.GlobalAveragePooling2D()(base)
    
    inner = keras.layers.Dense(size_inner, activation='relu')(vectors)
    
    outputs = keras.layers.Dense(10)(inner)
    
    model = keras.Model(inputs, outputs)
    
    #########################################
    optimizer = keras.optimizers.Adam(learning_rate=learning_rate)
    loss = keras.losses.CategoricalCrossentropy(from_logits=True)
    model.compile(
        optimizer=optimizer,
        loss=loss,
        metrics=['accuracy']
    )
    
    return model

Below we are going to try a few different sizes for our inner layer.

In [60]:

learning_rate = 0.001
scores = {}
for size in [10, 100, 1000]:
    print(size)
    model = make_model(learning_rate=learning_rate, size_inner=size)
    history = model.fit(train_ds, epochs=10, validation_data=val_ds)
    scores[size] = history.history
    print()
    print()

10
Epoch 1/10
96/96 [==============================] - 14s 110ms/step - loss: 1.5467 - accuracy: 0.5075 - val_loss: 1.1570 - val_accuracy: 0.6158
Epoch 2/10
96/96 [==============================] - 10s 108ms/step - loss: 1.0343 - accuracy: 0.6649 - val_loss: 0.8854 - val_accuracy: 0.7566
Epoch 3/10
96/96 [==============================] - 10s 107ms/step - loss: 0.7476 - accuracy: 0.7614 - val_loss: 0.6964 - val_accuracy: 0.7889
Epoch 4/10
96/96 [==============================] - 10s 104ms/step - loss: 0.5903 - accuracy: 0.8093 - val_loss: 0.6480 - val_accuracy: 0.7801
Epoch 5/10
96/96 [==============================] - 10s 104ms/step - loss: 0.4974 - accuracy: 0.8364 - val_loss: 0.5932 - val_accuracy: 0.7859
Epoch 6/10
96/96 [==============================] - 10s 105ms/step - loss: 0.4166 - accuracy: 0.8713 - val_loss: 0.5700 - val_accuracy: 0.8152
Epoch 7/10
96/96 [==============================] - 10s 104ms/step - loss: 0.3606 - accuracy: 0.8898 - val_loss: 0.5719 - val_accuracy: 0.7889
Epoch 8/10
96/96 [==============================] - 10s 105ms/step - loss: 0.3142 - accuracy: 0.9097 - val_loss: 0.5782 - val_accuracy: 0.7889
Epoch 9/10
96/96 [==============================] - 10s 108ms/step - loss: 0.2778 - accuracy: 0.9211 - val_loss: 0.5600 - val_accuracy: 0.7977
Epoch 10/10
96/96 [==============================] - 10s 103ms/step - loss: 0.2415 - accuracy: 0.9351 - val_loss: 0.5197 - val_accuracy: 0.8211
100
Epoch 1/10
96/96 [==============================] - 14s 110ms/step - loss: 0.9562 - accuracy: 0.6718 - val_loss: 0.6484 - val_accuracy: 0.7830
Epoch 2/10
96/96 [==============================] - 10s 103ms/step - loss: 0.5109 - accuracy: 0.8145 - val_loss: 0.6565 - val_accuracy: 0.7859
Epoch 3/10
96/96 [==============================] - 10s 100ms/step - loss: 0.3621 - accuracy: 0.8794 - val_loss: 0.5475 - val_accuracy: 0.8152
Epoch 4/10
96/96 [==============================] - 9s 99ms/step - loss: 0.2489 - accuracy: 0.9276 - val_loss: 0.5335 - val_accuracy: 0.8123
Epoch 5/10
96/96 [==============================] - 9s 98ms/step - loss: 0.1932 - accuracy: 0.9433 - val_loss: 0.5554 - val_accuracy: 0.8270
Epoch 6/10
96/96 [==============================] - 10s 101ms/step - loss: 0.1332 - accuracy: 0.9668 - val_loss: 0.5641 - val_accuracy: 0.8240
Epoch 7/10
96/96 [==============================] - 10s 103ms/step - loss: 0.0906 - accuracy: 0.9834 - val_loss: 0.6097 - val_accuracy: 0.8152
Epoch 8/10
96/96 [==============================] - 10s 102ms/step - loss: 0.0632 - accuracy: 0.9915 - val_loss: 0.5985 - val_accuracy: 0.8182
Epoch 9/10
96/96 [==============================] - 10s 104ms/step - loss: 0.0433 - accuracy: 0.9971 - val_loss: 0.6334 - val_accuracy: 0.8182
Epoch 10/10
96/96 [==============================] - 10s 103ms/step - loss: 0.0331 - accuracy: 0.9984 - val_loss: 0.6449 - val_accuracy: 0.8182
1000
Epoch 1/10
96/96 [==============================] - 14s 118ms/step - loss: 0.9032 - accuracy: 0.6946 - val_loss: 0.6289 - val_accuracy: 0.7830
Epoch 2/10
96/96 [==============================] - 10s 106ms/step - loss: 0.4371 - accuracy: 0.8445 - val_loss: 0.6548 - val_accuracy: 0.7977
Epoch 3/10
96/96 [==============================] - 10s 103ms/step - loss: 0.2766 - accuracy: 0.9029 - val_loss: 0.6154 - val_accuracy: 0.8094
Epoch 4/10
96/96 [==============================] - 10s 108ms/step - loss: 0.1445 - accuracy: 0.9570 - val_loss: 0.6362 - val_accuracy: 0.8035
Epoch 5/10
96/96 [==============================] - 10s 105ms/step - loss: 0.0890 - accuracy: 0.9798 - val_loss: 0.7288 - val_accuracy: 0.7889
Epoch 6/10
96/96 [==============================] - 10s 106ms/step - loss: 0.0553 - accuracy: 0.9870 - val_loss: 0.7045 - val_accuracy: 0.8065
Epoch 7/10
96/96 [==============================] - 10s 104ms/step - loss: 0.0276 - accuracy: 0.9967 - val_loss: 0.6982 - val_accuracy: 0.8182
Epoch 8/10
96/96 [==============================] - 10s 106ms/step - loss: 0.0202 - accuracy: 0.9977 - val_loss: 0.6742 - val_accuracy: 0.8211
Epoch 9/10
96/96 [==============================] - 10s 103ms/step - loss: 0.0075 - accuracy: 0.9997 - val_loss: 0.6931 - val_accuracy: 0.8182
Epoch 10/10
96/96 [==============================] - 10s 105ms/step - loss: 0.0131 - accuracy: 0.9980 - val_loss: 0.6761 - val_accuracy: 0.8211

In [61]:

for size, hist in scores.items():
    plt.plot(hist['val_accuracy'], label=('val=%s' % size))
plt.xticks(np.arange(10))
plt.yticks([0.78, 0.80, 0.82, 0.825, 0.83, 0.84])
plt.legend()

Out[61]:

<matplotlib.legend.Legend at 0x1560c21cfd0>

Adding the inner layer to our model didn’t improve our accuracy any. We added complexity to the model with no benfits. Simple is better.

8.9 Regularization and dropout¶

Regularizing by freezing a part of the network
Adding dropout to our model
Experimenting with different values

Our t-shirt has a logo on it. Without regularization our model might classify images based on the logo. We want to prevent the model from paying attention to one particular detail to classify other classes to the t-shirt class. We will use regularization to remove portions of the image and train on that.

We can also use dropout to freeze ‘nodes’ of the network so that specific details will be blocked from the training.

Between dropout and regularization we can force the model to look at the overall image and not pay attention to small details, such as a logo, which could help our model be more accurate.

In [62]:

def make_model(learning_rate=0.001, size_inner=100, droprate=0.5):
    base_model = Xception(
        weights='imagenet',
        include_top=False,
        input_shape=(150, 150, 3)
    )
    base_model.trainable = False
    #########################################
    inputs = keras.Input(shape=(150, 150, 3))
    base = base_model(inputs, training=False)
    vectors = keras.layers.GlobalAveragePooling2D()(base)
    
    inner = keras.layers.Dense(size_inner, activation='relu')(vectors)
    drop = keras.layers.Dropout(droprate)(inner)
    
    outputs = keras.layers.Dense(10)(drop)
    
    model = keras.Model(inputs, outputs)
    
    #########################################
    optimizer = keras.optimizers.Adam(learning_rate=learning_rate)
    loss = keras.losses.CategoricalCrossentropy(from_logits=True)
    model.compile(
        optimizer=optimizer,
        loss=loss,
        metrics=['accuracy']
    )
    
    return model

Because we are using dropout we need to increase the number of epochs for training.

In [63]:

learning_rate = 0.001
size = 100
scores = {}
for droprate in [0.0, 0.2, 0.5, 0.8]:
    print(droprate)
    model = make_model(
        learning_rate=learning_rate,
        size_inner=size,
        droprate=droprate
    )
    history = model.fit(train_ds, epochs=30, validation_data=val_ds)
    scores[droprate] = history.history
    print()
    print()

0.0
Epoch 1/30
96/96 [==============================] - 14s 113ms/step - loss: 0.9516 - accuracy: 0.6793 - val_loss: 0.6238 - val_accuracy: 0.7977
Epoch 2/30
96/96 [==============================] - 10s 103ms/step - loss: 0.5020 - accuracy: 0.8227 - val_loss: 0.5527 - val_accuracy: 0.8094
Epoch 3/30
96/96 [==============================] - 10s 106ms/step - loss: 0.3503 - accuracy: 0.8791 - val_loss: 0.5815 - val_accuracy: 0.8094
Epoch 4/30
96/96 [==============================] - 10s 103ms/step - loss: 0.2563 - accuracy: 0.9159 - val_loss: 0.6035 - val_accuracy: 0.7977
Epoch 5/30
96/96 [==============================] - 10s 102ms/step - loss: 0.1827 - accuracy: 0.9462 - val_loss: 0.5835 - val_accuracy: 0.8006
Epoch 6/30
96/96 [==============================] - 10s 104ms/step - loss: 0.1228 - accuracy: 0.9729 - val_loss: 0.5887 - val_accuracy: 0.8299
Epoch 7/30
96/96 [==============================] - 10s 104ms/step - loss: 0.0788 - accuracy: 0.9883 - val_loss: 0.5653 - val_accuracy: 0.8182
Epoch 8/30
96/96 [==============================] - 10s 104ms/step - loss: 0.0551 - accuracy: 0.9948 - val_loss: 0.6041 - val_accuracy: 0.8065
Epoch 9/30
96/96 [==============================] - 10s 101ms/step - loss: 0.0402 - accuracy: 0.9974 - val_loss: 0.6137 - val_accuracy: 0.8123
Epoch 10/30
96/96 [==============================] - 10s 103ms/step - loss: 0.0307 - accuracy: 0.9980 - val_loss: 0.6338 - val_accuracy: 0.8094
Epoch 11/30
96/96 [==============================] - 10s 104ms/step - loss: 0.0208 - accuracy: 0.9993 - val_loss: 0.6775 - val_accuracy: 0.8152
Epoch 12/30
96/96 [==============================] - 10s 104ms/step - loss: 0.0195 - accuracy: 0.9993 - val_loss: 0.6558 - val_accuracy: 0.8152
Epoch 13/30
96/96 [==============================] - 10s 103ms/step - loss: 0.0154 - accuracy: 0.9993 - val_loss: 0.6864 - val_accuracy: 0.8123
Epoch 14/30
96/96 [==============================] - 10s 106ms/step - loss: 0.0137 - accuracy: 0.9990 - val_loss: 0.6740 - val_accuracy: 0.8065
Epoch 15/30
96/96 [==============================] - 10s 102ms/step - loss: 0.0105 - accuracy: 0.9997 - val_loss: 0.7072 - val_accuracy: 0.8152
Epoch 16/30
96/96 [==============================] - 10s 106ms/step - loss: 0.0094 - accuracy: 0.9997 - val_loss: 0.6863 - val_accuracy: 0.8123
Epoch 17/30
96/96 [==============================] - 10s 102ms/step - loss: 0.0075 - accuracy: 0.9993 - val_loss: 0.7236 - val_accuracy: 0.8299
Epoch 18/30
96/96 [==============================] - 10s 105ms/step - loss: 0.0114 - accuracy: 0.9990 - val_loss: 0.6840 - val_accuracy: 0.8299
Epoch 19/30
96/96 [==============================] - 10s 102ms/step - loss: 0.0082 - accuracy: 0.9993 - val_loss: 0.7030 - val_accuracy: 0.8152
Epoch 20/30
96/96 [==============================] - 10s 104ms/step - loss: 0.0083 - accuracy: 0.9993 - val_loss: 0.7207 - val_accuracy: 0.8240
Epoch 21/30
96/96 [==============================] - 10s 102ms/step - loss: 0.0120 - accuracy: 0.9974 - val_loss: 0.8355 - val_accuracy: 0.8035
Epoch 22/30
96/96 [==============================] - 10s 106ms/step - loss: 0.0121 - accuracy: 0.9984 - val_loss: 0.7573 - val_accuracy: 0.8240
Epoch 23/30
96/96 [==============================] - 10s 105ms/step - loss: 0.0086 - accuracy: 0.9990 - val_loss: 0.7832 - val_accuracy: 0.8182
Epoch 24/30
96/96 [==============================] - 10s 103ms/step - loss: 0.0067 - accuracy: 0.9993 - val_loss: 0.8419 - val_accuracy: 0.8006
Epoch 25/30
96/96 [==============================] - 10s 102ms/step - loss: 0.0063 - accuracy: 0.9993 - val_loss: 0.7777 - val_accuracy: 0.8182
Epoch 26/30
96/96 [==============================] - 10s 103ms/step - loss: 0.0083 - accuracy: 0.9990 - val_loss: 0.8300 - val_accuracy: 0.8123
Epoch 27/30
96/96 [==============================] - 10s 103ms/step - loss: 0.0100 - accuracy: 0.9984 - val_loss: 0.8156 - val_accuracy: 0.8211
Epoch 28/30
96/96 [==============================] - 10s 107ms/step - loss: 0.0059 - accuracy: 0.9987 - val_loss: 0.8117 - val_accuracy: 0.8240
Epoch 29/30
96/96 [==============================] - 10s 99ms/step - loss: 0.0052 - accuracy: 0.9993 - val_loss: 0.8297 - val_accuracy: 0.8094
Epoch 30/30
96/96 [==============================] - 10s 104ms/step - loss: 0.0079 - accuracy: 0.9990 - val_loss: 0.8519 - val_accuracy: 0.8152
0.2
Epoch 1/30
96/96 [==============================] - 15s 115ms/step - loss: 1.0668 - accuracy: 0.6362 - val_loss: 0.6718 - val_accuracy: 0.7742
Epoch 2/30
96/96 [==============================] - 10s 107ms/step - loss: 0.6240 - accuracy: 0.7806 - val_loss: 0.5700 - val_accuracy: 0.8211
Epoch 3/30
96/96 [==============================] - 11s 113ms/step - loss: 0.4664 - accuracy: 0.8413 - val_loss: 0.5374 - val_accuracy: 0.8094
Epoch 4/30
96/96 [==============================] - 10s 103ms/step - loss: 0.3668 - accuracy: 0.8797 - val_loss: 0.5304 - val_accuracy: 0.8387
Epoch 5/30
96/96 [==============================] - 10s 105ms/step - loss: 0.2953 - accuracy: 0.9117 - val_loss: 0.5477 - val_accuracy: 0.8123
Epoch 6/30
96/96 [==============================] - 11s 110ms/step - loss: 0.2335 - accuracy: 0.9250 - val_loss: 0.5363 - val_accuracy: 0.8182
Epoch 7/30
96/96 [==============================] - 10s 108ms/step - loss: 0.1825 - accuracy: 0.9459 - val_loss: 0.5177 - val_accuracy: 0.8211
Epoch 8/30
96/96 [==============================] - 11s 111ms/step - loss: 0.1541 - accuracy: 0.9580 - val_loss: 0.5542 - val_accuracy: 0.8270
Epoch 9/30
96/96 [==============================] - 11s 115ms/step - loss: 0.1272 - accuracy: 0.9651 - val_loss: 0.5481 - val_accuracy: 0.8328
Epoch 10/30
96/96 [==============================] - 11s 116ms/step - loss: 0.0984 - accuracy: 0.9736 - val_loss: 0.6013 - val_accuracy: 0.8270
Epoch 11/30
96/96 [==============================] - 11s 109ms/step - loss: 0.0819 - accuracy: 0.9824 - val_loss: 0.5988 - val_accuracy: 0.8328
Epoch 12/30
96/96 [==============================] - 11s 115ms/step - loss: 0.0815 - accuracy: 0.9814 - val_loss: 0.5403 - val_accuracy: 0.8387
Epoch 13/30
96/96 [==============================] - 11s 116ms/step - loss: 0.0640 - accuracy: 0.9857 - val_loss: 0.6254 - val_accuracy: 0.8299
Epoch 14/30
96/96 [==============================] - 11s 116ms/step - loss: 0.0578 - accuracy: 0.9873 - val_loss: 0.6290 - val_accuracy: 0.8211
Epoch 15/30
96/96 [==============================] - 11s 115ms/step - loss: 0.0510 - accuracy: 0.9902 - val_loss: 0.6171 - val_accuracy: 0.8358
Epoch 16/30
96/96 [==============================] - 11s 117ms/step - loss: 0.0406 - accuracy: 0.9912 - val_loss: 0.6137 - val_accuracy: 0.8270
Epoch 17/30
96/96 [==============================] - 11s 116ms/step - loss: 0.0412 - accuracy: 0.9909 - val_loss: 0.6575 - val_accuracy: 0.8328
Epoch 18/30
96/96 [==============================] - 11s 115ms/step - loss: 0.0310 - accuracy: 0.9961 - val_loss: 0.6764 - val_accuracy: 0.8211
Epoch 19/30
96/96 [==============================] - 11s 116ms/step - loss: 0.0331 - accuracy: 0.9938 - val_loss: 0.6851 - val_accuracy: 0.8123
Epoch 20/30
96/96 [==============================] - 12s 127ms/step - loss: 0.0282 - accuracy: 0.9941 - val_loss: 0.6451 - val_accuracy: 0.8270
Epoch 21/30
96/96 [==============================] - 17s 178ms/step - loss: 0.0237 - accuracy: 0.9961 - val_loss: 0.6092 - val_accuracy: 0.8387
Epoch 22/30
96/96 [==============================] - 13s 139ms/step - loss: 0.0264 - accuracy: 0.9951 - val_loss: 0.6526 - val_accuracy: 0.8211
Epoch 23/30
96/96 [==============================] - 11s 111ms/step - loss: 0.0251 - accuracy: 0.9958 - val_loss: 0.6701 - val_accuracy: 0.8299
Epoch 24/30
96/96 [==============================] - 11s 111ms/step - loss: 0.0225 - accuracy: 0.9961 - val_loss: 0.6821 - val_accuracy: 0.8211
Epoch 25/30
96/96 [==============================] - 10s 99ms/step - loss: 0.0215 - accuracy: 0.9961 - val_loss: 0.6983 - val_accuracy: 0.8328
Epoch 26/30
96/96 [==============================] - 10s 100ms/step - loss: 0.0200 - accuracy: 0.9958 - val_loss: 0.7235 - val_accuracy: 0.8270
Epoch 27/30
96/96 [==============================] - 10s 100ms/step - loss: 0.0260 - accuracy: 0.9932 - val_loss: 0.7174 - val_accuracy: 0.8240
Epoch 28/30
96/96 [==============================] - 10s 99ms/step - loss: 0.0197 - accuracy: 0.9964 - val_loss: 0.7274 - val_accuracy: 0.8416
Epoch 29/30
96/96 [==============================] - 10s 99ms/step - loss: 0.0307 - accuracy: 0.9896 - val_loss: 0.7749 - val_accuracy: 0.8123
Epoch 30/30
96/96 [==============================] - 9s 98ms/step - loss: 0.0307 - accuracy: 0.9928 - val_loss: 0.7421 - val_accuracy: 0.8270
0.5
Epoch 1/30
96/96 [==============================] - 14s 115ms/step - loss: 1.2635 - accuracy: 0.5821 - val_loss: 0.7201 - val_accuracy: 0.7947
Epoch 2/30
96/96 [==============================] - 10s 102ms/step - loss: 0.8069 - accuracy: 0.7223 - val_loss: 0.6785 - val_accuracy: 0.7742
Epoch 3/30
96/96 [==============================] - 10s 103ms/step - loss: 0.6699 - accuracy: 0.7656 - val_loss: 0.6077 - val_accuracy: 0.7859
Epoch 4/30
96/96 [==============================] - 10s 101ms/step - loss: 0.5592 - accuracy: 0.8051 - val_loss: 0.5997 - val_accuracy: 0.8152
Epoch 5/30
96/96 [==============================] - 10s 103ms/step - loss: 0.5104 - accuracy: 0.8269 - val_loss: 0.5764 - val_accuracy: 0.8006
Epoch 6/30
96/96 [==============================] - 10s 102ms/step - loss: 0.4454 - accuracy: 0.8422 - val_loss: 0.5802 - val_accuracy: 0.8006
Epoch 7/30
96/96 [==============================] - 10s 101ms/step - loss: 0.3782 - accuracy: 0.8673 - val_loss: 0.5376 - val_accuracy: 0.8299
Epoch 8/30
96/96 [==============================] - 10s 101ms/step - loss: 0.3463 - accuracy: 0.8814 - val_loss: 0.5434 - val_accuracy: 0.8240
Epoch 9/30
96/96 [==============================] - 10s 102ms/step - loss: 0.3227 - accuracy: 0.8908 - val_loss: 0.5423 - val_accuracy: 0.8270
Epoch 10/30
96/96 [==============================] - 10s 103ms/step - loss: 0.2815 - accuracy: 0.9071 - val_loss: 0.5822 - val_accuracy: 0.8182
Epoch 11/30
96/96 [==============================] - 10s 102ms/step - loss: 0.2631 - accuracy: 0.9097 - val_loss: 0.5391 - val_accuracy: 0.8270
Epoch 12/30
96/96 [==============================] - 10s 103ms/step - loss: 0.2365 - accuracy: 0.9175 - val_loss: 0.5790 - val_accuracy: 0.8182
Epoch 13/30
96/96 [==============================] - 10s 102ms/step - loss: 0.2215 - accuracy: 0.9231 - val_loss: 0.6100 - val_accuracy: 0.8152
Epoch 14/30
96/96 [==============================] - 10s 102ms/step - loss: 0.1947 - accuracy: 0.9374 - val_loss: 0.5408 - val_accuracy: 0.8504
Epoch 15/30
96/96 [==============================] - 10s 102ms/step - loss: 0.1770 - accuracy: 0.9433 - val_loss: 0.5827 - val_accuracy: 0.8240
Epoch 16/30
96/96 [==============================] - 10s 102ms/step - loss: 0.1690 - accuracy: 0.9443 - val_loss: 0.5931 - val_accuracy: 0.8270
Epoch 17/30
96/96 [==============================] - 10s 103ms/step - loss: 0.1368 - accuracy: 0.9547 - val_loss: 0.5773 - val_accuracy: 0.8328
Epoch 18/30
96/96 [==============================] - 10s 101ms/step - loss: 0.1339 - accuracy: 0.9580 - val_loss: 0.6091 - val_accuracy: 0.8240
Epoch 19/30
96/96 [==============================] - 10s 100ms/step - loss: 0.1397 - accuracy: 0.9566 - val_loss: 0.5926 - val_accuracy: 0.8299
Epoch 20/30
96/96 [==============================] - 10s 103ms/step - loss: 0.1271 - accuracy: 0.9586 - val_loss: 0.6286 - val_accuracy: 0.8182
Epoch 21/30
96/96 [==============================] - 10s 104ms/step - loss: 0.1266 - accuracy: 0.9576 - val_loss: 0.6405 - val_accuracy: 0.8182
Epoch 22/30
96/96 [==============================] - 10s 102ms/step - loss: 0.1216 - accuracy: 0.9593 - val_loss: 0.6536 - val_accuracy: 0.8211
Epoch 23/30
96/96 [==============================] - 10s 102ms/step - loss: 0.1054 - accuracy: 0.9684 - val_loss: 0.6455 - val_accuracy: 0.8270
Epoch 24/30
96/96 [==============================] - 10s 103ms/step - loss: 0.0976 - accuracy: 0.9674 - val_loss: 0.6854 - val_accuracy: 0.8211
Epoch 25/30
96/96 [==============================] - 10s 103ms/step - loss: 0.1025 - accuracy: 0.9684 - val_loss: 0.7010 - val_accuracy: 0.8240
Epoch 26/30
96/96 [==============================] - 10s 101ms/step - loss: 0.0981 - accuracy: 0.9661 - val_loss: 0.6834 - val_accuracy: 0.8240
Epoch 27/30
96/96 [==============================] - 10s 102ms/step - loss: 0.0962 - accuracy: 0.9681 - val_loss: 0.7092 - val_accuracy: 0.8182
Epoch 28/30
96/96 [==============================] - 10s 101ms/step - loss: 0.0941 - accuracy: 0.9694 - val_loss: 0.7554 - val_accuracy: 0.8270
Epoch 29/30
96/96 [==============================] - 10s 102ms/step - loss: 0.0883 - accuracy: 0.9716 - val_loss: 0.6981 - val_accuracy: 0.8270
Epoch 30/30
96/96 [==============================] - 10s 103ms/step - loss: 0.0878 - accuracy: 0.9716 - val_loss: 0.7186 - val_accuracy: 0.8211
0.8
Epoch 1/30
96/96 [==============================] - 14s 111ms/step - loss: 1.7719 - accuracy: 0.4016 - val_loss: 1.0677 - val_accuracy: 0.6598
Epoch 2/30
96/96 [==============================] - 10s 103ms/step - loss: 1.3711 - accuracy: 0.5297 - val_loss: 0.9565 - val_accuracy: 0.7097
Epoch 3/30
96/96 [==============================] - 10s 101ms/step - loss: 1.2479 - accuracy: 0.5407 - val_loss: 0.8560 - val_accuracy: 0.7126
Epoch 4/30
96/96 [==============================] - 10s 103ms/step - loss: 1.1770 - accuracy: 0.5743 - val_loss: 0.7593 - val_accuracy: 0.7625
Epoch 5/30
96/96 [==============================] - 10s 102ms/step - loss: 1.1121 - accuracy: 0.5874 - val_loss: 0.7429 - val_accuracy: 0.7683
Epoch 6/30
96/96 [==============================] - 10s 102ms/step - loss: 1.0110 - accuracy: 0.6291 - val_loss: 0.7363 - val_accuracy: 0.7625
Epoch 7/30
96/96 [==============================] - 10s 100ms/step - loss: 1.0057 - accuracy: 0.6287 - val_loss: 0.7099 - val_accuracy: 0.7771
Epoch 8/30
96/96 [==============================] - 10s 103ms/step - loss: 0.9733 - accuracy: 0.6411 - val_loss: 0.6789 - val_accuracy: 0.7625
Epoch 9/30
96/96 [==============================] - 10s 100ms/step - loss: 0.9314 - accuracy: 0.6548 - val_loss: 0.6515 - val_accuracy: 0.7801
Epoch 10/30
96/96 [==============================] - 10s 102ms/step - loss: 0.9034 - accuracy: 0.6714 - val_loss: 0.6467 - val_accuracy: 0.7830
Epoch 11/30
96/96 [==============================] - 10s 100ms/step - loss: 0.8977 - accuracy: 0.6617 - val_loss: 0.6213 - val_accuracy: 0.7742
Epoch 12/30
96/96 [==============================] - 10s 102ms/step - loss: 0.8654 - accuracy: 0.6701 - val_loss: 0.6104 - val_accuracy: 0.7889
Epoch 13/30
96/96 [==============================] - 10s 102ms/step - loss: 0.8052 - accuracy: 0.6806 - val_loss: 0.6084 - val_accuracy: 0.7830
Epoch 14/30
96/96 [==============================] - 10s 103ms/step - loss: 0.8270 - accuracy: 0.6819 - val_loss: 0.5853 - val_accuracy: 0.8035
Epoch 15/30
96/96 [==============================] - 10s 100ms/step - loss: 0.8182 - accuracy: 0.6822 - val_loss: 0.5886 - val_accuracy: 0.8094
Epoch 16/30
96/96 [==============================] - 10s 103ms/step - loss: 0.7951 - accuracy: 0.6926 - val_loss: 0.6199 - val_accuracy: 0.7977
Epoch 17/30
96/96 [==============================] - 10s 102ms/step - loss: 0.7807 - accuracy: 0.7119 - val_loss: 0.5882 - val_accuracy: 0.7977
Epoch 18/30
96/96 [==============================] - 10s 101ms/step - loss: 0.7702 - accuracy: 0.6998 - val_loss: 0.5886 - val_accuracy: 0.7977
Epoch 19/30
96/96 [==============================] - 10s 99ms/step - loss: 0.7265 - accuracy: 0.7226 - val_loss: 0.5710 - val_accuracy: 0.8211
Epoch 20/30
96/96 [==============================] - 10s 102ms/step - loss: 0.7518 - accuracy: 0.7089 - val_loss: 0.5807 - val_accuracy: 0.7918
Epoch 21/30
96/96 [==============================] - 10s 100ms/step - loss: 0.7306 - accuracy: 0.7148 - val_loss: 0.5765 - val_accuracy: 0.8094
Epoch 22/30
96/96 [==============================] - 10s 100ms/step - loss: 0.7368 - accuracy: 0.7021 - val_loss: 0.5790 - val_accuracy: 0.7947
Epoch 23/30
96/96 [==============================] - 10s 100ms/step - loss: 0.6902 - accuracy: 0.7275 - val_loss: 0.5630 - val_accuracy: 0.8065
Epoch 24/30
96/96 [==============================] - 10s 102ms/step - loss: 0.6683 - accuracy: 0.7399 - val_loss: 0.5552 - val_accuracy: 0.8065
Epoch 25/30
96/96 [==============================] - 10s 101ms/step - loss: 0.6703 - accuracy: 0.7379 - val_loss: 0.5690 - val_accuracy: 0.8152
Epoch 26/30
96/96 [==============================] - 10s 102ms/step - loss: 0.6656 - accuracy: 0.7321 - val_loss: 0.5315 - val_accuracy: 0.8182
Epoch 27/30
96/96 [==============================] - 10s 100ms/step - loss: 0.6517 - accuracy: 0.7379 - val_loss: 0.5552 - val_accuracy: 0.8123
Epoch 28/30
96/96 [==============================] - 10s 101ms/step - loss: 0.6539 - accuracy: 0.7376 - val_loss: 0.5527 - val_accuracy: 0.8123
Epoch 29/30
96/96 [==============================] - 9s 98ms/step - loss: 0.5929 - accuracy: 0.7614 - val_loss: 0.5314 - val_accuracy: 0.8211
Epoch 30/30
96/96 [==============================] - 10s 102ms/step - loss: 0.6131 - accuracy: 0.7568 - val_loss: 0.5628 - val_accuracy: 0.8152

The results were sort of a mixed bag. All of the training, but the 0.8, were almost at 100% while the scores on the validation set were pretty close to each other. We know we are dropping the 0.8 due to a slow ramp up.

In [64]:

for droprate, hist in scores.items():
    plt.plot(hist['val_accuracy'], label=('val=%s' % droprate))
plt.ylim(0.78, 0.86)
plt.legend()

Out[64]:

<matplotlib.legend.Legend at 0x1560505e820>

Looking at 0.0 and 0.2 we can see that 0.2 has a higher average, somewhere between 0.83 and 0.84. We will choose 0.2 as our dropout rate.

In [65]:

hist = scores[0.0]
plt.plot(hist['val_accuracy'], label=0.0)
hist = scores[0.2]
plt.plot(hist['val_accuracy'], label=0.2)
plt.legend()
#plt.plot(hist['accuracy'], label=('val=%s' % droprate))

Out[65]:

<matplotlib.legend.Legend at 0x153850097c0>

8.10 Data augmentation¶

Different data augmentations
Training a model with augmentations
How to select data augmentations?

With data augmentation we can create more data from existing data. We can take our original images and flip, mirror, skew and rotate them. After testing a few we didn’t really find any improvement of our model using the smaller images, so for simplicity sake we will move forward without any augmentation with the smaller images.

In [66]:

train_gen = ImageDataGenerator(
    preprocessing_function=preprocess_input,
#     vertical_flip=True,
)
train_ds = train_gen.flow_from_directory(
    './clothing-dataset-small/train',
    target_size=(150, 150),
    batch_size=32
)
val_gen = ImageDataGenerator(preprocessing_function=preprocess_input)
val_ds = val_gen.flow_from_directory(
    './clothing-dataset-small/validation',
    target_size=(150, 150),
    batch_size=32,
    shuffle=False
)

Found 3068 images belonging to 10 classes.
Found 341 images belonging to 10 classes.

In [67]:

learning_rate = 0.001
size = 100
droprate = 0.2
model = make_model(
    learning_rate=learning_rate,
    size_inner=size,
    droprate=droprate
)
history = model.fit(train_ds, epochs=50, validation_data=val_ds)

Epoch 1/50
96/96 [==============================] - 14s 111ms/step - loss: 1.0762 - accuracy: 0.6385 - val_loss: 0.6632 - val_accuracy: 0.7918
Epoch 2/50
96/96 [==============================] - 10s 103ms/step - loss: 0.6313 - accuracy: 0.7777 - val_loss: 0.6413 - val_accuracy: 0.7830
Epoch 3/50
96/96 [==============================] - 10s 101ms/step - loss: 0.4719 - accuracy: 0.8338 - val_loss: 0.5595 - val_accuracy: 0.8152
Epoch 4/50
96/96 [==============================] - 10s 101ms/step - loss: 0.3677 - accuracy: 0.8755 - val_loss: 0.5768 - val_accuracy: 0.7977
Epoch 5/50
96/96 [==============================] - 10s 100ms/step - loss: 0.2990 - accuracy: 0.9042 - val_loss: 0.5569 - val_accuracy: 0.8270
Epoch 6/50
96/96 [==============================] - 10s 102ms/step - loss: 0.2511 - accuracy: 0.9228 - val_loss: 0.5515 - val_accuracy: 0.8328
Epoch 7/50
96/96 [==============================] - 10s 100ms/step - loss: 0.2001 - accuracy: 0.9400 - val_loss: 0.5870 - val_accuracy: 0.8094
Epoch 8/50
96/96 [==============================] - 10s 100ms/step - loss: 0.1556 - accuracy: 0.9537 - val_loss: 0.5857 - val_accuracy: 0.8270
Epoch 9/50
96/96 [==============================] - 10s 101ms/step - loss: 0.1277 - accuracy: 0.9674 - val_loss: 0.5614 - val_accuracy: 0.8240
Epoch 10/50
96/96 [==============================] - 10s 101ms/step - loss: 0.1036 - accuracy: 0.9765 - val_loss: 0.5594 - val_accuracy: 0.8358
Epoch 11/50
96/96 [==============================] - 10s 99ms/step - loss: 0.0879 - accuracy: 0.9775 - val_loss: 0.5949 - val_accuracy: 0.8270
Epoch 12/50
96/96 [==============================] - 10s 100ms/step - loss: 0.0711 - accuracy: 0.9866 - val_loss: 0.6347 - val_accuracy: 0.8328
Epoch 13/50
96/96 [==============================] - 10s 99ms/step - loss: 0.0633 - accuracy: 0.9866 - val_loss: 0.5978 - val_accuracy: 0.8416
Epoch 14/50
96/96 [==============================] - 10s 100ms/step - loss: 0.0521 - accuracy: 0.9919 - val_loss: 0.6938 - val_accuracy: 0.8211
Epoch 15/50
96/96 [==============================] - 10s 100ms/step - loss: 0.0524 - accuracy: 0.9899 - val_loss: 0.6467 - val_accuracy: 0.8299
Epoch 16/50
96/96 [==============================] - 10s 105ms/step - loss: 0.0392 - accuracy: 0.9941 - val_loss: 0.6674 - val_accuracy: 0.8123
Epoch 17/50
96/96 [==============================] - 10s 103ms/step - loss: 0.0387 - accuracy: 0.9925 - val_loss: 0.6757 - val_accuracy: 0.8240
Epoch 18/50
96/96 [==============================] - 10s 102ms/step - loss: 0.0414 - accuracy: 0.9905 - val_loss: 0.6493 - val_accuracy: 0.8299
Epoch 19/50
96/96 [==============================] - 10s 100ms/step - loss: 0.0309 - accuracy: 0.9941 - val_loss: 0.6633 - val_accuracy: 0.8299
Epoch 20/50
96/96 [==============================] - 10s 103ms/step - loss: 0.0393 - accuracy: 0.9879 - val_loss: 0.6895 - val_accuracy: 0.8328
Epoch 21/50
96/96 [==============================] - 10s 102ms/step - loss: 0.0260 - accuracy: 0.9974 - val_loss: 0.6918 - val_accuracy: 0.8358
Epoch 22/50
96/96 [==============================] - 10s 105ms/step - loss: 0.0299 - accuracy: 0.9945 - val_loss: 0.6693 - val_accuracy: 0.8299
Epoch 23/50
96/96 [==============================] - 10s 101ms/step - loss: 0.0320 - accuracy: 0.9948 - val_loss: 0.7075 - val_accuracy: 0.8152
Epoch 24/50
96/96 [==============================] - 10s 103ms/step - loss: 0.0249 - accuracy: 0.9945 - val_loss: 0.6975 - val_accuracy: 0.8328
Epoch 25/50
96/96 [==============================] - 10s 100ms/step - loss: 0.0221 - accuracy: 0.9954 - val_loss: 0.7313 - val_accuracy: 0.8123
Epoch 26/50
96/96 [==============================] - 10s 102ms/step - loss: 0.0305 - accuracy: 0.9922 - val_loss: 0.7896 - val_accuracy: 0.8152
Epoch 27/50
96/96 [==============================] - 10s 100ms/step - loss: 0.0211 - accuracy: 0.9958 - val_loss: 0.7599 - val_accuracy: 0.8387
Epoch 28/50
96/96 [==============================] - 10s 103ms/step - loss: 0.0175 - accuracy: 0.9977 - val_loss: 0.7163 - val_accuracy: 0.8211
Epoch 29/50
96/96 [==============================] - 10s 100ms/step - loss: 0.0201 - accuracy: 0.9961 - val_loss: 0.8015 - val_accuracy: 0.8270
Epoch 30/50
96/96 [==============================] - 10s 102ms/step - loss: 0.0168 - accuracy: 0.9961 - val_loss: 0.7960 - val_accuracy: 0.8240
Epoch 31/50
96/96 [==============================] - 10s 100ms/step - loss: 0.0202 - accuracy: 0.9958 - val_loss: 0.7934 - val_accuracy: 0.8270
Epoch 32/50
96/96 [==============================] - 10s 102ms/step - loss: 0.0178 - accuracy: 0.9958 - val_loss: 0.8182 - val_accuracy: 0.8182
Epoch 33/50
96/96 [==============================] - 10s 99ms/step - loss: 0.0269 - accuracy: 0.9919 - val_loss: 0.8080 - val_accuracy: 0.8240
Epoch 34/50
96/96 [==============================] - 10s 101ms/step - loss: 0.0201 - accuracy: 0.9951 - val_loss: 0.8864 - val_accuracy: 0.8152
Epoch 35/50
96/96 [==============================] - 10s 100ms/step - loss: 0.0279 - accuracy: 0.9922 - val_loss: 0.7993 - val_accuracy: 0.8152
Epoch 36/50
96/96 [==============================] - 10s 102ms/step - loss: 0.0259 - accuracy: 0.9925 - val_loss: 0.8792 - val_accuracy: 0.8035
Epoch 37/50
96/96 [==============================] - 10s 100ms/step - loss: 0.0196 - accuracy: 0.9954 - val_loss: 0.9022 - val_accuracy: 0.8211
Epoch 38/50
96/96 [==============================] - 10s 102ms/step - loss: 0.0274 - accuracy: 0.9922 - val_loss: 0.9429 - val_accuracy: 0.8270
Epoch 39/50
96/96 [==============================] - 10s 100ms/step - loss: 0.0326 - accuracy: 0.9899 - val_loss: 0.9029 - val_accuracy: 0.8270
Epoch 40/50
96/96 [==============================] - 10s 101ms/step - loss: 0.0251 - accuracy: 0.9922 - val_loss: 0.9325 - val_accuracy: 0.8240
Epoch 41/50
96/96 [==============================] - 10s 99ms/step - loss: 0.0218 - accuracy: 0.9945 - val_loss: 0.8633 - val_accuracy: 0.8270
Epoch 42/50
96/96 [==============================] - 10s 100ms/step - loss: 0.0199 - accuracy: 0.9935 - val_loss: 1.0232 - val_accuracy: 0.8123
Epoch 43/50
96/96 [==============================] - 10s 100ms/step - loss: 0.0215 - accuracy: 0.9938 - val_loss: 1.0539 - val_accuracy: 0.8065
Epoch 44/50
96/96 [==============================] - 10s 101ms/step - loss: 0.0403 - accuracy: 0.9873 - val_loss: 1.0801 - val_accuracy: 0.7830
Epoch 45/50
96/96 [==============================] - 10s 100ms/step - loss: 0.0250 - accuracy: 0.9919 - val_loss: 0.9836 - val_accuracy: 0.8152
Epoch 46/50
96/96 [==============================] - 10s 101ms/step - loss: 0.0125 - accuracy: 0.9977 - val_loss: 1.0975 - val_accuracy: 0.8182
Epoch 47/50
96/96 [==============================] - 10s 99ms/step - loss: 0.0165 - accuracy: 0.9948 - val_loss: 1.0748 - val_accuracy: 0.8123
Epoch 48/50
96/96 [==============================] - 10s 100ms/step - loss: 0.0213 - accuracy: 0.9932 - val_loss: 0.9382 - val_accuracy: 0.8270
Epoch 49/50
96/96 [==============================] - 10s 99ms/step - loss: 0.0133 - accuracy: 0.9961 - val_loss: 1.1036 - val_accuracy: 0.8182
Epoch 50/50
96/96 [==============================] - 10s 102ms/step - loss: 0.0140 - accuracy: 0.9954 - val_loss: 1.0177 - val_accuracy: 0.8240

In [68]:

hist = history.history
plt.plot(hist['val_accuracy'], label='val')
plt.plot(hist['accuracy'], label='train')
plt.legend()

Out[68]:

<matplotlib.legend.Legend at 0x15610e7dcd0>

8.11 Training a larger model¶

Train a 299×299 model

In [69]:

def make_model(input_size=150, learning_rate=0.01, size_inner=100,
               droprate=0.5):
    base_model = Xception(
        weights='imagenet',
        include_top=False,
        input_shape=(input_size, input_size, 3)
    )
    base_model.trainable = False
    #########################################
    inputs = keras.Input(shape=(input_size, input_size, 3))
    base = base_model(inputs, training=False)
    vectors = keras.layers.GlobalAveragePooling2D()(base)
    
    inner = keras.layers.Dense(size_inner, activation='relu')(vectors)
    drop = keras.layers.Dropout(droprate)(inner)
    
    outputs = keras.layers.Dense(10)(drop)
    
    model = keras.Model(inputs, outputs)
    
    #########################################
    optimizer = keras.optimizers.Adam(learning_rate=learning_rate)
    loss = keras.losses.CategoricalCrossentropy(from_logits=True)
    model.compile(
        optimizer=optimizer,
        loss=loss,
        metrics=['accuracy']
    )
    
    return model

In [70]:

input_size = 299

On our larger images we will see an improvement with the augmentation. We will use shear_range, zoom_range and horizontal_flip.

In [71]:

train_gen = ImageDataGenerator(
    preprocessing_function=preprocess_input,
    shear_range=10,
    zoom_range=0.1,
    horizontal_flip=True
)
train_ds = train_gen.flow_from_directory(
    './clothing-dataset-small/train',
    target_size=(input_size, input_size),
    batch_size=32
)
val_gen = ImageDataGenerator(preprocessing_function=preprocess_input)
val_ds = train_gen.flow_from_directory(
    './clothing-dataset-small/validation',
    target_size=(input_size, input_size),
    batch_size=32,
    shuffle=False
)

Found 3068 images belonging to 10 classes.
Found 341 images belonging to 10 classes.

We will now add back in our checkpointing to save the models.

In [72]:

checkpoint = keras.callbacks.ModelCheckpoint(
    'xception_v4_1_{epoch:02d}_{val_accuracy:.3f}.h5',
    save_best_only=True,
    monitor='val_accuracy',
    mode='max'
)

In [73]:

learning_rate = 0.0005
size = 100
droprate = 0.2
model = make_model(
    input_size=input_size,
    learning_rate=learning_rate,
    size_inner=size,
    droprate=droprate
)
history = model.fit(train_ds, epochs=50, validation_data=val_ds,
                   callbacks=[checkpoint])

Epoch 1/50
96/96 [==============================] - 68s 658ms/step - loss: 1.0067 - accuracy: 0.6832 - val_loss: 0.5696 - val_accuracy: 0.8123
Epoch 2/50
96/96 [==============================] - 60s 625ms/step - loss: 0.5302 - accuracy: 0.8276 - val_loss: 0.4511 - val_accuracy: 0.8504
Epoch 3/50
96/96 [==============================] - 60s 619ms/step - loss: 0.4221 - accuracy: 0.8514 - val_loss: 0.4058 - val_accuracy: 0.8798
Epoch 4/50
96/96 [==============================] - 60s 626ms/step - loss: 0.3849 - accuracy: 0.8660 - val_loss: 0.3759 - val_accuracy: 0.8886
Epoch 5/50
96/96 [==============================] - 60s 624ms/step - loss: 0.3494 - accuracy: 0.8784 - val_loss: 0.3991 - val_accuracy: 0.8680
Epoch 6/50
96/96 [==============================] - 60s 623ms/step - loss: 0.3118 - accuracy: 0.8911 - val_loss: 0.3669 - val_accuracy: 0.8798
Epoch 7/50
96/96 [==============================] - 60s 620ms/step - loss: 0.2963 - accuracy: 0.9012 - val_loss: 0.3622 - val_accuracy: 0.8856
Epoch 8/50
96/96 [==============================] - 60s 627ms/step - loss: 0.2803 - accuracy: 0.9061 - val_loss: 0.3552 - val_accuracy: 0.8710
Epoch 9/50
96/96 [==============================] - 60s 622ms/step - loss: 0.2487 - accuracy: 0.9133 - val_loss: 0.3900 - val_accuracy: 0.8827
Epoch 10/50
96/96 [==============================] - 60s 621ms/step - loss: 0.2249 - accuracy: 0.9263 - val_loss: 0.3760 - val_accuracy: 0.8827
Epoch 11/50
96/96 [==============================] - 60s 628ms/step - loss: 0.2203 - accuracy: 0.9205 - val_loss: 0.3704 - val_accuracy: 0.8622
Epoch 12/50
96/96 [==============================] - 60s 623ms/step - loss: 0.2160 - accuracy: 0.9254 - val_loss: 0.3603 - val_accuracy: 0.8710
Epoch 13/50
96/96 [==============================] - 60s 624ms/step - loss: 0.2016 - accuracy: 0.9312 - val_loss: 0.3553 - val_accuracy: 0.8915
Epoch 14/50
96/96 [==============================] - 67s 697ms/step - loss: 0.1847 - accuracy: 0.9351 - val_loss: 0.3623 - val_accuracy: 0.8739
Epoch 15/50
96/96 [==============================] - 68s 708ms/step - loss: 0.1909 - accuracy: 0.9364 - val_loss: 0.3826 - val_accuracy: 0.8798
Epoch 16/50
96/96 [==============================] - 68s 713ms/step - loss: 0.1762 - accuracy: 0.9400 - val_loss: 0.3862 - val_accuracy: 0.8886
Epoch 17/50
96/96 [==============================] - 68s 706ms/step - loss: 0.1674 - accuracy: 0.9433 - val_loss: 0.3731 - val_accuracy: 0.8856
Epoch 18/50
96/96 [==============================] - 67s 701ms/step - loss: 0.1588 - accuracy: 0.9443 - val_loss: 0.3703 - val_accuracy: 0.8886
Epoch 19/50
96/96 [==============================] - 67s 700ms/step - loss: 0.1458 - accuracy: 0.9544 - val_loss: 0.3719 - val_accuracy: 0.8798
Epoch 20/50
96/96 [==============================] - 68s 704ms/step - loss: 0.1410 - accuracy: 0.9560 - val_loss: 0.3741 - val_accuracy: 0.8856
Epoch 21/50
96/96 [==============================] - 63s 660ms/step - loss: 0.1372 - accuracy: 0.9540 - val_loss: 0.3788 - val_accuracy: 0.8768
Epoch 22/50
96/96 [==============================] - 60s 626ms/step - loss: 0.1320 - accuracy: 0.9573 - val_loss: 0.3554 - val_accuracy: 0.8827
Epoch 23/50
96/96 [==============================] - 60s 627ms/step - loss: 0.1372 - accuracy: 0.9524 - val_loss: 0.3486 - val_accuracy: 0.8768
Epoch 24/50
96/96 [==============================] - 60s 624ms/step - loss: 0.1189 - accuracy: 0.9641 - val_loss: 0.3488 - val_accuracy: 0.9032
Epoch 25/50
96/96 [==============================] - 60s 625ms/step - loss: 0.1084 - accuracy: 0.9638 - val_loss: 0.4279 - val_accuracy: 0.8710
Epoch 26/50
96/96 [==============================] - 60s 622ms/step - loss: 0.1183 - accuracy: 0.9635 - val_loss: 0.4132 - val_accuracy: 0.8739
Epoch 27/50
96/96 [==============================] - 60s 624ms/step - loss: 0.1074 - accuracy: 0.9681 - val_loss: 0.4173 - val_accuracy: 0.8592
Epoch 28/50
96/96 [==============================] - 61s 633ms/step - loss: 0.0949 - accuracy: 0.9716 - val_loss: 0.4151 - val_accuracy: 0.8622
Epoch 29/50
96/96 [==============================] - 59s 619ms/step - loss: 0.1013 - accuracy: 0.9684 - val_loss: 0.3708 - val_accuracy: 0.8827
Epoch 30/50
96/96 [==============================] - 60s 628ms/step - loss: 0.0976 - accuracy: 0.9707 - val_loss: 0.3975 - val_accuracy: 0.8768
Epoch 31/50
96/96 [==============================] - 60s 619ms/step - loss: 0.0936 - accuracy: 0.9756 - val_loss: 0.4680 - val_accuracy: 0.8710
Epoch 32/50
96/96 [==============================] - 60s 621ms/step - loss: 0.0889 - accuracy: 0.9733 - val_loss: 0.3743 - val_accuracy: 0.8798
Epoch 33/50
96/96 [==============================] - 60s 622ms/step - loss: 0.0871 - accuracy: 0.9762 - val_loss: 0.4245 - val_accuracy: 0.8739
Epoch 34/50
96/96 [==============================] - 59s 619ms/step - loss: 0.0811 - accuracy: 0.9743 - val_loss: 0.3928 - val_accuracy: 0.8886
Epoch 35/50
96/96 [==============================] - 59s 618ms/step - loss: 0.0858 - accuracy: 0.9720 - val_loss: 0.4123 - val_accuracy: 0.8856
Epoch 36/50
96/96 [==============================] - 60s 620ms/step - loss: 0.0730 - accuracy: 0.9785 - val_loss: 0.3958 - val_accuracy: 0.8768
Epoch 37/50
96/96 [==============================] - 60s 625ms/step - loss: 0.0725 - accuracy: 0.9791 - val_loss: 0.3857 - val_accuracy: 0.8886
Epoch 38/50
96/96 [==============================] - 60s 623ms/step - loss: 0.0695 - accuracy: 0.9804 - val_loss: 0.4276 - val_accuracy: 0.8915
Epoch 39/50
96/96 [==============================] - 60s 621ms/step - loss: 0.0729 - accuracy: 0.9791 - val_loss: 0.4464 - val_accuracy: 0.9003
Epoch 40/50
96/96 [==============================] - 60s 626ms/step - loss: 0.0582 - accuracy: 0.9824 - val_loss: 0.4047 - val_accuracy: 0.8856
Epoch 41/50
96/96 [==============================] - 60s 623ms/step - loss: 0.0564 - accuracy: 0.9860 - val_loss: 0.4544 - val_accuracy: 0.8622
Epoch 42/50
96/96 [==============================] - 60s 624ms/step - loss: 0.0579 - accuracy: 0.9824 - val_loss: 0.3932 - val_accuracy: 0.8886
Epoch 43/50
96/96 [==============================] - 60s 625ms/step - loss: 0.0583 - accuracy: 0.9847 - val_loss: 0.3928 - val_accuracy: 0.8798
Epoch 44/50
96/96 [==============================] - 59s 612ms/step - loss: 0.0574 - accuracy: 0.9837 - val_loss: 0.3782 - val_accuracy: 0.8886
Epoch 45/50
96/96 [==============================] - 60s 627ms/step - loss: 0.0501 - accuracy: 0.9889 - val_loss: 0.4579 - val_accuracy: 0.8768
Epoch 46/50
96/96 [==============================] - 59s 616ms/step - loss: 0.0500 - accuracy: 0.9883 - val_loss: 0.4178 - val_accuracy: 0.9032
Epoch 47/50
96/96 [==============================] - 59s 614ms/step - loss: 0.0503 - accuracy: 0.9870 - val_loss: 0.4474 - val_accuracy: 0.8710
Epoch 48/50
96/96 [==============================] - 59s 614ms/step - loss: 0.0556 - accuracy: 0.9844 - val_loss: 0.5131 - val_accuracy: 0.8651
Epoch 49/50
96/96 [==============================] - 59s 618ms/step - loss: 0.0487 - accuracy: 0.9876 - val_loss: 0.5123 - val_accuracy: 0.8680
Epoch 50/50
96/96 [==============================] - 59s 617ms/step - loss: 0.0505 - accuracy: 0.9847 - val_loss: 0.5240 - val_accuracy: 0.8563

8.12 Using the model¶

Loading the model
Evaluating the model
Getting predictions

In [74]:

import tensorflow as tf
from tensorflow import keras

In [75]:

from tensorflow.keras.preprocessing.image import ImageDataGenerator
from tensorflow.keras.preprocessing.image import load_img
from tensorflow.keras.applications.xception import preprocess_input

In [76]:

test_gen = ImageDataGenerator(preprocessing_function=preprocess_input)
test_ds = test_gen.flow_from_directory(
    './clothing-dataset-small/test',
    target_size=(299, 299),
    batch_size=32,
    shuffle=False
)

Found 372 images belonging to 10 classes.

We can load the highest val_accuracy model using the keras.models.load_model.

In [78]:

model = keras.models.load_model('xception_v4_1_24_0.903.h5')

In [79]:

model.evaluate(test_ds)

12/12 [==============================] - 6s 399ms/step - loss: 0.2698 - accuracy: 0.9005

Out[79]:

[0.269768089056015, 0.9005376100540161]

Above we can see the 2nd number is the accuracy of the model on the test data set. The 1st number is the loss fuction.

Let’s load an image to use our model on.

In [80]:

path = 'clothing-dataset-small/test/pants/c8d21106-bbdb-4e8d-83e4-bf3d14e54c16.jpg'

In [81]:

img = load_img(path, target_size=(299, 299))
img

Out[81]:

Convert the image to a numpy array.

In [82]:

import numpy as np

In [83]:

x = np.array(img)
X = np.array([x])
X.shape

Out[83]:

(1, 299, 299, 3)

Preprocess the image.

In [84]:

X = preprocess_input(X)

Make the prediction

In [85]:

pred = model.predict(X)

Set the classes to a list.

In [86]:

classes = [
    'dress',
    'hat',
    'longsleeve',
    'outwear',
    'pants',
    'shirt',
    'shoes',
    'shorts',
    'skirt',
    't-shirt'
]

zip the two to a dictionary and show the results.

In [87]:

dict(zip(classes, pred[0]))

Out[87]:

{'dress': -4.1065955,
 'hat': -7.2213717,
 'longsleeve': -2.8130648,
 'outwear': -4.012348,
 'pants': 9.626171,
 'shirt': -4.4783163,
 'shoes': -6.654378,
 'shorts': 2.1260917,
 'skirt': -6.1750555,
 't-shirt': -5.297008}

We can see that our model predicts the image that we loaded and ran the model on it is in the pants class.

8.13 Summary¶

We can use pre-trained models for general image classification
Convolutional layers let us turn an image into a vector
Dense layers use the vector to make the predictions
Instead of training a model from scratch, we can use transfer learning and re-use already trained convolutional layers
First, train a small model (150×150) before training a big one (299×299)
Learning rate – how fast the model trians. Fast learners aren’t always best ones
We can save the best model using callbacks and checkpointing
To avoid overfitting, use dropout and augmentation

In [ ]:

8.14 Explore more¶

Add more data, e.g. Zalando, etc (ADD LINKS)
Albumentations – another way of generating augmentations
Use PyTorch or MXNet instead of TensorFlow/Keras
In addition to Xception, there are others architectures – try them

Other projects:

cats vs dogs
Hotdog vs not hotdog
Category of images