EuroSAT Land Use and Land Cover Classification using Deep Learning

In this exercise, I implemented deep learning models to solve a typical problem in satellite imaging using a benchmark dataset. Both multilayer perceptron (MLP) and convolutional neural network (CNN) are used to label the EuroSAT satellite images (27010 images) with the correct land use classifications. This notebook consists of several sections that train the EuroSAT images with increasingly more complex models, from the single-layer neural net to transfer learning of VGG16. The accuracies improved from 16.8% to 88.8%.

S1:

  • Visit the EuroSAT data description page and download the data: https://github.com/phelber/eurosat

  • Split the data into training (50%) and testing sets (50%), stratified on class labels (equal percentage of each class type in train and test sets).

In [0]:
from google.colab import drive
drive.mount('/content/drive')
Go to this URL in a browser: https://accounts.google.com/o/oauth2/auth?client_id=947318989803-6bn6qk8qdgf4n4g3pfee6491hc0brc4i.apps.googleusercontent.com&redirect_uri=urn%3aietf%3awg%3aoauth%3a2.0%3aoob&response_type=code&scope=email%20https%3a%2f%2fwww.googleapis.com%2fauth%2fdocs.test%20https%3a%2f%2fwww.googleapis.com%2fauth%2fdrive%20https%3a%2f%2fwww.googleapis.com%2fauth%2fdrive.photos.readonly%20https%3a%2f%2fwww.googleapis.com%2fauth%2fpeopleapi.readonly

Enter your authorization code:
··········
Mounted at /content/drive
In [0]:
from __future__ import print_function
import pandas as pd
import numpy as np
import os
from tqdm import tqdm

import keras
from keras.datasets import mnist
from keras.models import Sequential
from keras.layers import Dense, Dropout, Flatten, Conv2D, MaxPooling2D, Input, GlobalAveragePooling2D
from keras.optimizers import RMSprop, Adam, SGD, Adadelta
from keras.utils import plot_model
from keras import backend as K

import matplotlib.image as mpimg
import matplotlib.pyplot as plt
from matplotlib.pyplot import imread, imshow, subplots, show

from skimage.color import rgb2gray
from sklearn.metrics import accuracy_score, confusion_matrix, roc_auc_score
from sklearn.preprocessing import StandardScaler
from sklearn.preprocessing import MinMaxScaler
from sklearn.decomposition import PCA
from sklearn.model_selection import train_test_split, StratifiedShuffleSplit, StratifiedKFold, cross_val_score, cross_validate, cross_val_predict
Using TensorFlow backend.
In [0]:
# reading the data
import glob 
import os 
imgdir='/content/drive/My Drive/Colab Notebooks/MUSA650/EuroSAT' 
imgfiles = [] 
for file in glob.glob(imgdir + os.sep + "*" + os.sep + "*.jpg"):
  imgfiles.append(file)
In [0]:
# plot the first img
tmpimg = imread(imgfiles[0])
imshow(tmpimg)
show()
In [0]:
### TAKES A LONG TIME TO RUN, AVOID RE-RUN !!! ###
eurosat = []
eurosat_label = []
for row in imgfiles:
  tmpimg = imread(row)
  gray_img = rgb2gray(tmpimg).reshape(1, 4096).squeeze().flatten() #flatten and convert to gray scale 
  eurosat.append(gray_img) #append to the feature data set
  label = row.split("/")[7]
  eurosat_label.append(label) #append to the label list
In [0]:
# flatten the list of arrays into matrix
eurosat_f = np.vstack(eurosat)
eurosat_label_f = np.vstack(eurosat_label).reshape(27010,)
In [0]:
# change label to integers
dictionary = {"AnnualCrop":0, "Forest":1, "HerbaceousVegetation":2, "Highway":3, 
              "Industrial":4, "Pasture":5, "PermanentCrop":6, "Residential":7, "River":8, "SeaLake":9} 
eurosat_label_f = [dictionary[letter] for letter in eurosat_label_f]
In [0]:
eurosat_label_f = np.vstack(eurosat_label_f).reshape(27010,)
In [0]:
# split into 50%/50% training and testing set
X_train, X_test, y_train, y_test = train_test_split(eurosat_f, eurosat_label_f, stratify = eurosat_label_f, train_size = 0.5, random_state= 10)
In [0]:
X_train.shape
Out[0]:
(13505, 4096)
In [0]:
y_train.shape
Out[0]:
(13505,)

S2:

  • Convert each RBG image to grayscale and flatten the images into a data matrix (n x p: n = #samples, p = #pixels in each image)

  • Implement a first deep learning model (M.1) using a fully connected network with a single fully connected layer (i.e: input layer + fully connected layer as the output layer).

  • Q1: Calculate classification accuracy on the test data.

    16.8%

In [0]:
# first step done in S1
In [0]:
# set global params
batch_size = 1200
num_classes = 10
epochs = 10
In [0]:
# convert class vectors to binary class matrices
y_train = keras.utils.to_categorical(y_train, num_classes)
y_test = keras.utils.to_categorical(y_test, num_classes)
In [0]:
## First deep learning model M.1
model = Sequential()
model.add(Dense(num_classes, activation='softmax', input_shape=(4096,)))

model.summary()
Model: "sequential_21"
_________________________________________________________________
Layer (type)                 Output Shape              Param #   
=================================================================
dense_36 (Dense)             (None, 10)                40970     
=================================================================
Total params: 40,970
Trainable params: 40,970
Non-trainable params: 0
_________________________________________________________________
In [0]:
# visualize model
plot_model(model, show_shapes=True, show_layer_names=True)
Out[0]:
In [0]:
# complie the model
model.compile(loss='categorical_crossentropy',
              optimizer=Adam(),
              metrics=['accuracy'])
In [0]:
# fit the model
history = model.fit(X_train, y_train,
                    batch_size=batch_size,
                    epochs=epochs,
                    verbose=1,
                    validation_data=(X_test, y_test))
Train on 13505 samples, validate on 13505 samples
Epoch 1/10
13505/13505 [==============================] - 1s 72us/step - loss: 2.4170 - accuracy: 0.1325 - val_loss: 2.2695 - val_accuracy: 0.1616
Epoch 2/10
13505/13505 [==============================] - 1s 66us/step - loss: 2.2518 - accuracy: 0.1417 - val_loss: 2.2592 - val_accuracy: 0.1340
Epoch 3/10
13505/13505 [==============================] - 1s 67us/step - loss: 2.2150 - accuracy: 0.1460 - val_loss: 2.2182 - val_accuracy: 0.1526
Epoch 4/10
13505/13505 [==============================] - 1s 68us/step - loss: 2.1999 - accuracy: 0.1544 - val_loss: 2.2043 - val_accuracy: 0.1644
Epoch 5/10
13505/13505 [==============================] - 1s 72us/step - loss: 2.1919 - accuracy: 0.1563 - val_loss: 2.2033 - val_accuracy: 0.1541
Epoch 6/10
13505/13505 [==============================] - 1s 72us/step - loss: 2.1920 - accuracy: 0.1630 - val_loss: 2.2058 - val_accuracy: 0.1463
Epoch 7/10
13505/13505 [==============================] - 1s 72us/step - loss: 2.1889 - accuracy: 0.1579 - val_loss: 2.2065 - val_accuracy: 0.1580
Epoch 8/10
13505/13505 [==============================] - 1s 69us/step - loss: 2.1842 - accuracy: 0.1669 - val_loss: 2.1919 - val_accuracy: 0.1466
Epoch 9/10
13505/13505 [==============================] - 1s 72us/step - loss: 2.1741 - accuracy: 0.1642 - val_loss: 2.1926 - val_accuracy: 0.1740
Epoch 10/10
13505/13505 [==============================] - 1s 70us/step - loss: 2.1776 - accuracy: 0.1656 - val_loss: 2.1949 - val_accuracy: 0.1681
In [0]:
# evaluate the model
score = model.evaluate(X_test, y_test, verbose=0)
print('Test loss:', score[0])
print('Test accuracy:', score[1])
Test loss: 2.1948902905141807
Test accuracy: 0.16808588802814484

S3:

  • Implement a second deep learning model (M.2) adding an additional fully connected hidden layer (with an arbitrary number of nodes) to the previous model.

  • Q1: Calculate classification accuracy on the test data.

21.2%

In [0]:
## Second deep learning model M.2
model = Sequential()
model.add(Dense(800, activation='relu', input_shape=(4096,)))
model.add(Dense(num_classes, activation='softmax'))

model.summary()
Model: "sequential_22"
_________________________________________________________________
Layer (type)                 Output Shape              Param #   
=================================================================
dense_37 (Dense)             (None, 800)               3277600   
_________________________________________________________________
dense_38 (Dense)             (None, 10)                8010      
=================================================================
Total params: 3,285,610
Trainable params: 3,285,610
Non-trainable params: 0
_________________________________________________________________
In [0]:
# visualize model
plot_model(model, show_shapes=True, show_layer_names=True)
Out[0]:
In [0]:
# complie the model
model.compile(loss='categorical_crossentropy',
              optimizer=Adam(),
              metrics=['accuracy'])
In [0]:
# fit the model
history = model.fit(X_train, y_train,
                    batch_size=batch_size,
                    epochs=15,
                    verbose=1,
                    validation_data=(X_test, y_test))
Train on 13505 samples, validate on 13505 samples
Epoch 1/15
13505/13505 [==============================] - 5s 383us/step - loss: 2.1570 - accuracy: 0.1747 - val_loss: 2.1712 - val_accuracy: 0.1728
Epoch 2/15
13505/13505 [==============================] - 5s 399us/step - loss: 2.1628 - accuracy: 0.1697 - val_loss: 2.1936 - val_accuracy: 0.1496
Epoch 3/15
13505/13505 [==============================] - 6s 472us/step - loss: 2.1670 - accuracy: 0.1751 - val_loss: 2.1644 - val_accuracy: 0.1870
Epoch 4/15
13505/13505 [==============================] - 5s 378us/step - loss: 2.1468 - accuracy: 0.1796 - val_loss: 2.1707 - val_accuracy: 0.2110
Epoch 5/15
13505/13505 [==============================] - 5s 375us/step - loss: 2.1433 - accuracy: 0.1973 - val_loss: 2.1695 - val_accuracy: 0.1584
Epoch 6/15
13505/13505 [==============================] - 5s 380us/step - loss: 2.1436 - accuracy: 0.1890 - val_loss: 2.1839 - val_accuracy: 0.2125
Epoch 7/15
13505/13505 [==============================] - 5s 374us/step - loss: 2.1508 - accuracy: 0.2133 - val_loss: 2.1750 - val_accuracy: 0.1498
Epoch 8/15
13505/13505 [==============================] - 5s 379us/step - loss: 2.1454 - accuracy: 0.2027 - val_loss: 2.1639 - val_accuracy: 0.1539
Epoch 9/15
13505/13505 [==============================] - 5s 379us/step - loss: 2.1342 - accuracy: 0.1989 - val_loss: 2.1617 - val_accuracy: 0.1956
Epoch 10/15
13505/13505 [==============================] - 5s 383us/step - loss: 2.1245 - accuracy: 0.2087 - val_loss: 2.1498 - val_accuracy: 0.1899
Epoch 11/15
13505/13505 [==============================] - 5s 381us/step - loss: 2.1197 - accuracy: 0.2129 - val_loss: 2.1442 - val_accuracy: 0.2007
Epoch 12/15
13505/13505 [==============================] - 5s 383us/step - loss: 2.1211 - accuracy: 0.2101 - val_loss: 2.1657 - val_accuracy: 0.1849
Epoch 13/15
13505/13505 [==============================] - 5s 381us/step - loss: 2.1338 - accuracy: 0.2161 - val_loss: 2.1538 - val_accuracy: 0.1528
Epoch 14/15
13505/13505 [==============================] - 5s 385us/step - loss: 2.1270 - accuracy: 0.2161 - val_loss: 2.1515 - val_accuracy: 0.2173
Epoch 15/15
13505/13505 [==============================] - 5s 390us/step - loss: 2.1115 - accuracy: 0.2267 - val_loss: 2.1401 - val_accuracy: 0.2119
In [0]:
# evaluate the model
score = model.evaluate(X_test, y_test, verbose=0)
print('Test loss:', score[0])
print('Test accuracy:', score[1])
Test loss: 2.1400763732333927
Test accuracy: 0.21192151308059692

S4:

  • Implement a third deep learning model (M.3) adding two additional fully connected hidden layers (with arbitrary number of nodes) as well as drop-out layers to the previous model.

  • Q1: Calculate classification accuracy on the test data.

    28.0%

  • Q2: Compare against previous models. Which model was the "best"? Why?

The M.3 model was the "best". It has the highest accuracy with only 10 epochs. The multiple layers of neurons as well as the drop outs make the model more robust and avoid overfitting.

In [0]:
## Third deep learning model M.3
model = Sequential()
model.add(Dense(2700, activation='relu', input_shape=(4096,)))
model.add(Dropout(0.2))
model.add(Dense(1800, activation='relu'))
model.add(Dropout(0.2))
model.add(Dense(800, activation='relu'))
model.add(Dropout(0.2))
model.add(Dense(num_classes, activation='softmax'))

model.summary()
Model: "sequential_23"
_________________________________________________________________
Layer (type)                 Output Shape              Param #   
=================================================================
dense_39 (Dense)             (None, 2700)              11061900  
_________________________________________________________________
dropout_4 (Dropout)          (None, 2700)              0         
_________________________________________________________________
dense_40 (Dense)             (None, 1800)              4861800   
_________________________________________________________________
dropout_5 (Dropout)          (None, 1800)              0         
_________________________________________________________________
dense_41 (Dense)             (None, 800)               1440800   
_________________________________________________________________
dropout_6 (Dropout)          (None, 800)               0         
_________________________________________________________________
dense_42 (Dense)             (None, 10)                8010      
=================================================================
Total params: 17,372,510
Trainable params: 17,372,510
Non-trainable params: 0
_________________________________________________________________
In [0]:
# visualize model
plot_model(model, show_shapes=True, show_layer_names=True)
Out[0]:
In [0]:
# complie the model
model.compile(loss='categorical_crossentropy',
              optimizer=Adam(),
              metrics=['accuracy'])
In [0]:
# fit the model
history = model.fit(X_train, y_train,
                    batch_size=batch_size,
                    epochs=epochs,
                    verbose=1,
                    validation_data=(X_test, y_test))
Train on 13505 samples, validate on 13505 samples
Epoch 1/10
13505/13505 [==============================] - 29s 2ms/step - loss: 4.9121 - accuracy: 0.1044 - val_loss: 2.2943 - val_accuracy: 0.1114
Epoch 2/10
13505/13505 [==============================] - 29s 2ms/step - loss: 2.2971 - accuracy: 0.1053 - val_loss: 2.2734 - val_accuracy: 0.1114
Epoch 3/10
13505/13505 [==============================] - 29s 2ms/step - loss: 2.2779 - accuracy: 0.1091 - val_loss: 2.2678 - val_accuracy: 0.1117
Epoch 4/10
13505/13505 [==============================] - 29s 2ms/step - loss: 2.2677 - accuracy: 0.1183 - val_loss: 2.2520 - val_accuracy: 0.1468
Epoch 5/10
13505/13505 [==============================] - 29s 2ms/step - loss: 2.2452 - accuracy: 0.1338 - val_loss: 2.2111 - val_accuracy: 0.1456
Epoch 6/10
13505/13505 [==============================] - 29s 2ms/step - loss: 2.2167 - accuracy: 0.1392 - val_loss: 2.1806 - val_accuracy: 0.1490
Epoch 7/10
13505/13505 [==============================] - 29s 2ms/step - loss: 2.1911 - accuracy: 0.1593 - val_loss: 2.1801 - val_accuracy: 0.1452
Epoch 8/10
13505/13505 [==============================] - 29s 2ms/step - loss: 2.1645 - accuracy: 0.1815 - val_loss: 2.1342 - val_accuracy: 0.2041
Epoch 9/10
13505/13505 [==============================] - 29s 2ms/step - loss: 2.1275 - accuracy: 0.2205 - val_loss: 2.0795 - val_accuracy: 0.2798
Epoch 10/10
13505/13505 [==============================] - 29s 2ms/step - loss: 2.0737 - accuracy: 0.2332 - val_loss: 1.9996 - val_accuracy: 0.2803
In [0]:
# evaluate the model
score = model.evaluate(X_test, y_test, verbose=0)
print('Test loss:', score[0])
print('Test accuracy:', score[1])
Test loss: 1.9996189414814904
Test accuracy: 0.28034061193466187

S5:

  • Using RGB images (without vectorizing them), implement a fourth CNN model (M.4) that includes the following layers: Conv2D, MaxPooling2D, Dropout, Flatten, Dense.

  • Q1: Calculate classification accuracy on the test data.

50.3%

  • Q2: Compare against previous models. Which model was the "best"? Why?

This model is much better than the previous models. This is because the inclusion of the convolutional layers that captures the image relational features.

In [0]:
# load RGB images
RGB_eurosat = []
for row in imgfiles:
  tmpimg = imread(row)
  RGB_eurosat.append(tmpimg) #append to the feature data set
In [0]:
# list to matrix
RGB_eurosat_f = np.stack(RGB_eurosat)
In [0]:
input_shape = (64, 64, 3)
In [0]:
# split into 50%/50% training and testing set
X_train, X_test, y_train, y_test = train_test_split(RGB_eurosat_f, eurosat_label_f, stratify = eurosat_label_f, train_size = 0.5, random_state= 10)
In [0]:
X_train.shape
Out[0]:
(13505, 64, 64, 3)
In [0]:
# convert class vectors to binary class matrices
y_train = keras.utils.to_categorical(y_train, num_classes)
y_test = keras.utils.to_categorical(y_test, num_classes)
In [0]:
## Fourth deep learning model M.4
model = Sequential()
model.add(Conv2D(64, kernel_size=(3, 3),
                 activation='relu',
                 input_shape=input_shape, 
                 data_format='channels_last', padding = "same"))
model.add(MaxPooling2D(pool_size=(3, 3), padding = "same"))
model.add(Dropout(0.25))
model.add(Flatten())
model.add(Dense(num_classes, activation='softmax'))
In [0]:
# visualize model
plot_model(model, show_shapes=True, show_layer_names=True)
Out[0]:
In [0]:
# complie the model
model.compile(loss='categorical_crossentropy',
              optimizer=Adam(),
              metrics=['accuracy'])
In [0]:
# fit the model
history = model.fit(X_train, y_train,
                    batch_size=batch_size,
                    epochs=epochs,
                    verbose=1,
                    validation_data=(X_test, y_test))
Train on 13505 samples, validate on 13505 samples
Epoch 1/10
13505/13505 [==============================] - 58s 4ms/step - loss: 340.2240 - accuracy: 0.1228 - val_loss: 187.7995 - val_accuracy: 0.0926
Epoch 2/10
13505/13505 [==============================] - 55s 4ms/step - loss: 88.2401 - accuracy: 0.2013 - val_loss: 31.6399 - val_accuracy: 0.3569
Epoch 3/10
13505/13505 [==============================] - 54s 4ms/step - loss: 14.9956 - accuracy: 0.3054 - val_loss: 5.2373 - val_accuracy: 0.3204
Epoch 4/10
13505/13505 [==============================] - 54s 4ms/step - loss: 3.5793 - accuracy: 0.3022 - val_loss: 1.9519 - val_accuracy: 0.3570
Epoch 5/10
13505/13505 [==============================] - 53s 4ms/step - loss: 2.2431 - accuracy: 0.2994 - val_loss: 1.6913 - val_accuracy: 0.4004
Epoch 6/10
13505/13505 [==============================] - 53s 4ms/step - loss: 1.8884 - accuracy: 0.3648 - val_loss: 1.5904 - val_accuracy: 0.4284
Epoch 7/10
13505/13505 [==============================] - 53s 4ms/step - loss: 1.7220 - accuracy: 0.3947 - val_loss: 1.5322 - val_accuracy: 0.4692
Epoch 8/10
13505/13505 [==============================] - 53s 4ms/step - loss: 1.6130 - accuracy: 0.4290 - val_loss: 1.4749 - val_accuracy: 0.4856
Epoch 9/10
13505/13505 [==============================] - 53s 4ms/step - loss: 1.5095 - accuracy: 0.4654 - val_loss: 1.4318 - val_accuracy: 0.4990
Epoch 10/10
13505/13505 [==============================] - 53s 4ms/step - loss: 1.4226 - accuracy: 0.4947 - val_loss: 1.3996 - val_accuracy: 0.5031
In [0]:
# evaluate the model
score = model.evaluate(X_test, y_test, verbose=0)
print('Test loss:', score[0])
print('Test accuracy:', score[1])
Test loss: 1.3995577849974592
Test accuracy: 0.5030729174613953

S6:

  • Using RGB images (without vectorizing them), implement a fifth deep learning model (M.5) targeting accuracy that will outperform all previous models. You are free to use any tools and techniques, as well as pre-trained models for transfer learning.

  • Q1: Describe the model you built, and why you chose it.

    The first trial uses multiple convolutional layers, as well as max pooling, dense, and drop out layers. A second trial was done with transfer learning with the VGG16 model which gives the best model for this section. This model relies on the pre-trained VGG16 model weights. The architecture here uses the VGG16 model weights with variation in the output class numbers. VGG16 was chosen because it has trained on a large dataset with multiple kinds of images, which means it captures the basic image features(filters) well.

  • Q2: Calculate classification accuracy on the test data.

    88.8%

  • Q3: Compare against previous models. Which model was the "best"? Why?

    The model using VGG16 weights outperforms all the previous model. This is because VGG16 was trained on a large dataset with an neural net architecture that has tested to be good at capturing the image features.

  • Q4: What are the two classes with the highest labeling error? Explain using data and showing mis-classified examples.

    Class 6(PermanentCrop) is prone to be misclassified as class 2(HerbaceousVegetation), and class 8(River) is prone to be misclassfied as class 3(Highway).

In [0]:
## Fifth deep learning model M.5 trial 1
model = Sequential()
model.add(Conv2D(64, kernel_size=(3, 3),
                 activation='relu',
                 input_shape=input_shape, 
                 data_format='channels_last', padding = "same"))
model.add(Conv2D(64, kernel_size=(3, 3),
                 activation='relu',
                 data_format='channels_last', padding = "same"))
model.add(MaxPooling2D(pool_size=(3, 3), padding = "same"))
model.add(Conv2D(128, kernel_size=(3, 3),
                 activation='relu',
                 data_format='channels_last', padding = "same"))
model.add(MaxPooling2D(pool_size=(3, 3), padding = "same"))
model.add(Dropout(0.25))
model.add(Flatten())
model.add(Dense(800, activation='relu'))
model.add(Dropout(0.25))
model.add(Dense(num_classes, activation='softmax'))
In [0]:
# visualize model
plot_model(model, show_shapes=True, show_layer_names=True)
Out[0]:
In [0]:
# complie the model
model.compile(loss='categorical_crossentropy',
              optimizer=Adam(),
              metrics=['accuracy'])
In [0]:
# fit the model
history = model.fit(X_train, y_train,
                    batch_size=batch_size,
                    epochs=epochs,
                    verbose=1,
                    validation_data=(X_test, y_test))
Train on 13505 samples, validate on 13505 samples
Epoch 1/10
13505/13505 [==============================] - 501s 37ms/step - loss: 27.3033 - accuracy: 0.1719 - val_loss: 1.9956 - val_accuracy: 0.3362
Epoch 2/10
13505/13505 [==============================] - 489s 36ms/step - loss: 1.7079 - accuracy: 0.3757 - val_loss: 1.4062 - val_accuracy: 0.4943
Epoch 3/10
13505/13505 [==============================] - 492s 36ms/step - loss: 1.3455 - accuracy: 0.5183 - val_loss: 1.5257 - val_accuracy: 0.4529
Epoch 4/10
13505/13505 [==============================] - 489s 36ms/step - loss: 1.2705 - accuracy: 0.5551 - val_loss: 1.1560 - val_accuracy: 0.5882
Epoch 5/10
13505/13505 [==============================] - 492s 36ms/step - loss: 1.0888 - accuracy: 0.6136 - val_loss: 0.9816 - val_accuracy: 0.6418
Epoch 6/10
13505/13505 [==============================] - 489s 36ms/step - loss: 1.3437 - accuracy: 0.5312 - val_loss: 1.1140 - val_accuracy: 0.5964
Epoch 7/10
13505/13505 [==============================] - 490s 36ms/step - loss: 1.0573 - accuracy: 0.6184 - val_loss: 0.9767 - val_accuracy: 0.6466
Epoch 8/10
13505/13505 [==============================] - 496s 37ms/step - loss: 1.1360 - accuracy: 0.6017 - val_loss: 1.1099 - val_accuracy: 0.6100
Epoch 9/10
13505/13505 [==============================] - 500s 37ms/step - loss: 0.9952 - accuracy: 0.6514 - val_loss: 0.8773 - val_accuracy: 0.6801
Epoch 10/10
13505/13505 [==============================] - 502s 37ms/step - loss: 0.8612 - accuracy: 0.6899 - val_loss: 0.8139 - val_accuracy: 0.7115
In [0]:
# evaluate the model
score = model.evaluate(X_test, y_test, verbose=0)
print('Test loss:', score[0])
print('Test accuracy:', score[1])
Test loss: 0.8138929843108154
Test accuracy: 0.7115142345428467
In [0]:
## Fifth deep learning model M.5 trial 2 (with VGG16 model weights)
from keras.models import Model
# load vgg model
from keras.applications.vgg16 import VGG16, preprocess_input
from keras.applications.inception_v3 import InceptionV3, preprocess_input
In [0]:
# load the model
base_model = VGG16(weights='imagenet', include_top=False)
#base_model = InceptionV3(weights='imagenet', include_top=False)
In [0]:
base_model.summary()
Model: "vgg16"
_________________________________________________________________
Layer (type)                 Output Shape              Param #   
=================================================================
input_1 (InputLayer)         (None, None, None, 3)     0         
_________________________________________________________________
block1_conv1 (Conv2D)        (None, None, None, 64)    1792      
_________________________________________________________________
block1_conv2 (Conv2D)        (None, None, None, 64)    36928     
_________________________________________________________________
block1_pool (MaxPooling2D)   (None, None, None, 64)    0         
_________________________________________________________________
block2_conv1 (Conv2D)        (None, None, None, 128)   73856     
_________________________________________________________________
block2_conv2 (Conv2D)        (None, None, None, 128)   147584    
_________________________________________________________________
block2_pool (MaxPooling2D)   (None, None, None, 128)   0         
_________________________________________________________________
block3_conv1 (Conv2D)        (None, None, None, 256)   295168    
_________________________________________________________________
block3_conv2 (Conv2D)        (None, None, None, 256)   590080    
_________________________________________________________________
block3_conv3 (Conv2D)        (None, None, None, 256)   590080    
_________________________________________________________________
block3_pool (MaxPooling2D)   (None, None, None, 256)   0         
_________________________________________________________________
block4_conv1 (Conv2D)        (None, None, None, 512)   1180160   
_________________________________________________________________
block4_conv2 (Conv2D)        (None, None, None, 512)   2359808   
_________________________________________________________________
block4_conv3 (Conv2D)        (None, None, None, 512)   2359808   
_________________________________________________________________
block4_pool (MaxPooling2D)   (None, None, None, 512)   0         
_________________________________________________________________
block5_conv1 (Conv2D)        (None, None, None, 512)   2359808   
_________________________________________________________________
block5_conv2 (Conv2D)        (None, None, None, 512)   2359808   
_________________________________________________________________
block5_conv3 (Conv2D)        (None, None, None, 512)   2359808   
_________________________________________________________________
block5_pool (MaxPooling2D)   (None, None, None, 512)   0         
=================================================================
Total params: 14,714,688
Trainable params: 14,714,688
Non-trainable params: 0
_________________________________________________________________
In [0]:
# freeze the model weights
for layer in base_model.layers:
    layer.trainable = False
In [0]:
# construct model
x = base_model.output
x = GlobalAveragePooling2D(name='avg_pool')(x)
x = Dropout(0.4)(x)
predictions = Dense(num_classes, activation='softmax')(x)
model = Model(inputs=base_model.input, outputs=predictions)
In [0]:
model.compile(optimizer=Adam(),
              loss='categorical_crossentropy',
              metrics=['accuracy'])
In [0]:
model.summary()
Model: "model_8"
_________________________________________________________________
Layer (type)                 Output Shape              Param #   
=================================================================
input_6 (InputLayer)         (None, None, None, 3)     0         
_________________________________________________________________
block1_conv1 (Conv2D)        (None, None, None, 64)    1792      
_________________________________________________________________
block1_conv2 (Conv2D)        (None, None, None, 64)    36928     
_________________________________________________________________
block1_pool (MaxPooling2D)   (None, None, None, 64)    0         
_________________________________________________________________
block2_conv1 (Conv2D)        (None, None, None, 128)   73856     
_________________________________________________________________
block2_conv2 (Conv2D)        (None, None, None, 128)   147584    
_________________________________________________________________
block2_pool (MaxPooling2D)   (None, None, None, 128)   0         
_________________________________________________________________
block3_conv1 (Conv2D)        (None, None, None, 256)   295168    
_________________________________________________________________
block3_conv2 (Conv2D)        (None, None, None, 256)   590080    
_________________________________________________________________
block3_conv3 (Conv2D)        (None, None, None, 256)   590080    
_________________________________________________________________
block3_pool (MaxPooling2D)   (None, None, None, 256)   0         
_________________________________________________________________
block4_conv1 (Conv2D)        (None, None, None, 512)   1180160   
_________________________________________________________________
block4_conv2 (Conv2D)        (None, None, None, 512)   2359808   
_________________________________________________________________
block4_conv3 (Conv2D)        (None, None, None, 512)   2359808   
_________________________________________________________________
block4_pool (MaxPooling2D)   (None, None, None, 512)   0         
_________________________________________________________________
block5_conv1 (Conv2D)        (None, None, None, 512)   2359808   
_________________________________________________________________
block5_conv2 (Conv2D)        (None, None, None, 512)   2359808   
_________________________________________________________________
block5_conv3 (Conv2D)        (None, None, None, 512)   2359808   
_________________________________________________________________
block5_pool (MaxPooling2D)   (None, None, None, 512)   0         
_________________________________________________________________
avg_pool (GlobalAveragePooli (None, 512)               0         
_________________________________________________________________
dropout_5 (Dropout)          (None, 512)               0         
_________________________________________________________________
dense_4 (Dense)              (None, 10)                5130      
=================================================================
Total params: 14,719,818
Trainable params: 5,130
Non-trainable params: 14,714,688
_________________________________________________________________
In [0]:
# fit the model
history = model.fit(X_train, y_train,
                    batch_size=500,
                    epochs=12,
                    verbose=1,
                    validation_data=(X_test, y_test))
Train on 13505 samples, validate on 13505 samples
Epoch 1/12
13505/13505 [==============================] - 6s 414us/step - loss: 0.5167 - accuracy: 0.8501 - val_loss: 0.3863 - val_accuracy: 0.8910
Epoch 2/12
13505/13505 [==============================] - 6s 411us/step - loss: 0.4885 - accuracy: 0.8534 - val_loss: 0.3873 - val_accuracy: 0.8905
Epoch 3/12
13505/13505 [==============================] - 6s 412us/step - loss: 0.5240 - accuracy: 0.8459 - val_loss: 0.4000 - val_accuracy: 0.8874
Epoch 4/12
13505/13505 [==============================] - 6s 413us/step - loss: 0.4746 - accuracy: 0.8552 - val_loss: 0.3792 - val_accuracy: 0.8940
Epoch 5/12
13505/13505 [==============================] - 6s 412us/step - loss: 0.4769 - accuracy: 0.8538 - val_loss: 0.3757 - val_accuracy: 0.8929
Epoch 6/12
13505/13505 [==============================] - 6s 411us/step - loss: 0.4766 - accuracy: 0.8499 - val_loss: 0.3733 - val_accuracy: 0.8934
Epoch 7/12
13505/13505 [==============================] - 6s 413us/step - loss: 0.4604 - accuracy: 0.8569 - val_loss: 0.3739 - val_accuracy: 0.8915
Epoch 8/12
13505/13505 [==============================] - 6s 412us/step - loss: 0.4370 - accuracy: 0.8632 - val_loss: 0.3708 - val_accuracy: 0.8907
Epoch 9/12
13505/13505 [==============================] - 6s 413us/step - loss: 0.4508 - accuracy: 0.8596 - val_loss: 0.3787 - val_accuracy: 0.8872
Epoch 10/12
13505/13505 [==============================] - 6s 415us/step - loss: 0.4645 - accuracy: 0.8485 - val_loss: 0.3755 - val_accuracy: 0.8922
Epoch 11/12
13505/13505 [==============================] - 6s 411us/step - loss: 0.4492 - accuracy: 0.8618 - val_loss: 0.3659 - val_accuracy: 0.8942
Epoch 12/12
13505/13505 [==============================] - 6s 412us/step - loss: 0.4497 - accuracy: 0.8588 - val_loss: 0.3767 - val_accuracy: 0.8883
In [0]:
# evaluate the model
score = model.evaluate(X_test, y_test, verbose=0)
print('Test loss:', score[0])
print('Test accuracy:', score[1])
Test loss: 0.3766734523619382
Test accuracy: 0.8883376717567444
In [0]:
# check labeling error of each class
results = model.predict(X_test)
In [0]:
results
Out[0]:
array([[2.4133356e-01, 7.3351913e-10, 8.1162510e-10, ..., 5.9573291e-10,
        7.5440884e-01, 1.6310478e-08],
       [1.9804563e-01, 2.4410190e-02, 4.3836786e-07, ..., 9.6854683e-06,
        1.5316344e-02, 1.9544849e-02],
       [2.0079828e-04, 8.7549537e-02, 1.2287562e-03, ..., 5.4740848e-04,
        3.4269638e-04, 9.0904033e-01],
       ...,
       [9.3672252e-01, 8.8067819e-23, 3.0530498e-11, ..., 9.3517274e-09,
        3.2298265e-06, 3.3727824e-20],
       [2.2811066e-16, 0.0000000e+00, 2.0683032e-26, ..., 3.4997865e-09,
        3.4882233e-17, 0.0000000e+00],
       [6.8968498e-12, 4.7158712e-24, 6.3914107e-10, ..., 9.1521198e-01,
        4.2880635e-07, 2.5439743e-23]], dtype=float32)
In [0]:
y_pred = np.argmax(results,axis=1)
In [0]:
y_test1 = np.argmax(y_test,axis=1)
In [0]:
confusion_matrix(y_test1, y_pred)
Out[0]:
array([[1346,    2,    0,   34,    1,   49,   49,    0,   19,    5],
       [   1, 1419,   17,    2,    0,   35,    1,    2,    4,   19],
       [   1,   22, 1391,    5,    2,   24,   32,   18,    5,    0],
       [  37,    4,   27, 1006,   25,   36,   33,   19,   62,    1],
       [   5,    0,    1,   29, 1129,    3,   26,   51,    6,    0],
       [   4,   24,   18,   10,    0,  928,    7,    1,    7,    1],
       [  38,    1,  128,   30,   21,   51,  944,   25,   12,    0],
       [   0,   15,   28,    7,   32,    7,    8, 1398,    5,    0],
       [  35,    4,   21,  122,    9,   55,    4,    5,  988,    7],
       [   2,   24,    4,    5,    0,    7,    0,    0,   10, 1448]])
In [0]:
import seaborn as sn
import pandas as pd
import matplotlib.pyplot as plt

array = confusion_matrix(y_test1, y_pred)

df_cm = pd.DataFrame(array, range(10), range(10))

plt.figure(figsize=(11,9))
sn.set(font_scale=1) # for label size
sn.heatmap(df_cm, annot=True, annot_kws={"size": 12}) # font size

plt.show()
In [0]:
dictionary
Out[0]:
{'AnnualCrop': 0,
 'Forest': 1,
 'HerbaceousVegetation': 2,
 'Highway': 3,
 'Industrial': 4,
 'Pasture': 5,
 'PermanentCrop': 6,
 'Residential': 7,
 'River': 8,
 'SeaLake': 9}
In [0]:
# Class 6(PermanentCrop) is prone to be misclassified as class 2(HerbaceousVegetation), 
# and class 8(River) is prone to be misclassfied as class 3(Highway).
In [0]:
# successfully identified class 6 (PermanentCrop) image example:
trial1 = np.where((y_test1== 6) & (y_pred == 6))[0][1]
tmpimg = imread(imgfiles[trial1])
imshow(tmpimg)
show()
In [0]:
# successfully identified class 2 (HerbaceousVegetation) image example:
trial1 = np.where((y_test1== 2) & (y_pred == 2))[0][10]
tmpimg = imread(imgfiles[trial1])
imshow(tmpimg)
show()
In [0]:
# show some misclassified examples:
# select a few class 6 images that were miclassified as class 2: 
trial1 = np.where((y_test1== 6) & (y_pred == 2))[0][1]
tmpimg = imread(imgfiles[trial1])
imshow(tmpimg)
show()
trial2 = np.where((y_test1== 6) & (y_pred == 2))[0][5]
tmpimg = imread(imgfiles[trial2])
imshow(tmpimg)
show()
In [0]:
# successfully identified class 8 (River) image example:
trial1 = np.where((y_test1== 8) & (y_pred == 8))[0][15]
tmpimg = imread(imgfiles[trial1])
imshow(tmpimg)
show()
In [0]:
# successfully identified class 3 (Highway) image example:
trial1 = np.where((y_test1== 3) & (y_pred == 3))[0][15]
tmpimg = imread(imgfiles[trial1])
imshow(tmpimg)
show()
In [0]:
# select a few class 8 images that were miclassified as class 3: 
trial1 = np.where((y_test1== 8) & (y_pred == 3))[0][1]
tmpimg = imread(imgfiles[trial1])
imshow(tmpimg)
show()
trial2 = np.where((y_test1== 8) & (y_pred == 3))[0][5]
tmpimg = imread(imgfiles[trial2])
imshow(tmpimg)
show()