EuroSAT Land Use and Land Cover Classification using Deep Learning¶
In this exercise, I implemented deep learning models to solve a typical problem in satellite imaging using a benchmark dataset. Both multilayer perceptron (MLP) and convolutional neural network (CNN) are used to label the EuroSAT satellite images (27010 images) with the correct land use classifications. This notebook consists of several sections that train the EuroSAT images with increasingly more complex models, from the single-layer neural net to transfer learning of VGG16. The accuracies improved from 16.8% to 88.8%.
S1:¶
Visit the EuroSAT data description page and download the data: https://github.com/phelber/eurosat
Split the data into training (50%) and testing sets (50%), stratified on class labels (equal percentage of each class type in train and test sets).
from google.colab import drive
drive.mount('/content/drive')
from __future__ import print_function
import pandas as pd
import numpy as np
import os
from tqdm import tqdm
import keras
from keras.datasets import mnist
from keras.models import Sequential
from keras.layers import Dense, Dropout, Flatten, Conv2D, MaxPooling2D, Input, GlobalAveragePooling2D
from keras.optimizers import RMSprop, Adam, SGD, Adadelta
from keras.utils import plot_model
from keras import backend as K
import matplotlib.image as mpimg
import matplotlib.pyplot as plt
from matplotlib.pyplot import imread, imshow, subplots, show
from skimage.color import rgb2gray
from sklearn.metrics import accuracy_score, confusion_matrix, roc_auc_score
from sklearn.preprocessing import StandardScaler
from sklearn.preprocessing import MinMaxScaler
from sklearn.decomposition import PCA
from sklearn.model_selection import train_test_split, StratifiedShuffleSplit, StratifiedKFold, cross_val_score, cross_validate, cross_val_predict
# reading the data
import glob
import os
imgdir='/content/drive/My Drive/Colab Notebooks/MUSA650/EuroSAT'
imgfiles = []
for file in glob.glob(imgdir + os.sep + "*" + os.sep + "*.jpg"):
imgfiles.append(file)
# plot the first img
tmpimg = imread(imgfiles[0])
imshow(tmpimg)
show()
### TAKES A LONG TIME TO RUN, AVOID RE-RUN !!! ###
eurosat = []
eurosat_label = []
for row in imgfiles:
tmpimg = imread(row)
gray_img = rgb2gray(tmpimg).reshape(1, 4096).squeeze().flatten() #flatten and convert to gray scale
eurosat.append(gray_img) #append to the feature data set
label = row.split("/")[7]
eurosat_label.append(label) #append to the label list
# flatten the list of arrays into matrix
eurosat_f = np.vstack(eurosat)
eurosat_label_f = np.vstack(eurosat_label).reshape(27010,)
# change label to integers
dictionary = {"AnnualCrop":0, "Forest":1, "HerbaceousVegetation":2, "Highway":3,
"Industrial":4, "Pasture":5, "PermanentCrop":6, "Residential":7, "River":8, "SeaLake":9}
eurosat_label_f = [dictionary[letter] for letter in eurosat_label_f]
eurosat_label_f = np.vstack(eurosat_label_f).reshape(27010,)
# split into 50%/50% training and testing set
X_train, X_test, y_train, y_test = train_test_split(eurosat_f, eurosat_label_f, stratify = eurosat_label_f, train_size = 0.5, random_state= 10)
X_train.shape
y_train.shape
S2:¶
Convert each RBG image to grayscale and flatten the images into a data matrix (n x p: n = #samples, p = #pixels in each image)
Implement a first deep learning model (M.1) using a fully connected network with a single fully connected layer (i.e: input layer + fully connected layer as the output layer).
Q1: Calculate classification accuracy on the test data.
16.8%
# first step done in S1
# set global params
batch_size = 1200
num_classes = 10
epochs = 10
# convert class vectors to binary class matrices
y_train = keras.utils.to_categorical(y_train, num_classes)
y_test = keras.utils.to_categorical(y_test, num_classes)
## First deep learning model M.1
model = Sequential()
model.add(Dense(num_classes, activation='softmax', input_shape=(4096,)))
model.summary()
# visualize model
plot_model(model, show_shapes=True, show_layer_names=True)
# complie the model
model.compile(loss='categorical_crossentropy',
optimizer=Adam(),
metrics=['accuracy'])
# fit the model
history = model.fit(X_train, y_train,
batch_size=batch_size,
epochs=epochs,
verbose=1,
validation_data=(X_test, y_test))
# evaluate the model
score = model.evaluate(X_test, y_test, verbose=0)
print('Test loss:', score[0])
print('Test accuracy:', score[1])
S3:¶
Implement a second deep learning model (M.2) adding an additional fully connected hidden layer (with an arbitrary number of nodes) to the previous model.
Q1: Calculate classification accuracy on the test data.
21.2%
## Second deep learning model M.2
model = Sequential()
model.add(Dense(800, activation='relu', input_shape=(4096,)))
model.add(Dense(num_classes, activation='softmax'))
model.summary()
# visualize model
plot_model(model, show_shapes=True, show_layer_names=True)
# complie the model
model.compile(loss='categorical_crossentropy',
optimizer=Adam(),
metrics=['accuracy'])
# fit the model
history = model.fit(X_train, y_train,
batch_size=batch_size,
epochs=15,
verbose=1,
validation_data=(X_test, y_test))
# evaluate the model
score = model.evaluate(X_test, y_test, verbose=0)
print('Test loss:', score[0])
print('Test accuracy:', score[1])
S4:¶
Implement a third deep learning model (M.3) adding two additional fully connected hidden layers (with arbitrary number of nodes) as well as drop-out layers to the previous model.
Q1: Calculate classification accuracy on the test data.
28.0%
Q2: Compare against previous models. Which model was the "best"? Why?
The M.3 model was the "best". It has the highest accuracy with only 10 epochs. The multiple layers of neurons as well as the drop outs make the model more robust and avoid overfitting.
## Third deep learning model M.3
model = Sequential()
model.add(Dense(2700, activation='relu', input_shape=(4096,)))
model.add(Dropout(0.2))
model.add(Dense(1800, activation='relu'))
model.add(Dropout(0.2))
model.add(Dense(800, activation='relu'))
model.add(Dropout(0.2))
model.add(Dense(num_classes, activation='softmax'))
model.summary()
# visualize model
plot_model(model, show_shapes=True, show_layer_names=True)
# complie the model
model.compile(loss='categorical_crossentropy',
optimizer=Adam(),
metrics=['accuracy'])
# fit the model
history = model.fit(X_train, y_train,
batch_size=batch_size,
epochs=epochs,
verbose=1,
validation_data=(X_test, y_test))
# evaluate the model
score = model.evaluate(X_test, y_test, verbose=0)
print('Test loss:', score[0])
print('Test accuracy:', score[1])
S5:¶
Using RGB images (without vectorizing them), implement a fourth CNN model (M.4) that includes the following layers: Conv2D, MaxPooling2D, Dropout, Flatten, Dense.
Q1: Calculate classification accuracy on the test data.
50.3%
- Q2: Compare against previous models. Which model was the "best"? Why?
This model is much better than the previous models. This is because the inclusion of the convolutional layers that captures the image relational features.
# load RGB images
RGB_eurosat = []
for row in imgfiles:
tmpimg = imread(row)
RGB_eurosat.append(tmpimg) #append to the feature data set
# list to matrix
RGB_eurosat_f = np.stack(RGB_eurosat)
input_shape = (64, 64, 3)
# split into 50%/50% training and testing set
X_train, X_test, y_train, y_test = train_test_split(RGB_eurosat_f, eurosat_label_f, stratify = eurosat_label_f, train_size = 0.5, random_state= 10)
X_train.shape
# convert class vectors to binary class matrices
y_train = keras.utils.to_categorical(y_train, num_classes)
y_test = keras.utils.to_categorical(y_test, num_classes)
## Fourth deep learning model M.4
model = Sequential()
model.add(Conv2D(64, kernel_size=(3, 3),
activation='relu',
input_shape=input_shape,
data_format='channels_last', padding = "same"))
model.add(MaxPooling2D(pool_size=(3, 3), padding = "same"))
model.add(Dropout(0.25))
model.add(Flatten())
model.add(Dense(num_classes, activation='softmax'))
# visualize model
plot_model(model, show_shapes=True, show_layer_names=True)
# complie the model
model.compile(loss='categorical_crossentropy',
optimizer=Adam(),
metrics=['accuracy'])
# fit the model
history = model.fit(X_train, y_train,
batch_size=batch_size,
epochs=epochs,
verbose=1,
validation_data=(X_test, y_test))
# evaluate the model
score = model.evaluate(X_test, y_test, verbose=0)
print('Test loss:', score[0])
print('Test accuracy:', score[1])
S6:¶
Using RGB images (without vectorizing them), implement a fifth deep learning model (M.5) targeting accuracy that will outperform all previous models. You are free to use any tools and techniques, as well as pre-trained models for transfer learning.
Q1: Describe the model you built, and why you chose it.
The first trial uses multiple convolutional layers, as well as max pooling, dense, and drop out layers. A second trial was done with transfer learning with the VGG16 model which gives the best model for this section. This model relies on the pre-trained VGG16 model weights. The architecture here uses the VGG16 model weights with variation in the output class numbers. VGG16 was chosen because it has trained on a large dataset with multiple kinds of images, which means it captures the basic image features(filters) well.
Q2: Calculate classification accuracy on the test data.
88.8%
Q3: Compare against previous models. Which model was the "best"? Why?
The model using VGG16 weights outperforms all the previous model. This is because VGG16 was trained on a large dataset with an neural net architecture that has tested to be good at capturing the image features.
Q4: What are the two classes with the highest labeling error? Explain using data and showing mis-classified examples.
Class 6(PermanentCrop) is prone to be misclassified as class 2(HerbaceousVegetation), and class 8(River) is prone to be misclassfied as class 3(Highway).
## Fifth deep learning model M.5 trial 1
model = Sequential()
model.add(Conv2D(64, kernel_size=(3, 3),
activation='relu',
input_shape=input_shape,
data_format='channels_last', padding = "same"))
model.add(Conv2D(64, kernel_size=(3, 3),
activation='relu',
data_format='channels_last', padding = "same"))
model.add(MaxPooling2D(pool_size=(3, 3), padding = "same"))
model.add(Conv2D(128, kernel_size=(3, 3),
activation='relu',
data_format='channels_last', padding = "same"))
model.add(MaxPooling2D(pool_size=(3, 3), padding = "same"))
model.add(Dropout(0.25))
model.add(Flatten())
model.add(Dense(800, activation='relu'))
model.add(Dropout(0.25))
model.add(Dense(num_classes, activation='softmax'))
# visualize model
plot_model(model, show_shapes=True, show_layer_names=True)
# complie the model
model.compile(loss='categorical_crossentropy',
optimizer=Adam(),
metrics=['accuracy'])
# fit the model
history = model.fit(X_train, y_train,
batch_size=batch_size,
epochs=epochs,
verbose=1,
validation_data=(X_test, y_test))
# evaluate the model
score = model.evaluate(X_test, y_test, verbose=0)
print('Test loss:', score[0])
print('Test accuracy:', score[1])
## Fifth deep learning model M.5 trial 2 (with VGG16 model weights)
from keras.models import Model
# load vgg model
from keras.applications.vgg16 import VGG16, preprocess_input
from keras.applications.inception_v3 import InceptionV3, preprocess_input
# load the model
base_model = VGG16(weights='imagenet', include_top=False)
#base_model = InceptionV3(weights='imagenet', include_top=False)
base_model.summary()
# freeze the model weights
for layer in base_model.layers:
layer.trainable = False
# construct model
x = base_model.output
x = GlobalAveragePooling2D(name='avg_pool')(x)
x = Dropout(0.4)(x)
predictions = Dense(num_classes, activation='softmax')(x)
model = Model(inputs=base_model.input, outputs=predictions)
model.compile(optimizer=Adam(),
loss='categorical_crossentropy',
metrics=['accuracy'])
model.summary()
# fit the model
history = model.fit(X_train, y_train,
batch_size=500,
epochs=12,
verbose=1,
validation_data=(X_test, y_test))
# evaluate the model
score = model.evaluate(X_test, y_test, verbose=0)
print('Test loss:', score[0])
print('Test accuracy:', score[1])
# check labeling error of each class
results = model.predict(X_test)
results
y_pred = np.argmax(results,axis=1)
y_test1 = np.argmax(y_test,axis=1)
confusion_matrix(y_test1, y_pred)
import seaborn as sn
import pandas as pd
import matplotlib.pyplot as plt
array = confusion_matrix(y_test1, y_pred)
df_cm = pd.DataFrame(array, range(10), range(10))
plt.figure(figsize=(11,9))
sn.set(font_scale=1) # for label size
sn.heatmap(df_cm, annot=True, annot_kws={"size": 12}) # font size
plt.show()
dictionary
# Class 6(PermanentCrop) is prone to be misclassified as class 2(HerbaceousVegetation),
# and class 8(River) is prone to be misclassfied as class 3(Highway).
# successfully identified class 6 (PermanentCrop) image example:
trial1 = np.where((y_test1== 6) & (y_pred == 6))[0][1]
tmpimg = imread(imgfiles[trial1])
imshow(tmpimg)
show()
# successfully identified class 2 (HerbaceousVegetation) image example:
trial1 = np.where((y_test1== 2) & (y_pred == 2))[0][10]
tmpimg = imread(imgfiles[trial1])
imshow(tmpimg)
show()
# show some misclassified examples:
# select a few class 6 images that were miclassified as class 2:
trial1 = np.where((y_test1== 6) & (y_pred == 2))[0][1]
tmpimg = imread(imgfiles[trial1])
imshow(tmpimg)
show()
trial2 = np.where((y_test1== 6) & (y_pred == 2))[0][5]
tmpimg = imread(imgfiles[trial2])
imshow(tmpimg)
show()
# successfully identified class 8 (River) image example:
trial1 = np.where((y_test1== 8) & (y_pred == 8))[0][15]
tmpimg = imread(imgfiles[trial1])
imshow(tmpimg)
show()
# successfully identified class 3 (Highway) image example:
trial1 = np.where((y_test1== 3) & (y_pred == 3))[0][15]
tmpimg = imread(imgfiles[trial1])
imshow(tmpimg)
show()
# select a few class 8 images that were miclassified as class 3:
trial1 = np.where((y_test1== 8) & (y_pred == 3))[0][1]
tmpimg = imread(imgfiles[trial1])
imshow(tmpimg)
show()
trial2 = np.where((y_test1== 8) & (y_pred == 3))[0][5]
tmpimg = imread(imgfiles[trial2])
imshow(tmpimg)
show()
