Build a letter recognition classifier
Dec 28, 2017 00:00 · 14089 words · 67 minute read
In this analysis i’ll develop a model that will predict the hand-written capital letters (English) displayed in a picture, using various of it’s attributes.
INTRODUCTION
Dataset Information
This data set was originally used in the paper of P. W. Frey and D. J. Slate.“Letter Recognition Using Holland-style Adaptive Classifiers” (Machine Learning Vol 6 #2 March 91). It investigated the ability of several variations of Holland-style adaptive classifier systems to learn to correctly guess the letter categories associated with vectors of 16 integer attributes extracted from raster scan images of the letters. The actual data can be found at Letter Recognition Data Set Machine Learning Repository: https://archive.ics.uci.edu/ml/datasets/Letter+Recognition
The objective is to identify each of a large number of black-and-white rectangular pixel displays as one of the 26 capital letters in the English alphabet. The character images were based on 20 different fonts and each letter within these 20 fonts was randomly distorted to produce a file of unique stimuli. Each stimulus was converted into primitive numerical attributes (statistical moments and edge counts) which were then scaled to fit into a range of integer values from 0 through 15. So in total there are 20000 observations of 17 variables. There are no missing values in the data set.
The variables
- lettr capital letter (26 values from A to Z)
- x-box horizontal position of box (integer)
- y-box vertical position of box (integer)
- width width of box (integer)
- high height of box (integer)
- onpix total # on pixels (integer)
- x-bar mean x of on pixels in box (integer)
- y-bar mean y of on pixels in box (integer)
- x2bar mean x variance (integer)
- y2bar mean y variance (integer)
- xybar mean x y correlation (integer)
- x2ybr mean of x * x * y (integer)
- xy2br mean of x * y * y (integer)
- x-ege mean edge count left to right (integer)
- xegvy correlation of x-ege with y (integer)
- y-ege mean edge count bottom to top (integer)
- yegvx correlation of y-ege with x (integer)
PRE-PROCCESSING
Since the original data set didn’t contain any names on the variables, after i imported the data in R environment, i assigned the appropriate variable names. That was completed with the help of letter-recognition.names text file which accompanied the original data.
During the development of the prediction model, in order to predict the letter on new observations, at first we’ve used all variables. Then we excluded some of the 4 correlated variables and finally we’ve used the variables created from the SVD algorithm instead of the original.
Before we apply the hierarchical clustering, we created a data set with each letter as one observation and the average of each variable, as a distinct variable (named letter.cluster).
# Load Libraries
library(tidyverse)
library(stringr)
library(scales)
library(ggthemes)
library(corrplot)
library(caret)
library(gmodels)
library(class)
library(ggdendro)
# DATA WRANGLING ###############################################################
# Insert dataset
letter <- read.table("/Users/manos/OneDrive/Projects/R/Data/letter-recognition.data",
sep = ",")
# Create vector with new variable names
name <- c("lettr", "x-box", "y-box", "width", "high", "onpix", "x-bar", "y-bar",
"x2bar", "y2bar", "xybar", "x2ybr", "xy2br", "x-ege", "xegvy", "y-ege",
"yegvx")
# Change the variable names of the dataset
names(letter) <- name
# Make it a tibble
letter <- as_data_frame(letter)
# Check for missing values
sum(is.na(letter))## [1] 0# Create the summary dataset
letter.cluster <- letter %>%
group_by(lettr) %>%
summarise_each(funs(mean))EXPLORATORY ANALYSIS
In order to have an overview of the data-set’s variability, i produced a box-plot matrix of all variables and each letter on the x-axis
# Plot a box-plot for each letter and variable
letter %>%
gather("type", "n", 2:17) %>%
ggplot()+
geom_boxplot(aes(x = lettr, y = n), outlier.size = .5, fill = "steelblue2", alpha = .7)+
facet_grid(type ~.)+
labs(y = "", x = "",
title = "Box-Plots of all variables for each letter",
subtitle = "")+
theme_fivethirtyeight()
It looks that in some variables, the letters levels have significant different variance, than others. In particular, in y-box and high variables the different letters don’t seem to have any significant differences. On the other hand in x-bar, x2ybr, y-bar, y-ege variables the different letters seem to have large differences.
We should check for correlated variables in the data set. Below you can see a correlogram of the data set variables.
# Plot a correlogram to check for correlations
corrplot(cor(letter[,2:17]), method="number", type = "lower", number.cex = .6)
It seems that the first 4 variables (x-box, y-box, width, high) are quite correlated. Especially x-box & width (0.85) and y-box & high (0.82) are highly correlated.
MODELLING
An interesting insight that could be obtained from the data set is to try to predict the letter based on the variables produced by the digitized image. Classification is the problem of identifying to which of a set of categories (sub-populations) a new observation belongs, on the basis of a training set of data containing observations (or instances) whose category membership is known. So we must develop a model that classifies (categorize) every observation (case) to one of the 26 letters of the alphabet.
K-Nearest Neighbors (k-NN)
The K-nearest neighbors (kNN) algorithm used, in order to classify observations in a certain category (letter) by using the rest of the variables in the data set. It begins with a training data set made up of examples that are classified into several categories, as labeled by a nominal variable. Assume that we have a test data set containing unlabeled examples that otherwise have the same features as the training data. For each record in the test data set, kNN identifies k records in the training data that are the “nearest” in similarity, where k is an integer specified in advance. The unlabeled test instance is assigned the class of the majority of the k-nearest neighbors. We splitted the data set into a training data set (containing 70 % of the original data set observations) in which we build the classification model and a testing data set (containing the rest of the original data set observations - 30%) in which we tested the model we build before. The results are presented in the “results” section.
# Create a vector with the 70% of the dataset with respect to letter
set.seed(10)
inTrain = createDataPartition(letter$lettr, p = .7)[[1]]
# Assign the 70% of observations to training data
training <- letter[inTrain, -1]
training.lettr <- c(t(letter[inTrain, 1]))
# Assign the remaining 30 % of observations to testing data
testing <- letter[-inTrain, -1]
testing.lettr <- c(t(letter[-inTrain, 1]))
# Run knn algorithm on training dataset
# Create the knn model
knn_model <- knn(train = training, test = testing, cl = training.lettr, k = 3)
# Create a table in order to check the performance of the classification model
t <- CrossTable(x = testing.lettr,y = knn_model,
prop.chisq=FALSE)##
##
## Cell Contents
## |-------------------------|
## | N |
## | N / Row Total |
## | N / Col Total |
## | N / Table Total |
## |-------------------------|
##
##
## Total Observations in Table: 5985
##
##
## | knn_model
## testing.lettr | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z | Row Total |
## --------------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|
## A | 230 | 0 | 2 | 0 | 0 | 0 | 0 | 0 | 1 | 0 | 0 | 1 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 1 | 0 | 0 | 0 | 0 | 1 | 0 | 236 |
## | 0.975 | 0.000 | 0.008 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.004 | 0.000 | 0.000 | 0.004 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.004 | 0.000 | 0.000 | 0.000 | 0.000 | 0.004 | 0.000 | 0.039 |
## | 0.991 | 0.000 | 0.009 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.005 | 0.000 | 0.000 | 0.004 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.004 | 0.000 | 0.000 | 0.000 | 0.000 | 0.004 | 0.000 | |
## | 0.038 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | |
## --------------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|
## B | 0 | 210 | 0 | 1 | 1 | 1 | 0 | 2 | 0 | 0 | 2 | 0 | 1 | 1 | 0 | 0 | 0 | 7 | 0 | 0 | 0 | 3 | 0 | 0 | 0 | 0 | 229 |
## | 0.000 | 0.917 | 0.000 | 0.004 | 0.004 | 0.004 | 0.000 | 0.009 | 0.000 | 0.000 | 0.009 | 0.000 | 0.004 | 0.004 | 0.000 | 0.000 | 0.000 | 0.031 | 0.000 | 0.000 | 0.000 | 0.013 | 0.000 | 0.000 | 0.000 | 0.000 | 0.038 |
## | 0.000 | 0.913 | 0.000 | 0.004 | 0.004 | 0.005 | 0.000 | 0.010 | 0.000 | 0.000 | 0.009 | 0.000 | 0.004 | 0.004 | 0.000 | 0.000 | 0.000 | 0.028 | 0.000 | 0.000 | 0.000 | 0.013 | 0.000 | 0.000 | 0.000 | 0.000 | |
## | 0.000 | 0.035 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.001 | 0.000 | 0.000 | 0.000 | 0.001 | 0.000 | 0.000 | 0.000 | 0.000 | |
## --------------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|
## C | 0 | 0 | 210 | 0 | 4 | 0 | 1 | 0 | 0 | 0 | 0 | 1 | 0 | 0 | 3 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 1 | 0 | 0 | 220 |
## | 0.000 | 0.000 | 0.955 | 0.000 | 0.018 | 0.000 | 0.005 | 0.000 | 0.000 | 0.000 | 0.000 | 0.005 | 0.000 | 0.000 | 0.014 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.005 | 0.000 | 0.000 | 0.037 |
## | 0.000 | 0.000 | 0.972 | 0.000 | 0.017 | 0.000 | 0.004 | 0.000 | 0.000 | 0.000 | 0.000 | 0.004 | 0.000 | 0.000 | 0.013 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.004 | 0.000 | 0.000 | |
## | 0.000 | 0.000 | 0.035 | 0.000 | 0.001 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.001 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | |
## --------------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|
## D | 0 | 1 | 0 | 233 | 0 | 0 | 0 | 1 | 0 | 0 | 1 | 0 | 0 | 2 | 2 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 1 | 0 | 0 | 241 |
## | 0.000 | 0.004 | 0.000 | 0.967 | 0.000 | 0.000 | 0.000 | 0.004 | 0.000 | 0.000 | 0.004 | 0.000 | 0.000 | 0.008 | 0.008 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.004 | 0.000 | 0.000 | 0.040 |
## | 0.000 | 0.004 | 0.000 | 0.940 | 0.000 | 0.000 | 0.000 | 0.005 | 0.000 | 0.000 | 0.005 | 0.000 | 0.000 | 0.008 | 0.009 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.004 | 0.000 | 0.000 | |
## | 0.000 | 0.000 | 0.000 | 0.039 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | |
## --------------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|
## E | 0 | 1 | 0 | 0 | 217 | 0 | 4 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 2 | 0 | 0 | 0 | 0 | 1 | 0 | 5 | 230 |
## | 0.000 | 0.004 | 0.000 | 0.000 | 0.943 | 0.000 | 0.017 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.009 | 0.000 | 0.000 | 0.000 | 0.000 | 0.004 | 0.000 | 0.022 | 0.038 |
## | 0.000 | 0.004 | 0.000 | 0.000 | 0.935 | 0.000 | 0.017 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.009 | 0.000 | 0.000 | 0.000 | 0.000 | 0.004 | 0.000 | 0.022 | |
## | 0.000 | 0.000 | 0.000 | 0.000 | 0.036 | 0.000 | 0.001 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.001 | |
## --------------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|
## F | 0 | 0 | 0 | 0 | 0 | 206 | 0 | 1 | 0 | 1 | 0 | 0 | 0 | 2 | 0 | 12 | 0 | 2 | 0 | 4 | 0 | 1 | 1 | 0 | 1 | 1 | 232 |
## | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.888 | 0.000 | 0.004 | 0.000 | 0.004 | 0.000 | 0.000 | 0.000 | 0.009 | 0.000 | 0.052 | 0.000 | 0.009 | 0.000 | 0.017 | 0.000 | 0.004 | 0.004 | 0.000 | 0.004 | 0.004 | 0.039 |
## | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.932 | 0.000 | 0.005 | 0.000 | 0.004 | 0.000 | 0.000 | 0.000 | 0.008 | 0.000 | 0.049 | 0.000 | 0.008 | 0.000 | 0.017 | 0.000 | 0.004 | 0.004 | 0.000 | 0.004 | 0.004 | |
## | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.034 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.002 | 0.000 | 0.000 | 0.000 | 0.001 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | |
## --------------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|
## G | 0 | 1 | 1 | 1 | 0 | 1 | 221 | 1 | 0 | 0 | 1 | 0 | 0 | 0 | 2 | 0 | 0 | 1 | 0 | 0 | 0 | 0 | 1 | 0 | 0 | 0 | 231 |
## | 0.000 | 0.004 | 0.004 | 0.004 | 0.000 | 0.004 | 0.957 | 0.004 | 0.000 | 0.000 | 0.004 | 0.000 | 0.000 | 0.000 | 0.009 | 0.000 | 0.000 | 0.004 | 0.000 | 0.000 | 0.000 | 0.000 | 0.004 | 0.000 | 0.000 | 0.000 | 0.039 |
## | 0.000 | 0.004 | 0.005 | 0.004 | 0.000 | 0.005 | 0.957 | 0.005 | 0.000 | 0.000 | 0.005 | 0.000 | 0.000 | 0.000 | 0.009 | 0.000 | 0.000 | 0.004 | 0.000 | 0.000 | 0.000 | 0.000 | 0.004 | 0.000 | 0.000 | 0.000 | |
## | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.037 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | |
## --------------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|
## H | 0 | 1 | 0 | 6 | 0 | 0 | 2 | 186 | 0 | 0 | 12 | 0 | 0 | 1 | 1 | 1 | 0 | 9 | 1 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 220 |
## | 0.000 | 0.005 | 0.000 | 0.027 | 0.000 | 0.000 | 0.009 | 0.845 | 0.000 | 0.000 | 0.055 | 0.000 | 0.000 | 0.005 | 0.005 | 0.005 | 0.000 | 0.041 | 0.005 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.037 |
## | 0.000 | 0.004 | 0.000 | 0.024 | 0.000 | 0.000 | 0.009 | 0.899 | 0.000 | 0.000 | 0.055 | 0.000 | 0.000 | 0.004 | 0.004 | 0.004 | 0.000 | 0.037 | 0.005 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | |
## | 0.000 | 0.000 | 0.000 | 0.001 | 0.000 | 0.000 | 0.000 | 0.031 | 0.000 | 0.000 | 0.002 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.002 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | |
## --------------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|
## I | 0 | 0 | 0 | 1 | 0 | 2 | 0 | 0 | 211 | 12 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 226 |
## | 0.000 | 0.000 | 0.000 | 0.004 | 0.000 | 0.009 | 0.000 | 0.000 | 0.934 | 0.053 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.038 |
## | 0.000 | 0.000 | 0.000 | 0.004 | 0.000 | 0.009 | 0.000 | 0.000 | 0.955 | 0.053 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | |
## | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.035 | 0.002 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | |
## --------------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|
## J | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 2 | 7 | 212 | 0 | 1 | 0 | 0 | 1 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 1 | 224 |
## | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.009 | 0.031 | 0.946 | 0.000 | 0.004 | 0.000 | 0.000 | 0.004 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.004 | 0.037 |
## | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.010 | 0.032 | 0.942 | 0.000 | 0.004 | 0.000 | 0.000 | 0.004 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.004 | |
## | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.001 | 0.035 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | |
## --------------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|
## K | 0 | 1 | 0 | 0 | 3 | 0 | 0 | 9 | 0 | 0 | 195 | 0 | 0 | 0 | 0 | 0 | 0 | 7 | 0 | 0 | 0 | 1 | 0 | 5 | 0 | 0 | 221 |
## | 0.000 | 0.005 | 0.000 | 0.000 | 0.014 | 0.000 | 0.000 | 0.041 | 0.000 | 0.000 | 0.882 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.032 | 0.000 | 0.000 | 0.000 | 0.005 | 0.000 | 0.023 | 0.000 | 0.000 | 0.037 |
## | 0.000 | 0.004 | 0.000 | 0.000 | 0.013 | 0.000 | 0.000 | 0.043 | 0.000 | 0.000 | 0.886 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.028 | 0.000 | 0.000 | 0.000 | 0.004 | 0.000 | 0.022 | 0.000 | 0.000 | |
## | 0.000 | 0.000 | 0.000 | 0.000 | 0.001 | 0.000 | 0.000 | 0.002 | 0.000 | 0.000 | 0.033 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.001 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.001 | 0.000 | 0.000 | |
## --------------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|
## L | 0 | 0 | 0 | 0 | 1 | 0 | 1 | 1 | 0 | 0 | 0 | 224 | 0 | 0 | 0 | 0 | 0 | 1 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 228 |
## | 0.000 | 0.000 | 0.000 | 0.000 | 0.004 | 0.000 | 0.004 | 0.004 | 0.000 | 0.000 | 0.000 | 0.982 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.004 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.038 |
## | 0.000 | 0.000 | 0.000 | 0.000 | 0.004 | 0.000 | 0.004 | 0.005 | 0.000 | 0.000 | 0.000 | 0.987 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.004 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | |
## | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.037 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | |
## --------------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|
## M | 0 | 2 | 0 | 0 | 0 | 0 | 1 | 0 | 0 | 0 | 0 | 0 | 231 | 1 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 2 | 0 | 0 | 0 | 237 |
## | 0.000 | 0.008 | 0.000 | 0.000 | 0.000 | 0.000 | 0.004 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.975 | 0.004 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.008 | 0.000 | 0.000 | 0.000 | 0.040 |
## | 0.000 | 0.009 | 0.000 | 0.000 | 0.000 | 0.000 | 0.004 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.979 | 0.004 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.009 | 0.000 | 0.000 | 0.000 | |
## | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.039 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | |
## --------------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|
## N | 1 | 0 | 0 | 0 | 0 | 0 | 0 | 1 | 0 | 0 | 0 | 0 | 0 | 228 | 1 | 0 | 0 | 2 | 0 | 0 | 0 | 0 | 1 | 0 | 0 | 0 | 234 |
## | 0.004 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.004 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.974 | 0.004 | 0.000 | 0.000 | 0.009 | 0.000 | 0.000 | 0.000 | 0.000 | 0.004 | 0.000 | 0.000 | 0.000 | 0.039 |
## | 0.004 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.005 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.966 | 0.004 | 0.000 | 0.000 | 0.008 | 0.000 | 0.000 | 0.000 | 0.000 | 0.004 | 0.000 | 0.000 | 0.000 | |
## | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.038 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | |
## --------------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|
## O | 0 | 0 | 1 | 3 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 216 | 0 | 3 | 0 | 0 | 0 | 2 | 0 | 0 | 0 | 0 | 0 | 225 |
## | 0.000 | 0.000 | 0.004 | 0.013 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.960 | 0.000 | 0.013 | 0.000 | 0.000 | 0.000 | 0.009 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.038 |
## | 0.000 | 0.000 | 0.005 | 0.012 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.935 | 0.000 | 0.013 | 0.000 | 0.000 | 0.000 | 0.008 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | |
## | 0.000 | 0.000 | 0.000 | 0.001 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.036 | 0.000 | 0.001 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | |
## --------------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|
## P | 0 | 1 | 0 | 1 | 0 | 8 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 229 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 1 | 0 | 240 |
## | 0.000 | 0.004 | 0.000 | 0.004 | 0.000 | 0.033 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.954 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.004 | 0.000 | 0.040 |
## | 0.000 | 0.004 | 0.000 | 0.004 | 0.000 | 0.036 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.939 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.004 | 0.000 | |
## | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.001 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.038 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | |
## --------------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|
## Q | 0 | 0 | 0 | 0 | 0 | 0 | 1 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 4 | 2 | 225 | 2 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 234 |
## | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.004 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.017 | 0.009 | 0.962 | 0.009 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.039 |
## | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.004 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.017 | 0.008 | 0.978 | 0.008 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | |
## | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.001 | 0.000 | 0.038 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | |
## --------------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|
## R | 0 | 8 | 0 | 1 | 0 | 0 | 0 | 1 | 0 | 0 | 3 | 0 | 0 | 0 | 0 | 0 | 0 | 213 | 0 | 0 | 1 | 0 | 0 | 0 | 0 | 0 | 227 |
## | 0.000 | 0.035 | 0.000 | 0.004 | 0.000 | 0.000 | 0.000 | 0.004 | 0.000 | 0.000 | 0.013 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.938 | 0.000 | 0.000 | 0.004 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.038 |
## | 0.000 | 0.035 | 0.000 | 0.004 | 0.000 | 0.000 | 0.000 | 0.005 | 0.000 | 0.000 | 0.014 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.866 | 0.000 | 0.000 | 0.004 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | |
## | 0.000 | 0.001 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.001 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.036 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | |
## --------------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|
## S | 0 | 1 | 0 | 0 | 2 | 0 | 0 | 0 | 1 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 218 | 0 | 1 | 0 | 0 | 0 | 0 | 1 | 224 |
## | 0.000 | 0.004 | 0.000 | 0.000 | 0.009 | 0.000 | 0.000 | 0.000 | 0.004 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.973 | 0.000 | 0.004 | 0.000 | 0.000 | 0.000 | 0.000 | 0.004 | 0.037 |
## | 0.000 | 0.004 | 0.000 | 0.000 | 0.009 | 0.000 | 0.000 | 0.000 | 0.005 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.986 | 0.000 | 0.004 | 0.000 | 0.000 | 0.000 | 0.000 | 0.004 | |
## | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.036 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | |
## --------------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|
## T | 0 | 0 | 1 | 1 | 0 | 2 | 0 | 1 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 227 | 0 | 0 | 0 | 0 | 5 | 1 | 238 |
## | 0.000 | 0.000 | 0.004 | 0.004 | 0.000 | 0.008 | 0.000 | 0.004 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.954 | 0.000 | 0.000 | 0.000 | 0.000 | 0.021 | 0.004 | 0.040 |
## | 0.000 | 0.000 | 0.005 | 0.004 | 0.000 | 0.009 | 0.000 | 0.005 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.970 | 0.000 | 0.000 | 0.000 | 0.000 | 0.021 | 0.004 | |
## | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.038 | 0.000 | 0.000 | 0.000 | 0.000 | 0.001 | 0.000 | |
## --------------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|
## U | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 1 | 0 | 2 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 240 | 0 | 0 | 0 | 0 | 0 | 243 |
## | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.004 | 0.000 | 0.008 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.988 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.041 |
## | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.005 | 0.000 | 0.008 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.980 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | |
## | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.040 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | |
## --------------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|
## V | 0 | 3 | 0 | 0 | 0 | 1 | 0 | 0 | 0 | 0 | 0 | 0 | 1 | 1 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 221 | 0 | 0 | 2 | 0 | 229 |
## | 0.000 | 0.013 | 0.000 | 0.000 | 0.000 | 0.004 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.004 | 0.004 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.965 | 0.000 | 0.000 | 0.009 | 0.000 | 0.038 |
## | 0.000 | 0.013 | 0.000 | 0.000 | 0.000 | 0.005 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.004 | 0.004 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.978 | 0.000 | 0.000 | 0.008 | 0.000 | |
## | 0.000 | 0.001 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.037 | 0.000 | 0.000 | 0.000 | 0.000 | |
## --------------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|
## W | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 1 | 0 | 0 | 0 | 0 | 1 | 0 | 1 | 0 | 0 | 0 | 0 | 0 | 1 | 0 | 221 | 0 | 0 | 0 | 225 |
## | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.004 | 0.000 | 0.000 | 0.000 | 0.000 | 0.004 | 0.000 | 0.004 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.004 | 0.000 | 0.982 | 0.000 | 0.000 | 0.000 | 0.038 |
## | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.005 | 0.000 | 0.000 | 0.000 | 0.000 | 0.004 | 0.000 | 0.004 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.004 | 0.000 | 0.978 | 0.000 | 0.000 | 0.000 | |
## | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.037 | 0.000 | 0.000 | 0.000 | |
## --------------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|
## X | 1 | 0 | 1 | 0 | 3 | 0 | 0 | 0 | 1 | 0 | 5 | 0 | 0 | 0 | 0 | 0 | 0 | 2 | 0 | 0 | 0 | 0 | 0 | 221 | 0 | 2 | 236 |
## | 0.004 | 0.000 | 0.004 | 0.000 | 0.013 | 0.000 | 0.000 | 0.000 | 0.004 | 0.000 | 0.021 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.008 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.936 | 0.000 | 0.008 | 0.039 |
## | 0.004 | 0.000 | 0.005 | 0.000 | 0.013 | 0.000 | 0.000 | 0.000 | 0.005 | 0.000 | 0.023 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.008 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.965 | 0.000 | 0.009 | |
## | 0.000 | 0.000 | 0.000 | 0.000 | 0.001 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.001 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.037 | 0.000 | 0.000 | |
## --------------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|
## Y | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 2 | 0 | 0 | 0 | 0 | 233 | 0 | 235 |
## | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.009 | 0.000 | 0.000 | 0.000 | 0.000 | 0.991 | 0.000 | 0.039 |
## | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.009 | 0.000 | 0.000 | 0.000 | 0.000 | 0.959 | 0.000 | |
## | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.039 | 0.000 | |
## --------------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|
## Z | 0 | 0 | 0 | 0 | 1 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 2 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 217 | 220 |
## | 0.000 | 0.000 | 0.000 | 0.000 | 0.005 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.009 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.986 | 0.037 |
## | 0.000 | 0.000 | 0.000 | 0.000 | 0.004 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.009 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.952 | |
## | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.036 | |
## --------------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|
## Column Total | 232 | 230 | 216 | 248 | 232 | 221 | 231 | 207 | 221 | 225 | 220 | 227 | 236 | 236 | 231 | 244 | 230 | 246 | 221 | 234 | 245 | 226 | 226 | 229 | 243 | 228 | 5985 |
## | 0.039 | 0.038 | 0.036 | 0.041 | 0.039 | 0.037 | 0.039 | 0.035 | 0.037 | 0.038 | 0.037 | 0.038 | 0.039 | 0.039 | 0.039 | 0.041 | 0.038 | 0.041 | 0.037 | 0.039 | 0.041 | 0.038 | 0.038 | 0.038 | 0.041 | 0.038 | |
## --------------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|
##
## # Calculate prediction success rate for all letters
mean(diag(t$prop.col))## [1] 0.951506# Check prediction success rate for each letter
diag(t$prop.col)## A B C D E F G H
## 0.9913793 0.9130435 0.9722222 0.9395161 0.9353448 0.9321267 0.9567100 0.8985507
## I J K L M N O P
## 0.9547511 0.9422222 0.8863636 0.9867841 0.9788136 0.9661017 0.9350649 0.9385246
## Q R S T U V W X
## 0.9782609 0.8658537 0.9864253 0.9700855 0.9795918 0.9778761 0.9778761 0.9650655
## Y Z
## 0.9588477 0.9517544The accuracy obtained from the nearest neighbors classification algorithm is around 94,8%. I’ve used 3 nearest neighbors (k=3) for building the final model. Alternative number of neighbors (4,5,6,7 etc) produced slightly lower accuracy rate (94.3% - 94,7%). The model is better at predicting some letters, such as L, M, U, Z with more than 98% accuracy rate. On the other hand, other letters such as H, R, B are more difficult to predict (<90% accuracy rate). In the plot below you can see each letter accuracy rate of the final model.
# Create a data set with the accuracy rate of each letter
letters <- diag(t$prop.col) %>%
as.data.frame()
# Make the appropriate transformations in order to plot
letters$letter <- rownames(letters)
# Plot the accuracy rate per letter
ggplot(letters)+
geom_bar(aes(x = letter, y = .), stat="identity", fill = "steelblue2", alpha = .7)+
scale_y_continuous(labels=percent) +
labs(title = "Accuracy rate % per letter (K-NN algorithm)",
x = "", y = "Accuracy rate %", subtittle = "")+
theme_fivethirtyeight()
Singular Value Decomposition (SVD)
Since i discovered that there are some correlated variables in the data set, i applied the SVD algorithm in order to create a data set with uncorrelated variables. SVD is an algebraic tool that has many potential uses such as:
- Dimensionality reduction
- Indexing (LSI)
- Visualization/clustering of high-dimensional objects
- Similarity computations/outlier detection
- Rule mining, treatment of missing/wrong values
Below i created a new dataset with the new SVD variables and build a new model using k-NN algorithm.
# Create an in Dataset
letter.svd <- svd(letter[,2:17])
# Create a dataset with the SVD values
letter2 <- as_data_frame(cbind(letter[,1],letter.svd$u))
# Keep specific SVD variables
letter2 <- letter2[,1:8]
# Create a vector with the 70% of the dataset with respect to letter
set.seed(10)
inTrain = createDataPartition(letter2$lettr, p = .7)[[1]]
# Assign the 70% of observations to training data
training <- letter2[inTrain, -1]
training.lettr <- c(t(letter2[inTrain, 1]))
# Assign the remaining 30 % of observations to testing data
testing <- letter2[-inTrain, -1]
testing.lettr <- c(t(letter2[-inTrain, 1]))
# Run knn algorithm on training dataset
# Create the knn model
knn_model.svd <- knn(train = training, test = testing, cl = training.lettr, k = 3)
# Create a table in order to check the performance of the classification model
s <- CrossTable(x = testing.lettr,y = knn_model.svd,
prop.chisq=FALSE)##
##
## Cell Contents
## |-------------------------|
## | N |
## | N / Row Total |
## | N / Col Total |
## | N / Table Total |
## |-------------------------|
##
##
## Total Observations in Table: 5985
##
##
## | knn_model.svd
## testing.lettr | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z | Row Total |
## --------------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|
## A | 210 | 0 | 0 | 4 | 0 | 0 | 1 | 3 | 0 | 1 | 0 | 0 | 2 | 4 | 0 | 1 | 1 | 4 | 0 | 1 | 0 | 0 | 0 | 0 | 2 | 2 | 236 |
## | 0.890 | 0.000 | 0.000 | 0.017 | 0.000 | 0.000 | 0.004 | 0.013 | 0.000 | 0.004 | 0.000 | 0.000 | 0.008 | 0.017 | 0.000 | 0.004 | 0.004 | 0.017 | 0.000 | 0.004 | 0.000 | 0.000 | 0.000 | 0.000 | 0.008 | 0.008 | 0.039 |
## | 0.871 | 0.000 | 0.000 | 0.015 | 0.000 | 0.000 | 0.004 | 0.014 | 0.000 | 0.004 | 0.000 | 0.000 | 0.008 | 0.018 | 0.000 | 0.004 | 0.004 | 0.016 | 0.000 | 0.004 | 0.000 | 0.000 | 0.000 | 0.000 | 0.008 | 0.009 | |
## | 0.035 | 0.000 | 0.000 | 0.001 | 0.000 | 0.000 | 0.000 | 0.001 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.001 | 0.000 | 0.000 | 0.000 | 0.001 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | |
## --------------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|
## B | 1 | 173 | 0 | 4 | 1 | 4 | 4 | 7 | 1 | 0 | 1 | 0 | 0 | 0 | 1 | 0 | 1 | 18 | 7 | 0 | 0 | 1 | 0 | 1 | 3 | 1 | 229 |
## | 0.004 | 0.755 | 0.000 | 0.017 | 0.004 | 0.017 | 0.017 | 0.031 | 0.004 | 0.000 | 0.004 | 0.000 | 0.000 | 0.000 | 0.004 | 0.000 | 0.004 | 0.079 | 0.031 | 0.000 | 0.000 | 0.004 | 0.000 | 0.004 | 0.013 | 0.004 | 0.038 |
## | 0.004 | 0.718 | 0.000 | 0.015 | 0.005 | 0.018 | 0.017 | 0.032 | 0.005 | 0.000 | 0.005 | 0.000 | 0.000 | 0.000 | 0.005 | 0.000 | 0.004 | 0.071 | 0.031 | 0.000 | 0.000 | 0.004 | 0.000 | 0.004 | 0.012 | 0.005 | |
## | 0.000 | 0.029 | 0.000 | 0.001 | 0.000 | 0.001 | 0.001 | 0.001 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.003 | 0.001 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.001 | 0.000 | |
## --------------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|
## C | 0 | 0 | 185 | 1 | 4 | 1 | 7 | 3 | 0 | 0 | 3 | 3 | 1 | 0 | 2 | 0 | 2 | 0 | 1 | 1 | 6 | 0 | 0 | 0 | 0 | 0 | 220 |
## | 0.000 | 0.000 | 0.841 | 0.005 | 0.018 | 0.005 | 0.032 | 0.014 | 0.000 | 0.000 | 0.014 | 0.014 | 0.005 | 0.000 | 0.009 | 0.000 | 0.009 | 0.000 | 0.005 | 0.005 | 0.027 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.037 |
## | 0.000 | 0.000 | 0.920 | 0.004 | 0.019 | 0.005 | 0.030 | 0.014 | 0.000 | 0.000 | 0.014 | 0.013 | 0.004 | 0.000 | 0.009 | 0.000 | 0.008 | 0.000 | 0.004 | 0.004 | 0.024 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | |
## | 0.000 | 0.000 | 0.031 | 0.000 | 0.001 | 0.000 | 0.001 | 0.001 | 0.000 | 0.000 | 0.001 | 0.001 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.001 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | |
## --------------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|
## D | 5 | 3 | 1 | 183 | 0 | 1 | 5 | 6 | 0 | 2 | 3 | 2 | 0 | 1 | 5 | 0 | 0 | 16 | 2 | 1 | 0 | 0 | 0 | 4 | 0 | 1 | 241 |
## | 0.021 | 0.012 | 0.004 | 0.759 | 0.000 | 0.004 | 0.021 | 0.025 | 0.000 | 0.008 | 0.012 | 0.008 | 0.000 | 0.004 | 0.021 | 0.000 | 0.000 | 0.066 | 0.008 | 0.004 | 0.000 | 0.000 | 0.000 | 0.017 | 0.000 | 0.004 | 0.040 |
## | 0.021 | 0.012 | 0.005 | 0.707 | 0.000 | 0.005 | 0.022 | 0.027 | 0.000 | 0.009 | 0.014 | 0.009 | 0.000 | 0.004 | 0.023 | 0.000 | 0.000 | 0.063 | 0.009 | 0.004 | 0.000 | 0.000 | 0.000 | 0.016 | 0.000 | 0.005 | |
## | 0.001 | 0.001 | 0.000 | 0.031 | 0.000 | 0.000 | 0.001 | 0.001 | 0.000 | 0.000 | 0.001 | 0.000 | 0.000 | 0.000 | 0.001 | 0.000 | 0.000 | 0.003 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.001 | 0.000 | 0.000 | |
## --------------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|
## E | 0 | 4 | 5 | 1 | 174 | 1 | 3 | 1 | 4 | 1 | 4 | 2 | 0 | 0 | 0 | 0 | 1 | 3 | 9 | 1 | 0 | 2 | 0 | 12 | 0 | 2 | 230 |
## | 0.000 | 0.017 | 0.022 | 0.004 | 0.757 | 0.004 | 0.013 | 0.004 | 0.017 | 0.004 | 0.017 | 0.009 | 0.000 | 0.000 | 0.000 | 0.000 | 0.004 | 0.013 | 0.039 | 0.004 | 0.000 | 0.009 | 0.000 | 0.052 | 0.000 | 0.009 | 0.038 |
## | 0.000 | 0.017 | 0.025 | 0.004 | 0.821 | 0.005 | 0.013 | 0.005 | 0.018 | 0.004 | 0.019 | 0.009 | 0.000 | 0.000 | 0.000 | 0.000 | 0.004 | 0.012 | 0.040 | 0.004 | 0.000 | 0.009 | 0.000 | 0.049 | 0.000 | 0.009 | |
## | 0.000 | 0.001 | 0.001 | 0.000 | 0.029 | 0.000 | 0.001 | 0.000 | 0.001 | 0.000 | 0.001 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.001 | 0.002 | 0.000 | 0.000 | 0.000 | 0.000 | 0.002 | 0.000 | 0.000 | |
## --------------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|
## F | 3 | 6 | 0 | 4 | 0 | 182 | 0 | 1 | 2 | 2 | 1 | 0 | 1 | 2 | 0 | 10 | 0 | 0 | 1 | 4 | 0 | 0 | 2 | 1 | 10 | 0 | 232 |
## | 0.013 | 0.026 | 0.000 | 0.017 | 0.000 | 0.784 | 0.000 | 0.004 | 0.009 | 0.009 | 0.004 | 0.000 | 0.004 | 0.009 | 0.000 | 0.043 | 0.000 | 0.000 | 0.004 | 0.017 | 0.000 | 0.000 | 0.009 | 0.004 | 0.043 | 0.000 | 0.039 |
## | 0.012 | 0.025 | 0.000 | 0.015 | 0.000 | 0.835 | 0.000 | 0.005 | 0.009 | 0.009 | 0.005 | 0.000 | 0.004 | 0.009 | 0.000 | 0.042 | 0.000 | 0.000 | 0.004 | 0.018 | 0.000 | 0.000 | 0.009 | 0.004 | 0.040 | 0.000 | |
## | 0.001 | 0.001 | 0.000 | 0.001 | 0.000 | 0.030 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.002 | 0.000 | 0.000 | 0.000 | 0.001 | 0.000 | 0.000 | 0.000 | 0.000 | 0.002 | 0.000 | |
## --------------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|
## G | 0 | 4 | 1 | 3 | 3 | 0 | 179 | 4 | 0 | 1 | 1 | 1 | 4 | 0 | 5 | 0 | 5 | 4 | 2 | 0 | 1 | 0 | 1 | 3 | 9 | 0 | 231 |
## | 0.000 | 0.017 | 0.004 | 0.013 | 0.013 | 0.000 | 0.775 | 0.017 | 0.000 | 0.004 | 0.004 | 0.004 | 0.017 | 0.000 | 0.022 | 0.000 | 0.022 | 0.017 | 0.009 | 0.000 | 0.004 | 0.000 | 0.004 | 0.013 | 0.039 | 0.000 | 0.039 |
## | 0.000 | 0.017 | 0.005 | 0.012 | 0.014 | 0.000 | 0.772 | 0.018 | 0.000 | 0.004 | 0.005 | 0.004 | 0.017 | 0.000 | 0.023 | 0.000 | 0.021 | 0.016 | 0.009 | 0.000 | 0.004 | 0.000 | 0.004 | 0.012 | 0.036 | 0.000 | |
## | 0.000 | 0.001 | 0.000 | 0.001 | 0.001 | 0.000 | 0.030 | 0.001 | 0.000 | 0.000 | 0.000 | 0.000 | 0.001 | 0.000 | 0.001 | 0.000 | 0.001 | 0.001 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.001 | 0.002 | 0.000 | |
## --------------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|
## H | 4 | 3 | 1 | 5 | 2 | 0 | 8 | 154 | 0 | 1 | 4 | 1 | 0 | 5 | 1 | 1 | 7 | 8 | 0 | 1 | 3 | 2 | 0 | 5 | 4 | 0 | 220 |
## | 0.018 | 0.014 | 0.005 | 0.023 | 0.009 | 0.000 | 0.036 | 0.700 | 0.000 | 0.005 | 0.018 | 0.005 | 0.000 | 0.023 | 0.005 | 0.005 | 0.032 | 0.036 | 0.000 | 0.005 | 0.014 | 0.009 | 0.000 | 0.023 | 0.018 | 0.000 | 0.037 |
## | 0.017 | 0.012 | 0.005 | 0.019 | 0.009 | 0.000 | 0.034 | 0.703 | 0.000 | 0.004 | 0.019 | 0.004 | 0.000 | 0.022 | 0.005 | 0.004 | 0.030 | 0.032 | 0.000 | 0.004 | 0.012 | 0.009 | 0.000 | 0.020 | 0.016 | 0.000 | |
## | 0.001 | 0.001 | 0.000 | 0.001 | 0.000 | 0.000 | 0.001 | 0.026 | 0.000 | 0.000 | 0.001 | 0.000 | 0.000 | 0.001 | 0.000 | 0.000 | 0.001 | 0.001 | 0.000 | 0.000 | 0.001 | 0.000 | 0.000 | 0.001 | 0.001 | 0.000 | |
## --------------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|
## I | 0 | 2 | 0 | 2 | 0 | 1 | 0 | 0 | 200 | 13 | 0 | 0 | 0 | 0 | 0 | 1 | 0 | 1 | 1 | 0 | 0 | 0 | 0 | 3 | 1 | 1 | 226 |
## | 0.000 | 0.009 | 0.000 | 0.009 | 0.000 | 0.004 | 0.000 | 0.000 | 0.885 | 0.058 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.004 | 0.000 | 0.004 | 0.004 | 0.000 | 0.000 | 0.000 | 0.000 | 0.013 | 0.004 | 0.004 | 0.038 |
## | 0.000 | 0.008 | 0.000 | 0.008 | 0.000 | 0.005 | 0.000 | 0.000 | 0.917 | 0.057 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.004 | 0.000 | 0.004 | 0.004 | 0.000 | 0.000 | 0.000 | 0.000 | 0.012 | 0.004 | 0.005 | |
## | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.033 | 0.002 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.001 | 0.000 | 0.000 | |
## --------------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|
## J | 0 | 2 | 0 | 6 | 0 | 1 | 1 | 0 | 9 | 192 | 0 | 0 | 0 | 0 | 4 | 4 | 0 | 4 | 0 | 0 | 0 | 0 | 0 | 1 | 0 | 0 | 224 |
## | 0.000 | 0.009 | 0.000 | 0.027 | 0.000 | 0.004 | 0.004 | 0.000 | 0.040 | 0.857 | 0.000 | 0.000 | 0.000 | 0.000 | 0.018 | 0.018 | 0.000 | 0.018 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.004 | 0.000 | 0.000 | 0.037 |
## | 0.000 | 0.008 | 0.000 | 0.023 | 0.000 | 0.005 | 0.004 | 0.000 | 0.041 | 0.838 | 0.000 | 0.000 | 0.000 | 0.000 | 0.019 | 0.017 | 0.000 | 0.016 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.004 | 0.000 | 0.000 | |
## | 0.000 | 0.000 | 0.000 | 0.001 | 0.000 | 0.000 | 0.000 | 0.000 | 0.002 | 0.032 | 0.000 | 0.000 | 0.000 | 0.000 | 0.001 | 0.001 | 0.000 | 0.001 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | |
## --------------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|
## K | 1 | 3 | 0 | 3 | 6 | 0 | 1 | 13 | 0 | 0 | 174 | 1 | 0 | 0 | 0 | 0 | 0 | 4 | 1 | 0 | 3 | 0 | 0 | 8 | 2 | 1 | 221 |
## | 0.005 | 0.014 | 0.000 | 0.014 | 0.027 | 0.000 | 0.005 | 0.059 | 0.000 | 0.000 | 0.787 | 0.005 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.018 | 0.005 | 0.000 | 0.014 | 0.000 | 0.000 | 0.036 | 0.009 | 0.005 | 0.037 |
## | 0.004 | 0.012 | 0.000 | 0.012 | 0.028 | 0.000 | 0.004 | 0.059 | 0.000 | 0.000 | 0.813 | 0.004 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.016 | 0.004 | 0.000 | 0.012 | 0.000 | 0.000 | 0.033 | 0.008 | 0.005 | |
## | 0.000 | 0.001 | 0.000 | 0.001 | 0.001 | 0.000 | 0.000 | 0.002 | 0.000 | 0.000 | 0.029 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.001 | 0.000 | 0.000 | 0.001 | 0.000 | 0.000 | 0.001 | 0.000 | 0.000 | |
## --------------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|
## L | 1 | 0 | 1 | 0 | 2 | 0 | 1 | 1 | 0 | 1 | 1 | 205 | 0 | 0 | 0 | 0 | 7 | 2 | 0 | 1 | 1 | 0 | 0 | 4 | 0 | 0 | 228 |
## | 0.004 | 0.000 | 0.004 | 0.000 | 0.009 | 0.000 | 0.004 | 0.004 | 0.000 | 0.004 | 0.004 | 0.899 | 0.000 | 0.000 | 0.000 | 0.000 | 0.031 | 0.009 | 0.000 | 0.004 | 0.004 | 0.000 | 0.000 | 0.018 | 0.000 | 0.000 | 0.038 |
## | 0.004 | 0.000 | 0.005 | 0.000 | 0.009 | 0.000 | 0.004 | 0.005 | 0.000 | 0.004 | 0.005 | 0.915 | 0.000 | 0.000 | 0.000 | 0.000 | 0.030 | 0.008 | 0.000 | 0.004 | 0.004 | 0.000 | 0.000 | 0.016 | 0.000 | 0.000 | |
## | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.034 | 0.000 | 0.000 | 0.000 | 0.000 | 0.001 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.001 | 0.000 | 0.000 | |
## --------------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|
## M | 2 | 0 | 0 | 0 | 0 | 0 | 0 | 2 | 0 | 0 | 1 | 0 | 220 | 2 | 1 | 1 | 0 | 0 | 0 | 0 | 0 | 3 | 5 | 0 | 0 | 0 | 237 |
## | 0.008 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.008 | 0.000 | 0.000 | 0.004 | 0.000 | 0.928 | 0.008 | 0.004 | 0.004 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.013 | 0.021 | 0.000 | 0.000 | 0.000 | 0.040 |
## | 0.008 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.009 | 0.000 | 0.000 | 0.005 | 0.000 | 0.921 | 0.009 | 0.005 | 0.004 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.013 | 0.022 | 0.000 | 0.000 | 0.000 | |
## | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.037 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.001 | 0.001 | 0.000 | 0.000 | 0.000 | |
## --------------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|
## N | 2 | 1 | 0 | 1 | 0 | 1 | 1 | 2 | 0 | 1 | 0 | 0 | 1 | 206 | 3 | 1 | 1 | 0 | 0 | 0 | 4 | 1 | 4 | 2 | 2 | 0 | 234 |
## | 0.009 | 0.004 | 0.000 | 0.004 | 0.000 | 0.004 | 0.004 | 0.009 | 0.000 | 0.004 | 0.000 | 0.000 | 0.004 | 0.880 | 0.013 | 0.004 | 0.004 | 0.000 | 0.000 | 0.000 | 0.017 | 0.004 | 0.017 | 0.009 | 0.009 | 0.000 | 0.039 |
## | 0.008 | 0.004 | 0.000 | 0.004 | 0.000 | 0.005 | 0.004 | 0.009 | 0.000 | 0.004 | 0.000 | 0.000 | 0.004 | 0.907 | 0.014 | 0.004 | 0.004 | 0.000 | 0.000 | 0.000 | 0.016 | 0.004 | 0.017 | 0.008 | 0.008 | 0.000 | |
## | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.034 | 0.001 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.001 | 0.000 | 0.001 | 0.000 | 0.000 | 0.000 | |
## --------------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|
## O | 3 | 1 | 2 | 8 | 0 | 0 | 5 | 3 | 0 | 2 | 0 | 0 | 0 | 1 | 172 | 1 | 10 | 11 | 0 | 0 | 2 | 1 | 2 | 1 | 0 | 0 | 225 |
## | 0.013 | 0.004 | 0.009 | 0.036 | 0.000 | 0.000 | 0.022 | 0.013 | 0.000 | 0.009 | 0.000 | 0.000 | 0.000 | 0.004 | 0.764 | 0.004 | 0.044 | 0.049 | 0.000 | 0.000 | 0.009 | 0.004 | 0.009 | 0.004 | 0.000 | 0.000 | 0.038 |
## | 0.012 | 0.004 | 0.010 | 0.031 | 0.000 | 0.000 | 0.022 | 0.014 | 0.000 | 0.009 | 0.000 | 0.000 | 0.000 | 0.004 | 0.796 | 0.004 | 0.042 | 0.043 | 0.000 | 0.000 | 0.008 | 0.004 | 0.009 | 0.004 | 0.000 | 0.000 | |
## | 0.001 | 0.000 | 0.000 | 0.001 | 0.000 | 0.000 | 0.001 | 0.001 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.029 | 0.000 | 0.002 | 0.002 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | |
## --------------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|
## P | 4 | 1 | 0 | 2 | 0 | 11 | 2 | 0 | 0 | 4 | 1 | 0 | 0 | 0 | 1 | 206 | 2 | 1 | 0 | 1 | 1 | 0 | 0 | 0 | 3 | 0 | 240 |
## | 0.017 | 0.004 | 0.000 | 0.008 | 0.000 | 0.046 | 0.008 | 0.000 | 0.000 | 0.017 | 0.004 | 0.000 | 0.000 | 0.000 | 0.004 | 0.858 | 0.008 | 0.004 | 0.000 | 0.004 | 0.004 | 0.000 | 0.000 | 0.000 | 0.013 | 0.000 | 0.040 |
## | 0.017 | 0.004 | 0.000 | 0.008 | 0.000 | 0.050 | 0.009 | 0.000 | 0.000 | 0.017 | 0.005 | 0.000 | 0.000 | 0.000 | 0.005 | 0.873 | 0.008 | 0.004 | 0.000 | 0.004 | 0.004 | 0.000 | 0.000 | 0.000 | 0.012 | 0.000 | |
## | 0.001 | 0.000 | 0.000 | 0.000 | 0.000 | 0.002 | 0.000 | 0.000 | 0.000 | 0.001 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.034 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.001 | 0.000 | |
## --------------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|
## Q | 0 | 3 | 0 | 1 | 0 | 0 | 3 | 3 | 0 | 0 | 1 | 2 | 0 | 1 | 13 | 2 | 193 | 3 | 2 | 2 | 2 | 0 | 0 | 0 | 2 | 1 | 234 |
## | 0.000 | 0.013 | 0.000 | 0.004 | 0.000 | 0.000 | 0.013 | 0.013 | 0.000 | 0.000 | 0.004 | 0.009 | 0.000 | 0.004 | 0.056 | 0.009 | 0.825 | 0.013 | 0.009 | 0.009 | 0.009 | 0.000 | 0.000 | 0.000 | 0.009 | 0.004 | 0.039 |
## | 0.000 | 0.012 | 0.000 | 0.004 | 0.000 | 0.000 | 0.013 | 0.014 | 0.000 | 0.000 | 0.005 | 0.009 | 0.000 | 0.004 | 0.060 | 0.008 | 0.814 | 0.012 | 0.009 | 0.009 | 0.008 | 0.000 | 0.000 | 0.000 | 0.008 | 0.005 | |
## | 0.000 | 0.001 | 0.000 | 0.000 | 0.000 | 0.000 | 0.001 | 0.001 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.002 | 0.000 | 0.032 | 0.001 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | |
## --------------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|
## R | 2 | 14 | 2 | 15 | 0 | 0 | 4 | 8 | 0 | 1 | 2 | 2 | 0 | 0 | 2 | 1 | 2 | 168 | 2 | 1 | 1 | 0 | 0 | 0 | 0 | 0 | 227 |
## | 0.009 | 0.062 | 0.009 | 0.066 | 0.000 | 0.000 | 0.018 | 0.035 | 0.000 | 0.004 | 0.009 | 0.009 | 0.000 | 0.000 | 0.009 | 0.004 | 0.009 | 0.740 | 0.009 | 0.004 | 0.004 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.038 |
## | 0.008 | 0.058 | 0.010 | 0.058 | 0.000 | 0.000 | 0.017 | 0.037 | 0.000 | 0.004 | 0.009 | 0.009 | 0.000 | 0.000 | 0.009 | 0.004 | 0.008 | 0.664 | 0.009 | 0.004 | 0.004 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | |
## | 0.000 | 0.002 | 0.000 | 0.003 | 0.000 | 0.000 | 0.001 | 0.001 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.028 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | |
## --------------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|
## S | 1 | 8 | 0 | 3 | 5 | 4 | 1 | 1 | 1 | 3 | 2 | 0 | 0 | 0 | 0 | 0 | 0 | 1 | 186 | 1 | 1 | 0 | 0 | 2 | 0 | 4 | 224 |
## | 0.004 | 0.036 | 0.000 | 0.013 | 0.022 | 0.018 | 0.004 | 0.004 | 0.004 | 0.013 | 0.009 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.004 | 0.830 | 0.004 | 0.004 | 0.000 | 0.000 | 0.009 | 0.000 | 0.018 | 0.037 |
## | 0.004 | 0.033 | 0.000 | 0.012 | 0.024 | 0.018 | 0.004 | 0.005 | 0.005 | 0.013 | 0.009 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.004 | 0.834 | 0.004 | 0.004 | 0.000 | 0.000 | 0.008 | 0.000 | 0.019 | |
## | 0.000 | 0.001 | 0.000 | 0.001 | 0.001 | 0.001 | 0.000 | 0.000 | 0.000 | 0.001 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.031 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.001 | |
## --------------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|
## T | 0 | 0 | 2 | 3 | 0 | 9 | 0 | 1 | 0 | 0 | 1 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 1 | 204 | 0 | 2 | 0 | 2 | 13 | 0 | 238 |
## | 0.000 | 0.000 | 0.008 | 0.013 | 0.000 | 0.038 | 0.000 | 0.004 | 0.000 | 0.000 | 0.004 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.004 | 0.857 | 0.000 | 0.008 | 0.000 | 0.008 | 0.055 | 0.000 | 0.040 |
## | 0.000 | 0.000 | 0.010 | 0.012 | 0.000 | 0.041 | 0.000 | 0.005 | 0.000 | 0.000 | 0.005 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.004 | 0.895 | 0.000 | 0.009 | 0.000 | 0.008 | 0.052 | 0.000 | |
## | 0.000 | 0.000 | 0.000 | 0.001 | 0.000 | 0.002 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.034 | 0.000 | 0.000 | 0.000 | 0.000 | 0.002 | 0.000 | |
## --------------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|
## U | 0 | 0 | 1 | 1 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 1 | 3 | 3 | 1 | 0 | 2 | 0 | 0 | 1 | 228 | 0 | 1 | 0 | 1 | 0 | 243 |
## | 0.000 | 0.000 | 0.004 | 0.004 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.004 | 0.012 | 0.012 | 0.004 | 0.000 | 0.008 | 0.000 | 0.000 | 0.004 | 0.938 | 0.000 | 0.004 | 0.000 | 0.004 | 0.000 | 0.041 |
## | 0.000 | 0.000 | 0.005 | 0.004 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.004 | 0.013 | 0.013 | 0.005 | 0.000 | 0.008 | 0.000 | 0.000 | 0.004 | 0.898 | 0.000 | 0.004 | 0.000 | 0.004 | 0.000 | |
## | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.001 | 0.001 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.038 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | |
## --------------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|
## V | 0 | 4 | 0 | 0 | 1 | 0 | 1 | 1 | 0 | 0 | 0 | 0 | 3 | 1 | 0 | 1 | 0 | 2 | 0 | 0 | 1 | 197 | 6 | 0 | 11 | 0 | 229 |
## | 0.000 | 0.017 | 0.000 | 0.000 | 0.004 | 0.000 | 0.004 | 0.004 | 0.000 | 0.000 | 0.000 | 0.000 | 0.013 | 0.004 | 0.000 | 0.004 | 0.000 | 0.009 | 0.000 | 0.000 | 0.004 | 0.860 | 0.026 | 0.000 | 0.048 | 0.000 | 0.038 |
## | 0.000 | 0.017 | 0.000 | 0.000 | 0.005 | 0.000 | 0.004 | 0.005 | 0.000 | 0.000 | 0.000 | 0.000 | 0.013 | 0.004 | 0.000 | 0.004 | 0.000 | 0.008 | 0.000 | 0.000 | 0.004 | 0.864 | 0.026 | 0.000 | 0.044 | 0.000 | |
## | 0.000 | 0.001 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.001 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.033 | 0.001 | 0.000 | 0.002 | 0.000 | |
## --------------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|
## W | 0 | 1 | 0 | 0 | 0 | 0 | 0 | 1 | 0 | 0 | 0 | 0 | 4 | 0 | 3 | 1 | 0 | 0 | 0 | 0 | 0 | 5 | 210 | 0 | 0 | 0 | 225 |
## | 0.000 | 0.004 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.004 | 0.000 | 0.000 | 0.000 | 0.000 | 0.018 | 0.000 | 0.013 | 0.004 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.022 | 0.933 | 0.000 | 0.000 | 0.000 | 0.038 |
## | 0.000 | 0.004 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.005 | 0.000 | 0.000 | 0.000 | 0.000 | 0.017 | 0.000 | 0.014 | 0.004 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.022 | 0.909 | 0.000 | 0.000 | 0.000 | |
## | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.001 | 0.000 | 0.001 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.001 | 0.035 | 0.000 | 0.000 | 0.000 | |
## --------------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|
## X | 0 | 1 | 0 | 1 | 11 | 0 | 0 | 4 | 0 | 3 | 11 | 3 | 0 | 0 | 0 | 0 | 0 | 0 | 5 | 0 | 0 | 0 | 0 | 192 | 1 | 4 | 236 |
## | 0.000 | 0.004 | 0.000 | 0.004 | 0.047 | 0.000 | 0.000 | 0.017 | 0.000 | 0.013 | 0.047 | 0.013 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.021 | 0.000 | 0.000 | 0.000 | 0.000 | 0.814 | 0.004 | 0.017 | 0.039 |
## | 0.000 | 0.004 | 0.000 | 0.004 | 0.052 | 0.000 | 0.000 | 0.018 | 0.000 | 0.013 | 0.051 | 0.013 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.022 | 0.000 | 0.000 | 0.000 | 0.000 | 0.787 | 0.004 | 0.019 | |
## | 0.000 | 0.000 | 0.000 | 0.000 | 0.002 | 0.000 | 0.000 | 0.001 | 0.000 | 0.001 | 0.002 | 0.001 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.001 | 0.000 | 0.000 | 0.000 | 0.000 | 0.032 | 0.000 | 0.001 | |
## --------------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|
## Y | 2 | 2 | 0 | 3 | 1 | 1 | 4 | 0 | 0 | 1 | 3 | 0 | 0 | 1 | 2 | 5 | 0 | 2 | 0 | 8 | 0 | 14 | 0 | 0 | 186 | 0 | 235 |
## | 0.009 | 0.009 | 0.000 | 0.013 | 0.004 | 0.004 | 0.017 | 0.000 | 0.000 | 0.004 | 0.013 | 0.000 | 0.000 | 0.004 | 0.009 | 0.021 | 0.000 | 0.009 | 0.000 | 0.034 | 0.000 | 0.060 | 0.000 | 0.000 | 0.791 | 0.000 | 0.039 |
## | 0.008 | 0.008 | 0.000 | 0.012 | 0.005 | 0.005 | 0.017 | 0.000 | 0.000 | 0.004 | 0.014 | 0.000 | 0.000 | 0.004 | 0.009 | 0.021 | 0.000 | 0.008 | 0.000 | 0.035 | 0.000 | 0.061 | 0.000 | 0.000 | 0.744 | 0.000 | |
## | 0.000 | 0.000 | 0.000 | 0.001 | 0.000 | 0.000 | 0.001 | 0.000 | 0.000 | 0.000 | 0.001 | 0.000 | 0.000 | 0.000 | 0.000 | 0.001 | 0.000 | 0.000 | 0.000 | 0.001 | 0.000 | 0.002 | 0.000 | 0.000 | 0.031 | 0.000 | |
## --------------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|
## Z | 0 | 5 | 0 | 5 | 2 | 1 | 1 | 0 | 1 | 0 | 0 | 1 | 0 | 0 | 0 | 0 | 3 | 1 | 3 | 0 | 0 | 0 | 0 | 3 | 0 | 194 | 220 |
## | 0.000 | 0.023 | 0.000 | 0.023 | 0.009 | 0.005 | 0.005 | 0.000 | 0.005 | 0.000 | 0.000 | 0.005 | 0.000 | 0.000 | 0.000 | 0.000 | 0.014 | 0.005 | 0.014 | 0.000 | 0.000 | 0.000 | 0.000 | 0.014 | 0.000 | 0.882 | 0.037 |
## | 0.000 | 0.021 | 0.000 | 0.019 | 0.009 | 0.005 | 0.004 | 0.000 | 0.005 | 0.000 | 0.000 | 0.004 | 0.000 | 0.000 | 0.000 | 0.000 | 0.013 | 0.004 | 0.013 | 0.000 | 0.000 | 0.000 | 0.000 | 0.012 | 0.000 | 0.919 | |
## | 0.000 | 0.001 | 0.000 | 0.001 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.001 | 0.000 | 0.001 | 0.000 | 0.000 | 0.000 | 0.000 | 0.001 | 0.000 | 0.032 | |
## --------------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|
## Column Total | 241 | 241 | 201 | 259 | 212 | 218 | 232 | 219 | 218 | 229 | 214 | 224 | 239 | 227 | 216 | 236 | 237 | 253 | 223 | 228 | 254 | 228 | 231 | 244 | 250 | 211 | 5985 |
## | 0.040 | 0.040 | 0.034 | 0.043 | 0.035 | 0.036 | 0.039 | 0.037 | 0.036 | 0.038 | 0.036 | 0.037 | 0.040 | 0.038 | 0.036 | 0.039 | 0.040 | 0.042 | 0.037 | 0.038 | 0.042 | 0.038 | 0.039 | 0.041 | 0.042 | 0.035 | |
## --------------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|
##
## # Calculate prediction success rate for all letters
mean(diag(s$prop.col))## [1] 0.8329271# Check prediction success rate for each letter
diag(s$prop.col)## A B C D E F G H
## 0.8713693 0.7178423 0.9203980 0.7065637 0.8207547 0.8348624 0.7715517 0.7031963
## I J K L M N O P
## 0.9174312 0.8384279 0.8130841 0.9151786 0.9205021 0.9074890 0.7962963 0.8728814
## Q R S T U V W X
## 0.8143460 0.6640316 0.8340807 0.8947368 0.8976378 0.8640351 0.9090909 0.7868852
## Y Z
## 0.7440000 0.9194313Even when all SVD variables (16) were used, the prediction rate is around 93.4% (lower than the original of 94.8%). When we use less (e.g. 8), the prediction rate is getting worst (83.4%). It seems that SVD does not help us improve the prediction model. Almost all original variables have variability (add information) so they are useful for predicting the letter
Hierarchical clustering
In this part of the analysis the objective is to assign the cases into clusters. Cluster analysis or clustering is the task of grouping a set of objects in such a way that objects in the same group (called a cluster) are more similar (in some sense or another) to each other than to those in other groups (clusters). Finally i build a hierarchical clustering model in order to understand which letters are “closer” to others and maybe misclassified. We applied it on the pre-processed data set (letter.cluster).
# Tranform the summary dataset to a dataframe
letter.cluster <- as.data.frame(letter.cluster)
# Assign letters variable as rownames
rownames(letter.cluster) <- letter.cluster$lettr
# Run the hierarchical clustering
clusters <- hclust(dist(letter.cluster))At the plot below you can see the hierarchical clustering output. When we compare these results with the misclassification rates of the original model, it seems that for some letters like “B”, “D”, “F” it looks relevant. For other letters such “A”, “C” don’t match.
# Plot the dendogram
ggdendrogram(clusters, rotate = TRUE, size = 2)+
labs(title = "Dendogram of hierarchical clustering model")
RESULTS
Finally, after testing various models using k-NN algorithm, the best model used the original predictors & 3 nearest neighbors (k=3). The accuracy obtained from from the final model is 94,8%. I also tried the SVD technique in order to check if the new uncorrelated variables would help us improve the prediction accuracy rate of the letter in a new observation. But the model wasn’t improved.