Build a letter recognition classifier

Dec 28, 2017 00:00 · 14089 words · 67 minute read classification clustering k-NN SVD

In this analysis i’ll develop a model that will predict the hand-written capital letters (English) displayed in a picture, using various of it’s attributes.

INTRODUCTION

Dataset Information

This data set was originally used in the paper of P. W. Frey and D. J. Slate.“Letter Recognition Using Holland-style Adaptive Classifiers” (Machine Learning Vol 6 #2 March 91). It investigated the ability of several variations of Holland-style adaptive classifier systems to learn to correctly guess the letter categories associated with vectors of 16 integer attributes extracted from raster scan images of the letters. The actual data can be found at Letter Recognition Data Set Machine Learning Repository: https://archive.ics.uci.edu/ml/datasets/Letter+Recognition

The objective is to identify each of a large number of black-and-white rectangular pixel displays as one of the 26 capital letters in the English alphabet. The character images were based on 20 different fonts and each letter within these 20 fonts was randomly distorted to produce a file of unique stimuli. Each stimulus was converted into primitive numerical attributes (statistical moments and edge counts) which were then scaled to fit into a range of integer values from 0 through 15. So in total there are 20000 observations of 17 variables. There are no missing values in the data set.

The variables

  1. lettr capital letter (26 values from A to Z)
  2. x-box horizontal position of box (integer)
  3. y-box vertical position of box (integer)
  4. width width of box (integer)
  5. high height of box (integer)
  6. onpix total # on pixels (integer)
  7. x-bar mean x of on pixels in box (integer)
  8. y-bar mean y of on pixels in box (integer)
  9. x2bar mean x variance (integer)
  10. y2bar mean y variance (integer)
  11. xybar mean x y correlation (integer)
  12. x2ybr mean of x * x * y (integer)
  13. xy2br mean of x * y * y (integer)
  14. x-ege mean edge count left to right (integer)
  15. xegvy correlation of x-ege with y (integer)
  16. y-ege mean edge count bottom to top (integer)
  17. yegvx correlation of y-ege with x (integer)

PRE-PROCCESSING

Since the original data set didn’t contain any names on the variables, after i imported the data in R environment, i assigned the appropriate variable names. That was completed with the help of letter-recognition.names text file which accompanied the original data.

During the development of the prediction model, in order to predict the letter on new observations, at first we’ve used all variables. Then we excluded some of the 4 correlated variables and finally we’ve used the variables created from the SVD algorithm instead of the original.

Before we apply the hierarchical clustering, we created a data set with each letter as one observation and the average of each variable, as a distinct variable (named letter.cluster).

# Load Libraries
library(tidyverse)
library(stringr)
library(scales)
library(ggthemes)
library(corrplot)
library(caret)
library(gmodels)
library(class)
library(ggdendro)


# DATA WRANGLING ###############################################################
# Insert dataset
letter <- read.table("/Users/manos/Dropbox/Projects/R/Data/letter-recognition.data",
                  sep = ",")

# Create vector with new variable names
name <- c("lettr", "x-box", "y-box", "width", "high", "onpix", "x-bar", "y-bar", 
          "x2bar", "y2bar", "xybar", "x2ybr", "xy2br", "x-ege", "xegvy", "y-ege",
          "yegvx")

# Change the variable names of the dataset
names(letter) <- name

# Make it a tibble
letter <- as_data_frame(letter)

# Check for missing values
sum(is.na(letter))
## [1] 0
# Create the summary dataset
letter.cluster <- letter %>% 
  group_by(lettr) %>% 
  summarise_each(funs(mean))

EXPLORATORY ANALYSIS

In order to have an overview of the data-set’s variability, i produced a box-plot matrix of all variables and each letter on the x-axis

# Plot a box-plot for each letter and variable
letter %>% 
  gather("type", "n", 2:17) %>% 
  ggplot()+
  geom_boxplot(aes(x = lettr, y = n), outlier.size = .5, fill = "steelblue2", alpha = .7)+
  facet_grid(type ~.)+
  labs(y = "", x = "", 
       title = "Box-Plots of all variables for each letter", 
       subtitle = "")+
 theme_fivethirtyeight()

It looks that in some variables, the letters levels have significant different variance, than others. In particular, in y-box and high variables the different letters don’t seem to have any significant differences. On the other hand in x-bar, x2ybr, y-bar, y-ege variables the different letters seem to have large differences.

We should check for correlated variables in the data set. Below you can see a correlogram of the data set variables.

# Plot a correlogram to check for correlations
corrplot(cor(letter[,2:17]), method="number", type = "lower", number.cex = .6)

It seems that the first 4 variables (x-box, y-box, width, high) are quite correlated. Especially x-box & width (0.85) and y-box & high (0.82) are highly correlated.

MODELLING

An interesting insight that could be obtained from the data set is to try to predict the letter based on the variables produced by the digitized image. Classification is the problem of identifying to which of a set of categories (sub-populations) a new observation belongs, on the basis of a training set of data containing observations (or instances) whose category membership is known. So we must develop a model that classifies (categorize) every observation (case) to one of the 26 letters of the alphabet.

K-Nearest Neighbors (k-NN)

The K-nearest neighbors (kNN) algorithm used, in order to classify observations in a certain category (letter) by using the rest of the variables in the data set. It begins with a training data set made up of examples that are classified into several categories, as labeled by a nominal variable. Assume that we have a test data set containing unlabeled examples that otherwise have the same features as the training data. For each record in the test data set, kNN identifies k records in the training data that are the “nearest” in similarity, where k is an integer specified in advance. The unlabeled test instance is assigned the class of the majority of the k-nearest neighbors. We splitted the data set into a training data set (containing 70 % of the original data set observations) in which we build the classification model and a testing data set (containing the rest of the original data set observations - 30%) in which we tested the model we build before. The results are presented in the “results” section.

# Create a vector with the 70% of the dataset with respect to letter
set.seed(10)
inTrain = createDataPartition(letter$lettr, p = .7)[[1]]

# Assign the 70% of observations to training data
training <- letter[inTrain, -1]
training.lettr <- c(t(letter[inTrain, 1]))

# Assign the remaining 30 % of observations to testing data
testing <- letter[-inTrain, -1]
testing.lettr <- c(t(letter[-inTrain, 1]))


# Run knn algorithm on training dataset
# Create the knn model
knn_model <- knn(train = training, test = testing, cl = training.lettr, k = 3)

# Create a table in order to check the performance of the classification model
t <- CrossTable(x = testing.lettr,y = knn_model,
           prop.chisq=FALSE)
## 
##  
##    Cell Contents
## |-------------------------|
## |                       N |
## |           N / Row Total |
## |           N / Col Total |
## |         N / Table Total |
## |-------------------------|
## 
##  
## Total Observations in Table:  5985 
## 
##  
##               | knn_model 
## testing.lettr |         A |         B |         C |         D |         E |         F |         G |         H |         I |         J |         K |         L |         M |         N |         O |         P |         Q |         R |         S |         T |         U |         V |         W |         X |         Y |         Z | Row Total | 
## --------------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|
##             A |       235 |         1 |         0 |         0 |         0 |         0 |         0 |         0 |         0 |         0 |         0 |         0 |         0 |         0 |         0 |         0 |         0 |         0 |         0 |         0 |         0 |         0 |         0 |         0 |         0 |         0 |       236 | 
##               |     0.996 |     0.004 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.039 | 
##               |     0.975 |     0.004 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |           | 
##               |     0.039 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |           | 
## --------------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|
##             B |         0 |       212 |         0 |         0 |         3 |         0 |         0 |         7 |         0 |         0 |         0 |         0 |         1 |         0 |         0 |         0 |         0 |         2 |         3 |         0 |         0 |         1 |         0 |         0 |         0 |         0 |       229 | 
##               |     0.000 |     0.926 |     0.000 |     0.000 |     0.013 |     0.000 |     0.000 |     0.031 |     0.000 |     0.000 |     0.000 |     0.000 |     0.004 |     0.000 |     0.000 |     0.000 |     0.000 |     0.009 |     0.013 |     0.000 |     0.000 |     0.004 |     0.000 |     0.000 |     0.000 |     0.000 |     0.038 | 
##               |     0.000 |     0.895 |     0.000 |     0.000 |     0.013 |     0.000 |     0.000 |     0.033 |     0.000 |     0.000 |     0.000 |     0.000 |     0.004 |     0.000 |     0.000 |     0.000 |     0.000 |     0.009 |     0.014 |     0.000 |     0.000 |     0.004 |     0.000 |     0.000 |     0.000 |     0.000 |           | 
##               |     0.000 |     0.035 |     0.000 |     0.000 |     0.001 |     0.000 |     0.000 |     0.001 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.001 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |           | 
## --------------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|
##             C |         0 |         0 |       214 |         0 |         1 |         0 |         1 |         0 |         0 |         0 |         0 |         1 |         0 |         0 |         2 |         0 |         0 |         0 |         0 |         0 |         1 |         0 |         0 |         0 |         0 |         0 |       220 | 
##               |     0.000 |     0.000 |     0.973 |     0.000 |     0.005 |     0.000 |     0.005 |     0.000 |     0.000 |     0.000 |     0.000 |     0.005 |     0.000 |     0.000 |     0.009 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.005 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.037 | 
##               |     0.000 |     0.000 |     0.977 |     0.000 |     0.004 |     0.000 |     0.004 |     0.000 |     0.000 |     0.000 |     0.000 |     0.004 |     0.000 |     0.000 |     0.008 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.004 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |           | 
##               |     0.000 |     0.000 |     0.036 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |           | 
## --------------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|
##             D |         0 |         2 |         0 |       223 |         0 |         0 |         0 |         8 |         0 |         0 |         0 |         0 |         0 |         2 |         1 |         0 |         0 |         3 |         1 |         0 |         0 |         0 |         0 |         1 |         0 |         0 |       241 | 
##               |     0.000 |     0.008 |     0.000 |     0.925 |     0.000 |     0.000 |     0.000 |     0.033 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.008 |     0.004 |     0.000 |     0.000 |     0.012 |     0.004 |     0.000 |     0.000 |     0.000 |     0.000 |     0.004 |     0.000 |     0.000 |     0.040 | 
##               |     0.000 |     0.008 |     0.000 |     0.914 |     0.000 |     0.000 |     0.000 |     0.038 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.009 |     0.004 |     0.000 |     0.000 |     0.013 |     0.005 |     0.000 |     0.000 |     0.000 |     0.000 |     0.004 |     0.000 |     0.000 |           | 
##               |     0.000 |     0.000 |     0.000 |     0.037 |     0.000 |     0.000 |     0.000 |     0.001 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.001 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |           | 
## --------------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|
##             E |         0 |         1 |         0 |         0 |       221 |         2 |         2 |         0 |         0 |         0 |         0 |         0 |         0 |         0 |         0 |         0 |         0 |         0 |         0 |         0 |         0 |         1 |         0 |         2 |         0 |         1 |       230 | 
##               |     0.000 |     0.004 |     0.000 |     0.000 |     0.961 |     0.009 |     0.009 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.004 |     0.000 |     0.009 |     0.000 |     0.004 |     0.038 | 
##               |     0.000 |     0.004 |     0.000 |     0.000 |     0.932 |     0.008 |     0.009 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.004 |     0.000 |     0.009 |     0.000 |     0.005 |           | 
##               |     0.000 |     0.000 |     0.000 |     0.000 |     0.037 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |           | 
## --------------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|
##             F |         0 |         1 |         1 |         0 |         0 |       219 |         0 |         0 |         0 |         0 |         0 |         0 |         0 |         1 |         0 |         5 |         0 |         0 |         0 |         3 |         0 |         1 |         0 |         1 |         0 |         0 |       232 | 
##               |     0.000 |     0.004 |     0.004 |     0.000 |     0.000 |     0.944 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.004 |     0.000 |     0.022 |     0.000 |     0.000 |     0.000 |     0.013 |     0.000 |     0.004 |     0.000 |     0.004 |     0.000 |     0.000 |     0.039 | 
##               |     0.000 |     0.004 |     0.005 |     0.000 |     0.000 |     0.905 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.004 |     0.000 |     0.022 |     0.000 |     0.000 |     0.000 |     0.013 |     0.000 |     0.004 |     0.000 |     0.004 |     0.000 |     0.000 |           | 
##               |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.037 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.001 |     0.000 |     0.000 |     0.000 |     0.001 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |           | 
## --------------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|
##             G |         0 |         0 |         1 |         0 |         3 |         0 |       221 |         0 |         0 |         0 |         1 |         0 |         0 |         0 |         2 |         0 |         1 |         0 |         1 |         0 |         0 |         0 |         1 |         0 |         0 |         0 |       231 | 
##               |     0.000 |     0.000 |     0.004 |     0.000 |     0.013 |     0.000 |     0.957 |     0.000 |     0.000 |     0.000 |     0.004 |     0.000 |     0.000 |     0.000 |     0.009 |     0.000 |     0.004 |     0.000 |     0.004 |     0.000 |     0.000 |     0.000 |     0.004 |     0.000 |     0.000 |     0.000 |     0.039 | 
##               |     0.000 |     0.000 |     0.005 |     0.000 |     0.013 |     0.000 |     0.944 |     0.000 |     0.000 |     0.000 |     0.005 |     0.000 |     0.000 |     0.000 |     0.008 |     0.000 |     0.004 |     0.000 |     0.005 |     0.000 |     0.000 |     0.000 |     0.004 |     0.000 |     0.000 |     0.000 |           | 
##               |     0.000 |     0.000 |     0.000 |     0.000 |     0.001 |     0.000 |     0.037 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |           | 
## --------------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|
##             H |         0 |         1 |         0 |         8 |         0 |         0 |         5 |       185 |         0 |         0 |         9 |         0 |         1 |         0 |         1 |         2 |         0 |         7 |         0 |         0 |         0 |         1 |         0 |         0 |         0 |         0 |       220 | 
##               |     0.000 |     0.005 |     0.000 |     0.036 |     0.000 |     0.000 |     0.023 |     0.841 |     0.000 |     0.000 |     0.041 |     0.000 |     0.005 |     0.000 |     0.005 |     0.009 |     0.000 |     0.032 |     0.000 |     0.000 |     0.000 |     0.005 |     0.000 |     0.000 |     0.000 |     0.000 |     0.037 | 
##               |     0.000 |     0.004 |     0.000 |     0.033 |     0.000 |     0.000 |     0.021 |     0.869 |     0.000 |     0.000 |     0.041 |     0.000 |     0.004 |     0.000 |     0.004 |     0.009 |     0.000 |     0.031 |     0.000 |     0.000 |     0.000 |     0.004 |     0.000 |     0.000 |     0.000 |     0.000 |           | 
##               |     0.000 |     0.000 |     0.000 |     0.001 |     0.000 |     0.000 |     0.001 |     0.031 |     0.000 |     0.000 |     0.002 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.001 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |           | 
## --------------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|
##             I |         0 |         1 |         0 |         1 |         0 |         0 |         0 |         0 |       217 |         7 |         0 |         0 |         0 |         0 |         0 |         0 |         0 |         0 |         0 |         0 |         0 |         0 |         0 |         0 |         0 |         0 |       226 | 
##               |     0.000 |     0.004 |     0.000 |     0.004 |     0.000 |     0.000 |     0.000 |     0.000 |     0.960 |     0.031 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.038 | 
##               |     0.000 |     0.004 |     0.000 |     0.004 |     0.000 |     0.000 |     0.000 |     0.000 |     0.956 |     0.032 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |           | 
##               |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.036 |     0.001 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |           | 
## --------------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|
##             J |         0 |         0 |         0 |         0 |         0 |         0 |         0 |         1 |        10 |       212 |         0 |         0 |         0 |         0 |         1 |         0 |         0 |         0 |         0 |         0 |         0 |         0 |         0 |         0 |         0 |         0 |       224 | 
##               |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.004 |     0.045 |     0.946 |     0.000 |     0.000 |     0.000 |     0.000 |     0.004 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.037 | 
##               |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.005 |     0.044 |     0.968 |     0.000 |     0.000 |     0.000 |     0.000 |     0.004 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |           | 
##               |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.002 |     0.035 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |           | 
## --------------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|
##             K |         0 |         0 |         0 |         1 |         4 |         0 |         0 |         4 |         0 |         0 |       199 |         1 |         0 |         0 |         0 |         0 |         0 |         7 |         1 |         0 |         0 |         0 |         0 |         4 |         0 |         0 |       221 | 
##               |     0.000 |     0.000 |     0.000 |     0.005 |     0.018 |     0.000 |     0.000 |     0.018 |     0.000 |     0.000 |     0.900 |     0.005 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.032 |     0.005 |     0.000 |     0.000 |     0.000 |     0.000 |     0.018 |     0.000 |     0.000 |     0.037 | 
##               |     0.000 |     0.000 |     0.000 |     0.004 |     0.017 |     0.000 |     0.000 |     0.019 |     0.000 |     0.000 |     0.900 |     0.004 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.031 |     0.005 |     0.000 |     0.000 |     0.000 |     0.000 |     0.017 |     0.000 |     0.000 |           | 
##               |     0.000 |     0.000 |     0.000 |     0.000 |     0.001 |     0.000 |     0.000 |     0.001 |     0.000 |     0.000 |     0.033 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.001 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.001 |     0.000 |     0.000 |           | 
## --------------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|
##             L |         0 |         0 |         0 |         0 |         1 |         0 |         3 |         0 |         0 |         0 |         2 |       220 |         0 |         0 |         0 |         0 |         0 |         2 |         0 |         0 |         0 |         0 |         0 |         0 |         0 |         0 |       228 | 
##               |     0.000 |     0.000 |     0.000 |     0.000 |     0.004 |     0.000 |     0.013 |     0.000 |     0.000 |     0.000 |     0.009 |     0.965 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.009 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.038 | 
##               |     0.000 |     0.000 |     0.000 |     0.000 |     0.004 |     0.000 |     0.013 |     0.000 |     0.000 |     0.000 |     0.009 |     0.982 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.009 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |           | 
##               |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.001 |     0.000 |     0.000 |     0.000 |     0.000 |     0.037 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |           | 
## --------------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|
##             M |         0 |         0 |         0 |         0 |         0 |         0 |         0 |         0 |         0 |         0 |         0 |         0 |       234 |         1 |         1 |         0 |         0 |         0 |         0 |         0 |         0 |         1 |         0 |         0 |         0 |         0 |       237 | 
##               |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.987 |     0.004 |     0.004 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.004 |     0.000 |     0.000 |     0.000 |     0.000 |     0.040 | 
##               |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.987 |     0.004 |     0.004 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.004 |     0.000 |     0.000 |     0.000 |     0.000 |           | 
##               |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.039 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |           | 
## --------------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|
##             N |         1 |         0 |         0 |         1 |         0 |         0 |         0 |         3 |         0 |         0 |         0 |         1 |         1 |       219 |         3 |         0 |         1 |         3 |         0 |         0 |         0 |         1 |         0 |         0 |         0 |         0 |       234 | 
##               |     0.004 |     0.000 |     0.000 |     0.004 |     0.000 |     0.000 |     0.000 |     0.013 |     0.000 |     0.000 |     0.000 |     0.004 |     0.004 |     0.936 |     0.013 |     0.000 |     0.004 |     0.013 |     0.000 |     0.000 |     0.000 |     0.004 |     0.000 |     0.000 |     0.000 |     0.000 |     0.039 | 
##               |     0.004 |     0.000 |     0.000 |     0.004 |     0.000 |     0.000 |     0.000 |     0.014 |     0.000 |     0.000 |     0.000 |     0.004 |     0.004 |     0.948 |     0.013 |     0.000 |     0.004 |     0.013 |     0.000 |     0.000 |     0.000 |     0.004 |     0.000 |     0.000 |     0.000 |     0.000 |           | 
##               |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.001 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.037 |     0.001 |     0.000 |     0.000 |     0.001 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |           | 
## --------------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|
##             O |         0 |         0 |         3 |         2 |         0 |         0 |         1 |         0 |         0 |         0 |         0 |         0 |         0 |         1 |       217 |         0 |         0 |         0 |         0 |         0 |         0 |         0 |         1 |         0 |         0 |         0 |       225 | 
##               |     0.000 |     0.000 |     0.013 |     0.009 |     0.000 |     0.000 |     0.004 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.004 |     0.964 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.004 |     0.000 |     0.000 |     0.000 |     0.038 | 
##               |     0.000 |     0.000 |     0.014 |     0.008 |     0.000 |     0.000 |     0.004 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.004 |     0.916 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.004 |     0.000 |     0.000 |     0.000 |           | 
##               |     0.000 |     0.000 |     0.001 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.036 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |           | 
## --------------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|
##             P |         0 |         0 |         0 |         0 |         0 |        19 |         0 |         0 |         0 |         0 |         0 |         1 |         0 |         0 |         0 |       216 |         2 |         1 |         0 |         0 |         0 |         0 |         0 |         0 |         1 |         0 |       240 | 
##               |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.079 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.004 |     0.000 |     0.000 |     0.000 |     0.900 |     0.008 |     0.004 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.004 |     0.000 |     0.040 | 
##               |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.079 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.004 |     0.000 |     0.000 |     0.000 |     0.964 |     0.009 |     0.004 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.004 |     0.000 |           | 
##               |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.003 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.036 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |           | 
## --------------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|
##             Q |         0 |         0 |         0 |         0 |         0 |         0 |         0 |         0 |         0 |         0 |         0 |         0 |         0 |         0 |         8 |         0 |       224 |         1 |         0 |         0 |         0 |         0 |         0 |         0 |         1 |         0 |       234 | 
##               |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.034 |     0.000 |     0.957 |     0.004 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.004 |     0.000 |     0.039 | 
##               |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.034 |     0.000 |     0.974 |     0.004 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.004 |     0.000 |           | 
##               |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.001 |     0.000 |     0.037 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |           | 
## --------------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|
##             R |         0 |        11 |         0 |         3 |         0 |         0 |         0 |         2 |         0 |         0 |         6 |         0 |         0 |         6 |         0 |         0 |         0 |       198 |         1 |         0 |         0 |         0 |         0 |         0 |         0 |         0 |       227 | 
##               |     0.000 |     0.048 |     0.000 |     0.013 |     0.000 |     0.000 |     0.000 |     0.009 |     0.000 |     0.000 |     0.026 |     0.000 |     0.000 |     0.026 |     0.000 |     0.000 |     0.000 |     0.872 |     0.004 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.038 | 
##               |     0.000 |     0.046 |     0.000 |     0.012 |     0.000 |     0.000 |     0.000 |     0.009 |     0.000 |     0.000 |     0.027 |     0.000 |     0.000 |     0.026 |     0.000 |     0.000 |     0.000 |     0.872 |     0.005 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |           | 
##               |     0.000 |     0.002 |     0.000 |     0.001 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.001 |     0.000 |     0.000 |     0.001 |     0.000 |     0.000 |     0.000 |     0.033 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |           | 
## --------------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|
##             S |         0 |         1 |         0 |         4 |         1 |         0 |         1 |         2 |         0 |         0 |         0 |         0 |         0 |         0 |         0 |         0 |         0 |         1 |       213 |         0 |         0 |         0 |         0 |         0 |         0 |         1 |       224 | 
##               |     0.000 |     0.004 |     0.000 |     0.018 |     0.004 |     0.000 |     0.004 |     0.009 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.004 |     0.951 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.004 |     0.037 | 
##               |     0.000 |     0.004 |     0.000 |     0.016 |     0.004 |     0.000 |     0.004 |     0.009 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.004 |     0.964 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.005 |           | 
##               |     0.000 |     0.000 |     0.000 |     0.001 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.036 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |           | 
## --------------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|
##             T |         0 |         1 |         0 |         1 |         0 |         1 |         0 |         0 |         0 |         0 |         0 |         0 |         0 |         0 |         0 |         0 |         0 |         1 |         0 |       229 |         0 |         1 |         0 |         1 |         3 |         0 |       238 | 
##               |     0.000 |     0.004 |     0.000 |     0.004 |     0.000 |     0.004 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.004 |     0.000 |     0.962 |     0.000 |     0.004 |     0.000 |     0.004 |     0.013 |     0.000 |     0.040 | 
##               |     0.000 |     0.004 |     0.000 |     0.004 |     0.000 |     0.004 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.004 |     0.000 |     0.954 |     0.000 |     0.004 |     0.000 |     0.004 |     0.013 |     0.000 |           | 
##               |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.038 |     0.000 |     0.000 |     0.000 |     0.000 |     0.001 |     0.000 |           | 
## --------------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|
##             U |         4 |         0 |         0 |         0 |         0 |         0 |         0 |         0 |         0 |         0 |         0 |         0 |         0 |         0 |         0 |         0 |         0 |         0 |         0 |         0 |       239 |         0 |         0 |         0 |         0 |         0 |       243 | 
##               |     0.016 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.984 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.041 | 
##               |     0.017 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.988 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |           | 
##               |     0.001 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.040 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |           | 
## --------------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|
##             V |         0 |         4 |         0 |         0 |         0 |         1 |         0 |         0 |         0 |         0 |         0 |         0 |         0 |         1 |         0 |         1 |         0 |         0 |         0 |         0 |         0 |       218 |         3 |         0 |         1 |         0 |       229 | 
##               |     0.000 |     0.017 |     0.000 |     0.000 |     0.000 |     0.004 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.004 |     0.000 |     0.004 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.952 |     0.013 |     0.000 |     0.004 |     0.000 |     0.038 | 
##               |     0.000 |     0.017 |     0.000 |     0.000 |     0.000 |     0.004 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.004 |     0.000 |     0.004 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.956 |     0.013 |     0.000 |     0.004 |     0.000 |           | 
##               |     0.000 |     0.001 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.036 |     0.001 |     0.000 |     0.000 |     0.000 |           | 
## --------------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|
##             W |         0 |         1 |         0 |         0 |         0 |         0 |         0 |         0 |         0 |         0 |         0 |         0 |         0 |         0 |         1 |         0 |         0 |         1 |         0 |         0 |         0 |         3 |       219 |         0 |         0 |         0 |       225 | 
##               |     0.000 |     0.004 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.004 |     0.000 |     0.000 |     0.004 |     0.000 |     0.000 |     0.000 |     0.013 |     0.973 |     0.000 |     0.000 |     0.000 |     0.038 | 
##               |     0.000 |     0.004 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.004 |     0.000 |     0.000 |     0.004 |     0.000 |     0.000 |     0.000 |     0.013 |     0.978 |     0.000 |     0.000 |     0.000 |           | 
##               |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.001 |     0.037 |     0.000 |     0.000 |     0.000 |           | 
## --------------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|
##             X |         0 |         0 |         0 |         0 |         3 |         0 |         0 |         0 |         0 |         0 |         4 |         0 |         0 |         0 |         0 |         0 |         1 |         0 |         1 |         1 |         2 |         0 |         0 |       224 |         0 |         0 |       236 | 
##               |     0.000 |     0.000 |     0.000 |     0.000 |     0.013 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.017 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.004 |     0.000 |     0.004 |     0.004 |     0.008 |     0.000 |     0.000 |     0.949 |     0.000 |     0.000 |     0.039 | 
##               |     0.000 |     0.000 |     0.000 |     0.000 |     0.013 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.018 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.004 |     0.000 |     0.005 |     0.004 |     0.008 |     0.000 |     0.000 |     0.961 |     0.000 |     0.000 |           | 
##               |     0.000 |     0.000 |     0.000 |     0.000 |     0.001 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.001 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.037 |     0.000 |     0.000 |           | 
## --------------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|
##             Y |         1 |         0 |         0 |         0 |         0 |         0 |         0 |         1 |         0 |         0 |         0 |         0 |         0 |         0 |         0 |         0 |         0 |         0 |         0 |         5 |         0 |         0 |         0 |         0 |       228 |         0 |       235 | 
##               |     0.004 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.004 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.021 |     0.000 |     0.000 |     0.000 |     0.000 |     0.970 |     0.000 |     0.039 | 
##               |     0.004 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.005 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.021 |     0.000 |     0.000 |     0.000 |     0.000 |     0.974 |     0.000 |           | 
##               |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.001 |     0.000 |     0.000 |     0.000 |     0.000 |     0.038 |     0.000 |           | 
## --------------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|
##             Z |         0 |         0 |         0 |         0 |         0 |         0 |         0 |         0 |         0 |         0 |         0 |         0 |         0 |         0 |         0 |         0 |         1 |         0 |         0 |         2 |         0 |         0 |         0 |         0 |         0 |       217 |       220 | 
##               |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.005 |     0.000 |     0.000 |     0.009 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.986 |     0.037 | 
##               |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.004 |     0.000 |     0.000 |     0.008 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.991 |           | 
##               |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.036 |           | 
## --------------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|
##  Column Total |       241 |       237 |       219 |       244 |       237 |       242 |       234 |       213 |       227 |       219 |       221 |       224 |       237 |       231 |       237 |       224 |       230 |       227 |       221 |       240 |       242 |       228 |       224 |       233 |       234 |       219 |      5985 | 
##               |     0.040 |     0.040 |     0.037 |     0.041 |     0.040 |     0.040 |     0.039 |     0.036 |     0.038 |     0.037 |     0.037 |     0.037 |     0.040 |     0.039 |     0.040 |     0.037 |     0.038 |     0.038 |     0.037 |     0.040 |     0.040 |     0.038 |     0.037 |     0.039 |     0.039 |     0.037 |           | 
## --------------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|
## 
## 
# Calculate prediction success rate for all letters
mean(diag(t$prop.col))
## [1] 0.9478915
# Check prediction success rate for each letter
diag(t$prop.col)
##         A         B         C         D         E         F         G 
## 0.9751037 0.8945148 0.9771689 0.9139344 0.9324895 0.9049587 0.9444444 
##         H         I         J         K         L         M         N 
## 0.8685446 0.9559471 0.9680365 0.9004525 0.9821429 0.9873418 0.9480519 
##         O         P         Q         R         S         T         U 
## 0.9156118 0.9642857 0.9739130 0.8722467 0.9638009 0.9541667 0.9876033 
##         V         W         X         Y         Z 
## 0.9561404 0.9776786 0.9613734 0.9743590 0.9908676

The accuracy obtained from the nearest neighbors classification algorithm is around 94,8%. I’ve used 3 nearest neighbors (k=3) for building the final model. Alternative number of neighbors (4,5,6,7 etc) produced slightly lower accuracy rate (94.3% - 94,7%). The model is better at predicting some letters, such as L, M, U, Z with more than 98% accuracy rate. On the other hand, other letters such as H, R, B are more difficult to predict (<90% accuracy rate). In the plot below you can see each letter accuracy rate of the final model.

# Create a data set with the accuracy rate of each letter
letters <- diag(t$prop.col) %>% 
  as.data.frame()

# Make the appropriate transformations in order to plot
letters$letter <- rownames(letters)

# Plot the accuracy rate per letter
ggplot(letters)+
  geom_bar(aes(x = letter, y = .), stat="identity", fill = "steelblue2", alpha = .7)+
    scale_y_continuous(labels=percent) +
  labs(title = "Accuracy rate % per letter (K-NN algorithm)", 
       x = "", y = "Accuracy rate %", subtittle = "")+
 theme_fivethirtyeight()

Singular Value Decomposition (SVD)

Since i discovered that there are some correlated variables in the data set, i applied the SVD algorithm in order to create a data set with uncorrelated variables. SVD is an algebraic tool that has many potential uses such as:

  • Dimensionality reduction
  • Indexing (LSI)
  • Visualization/clustering of high-dimensional objects
  • Similarity computations/outlier detection
  • Rule mining, treatment of missing/wrong values

Below i created a new dataset with the new SVD variables and build a new model using k-NN algorithm.

# Create an in Dataset 
letter.svd <- svd(letter[,2:17])

# Create a dataset with the SVD values
letter2 <- as_data_frame(cbind(letter[,1],letter.svd$u))

# Keep specific SVD variables
letter2 <- letter2[,1:8]

# Create a vector with the 70% of the dataset with respect to letter
set.seed(10)
inTrain = createDataPartition(letter2$lettr, p = .7)[[1]]

# Assign the 70% of observations to training data
training <- letter2[inTrain, -1]
training.lettr <- c(t(letter2[inTrain, 1]))

# Assign the remaining 30 % of observations to testing data
testing <- letter2[-inTrain, -1]
testing.lettr <- c(t(letter2[-inTrain, 1]))


# Run knn algorithm on training dataset
# Create the knn model
knn_model.svd <- knn(train = training, test = testing, cl = training.lettr, k = 3)

# Create a table in order to check the performance of the classification model
s <- CrossTable(x = testing.lettr,y = knn_model.svd,
           prop.chisq=FALSE)
## 
##  
##    Cell Contents
## |-------------------------|
## |                       N |
## |           N / Row Total |
## |           N / Col Total |
## |         N / Table Total |
## |-------------------------|
## 
##  
## Total Observations in Table:  5985 
## 
##  
##               | knn_model.svd 
## testing.lettr |         A |         B |         C |         D |         E |         F |         G |         H |         I |         J |         K |         L |         M |         N |         O |         P |         Q |         R |         S |         T |         U |         V |         W |         X |         Y |         Z | Row Total | 
## --------------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|
##             A |       212 |         2 |         0 |         2 |         0 |         0 |         3 |         3 |         0 |         0 |         1 |         0 |         1 |         1 |         2 |         1 |         0 |         4 |         0 |         1 |         0 |         0 |         0 |         0 |         3 |         0 |       236 | 
##               |     0.898 |     0.008 |     0.000 |     0.008 |     0.000 |     0.000 |     0.013 |     0.013 |     0.000 |     0.000 |     0.004 |     0.000 |     0.004 |     0.004 |     0.008 |     0.004 |     0.000 |     0.017 |     0.000 |     0.004 |     0.000 |     0.000 |     0.000 |     0.000 |     0.013 |     0.000 |     0.039 | 
##               |     0.910 |     0.008 |     0.000 |     0.008 |     0.000 |     0.000 |     0.012 |     0.015 |     0.000 |     0.000 |     0.005 |     0.000 |     0.004 |     0.004 |     0.008 |     0.005 |     0.000 |     0.016 |     0.000 |     0.004 |     0.000 |     0.000 |     0.000 |     0.000 |     0.013 |     0.000 |           | 
##               |     0.035 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.001 |     0.001 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.001 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.001 |     0.000 |           | 
## --------------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|
##             B |         1 |       165 |         0 |         3 |         3 |         1 |         2 |         4 |         3 |         1 |         2 |         0 |         1 |         1 |         1 |         0 |         2 |        14 |        17 |         0 |         0 |         2 |         1 |         1 |         1 |         3 |       229 | 
##               |     0.004 |     0.721 |     0.000 |     0.013 |     0.013 |     0.004 |     0.009 |     0.017 |     0.013 |     0.004 |     0.009 |     0.000 |     0.004 |     0.004 |     0.004 |     0.000 |     0.009 |     0.061 |     0.074 |     0.000 |     0.000 |     0.009 |     0.004 |     0.004 |     0.004 |     0.013 |     0.038 | 
##               |     0.004 |     0.655 |     0.000 |     0.012 |     0.013 |     0.004 |     0.008 |     0.021 |     0.013 |     0.005 |     0.010 |     0.000 |     0.004 |     0.004 |     0.004 |     0.000 |     0.009 |     0.055 |     0.074 |     0.000 |     0.000 |     0.009 |     0.004 |     0.004 |     0.004 |     0.014 |           | 
##               |     0.000 |     0.028 |     0.000 |     0.001 |     0.001 |     0.000 |     0.000 |     0.001 |     0.001 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.002 |     0.003 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.001 |           | 
## --------------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|
##             C |         0 |         0 |       198 |         0 |         3 |         0 |         8 |         1 |         0 |         0 |         0 |         1 |         0 |         0 |         3 |         0 |         1 |         1 |         0 |         1 |         3 |         0 |         0 |         0 |         0 |         0 |       220 | 
##               |     0.000 |     0.000 |     0.900 |     0.000 |     0.014 |     0.000 |     0.036 |     0.005 |     0.000 |     0.000 |     0.000 |     0.005 |     0.000 |     0.000 |     0.014 |     0.000 |     0.005 |     0.005 |     0.000 |     0.005 |     0.014 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.037 | 
##               |     0.000 |     0.000 |     0.925 |     0.000 |     0.013 |     0.000 |     0.033 |     0.005 |     0.000 |     0.000 |     0.000 |     0.005 |     0.000 |     0.000 |     0.013 |     0.000 |     0.004 |     0.004 |     0.000 |     0.004 |     0.012 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |           | 
##               |     0.000 |     0.000 |     0.033 |     0.000 |     0.001 |     0.000 |     0.001 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.001 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.001 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |           | 
## --------------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|
##             D |         3 |         6 |         0 |       184 |         2 |         2 |         2 |         3 |         2 |         2 |         2 |         1 |         0 |         1 |         4 |         0 |         1 |        16 |         3 |         0 |         0 |         0 |         0 |         3 |         1 |         3 |       241 | 
##               |     0.012 |     0.025 |     0.000 |     0.763 |     0.008 |     0.008 |     0.008 |     0.012 |     0.008 |     0.008 |     0.008 |     0.004 |     0.000 |     0.004 |     0.017 |     0.000 |     0.004 |     0.066 |     0.012 |     0.000 |     0.000 |     0.000 |     0.000 |     0.012 |     0.004 |     0.012 |     0.040 | 
##               |     0.013 |     0.024 |     0.000 |     0.751 |     0.008 |     0.009 |     0.008 |     0.015 |     0.009 |     0.009 |     0.010 |     0.005 |     0.000 |     0.004 |     0.017 |     0.000 |     0.004 |     0.063 |     0.013 |     0.000 |     0.000 |     0.000 |     0.000 |     0.012 |     0.004 |     0.014 |           | 
##               |     0.001 |     0.001 |     0.000 |     0.031 |     0.000 |     0.000 |     0.000 |     0.001 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.001 |     0.000 |     0.000 |     0.003 |     0.001 |     0.000 |     0.000 |     0.000 |     0.000 |     0.001 |     0.000 |     0.001 |           | 
## --------------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|
##             E |         0 |         3 |         3 |         0 |       178 |         1 |         5 |         0 |         2 |         0 |         1 |         2 |         0 |         0 |         0 |         0 |         2 |         2 |         8 |         0 |         0 |         2 |         0 |        19 |         0 |         2 |       230 | 
##               |     0.000 |     0.013 |     0.013 |     0.000 |     0.774 |     0.004 |     0.022 |     0.000 |     0.009 |     0.000 |     0.004 |     0.009 |     0.000 |     0.000 |     0.000 |     0.000 |     0.009 |     0.009 |     0.035 |     0.000 |     0.000 |     0.009 |     0.000 |     0.083 |     0.000 |     0.009 |     0.038 | 
##               |     0.000 |     0.012 |     0.014 |     0.000 |     0.748 |     0.004 |     0.020 |     0.000 |     0.009 |     0.000 |     0.005 |     0.009 |     0.000 |     0.000 |     0.000 |     0.000 |     0.009 |     0.008 |     0.035 |     0.000 |     0.000 |     0.009 |     0.000 |     0.075 |     0.000 |     0.010 |           | 
##               |     0.000 |     0.001 |     0.001 |     0.000 |     0.030 |     0.000 |     0.001 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.001 |     0.000 |     0.000 |     0.000 |     0.000 |     0.003 |     0.000 |     0.000 |           | 
## --------------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|
##             F |         0 |         4 |         0 |         2 |         1 |       186 |         0 |         1 |         4 |         3 |         0 |         1 |         0 |         1 |         0 |         7 |         0 |         0 |         3 |        11 |         0 |         2 |         0 |         0 |         5 |         1 |       232 | 
##               |     0.000 |     0.017 |     0.000 |     0.009 |     0.004 |     0.802 |     0.000 |     0.004 |     0.017 |     0.013 |     0.000 |     0.004 |     0.000 |     0.004 |     0.000 |     0.030 |     0.000 |     0.000 |     0.013 |     0.047 |     0.000 |     0.009 |     0.000 |     0.000 |     0.022 |     0.004 |     0.039 | 
##               |     0.000 |     0.016 |     0.000 |     0.008 |     0.004 |     0.802 |     0.000 |     0.005 |     0.018 |     0.014 |     0.000 |     0.005 |     0.000 |     0.004 |     0.000 |     0.032 |     0.000 |     0.000 |     0.013 |     0.044 |     0.000 |     0.009 |     0.000 |     0.000 |     0.022 |     0.005 |           | 
##               |     0.000 |     0.001 |     0.000 |     0.000 |     0.000 |     0.031 |     0.000 |     0.000 |     0.001 |     0.001 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.001 |     0.000 |     0.000 |     0.001 |     0.002 |     0.000 |     0.000 |     0.000 |     0.000 |     0.001 |     0.000 |           | 
## --------------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|
##             G |         1 |         6 |         1 |         1 |         1 |         0 |       187 |         2 |         0 |         0 |         3 |         1 |         5 |         1 |         4 |         0 |         6 |         6 |         1 |         0 |         2 |         1 |         1 |         1 |         0 |         0 |       231 | 
##               |     0.004 |     0.026 |     0.004 |     0.004 |     0.004 |     0.000 |     0.810 |     0.009 |     0.000 |     0.000 |     0.013 |     0.004 |     0.022 |     0.004 |     0.017 |     0.000 |     0.026 |     0.026 |     0.004 |     0.000 |     0.009 |     0.004 |     0.004 |     0.004 |     0.000 |     0.000 |     0.039 | 
##               |     0.004 |     0.024 |     0.005 |     0.004 |     0.004 |     0.000 |     0.760 |     0.010 |     0.000 |     0.000 |     0.015 |     0.005 |     0.022 |     0.004 |     0.017 |     0.000 |     0.026 |     0.024 |     0.004 |     0.000 |     0.008 |     0.004 |     0.004 |     0.004 |     0.000 |     0.000 |           | 
##               |     0.000 |     0.001 |     0.000 |     0.000 |     0.000 |     0.000 |     0.031 |     0.000 |     0.000 |     0.000 |     0.001 |     0.000 |     0.001 |     0.000 |     0.001 |     0.000 |     0.001 |     0.001 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |           | 
## --------------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|
##             H |         2 |        11 |         1 |        10 |         1 |         1 |         8 |       140 |         0 |         2 |         6 |         1 |         1 |         4 |         1 |         3 |         2 |        15 |         0 |         3 |         0 |         2 |         0 |         2 |         4 |         0 |       220 | 
##               |     0.009 |     0.050 |     0.005 |     0.045 |     0.005 |     0.005 |     0.036 |     0.636 |     0.000 |     0.009 |     0.027 |     0.005 |     0.005 |     0.018 |     0.005 |     0.014 |     0.009 |     0.068 |     0.000 |     0.014 |     0.000 |     0.009 |     0.000 |     0.009 |     0.018 |     0.000 |     0.037 | 
##               |     0.009 |     0.044 |     0.005 |     0.041 |     0.004 |     0.004 |     0.033 |     0.722 |     0.000 |     0.009 |     0.030 |     0.005 |     0.004 |     0.017 |     0.004 |     0.014 |     0.009 |     0.059 |     0.000 |     0.012 |     0.000 |     0.009 |     0.000 |     0.008 |     0.018 |     0.000 |           | 
##               |     0.000 |     0.002 |     0.000 |     0.002 |     0.000 |     0.000 |     0.001 |     0.023 |     0.000 |     0.000 |     0.001 |     0.000 |     0.000 |     0.001 |     0.000 |     0.001 |     0.000 |     0.003 |     0.000 |     0.001 |     0.000 |     0.000 |     0.000 |     0.000 |     0.001 |     0.000 |           | 
## --------------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|
##             I |         0 |         3 |         0 |         1 |         3 |         2 |         0 |         0 |       202 |         3 |         1 |         0 |         0 |         0 |         0 |         4 |         0 |         2 |         3 |         0 |         0 |         0 |         0 |         1 |         0 |         1 |       226 | 
##               |     0.000 |     0.013 |     0.000 |     0.004 |     0.013 |     0.009 |     0.000 |     0.000 |     0.894 |     0.013 |     0.004 |     0.000 |     0.000 |     0.000 |     0.000 |     0.018 |     0.000 |     0.009 |     0.013 |     0.000 |     0.000 |     0.000 |     0.000 |     0.004 |     0.000 |     0.004 |     0.038 | 
##               |     0.000 |     0.012 |     0.000 |     0.004 |     0.013 |     0.009 |     0.000 |     0.000 |     0.906 |     0.014 |     0.005 |     0.000 |     0.000 |     0.000 |     0.000 |     0.018 |     0.000 |     0.008 |     0.013 |     0.000 |     0.000 |     0.000 |     0.000 |     0.004 |     0.000 |     0.005 |           | 
##               |     0.000 |     0.001 |     0.000 |     0.000 |     0.001 |     0.000 |     0.000 |     0.000 |     0.034 |     0.001 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.001 |     0.000 |     0.000 |     0.001 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |           | 
## --------------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|
##             J |         0 |         5 |         0 |         4 |         1 |         0 |         1 |         1 |         5 |       194 |         0 |         1 |         0 |         0 |         4 |         1 |         0 |         2 |         1 |         2 |         0 |         0 |         0 |         2 |         0 |         0 |       224 | 
##               |     0.000 |     0.022 |     0.000 |     0.018 |     0.004 |     0.000 |     0.004 |     0.004 |     0.022 |     0.866 |     0.000 |     0.004 |     0.000 |     0.000 |     0.018 |     0.004 |     0.000 |     0.009 |     0.004 |     0.009 |     0.000 |     0.000 |     0.000 |     0.009 |     0.000 |     0.000 |     0.037 | 
##               |     0.000 |     0.020 |     0.000 |     0.016 |     0.004 |     0.000 |     0.004 |     0.005 |     0.022 |     0.915 |     0.000 |     0.005 |     0.000 |     0.000 |     0.017 |     0.005 |     0.000 |     0.008 |     0.004 |     0.008 |     0.000 |     0.000 |     0.000 |     0.008 |     0.000 |     0.000 |           | 
##               |     0.000 |     0.001 |     0.000 |     0.001 |     0.000 |     0.000 |     0.000 |     0.000 |     0.001 |     0.032 |     0.000 |     0.000 |     0.000 |     0.000 |     0.001 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |           | 
## --------------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|
##             K |         1 |         1 |         2 |         0 |         6 |         2 |         2 |         9 |         1 |         0 |       177 |         0 |         0 |         0 |         0 |         0 |         0 |         3 |         3 |         0 |         1 |         2 |         0 |         8 |         3 |         0 |       221 | 
##               |     0.005 |     0.005 |     0.009 |     0.000 |     0.027 |     0.009 |     0.009 |     0.041 |     0.005 |     0.000 |     0.801 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.014 |     0.014 |     0.000 |     0.005 |     0.009 |     0.000 |     0.036 |     0.014 |     0.000 |     0.037 | 
##               |     0.004 |     0.004 |     0.009 |     0.000 |     0.025 |     0.009 |     0.008 |     0.046 |     0.004 |     0.000 |     0.876 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.012 |     0.013 |     0.000 |     0.004 |     0.009 |     0.000 |     0.032 |     0.013 |     0.000 |           | 
##               |     0.000 |     0.000 |     0.000 |     0.000 |     0.001 |     0.000 |     0.000 |     0.002 |     0.000 |     0.000 |     0.030 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.001 |     0.001 |     0.000 |     0.000 |     0.000 |     0.000 |     0.001 |     0.001 |     0.000 |           | 
## --------------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|
##             L |         0 |         0 |         0 |         0 |         4 |         0 |         3 |         2 |         2 |         1 |         0 |       197 |         0 |         1 |         0 |         0 |         7 |         5 |         0 |         2 |         1 |         0 |         0 |         2 |         0 |         1 |       228 | 
##               |     0.000 |     0.000 |     0.000 |     0.000 |     0.018 |     0.000 |     0.013 |     0.009 |     0.009 |     0.004 |     0.000 |     0.864 |     0.000 |     0.004 |     0.000 |     0.000 |     0.031 |     0.022 |     0.000 |     0.009 |     0.004 |     0.000 |     0.000 |     0.009 |     0.000 |     0.004 |     0.038 | 
##               |     0.000 |     0.000 |     0.000 |     0.000 |     0.017 |     0.000 |     0.012 |     0.010 |     0.009 |     0.005 |     0.000 |     0.925 |     0.000 |     0.004 |     0.000 |     0.000 |     0.031 |     0.020 |     0.000 |     0.008 |     0.004 |     0.000 |     0.000 |     0.008 |     0.000 |     0.005 |           | 
##               |     0.000 |     0.000 |     0.000 |     0.000 |     0.001 |     0.000 |     0.001 |     0.000 |     0.000 |     0.000 |     0.000 |     0.033 |     0.000 |     0.000 |     0.000 |     0.000 |     0.001 |     0.001 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |           | 
## --------------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|
##             M |         1 |         0 |         0 |         0 |         1 |         0 |         2 |         1 |         0 |         0 |         0 |         0 |       215 |         1 |         0 |         2 |         0 |         1 |         0 |         0 |         4 |         3 |         6 |         0 |         0 |         0 |       237 | 
##               |     0.004 |     0.000 |     0.000 |     0.000 |     0.004 |     0.000 |     0.008 |     0.004 |     0.000 |     0.000 |     0.000 |     0.000 |     0.907 |     0.004 |     0.000 |     0.008 |     0.000 |     0.004 |     0.000 |     0.000 |     0.017 |     0.013 |     0.025 |     0.000 |     0.000 |     0.000 |     0.040 | 
##               |     0.004 |     0.000 |     0.000 |     0.000 |     0.004 |     0.000 |     0.008 |     0.005 |     0.000 |     0.000 |     0.000 |     0.000 |     0.931 |     0.004 |     0.000 |     0.009 |     0.000 |     0.004 |     0.000 |     0.000 |     0.016 |     0.013 |     0.026 |     0.000 |     0.000 |     0.000 |           | 
##               |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.036 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.001 |     0.001 |     0.001 |     0.000 |     0.000 |     0.000 |           | 
## --------------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|
##             N |         3 |         0 |         0 |         1 |         0 |         0 |         0 |         3 |         0 |         0 |         0 |         0 |         0 |       212 |         5 |         0 |         2 |         0 |         0 |         0 |         4 |         1 |         2 |         0 |         1 |         0 |       234 | 
##               |     0.013 |     0.000 |     0.000 |     0.004 |     0.000 |     0.000 |     0.000 |     0.013 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.906 |     0.021 |     0.000 |     0.009 |     0.000 |     0.000 |     0.000 |     0.017 |     0.004 |     0.009 |     0.000 |     0.004 |     0.000 |     0.039 | 
##               |     0.013 |     0.000 |     0.000 |     0.004 |     0.000 |     0.000 |     0.000 |     0.015 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.876 |     0.021 |     0.000 |     0.009 |     0.000 |     0.000 |     0.000 |     0.016 |     0.004 |     0.009 |     0.000 |     0.004 |     0.000 |           | 
##               |     0.001 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.001 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.035 |     0.001 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.001 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |           | 
## --------------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|
##             O |         0 |         1 |         2 |         8 |         0 |         0 |         2 |         1 |         0 |         0 |         0 |         0 |         0 |         4 |       186 |         0 |        10 |         6 |         0 |         0 |         3 |         0 |         0 |         2 |         0 |         0 |       225 | 
##               |     0.000 |     0.004 |     0.009 |     0.036 |     0.000 |     0.000 |     0.009 |     0.004 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.018 |     0.827 |     0.000 |     0.044 |     0.027 |     0.000 |     0.000 |     0.013 |     0.000 |     0.000 |     0.009 |     0.000 |     0.000 |     0.038 | 
##               |     0.000 |     0.004 |     0.009 |     0.033 |     0.000 |     0.000 |     0.008 |     0.005 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.017 |     0.788 |     0.000 |     0.044 |     0.024 |     0.000 |     0.000 |     0.012 |     0.000 |     0.000 |     0.008 |     0.000 |     0.000 |           | 
##               |     0.000 |     0.000 |     0.000 |     0.001 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.001 |     0.031 |     0.000 |     0.002 |     0.001 |     0.000 |     0.000 |     0.001 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |           | 
## --------------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|
##             P |         1 |         1 |         0 |         2 |         0 |        22 |         2 |         3 |         0 |         3 |         0 |         0 |         1 |         1 |         2 |       193 |         3 |         3 |         0 |         1 |         1 |         0 |         1 |         0 |         0 |         0 |       240 | 
##               |     0.004 |     0.004 |     0.000 |     0.008 |     0.000 |     0.092 |     0.008 |     0.013 |     0.000 |     0.013 |     0.000 |     0.000 |     0.004 |     0.004 |     0.008 |     0.804 |     0.013 |     0.013 |     0.000 |     0.004 |     0.004 |     0.000 |     0.004 |     0.000 |     0.000 |     0.000 |     0.040 | 
##               |     0.004 |     0.004 |     0.000 |     0.008 |     0.000 |     0.095 |     0.008 |     0.015 |     0.000 |     0.014 |     0.000 |     0.000 |     0.004 |     0.004 |     0.008 |     0.889 |     0.013 |     0.012 |     0.000 |     0.004 |     0.004 |     0.000 |     0.004 |     0.000 |     0.000 |     0.000 |           | 
##               |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.004 |     0.000 |     0.001 |     0.000 |     0.001 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.032 |     0.001 |     0.001 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |           | 
## --------------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|
##             Q |         2 |         1 |         2 |         1 |         2 |         0 |         5 |         6 |         0 |         1 |         0 |         1 |         2 |         1 |        17 |         1 |       180 |         4 |         1 |         0 |         3 |         1 |         0 |         0 |         0 |         3 |       234 | 
##               |     0.009 |     0.004 |     0.009 |     0.004 |     0.009 |     0.000 |     0.021 |     0.026 |     0.000 |     0.004 |     0.000 |     0.004 |     0.009 |     0.004 |     0.073 |     0.004 |     0.769 |     0.017 |     0.004 |     0.000 |     0.013 |     0.004 |     0.000 |     0.000 |     0.000 |     0.013 |     0.039 | 
##               |     0.009 |     0.004 |     0.009 |     0.004 |     0.008 |     0.000 |     0.020 |     0.031 |     0.000 |     0.005 |     0.000 |     0.005 |     0.009 |     0.004 |     0.072 |     0.005 |     0.793 |     0.016 |     0.004 |     0.000 |     0.012 |     0.004 |     0.000 |     0.000 |     0.000 |     0.014 |           | 
##               |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.001 |     0.001 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.003 |     0.000 |     0.030 |     0.001 |     0.000 |     0.000 |     0.001 |     0.000 |     0.000 |     0.000 |     0.000 |     0.001 |           | 
## --------------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|
##             R |         3 |        19 |         2 |        12 |         2 |         0 |         3 |         5 |         0 |         0 |         3 |         2 |         0 |         5 |         2 |         0 |         3 |       162 |         0 |         1 |         3 |         0 |         0 |         0 |         0 |         0 |       227 | 
##               |     0.013 |     0.084 |     0.009 |     0.053 |     0.009 |     0.000 |     0.013 |     0.022 |     0.000 |     0.000 |     0.013 |     0.009 |     0.000 |     0.022 |     0.009 |     0.000 |     0.013 |     0.714 |     0.000 |     0.004 |     0.013 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.038 | 
##               |     0.013 |     0.075 |     0.009 |     0.049 |     0.008 |     0.000 |     0.012 |     0.026 |     0.000 |     0.000 |     0.015 |     0.009 |     0.000 |     0.021 |     0.008 |     0.000 |     0.013 |     0.638 |     0.000 |     0.004 |     0.012 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |           | 
##               |     0.001 |     0.003 |     0.000 |     0.002 |     0.000 |     0.000 |     0.001 |     0.001 |     0.000 |     0.000 |     0.001 |     0.000 |     0.000 |     0.001 |     0.000 |     0.000 |     0.001 |     0.027 |     0.000 |     0.000 |     0.001 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |           | 
## --------------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|
##             S |         2 |         7 |         0 |         4 |        10 |         2 |         1 |         0 |         1 |         0 |         1 |         1 |         0 |         0 |         1 |         0 |         0 |         2 |       185 |         0 |         0 |         0 |         0 |         2 |         0 |         5 |       224 | 
##               |     0.009 |     0.031 |     0.000 |     0.018 |     0.045 |     0.009 |     0.004 |     0.000 |     0.004 |     0.000 |     0.004 |     0.004 |     0.000 |     0.000 |     0.004 |     0.000 |     0.000 |     0.009 |     0.826 |     0.000 |     0.000 |     0.000 |     0.000 |     0.009 |     0.000 |     0.022 |     0.037 | 
##               |     0.009 |     0.028 |     0.000 |     0.016 |     0.042 |     0.009 |     0.004 |     0.000 |     0.004 |     0.000 |     0.005 |     0.005 |     0.000 |     0.000 |     0.004 |     0.000 |     0.000 |     0.008 |     0.808 |     0.000 |     0.000 |     0.000 |     0.000 |     0.008 |     0.000 |     0.024 |           | 
##               |     0.000 |     0.001 |     0.000 |     0.001 |     0.002 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.031 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.001 |           | 
## --------------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|
##             T |         0 |         0 |         0 |         2 |         1 |         7 |         2 |         0 |         1 |         0 |         0 |         2 |         0 |         0 |         0 |         2 |         1 |         1 |         1 |       205 |         2 |         1 |         0 |         4 |         5 |         1 |       238 | 
##               |     0.000 |     0.000 |     0.000 |     0.008 |     0.004 |     0.029 |     0.008 |     0.000 |     0.004 |     0.000 |     0.000 |     0.008 |     0.000 |     0.000 |     0.000 |     0.008 |     0.004 |     0.004 |     0.004 |     0.861 |     0.008 |     0.004 |     0.000 |     0.017 |     0.021 |     0.004 |     0.040 | 
##               |     0.000 |     0.000 |     0.000 |     0.008 |     0.004 |     0.030 |     0.008 |     0.000 |     0.004 |     0.000 |     0.000 |     0.009 |     0.000 |     0.000 |     0.000 |     0.009 |     0.004 |     0.004 |     0.004 |     0.827 |     0.008 |     0.004 |     0.000 |     0.016 |     0.022 |     0.005 |           | 
##               |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.001 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.034 |     0.000 |     0.000 |     0.000 |     0.001 |     0.001 |     0.000 |           | 
## --------------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|
##             U |         1 |         0 |         3 |         0 |         0 |         0 |         2 |         3 |         0 |         0 |         0 |         1 |         0 |         5 |         1 |         0 |         3 |         0 |         0 |         2 |       220 |         1 |         0 |         0 |         1 |         0 |       243 | 
##               |     0.004 |     0.000 |     0.012 |     0.000 |     0.000 |     0.000 |     0.008 |     0.012 |     0.000 |     0.000 |     0.000 |     0.004 |     0.000 |     0.021 |     0.004 |     0.000 |     0.012 |     0.000 |     0.000 |     0.008 |     0.905 |     0.004 |     0.000 |     0.000 |     0.004 |     0.000 |     0.041 | 
##               |     0.004 |     0.000 |     0.014 |     0.000 |     0.000 |     0.000 |     0.008 |     0.015 |     0.000 |     0.000 |     0.000 |     0.005 |     0.000 |     0.021 |     0.004 |     0.000 |     0.013 |     0.000 |     0.000 |     0.008 |     0.887 |     0.004 |     0.000 |     0.000 |     0.004 |     0.000 |           | 
##               |     0.000 |     0.000 |     0.001 |     0.000 |     0.000 |     0.000 |     0.000 |     0.001 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.001 |     0.000 |     0.000 |     0.001 |     0.000 |     0.000 |     0.000 |     0.037 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |           | 
## --------------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|
##             V |         0 |         2 |         0 |         0 |         2 |         0 |         1 |         1 |         0 |         0 |         0 |         0 |         0 |         0 |         0 |         1 |         1 |         1 |         0 |         1 |         0 |       203 |         8 |         0 |         8 |         0 |       229 | 
##               |     0.000 |     0.009 |     0.000 |     0.000 |     0.009 |     0.000 |     0.004 |     0.004 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.004 |     0.004 |     0.004 |     0.000 |     0.004 |     0.000 |     0.886 |     0.035 |     0.000 |     0.035 |     0.000 |     0.038 | 
##               |     0.000 |     0.008 |     0.000 |     0.000 |     0.008 |     0.000 |     0.004 |     0.005 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.005 |     0.004 |     0.004 |     0.000 |     0.004 |     0.000 |     0.890 |     0.035 |     0.000 |     0.035 |     0.000 |           | 
##               |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.034 |     0.001 |     0.000 |     0.001 |     0.000 |           | 
## --------------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|
##             W |         0 |         0 |         0 |         0 |         1 |         0 |         1 |         0 |         0 |         0 |         0 |         0 |         4 |         1 |         2 |         1 |         0 |         0 |         0 |         0 |         0 |         3 |       212 |         0 |         0 |         0 |       225 | 
##               |     0.000 |     0.000 |     0.000 |     0.000 |     0.004 |     0.000 |     0.004 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.018 |     0.004 |     0.009 |     0.004 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.013 |     0.942 |     0.000 |     0.000 |     0.000 |     0.038 | 
##               |     0.000 |     0.000 |     0.000 |     0.000 |     0.004 |     0.000 |     0.004 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.017 |     0.004 |     0.008 |     0.005 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.013 |     0.918 |     0.000 |     0.000 |     0.000 |           | 
##               |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.001 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.001 |     0.035 |     0.000 |     0.000 |     0.000 |           | 
## --------------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|
##             X |         0 |         4 |         0 |         2 |        13 |         0 |         0 |         3 |         0 |         1 |         3 |         1 |         0 |         0 |         1 |         0 |         1 |         0 |         1 |         4 |         1 |         0 |         0 |       199 |         2 |         0 |       236 | 
##               |     0.000 |     0.017 |     0.000 |     0.008 |     0.055 |     0.000 |     0.000 |     0.013 |     0.000 |     0.004 |     0.013 |     0.004 |     0.000 |     0.000 |     0.004 |     0.000 |     0.004 |     0.000 |     0.004 |     0.017 |     0.004 |     0.000 |     0.000 |     0.843 |     0.008 |     0.000 |     0.039 | 
##               |     0.000 |     0.016 |     0.000 |     0.008 |     0.055 |     0.000 |     0.000 |     0.015 |     0.000 |     0.005 |     0.015 |     0.005 |     0.000 |     0.000 |     0.004 |     0.000 |     0.004 |     0.000 |     0.004 |     0.016 |     0.004 |     0.000 |     0.000 |     0.787 |     0.009 |     0.000 |           | 
##               |     0.000 |     0.001 |     0.000 |     0.000 |     0.002 |     0.000 |     0.000 |     0.001 |     0.000 |     0.000 |     0.001 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.001 |     0.000 |     0.000 |     0.000 |     0.033 |     0.000 |     0.000 |           | 
## --------------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|
##             Y |         0 |         3 |         0 |         3 |         0 |         3 |         3 |         1 |         0 |         1 |         2 |         0 |         1 |         2 |         0 |         1 |         1 |         3 |         0 |        14 |         0 |         4 |         0 |         1 |       192 |         0 |       235 | 
##               |     0.000 |     0.013 |     0.000 |     0.013 |     0.000 |     0.013 |     0.013 |     0.004 |     0.000 |     0.004 |     0.009 |     0.000 |     0.004 |     0.009 |     0.000 |     0.004 |     0.004 |     0.013 |     0.000 |     0.060 |     0.000 |     0.017 |     0.000 |     0.004 |     0.817 |     0.000 |     0.039 | 
##               |     0.000 |     0.012 |     0.000 |     0.012 |     0.000 |     0.013 |     0.012 |     0.005 |     0.000 |     0.005 |     0.010 |     0.000 |     0.004 |     0.008 |     0.000 |     0.005 |     0.004 |     0.012 |     0.000 |     0.056 |     0.000 |     0.018 |     0.000 |     0.004 |     0.846 |     0.000 |           | 
##               |     0.000 |     0.001 |     0.000 |     0.001 |     0.000 |     0.001 |     0.001 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.001 |     0.000 |     0.002 |     0.000 |     0.001 |     0.000 |     0.000 |     0.032 |     0.000 |           | 
## --------------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|
##             Z |         0 |         8 |         0 |         3 |         3 |         3 |         1 |         1 |         0 |         0 |         0 |         0 |         0 |         0 |         0 |         0 |         1 |         1 |         2 |         0 |         0 |         0 |         0 |         6 |         1 |       190 |       220 | 
##               |     0.000 |     0.036 |     0.000 |     0.014 |     0.014 |     0.014 |     0.005 |     0.005 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.005 |     0.005 |     0.009 |     0.000 |     0.000 |     0.000 |     0.000 |     0.027 |     0.005 |     0.864 |     0.037 | 
##               |     0.000 |     0.032 |     0.000 |     0.012 |     0.013 |     0.013 |     0.004 |     0.005 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.004 |     0.004 |     0.009 |     0.000 |     0.000 |     0.000 |     0.000 |     0.024 |     0.004 |     0.905 |           | 
##               |     0.000 |     0.001 |     0.000 |     0.001 |     0.001 |     0.001 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.000 |     0.001 |     0.000 |     0.032 |           | 
## --------------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|
##  Column Total |       233 |       252 |       214 |       245 |       238 |       232 |       246 |       194 |       223 |       212 |       202 |       213 |       231 |       242 |       236 |       217 |       227 |       254 |       229 |       248 |       248 |       228 |       231 |       253 |       227 |       210 |      5985 | 
##               |     0.039 |     0.042 |     0.036 |     0.041 |     0.040 |     0.039 |     0.041 |     0.032 |     0.037 |     0.035 |     0.034 |     0.036 |     0.039 |     0.040 |     0.039 |     0.036 |     0.038 |     0.042 |     0.038 |     0.041 |     0.041 |     0.038 |     0.039 |     0.042 |     0.038 |     0.035 |           | 
## --------------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|
## 
## 
# Calculate prediction success rate for all letters
mean(diag(s$prop.col))
## [1] 0.833701
# Check prediction success rate for each letter
diag(s$prop.col)
##         A         B         C         D         E         F         G 
## 0.9098712 0.6547619 0.9252336 0.7510204 0.7478992 0.8017241 0.7601626 
##         H         I         J         K         L         M         N 
## 0.7216495 0.9058296 0.9150943 0.8762376 0.9248826 0.9307359 0.8760331 
##         O         P         Q         R         S         T         U 
## 0.7881356 0.8894009 0.7929515 0.6377953 0.8078603 0.8266129 0.8870968 
##         V         W         X         Y         Z 
## 0.8903509 0.9177489 0.7865613 0.8458150 0.9047619

Even when all SVD variables (16) were used, the prediction rate is around 93.4% (lower than the original of 94.8%). When we use less (e.g. 8), the prediction rate is getting worst (83.4%). It seems that SVD does not help us improve the prediction model. Almost all original variables have variability (add information) so they are useful for predicting the letter

Hierarchical clustering

In this part of the analysis the objective is to assign the cases into clusters. Cluster analysis or clustering is the task of grouping a set of objects in such a way that objects in the same group (called a cluster) are more similar (in some sense or another) to each other than to those in other groups (clusters). Finally i build a hierarchical clustering model in order to understand which letters are “closer” to others and maybe misclassified. We applied it on the pre-processed data set (letter.cluster).

# Tranform the summary dataset to a dataframe
letter.cluster <- as.data.frame(letter.cluster)

# Assign letters variable as rownames
rownames(letter.cluster) <- letter.cluster$lettr

# Run the hierarchical clustering 
clusters <- hclust(dist(letter.cluster))

At the plot below you can see the hierarchical clustering output. When we compare these results with the misclassification rates of the original model, it seems that for some letters like “B”, “D”, “F” it looks relevant. For other letters such “A”, “C” don’t match.

# Plot the dendogram
ggdendrogram(clusters, rotate = TRUE, size = 2)+
  labs(title = "Dendogram of hierarchical clustering model")

RESULTS

Finally, after testing various models using k-NN algorithm, the best model used the original predictors & 3 nearest neighbors (k=3). The accuracy obtained from from the final model is 94,8%. I also tried the SVD technique in order to check if the new uncorrelated variables would help us improve the prediction accuracy rate of the letter in a new observation. But the model wasn’t improved.