Introduction

We recently released Neighbr, a package for performing k-nearest neighbor classification and regression. Highlights of version 1.0 include:

• comparison measures that support continuous or logical features
• support for categorical and continuous targets
• neighbor ranking

Neighbr models can also be converted to the PMML (Predictive Model Markup Language) standard using the pmml R package.

In this blog post, we will provide some examples of how to use neighbr to create knn models.

Examples

First, load necessary libraries and set the seed and number display options. knitr::kable is used to display data frames.

library(neighbr)
library(knitr)
set.seed(123)
options(digits=3)

Continuous features and categorical target

This example shows using squared euclidean distance with 3 neighbors to classify the Species of flowers in the iris dataset. Each training instance consists of 4 features and 1 class variable. The categorical target is predicted by a majority vote from the closest k neighbors. The knn() function requires that all columns in test_set are feature columns, and have the same names and are in the same order as the features in train_set. The train_set is assumed to only contain features and targets (one categorical, one continuous, and/or ID for neighbor ranking); i.e., if a column name is not specified as a target, it is assumed to be a feature. The fit object contains predictions for test_set in fit$test_set_scores (there is no predict method for knn). data(iris) train_set <- iris[1:147,] #train set contains all targets and features test_set <- iris[148:150,!names(iris) %in% c("Species")] #test set does not contain any targets #run knn function fit <- knn(train_set=train_set,test_set=test_set, k=3, categorical_target="Species", comparison_measure="squared_euclidean") #show predictions kable(fit$test_set_scores)
categorical_target
148 virginica
149 virginica
150 virginica

The returned data frame contains predictions for the categorical target (Species).

Mixed targets and neighbor ranking

It is possible to predict categorical and continuous targets simultaneously, as well as to return the IDs of closest neighbors of a given instance. In the next example, an ID column is added to the data for ranking, and Petal.Width is used as a continuous target. By default, the prediction for the continuous target is calculated by averaging the closest k neighbors.

data(iris)
iris$ID <- c(1:150) #an ID column is necessary if ranks are to be calculated train_set <- iris[1:147,] #train set contains all predicted variables, features, and ID column test_set <- iris[148:150,!names(iris) %in% c("Petal.Width","Species","ID")] #test set does not contain predicted variables or ID column fit <- knn(train_set=train_set,test_set=test_set, k=3, categorical_target="Species", continuous_target= "Petal.Width", comparison_measure="squared_euclidean", return_ranked_neighbors=3, id="ID") kable(fit$test_set_scores)
categorical_target continuous_target neighbor1 neighbor2 neighbor3
148 virginica 2.20 146 111 116
149 virginica 2.17 137 116 138
150 virginica 1.93 115 128 84

The ranked neighbor IDs are returned along with the categorical and continuous targets, with neghbor1 being the closest in terms of distance. If a similarity measure were being used, neighbor1 would be the most similar. Any number of neighbors can be returned, as long as return_ranked_neighbors <= k.

Neighbor ranking without targets

It is possible to get neighbor ranks without a target variable. In this unsupervised learning case, continuous_target and categorical_target are left as NULL by default.

data(iris)
iris$ID <- c(1:150) #an ID column is necessary if ranks are to be calculated train_set <- iris[1:147,-c(5)] #remove Species categorical variable test_set <- iris[148:150,!names(iris) %in% c("Species","ID")] #test set does not contain predicted variables or ID column fit <- knn(train_set=train_set,test_set=test_set, k=5, comparison_measure="squared_euclidean", return_ranked_neighbors=4, id="ID") kable(fit$test_set_scores)
neighbor1 neighbor2 neighbor3 neighbor4
148 111 112 117 146
149 137 116 111 141
150 128 139 102 143

Logical features

The package supports logical features, to be used with an appropriate similarity measure. This example demonstrates predicting a categorical target and ranking neighbors for the HouseVotes84 dataset (from the mlbench package). The features may be logical consisting of {TRUE, FALSE} or numeric vectors consisting of {0,1}, but not factors. In this example, the factor features are converted to numeric vectors.

library(mlbench)

# change all {yes,no} factors to {0,1}
feature_names <- names(dat)[!names(dat) %in% c("Class","ID")]
for (n in feature_names) {
levels(dat[,n])[levels(dat[,n])=="n"] <- 0
levels(dat[,n])[levels(dat[,n])=="y"] <- 1
}

# change factors to numeric
for (n in feature_names) {dat[,n] <- as.numeric(levels(dat[,n]))[dat[,n]]}

dat$ID <- c(1:nrow(dat)) #an ID column is necessary if ranks are to be calculated train_set <- dat[1:227,] test_set <- dat[228:232,!names(dat) %in% c("Class","ID")] #test set does not contain predicted variables or ID column house_fit <- knn(train_set=train_set,test_set=test_set, k=7, categorical_target = "Class", comparison_measure="jaccard", return_ranked_neighbors=3, id="ID") kable(house_fit$test_set_scores)
categorical_target neighbor1 neighbor2 neighbor3
424 democrat 114 96 112
427 democrat 5 47 91
428 republican 70 156 155
431 republican 115 117 152
432 democrat 57 130 135

Comparison measures

Distance measures are used for vectors with continuous elements. Similarity measures are used for logical vectors. The comparison measures used in neighbr are based on those defined in the PMML standard.

Functions in neighbr can be used to calculate distances or similarities between vectors directly:

distance(c(1,2.3,2.9,0.4),c(-0.3,5.3,2.9,3.3),"euclidean")
#> [1] 4.37
similarity(c(0,1,0,1,1,1),c(1,1,0,1,1,0),"tanimoto")
#> [1] 0.5
similarity(c(0,1,0,1,1,1),c(1,1,0,1,1,0),"jaccard")
#> [1] 0.6

To check which measures are available, run ?distance and ?similarity in your R session.

Neighbr and PMML

This package was developed following the KNN specification in the PMML (Predictive Model Markup Language) standard. The models produced by neighbr can be converted to PMML (using the pmml R package).

For example, to convert the model for HouseVotes84 data above:

library(pmml)
house_fit_pmml <- pmml(house_fit)

Additional examples and details are available in the neighbr vignette, which can also be accessed from an R session by running vignette("neighbr-help").

For additional examples on converting neighbr models to PMML, run ?pmml.neighbr after loading the pmml package in R.