ISYE 7406 Homework 1
1. Introduction
This homework examines fundamental classification methods applied to a real-world, high-dimensional
dataset and evaluates their performance using training, testing, and Monte Carlo cross-validation. The
objective is to distinguish between handwritten digits 2 and 7 from the well-known ZIP code dataset, a
classic benchmark in pattern recognition and image classification. Each observation consists of a 16×16
grayscale image, highlighting common challenges in supervised learning such as high dimensionality, noise,
and model complexity.
Two families of classifiers are examined: linear regression used as a classifier and the k-nearest neighbors
(KNN) method with multiple choices of k. By comparing these approaches under different error estimation
schemes, the analysis illustrates how model flexibility and tuning parameters influence predictive
accuracy, overfitting, and robustness.
2. Exploratory data analysis
The training dataset, denoted as ziptrain27, contains 1,376 observations and 257 variables. The first
column represents the digit label (2 or 7), while the remaining 256 columns correspond to pixel intensities
of a 16×16 grayscale image, with values bounded in the interval [−1,1].
Among the training observations, 731 correspond to digit 2 and 645 to digit 7, indicating a reasonably
balanced class distribution. Because the response is balanced and misclassification costs are assumed to
be symmetric, overall classification error is an appropriate performance metric, and no class weighting or
resampling is required.
Each observation can be reshaped into a 16×16 matrix and visualized as a grayscale image. Visual
inspection of several examples (for instance, row 5, which clearly represents digit 7) confirms that digits
2 and 7 exhibit distinct structural patterns. This observation supports the feasibility of accurate
classification despite the high dimensionality of the feature space. Since all pixel features are measured
on a comparable numeric scale within [−1,1], no additional standardization was performed. Shown below
are visual representations of the digit 7 using two different grayscale parameter ranges (left: 0–1; right:
0–32/32). These representations demonstrate how grayscale scaling affects the sharpness and clarity of
the digit.
-1-
, 3. Methods
3.1 Linear regression model classifier
A linear regression model was fitted using the training dataset ziptrain27, with the digit label Y∈{2,7} as
the response and the 256 pixel intensities as predictors. Predicted values of Y were converted to class
labels by applying a threshold of 4.5, the midpoint between 2 and 7: observations with Ŷ < 4.5 were
classified as digit 2, and those with Ŷ ≥ 4.5 were classified as digit 7.
This approach treats linear regression as a simple parametric classifier, inducing a linear decision boundary
in the 256-dimensional feature space. Although it does not capture potential nonlinear structure in the
data, the model provides a useful baseline with relatively low variance compared to more flexible,
nonparametric methods.
3.2 K-nearest neighbors (KNN)
The k-nearest neighbors (KNN) classifier was implemented using Euclidean distance in the original pixel
feature space. Eight values of the tuning parameter were considered: k∈{1,3,5,7,9,11,13,15}.
For each test observation, the classifier identifies the k nearest training samples and assigns the class label
by majority vote among their corresponding labels. Smaller values of k produce highly flexible decision
rules with low bias but high variance, whereas larger values of k yield smoother classifiers characterized
by higher bias and lower variance.
3.3 Error estimation procedures
Model performance was evaluated using three complementary approaches.
• Training error
For each method, the empirical misclassification rate was computed on the original training
dataset ziptrain27. While this measure reflects goodness-of-fit, it typically underestimates the
true predictive error.
• Testing error on an independent test set
An independent testing dataset, ziptest27, was constructed by subsetting the original ZIP test data
to digits 2 and 7. Misclassification rates were computed on ziptest27 using models trained
exclusively on ziptrain27.
• Monte Carlo cross-validation (MC-CV)
The full dataset of digits 2 and 7 was formed by combining ziptrain27 and ziptest27 into zip27full,
yielding a total sample size of 1,721. For each of B = 100 runs, 1,376 observations were randomly
selected as a temporary training set, with the remaining 345 used as a temporary test set. All nine
models (linear regression and eight KNN variants) were trained on the temporary training sets
and evaluated on the corresponding test sets. Average cross-validation testing errors and their
sample variances were then computed across the B runs for each method.
An alternative Monte Carlo cross-validation (MC-CV) scheme, also implemented in the provided code,
resamples digits 2 and 7 separately to preserve class balance in each split and records both average
training and testing errors across runs. This approach leads to qualitatively similar conclusions; therefore,
detailed numerical results are omitted for brevity.
4. Results
4.1 Training and testing errors
Table 1 summarizes the training errors (on ziptrain27) and the testing errors (on ziptest27) for all nine
models.
-2-
1. Introduction
This homework examines fundamental classification methods applied to a real-world, high-dimensional
dataset and evaluates their performance using training, testing, and Monte Carlo cross-validation. The
objective is to distinguish between handwritten digits 2 and 7 from the well-known ZIP code dataset, a
classic benchmark in pattern recognition and image classification. Each observation consists of a 16×16
grayscale image, highlighting common challenges in supervised learning such as high dimensionality, noise,
and model complexity.
Two families of classifiers are examined: linear regression used as a classifier and the k-nearest neighbors
(KNN) method with multiple choices of k. By comparing these approaches under different error estimation
schemes, the analysis illustrates how model flexibility and tuning parameters influence predictive
accuracy, overfitting, and robustness.
2. Exploratory data analysis
The training dataset, denoted as ziptrain27, contains 1,376 observations and 257 variables. The first
column represents the digit label (2 or 7), while the remaining 256 columns correspond to pixel intensities
of a 16×16 grayscale image, with values bounded in the interval [−1,1].
Among the training observations, 731 correspond to digit 2 and 645 to digit 7, indicating a reasonably
balanced class distribution. Because the response is balanced and misclassification costs are assumed to
be symmetric, overall classification error is an appropriate performance metric, and no class weighting or
resampling is required.
Each observation can be reshaped into a 16×16 matrix and visualized as a grayscale image. Visual
inspection of several examples (for instance, row 5, which clearly represents digit 7) confirms that digits
2 and 7 exhibit distinct structural patterns. This observation supports the feasibility of accurate
classification despite the high dimensionality of the feature space. Since all pixel features are measured
on a comparable numeric scale within [−1,1], no additional standardization was performed. Shown below
are visual representations of the digit 7 using two different grayscale parameter ranges (left: 0–1; right:
0–32/32). These representations demonstrate how grayscale scaling affects the sharpness and clarity of
the digit.
-1-
, 3. Methods
3.1 Linear regression model classifier
A linear regression model was fitted using the training dataset ziptrain27, with the digit label Y∈{2,7} as
the response and the 256 pixel intensities as predictors. Predicted values of Y were converted to class
labels by applying a threshold of 4.5, the midpoint between 2 and 7: observations with Ŷ < 4.5 were
classified as digit 2, and those with Ŷ ≥ 4.5 were classified as digit 7.
This approach treats linear regression as a simple parametric classifier, inducing a linear decision boundary
in the 256-dimensional feature space. Although it does not capture potential nonlinear structure in the
data, the model provides a useful baseline with relatively low variance compared to more flexible,
nonparametric methods.
3.2 K-nearest neighbors (KNN)
The k-nearest neighbors (KNN) classifier was implemented using Euclidean distance in the original pixel
feature space. Eight values of the tuning parameter were considered: k∈{1,3,5,7,9,11,13,15}.
For each test observation, the classifier identifies the k nearest training samples and assigns the class label
by majority vote among their corresponding labels. Smaller values of k produce highly flexible decision
rules with low bias but high variance, whereas larger values of k yield smoother classifiers characterized
by higher bias and lower variance.
3.3 Error estimation procedures
Model performance was evaluated using three complementary approaches.
• Training error
For each method, the empirical misclassification rate was computed on the original training
dataset ziptrain27. While this measure reflects goodness-of-fit, it typically underestimates the
true predictive error.
• Testing error on an independent test set
An independent testing dataset, ziptest27, was constructed by subsetting the original ZIP test data
to digits 2 and 7. Misclassification rates were computed on ziptest27 using models trained
exclusively on ziptrain27.
• Monte Carlo cross-validation (MC-CV)
The full dataset of digits 2 and 7 was formed by combining ziptrain27 and ziptest27 into zip27full,
yielding a total sample size of 1,721. For each of B = 100 runs, 1,376 observations were randomly
selected as a temporary training set, with the remaining 345 used as a temporary test set. All nine
models (linear regression and eight KNN variants) were trained on the temporary training sets
and evaluated on the corresponding test sets. Average cross-validation testing errors and their
sample variances were then computed across the B runs for each method.
An alternative Monte Carlo cross-validation (MC-CV) scheme, also implemented in the provided code,
resamples digits 2 and 7 separately to preserve class balance in each split and records both average
training and testing errors across runs. This approach leads to qualitatively similar conclusions; therefore,
detailed numerical results are omitted for brevity.
4. Results
4.1 Training and testing errors
Table 1 summarizes the training errors (on ziptrain27) and the testing errors (on ziptest27) for all nine
models.
-2-
