ISYE 6501 - Midterm 2
Study online at https://quizlet.com/_4o5wwk
1. when might overfitting occur when the # of factors is close to or larger
than the # of data points causing the
model to potentially fit too closely to
random effects
2. Why are simple models better than complex less data is required; less chance of in-
ones significant factors and easier to inter-
pret
3. what is forward selection we select the best new factor and see if
it's good enough (R^2, AIC, or p-value)
add it to our model and fit the model
with the current set of factors. Then at
the end we remove factors that are low-
er than a certain threshold
4. what is backward elimination we start with all factors and find the
worst on a supplied threshold (p =
0.15). If it is worse we remove it and
start the process over. We do that un-
til we have the number of factors that
we want and then we move the factors
lower than a second threshold (p = .05)
and fit the model with all set of factors
5. what is stepwise regression
, ISYE 6501 - Midterm 2
Study online at https://quizlet.com/_4o5wwk
it is a combination of forward selection
and backward elimination. We can ei-
ther start with all factors or no factors
and at each step we remove or add a
factor. As we go through the procedure
after adding each new factor and at the
end we eliminate right away factors that
no longer appear.
6. what type of algorithms are stepwise selec- Greedy algorithms - at each step they
tion? take one thing that looks best
7. what is LASSO a variable selection method where the
coefficients are determined by both
minimizing the squared error and the
sum of their absolute value not being
over a certain threshold t
8. How do you choose t in LASSO use the lasso approach with different
values of t and see which gives the best
trade off
9. why do we have to scale the data for LASSO if we don't the measure of the data will
artificially affect how big the coefficients
need to be
10. What is elastic net? A variable selection method that works
by minimizing the squared error and
constraining the combination of ab-
, ISYE 6501 - Midterm 2
Study online at https://quizlet.com/_4o5wwk
solute values of coefficients and their
squares
11. what is a key difference between stepwise If the data is not scaled, the coefficients
regresson and lasso regression can have artificially different orders of
magnitude, which means they'll have
unbalanced effects on the lasso con-
straint.
12. Why doesn't Ridge Regression perform vari- The coefficients values are squared so
able selection? they go closer to zero or regularizes
them
13. What are the pros and cons of Greedy Algo- Good for initial analysis but often don't
rithms (Forward selection, stepwise elimina- perform as well on other data because
tion, stepwise regression) they fit more to random effects than
you'd like and appear to have a better
fit
14. What are the pros and cons of LASSO and They are slower but help make models
elastic net that make better predictions
15. Which two methods does elastic net look like Ridge Regression and LASSO.
it combines and what are the downsides from
it? Advantages: variable selection from
LASSO and Predictive benefits of LAS-
SO.
Disadvantages: Arbitrarily rules out
some correlated variables like LASSO
(don't know which one that is left out
should be); Underestimates coefficients
Study online at https://quizlet.com/_4o5wwk
1. when might overfitting occur when the # of factors is close to or larger
than the # of data points causing the
model to potentially fit too closely to
random effects
2. Why are simple models better than complex less data is required; less chance of in-
ones significant factors and easier to inter-
pret
3. what is forward selection we select the best new factor and see if
it's good enough (R^2, AIC, or p-value)
add it to our model and fit the model
with the current set of factors. Then at
the end we remove factors that are low-
er than a certain threshold
4. what is backward elimination we start with all factors and find the
worst on a supplied threshold (p =
0.15). If it is worse we remove it and
start the process over. We do that un-
til we have the number of factors that
we want and then we move the factors
lower than a second threshold (p = .05)
and fit the model with all set of factors
5. what is stepwise regression
, ISYE 6501 - Midterm 2
Study online at https://quizlet.com/_4o5wwk
it is a combination of forward selection
and backward elimination. We can ei-
ther start with all factors or no factors
and at each step we remove or add a
factor. As we go through the procedure
after adding each new factor and at the
end we eliminate right away factors that
no longer appear.
6. what type of algorithms are stepwise selec- Greedy algorithms - at each step they
tion? take one thing that looks best
7. what is LASSO a variable selection method where the
coefficients are determined by both
minimizing the squared error and the
sum of their absolute value not being
over a certain threshold t
8. How do you choose t in LASSO use the lasso approach with different
values of t and see which gives the best
trade off
9. why do we have to scale the data for LASSO if we don't the measure of the data will
artificially affect how big the coefficients
need to be
10. What is elastic net? A variable selection method that works
by minimizing the squared error and
constraining the combination of ab-
, ISYE 6501 - Midterm 2
Study online at https://quizlet.com/_4o5wwk
solute values of coefficients and their
squares
11. what is a key difference between stepwise If the data is not scaled, the coefficients
regresson and lasso regression can have artificially different orders of
magnitude, which means they'll have
unbalanced effects on the lasso con-
straint.
12. Why doesn't Ridge Regression perform vari- The coefficients values are squared so
able selection? they go closer to zero or regularizes
them
13. What are the pros and cons of Greedy Algo- Good for initial analysis but often don't
rithms (Forward selection, stepwise elimina- perform as well on other data because
tion, stepwise regression) they fit more to random effects than
you'd like and appear to have a better
fit
14. What are the pros and cons of LASSO and They are slower but help make models
elastic net that make better predictions
15. Which two methods does elastic net look like Ridge Regression and LASSO.
it combines and what are the downsides from
it? Advantages: variable selection from
LASSO and Predictive benefits of LAS-
SO.
Disadvantages: Arbitrarily rules out
some correlated variables like LASSO
(don't know which one that is left out
should be); Underestimates coefficients
