ISYE 6501 Final
Study online at https://quizlet.com/_7l1l0g
1. Factor Based classification, clustering, regression. Implicitly assumed that we have a lot of factors
Models in the final model
2. Why limit num- overfitting: when # of factors is close to or larger than # of data points. Model may
ber of factors in a fit too closely to random effects
model? 2 reasons simplicity: simple models are usually better
3. Classical vari- 1. Forward selection
able selection ap- 2. Backwards elimination
proaches 3. Stepwise regression
greedy algorithms
4. Backward elimi- variable selection; classical
nation Opposite of forward selection. Start with model with all factors, at each step find
worst factor and remove from model. Continue until no more to add, # of factor
threshold is satisfied. Remove factors at the end that were not good enough
5. Forward selec- variable selection; classical
tion Start with model with no factors, at each step find best new factor to add. Continue
until none bad enough to remove, # of factor threshold is satisfied. Remove factors
at the end that were not good enough
6. Stepwise regres- variable selection; classical
sion Combination of forward selection and backwards elimination. Start with all or no
factors. Each step remove/add a factor. As it continues, after adding in new factor
we eliminate right away any factors that may be good. Helps model adjust when
new factors are added, goodness values change
7. Ways of deter- p-value, Rsquared, AIC, BIC
mining if factors
are good enough
in variable selec-
tion
, ISYE 6501 Final
Study online at https://quizlet.com/_7l1l0g
8. Greedy algorithm At each step, it does the one thing that looks best
without taking future options into consideration. Good for initial analysis
1. Forward selection
2. Backwards elimination
3. Stepwise regression
9. Global variable 1. LASSO
selection ap- 2. Elastic Net
proaches
Slower, but tend to give better predictive models
10. LASSO variable selection; global
- SCALE the date (as with any constrained sum of coefficients)
- add a constraint to the standard regression equation
- minimize sum of squared errors
- T = limit or "budget" on how large the sum of squared errors can get. Budget
will be used on most important coefficients
- Method for limiting the number of variables in a model by limiting the sum of
all coefficients' absolute values. Can be very helpful when number of data points
is less than number of factors.
11. Elastic Net variable selection; global
- SCALE the date (as with any constrained sum of coefficients)
- T = limit or "budget" on how large the sum of squared errors can get. Budget
will be used on most important coefficients
- Combination of lasso and ridge regression.
- Variable selection benefits of LASSO
- Predictive benefits of ridge regression
12. Ridge Regression
, ISYE 6501 Final
Study online at https://quizlet.com/_7l1l0g
- Method of regularization by limiting the sum of the squares of the coefficients.
Will reduce the magnitude of coefficients, not the number of variables chosen.
- The quadratic term in ridge regression
tends to shrink the coefficient values i.e Whatever the basic regression model
coefficients would be,
the quadratic constraint pushes them toward zero
or regularizes them.
13. Design of Experi- How can we still have a representative sample of each combination of factors, while
ments (DOE) only surveying 600 people?
How to determine which of the several factors are most
important to predicting someone's answers?
comparison to measure difference
control for other factors and effects
blocking factors that account for the variation between factors (red sports car vs
red minivan example)
14. A/B testing Whenever we want to choose between 2 alternatives.
As long as the following 3 things are true:
1st, we need to be able to collect a lot of data quickly enough to get an answer in
time to use it.
2nd, the data we collect has to be from a representative sample of the whole
3rd, the amount of data we collect has to be small compared to the total population
we want to use the answer on.
Before modeling and before collecting data
15. (Full) Factorial Test every combination of variables in an experiment to find each one's effect, and
Design interaction effects on the outcome.
16. Fractional Factor- A subset of combinations to test - selected combinations give same result as full
ial Design factorial design i.e a balanced design
Before modeling and before collecting data
Study online at https://quizlet.com/_7l1l0g
1. Factor Based classification, clustering, regression. Implicitly assumed that we have a lot of factors
Models in the final model
2. Why limit num- overfitting: when # of factors is close to or larger than # of data points. Model may
ber of factors in a fit too closely to random effects
model? 2 reasons simplicity: simple models are usually better
3. Classical vari- 1. Forward selection
able selection ap- 2. Backwards elimination
proaches 3. Stepwise regression
greedy algorithms
4. Backward elimi- variable selection; classical
nation Opposite of forward selection. Start with model with all factors, at each step find
worst factor and remove from model. Continue until no more to add, # of factor
threshold is satisfied. Remove factors at the end that were not good enough
5. Forward selec- variable selection; classical
tion Start with model with no factors, at each step find best new factor to add. Continue
until none bad enough to remove, # of factor threshold is satisfied. Remove factors
at the end that were not good enough
6. Stepwise regres- variable selection; classical
sion Combination of forward selection and backwards elimination. Start with all or no
factors. Each step remove/add a factor. As it continues, after adding in new factor
we eliminate right away any factors that may be good. Helps model adjust when
new factors are added, goodness values change
7. Ways of deter- p-value, Rsquared, AIC, BIC
mining if factors
are good enough
in variable selec-
tion
, ISYE 6501 Final
Study online at https://quizlet.com/_7l1l0g
8. Greedy algorithm At each step, it does the one thing that looks best
without taking future options into consideration. Good for initial analysis
1. Forward selection
2. Backwards elimination
3. Stepwise regression
9. Global variable 1. LASSO
selection ap- 2. Elastic Net
proaches
Slower, but tend to give better predictive models
10. LASSO variable selection; global
- SCALE the date (as with any constrained sum of coefficients)
- add a constraint to the standard regression equation
- minimize sum of squared errors
- T = limit or "budget" on how large the sum of squared errors can get. Budget
will be used on most important coefficients
- Method for limiting the number of variables in a model by limiting the sum of
all coefficients' absolute values. Can be very helpful when number of data points
is less than number of factors.
11. Elastic Net variable selection; global
- SCALE the date (as with any constrained sum of coefficients)
- T = limit or "budget" on how large the sum of squared errors can get. Budget
will be used on most important coefficients
- Combination of lasso and ridge regression.
- Variable selection benefits of LASSO
- Predictive benefits of ridge regression
12. Ridge Regression
, ISYE 6501 Final
Study online at https://quizlet.com/_7l1l0g
- Method of regularization by limiting the sum of the squares of the coefficients.
Will reduce the magnitude of coefficients, not the number of variables chosen.
- The quadratic term in ridge regression
tends to shrink the coefficient values i.e Whatever the basic regression model
coefficients would be,
the quadratic constraint pushes them toward zero
or regularizes them.
13. Design of Experi- How can we still have a representative sample of each combination of factors, while
ments (DOE) only surveying 600 people?
How to determine which of the several factors are most
important to predicting someone's answers?
comparison to measure difference
control for other factors and effects
blocking factors that account for the variation between factors (red sports car vs
red minivan example)
14. A/B testing Whenever we want to choose between 2 alternatives.
As long as the following 3 things are true:
1st, we need to be able to collect a lot of data quickly enough to get an answer in
time to use it.
2nd, the data we collect has to be from a representative sample of the whole
3rd, the amount of data we collect has to be small compared to the total population
we want to use the answer on.
Before modeling and before collecting data
15. (Full) Factorial Test every combination of variables in an experiment to find each one's effect, and
Design interaction effects on the outcome.
16. Fractional Factor- A subset of combinations to test - selected combinations give same result as full
ial Design factorial design i.e a balanced design
Before modeling and before collecting data
