Discussion

Announcements

• Application exam available (due Friday, May 9th at noon)
• Readings due by next Tuesday
• Will discuss readings with me in “lab” next Tuesday
• Quiz still due on Wednesday
• Review session in dicussion next Thursday
• Concepts exam on Tuesday May 6th (typical lab meeting) in this room

Sets and iteration

• train/val(dev)/test vs kfold/bootstrap with test vs nested cv

• choosing sample sizes for train, val, test

• Random vs non-random splits into train/val/test. what needs to be true? what is less important?

• overfit val? Use test more than once? Do vs. don’t and risks

• Review 12 Takeaways

Error analysis

• why

• start with baseline/basic system first- why?

• eyeball and black box dev sets

• size of eyeball sample?

• risk of using all dev as eyeball sample

• error analysis for regression?

• review 19 takeaways

Bias/Variance

We do not use training error as an estimate of model performance in new data. Why?

We can compare training and val error (e.g., rmse) to coarsely partition error into separate sources (bias vs. variance)

• 1/11
• 15/16
• 15/30
• 1/2

Also discuss

• role of optimal error rate (and human error rate?)
• avoidable bias and variance using train/val/ and optimal error
• how to fix avoidable bias (25)
• how to fix variance (27)

Learning Curves

• How and why
• how to calculate with small samples? issues with large samples?
• review figs on next slides

example 1

example 2

example 3

Pipeline vs End-to-End

• when have we seen each?
• costs and benefits vs. possible?