IAML Unit 11: Discussion

Announcements

• Permutation test quiz question - full credit for all!
• Neural net winner

Bayesian Model Comparison

• What is the difference between Bayesian and Frequentist approaches?
  • NHST does not provide the probabilities of the null and alternative hypotheses.
    • That is what we want
    • NHST gives us the probability of our data given the null
  • NHST focuses on a point-wise comparison (no difference) that is almost never true.
  • NHST yields no information about the null hypothesis (i.e., when we fail to reject)
  • The inference depends on the sampling and testing intention (think about Bonferonni correction)
• Role of feature ablation
• How to interpret results
  • Credible intervals
  • Posterior probabilities for model comparisons
  • Use of ROPE

PDP

• How calculated and interpreted
• Problems?

ALE

• How calculated and interpreted
• When to use?

Feature Importance Approaches

Model specific vs. agnostic

Permutation Feature Importance

• How does it work
• What does it give you

SHAP

• How does it work
• Local vs. global importance
• Individual vs. categories of features
• Useful plots
  • Global importance
  • Sina plots
  • SHAP dependence plots
  • Individual predictions

Group discussion

Get into small groups of 2-3. Think about one of the data sets we have worked with in class or your own! Create a research question you might ask using each of the following explanatory methods.

1. Bayesian model comparison
2. ALE or PDP plots
3. SHAP