Permutation importance на собеседовании Data Scientist

Зачем разбирать на собесе

Permutation importance — robust способ measure feature importance. На собесе DS: «отличие от feature_importances_», «когда permutation».

Алгоритм

1. Train model on training data.
2. Measure baseline metric (e.g., AUC) на validation set.
3. For каждой feature f:
   a. Permute f (shuffle values randomly).
   b. Predict, measure metric.
   c. Importance(f) = baseline - permuted metric.
   d. Restore feature.
4. Repeat several times, average.

from sklearn.inspection import permutation_importance
result = permutation_importance(model, X_val, y_val, n_repeats=10)
importance = result.importances_mean

Отличие от feature_importances_

model.feature_importances_ в RF / GB — based на split frequency / impurity reduction. Doesn't measure actual prediction impact.

Permutation importance — directly measures «how much model loses когда feature destroyed».

Bias feature_importances_:

• High-cardinality features («ID column») show fake importance.
• Doesn't account for feature interactions accurately.
• Computed на training data.

Permutation:

• Model-agnostic — works for any model.
• На validation data — reflects generalization.
• Captures real prediction impact.

На train или test?

Train. Tells which features model uses. Может show high importance overfitted features.

Validation / Test. Tells which features actually generalize. Production-relevant.

Production interest — обычно validation.

Ошибки

Correlated features. Permutation одной — другая takes её place. Joint importance hidden. Решение — group permutation.

Multiple correlated. «Если оба важные, permutation одной → low importance каждая». Incorrect impression.

Slow на больших data. N features × N repeats × inference. Sample dataset для speed.

Не учитывают interactions. Permutation measures marginal feature effect.

Связанные темы

• SHAP и interpretability на собесе DS
• Feature engineering для DS
• Decision trees для DS
• Bagging vs Boosting для DS
• Подготовка к собесу Data Scientist

FAQ

Это официальная информация?

Нет. Статья основана на работах Breiman 2001, документации scikit-learn.

Тренируйте Data Science — откройте тренажёр с 1500+ вопросами для собесов.