cas-pml/SL/aufgaben/template/2_Code/4.5 Validierung und mehr - Unbalancierte Daten.ipynb

Feature Engineering

Klassifikation

Regression

Validierung und mehr

Sampling und Resampling

Validierungstechniken

Grid Search und Random Search

Performancemetriken

Unbalancierte Daten

Motivation und Vorbereitung

Cell:

[Cell type raw - unsupported, skipped]

In [1]:

## import libraries
import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
import seaborn as sns; sns.set()
%matplotlib inline

In [2]:

## read and prepare data
datapath = '../3_data'
from os import chdir; chdir(datapath)
data = pd.read_csv('bank-additional-full.csv', sep=';')
print('dim =', data.shape)
print(data.y.value_counts(normalize=True)) ## proportion

X_full = data.drop('y', axis=1)
y_full = data['y']             

dim = (41188, 21)
y
no     0.887346
yes    0.112654
Name: proportion, dtype: float64

In [3]:

## minimal feature engineering: one hot encoding for not numerical features
X_full = pd.get_dummies(X_full, drop_first=True)

In [4]:

## test - train - split
from sklearn.model_selection import train_test_split
X_full_train, X_full_test, y_full_train, y_full_test, = train_test_split(
    X_full,
    y_full,
    train_size=2/3,
    random_state=1234)

In [5]:

## function for evaluate different sampling methods
##   train a RandomForestClassifier model with train data
##   return
##     internal scorer (accuracy) for test data
##     proportion of classes after resampling

from sklearn.ensemble import RandomForestClassifier
def getResampledRfScore(X_train, y_train, X_test, y_test):
    model = RandomForestClassifier(random_state=1234)
    model.fit(X_train, y_train)
    print('score ', model.score(X_test, y_test))
    print(y_train.value_counts(normalize=True)) 

In [6]:

## test call (without resampling)
getResampledRfScore(X_full_train, y_full_train, X_full_test, y_full_test)

score  0.912163146394756
y
no     0.886773
yes    0.113227
Name: proportion, dtype: float64

Random under-sampling

In [7]:

!pip install imblearn

Requirement already satisfied: imblearn in /Users/vgv1/.pyenv/versions/teaching/lib/python3.13/site-packages (0.0)
Requirement already satisfied: imbalanced-learn in /Users/vgv1/.pyenv/versions/teaching/lib/python3.13/site-packages (from imblearn) (0.13.0)
Requirement already satisfied: numpy<3,>=1.24.3 in /Users/vgv1/.pyenv/versions/teaching/lib/python3.13/site-packages (from imbalanced-learn->imblearn) (2.2.6)
Requirement already satisfied: scipy<2,>=1.10.1 in /Users/vgv1/.pyenv/versions/teaching/lib/python3.13/site-packages (from imbalanced-learn->imblearn) (1.15.3)
Requirement already satisfied: scikit-learn<2,>=1.3.2 in /Users/vgv1/.pyenv/versions/teaching/lib/python3.13/site-packages (from imbalanced-learn->imblearn) (1.6.1)
Requirement already satisfied: sklearn-compat<1,>=0.1 in /Users/vgv1/.pyenv/versions/teaching/lib/python3.13/site-packages (from imbalanced-learn->imblearn) (0.1.3)
Requirement already satisfied: joblib<2,>=1.1.1 in /Users/vgv1/.pyenv/versions/teaching/lib/python3.13/site-packages (from imbalanced-learn->imblearn) (1.5.1)
Requirement already satisfied: threadpoolctl<4,>=2.0.0 in /Users/vgv1/.pyenv/versions/teaching/lib/python3.13/site-packages (from imbalanced-learn->imblearn) (3.6.0)

In [8]:

from imblearn.under_sampling import RandomUnderSampler
rus = RandomUnderSampler(random_state=1234)
X_resampled_train, y_resampled_train =\
    rus.fit_resample(X_full_train, y_full_train)
getResampledRfScore(
    X_resampled_train, y_resampled_train, X_full_test, y_full_test)

score  0.847632920611799
y
no     0.5
yes    0.5
Name: proportion, dtype: float64

Random over-sampling

In [9]:

from imblearn.over_sampling import\
    RandomOverSampler
ros = RandomOverSampler(random_state=1234)
X_resampled_train, y_resampled_train =\
    ros.fit_resample(X_full_train, y_full_train)
getResampledRfScore(
    X_resampled_train, y_resampled_train, X_full_test, y_full_test)

score  0.9041514930808449
y
no     0.5
yes    0.5
Name: proportion, dtype: float64

Undersampling mit Tomek Links

In [10]:

from imblearn.under_sampling import TomekLinks
tl = TomekLinks()
X_resampled_train, y_resampled_train = tl.fit_resample(
    X_full_train, y_full_train)
getResampledRfScore(
    X_resampled_train, y_resampled_train, X_full_test, y_full_test)

score  0.9115076474872542
y
no     0.883063
yes    0.116937
Name: proportion, dtype: float64

Oversampling mit SMOTE

In [11]:

from imblearn.over_sampling import SMOTE
sm = SMOTE()
X_resampled_train, y_resampled_train = sm.fit_resample(
    X_full_train, y_full_train)
getResampledRfScore(
    X_resampled_train, y_resampled_train, X_full_test, y_full_test)

score  0.9038601602330663
y
no     0.5
yes    0.5
Name: proportion, dtype: float64

Weights beim Trainieren

ref: https://scikit-learn.org/stable/modules/generated/sklearn.ensemble.RandomForestClassifier.html

• the formula for class_weights:

    n_samples / (n_classes * np.bincount(y))
  

• the weights of y are calculated inversely proportional to the frequencies of the present classes

In [12]:

## with weights: balanced
model = RandomForestClassifier(n_estimators=100, class_weight='balanced', random_state=1234)
model.fit(X_full_train, y_full_train)
print(model.score(X_full_test, y_full_test))

0.9104151493080845

In [13]:

## with weights: balanced: mannualy set
n_no = y_full_train.value_counts()['no']
n_yes = y_full_train.value_counts()['yes']
weight_no = len(y_full_train) / (2 * n_no)
weight_yes = len(y_full_train) / (2 * n_yes)
print(weight_no, weight_yes)

model = RandomForestClassifier(
    n_estimators=100,
    class_weight={'no': weight_no,
                  'yes': weight_yes}, 
    random_state=1234)

model.fit(X_full_train, y_full_train)
print(model.score(X_full_test, y_full_test))

0.5638424575957945 4.415889353489868
0.9104151493080845