# logregwithKFold

course: Module 2 — Machine Learning Algorithms
module: Module-2-Machine-Learning-Algorithms
type: notebook
source_url: https://personal-learn.armco.dev/files/Module-2-Machine-Learning-Algorithms/General/Lab_Materials/logregwithKFold.ipynb

---
[cell 1 markdown]
# **Logistic Regression with KFold Cross-Validation fotr Binary Classification**

[cell 2 markdown]
# E-Commerce Shipping Dataset Overview

The **E-Commerce Shipping** dataset provides shipping and delivery data for products ordered through an e-commerce platform. The goal is to **predict whether a product shipment is delivered on time**, using information related to the shipment process, customer interaction, and product details.

## Source:

Available on Kaggle as a public dataset for classification modeling and EDA:

* [Kaggle Dataset Page](https://www.kaggle.com/datasets/prachi13/customer-analytics)

---

# Target Variable:

* **Reached.on.Time\_Y.N** *(binary)*: Whether the shipment was delivered on time (`0` = Yes, `1` = No)

---

### Features:

| Column Name            | Description                                            | Data Type         |
| ---------------------- | ------------------------------------------------------ | ----------------- |
| ID                     | Unique identifier for each product shipment            | *String*          |
| Warehouse\_block       | Zone of the warehouse handling the shipment            | *Categorical*     |
| Mode\_of\_Shipment     | Shipping method used (`Ship`, `Flight`, `Road`)        | *Categorical*     |
| Customer\_care\_calls  | Number of customer care calls made about the shipment  | *Numeric (int)*   |
| Customer\_rating       | Customer's rating for the service (1 to 5)             | *Numeric (int)*   |
| Cost\_of\_the\_Product | Cost of the product ordered                            | *Numeric (int)*   |
| Prior\_purchases       | Number of prior purchases made by the customer         | *Numeric (int)*   |
| Product\_importance    | Perceived product importance (`low`, `medium`, `high`) | *Categorical*     |
| Gender                 | Gender of the customer (`M`, `F`)                      | *Categorical*     |
| Discount\_offered      | Discount applied to the product                        | *Numeric (int)*   |
| Weight\_in\_gms        | Weight of the product in grams                         | *Numeric (int)*   |
| Reached.on.Time\_Y.N   | Delivery status (`0` = On time, `1` = Delayed)         | **Binary Target** |

[cell 3 code]
import pandas as pd


df=pd.read_csv('ecom_shipping_v1.csv')
df

[cell 4 code]
df.info()

[cell 5 markdown]
# **Checking for missing values**

[cell 6 code]
df.isna().sum()

[cell 7 code]
(df == '').sum()

[cell 8 code]
df.apply(lambda x: x.astype(str).str.lower().isin(['none', 'null', 'na','?'])).sum()

[cell 9 markdown]
# **Displaying unique values of categorical columns**

[cell 10 code]
categorical_cols = df.select_dtypes(include='object').columns.tolist()

for col in categorical_cols:
    unique_vals = df[col].unique()
    print(f"{col} ({len(unique_vals)} unique): {unique_vals}")
    print('------------------------------------------------')

[cell 11 markdown]
# **Count plot for target variable**

[cell 12 code]
print(df['Reached.on.Time_Y.N'].value_counts(normalize=True))
df['Reached.on.Time_Y.N'].value_counts().plot(kind='bar')

[cell 13 markdown]
# **Optimal Number of KFolds (value of K) using Using Logistic Regression and Categorical features: Integer Encoded**

[cell 14 markdown]
We use **StratifiedKFold** to ensure that each fold has approximately the same proportion of class labels as the full dataset, and we evaluate fold counts in the range **3 to 7** to identify the optimal choice.

[cell 15 code]
from sklearn.model_selection import StratifiedKFold
from sklearn.preprocessing import StandardScaler, OrdinalEncoder, OneHotEncoder
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import accuracy_score, precision_score, recall_score, f1_score, roc_auc_score
import matplotlib.pyplot as plt
import numpy as np

# Separate features and target
X = df.drop(['ID', 'Reached.on.Time_Y.N'], axis=1)  # Features (excluding ID and target)
y = df['Reached.on.Time_Y.N']                       # Target column


# Identify column types
categorical_cols = X.select_dtypes(include=['object']).columns.tolist()
numerical_cols = X.select_dtypes(include=['int64', 'float64']).columns.tolist()

fold_values = [3,4, 5,6,7 ]  # Varying K-Fold splits


# Initialize metrics dictionary
metrics_kfold = {
    'folds': [],
    'Train Accuracy': [], 'Test Accuracy': [],
    'Train Precision': [], 'Test Precision': [],
    'Train Recall': [], 'Test Recall': [],
    'Train F1': [], 'Test F1': [],
    'Train AUC': [], 'Test AUC': []
}

# Loop over different fold counts
for n_folds in fold_values:
    train_acc, test_acc = [], []
    train_prec, test_prec = [], []
    train_rec, test_rec = [], []
    train_f1, test_f1 = [], []
    train_auc, test_auc = [], []

    kf = StratifiedKFold(n_splits=n_folds, shuffle=True, random_state=42)

    for train_idx, test_idx in kf.split(X, y):
        X_train_raw, X_test_raw = X.iloc[train_idx], X.iloc[test_idx]
        y_train, y_test = y.iloc[train_idx], y.iloc[test_idx]

        # Encode + Scale
        encoder = OrdinalEncoder(handle_unknown='use_encoded_value', unknown_value=-1)
        scaler = StandardScaler()

        X_train = X_train_raw.copy()
        X_test = X_test_raw.copy()

        X_train[categorical_cols] = encoder.fit_transform(X_train_raw[categorical_cols])
        X_test[categorical_cols] = encoder.transform(X_test_raw[categorical_cols])

        X_train[numerical_cols] = scaler.fit_transform(X_train[numerical_cols])
        X_test[numerical_cols] = scaler.transform(X_test[numerical_cols])

        # Train LR
        model= LogisticRegression()
        model.fit(X_train, y_train)

        # Predictions
        y_train_pred = model.predict(X_train)
        y_test_pred = model.predict(X_test)
        y_train_prob = model.predict_proba(X_train)[:, 1]
        y_test_prob = model.predict_proba(X_test)[:, 1]

        # Metrics
        train_acc.append(accuracy_score(y_train, y_train_pred))
        test_acc.append(accuracy_score(y_test, y_test_pred))

        train_prec.append(precision_score(y_train, y_train_pred, zero_division=0))
        test_prec.append(precision_score(y_test, y_test_pred, zero_division=0))

        train_rec.append(recall_score(y_train, y_train_pred, zero_division=0))
        test_rec.append(recall_score(y_test, y_test_pred, zero_division=0))

        train_f1.append(f1_score(y_train, y_train_pred, zero_division=0))
        test_f1.append(f1_score(y_test, y_test_pred, zero_division=0))

        train_auc.append(roc_auc_score(y_train, y_train_prob))
        test_auc.append(roc_auc_score(y_test, y_test_prob))

    # Store average metrics
    metrics_kfold['folds'].append(n_folds)
    metrics_kfold['Train Accuracy'].append(np.mean(train_acc))
    metrics_kfold['Test Accuracy'].append(np.mean(test_acc))
    metrics_kfold['Train Precision'].append(np.mean(train_prec))
    metrics_kfold['Test Precision'].append(np.mean(test_prec))
    metrics_kfold['Train Recall'].append(np.mean(train_rec))
    metrics_kfold['Test Recall'].append(np.mean(test_rec))
    metrics_kfold['Train F1'].append(np.mean(train_f1))
    metrics_kfold['Test F1'].append(np.mean(test_f1))
    metrics_kfold['Train AUC'].append(np.mean(train_auc))
    metrics_kfold['Test AUC'].append(np.mean(test_auc))


import pandas as pd

# Convert metrics to DataFrame
metrics_df = pd.DataFrame(metrics_kfold)

# Optional: round for readability
metrics_df = metrics_df.round(4)

# Print all metrics
metrics_df[['folds','Test Accuracy','Test Precision','Test Recall','Test F1','Test AUC']]

[cell 16 markdown]
### **Observations**
* Performance is almost identical across all folds, showing stable results with integer-encoded categorical features.
* Best test accuracy is observed at **5 folds**, but the improvement over other folds is marginal, showing that changing the number of folds does not significantly affect accuracy.

[cell 17 code]
# Plotting function
def plot_all_metrics_in_grid(metrics, x_values):
    metric_names = ["Accuracy", "Precision", "Recall", "F1", "AUC"]
    fig, axes = plt.subplots(2, 3, figsize=(14, 8))
    axes = axes.flatten()

    for i, name in enumerate(metric_names):
        ax = axes[i]
        ax.plot(x_values, metrics[f"Train {name}"], marker='o', label='Train')
        ax.plot(x_values, metrics[f"Test {name}"], marker='s', label='Test')
        ax.set_title(name)
        ax.set_xlabel("Number of Folds")
        ax.set_ylabel(name)
        ax.grid(True)
        ax.legend()

    if len(metric_names) < len(axes):
        axes[-1].axis("off")

    fig.suptitle("Train vs Test Metrics Across K-Fold Splits", fontsize=16)
    plt.tight_layout(rect=[0, 0, 1, 0.95])
    plt.show()

# Call the plotting function
plot_all_metrics_in_grid(metrics_kfold, metrics_kfold['folds'])

[cell 18 markdown]
### **Observations**
We select 5-fold cross validation as it performs slightly better than the other fold settings, even though the difference is marginal.

[cell 19 markdown]
# **Optimal Number of KFolds (value of K) using Using Logistic Regression and Categorical features: One Hot Encoded**

[cell 20 markdown]
We use **StratifiedKFold** to ensure that each fold has approximately the same proportion of class labels as the full dataset, and we evaluate fold counts in the range **3 to 7** to identify the optimal choice.

[cell 21 code]
from sklearn.model_selection import StratifiedKFold
from sklearn.preprocessing import StandardScaler, OrdinalEncoder, OneHotEncoder
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import accuracy_score, precision_score, recall_score, f1_score, roc_auc_score
import matplotlib.pyplot as plt
import numpy as np

# Separate features and target
X = df.drop(['ID', 'Reached.on.Time_Y.N'], axis=1)  # Features (excluding ID and target)
y = df['Reached.on.Time_Y.N']                       # Target column


# Identify column types
categorical_cols = X.select_dtypes(include=['object']).columns.tolist()
numerical_cols = X.select_dtypes(include=['int64', 'float64']).columns.tolist()

fold_values = [3,4, 5,6,7 ]  # Varying K-Fold splits


# Initialize metrics dictionary
metrics_kfold = {
    'folds': [],
    'Train Accuracy': [], 'Test Accuracy': [],
    'Train Precision': [], 'Test Precision': [],
    'Train Recall': [], 'Test Recall': [],
    'Train F1': [], 'Test F1': [],
    'Train AUC': [], 'Test AUC': []
}

# Loop over different fold counts
for n_folds in fold_values:
    train_acc, test_acc = [], []
    train_prec, test_prec = [], []
    train_rec, test_rec = [], []
    train_f1, test_f1 = [], []
    train_auc, test_auc = [], []

    kf = StratifiedKFold(n_splits=n_folds, shuffle=True, random_state=42)

    for train_idx, test_idx in kf.split(X, y):
        X_train_raw, X_test_raw = X.iloc[train_idx], X.iloc[test_idx]
        y_train, y_test = y.iloc[train_idx], y.iloc[test_idx]

        scaler = StandardScaler() # drop='first'
        encoder = OneHotEncoder(handle_unknown='ignore',sparse_output=False)

        # One-hot encode categorical columns
        X_train_cat = encoder.fit_transform(X_train_raw[categorical_cols])
        X_test_cat = encoder.transform(X_test_raw[categorical_cols])

        # Scale numerical columns
        X_train_num = scaler.fit_transform(X_train_raw[numerical_cols])
        X_test_num = scaler.transform(X_test_raw[numerical_cols])

        # Combine numeric + categorical
        X_train = np.hstack([X_train_num, X_train_cat])
        X_test = np.hstack([X_test_num, X_test_cat])



        # Train LR
        model= LogisticRegression()
        model.fit(X_train, y_train)

        # Predictions
        y_train_pred = model.predict(X_train)
        y_test_pred = model.predict(X_test)
        y_train_prob = model.predict_proba(X_train)[:, 1]
        y_test_prob = model.predict_proba(X_test)[:, 1]

        # Metrics
        train_acc.append(accuracy_score(y_train, y_train_pred))
        test_acc.append(accuracy_score(y_test, y_test_pred))

        train_prec.append(precision_score(y_train, y_train_pred, zero_division=0))
        test_prec.append(precision_score(y_test, y_test_pred, zero_division=0))

        train_rec.append(recall_score(y_train, y_train_pred, zero_division=0))
        test_rec.append(recall_score(y_test, y_test_pred, zero_division=0))

        train_f1.append(f1_score(y_train, y_train_pred, zero_division=0))
        test_f1.append(f1_score(y_test, y_test_pred, zero_division=0))

        train_auc.append(roc_auc_score(y_train, y_train_prob))
        test_auc.append(roc_auc_score(y_test, y_test_prob))

    # Store average metrics
    metrics_kfold['folds'].append(n_folds)
    metrics_kfold['Train Accuracy'].append(np.mean(train_acc))
    metrics_kfold['Test Accuracy'].append(np.mean(test_acc))
    metrics_kfold['Train Precision'].append(np.mean(train_prec))
    metrics_kfold['Test Precision'].append(np.mean(test_prec))
    metrics_kfold['Train Recall'].append(np.mean(train_rec))
    metrics_kfold['Test Recall'].append(np.mean(test_rec))
    metrics_kfold['Train F1'].append(np.mean(train_f1))
    metrics_kfold['Test F1'].append(np.mean(test_f1))
    metrics_kfold['Train AUC'].append(np.mean(train_auc))
    metrics_kfold['Test AUC'].append(np.mean(test_auc))


import pandas as pd

# Convert metrics to DataFrame
metrics_df = pd.DataFrame(metrics_kfold)

# Optional: round for readability
metrics_df = metrics_df.round(4)

# Print all metrics
metrics_df[['folds','Test Accuracy','Test Precision','Test Recall','Test F1','Test AUC']]

[cell 22 markdown]
### **Observations**
* Performance remains consistent across all fold values when categorical features are one-hot encoded.

* Overall performance is lower than the integer-encoded case, despite similar stability across folds.

[cell 23 code]
# Plotting function
def plot_all_metrics_in_grid(metrics, x_values):
    metric_names = ["Accuracy", "Precision", "Recall", "F1", "AUC"]
    fig, axes = plt.subplots(2, 3, figsize=(14, 8))
    axes = axes.flatten()

    for i, name in enumerate(metric_names):
        ax = axes[i]
        ax.plot(x_values, metrics[f"Train {name}"], marker='o', label='Train')
        ax.plot(x_values, metrics[f"Test {name}"], marker='s', label='Test')
        ax.set_title(name)
        ax.set_xlabel("Number of Folds")
        ax.set_ylabel(name)
        ax.grid(True)
        ax.legend()

    if len(metric_names) < len(axes):
        axes[-1].axis("off")

    fig.suptitle("Train vs Test Metrics Across K-Fold Splits", fontsize=16)
    plt.tight_layout(rect=[0, 0, 1, 0.95])
    plt.show()

# Call the plotting function
plot_all_metrics_in_grid(metrics_kfold, metrics_kfold['folds'])

[cell 24 markdown]
### **Observations**
We select 4-fold cross validation as it performs slightly better than the other fold settings, even though the difference is marginal.

[cell 25 markdown]
# **Final Model comparsion with Decision Tree, Random Forest Classifier**
* K=5, Features are Integer Encoded

[cell 26 code]
from sklearn.model_selection import StratifiedKFold
from sklearn.preprocessing import StandardScaler, OrdinalEncoder
from sklearn.linear_model import LogisticRegression
from sklearn.tree import DecisionTreeClassifier
from sklearn.ensemble import RandomForestClassifier
from sklearn.metrics import accuracy_score, precision_score, recall_score, f1_score, roc_auc_score
import numpy as np
import pandas as pd

# Separate features and target
X = df.drop(['ID', 'Reached.on.Time_Y.N'], axis=1)
y = df['Reached.on.Time_Y.N']

# Identify column types
categorical_cols = X.select_dtypes(include=['object']).columns.tolist()
numerical_cols = X.select_dtypes(include=['int64', 'float64']).columns.tolist()

# 5-fold CV
kf = StratifiedKFold(n_splits=5, shuffle=True, random_state=42)

models = {
    'Logistic Regression': LogisticRegression(),
    'Decision Tree': DecisionTreeClassifier(random_state=42),
    'Random Forest': RandomForestClassifier(n_estimators=100, random_state=42)
}

# Metrics storage
results = []

for model_name, model in models.items():

    acc, prec, rec, f1, auc = [], [], [], [], []

    for train_idx, test_idx in kf.split(X, y):
        X_train_raw, X_test_raw = X.iloc[train_idx], X.iloc[test_idx]
        y_train, y_test = y.iloc[train_idx], y.iloc[test_idx]

        # Encode + Scale
        encoder = OrdinalEncoder(handle_unknown='use_encoded_value', unknown_value=-1)
        scaler = StandardScaler()

        X_train = X_train_raw.copy()
        X_test = X_test_raw.copy()

        X_train[categorical_cols] = encoder.fit_transform(X_train_raw[categorical_cols])
        X_test[categorical_cols] = encoder.transform(X_test_raw[categorical_cols])

        X_train[numerical_cols] = scaler.fit_transform(X_train[numerical_cols])
        X_test[numerical_cols] = scaler.transform(X_test[numerical_cols])

        # Train
        model.fit(X_train, y_train)

        # Predict
        y_pred = model.predict(X_test)

        y_prob = model.predict_proba(X_test)[:, 1]

        auc.append(roc_auc_score(y_test, y_prob))

        acc.append(accuracy_score(y_test, y_pred))
        prec.append(precision_score(y_test, y_pred, zero_division=0))
        rec.append(recall_score(y_test, y_pred, zero_division=0))
        f1.append(f1_score(y_test, y_pred, zero_division=0))

    results.append({
        'Model': model_name,
        'Test Accuracy': np.mean(acc),
        'Test Precision': np.mean(prec),
        'Test Recall': np.mean(rec),
        'Test F1': np.mean(f1),
        'Test AUC': np.mean(auc)
    })

# Results table
results_df = pd.DataFrame(results).round(4)
results_df

[cell 27 markdown]
### **Observation**
Random Forest peforms best among all when categorical features are integer encoded for k-Fold=5.

[cell 28 markdown]
# **Final Model comparsion with Decision Tree, Random Forest Classifier**
* K=4, Features are One Hot Encoded

[cell 29 code]
from sklearn.model_selection import StratifiedKFold
from sklearn.preprocessing import StandardScaler, OneHotEncoder
from sklearn.linear_model import LogisticRegression
from sklearn.tree import DecisionTreeClassifier
from sklearn.ensemble import RandomForestClassifier
from sklearn.metrics import accuracy_score, precision_score, recall_score, f1_score, roc_auc_score
import numpy as np
import pandas as pd

# Separate features and target
X = df.drop(['ID', 'Reached.on.Time_Y.N'], axis=1)
y = df['Reached.on.Time_Y.N']

# Identify column types
categorical_cols = X.select_dtypes(include=['object']).columns.tolist()
numerical_cols = X.select_dtypes(include=['int64', 'float64']).columns.tolist()

# 4-fold CV
kf = StratifiedKFold(n_splits=4, shuffle=True, random_state=42)

models = {
    'Logistic Regression': LogisticRegression(),
    'Decision Tree': DecisionTreeClassifier(random_state=42),
    'Random Forest': RandomForestClassifier(n_estimators=100, random_state=42)
}

results = []

for model_name, model in models.items():

    acc, prec, rec, f1, auc = [], [], [], [], []

    for train_idx, test_idx in kf.split(X, y):
        X_train_raw, X_test_raw = X.iloc[train_idx], X.iloc[test_idx]
        y_train, y_test = y.iloc[train_idx], y.iloc[test_idx]

        # One-hot encode categorical
        encoder = OneHotEncoder(handle_unknown='ignore', sparse_output=False)
        scaler = StandardScaler()

        X_train_cat = encoder.fit_transform(X_train_raw[categorical_cols])
        X_test_cat = encoder.transform(X_test_raw[categorical_cols])

        # Scale numeric
        X_train_num = scaler.fit_transform(X_train_raw[numerical_cols])
        X_test_num = scaler.transform(X_test_raw[numerical_cols])

        # Combine
        X_train = np.hstack([X_train_num, X_train_cat])
        X_test = np.hstack([X_test_num, X_test_cat])

        # Train
        model.fit(X_train, y_train)

        # Predict
        y_pred = model.predict(X_test)
        y_prob = model.predict_proba(X_test)[:, 1]

        acc.append(accuracy_score(y_test, y_pred))
        prec.append(precision_score(y_test, y_pred, zero_division=0))
        rec.append(recall_score(y_test, y_pred, zero_division=0))
        f1.append(f1_score(y_test, y_pred, zero_division=0))
        auc.append(roc_auc_score(y_test, y_prob))

    results.append({
        'Model': model_name,
        'Test Accuracy': np.mean(acc),
        'Test Precision': np.mean(prec),
        'Test Recall': np.mean(rec),
        'Test F1': np.mean(f1),
        'Test AUC': np.mean(auc)
    })

# Results table
results_df = pd.DataFrame(results).round(4)
results_df

[cell 30 markdown]
### **Observation**
Random Forest peforms best among all when categorical features are one hot encoded for k-Fold=4.

[cell 31 markdown]
# **Logistic Regression with KFolds for Multi-Class Classification**

[cell 32 markdown]
# Car Evaluation Dataset Overview

The **Car Evaluation Dataset** comes from the automotive industry and is used to classify cars into four categories based on features like buying price, maintenance cost, seating capacity, luggage space, and safety.
The target classes are:

* `unacc`: Unacceptable
* `acc`: Acceptable
* `good`: Good
* `vgood`: Very Good


## Source:

Available via the UCI Machine Learning Repository:

* [UCI Dataset Page](https://archive.ics.uci.edu/ml/datasets/Car+Evaluation)



## Features:

| Column Name | Description                         | Data Type     | Values / Categories             |
| ----------- | ----------------------------------- | ------------- | ------------------------------- |
| buying      | Buying price level                  | *Categorical* | `vhigh`, `high`, `med`, `low`   |
| maint       | Maintenance cost level              | *Categorical* | `vhigh`, `high`, `med`, `low`   |
| doors       | Number of doors                     | *Categorical* | `2`, `3`, `4`, `5more`          |
| persons     | Passenger capacity                  | *Categorical* | `2`, `4`, `more`                |
| lug\_boot   | Luggage boot size                   | *Categorical* | `small`, `med`, `big`           |
| safety      | Safety rating                       | *Categorical* | `low`, `med`, `high`            |
| class       | **Target**: Car acceptability class | *Categorical* | `unacc`, `acc`, `good`, `vgood` |

[cell 33 markdown]
# **Loading the dataset**

[cell 34 code]
import pandas as pd

# Define column names based on dataset description
columns = ['buying', 'maint', 'doors', 'persons', 'lug_boot', 'safety', 'class']


df = pd.read_csv('car_v1.data', header=None, names=columns)


df

[cell 35 code]
df.info()

[cell 36 markdown]
# **Checking for missing values**

[cell 37 code]
df.isna().sum()

[cell 38 code]
(df == '').sum()

[cell 39 code]
df.apply(lambda x: x.astype(str).str.lower().isin(['none', 'null', 'na','?'])).sum()

[cell 40 markdown]
# **Displaying unique values of categorical columns**

[cell 41 code]
for col in df.columns:
    unique_vals = df[col].unique()
    print(f"{col} ({len(unique_vals)} unique): {unique_vals}")
    print('------------------------------------------------')

[cell 42 markdown]
# **Label Count for target variable**

[cell 43 code]
print(df['class'].value_counts(normalize=True))
df['class'].value_counts().plot(kind='bar')

[cell 44 markdown]
# **Optimal Number of KFolds (value of K) using Using Logistic Regression and Categorical features: Integer Encoded**

[cell 45 markdown]
We use **StratifiedKFold** to ensure that each fold has approximately the same proportion of class labels as the full dataset, and we evaluate fold counts in the range **3 to 10** to identify the optimal choice.

[cell 46 code]
from sklearn.model_selection import StratifiedKFold
from sklearn.preprocessing import StandardScaler, OrdinalEncoder, OneHotEncoder
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import accuracy_score, precision_score, recall_score, f1_score, roc_auc_score
import matplotlib.pyplot as plt
import numpy as np

# Separate features and target
# Drop 'ID' column and isolate features and target
X = df.drop(['class'], axis=1)  # Features (excluding ID and target)
y = df['class']                       # Target column
                     # Target column


# Identify column types
categorical_cols = X.select_dtypes(include=['object']).columns.tolist()


fold_values = [3,4, 5,6,7,8,9,10 ]  # Varying K-Fold splits


# Initialize metrics dictionary
metrics_kfold = {
    'folds': [],
    'Train Accuracy': [], 'Test Accuracy': [],
    'Train Precision': [], 'Test Precision': [],
    'Train Recall': [], 'Test Recall': [],
    'Train F1': [], 'Test F1': [],
    'Train AUC': [], 'Test AUC': []
}

# Loop over different fold counts
for n_folds in fold_values:
    train_acc, test_acc = [], []
    train_prec, test_prec = [], []
    train_rec, test_rec = [], []
    train_f1, test_f1 = [], []
    train_auc, test_auc = [], []

    kf = StratifiedKFold(n_splits=n_folds, shuffle=True, random_state=42)

    for train_idx, test_idx in kf.split(X, y):
        X_train_raw, X_test_raw = X.iloc[train_idx], X.iloc[test_idx]
        y_train, y_test = y.iloc[train_idx], y.iloc[test_idx]

        # Encode + Scale
        encoder = OrdinalEncoder(handle_unknown='use_encoded_value', unknown_value=-1)
        #scaler = StandardScaler()

        X_train = X_train_raw.copy()
        X_test = X_test_raw.copy()

        X_train[categorical_cols] = encoder.fit_transform(X_train_raw[categorical_cols])
        X_test[categorical_cols] = encoder.transform(X_test_raw[categorical_cols])

        #X_train[numerical_cols] = scaler.fit_transform(X_train[numerical_cols])
        #X_test[numerical_cols] = scaler.transform(X_test[numerical_cols])

        # Train LR
        model= LogisticRegression()
        model.fit(X_train, y_train)

        # Predictions
        y_train_pred = model.predict(X_train)
        y_test_pred = model.predict(X_test)
        y_train_prob = model.predict_proba(X_train)
        y_test_prob = model.predict_proba(X_test)

        # Metrics
        train_acc.append(accuracy_score(y_train, y_train_pred))
        test_acc.append(accuracy_score(y_test, y_test_pred))

        train_prec.append(precision_score(y_train, y_train_pred, zero_division=0,average='macro'))
        test_prec.append(precision_score(y_test, y_test_pred, zero_division=0,average='macro'))

        train_rec.append(recall_score(y_train, y_train_pred, zero_division=0,average='macro'))
        test_rec.append(recall_score(y_test, y_test_pred, zero_division=0,average='macro'))

        train_f1.append(f1_score(y_train, y_train_pred, zero_division=0,average='macro'))
        test_f1.append(f1_score(y_test, y_test_pred, zero_division=0,average='macro'))

        train_auc.append(roc_auc_score(y_train, y_train_prob,multi_class='ovr'))
        test_auc.append(roc_auc_score(y_test, y_test_prob,multi_class='ovr'))

    # Store average metrics
    metrics_kfold['folds'].append(n_folds)
    metrics_kfold['Train Accuracy'].append(np.mean(train_acc))
    metrics_kfold['Test Accuracy'].append(np.mean(test_acc))
    metrics_kfold['Train Precision'].append(np.mean(train_prec))
    metrics_kfold['Test Precision'].append(np.mean(test_prec))
    metrics_kfold['Train Recall'].append(np.mean(train_rec))
    metrics_kfold['Test Recall'].append(np.mean(test_rec))
    metrics_kfold['Train F1'].append(np.mean(train_f1))
    metrics_kfold['Test F1'].append(np.mean(test_f1))
    metrics_kfold['Train AUC'].append(np.mean(train_auc))
    metrics_kfold['Test AUC'].append(np.mean(test_auc))


import pandas as pd

# Convert metrics to DataFrame
metrics_df = pd.DataFrame(metrics_kfold)

# Optional: round for readability
metrics_df = metrics_df.round(4)

# Print all metrics
metrics_df[['folds','Test Accuracy','Test Precision','Test Recall','Test F1','Test AUC']]

[cell 47 markdown]
### **Observations**

1. **Integer encoding gives weak performance** for this fully categorical car dataset, as reflected by low precision, recall, and F1 scores despite moderate accuracy.

2. **Accuracy remains capped around 69 percent**, while nearly **70 percent of samples belong to a single class**, indicating that the model is unable to learn meaningful patterns beyond the class distribution.

3. **Changing the number of folds has negligible impact**, showing stable but consistently weak performance across different cross validation settings.

[cell 48 markdown]
# **Optimal Number of KFolds (value of K) using Using Logistic Regression and Categorical features: One Hot Encoded**

[cell 49 markdown]
We use **StratifiedKFold** to ensure that each fold has approximately the same proportion of class labels as the full dataset, and we evaluate fold counts in the range **3 to 10** to identify the optimal choice.

[cell 50 code]
from sklearn.model_selection import StratifiedKFold
from sklearn.preprocessing import StandardScaler, OrdinalEncoder, OneHotEncoder
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import accuracy_score, precision_score, recall_score, f1_score, roc_auc_score
import matplotlib.pyplot as plt
import numpy as np

# Separate features and target
# Drop 'ID' column and isolate features and target
X = df.drop(['class'], axis=1)  # Features (excluding ID and target)
y = df['class']                       # Target column
                     # Target column


# Identify column types
categorical_cols = X.select_dtypes(include=['object']).columns.tolist()


fold_values = [3,4, 5,6,7,8,9,10 ]  # Varying K-Fold splits


# Initialize metrics dictionary
metrics_kfold = {
    'folds': [],
    'Train Accuracy': [], 'Test Accuracy': [],
    'Train Precision': [], 'Test Precision': [],
    'Train Recall': [], 'Test Recall': [],
    'Train F1': [], 'Test F1': [],
    'Train AUC': [], 'Test AUC': []
}

# Loop over different fold counts
for n_folds in fold_values:
    train_acc, test_acc = [], []
    train_prec, test_prec = [], []
    train_rec, test_rec = [], []
    train_f1, test_f1 = [], []
    train_auc, test_auc = [], []

    kf = StratifiedKFold(n_splits=n_folds, shuffle=True, random_state=42)

    for train_idx, test_idx in kf.split(X, y):
        X_train_raw, X_test_raw = X.iloc[train_idx], X.iloc[test_idx]
        y_train, y_test = y.iloc[train_idx], y.iloc[test_idx]


        encoder = OneHotEncoder(handle_unknown='ignore', sparse_output=False)

        X_train_enc = encoder.fit_transform(X_train_raw[categorical_cols])
        X_test_enc  = encoder.transform(X_test_raw[categorical_cols])

        X_train = pd.DataFrame(
            X_train_enc,
            columns=encoder.get_feature_names_out(categorical_cols),
            index=X_train_raw.index
        )

        X_test = pd.DataFrame(
            X_test_enc,
            columns=encoder.get_feature_names_out(categorical_cols),
            index=X_test_raw.index
        )



        # Train LR
        model= LogisticRegression()
        model.fit(X_train, y_train)

        # Predictions
        y_train_pred = model.predict(X_train)
        y_test_pred = model.predict(X_test)
        y_train_prob = model.predict_proba(X_train)
        y_test_prob = model.predict_proba(X_test)

        # Metrics
        train_acc.append(accuracy_score(y_train, y_train_pred))
        test_acc.append(accuracy_score(y_test, y_test_pred))

        train_prec.append(precision_score(y_train, y_train_pred, zero_division=0,average='macro'))
        test_prec.append(precision_score(y_test, y_test_pred, zero_division=0,average='macro'))

        train_rec.append(recall_score(y_train, y_train_pred, zero_division=0,average='macro'))
        test_rec.append(recall_score(y_test, y_test_pred, zero_division=0,average='macro'))

        train_f1.append(f1_score(y_train, y_train_pred, zero_division=0,average='macro'))
        test_f1.append(f1_score(y_test, y_test_pred, zero_division=0,average='macro'))

        train_auc.append(roc_auc_score(y_train, y_train_prob,multi_class='ovr'))
        test_auc.append(roc_auc_score(y_test, y_test_prob,multi_class='ovr'))

    # Store average metrics
    metrics_kfold['folds'].append(n_folds)
    metrics_kfold['Train Accuracy'].append(np.mean(train_acc))
    metrics_kfold['Test Accuracy'].append(np.mean(test_acc))
    metrics_kfold['Train Precision'].append(np.mean(train_prec))
    metrics_kfold['Test Precision'].append(np.mean(test_prec))
    metrics_kfold['Train Recall'].append(np.mean(train_rec))
    metrics_kfold['Test Recall'].append(np.mean(test_rec))
    metrics_kfold['Train F1'].append(np.mean(train_f1))
    metrics_kfold['Test F1'].append(np.mean(test_f1))
    metrics_kfold['Train AUC'].append(np.mean(train_auc))
    metrics_kfold['Test AUC'].append(np.mean(test_auc))


import pandas as pd

# Convert metrics to DataFrame
metrics_df = pd.DataFrame(metrics_kfold)

# Optional: round for readability
metrics_df = metrics_df.round(4)

# Print all metrics
metrics_df[['folds','Test Accuracy','Test Precision','Test Recall','Test F1','Test AUC']]

[cell 51 markdown]
Both 3-fold and 4-fold cross validation are comparable in performance, with 3-fold giving slightly higher accuracy while 4-fold shows marginally better balance across precision, so either can be reasonably selected.

[cell 52 markdown]
### **Observations**
1. Performance is consistently high across all fold values, indicating strong and stable learning behavior under cross validation.

2. Differences across folds are minimal, showing that the model’s performance is not sensitive to the choice of fold count.
3. **One hot encoding is clearly beneficial**, as it enables the model to learn meaningful patterns from the categorical features and achieve strong overall performance.