{ "cells": [ { "cell_type": "markdown", "id": "6c8c9bfe", "metadata": {}, "source": [ "# Practical 8 — AdaBoost and gradient boosting\n", "\n", "**Objectives**: implement a few AdaBoost rounds, explain how observation weights\n", "change the next tree, and compare boosted classifiers with scikit-learn and XGBoost.\n" ] }, { "cell_type": "code", "execution_count": null, "id": "feeada03", "metadata": {}, "outputs": [], "source": [ "import matplotlib.pyplot as plt\n", "import numpy as np\n", "import sklearn\n", "import xgboost\n", "from sklearn.datasets import load_breast_cancer\n", "from sklearn.dummy import DummyClassifier\n", "from sklearn.ensemble import AdaBoostClassifier, GradientBoostingClassifier\n", "from sklearn.metrics import (accuracy_score, ConfusionMatrixDisplay, f1_score,\n", " log_loss, precision_score, recall_score, roc_auc_score)\n", "from sklearn.model_selection import GridSearchCV, StratifiedKFold, train_test_split\n", "from sklearn.tree import DecisionTreeClassifier\n", "from xgboost import XGBClassifier\n", "\n", "SEED = 42\n", "BLUE, RED = \"#2474b5\", \"#e30613\"\n", "plt.rcParams.update({\"figure.figsize\": (7, 4), \"font.size\": 11})\n", "print(\"scikit-learn:\", sklearn.__version__, \"XGBoost:\", xgboost.__version__)\n" ] }, { "cell_type": "markdown", "id": "2720089d", "metadata": {}, "source": [ "## 1. A small AdaBoost example by hand — 20 min\n", "\n", "Use the eight observations from the lecture, with labels $-1/+1$. All initial\n", "observation weights are $q_i=1/8$. A depth-1 tree (a **stump**) is supplied by sklearn;\n", "your task is to implement the **AdaBoost loop**, without `AdaBoostClassifier` yet.\n", "\n", "At each round, fit a **new** stump with `sample_weight=q`. Then calculate:\n", "\n", "$$\\varepsilon_m=\\sum_iq_i\\mathbf1_{h_m(x_i)\\ne y_i},\\qquad\n", " a_m=\\frac12\\log\\frac{1-\\varepsilon_m}{\\varepsilon_m},$$\n", "$$\\widetilde q_i=q_i\\exp(-a_m y_i h_m(x_i)),\\qquad\n", " q_i^{new}=\\frac{\\widetilde q_i}{\\sum_j\\widetilde q_j}.$$\n", "\n", "1. Before coding, the first stump predicts -1 for $x\\leq3.5$, +1 otherwise.\n", " Identify its mistake, calculate $\\varepsilon_1$, $a_1$ and the new weights.\n", "2. Complete the three-round function below. Store the weights **before each fit**.\n", " `prediction != y` produces the error indicator; use `np.exp` and `np.log`.\n", "3. Run the supplied display cell. Explain why stump 2 chooses a different threshold.\n", " Which weights enter its fit: $q_i$ or $a_m$? Are earlier stumps retrained?\n", "4. Calculate the final score $F(x)=\\sum_ma_mh_m(x)$ and predict its sign.\n", " Check observation 6 explicitly. Does the last stump alone give the same result?\n", "\n", "The first stump is simple, but it is not deliberately fitted badly. Each round\n", "must still beat chance **under its current observation weights**.\n" ] }, { "cell_type": "code", "execution_count": null, "id": "dac14fe5", "metadata": {}, "outputs": [], "source": [ "# The eight samples : \n", "X_small = np.arange(1., 9.)[:, None]\n", "y_small = np.array([-1, -1, -1, 1, 1, -1, 1, 1]) \n", "\n", "print(\"x:\", X_small[:, 0])\n", "print(\"y:\", y_small)\n" ] }, { "cell_type": "code", "execution_count": null, "id": "de9b04c8", "metadata": { "tags": [ "exercise" ] }, "outputs": [], "source": [ "def fit_adaboost_small(X, y, rounds=3):\n", " q = np.full(len(y), 1 / len(y)) # everybody has the same weight at the beginning\n", " trees, coefficients, weight_history, errors = [], [], [], [] # recording\n", " for m in range(rounds):\n", " weight_history.append(q.copy())\n", " tree = DecisionTreeClassifier(max_depth=1, random_state=SEED)\n", " # TODO: fit tree on X, y with sample_weight=q and obtain its predictions.\n", " # TODO: compute the weighted error and the vote coefficient a.\n", " # TODO: multiply q by exp(-a * y * prediction), then normalize it.\n", " # TODO: append tree, a and error to the corresponding lists.\n", " return trees, np.array(coefficients), np.array(weight_history), np.array(errors)\n" ] }, { "cell_type": "code", "execution_count": null, "id": "871e452e", "metadata": {}, "outputs": [], "source": [ "# Run after completing fit_adaboost_small.\n", "trees, coefficients, weight_history, errors = fit_adaboost_small(X_small, y_small)\n", "print(f\"{'Round':>5} {'Threshold':>10} {'Weighted error':>16} {'Vote a':>10}\")\n", "for m, tree in enumerate(trees):\n", " print(f\"{m+1:5d} {tree.tree_.threshold[0]:10.1f} {errors[m]:16.6f} {coefficients[m]:10.6f}\")\n", "print(\"Observation weights before each fit:\")\n", "print(np.round(weight_history, 4))\n", "fig, axes = plt.subplots(1, 3, figsize=(11, 3), sharey=True)\n", "for m, ax in enumerate(axes):\n", " ax.bar(X_small[:, 0], weight_history[m], color=BLUE)\n", " ax.set(title=f\"Before round {m+1}\", xlabel=\"Observation\", xticks=np.arange(1, 9))\n", "axes[0].set_ylabel(\"Observation weight q\")\n", "plt.tight_layout()\n", "plt.show()\n" ] }, { "cell_type": "code", "execution_count": null, "id": "bb9cc58d", "metadata": { "tags": [ "exercise" ] }, "outputs": [], "source": [ "# TODO: build votes with shape (3 stumps, 8 observations).\n", "# TODO: multiply each row by its coefficient a, then sum over stumps.\n", "# TODO: predict +1 if score > 0, -1 otherwise (tie convention for score 0).\n", "# TODO: print ensemble predictions and errors\n" ] }, { "cell_type": "markdown", "id": "fc67177a", "metadata": {}, "source": [ "## 2. Connect the loop to scikit-learn — 10 min\n", "\n", "Fit `AdaBoostClassifier(estimator=DecisionTreeClassifier(max_depth=1),\n", "n_estimators=3, learning_rate=1., random_state=SEED)` on the same eight points.\n", "\n", "1. Compare predictions and thresholds (`estimators_`) with your manual loop.\n", "2. Compare `estimator_errors_` with your weighted errors.\n", "3. Compare `estimator_weights_` with your coefficients. Why is there a factor 2?\n", "\n", "For two classes, sklearn's **SAMME** coefficient is\n", "$\\log((1-\\varepsilon)/\\varepsilon)=2a$. Its update multiplies errors by $e^{2a}$\n", "and correct cases by 1. The manual update multiplies errors by $e^a$ and correct\n", "cases by $e^{-a}$; these are proportional, hence identical **after normalization**.\n", "A common positive factor in the vote coefficients preserves its sign.\n", "Compare class predictions rather than expecting identical `decision_function` values.\n" ] }, { "cell_type": "code", "execution_count": null, "id": "92169247", "metadata": { "tags": [ "exercise" ] }, "outputs": [], "source": [ "# TODO: fit ada_small using AdaBoostClassifier with the supplied settings.\n", "# TODO: print its thresholds, errors and estimator weights.\n", "# TODO: compare predictions with manual_predictions and weights with 2*coefficients.\n" ] }, { "cell_type": "markdown", "id": "2a885361", "metadata": {}, "source": [ "## 3. Same classification dataset as TP5 — 5 min\n", "\n", "Use the bundled breast-cancer dataset, with **malignant = 1** and **benign = 0**.\n", "XGBoost's binary classifier expects labels 0/1. This recodes benign -1 from TP5 as 0;\n", "the positive class and the stratified 20% test observations remain the same.\n", "\n", "All model choices use development data only. Keep the test set unused until Section 6.\n", "Report development class counts and feature count. Do these tree models need scaling?\n" ] }, { "cell_type": "code", "execution_count": null, "id": "35c06bab", "metadata": {}, "outputs": [], "source": [ "data = load_breast_cancer()\n", "X = data.data\n", "y = (data.target == 0).astype(int) # Malignant = 1, benign = 0.\n", "X_dev, X_test, y_dev, y_test = train_test_split(\n", " X, y, test_size=.2, stratify=y, random_state=SEED)\n", "cv = StratifiedKFold(3, shuffle=True, random_state=SEED)\n", "print(\"Development:\", X_dev.shape, \"test:\", X_test.shape)\n", "print(\"Development class counts:\", dict(zip(*np.unique(y_dev, return_counts=True))))\n", "\n", "# A separate development-only split for learning curves; never the final test set.\n", "X_fit, X_val, y_fit, y_val = train_test_split(\n", " X_dev, y_dev, test_size=.25, stratify=y_dev, random_state=SEED)\n", "\n", "def positive_probability(model, X):\n", " column = np.flatnonzero(model.classes_ == 1)[0]\n", " return model.predict_proba(X)[:, column]\n" ] }, { "cell_type": "markdown", "id": "a44f7858", "metadata": {}, "source": [ "## 4. Learning rate and number of trees — 15 min\n", "\n", "On `X_fit/y_fit` and `X_val/y_val` only, compare **GradientBoostingClassifier**\n", "and **XGBClassifier** at learning rates .05 and .2, with 150 depth-2 trees.\n", "\n", "- For sklearn, use `staged_predict_proba` to obtain predictions after each tree.\n", "- For XGBoost, fit once with `tree_method=\"hist\"`, `objective=\"binary:logistic\"`,\n", " `eval_metric=\"logloss\"`, `reg_lambda=1.`, `gamma=0.`, `n_jobs=1`, `random_state=SEED`.\n", " For the first m trees, use `predict_proba(X, iteration_range=(0, m))`.\n", " The upper bound m is excluded, so this includes trees 0 through m-1.\n", "- For each library and learning rate, calculate **training and validation log-loss**\n", " after each stage. Plot two panels, one per library. Lower log-loss is better.\n", "\n", "Which stage minimizes validation loss for each curve? Does training loss tell the\n", "same story? Why does a smaller learning rate often need more trees? What quantity\n", "does a new tree correct in logistic gradient boosting: the score or the probability?\n", "The two libraries need not give identical curves even with matching depth/rate/budget.\n" ] }, { "cell_type": "code", "execution_count": null, "id": "1c977736", "metadata": { "tags": [ "exercise" ] }, "outputs": [], "source": [ "# TODO: for each library and rate in [.05, .2], fit the specified model on X_fit/y_fit.\n", "# TODO: obtain probabilities at stages 1 through 150 on X_fit and X_val.\n", "# TODO: compute log_loss curves; plot training (dashed) and validation (solid).\n", "# TODO: report the validation minimum stage for each curve, without using X_test.\n" ] }, { "cell_type": "markdown", "id": "278a277b", "metadata": {}, "source": [ "## 5. Select learning procedures on development data — 25 min\n", "\n", "Compare the following procedures by **mean development CV ROC-AUC**, using the same\n", "three stratified folds. Set `return_train_score=True`, `n_jobs=1`, `error_score=\"raise\"`.\n", "Use all `X_dev/y_dev` for this CV; the previous small validation split was illustrative.\n", "\n", "| Procedure | Parameters to search |\n", "|---|---|\n", "| `DummyClassifier(strategy=\"prior\")` | none |\n", "| `DecisionTreeClassifier(random_state=SEED)` | `max_depth` in [1, 2, 3] |\n", "| `AdaBoostClassifier(estimator=DecisionTreeClassifier(random_state=SEED), random_state=SEED)` | `estimator__max_depth` in [1, 2], `n_estimators` in [50, 150], `learning_rate` in [.1, 1.] |\n", "| `GradientBoostingClassifier(random_state=SEED)` | `max_depth` in [1, 2], `n_estimators` in [50, 150], `learning_rate` in [.05, .2] |\n", "| `XGBClassifier(...)` | `max_depth` in [1, 2], `n_estimators` in [50, 150], `learning_rate` in [.05, .2], `reg_lambda` in [1., 5.] |\n", "\n", "For XGBoost, keep `objective=\"binary:logistic\"`, `eval_metric=\"logloss\"`,\n", "`tree_method=\"hist\"`, `gamma=0.`, `n_jobs=1`, `random_state=SEED` fixed.\n", "Use fixed stage budgets here, without early stopping or `eval_set` inside the searches.\n", "\n", "1. Store fitted searches in `searches`, keyed by procedure name.\n", "2. Print mean train AUC, mean CV AUC, CV standard deviation and chosen parameters.\n", " Set `winner` using CV only, then freeze all choices.\n", "3. Explain `learning_rate`, `max_depth`, `n_estimators`, and XGBoost's `reg_lambda`.\n", " Why is `estimator__max_depth` nested for AdaBoost? Does its learning rate replace\n", " observation weights? Is `reg_lambda` the same as the learning rate?\n", "4. Which procedures improve on a single shallow tree on these CV folds? Is the\n", " largest training AUC sufficient to choose the best procedure?\n", "\n", "`eval_metric=\"logloss\"` configures XGBoost's evaluation metric; `GridSearchCV`\n", "still selects with `scoring=\"roc_auc\"`. The compact grids are for this TP,\n", "not a claim of exhaustive optimization or equal computational budgets.\n" ] }, { "cell_type": "code", "execution_count": null, "id": "3d80bf6b", "metadata": { "tags": [ "exercise" ] }, "outputs": [], "source": [ "# TODO: define specs = {name: (estimator, parameter_grid)} for the five procedures.\n", "# TODO: fit GridSearchCV searches using X_dev, y_dev and the supplied cv.\n", "# TODO: print a NumPy/Python table with train AUC, CV AUC, CV std and parameters.\n", "# TODO: choose winner by CV only, before looking at test scores.\n" ] }, { "cell_type": "markdown", "id": "428fc9d4", "metadata": {}, "source": [ "## 6. Final test-performance table and error analysis — 15 min\n", "\n", "Freeze the five procedures and parameters from Section 5. Evaluate each **once** on\n", "the same held-out test set. Print a final table containing:\n", "\n", "- model name and selected parameters;\n", "- ROC-AUC, accuracy, malignant precision, malignant recall and malignant F1.\n", "\n", "Use `positive_probability` for AUC and `predict` for class metrics (`pos_label=1`).\n", "Do not use predicted classes as the scores for AUC. Do not tune decision thresholds\n", "or parameters on the test set. Keep the procedure `winner` selected by CV, even if\n", "another model has a larger test score.\n", "\n", "Display the CV winner's confusion matrix with labels `[0, 1]` and print its\n", "false-negative test-row indices. Explain what a false negative means here.\n", "Does the largest AUC necessarily imply the highest recall at the default threshold?\n" ] }, { "cell_type": "code", "execution_count": null, "id": "fc93016c", "metadata": { "tags": [ "exercise" ] }, "outputs": [], "source": [ "# TODO: build test_table (list of dictionaries) for all five frozen procedures.\n", "# TODO: print parameters, test ROC-AUC, accuracy, precision, recall and F1.\n", "# TODO: plot the CV winner's confusion matrix, order [benign, malignant].\n", "# TODO: print its false-negative test-row indices and interpret the metrics.\n" ] }, { "cell_type": "markdown", "id": "da916981", "metadata": {}, "source": [ "## Optional — Early stopping with XGBoost\n", "\n", "Use development data only; do not update the final test comparison.\n", "Create a **new** XGBClassifier with the CV-selected XGBoost parameters, a budget of\n", "500 trees and `early_stopping_rounds=15` in its **constructor**. Keep the other fixed\n", "XGBoost settings. Fit on `X_fit/y_fit` with\n", "`eval_set=[(X_fit, y_fit), (X_val, y_val)]`, `verbose=False`.\n", "The last evaluation set is the one monitored for stopping, here validation.\n", "\n", "1. Plot both log-loss histories from `evals_result()`.\n", "2. Print `best_iteration`, `best_score` and the number of trained rounds from\n", " `get_booster().num_boosted_rounds()`. Explain the difference between zero-based\n", " best iteration, selected stage count and the number of trained rounds.\n", "3. Why is patience useful? What would be wrong with passing `X_test/y_test` as eval_set?\n", "\n", "The sklearn XGBoost interface uses `best_iteration` automatically in its predictions\n", "when early stopping is active. This differs from keeping every trained round.\n", "This development experiment illustrates the mechanism; its validation split is not\n", "an independent final evaluation, since the parameters came from CV on all development data.\n" ] }, { "cell_type": "code", "execution_count": null, "id": "651c30f1", "metadata": { "tags": [ "exercise" ] }, "outputs": [], "source": [ "# TODO: copy searches[\"XGBoost\"].best_params_ and replace n_estimators by 500.\n", "# TODO: instantiate the new model with early_stopping_rounds=15; fit on development only.\n", "# TODO: plot evals_result() log-loss histories; print best iteration and trained rounds.\n" ] }, { "cell_type": "markdown", "id": "c5feb6cb", "metadata": {}, "source": [ "## Documentation\n", "\n", "- [scikit-learn: AdaBoostClassifier](https://scikit-learn.org/1.7/modules/generated/sklearn.ensemble.AdaBoostClassifier.html)\n", "- [scikit-learn: GradientBoostingClassifier](https://scikit-learn.org/1.7/modules/generated/sklearn.ensemble.GradientBoostingClassifier.html)\n", "- [XGBoost: sklearn interface and early stopping](https://xgboost.readthedocs.io/en/stable/python/sklearn_estimator.html)\n", "- [XGBoost: parameters](https://xgboost.readthedocs.io/en/stable/parameter.html)\n", "\n", "The numerical AdaBoost example uses the lecture's binary exponential-loss convention.\n" ] } ], "metadata": { "kernelspec": { "display_name": "insa-iml-practicals (3.10.20)", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.10.20" } }, "nbformat": 4, "nbformat_minor": 5 }