{ "cells": [ { "cell_type": "markdown", "id": "tp5-000", "metadata": {}, "source": "# Practical 5 — Linear SVM classification\n\n**Objectives**: connect margins to slacks, inspect supports, select a linear classifier with cross-validation, and compare it with logistic regression and k-NN.\n\nComplete the TODO cells. Submit the development-data comparison, the final test-performance table and the interpretation answers. The final section is optional.\n" }, { "cell_type": "code", "execution_count": null, "id": "tp5-001", "metadata": {}, "outputs": [], "source": [ "import matplotlib.pyplot as plt\n", "import numpy as np\n", "from sklearn.datasets import load_breast_cancer, make_blobs\n", "from sklearn.dummy import DummyClassifier\n", "from sklearn.linear_model import LogisticRegression\n", "from sklearn.metrics import (accuracy_score, confusion_matrix, precision_score, recall_score,\n", " f1_score, roc_auc_score)\n", "from sklearn.model_selection import GridSearchCV, StratifiedKFold, train_test_split\n", "from sklearn.neighbors import KNeighborsClassifier\n", "from sklearn.pipeline import Pipeline\n", "from sklearn.preprocessing import StandardScaler\n", "from sklearn.svm import SVC\n", "\n", "SEED = 42\n", "BLUE, RED = \"#2474b5\", \"#e30613\"\n", "plt.rcParams.update({\"figure.figsize\": (7, 4), \"font.size\": 11})\n" ] }, { "cell_type": "markdown", "id": "tp5-002", "metadata": {}, "source": "## 1. Margins, slacks and supports — 15 min\n\nFor labels $y_i\\in\\{-1,+1\\}$, $f_i=w^Tx_i+b$, the signed geometric margin is $y_i f_i/\\|w\\|$. The minimum feasible slack is $\\xi_i=\\max(0,1-y_i f_i)$.\n\n1. Compute these quantities for the four observations below.\n2. Identify points beyond the margin, correctly classified inside it, on the boundary, or misclassified.\n3. Multiply **both** $w,b$ by 3. Which quantities change?\n" }, { "cell_type": "code", "execution_count": null, "id": "tp5-003", "metadata": { "tags": [ "exercise" ] }, "outputs": [], "source": [ "X_small = np.array([[0., 1.], [1., .25], [0., 0.], [0., -.5]])\n", "y_small = np.array([1, 1, -1, 1])\n", "w_small, b_small = np.array([0., 2.]), 0.\n", "# TODO: compute score, functional_margin, geometric_margin and slack.\n", "# Display a table; repeat after multiplying w and b by 3.\n" ] }, { "cell_type": "code", "execution_count": null, "id": "tp5-006", "metadata": {}, "outputs": [], "source": [ "def boundary_plot(ax, estimator, X, y, supports=None, levels=(0,)):\n", " low, high = X.min(axis=0) - .6, X.max(axis=0) + .6\n", " xx, yy = np.meshgrid(np.linspace(low[0], high[0], 150),\n", " np.linspace(low[1], high[1], 150))\n", " grid = np.column_stack((xx.ravel(), yy.ravel()))\n", " z = estimator.decision_function(grid).reshape(xx.shape)\n", " ax.contourf(xx, yy, z, levels=[-np.inf, 0, np.inf], colors=[BLUE, RED], alpha=.1)\n", " visible = [v for v in levels if z.min() < v < z.max()]\n", " if visible:\n", " ax.contour(xx, yy, z, levels=visible, colors=\"black\",\n", " linestyles=[\"-\" if v == 0 else \"--\" for v in visible])\n", " for label, color in ((-1, BLUE), (1, RED)):\n", " ax.scatter(*X[y == label].T, color=color, s=20, label=f\"y={label:+d}\")\n", " if supports is not None:\n", " ax.scatter(*supports.T, s=80, facecolors=\"none\", edgecolors=\"black\")\n", " ax.set(xlabel=\"$x_1$\", ylabel=\"$x_2$\")\n" ] }, { "cell_type": "code", "execution_count": null, "id": "tp5-007", "metadata": { "tags": [ "exercise" ] }, "outputs": [], "source": [ "X_geometry, lab = make_blobs(n_samples=90, centers=[(-1, -.6), (1, .6)],\n", " cluster_std=.85, random_state=SEED)\n", "y_geometry = 2*lab - 1\n", "X_geometry = StandardScaler().fit_transform(X_geometry)\n", "# TODO: fit SVC(kernel=\"linear\") for C=.05 and C=10 on these fixed observations.\n", "# Call boundary_plot with model.support_vectors_ and levels=(-1, 0, 1).\n", "# Compare support counts, training errors and margin violations.\n" ] }, { "cell_type": "markdown", "id": "tp5-009", "metadata": {}, "source": "## 2. Same classification problem as TP4 — 10 min\n\nUse the bundled breast cancer dataset. As in TP4, recode **malignant as +1**, benign as -1, and keep the same 20% test split (seed 42, stratified). We now use all remaining development observations for cross-validation. Only the encoding changes; the test observations are the same.\n\nInspect class counts and feature scales. Explain why a scaler must be inside each pipeline. The geometry dataset above is independent and only illustrates margins.\n" }, { "cell_type": "code", "execution_count": null, "id": "tp5-010", "metadata": {}, "outputs": [], "source": [ "data = load_breast_cancer()\n", "X = data.data\n", "y = np.where(data.target == 0, 1, -1)\n", "X_dev, X_test, y_dev, y_test = train_test_split(X, y, test_size=.2,\n", " stratify=y, random_state=SEED)\n", "cv = StratifiedKFold(5, shuffle=True, random_state=SEED)\n", "print(\"Development shape:\", X_dev.shape, \"test shape:\", X_test.shape)\n", "print(\"Development malignant fraction:\", np.mean(y_dev == 1))\n" ] }, { "cell_type": "code", "execution_count": null, "id": "tp5-011", "metadata": { "tags": [ "exercise" ] }, "outputs": [], "source": [ "# TODO: report min/max/std for three named features on X_dev.\n", "# TODO: define svm_pipeline = Pipeline with StandardScaler and SVC(kernel=\"linear\").\n" ] }, { "cell_type": "markdown", "id": "tp5-013", "metadata": {}, "source": "## 3. Select C and compare learning procedures — 20 min\n\nSelect by **mean CV ROC-AUC on development data**. Use the same folds for each procedure. Compare linear SVM, logistic regression and k-NN, plus a dummy reference. Tune each real model rather than comparing a tuned SVM with arbitrary defaults.\n\nUse `GridSearchCV`: SVM and logistic `C` in `[.01, .1, 1, 10]`, k-NN `n_neighbors` in `[3, 7, 15, 31]`. AUC accepts SVM decision scores; `probability=True` is unnecessary. For k-NN, scores are positive-class probabilities.\n\nReport mean and standard deviation of CV AUC and selected parameters. Choose the procedure before the test evaluation. Does the largest training score guarantee the largest CV score?\n" }, { "cell_type": "code", "execution_count": null, "id": "tp5-014", "metadata": { "tags": [ "exercise" ] }, "outputs": [], "source": [ "# TODO: build scaled logistic and k-NN pipelines and a dummy pipeline.\n", "# TODO: run the four searches using cv=cv, scoring=\"roc_auc\", return_train_score=True.\n", "# Store them in searches, keyed by model name.\n", "# TODO: display a development comparison and set winner using CV results only.\n" ] }, { "cell_type": "markdown", "id": "tp5-016", "metadata": {}, "source": "## 4. Inspect the selected linear SVM — 20 min\n\nWork on development data only. Use `coef_[0]` and `intercept_[0]` to obtain $w$ and $b$ directly from the fitted SVC. These parameters and `support_vectors_` refer to **scaled coordinates**.\n\n1. Transform the development inputs with the fitted scaler and calculate $f(X)=Xw+b$. Compare with the pipeline's `decision_function`.\n2. Compute the signed margins and minimum slacks. Count misclassified observations and margin violations. Why are these counts different?\n3. Count the support vectors using `support_`. Inspect their signed margins. Are they all misclassified?\n\nKeep test data unused until Section 5.\n" }, { "cell_type": "code", "execution_count": null, "id": "tp5-017", "metadata": { "tags": [ "exercise" ] }, "outputs": [], "source": [ "# TODO: get the scaler and SVC from searches[\"Linear SVM\"].best_estimator_.\n", "# TODO: transform X_dev using the fitted scaler.\n", "# TODO: read w and b from coef_ and intercept_, then compute the scores.\n", "# TODO: compare your scores with the pipeline's decision_function.\n", "# TODO: count development errors, margin violations and support vectors.\n" ] }, { "cell_type": "markdown", "id": "tp5-020", "metadata": {}, "source": "## 5. Final test-performance table — 15 min\n\nFreeze the four procedures and parameters selected in Section 3. Evaluate each **once** on the same test set. Build a final printed table with:\n\n- model name and selected parameters;\n- ROC-AUC, accuracy, malignant precision, malignant recall and malignant F1 (`pos_label=1`).\n\nFor SVM and logistic regression, use `decision_function` for AUC. For k-NN and the dummy reference, use the probability column corresponding to class +1. Use `predict` for the other metrics; no threshold is tuned on test data.\n\nDisplay the confusion matrix of the CV-selected `winner` with labels `[-1, +1]`, then print the test-row indices misclassified by the selected SVM. Explain one difference between AUC and recall. Do not change the selected procedure after reading this table.\n" }, { "cell_type": "code", "execution_count": null, "id": "tp5-021", "metadata": { "tags": [ "exercise" ] }, "outputs": [], "source": [ "# TODO: for each fitted search, compute test predictions and positive-class scores.\n", "# TODO: print the final table: parameters, AUC, accuracy, precision, recall and F1.\n", "# TODO: display the winner's confusion matrix, with labels [-1, +1].\n", "# TODO: print the test-row indices misclassified by the selected SVM.\n" ] }, { "cell_type": "markdown", "id": "tp5-024", "metadata": { "tags": [ "optional" ] }, "source": "## Optional A — Distance and support vectors\n\nUse only the selected SVM and development data.\n\n1. Compute the geometric margins $y_i f_i/\\|w\\|$ for all development observations.\n2. Compare the minimum and maximum geometric margins among support vectors and among the other observations.\n3. Compute the distance between the score planes $f=-1$ and $f=+1$: $2/\\|w\\|$.\n4. Explain why a negative labelled margin differs from an observation that is correctly classified but inside the margin.\n\nUse `coef_`, `intercept_`, `decision_function` and `support_` only.\n" }, { "cell_type": "code", "execution_count": null, "id": "tp5-025", "metadata": { "tags": [ "exercise", "optional" ] }, "outputs": [], "source": [ "# TODO: calculate geometric margins and the width 2 / ||w||.\n", "# TODO: create a boolean support mask using svc.support_.\n", "# TODO: print the geometric-margin range for supports and non-supports.\n" ] }, { "cell_type": "markdown", "metadata": { "tags": [ "optional" ] }, "source": "## Optional B — Check numerical KKT conditions\n\nThis optional exercise uses dual coefficients; the core exercises do not need them. Work on development data only, with the selected linear SVC from Section 4.\n\nScikit-learn stores $\\beta_i=\\alpha_i y_i$ in `dual_coef_[0]` for support vectors. Recover $\\alpha_i=\\beta_i y_i$ for these observations and set the remaining coefficients to zero.\n\nCheck $0\\leq\\alpha_i\\leq C$, $\\sum_i\\alpha_i y_i=0$ and both complementarity conditions:\n\n$$\n\\alpha_i(y_i f_i-1+\\xi_i)=0,\n\\qquad (C-\\alpha_i)\\xi_i=0.\n$$\n\nUse the minimum slacks from Section 4. Report the largest absolute residual for each identity; allow numerical tolerance `1e-2` for this solver. Explain the three cases $\\alpha_i=0$, $0<\\alpha_i