{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "tp6-000",
   "metadata": {},
   "source": [
    "# Practical 6 — Kernel SVM: features, model selection and prediction\n",
    "\n",
    "Objectives: connect explicit features and kernels, evaluate Gram matrices, visualize the interaction of C and gamma, select the whole pipeline with CV, reconstruct predictions and distinguish fitting flexibility from generalization.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "tp6-001",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2026-10-05T19:23:10.530444Z",
     "iopub.status.busy": "2026-10-05T19:23:10.530092Z",
     "iopub.status.idle": "2026-10-05T19:23:11.188754Z",
     "shell.execute_reply": "2026-10-05T19:23:11.188148Z"
    }
   },
   "outputs": [],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "from sklearn.datasets import load_breast_cancer, make_blobs, make_circles\n",
    "from sklearn.dummy import DummyClassifier, DummyRegressor\n",
    "from sklearn.linear_model import LogisticRegression, LinearRegression\n",
    "from sklearn.metrics import (accuracy_score, confusion_matrix, recall_score,\n",
    "                             roc_auc_score, root_mean_squared_error)\n",
    "from sklearn.metrics.pairwise import rbf_kernel\n",
    "from sklearn.model_selection import GridSearchCV, StratifiedKFold, train_test_split\n",
    "from sklearn.neighbors import KNeighborsClassifier\n",
    "from sklearn.pipeline import Pipeline\n",
    "from sklearn.preprocessing import StandardScaler\n",
    "from sklearn.svm import SVC, SVR, OneClassSVM\n",
    "\n",
    "SEED = 42\n",
    "BLUE, RED = \"#2474b5\", \"#e30613\"\n",
    "plt.rcParams.update({\"figure.figsize\": (7, 4), \"font.size\": 11})\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "tp6-002",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2026-10-05T19:23:11.190416Z",
     "iopub.status.busy": "2026-10-05T19:23:11.190260Z",
     "iopub.status.idle": "2026-10-05T19:23:11.193962Z",
     "shell.execute_reply": "2026-10-05T19:23:11.193563Z"
    }
   },
   "outputs": [],
   "source": [
    "def boundary_plot(ax, estimator, X, y, supports=None, levels=(0,)):\n",
    "    low, high = X.min(axis=0) - .6, X.max(axis=0) + .6\n",
    "    xx, yy = np.meshgrid(np.linspace(low[0], high[0], 150),\n",
    "                         np.linspace(low[1], high[1], 150))\n",
    "    grid = np.column_stack((xx.ravel(), yy.ravel()))\n",
    "    z = estimator.decision_function(grid).reshape(xx.shape)\n",
    "    ax.contourf(xx, yy, z, levels=[-np.inf, 0, np.inf], colors=[BLUE, RED], alpha=.1)\n",
    "    visible = [v for v in levels if z.min() < v < z.max()]\n",
    "    if visible:\n",
    "        ax.contour(xx, yy, z, levels=visible, colors=\"black\",\n",
    "                   linestyles=[\"-\" if v == 0 else \"--\" for v in visible])\n",
    "    for label, color in ((-1, BLUE), (1, RED)):\n",
    "        ax.scatter(*X[y == label].T, color=color, s=20, label=f\"y={label:+d}\")\n",
    "    if supports is not None:\n",
    "        ax.scatter(*supports.T, s=80, facecolors=\"none\", edgecolors=\"black\")\n",
    "    ax.set(xlabel=\"$x_1$\", ylabel=\"$x_2$\")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "tp6-003",
   "metadata": {},
   "source": [
    "## 1. A nonlinear classification problem — 10 min\n",
    "\n",
    "Generate noisy concentric circles using `make_circles` and split raw inputs into development/test subsets. Plot development observations only. Fit a scaled linear SVM on development data and inspect its boundary. Why is a straight boundary a poor representation for this problem?\n",
    "\n",
    "All boundary plots in the core use **development observations**. They illustrate geometry and never replace CV or test evaluation.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "tp6-004",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2026-10-05T19:23:11.195123Z",
     "iopub.status.busy": "2026-10-05T19:23:11.195040Z",
     "iopub.status.idle": "2026-10-05T19:23:11.198072Z",
     "shell.execute_reply": "2026-10-05T19:23:11.197643Z"
    }
   },
   "outputs": [],
   "source": [
    "X, labels = make_circles(n_samples=400, factor=.4, noise=.18, random_state=SEED)\n",
    "y = 2*labels-1\n",
    "X_dev, X_test, y_dev, y_test = train_test_split(X,y,test_size=.25,stratify=y,random_state=SEED)\n",
    "cv = StratifiedKFold(5, shuffle=True, random_state=SEED)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "tp6-005",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2026-10-05T19:23:11.199282Z",
     "iopub.status.busy": "2026-10-05T19:23:11.199179Z",
     "iopub.status.idle": "2026-10-05T19:23:11.200750Z",
     "shell.execute_reply": "2026-10-05T19:23:11.200345Z"
    },
    "tags": [
     "exercise"
    ]
   },
   "outputs": [],
   "source": [
    "# TODO: plot X_dev by class, then fit a scaled linear SVC(C=1).\n",
    "# Use boundary_plot in raw coordinates with the fitted pipeline.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "tp6-007",
   "metadata": {},
   "source": [
    "## 2. Explicit features, implicit product — 15 min\n",
    "\n",
    "Implement $\\phi(x)=(x_1^2,\\sqrt2 x_1x_2,x_2^2)$. For $x=(1,2)$ and $x'=(3,1)$, compare $\\phi(x)^T\\phi(x')$ with $(x^Tx')^2$.\n",
    "\n",
    "On development inputs scaled **once**, compare `SVC(kernel=\"linear\")` on phi features with `SVC(kernel=\"poly\", degree=2, gamma=1, coef0=0)` on the scaled inputs. Use C=1 and tol=1e-8. Do not standardize phi again: that would change the kernel being compared. These are identity checks; later CV uses fold-specific pipelines.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "tp6-008",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2026-10-05T19:23:11.443177Z",
     "iopub.status.busy": "2026-10-05T19:23:11.443081Z",
     "iopub.status.idle": "2026-10-05T19:23:11.444776Z",
     "shell.execute_reply": "2026-10-05T19:23:11.444403Z"
    },
    "tags": [
     "exercise"
    ]
   },
   "outputs": [],
   "source": [
    "# TODO: implement phi(X), accepting an (n,2) array.\n",
    "# TODO: verify the lecture product and full Gram identity.\n",
    "# TODO: fit the explicit/implicit models and compare development decision scores.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "tp6-010",
   "metadata": {},
   "source": [
    "## 3. C and gamma: inspect flexibility — 15 min\n",
    "\n",
    "For RBF, $k(x,x')=e^{-\\gamma\\|x-x'\\|^2}$. Fit six scaled pipelines on development data with C in `[.1,100]` and gamma in `[.1,1,20]`. Plot the zero-score boundaries and count supports.\n",
    "\n",
    "Which settings produce smooth boundaries or small islands? Can you infer test accuracy just from these plots? Explain why gamma is not another name for C.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "tp6-011",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2026-10-05T19:23:11.454205Z",
     "iopub.status.busy": "2026-10-05T19:23:11.454127Z",
     "iopub.status.idle": "2026-10-05T19:23:11.455800Z",
     "shell.execute_reply": "2026-10-05T19:23:11.455453Z"
    },
    "tags": [
     "exercise"
    ]
   },
   "outputs": [],
   "source": [
    "C_values=[.1,100]\n",
    "gamma_values=[.1,1,20]\n",
    "# TODO: fit the 6 pipelines and call boundary_plot in a 2x3 figure.\n",
    "# Keep X_test and y_test unused.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "tp6-014",
   "metadata": {},
   "source": [
    "## 4. Select kernel and parameters together — 25 min\n",
    "\n",
    "Build a scaled SVC pipeline. Use a **list of parameter dictionaries** so each kernel gets only relevant parameters:\n",
    "\n",
    "- linear: C in `[.1,1,10]`;\n",
    "- homogeneous polynomial: degree in `[2,3]`, gamma=1, coef0=0, C in `[.1,1,10]`;\n",
    "- RBF: C in `[.1,1,10,100]`, gamma in `[.1,1,10,20]`.\n",
    "\n",
    "Use five-fold stratified CV on development data, ROC-AUC and `return_train_score=True`. Display the best result for each kernel family, plot mean CV AUC over the RBF grid and compare train/CV scores. Select the best procedure without looking at test results. If a grid optimum is on an edge, explain how you would extend the search on development data.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "tp6-015",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2026-10-05T19:23:12.022674Z",
     "iopub.status.busy": "2026-10-05T19:23:12.022555Z",
     "iopub.status.idle": "2026-10-05T19:23:12.024416Z",
     "shell.execute_reply": "2026-10-05T19:23:12.023978Z"
    },
    "tags": [
     "exercise"
    ]
   },
   "outputs": [],
   "source": [
    "# TODO: define kernel_pipeline and param_grid (list of 3 dictionaries).\n",
    "# TODO: run search=GridSearchCV(...), fit on X_dev,y_dev and inspect cv_results_.\n",
    "# TODO: report the best row for each family and plot RBF mean CV AUC as a heatmap.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "tp6-017",
   "metadata": {},
   "source": [
    "## 5. One final test evaluation — 10 min\n",
    "\n",
    "Freeze the selected procedure and its zero decision threshold. Evaluate once on X_test,y_test: ROC-AUC, accuracy, positive-class recall and confusion matrix. Compare with the fixed linear reference from Section 1, without choosing a new procedure from the test scores. Explain the distinction between a score and a probability.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "tp6-018",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2026-10-05T19:23:12.634394Z",
     "iopub.status.busy": "2026-10-05T19:23:12.634305Z",
     "iopub.status.idle": "2026-10-05T19:23:12.635894Z",
     "shell.execute_reply": "2026-10-05T19:23:12.635558Z"
    },
    "tags": [
     "exercise"
    ]
   },
   "outputs": [],
   "source": [
    "# TODO: evaluate search.best_estimator_ and linear_reference once on test data.\n",
    "# TODO: report the selected model's confusion matrix with labels [-1,+1].\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "tp6-020",
   "metadata": {},
   "source": [
    "## 6. Gram matrices and manual prediction — 15 min\n",
    "\n",
    "Use a **separate fixed RBF demonstration model** (C=1, gamma=.5) on scaled development inputs. It is not a second model-selection experiment.\n",
    "\n",
    "1. Compute its training Gram matrix and verify symmetry and nonnegative eigenvalues up to tolerance (`-1e-8`).\n",
    "2. Reconstruct scores for three new query inputs using support vectors and signed dual coefficients. Explain the dimensions.\n",
    "3. Fit `SVC(kernel=\"precomputed\")` on the full training Gram matrix and compare query scores. Predict expects all training columns, in their original order.\n",
    "4. For $K=\\begin{pmatrix}1&2\\\\2&1\\end{pmatrix}$ and $c=(1,-1)$, compute $c^TKc$. Why are positive entries insufficient?\n",
    "\n",
    "A numerical check on one Gram matrix is not a proof of kernel validity on all finite sets.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "tp6-021",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2026-10-05T19:23:12.648226Z",
     "iopub.status.busy": "2026-10-05T19:23:12.648051Z",
     "iopub.status.idle": "2026-10-05T19:23:12.650216Z",
     "shell.execute_reply": "2026-10-05T19:23:12.649706Z"
    }
   },
   "outputs": [],
   "source": [
    "queries_raw=np.array([[0.,0.],[.7,.7],[-1.3,.1]])\n",
    "gamma=.5\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "tp6-022",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2026-10-05T19:23:12.651182Z",
     "iopub.status.busy": "2026-10-05T19:23:12.651095Z",
     "iopub.status.idle": "2026-10-05T19:23:12.652750Z",
     "shell.execute_reply": "2026-10-05T19:23:12.652371Z"
    },
    "tags": [
     "exercise"
    ]
   },
   "outputs": [],
   "source": [
    "# TODO: fit a scaler on X_dev, transform development and query inputs.\n",
    "# TODO: fit fixed_rbf, compute K_train and K_query_support, reconstruct scores.\n",
    "# TODO: compare with a precomputed model using K_query_train.\n",
    "# TODO: evaluate the invalid 2x2 example.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "tp6-025",
   "metadata": {
    "tags": [
     "optional"
    ]
   },
   "source": [
    "## Optional A — RBF novelty detection after TP5\n",
    "\n",
    "Generate the same positive sensor reference cluster as TP5. Fit a scaled RBF One-Class SVM with gamma=.5, nu=.1 on references only. Predict the same three measurements. Plot the acceptance region and compare the high-valued point with the linear detector's unbounded half-space. Why can centered scaling be used with a distance-based RBF kernel?\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "tp6-026",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2026-10-05T19:23:12.818679Z",
     "iopub.status.busy": "2026-10-05T19:23:12.818573Z",
     "iopub.status.idle": "2026-10-05T19:23:12.821409Z",
     "shell.execute_reply": "2026-10-05T19:23:12.821065Z"
    },
    "tags": [
     "exercise"
    ]
   },
   "outputs": [],
   "source": [
    "rng=np.random.default_rng(SEED+1)\n",
    "X_reference=rng.normal(loc=[3.,3.],scale=[.25,.25],size=(100,2))\n",
    "X_sensor_new=np.array([[3.,3.],[.2,.2],[8.,8.]])\n",
    "# TODO: fit the RBF detector and compare its three scores and flags with TP5.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "tp6-028",
   "metadata": {
    "tags": [
     "optional"
    ]
   },
   "source": [
    "## Optional B — Construct a simple kernel on labelled graphs\n",
    "\n",
    "A graph is represented by node labels and a list of edges. Define shared features: counts of C nodes, O nodes, C--C edges and C--O edges. Construct the kernel $k(G,G')=\\phi(G)^T\\phi(G')$ for the three tiny graphs below.\n",
    "\n",
    "Compute the Gram matrix, check PSD numerically and normalize it with its diagonal. Explain why the explicit feature-map argument proves validity beyond these three graphs. Do these counts uniquely identify a graph? No graph library is required. This is a kernel construction exercise, not a benchmark on three objects.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "tp6-029",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2026-10-05T19:23:12.922222Z",
     "iopub.status.busy": "2026-10-05T19:23:12.922124Z",
     "iopub.status.idle": "2026-10-05T19:23:12.924454Z",
     "shell.execute_reply": "2026-10-05T19:23:12.924031Z"
    },
    "tags": [
     "exercise"
    ]
   },
   "outputs": [],
   "source": [
    "graphs=[\n",
    "    {\"nodes\":[\"C\",\"C\",\"O\"],\"edges\":[(0,1),(1,2)]},\n",
    "    {\"nodes\":[\"C\",\"O\"],\"edges\":[(0,1)]},\n",
    "    {\"nodes\":[\"C\",\"C\",\"C\"],\"edges\":[(0,1),(1,2),(0,2)]},\n",
    "]\n",
    "# TODO: implement graph_features(graph), stack features, compute and normalize the Gram matrix.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f58cdafb",
   "metadata": {},
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "e7db3fd7",
   "metadata": {},
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "insa-iml-practicals (3.10.20)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.20"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}