{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Practical 2 — Audit and build a reliable ML protocol\n",
    "\n",
    "In this practical, you will first audit a deliberately flawed experiment, then rebuild it as a valid learning and evaluation protocol.\n",
    "\n",
    "By the end of the session, you should be able to:\n",
    "\n",
    "- distinguish training, validation and test roles;\n",
    "- detect preprocessing, model-selection and duplicate leakage;\n",
    "- select a metric for an imbalanced classification problem;\n",
    "- keep related observations together during splitting;\n",
    "- combine preprocessing and prediction in a `Pipeline`;\n",
    "- select a hyperparameter with cross-validation;\n",
    "- compare a model with a simple baseline;\n",
    "- write a concise ML audit.\n",
    "\n",
    "> The intentionally flawed result is part of the exercise. Do not reuse it as evidence in the final conclusion."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "from sklearn import __version__ as sklearn_version\n",
    "\n",
    "RANDOM_STATE = 42\n",
    "print(\"scikit-learn version:\", sklearn_version)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1. Scenario and provided data\n",
    "\n",
    "A monitoring system must detect a rare machine failure. Each row contains measurements available before the maintenance decision. The positive class (`y = 1`) represents a failure.\n",
    "\n",
    "The dataset below is synthetic so that the methodological problems are controlled and reproducible. Some source records were accidentally exported twice. `source_id` identifies the original record.\n",
    "\n",
    "Documentation: [`make_classification`](https://scikit-learn.org/stable/modules/generated/sklearn.datasets.make_classification.html)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.datasets import make_classification\n",
    "\n",
    "X_original, y_original = make_classification(\n",
    "    n_samples=1_200,\n",
    "    n_features=12,\n",
    "    n_informative=6,\n",
    "    n_redundant=2,\n",
    "    weights=[0.85, 0.15],\n",
    "    class_sep=1.3,\n",
    "    flip_y=0.01,\n",
    "    random_state=RANDOM_STATE,\n",
    ")\n",
    "\n",
    "# Simulate heterogeneous physical units.\n",
    "feature_scales = np.array([1, 10, 100, 0.1, 50, 2, 0.01, 20, 5, 200, 0.5, 80])\n",
    "X_original = X_original * feature_scales\n",
    "\n",
    "# Simulate an accidental second export of 180 source records.\n",
    "rng = np.random.default_rng(RANDOM_STATE)\n",
    "duplicated_sources = rng.choice(len(X_original), size=180, replace=False)\n",
    "X = np.vstack([X_original, X_original[duplicated_sources]])\n",
    "y = np.concatenate([y_original, y_original[duplicated_sources]])\n",
    "source_id = np.concatenate([np.arange(len(X_original)), duplicated_sources])\n",
    "\n",
    "# Shuffle the exported rows.\n",
    "order = rng.permutation(len(X))\n",
    "X, y, source_id = X[order], y[order], source_id[order]\n",
    "feature_names = [f\"sensor_{j:02d}\" for j in range(X.shape[1])]\n",
    "\n",
    "print(\"X shape:\", X.shape)\n",
    "print(\"Positive-class prevalence:\", f\"{y.mean():.3%}\")\n",
    "print(\"Unique source records:\", np.unique(source_id).size)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2. First inspection\n",
    "\n",
    "Before modelling, answer the following questions.\n",
    "\n",
    "1. How many exported rows and original source records are present?\n",
    "2. How many positive and negative examples are there?\n",
    "3. Would accuracy alone be sufficient for this task?\n",
    "4. Do the features have comparable numerical scales?\n",
    "\n",
    "Display the class counts and a boxplot of the features."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": [
     "exercise"
    ]
   },
   "outputs": [],
   "source": [
    "# TODO: inspect the class distribution and feature scales.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3. The deliberately flawed experiment\n",
    "\n",
    "The following experiment was sent with this conclusion:\n",
    "\n",
    "> “The model reaches excellent accuracy. We tested many values of $k$ and retained the best one, so this model is validated and ready for deployment.”\n",
    "\n",
    "Run the code, an then audit it to identify errors."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": [
     "flawed-example"
    ]
   },
   "outputs": [],
   "source": [
    "from sklearn.model_selection import train_test_split\n",
    "from sklearn.preprocessing import StandardScaler\n",
    "from sklearn.neighbors import KNeighborsClassifier\n",
    "\n",
    "X_scaled = StandardScaler().fit_transform(X)\n",
    "\n",
    "X_train_bad, X_test_bad, y_train_bad, y_test_bad, id_train_bad, id_test_bad = train_test_split(\n",
    "    X_scaled, y, source_id, test_size=0.25, random_state=RANDOM_STATE\n",
    ")\n",
    "\n",
    "best_accuracy_bad = -np.inf\n",
    "best_k_bad = None\n",
    "best_model_bad = None\n",
    "\n",
    "for k in range(1, 31):\n",
    "    candidate = KNeighborsClassifier(n_neighbors=k)\n",
    "    candidate.fit(X_train_bad, y_train_bad)\n",
    "    test_accuracy = candidate.score(X_test_bad, y_test_bad)\n",
    "    if test_accuracy > best_accuracy_bad:\n",
    "        best_accuracy_bad = test_accuracy\n",
    "        best_k_bad = k\n",
    "        best_model_bad = candidate\n",
    "\n",
    "print(\"Selected k:\", best_k_bad)\n",
    "print(\"Reported test accuracy:\", f\"{best_accuracy_bad:.3f}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 4. Audit the protocol\n",
    "\n",
    "Identify at least five problems in the experiment or its conclusion. For each problem, state:\n",
    "\n",
    "- what information or assumption is problematic;\n",
    "- why it can bias the result;\n",
    "- how the protocol should be changed.\n",
    "\n",
    "Consider preprocessing, partitions, duplicated sources, metric choice, model selection and the strength of the final claim."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": [
     "exercise"
    ]
   },
   "outputs": [],
   "source": [
    "# TODO: write your audit as Python comments or in a new Markdown cell.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 5. Show why accuracy is insufficient\n",
    "\n",
    "Evaluate the flawed selected model with:\n",
    "\n",
    "- accuracy;\n",
    "- precision, recall and F1 score for the positive class;\n",
    "- ROC-AUC, computed from the positive-class scores;\n",
    "- a confusion matrix.\n",
    "\n",
    "Compare it with a classifier that always predicts the most frequent class. Recall that random ranking corresponds to a ROC-AUC of 0.5.\n",
    "\n",
    "Documentation: [classification metrics](https://scikit-learn.org/stable/modules/model_evaluation.html#classification-metrics), [`DummyClassifier`](https://scikit-learn.org/stable/modules/generated/sklearn.dummy.DummyClassifier.html)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": [
     "exercise"
    ]
   },
   "outputs": [],
   "source": [
    "# TODO: compute several metrics for best_model_bad and a most-frequent baseline.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 6. Measure duplicate leakage\n",
    "\n",
    "Use `id_train_bad` and `id_test_bad` to count how many original sources occur in both partitions.\n",
    "\n",
    "Why does this violate the independence expected between training and test data?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": [
     "exercise"
    ]
   },
   "outputs": [],
   "source": [
    "# TODO: find source identifiers shared by the flawed training and test sets.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 7. Build an independent final test set\n",
    "\n",
    "Use [`GroupShuffleSplit`](https://scikit-learn.org/stable/modules/generated/sklearn.model_selection.GroupShuffleSplit.html) to reserve approximately 25% of the source records for final testing. Pass `source_id` through the `groups` argument so that all copies of one source stay together.\n",
    "\n",
    "Requirements:\n",
    "\n",
    "- do not scale the data before splitting;\n",
    "- create `X_dev`, `X_test`, `y_dev`, `y_test`, `groups_dev`, and `groups_test`;\n",
    "- verify that the two group sets are disjoint;\n",
    "- inspect the positive-class prevalence in both partitions;\n",
    "- do not evaluate a model on `X_test` yet."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": [
     "exercise"
    ]
   },
   "outputs": [],
   "source": [
    "from sklearn.model_selection import GroupShuffleSplit\n",
    "\n",
    "# TODO: construct a group-aware development/test split.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 8. Define the complete learning pipeline\n",
    "\n",
    "Create a [`Pipeline`](https://scikit-learn.org/stable/modules/generated/sklearn.pipeline.Pipeline.html) containing:\n",
    "\n",
    "1. `StandardScaler`;\n",
    "2. `KNeighborsClassifier`.\n",
    "\n",
    "Why must the scaler be inside the object passed to cross-validation?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": [
     "exercise"
    ]
   },
   "outputs": [],
   "source": [
    "from sklearn.pipeline import Pipeline\n",
    "\n",
    "# TODO: create the pipeline in a variable named model.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 9. Select $k$ without touching the test set\n",
    "\n",
    "Use [`StratifiedGroupKFold`](https://scikit-learn.org/stable/modules/generated/sklearn.model_selection.StratifiedGroupKFold.html) inside [`GridSearchCV`](https://scikit-learn.org/stable/modules/generated/sklearn.model_selection.GridSearchCV.html).\n",
    "\n",
    "Requirements:\n",
    "\n",
    "- use five folds, shuffling and the course random seed;\n",
    "- keep duplicate sources together with `groups=groups_dev`;\n",
    "- try `knn__n_neighbors` from 1 to 30;\n",
    "- use F1 score as the primary selection metric;\n",
    "- fit the search on development data only.\n",
    "\n",
    "Explain the double underscore in `knn__n_neighbors`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": [
     "exercise"
    ]
   },
   "outputs": [],
   "source": [
    "from sklearn.model_selection import StratifiedGroupKFold, GridSearchCV\n",
    "\n",
    "# TODO: create cv and search, then fit search on the development data.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "\n",
    "The remainder of this pratical work is optionnal during the class, but recommended as homework to go further.\n",
    "\n",
    "---\n",
    "\n",
    "\n",
    "## 10. Optional — Inspect the cross-validation results\n",
    "\n",
    "`GridSearchCV.cv_results_` contains NumPy-compatible arrays with the fold results. Extract $k$, mean training score, mean validation score and validation standard deviation. Display the first five values, then plot training and validation performance against $k$.\n",
    "\n",
    "Questions:\n",
    "\n",
    "1. Is the selected value clearly superior to nearby values?\n",
    "2. What behaviour do you observe for very small $k$?\n",
    "3. Is the standard deviation across folds a confidence interval?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": [
     "exercise"
    ]
   },
   "outputs": [],
   "source": [
    "# TODO: inspect search.cv_results_ and plot the learning behaviour.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 11. Compare with a baseline under the same protocol\n",
    "\n",
    "Use [`cross_val_score`](https://scikit-learn.org/stable/modules/generated/sklearn.model_selection.cross_val_score.html) to evaluate a most-frequent `DummyClassifier` with the same folds, groups and F1 metric.\n",
    "\n",
    "Why is comparison under the same splitting protocol important?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": [
     "exercise"
    ]
   },
   "outputs": [],
   "source": [
    "from sklearn.model_selection import cross_val_score\n",
    "\n",
    "# TODO: evaluate a most-frequent baseline with the same cv object.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 12. Bonus — Choose a decision threshold on development data\n",
    "\n",
    "The classifier uses a threshold of 0.5 by default. For a rare failure, missing a positive case may be more costly than raising a false alarm. Suppose the operational requirement is therefore to obtain at least 80% recall on development data.\n",
    "\n",
    "Use [`cross_val_predict`](https://scikit-learn.org/stable/modules/generated/sklearn.model_selection.cross_val_predict.html) to obtain out-of-fold positive-class scores for the selected pipeline on `X_dev`. Then use [`roc_curve`](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.roc_curve.html) to:\n",
    "\n",
    "1. plot the ROC curve and identify the point associated with each threshold;\n",
    "2. among thresholds reaching at least 80% recall, retain the one with the lowest false-positive rate.\n",
    "\n",
    "Why must this threshold be fixed without looking at the final test set?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": [
     "exercise"
    ]
   },
   "outputs": [],
   "source": [
    "from sklearn.model_selection import cross_val_predict\n",
    "from sklearn.metrics import roc_curve\n",
    "\n",
    "# TODO: obtain out-of-fold scores and choose the decision threshold.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 13. Freeze the protocol and evaluate once\n",
    "\n",
    "Before revealing the final result, write down:\n",
    "\n",
    "- the selected learning procedure;\n",
    "- the primary metric;\n",
    "- the baseline;\n",
    "- the expected limitation of this experiment.\n",
    "\n",
    "Then use the fitted `search.best_estimator_` to evaluate `X_test` exactly once. Compare decisions obtained with the default threshold 0.5 and with the previously frozen `decision_threshold`. Report accuracy, precision, recall, F1 score, ROC-AUC and the confusion matrices. Plot the ROC curve and compare it with the random-ranking diagonal.\n",
    "\n",
    "> With the default `refit=True`, `GridSearchCV` has already refitted the selected pipeline on all development data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": [
     "exercise"
    ]
   },
   "outputs": [],
   "source": [
    "# The default keeps this section executable if the threshold bonus was skipped.\n",
    "decision_threshold = globals().get(\"decision_threshold\", 0.5)\n",
    "\n",
    "# TODO: perform the single final evaluation.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 14. Optional — Write the ML audit\n",
    "\n",
    "Complete this short audit in a new Markdown cell, one sentence for each point.\n",
    "\n",
    "### ML audit card\n",
    "\n",
    "- **Prediction task and positive class:**\n",
    "- **Independent unit:**\n",
    "- **Outer test design:**\n",
    "- **Preprocessing and leakage protection:**\n",
    "- **Hyperparameter-selection protocol:**\n",
    "- **Primary metric and justification:**\n",
    "- **Baseline:**\n",
    "- **Final result:**\n",
    "- **What the result does not establish:**\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 15. Bonus — Uncertainty by group bootstrap\n",
    "\n",
    "The final score is computed from one finite test sample. Estimate a 95% bootstrap interval for F1 score. Because rows from one source are dependent, resample **source identifiers**, not individual rows.\n",
    "\n",
    "For each bootstrap repetition:\n",
    "\n",
    "1. sample test source identifiers with replacement;\n",
    "2. collect all rows belonging to each sampled source;\n",
    "3. recompute F1 score.\n",
    "\n",
    "Does this interval quantify uncertainty caused by changing the training set or algorithm seed?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": [
     "exercise",
     "optional"
    ]
   },
   "outputs": [],
   "source": [
    "# TODO (optional): bootstrap independent test groups and build a percentile interval.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 16. Optional — Takeaways\n",
    "\n",
    "Complete each sentence in your own words.\n",
    "\n",
    "1. A test set becomes validation data when ...\n",
    "2. A pipeline prevents preprocessing leakage because ...\n",
    "3. Related observations must stay in the same partition because ...\n",
    "4. Accuracy was misleading here because ...\n",
    "5. Cross-validation helps model selection, but does not ...\n",
    "6. A high final score alone does not prove ...\n",
    "7. Lowering the decision threshold usually increases ... and decreases ...\n",
    "8. Changing the decision threshold leaves ROC-AUC unchanged because ..."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python (cours-ml)",
   "language": "python",
   "name": "cours-ml"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.12.13"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}