{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Practical 1 — A first machine-learning workflow\n",
    "\n",
    "By the end of this practical, you should be able to:\n",
    "\n",
    "- inspect a dataset represented by NumPy arrays;\n",
    "- identify samples, features and targets;\n",
    "- use the scikit-learn `fit` / `predict` / `score` interface;\n",
    "- separate training and test data;\n",
    "- tune $k$ without using the test set for model selection;\n",
    "- explain why feature scaling matters for a distance-based method;\n",
    "- prevent preprocessing leakage with a `Pipeline`;\n",
    "- make a simple experiment reproducible.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "import sklearn\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1. Working environment\n",
    "\n",
    "Before starting Jupyter, activate the course environment in a terminal:\n",
    "\n",
    "```bash\n",
    "source /opt/venv/iti-iml/bin/activate\n",
    "python -m ipykernel install --user --name cours-ml --display-name \"Python (cours-ml)\"\n",
    "```\n",
    "\n",
    "Then select the **Python (cours-ml)** kernel in Jupyter. The kernel installation is normally required only once.\n",
    "\n",
    "Useful references: [installing scikit-learn](https://scikit-learn.org/stable/install.html) and the [scikit-learn user guide](https://scikit-learn.org/stable/user_guide.html)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import sys\n",
    "# Display the python used by the jupyter kernel \n",
    "print(\"Python used by the jupyter kernel : \", sys.executable)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2. Scikit-learn 101\n",
    "\n",
    "NumPy represents numerical data using multidimensional arrays. In this practical:\n",
    "\n",
    "- `X` will be a two-dimensional array with shape `(n_samples, n_features)`;\n",
    "- `y` will be a one-dimensional array containing one target per sample.\n",
    "\n",
    "Most supervised scikit-learn estimators share the same interface:\n",
    "\n",
    "1. create an estimator, for example `model = SomeClassifier(...)`;\n",
    "2. learn from examples with `model.fit(X, y)`;\n",
    "3. predict with `model.predict(X)`;\n",
    "4. evaluate predictions with a suitable metric.\n",
    "\n",
    "Scikit-learn also provides some built-in datasets. Browse the available [toy datasets](https://scikit-learn.org/stable/datasets/toy_dataset.html) before continuing."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": [
     "exercise"
    ]
   },
   "outputs": [],
   "source": [
    "# NumPy reminder: predict the shape and value of each expression before running it.\n",
    "example = np.array([[1.0, 10.0, 100.0], [2.0, 20.0, 200.0]])\n",
    "\n",
    "print(\"shape:\", example.shape)\n",
    "print(\"first sample:\", example[0])\n",
    "print(\"second feature:\", example[:, 1])\n",
    "print(\"feature means:\", example.mean(axis=0))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3. Load the Wine dataset\n",
    "\n",
    "Read the [`load_wine` documentation](https://scikit-learn.org/stable/modules/generated/sklearn.datasets.load_wine.html).\n",
    "\n",
    "Load the feature matrix into `X` and the target vector into `y` using `return_X_y=True`. Also load the complete `Bunch` object into `wine`, because it contains feature names, class names and a description."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": [
     "exercise"
    ]
   },
   "outputs": [],
   "source": [
    "from sklearn.datasets import load_wine\n",
    "\n",
    "# TODO: create wine, X and y.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": [
     "correction"
    ]
   },
   "outputs": [],
   "source": [
    "wine = load_wine()\n",
    "X, y = load_wine(return_X_y=True)\n",
    "\n",
    "print(type(X), type(y))\n",
    "print(wine.DESCR.splitlines()[0])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 4. Understand the data\n",
    "\n",
    "Answer the following questions using array attributes and the dataset metadata.\n",
    "\n",
    "1. How many samples and features are there?\n",
    "2. What does one row represent?\n",
    "3. What are the feature names?\n",
    "4. What does `y` represent, and which values can it take?\n",
    "5. Are the classes balanced?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": [
     "exercise"
    ]
   },
   "outputs": [],
   "source": [
    "# TODO: inspect X.shape, y.shape, wine.feature_names, wine.target_names\n",
    "# and count the samples in each class with np.unique(..., return_counts=True).\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "tags": [
     "correction"
    ]
   },
   "source": [
    "### Proposed answer\n",
    "\n",
    "Each row describes one wine sample through 13 positive, real-valued chemical measurements. The target is one of three cultivar classes. The classes are not exactly balanced, although no class is extremely rare."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": [
     "correction"
    ]
   },
   "outputs": [],
   "source": [
    "\n",
    "print(f\"X shape: {X.shape}\")\n",
    "print(f\"y shape: {y.shape}\")\n",
    "print(\"Features:\", wine.feature_names)\n",
    "print(\"Target names:\", wine.target_names)\n",
    "\n",
    "classes, counts = np.unique(y, return_counts=True)\n",
    "print(\"Class counts:\", dict(zip(classes, counts)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 5. Visualise feature scales\n",
    "\n",
    "Create a boxplot with one box per feature.\n",
    "\n",
    "Before modelling, note whether all variables have comparable numerical ranges. A boxplot is useful here for comparing scales, but it is not a complete exploration of the dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": [
     "exercise"
    ]
   },
   "outputs": [],
   "source": [
    "# TODO: display a boxplot using Matplotlib and identify each feature.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "tags": [
     "correction"
    ]
   },
   "source": [
    "### Proposed answer\n",
    "\n",
    "The variables have very different magnitudes. For example, `proline` is measured on a much larger numerical scale than several other features. This observation will matter because k-NN compares samples through distances."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": [
     "correction"
    ]
   },
   "outputs": [],
   "source": [
    "plt.figure(figsize=(12, 5))\n",
    "plt.boxplot(X, tick_labels=wine.feature_names)\n",
    "plt.xticks(rotation=70, ha=\"right\")\n",
    "plt.ylabel(\"Observed value\")\n",
    "plt.title(\"Wine features before scaling\")\n",
    "plt.tight_layout()\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 6. Identify the learning problem and metric\n",
    "\n",
    "Is this supervised or unsupervised learning? Classification or regression? Binary or multiclass?\n",
    "\n",
    "Choose a first metric and justify it. You can consult the [classification metrics documentation](https://scikit-learn.org/stable/modules/model_evaluation.html#classification-metrics)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": [
     "exercise"
    ]
   },
   "outputs": [],
   "source": [
    "# TODO: write your answers as comments, then import your chosen metric.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "tags": [
     "correction"
    ]
   },
   "source": [
    "### Proposed answer\n",
    "\n",
    "This is supervised multiclass classification: every sample has a known class label. Accuracy is a reasonable first metric because the classes are fairly balanced and we have not been given asymmetric error costs. That choice would need revision if some errors were more serious than others."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": [
     "correction"
    ]
   },
   "outputs": [],
   "source": [
    "from sklearn.metrics import accuracy_score\n",
    "\n",
    "print(\"Selected metric: accuracy\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 7. A first k-NN model\n",
    "Find the appropriate estimator in the [nearest-neighbours documentation](https://scikit-learn.org/stable/modules/neighbors.html#nearest-neighbors-classification).\n",
    "\n",
    "1. Instantiate the classifier with its default parameters.\n",
    "2. Fit it on `X, y`.\n",
    "3. Predict the labels of `X` and compute accuracy.\n",
    "4. Read the [`KNeighborsClassifier` documentation](https://scikit-learn.org/stable/modules/generated/sklearn.neighbors.KNeighborsClassifier.html): what is the default value of $k$?\n",
    "5. Which fundamental evaluation principle does this experiment violate?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": [
     "exercise"
    ]
   },
   "outputs": [],
   "source": [
    "from sklearn.neighbors import KNeighborsClassifier\n",
    "\n",
    "# TODO: fit a default k-NN on all the data and evaluate it on the same data.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "tags": [
     "correction"
    ]
   },
   "source": [
    "### Proposed answer\n",
    "\n",
    "The default is $k=5$. The resulting accuracy is a **training score**, not evidence of generalisation: the same samples were used for fitting and evaluation. Even a high value would not tell us how the classifier behaves on unseen wines."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": [
     "correction"
    ]
   },
   "outputs": [],
   "source": [
    "knn_on_all_data = KNeighborsClassifier()\n",
    "knn_on_all_data.fit(X, y)\n",
    "predictions_on_training_data = knn_on_all_data.predict(X)\n",
    "\n",
    "print(\"Parameters:\", knn_on_all_data.get_params())\n",
    "print(\"Training accuracy:\", accuracy_score(y, predictions_on_training_data))\n",
    "# it's ok if students compute accuracy on their own."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 8. A first train/test protocol\n",
    "\n",
    "Use [`train_test_split`](https://scikit-learn.org/stable/modules/generated/sklearn.model_selection.train_test_split.html) to reserve 25% of the samples for testing.\n",
    "\n",
    "Requirements:\n",
    "- Even if not relevant here, you can preserve class proportions with `stratify=y`;\n",
    "- for now, do **not** set `random_state`;\n",
    "- fit the default k-NN using the training set **only**;\n",
    "- report both training and test accuracy.\n",
    "\n",
    "Why is the training score still useful, even though it cannot estimate generalisation on its own?\n",
    "\n",
    "> In this practical, we inspect the test score early for teaching purposes. Consequently, a later score on this same test set must be interpreted as illustrative rather than as a perfectly untouched final estimate."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": [
     "exercise"
    ]
   },
   "outputs": [],
   "source": [
    "from sklearn.model_selection import train_test_split\n",
    "\n",
    "# TODO: create X_train, X_test, y_train and y_test, then evaluate k-NN.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "tags": [
     "correction"
    ]
   },
   "source": [
    "### Proposed answer\n",
    "\n",
    "The test set is not used by `fit`, so its score is a first estimate of performance on unseen samples from the same sampling process. Comparing train and test scores can reveal a generalisation gap, although one split remains uncertain."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": [
     "correction"
    ]
   },
   "outputs": [],
   "source": [
    "X_train, X_test, y_train, y_test = train_test_split(\n",
    "    X, y, test_size=0.25, stratify=y\n",
    ")\n",
    "\n",
    "knn = KNeighborsClassifier()\n",
    "knn.fit(X_train, y_train)\n",
    "\n",
    "print(f\"Training accuracy: {knn.score(X_train, y_train):.3f}\")\n",
    "print(f\"Test accuracy:     {knn.score(X_test, y_test):.3f}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 9. Select the number of neighbours\n",
    "\n",
    "$k$ is a **hyperparameter**: `fit` does not learn it. Testing many values on `X_test` and keeping the best would turn the test set into validation data.\n",
    "\n",
    "Propose a strategy to fit $k$ while avoiding leakage to the test set.\n",
    "\n",
    "What is the best value of $k$ for this validation split?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": [
     "exercise"
    ]
   },
   "outputs": [],
   "source": [
    "# TODO: Fit k !"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": [
     "correction"
    ]
   },
   "outputs": [],
   "source": [
    "# Odd k does not guarantee the absence of ties in multiclass classification,\n",
    "# so we evaluate every integer value in the chosen range.\n",
    "k_values = np.arange(1, 31)\n",
    "print(k_values)\n",
    "\n",
    "# Create a separate validation set from the training data.\n",
    "X_train, X_val, y_train, y_val = train_test_split(\n",
    "    X_train, y_train, test_size=0.25, stratify=y_train\n",
    ")\n",
    "\n",
    "perf_train = []\n",
    "perf_val = []\n",
    "\n",
    "for k in k_values:\n",
    "    knn = KNeighborsClassifier(n_neighbors=k)\n",
    "    knn.fit(X_train, y_train)\n",
    "    perf_train.append(knn.score(X_train, y_train))\n",
    "    perf_val.append(knn.score(X_val, y_val))\n",
    "    print(f\"k={k:2d} | training accuracy: {perf_train[-1]:.3f} | validation accuracy: {perf_val[-1]:.3f}\")  \n",
    "\n",
    "plt.plot(k_values, perf_train, label=\"training accuracy\")\n",
    "plt.plot(k_values, perf_val, label=\"validation accuracy\")\n",
    "plt.xlabel(\"k\")\n",
    "plt.ylabel(\"accuracy\")\n",
    "plt.title(\"k-NN performance on training and validation sets\")\n",
    "plt.legend()\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 10. To scale or not to scale?\n",
    "\n",
    "Return to the boxplot. With the default Euclidean distance, which variables are likely to dominate? Propose a preprocessing step.\n",
    "\n",
    "Read the [`StandardScaler` documentation](https://scikit-learn.org/stable/modules/generated/sklearn.preprocessing.StandardScaler.html). Then answer:\n",
    "\n",
    "1. Which statistics does the scaler learn?\n",
    "2. Why would calling `scaler.fit(X)` before the train/test split leak information?\n",
    "3. Why is scaling particularly relevant for k-NN?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": [
     "exercise"
    ]
   },
   "outputs": [],
   "source": [
    "from sklearn.preprocessing import StandardScaler\n",
    "\n",
    "# TODO: fit a scaler on X_train only and inspect its mean_ and scale_ attributes.\n",
    "# Do not transform X_test yet.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "tags": [
     "correction"
    ]
   },
   "source": [
    "### Proposed answer\n",
    "\n",
    "`StandardScaler` learns one training-set mean and standard deviation per feature. Fitting it before splitting would let test-set values influence preprocessing. Because k-NN uses distances, a large-scale feature can otherwise dominate several informative small-scale features."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": [
     "correction"
    ]
   },
   "outputs": [],
   "source": [
    "scaler = StandardScaler().fit(X_train)\n",
    "print(\"Mean of each feature:\", scaler.mean_)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 11. Prevent leakage with a pipeline\n",
    "\n",
    "A [`Pipeline`](https://scikit-learn.org/stable/modules/generated/sklearn.pipeline.Pipeline.html) chains preprocessing and prediction. When the pipeline is fitted on `X_train`, the scaler learns from `X_train` only; the same transformation is then applied to validation and test samples.\n",
    "\n",
    "Build a pipeline with a scaler and k-NN. Jointly compare:\n",
    "\n",
    "- no scaling versus `StandardScaler`;\n",
    "- $k=1,\\ldots,30$.\n",
    "\n",
    "Select the best configuration on the validation set, refit it on the combined training and validation data, then report its performance on the test set. Remember that this score is illustrative because the test set was already inspected in Section 8."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": [
     "exercise"
    ]
   },
   "outputs": [],
   "source": [
    "from sklearn.pipeline import Pipeline\n",
    "\n",
    "# TODO: build the pipeline and search both scaling choices and all k values."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": [
     "correction"
    ]
   },
   "outputs": [],
   "source": [
    "# Manual search on the validation set\n",
    "best_score = -np.inf\n",
    "best_params = None\n",
    "best_model = None\n",
    "\n",
    "for k in k_values:\n",
    "    for scaling in [None, StandardScaler()]:\n",
    "        if scaling is not None:\n",
    "            model = Pipeline([\n",
    "                (\"scaler\", scaling),\n",
    "                (\"knn\", KNeighborsClassifier(n_neighbors=k))\n",
    "            ])\n",
    "        else:\n",
    "            model = Pipeline([\n",
    "                (\"knn\", KNeighborsClassifier(n_neighbors=k))\n",
    "            ])\n",
    "\n",
    "        model.fit(X_train, y_train)\n",
    "        score = model.score(X_val, y_val)\n",
    "\n",
    "        if score > best_score:\n",
    "            best_model = model\n",
    "            best_score = score\n",
    "            best_params = (k, scaling)\n",
    "\n",
    "print(\"Best configuration:\", best_params)\n",
    "print(f\"Validation accuracy: {best_score:.3f}\")\n",
    "\n",
    "\n",
    "# Retrain on X_train + X_val. There are always questions on the need to retrain on the full training set, so let's do it here.\n",
    "# we can argue that we will use more data, but are the best hyperparameters still adapted ? \n",
    "X_train_final = np.concatenate([X_train, X_val])\n",
    "y_train_final = np.concatenate([y_train, y_val])\n",
    "\n",
    "best_model.fit(X_train_final, y_train_final)\n",
    "\n",
    "print(f\"Test accuracy: {best_model.score(X_test, y_test):.3f}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 12. Reproducibility\n",
    "\n",
    "Restart the kernel and run the train/test section several times. Are the scores identical? Locate the source of randomness.\n",
    "\n",
    "Modify the protocol so that another person running the notebook obtains the same split.\n",
    "\n",
    "Does setting a seed prove that the conclusion is robust to the choice of split?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": [
     "exercise"
    ]
   },
   "outputs": [],
   "source": [
    "# TODO: recreate both random splits using random_state.\n",
    "# Check that rerunning the cells produces the same partitions and scores.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "tags": [
     "correction"
    ]
   },
   "source": [
    "### Proposed answer\n",
    "\n",
    "Set `random_state` to a fixed value in every stochastic splitting operation. This makes the experiment repeatable, but it does not prove that the conclusion is insensitive to the particular split."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": [
     "correction"
    ]
   },
   "outputs": [],
   "source": [
    "RANDOM_STATE = 42\n",
    "\n",
    "X_train_repro, X_test_repro, y_train_repro, y_test_repro = train_test_split(\n",
    "    X, y, test_size=0.25, stratify=y, random_state=RANDOM_STATE\n",
    ")\n",
    "X_train_repro, X_val_repro, y_train_repro, y_val_repro = train_test_split(\n",
    "    X_train_repro,\n",
    "    y_train_repro,\n",
    "    test_size=0.25,\n",
    "    stratify=y_train_repro,\n",
    "    random_state=RANDOM_STATE,\n",
    ")\n",
    "\n",
    "print(\"Train / validation / test sizes:\",\n",
    "      len(X_train_repro), len(X_val_repro), len(X_test_repro))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 13. Conclusions\n",
    "\n",
    "Complete the following statements in your own words:\n",
    "\n",
    "1. Evaluating a model on its training data is misleading because ...\n",
    "2. The test set must remain outside model selection because ...\n",
    "3. Scaling matters for k-NN because ...\n",
    "4. A pipeline prevents leakage by ...\n",
    "5. A fixed seed provides reproducibility, but not ...\n",
    "\n",
    "### Optional extension\n",
    "\n",
    "Compare your final model with a [`DummyClassifier`](https://scikit-learn.org/stable/modules/generated/sklearn.dummy.DummyClassifier.html). A useful model should improve on a simple baseline under the same evaluation protocol."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python (cours-ml)",
   "language": "python",
   "name": "cours-ml"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.12.13"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}