{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Practical 1 — A first machine-learning workflow\n",
    "\n",
    "By the end of this practical, you should be able to:\n",
    "\n",
    "- inspect a dataset represented by NumPy arrays;\n",
    "- identify samples, features and targets;\n",
    "- use the scikit-learn `fit` / `predict` / `score` interface;\n",
    "- separate training and test data;\n",
    "- tune $k$ without using the test set for model selection;\n",
    "- explain why feature scaling matters for a distance-based method;\n",
    "- prevent preprocessing leakage with a `Pipeline`;\n",
    "- make a simple experiment reproducible.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "import sklearn\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1. Working environment\n",
    "\n",
    "Before starting Jupyter, activate the course environment in a terminal:\n",
    "\n",
    "```bash\n",
    "source /opt/venv/iti-iml/bin/activate\n",
    "python -m ipykernel install --user --name cours-ml --display-name \"Python (cours-ml)\"\n",
    "```\n",
    "\n",
    "Then select the **Python (cours-ml)** kernel in Jupyter. The kernel installation is normally required only once.\n",
    "\n",
    "Useful references: [installing scikit-learn](https://scikit-learn.org/stable/install.html) and the [scikit-learn user guide](https://scikit-learn.org/stable/user_guide.html)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import sys\n",
    "# Display the python used by the jupyter kernel \n",
    "print(\"Python used by the jupyter kernel : \", sys.executable)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2. Scikit-learn 101\n",
    "\n",
    "NumPy represents numerical data using multidimensional arrays. In this practical:\n",
    "\n",
    "- `X` will be a two-dimensional array with shape `(n_samples, n_features)`;\n",
    "- `y` will be a one-dimensional array containing one target per sample.\n",
    "\n",
    "Most supervised scikit-learn estimators share the same interface:\n",
    "\n",
    "1. create an estimator, for example `model = SomeClassifier(...)`;\n",
    "2. learn from examples with `model.fit(X, y)`;\n",
    "3. predict with `model.predict(X)`;\n",
    "4. evaluate predictions with a suitable metric.\n",
    "\n",
    "Scikit-learn also provides some built-in datasets. Browse the available [toy datasets](https://scikit-learn.org/stable/datasets/toy_dataset.html) before continuing."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": [
     "exercise"
    ]
   },
   "outputs": [],
   "source": [
    "# NumPy reminder: predict the shape and value of each expression before running it.\n",
    "example = np.array([[1.0, 10.0, 100.0], [2.0, 20.0, 200.0]])\n",
    "\n",
    "print(\"shape:\", example.shape)\n",
    "print(\"first sample:\", example[0])\n",
    "print(\"second feature:\", example[:, 1])\n",
    "print(\"feature means:\", example.mean(axis=0))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3. Load the Wine dataset\n",
    "\n",
    "Read the [`load_wine` documentation](https://scikit-learn.org/stable/modules/generated/sklearn.datasets.load_wine.html).\n",
    "\n",
    "Load the feature matrix into `X` and the target vector into `y` using `return_X_y=True`. Also load the complete `Bunch` object into `wine`, because it contains feature names, class names and a description."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": [
     "exercise"
    ]
   },
   "outputs": [],
   "source": [
    "from sklearn.datasets import load_wine\n",
    "\n",
    "# TODO: create wine, X and y.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 4. Understand the data\n",
    "\n",
    "Answer the following questions using array attributes and the dataset metadata.\n",
    "\n",
    "1. How many samples and features are there?\n",
    "2. What does one row represent?\n",
    "3. What are the feature names?\n",
    "4. What does `y` represent, and which values can it take?\n",
    "5. Are the classes balanced?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": [
     "exercise"
    ]
   },
   "outputs": [],
   "source": [
    "# TODO: inspect X.shape, y.shape, wine.feature_names, wine.target_names\n",
    "# and count the samples in each class with np.unique(..., return_counts=True).\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 5. Visualise feature scales\n",
    "\n",
    "Create a boxplot with one box per feature.\n",
    "\n",
    "Before modelling, note whether all variables have comparable numerical ranges. A boxplot is useful here for comparing scales, but it is not a complete exploration of the dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": [
     "exercise"
    ]
   },
   "outputs": [],
   "source": [
    "# TODO: display a boxplot using Matplotlib and identify each feature.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 6. Identify the learning problem and metric\n",
    "\n",
    "Is this supervised or unsupervised learning? Classification or regression? Binary or multiclass?\n",
    "\n",
    "Choose a first metric and justify it. You can consult the [classification metrics documentation](https://scikit-learn.org/stable/modules/model_evaluation.html#classification-metrics)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": [
     "exercise"
    ]
   },
   "outputs": [],
   "source": [
    "# TODO: write your answers as comments, then import your chosen metric.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 7. A first k-NN model\n",
    "Find the appropriate estimator in the [nearest-neighbours documentation](https://scikit-learn.org/stable/modules/neighbors.html#nearest-neighbors-classification).\n",
    "\n",
    "1. Instantiate the classifier with its default parameters.\n",
    "2. Fit it on `X, y`.\n",
    "3. Predict the labels of `X` and compute accuracy.\n",
    "4. Read the [`KNeighborsClassifier` documentation](https://scikit-learn.org/stable/modules/generated/sklearn.neighbors.KNeighborsClassifier.html): what is the default value of $k$?\n",
    "5. Which fundamental evaluation principle does this experiment violate?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": [
     "exercise"
    ]
   },
   "outputs": [],
   "source": [
    "from sklearn.neighbors import KNeighborsClassifier\n",
    "\n",
    "# TODO: fit a default k-NN on all the data and evaluate it on the same data.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 8. A first train/test protocol\n",
    "\n",
    "Use [`train_test_split`](https://scikit-learn.org/stable/modules/generated/sklearn.model_selection.train_test_split.html) to reserve 25% of the samples for testing.\n",
    "\n",
    "Requirements:\n",
    "- Even if not relevant here, you can preserve class proportions with `stratify=y`;\n",
    "- for now, do **not** set `random_state`;\n",
    "- fit the default k-NN using the training set **only**;\n",
    "- report both training and test accuracy.\n",
    "\n",
    "Why is the training score still useful, even though it cannot estimate generalisation on its own?\n",
    "\n",
    "> In this practical, we inspect the test score early for teaching purposes. Consequently, a later score on this same test set must be interpreted as illustrative rather than as a perfectly untouched final estimate."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": [
     "exercise"
    ]
   },
   "outputs": [],
   "source": [
    "from sklearn.model_selection import train_test_split\n",
    "\n",
    "# TODO: create X_train, X_test, y_train and y_test, then evaluate k-NN.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 9. Select the number of neighbours\n",
    "\n",
    "$k$ is a **hyperparameter**: `fit` does not learn it. Testing many values on `X_test` and keeping the best would turn the test set into validation data.\n",
    "\n",
    "Propose a strategy to fit $k$ while avoiding leakage to the test set.\n",
    "\n",
    "What is the best value of $k$ for this validation split?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": [
     "exercise"
    ]
   },
   "outputs": [],
   "source": [
    "# TODO: Fit k !"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 10. To scale or not to scale?\n",
    "\n",
    "Return to the boxplot. With the default Euclidean distance, which variables are likely to dominate? Propose a preprocessing step.\n",
    "\n",
    "Read the [`StandardScaler` documentation](https://scikit-learn.org/stable/modules/generated/sklearn.preprocessing.StandardScaler.html). Then answer:\n",
    "\n",
    "1. Which statistics does the scaler learn?\n",
    "2. Why would calling `scaler.fit(X)` before the train/test split leak information?\n",
    "3. Why is scaling particularly relevant for k-NN?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": [
     "exercise"
    ]
   },
   "outputs": [],
   "source": [
    "from sklearn.preprocessing import StandardScaler\n",
    "\n",
    "# TODO: fit a scaler on X_train only and inspect its mean_ and scale_ attributes.\n",
    "# Do not transform X_test yet.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 11. Prevent leakage with a pipeline\n",
    "\n",
    "A [`Pipeline`](https://scikit-learn.org/stable/modules/generated/sklearn.pipeline.Pipeline.html) chains preprocessing and prediction. When the pipeline is fitted on `X_train`, the scaler learns from `X_train` only; the same transformation is then applied to validation and test samples.\n",
    "\n",
    "Build a pipeline with a scaler and k-NN. Jointly compare:\n",
    "\n",
    "- no scaling versus `StandardScaler`;\n",
    "- $k=1,\\ldots,30$.\n",
    "\n",
    "Select the best configuration on the validation set, refit it on the combined training and validation data, then report its performance on the test set. Remember that this score is illustrative because the test set was already inspected in Section 8."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": [
     "exercise"
    ]
   },
   "outputs": [],
   "source": [
    "from sklearn.pipeline import Pipeline\n",
    "\n",
    "# TODO: build the pipeline and search both scaling choices and all k values."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 12. Reproducibility\n",
    "\n",
    "Restart the kernel and run the train/test section several times. Are the scores identical? Locate the source of randomness.\n",
    "\n",
    "Modify the protocol so that another person running the notebook obtains the same split.\n",
    "\n",
    "Does setting a seed prove that the conclusion is robust to the choice of split?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": [
     "exercise"
    ]
   },
   "outputs": [],
   "source": [
    "# TODO: recreate both random splits using random_state.\n",
    "# Check that rerunning the cells produces the same partitions and scores.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 13. Conclusions\n",
    "\n",
    "Complete the following statements in your own words:\n",
    "\n",
    "1. Evaluating a model on its training data is misleading because ...\n",
    "2. The test set must remain outside model selection because ...\n",
    "3. Scaling matters for k-NN because ...\n",
    "4. A pipeline prevents leakage by ...\n",
    "5. A fixed seed provides reproducibility, but not ...\n",
    "\n",
    "### Optional extension\n",
    "\n",
    "Compare your final model with a [`DummyClassifier`](https://scikit-learn.org/stable/modules/generated/sklearn.dummy.DummyClassifier.html). A useful model should improve on a simple baseline under the same evaluation protocol."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python (cours-ml)",
   "language": "python",
   "name": "cours-ml"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.12.13"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}