{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "# TP Classifieur et Validation Croisée\n", "\n", "L'objectif de ce travail pratique est de manipuler la bibliothèque `scikit-learn` et d’implémenter un classifieur avec une stratégie de validation croisée pour optimiser les hyperparamètres. Veillez à ne pas introduire de biais lors de l'ajustement de vos (hyper)paramètres." ] }, { "cell_type": "markdown", "metadata": { "id": "ITxmD4v0K7tt" }, "source": [ "## Charger des données\n", "\n", "Nous allons d'abord charger des données dans notre environnement pour réaliser une classification. `scikit-learn` inclut des utilitaires de chargement de jeux de données dans le module `sklearn.datasets` [lien](https://scikit-learn.org/stable/modules/classes.html#module-sklearn.datasets). Prenez quelques minutes pour parcourir la liste des jeux de données et identifier les tâches de classification.\n", "\n", "Pour ce TP, nous allons utiliser le jeu de données \"wine\" [lien](https://scikit-learn.org/stable/datasets/toy_dataset.html#wine-dataset)" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "from sklearn.datasets import load_wine\n", "X,y = ..." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "ITxmD4v0K7tt", "tags": [ "correction" ] }, "outputs": [], "source": [ "from sklearn.datasets import load_wine\n", "X,y = load_wine(return_X_y=True)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Un premier classifieur SVM\n", "\n", "`sklearn` propose des implémentations de différents algorithmes d’apprentissage. Comme vu en cours, nous allons nous concentrer sur les problèmes de classification. Nous allons commencer par les SVM.\n", "\n", "Cette méthode est implémentée par la classe `SVC` du module `sklearn.svm`. Consultez la documentation pour vous familiariser et vérifier qu’elle correspond bien à la méthode de SVM vue en cours [lien](https://scikit-learn.org/stable/modules/generated/sklearn.svm.SVC.html).\n", "\n", "1) Créez une variable `model` correspondant à un `SVC`." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "from sklearn.svm import SVC\n", "model = ..." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "colab": { "base_uri": "https://localhost:8080/", "height": 486 }, "id": "4-YyiHtbIdLH", "outputId": "6189d98c-5dc4-4e18-aa39-d4f142807146", "tags": [ "correction" ] }, "outputs": [], "source": [ "from sklearn.svm import SVC\n", "model = SVC()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "2) Entraînez le modèle sur le jeu de données `X` et prédisez les propriétés correspondantes." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [] }, { "cell_type": "code", "execution_count": null, "metadata": { "colab": { "base_uri": "https://localhost:8080/", "height": 486 }, "id": "4-YyiHtbIdLH", "outputId": "6189d98c-5dc4-4e18-aa39-d4f142807146", "tags": [ "correction" ] }, "outputs": [], "source": [ "model.fit(X,y)\n", "pred = model.predict(X)\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "3) Évaluez la précision de votre prédiction. Vous pouvez utiliser la fonction `accuracy_score` du module `sklearn.metrics` [lien](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.accuracy_score.html)." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "from sklearn.metrics import accuracy_score" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "tags": [ "correction" ] }, "outputs": [], "source": [ "from sklearn.metrics import accuracy_score\n", "print(accuracy_score(y,pred))" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "tags": [ "correction" ] }, "outputs": [], "source": [ "# accuracy score est simplement la moyenne des bonnes classifs.\n", "import numpy as np\n", "acc = np.mean(y==pred)\n", "print(acc)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Prédire des nouvelles données\n", "\n", "Félicitations, vous avez appris un premier modèle. Toutefois, il est peu utile car il prédit uniquement sur les données d'entraînement. Prédire sur des données jamais vues est plus pertinent.\n", "\n", "1) Séparez votre jeu de données en un ensemble d'entraînement et un ensemble de test à l'aide de la fonction `train_test_split` du module `sklearn.model_selection`. Utilisez une `test_size` de 50% du jeu total, et fixez `random_state` à `42` pour la reproductibilité. Documentation [ici](https://scikit-learn.org/stable/modules/generated/sklearn.model_selection.train_test_split.html)\n", "2) Entraînez votre modèle sur l’ensemble d’entraînement\n", "3) Prédisez les valeurs de l’ensemble de test et affichez les performances" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "from sklearn.model_selection import train_test_split\n", "X_train, X_test, y_train, y_test = ...\n", "model = ...\n", "model.fit(...)\n", "perf_test = ...\n", "print(f\"Performance on test is {perf_test:.2f}\")" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "Wk9pCSXvI9m3", "tags": [ "correction" ] }, "outputs": [], "source": [ "from sklearn.model_selection import train_test_split\n", "X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.5, random_state=42)" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "Wk9pCSXvI9m3", "tags": [ "correction" ] }, "outputs": [], "source": [ "model = SVC()\n", "model.fit(X_train,y_train)\n", "pred = model.predict(X_test)\n", "perf_test = accuracy_score(y_test,pred)\n", "print(f\"Performance on test is {perf_test:.2f}\") # perf différente que sur le train" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Ajuster l’hyperparamètre `C`\n", "\n", "Lors de la création d’un nouvel objet `SVC`, on peut ajuster la valeur de `C` qui représente l’importance des erreurs dans le processus d’optimisation. L’objectif est de trouver la meilleure valeur de `C`, c’est-à-dire celle qui fonctionne le mieux sur des données non vues.\n", "\n", "1) Calculez la performance (accuracy) pour différentes valeurs de `C` sur les ensembles d’entraînement et de test. On utilisera 25% du jeu de données pour l'entraînement et une échelle logarithmique (`np.logscale`) pour les valeurs de `C`.\n", "\n", " a) Divisez le jeu en entraînement et test (25% pour l'entraînement)\n", "\n", " b) Normalisez les données avec la classe `StandardScaler` du module `sklearn.preprocessing`. Attention à ne pas utiliser l'ensemble de test pour calculer les paramètres !\n", "\n", " c) Calculez le taux d'erreur de classification pour chaque `C` dans `np.logspace(-2,1,25)`" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "from sklearn.svm import SVC\n", "import numpy as np\n", "from sklearn.model_selection import train_test_split\n", "import matplotlib.pyplot as plt\n", "from sklearn.metrics import accuracy_score as score\n", "from sklearn.preprocessing import StandardScaler\n", "\n", "\n", "X_train, X_test, y_train, y_test = ..." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "colab": { "base_uri": "https://localhost:8080/", "height": 304 }, "id": "v91XlXrwK7tu", "outputId": "98e9c779-9054-4bce-ebba-72bd2b45d398", "tags": [ "correction" ] }, "outputs": [], "source": [ "from sklearn.svm import SVC\n", "import numpy as np\n", "from sklearn.model_selection import train_test_split\n", "import matplotlib.pyplot as plt\n", "from sklearn.metrics import accuracy_score as score\n", "from sklearn.preprocessing import StandardScaler\n", "\n", "\n", "X_train, X_test, y_train, y_test = train_test_split(X, y, train_size=0.25, random_state=42)\n", "\n", "scaler = StandardScaler()\n", "X_train_norm = scaler.fit_transform(X_train)\n", "X_test_norm = scaler.transform(X_test)\n", "\n", "error_train = []\n", "error_test = []\n", "Cs = np.logspace(-2,1,25) # leur faire printer pour voir la tête du vecteur\n", "for C in Cs:\n", " model = SVC(C=C)\n", "\n", " model.fit(X_train_norm,y_train)\n", " \n", " y_hat_train = model.predict(X_train_norm)\n", " y_hat_test = model.predict(X_test_norm)\n", " \n", " error_train.append(1-score(y_train, y_hat_train)) # on veut des erreurs pour le plot : erreur = 1 - perf\n", " error_test.append(1-score(y_test, y_hat_test))\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "2) Tracez les erreurs d'entraînement et de test en fonction de la complexité du modèle. Rappel : de petites valeurs de `C` correspondent à des modèles simples ; de grandes valeurs à des modèles plus complexes avec moins d'erreurs. Vérifiez que l’erreur d'entraînement diminue avec la complexité. Commentez l’erreur de test. Quelle est la meilleure valeur de `C` ?\n", "\n", "**Indice** : utilisez la fonction `plot` du module `matplotlib.pyplot`" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "import matplotlib.pyplot as plt\n", "plt.plot(...)\n", "plt.xscale('log') # Recommandé pour afficher de manière lisible le plot" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "colab": { "base_uri": "https://localhost:8080/", "height": 304 }, "id": "v91XlXrwK7tu", "outputId": "98e9c779-9054-4bce-ebba-72bd2b45d398", "tags": [ "correction" ] }, "outputs": [], "source": [ "# Ici, on ne demande pas un plot annoté, mais juste les courbes et leurs interprétations.\n", "# Attention, j'ai choisi le random state et le ratio du split pour que ce soit parlant\n", "fig,ax = plt.subplots()\n", "ax.plot(Cs,error_train)\n", "ax.plot(Cs,error_test)\n", "ax.set_xscale('log') # ou plt.xscale('log') mais le ax.set_xscale est + propre\n", "\n", "\n", "arrow_style = { 'arrowstyle' : \"->\", 'connectionstyle' : \"arc3\"}\n", "\n", "ax.legend([f\" error on train\", f\" error on test\"]);\n", "ax.annotate(\"sous apprentissage\",xy=(0.05,0.64),xytext=(0.01, 0.4),\n", " arrowprops=arrow_style)\n", "ax.annotate(\"sur apprentissage\",xy=(7.0,0.0),xytext=(1,0.4),\n", " arrowprops=arrow_style)\n", "\n", "best_idx = np.argmin(error_test)\n", "ax.annotate(\"Optimal\",xy=(Cs[best_idx],error_test[best_idx]),\n", " xytext=(.5,0.2),\n", " arrowprops=arrow_style)\n", "\n", "ax.set_ylabel(\"Erreur\")\n", "ax.set_xlabel(\"Complexité du modèle\")\n", "\n", "print(f\"The best C value is {Cs[best_idx]:2f}\")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Protocole complet\n", "\n", "Dans l’exercice précédent, nous avons choisi la meilleure valeur de `C` a posteriori, en fonction de sa performance sur l’ensemble de test. En conditions réelles, nous ne connaissons pas la performance sur des données non vues. Pour limiter ce biais, on utilise des ensembles d'entraînement, validation et test.\n", "\n", "1) Divisez le jeu de données original en un ensemble d'entraînement et un ensemble de test (30% pour le test)\n", "\n", "2) Utilisez la classe `GridSearchCV` pour faire une validation croisée et trouver la meilleure valeur de `C`." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "from sklearn.svm import SVC\n", "import numpy as np\n", "from sklearn.model_selection import train_test_split\n", "import matplotlib.pyplot as plt\n", "from sklearn.model_selection import GridSearchCV\n", "\n" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "colab": { "base_uri": "https://localhost:8080/" }, "id": "NriTBab2K7tx", "outputId": "97d73f21-f805-49c2-967a-8384e97b6283", "tags": [ "correction" ] }, "outputs": [], "source": [ "from sklearn.svm import SVC\n", "import numpy as np\n", "from sklearn.model_selection import train_test_split\n", "import matplotlib.pyplot as plt\n", "from sklearn.model_selection import GridSearchCV\n", "\n", "scaler = StandardScaler()\n", "X_norm = scaler.fit_transform(X)\n", "\n", "X_train_norm, X_test_norm, y_train, y_test = train_test_split(X_norm, y, test_size=0.3, random_state=42)\n", "\n", "params={'C' : np.logspace(-2,2,25)}\n", "model=SVC()\n", "cv = GridSearchCV(model,param_grid=params)\n", "\n", "cv.fit(X_train_norm,y_train)\n", "cv.cv_results_ # a explorer par les étudiants" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "tags": [ "correction" ] }, "outputs": [], "source": [ "cv.best_estimator_" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "3) Prédisez la performance finale sur l’ensemble de test." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "\n", "perf_test = ...\n", "print(f\"Accuracy on test = {perf_test}\")" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "colab": { "base_uri": "https://localhost:8080/" }, "id": "NriTBab2K7tx", "outputId": "97d73f21-f805-49c2-967a-8384e97b6283", "tags": [ "correction" ] }, "outputs": [], "source": [ "y_hat_test = cv.predict(X_test_norm)\n", "perf_test = cv.score(X_test_norm,y_test)\n", "print(f\"Accuracy on test = {perf_test}\")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "4. Nous avons commis une erreur dans notre protocole. Nous avons enfreint la première règle : ne jamais utiliser l’ensemble de test pendant l’apprentissage. Pouvez-vous identifier le problème ?\n", "\n", "Pour le résoudre, consultez la classe `Pipeline` de scikit-learn : https://scikit-learn.org/stable/modules/generated/sklearn.pipeline.Pipeline.html" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [] }, { "cell_type": "code", "execution_count": null, "metadata": { "tags": [ "correction" ] }, "outputs": [], "source": [ "# We used part of test set to normalize our data. " ] }, { "cell_type": "code", "execution_count": null, "metadata": { "tags": [ "correction" ] }, "outputs": [], "source": [ "from sklearn.svm import SVC\n", "from sklearn.preprocessing import StandardScaler\n", "from sklearn.model_selection import train_test_split, GridSearchCV\n", "from sklearn.pipeline import Pipeline\n", "\n", "X_train, X_test, y_train, y_test = train_test_split(X, y,test_size=0.3,random_state=42)\n", "pipe = Pipeline(steps=[('scaler', StandardScaler()), ('classifier', SVC())])\n", "params={'classifier__C' : np.logspace(-2,2,25)}\n", "cv = GridSearchCV(pipe,param_grid=params)\n", "cv.fit(X_train, y_train)\n", "cv.score(X_test, y_test)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "5. **Bonus 1** Étendez le processus d’apprentissage pour ajuster le meilleur noyau" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [] }, { "cell_type": "code", "execution_count": null, "metadata": { "tags": [ "correction" ] }, "outputs": [], "source": [ "from sklearn.svm import SVC\n", "from sklearn.preprocessing import StandardScaler\n", "from sklearn.model_selection import train_test_split, GridSearchCV\n", "from sklearn.pipeline import Pipeline\n", "\n", "X_train, X_test, y_train, y_test = train_test_split(X, y,test_size=0.3,random_state=42)\n", "kernels= ['linear','poly','rbf','sigmoid'] # on pourra meme fitter chaque hyperparameter de chaque kernel (bonus)\n", "\n", "C_values = np.logspace(-2,2,25)\n", "# attention aux temps de calculs !\n", "pipe = Pipeline(steps=[('scaler', StandardScaler()), ('classifier', SVC())])\n", "params={'classifier__C' : C_values,\n", " 'classifier__kernel' : kernels,\n", " 'classifier__gamma' : ['scale','auto', 1e-5,1e-3,1e-1]\n", " }\n", "cv = GridSearchCV(pipe,param_grid=params)\n", "cv.fit(X_train, y_train)\n", "cv.score(X_test, y_test)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "7. **Bonus 2** Étendez le processus d’apprentissage pour comparer différents classifieurs" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [] }, { "cell_type": "code", "execution_count": null, "metadata": { "tags": [ "correction" ] }, "outputs": [], "source": [ "# long. Pour occuper les plus rapides" ] } ], "metadata": { "celltoolbar": "Tags", "colab": { "name": "Cross validation et Ridge", "provenance": [] }, "kernelspec": { "display_name": "venvi4-UFRmBqUw-py3.10", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.10.12" } }, "nbformat": 4, "nbformat_minor": 1 }