{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "# Deep Neural Networks on MNIST\n", "\n", "Let's build a 5 layer MLP (with resp. 256, 128, 64, 28 hidden units) and fit it on MNIST.\n", "\n", "1) How good is it in from a shallow 2 layer network ?\n", "\n", "2) Where can we track vanishing gradient on Tensorboard ?" ] }, { "cell_type": "code", "execution_count": 1, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Data File found... loading it into memory\n", "Data File loaded\n" ] }, { "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAAAVEAAAD8CAYAAADOg5fGAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4zLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvnQurowAAIABJREFUeJzt3Xu8VXP+x/HXR0hUKkySS7lNQu7E9EgzyiWXXEY0hYyRh3seGA1+hnELMx6Tu5BCj4mZqBhNGiL3KaZ50IViRJQQSqiJ7++Ps79rr13nsvdZe6+11zrv5+PR46z9XWuf9Tn7c86371rrezHnHCIi0jjrJR2AiEiaqRIVEYlAlaiISASqREVEIlAlKiISgSpREZEIVImKiEQQqRI1s8PN7B0zW2Bmw8oVlCRLec0u5bb8rLGd7c2sGfAu0AdYBMwABjjn5pQvPImb8ppdym1lrB/hvfsDC5xz7wOY2TigH1BnQsysqQ+P+tw5t0XSQTRAeS1dGvIKJeZWeS0ur1Eu5zsCH4VeL8qVSd0WJh1AEZTX0qUhr6DclqqovEZpiRbFzIYAQyp9HomX8ppNymvpolSiHwPbhF5vnSsr4JwbCYwEXR6khPKaXQ3mVnktXZTL+RnATmbW2cw2BE4GJpUnLEmQ8ppdym0FNLol6pxbY2bnAVOAZsAo59zsskUmiVBes0u5rYxGd3Fq1Ml0efCGc27fpIMoN+VVec2oovKqEUsiIhGoEhURiaDiXZxE4rLPPvsE2+eddx4Ap556KgAPPfQQALfffntwzJtvvhljdJJVaomKiESQ2QdLzZo1C7Y33XTTOo/zLZaNN94YgJ/+9KcAnHvuucExf/zjHwEYMGAAAN9//32wb/jw4QBcc801xYSlBxAVsOeeewLw3HPPBWWtW7eu9divv/462N5ss83KFYLyWkUOOeQQAMaOHQvAwQcfHOx75513SvlWerAkIlJpqkRFRCJI5YOlbbfdNtjecMMNATjooIMA6NGjBwBt2rQJjjnhhBOK/t6LFi0C4LbbbgvKjjvuOABWrFgBwH/+859g3wsvvFBS7FI++++/PwDjx48HCm/b+NtUPmerV68GCi/hu3fvDuQfMPljpHg9e/YECj/XJ554IqlwANhvv/0AmDFjRiznU0tURCSCVLVEa3uAUN9Do1L8+OOPAFx55ZUAfPPNN8E+f4N68eLFAHz55ZfBvhJvVEsj+Qd/e++9d1D2yCOPANChQ4c63zd//nwAbr75ZgDGjRsX7Hv55ZeBfM5vvPHGMkbcNPTq1QuAnXbaKShLoiW63nr59mDnzp0B2G677QAws8qeu6LfXUQk41LVEv3www8B+OKLL4KyUlqir7/+OgBfffVVUPbzn/8cyN8Pe/jhhyPHKeV37733AvluZsXyLdeWLVsChfewfSuqW7duZYiwafKDGV599dVE4whfjZx55plA/kpl3rx5FT23WqIiIhGoEhURiaDBy3kzGwUcBSx1zu2WK2sHPAp0Aj4A+jvnvqzre5TLsmXLALj00kuDsqOOOgqAf//730Bh1yRv1qxZAPTp0weAlStXBvt23XVXAC688MIKRFy9qimv9fHj4Y888kig9ocE/hL9ySefDMr8KLNPPvkEyP9+hB8K/uIXv6jze6ZZnLkNP9BJ0v33379OmX+oWGnFfAKjgcPXKhsGPOuc2wl4Nvda0mU0ymtWjUa5jU2DLVHn3HQz67RWcT+gV257DPA8cFkZ46rXhAkTgm3f3cl3qt5jjz0AOOOMM4JjfKsk3AL1Zs+umdh7yJCmtTZXNeY1zHdnmzp1KpAfCx+e62Hy5MlA/mFTeIy077bkWyifffYZUDhQwndr863ccPepNM/wFEdu/cO49u3bN/ZblFVtD5j9706lNfbpfHvn3OLc9hKgzk9SqwemivKaXUXlVnktXeQuTs45V99sL5VePXD58uUFr8Oz9Hi+y8Ojjz4K5FsgUrck8rrzzjsH2/6+t29hfP7550B+wAPAmDFjgPzAiL///e/BvvB2Q1q0aAHAxRdfHJQNHDiwpNjTpL7cFpvXvn37AvnPLim+Jew72Id9/PE6i9RWRGPvCn9qZh0Acl+Xli8kSZDyml3KbYU0tiU6CTgNGJ77OrFsEUV09dVXA4WznPt7Zb179wbgmWeeiT2ulEgkr82bNwfy964h39Lx97p9p+6ZM2cGx5S7FRSe2CaDyppbP++u558txM3/zoTvzb777rtA/nen0hpsiZrZX4BXgZ+a2SIzO4OaRPQxs/lA79xrSRHlNbuU23gV83S+rnF2h5Q5FomR8ppdym28UjV2vhi+G5N/mAT57ir33XcfANOmTQv2+cvDO++8EyjsQiPx2GuvvYD8JXxYv379AM3bWu0qOXdneKmXww+v6f46aNAgAA499NB1jr/22muBwjkyKqk6hhuIiKRU5lqi3nvvvRdsDx48GIAHH3wQgFNOOSXY57c32WQTIL+0brgrjVTWrbfeChQOv/Qtz0q2QP2QRXV5i65du3ZFHecHw/hc+4e9W2+9dXCMX63CdzMLDy397rvvgPyMbKtWrQJg/fXzVdkbb7xR+g8QgVqiIiIRZLYlGuZn2vYTEviWD+SXV73hhhuA/GzY119/fXBMXJ12mxo/eYwf4hm+Hz1p0qSKn9+3QP15/UQ10jDfIvSf3T333BPsu/zyy+t8nx8u6luia9asAeDbb78NjpkzZw4Ao0aNAgq7tfkrk08//RTIr4kW7u5W6flD16aWqIhIBKpERUQiaBKX897bb78NQP/+/YOyo48+Gsg/dDrrrLOAwoW3/DykUl7+Esw/SFi6ND8S0c9zUC5+VJQf0RbmZwL73e9+V9ZzZtk555wDwMKFC4H8kuUN8Uv8+JnY5s6dC8Brr71W0vn9rGtbbLEFAO+//35J7y8ntURFRCJoUi1RL9wJ1y9M5+ed9F0levbsGRzjFzR7/vnn4wmwifLdVaB8Xcx8C9TPLxpeFcE/lPjTn/4EFC6TLcW56aabEjmvfyDsjR8/PpE4QC1REZFImlRL1Hev+OUvfxmU7bfffkBhZ13Id7MAmD59egzRSTm7NfluU77ledJJJwEwcWJ+8qITTjihbOeTZPlujElQS1REJILMtkTD8x2ed955ABx//PEAbLnllnW+74cffgAK78lpWGBl+A7X/uuxxx4b7GvM6qsXXXRRsP1///d/QH5m/LFjxwL5eUlFyqWY+US3MbNpZjbHzGab2YW58nZmNtXM5ue+tq18uFIuyms2Ka/xK+Zyfg1wsXOuK9AdONfMuqIlWNNOec0m5TVmxUzKvBhYnNteYWZzgY5U0fK6kL9E98vn+kt4gE6dOjX4fj8+14+Zj2PsdpKqIa9+3LX/Gr7NcttttwH58dNffPEFAN27dw+O8TNw+ZmBwjMB+U7dU6ZMAeCuu+4q/w9Qhaohr3Hyt4LCixyW2nE/qpLuiebWst4LeB0twZoZyms2Ka/xKLoSNbOWwHhgqHNueXjux3IswVqK8KJUXbt2BeCOO+4AoEuXLg2+389FCHDLLbcA+a4vTe0hUjXltVmzZsG2H1bouyH5pbHDw3HX9sorrwTbfvWCq666qhyhpU415bWS/FVMeM7RuBV1ZjPbgJqEjHXOPZ4r1hKsKae8ZpPyGq8GW6JW81/YA8Bc59ytoV2xLa/rZ82+9957gXxHaoDtt9++wff7Foof3ufvk0F+XsSmphry+uqrrwL59Xn8wIcwf580fPXh+fuk48aNAxrXLSprqiGvSTjwwAOD7dGjR8d67mIu538GnAK8ZWZ+1trLqUnGY7nlWBcC/et4v1Qn5TWblNeYFfN0/iXA6titJVhTSnnNJuU1flU3YumAAw4ACmfb2X///QHo2LFjg+/3ywz4LjKQX/rDL6cs1cHPouRHkvm5XCE/69LaRowYEWzffffdACxYsKBSIUqVCz8wS4rGzouIRFB1LdHjjjuu4GttwjMsPfXUU0B+wSv/8Cg8Z6hUNz9PQXjW+dpmoBfxJk+eDMCJJ56YcCRqiYqIRGLhZWorfrIUdN6tsDecc/smHUS5Ka/Ka0YVlVe1REVEIlAlKiISgSpREZEIVImKiESgSlREJAJVoiIiEcTd2f5zYGXua9psTvS4tytHIFVIec0m5bUIsfYTBTCzmWnsU5fWuOOS1s8nrXHHJa2fT5xx63JeRCQCVaIiIhEkUYmOTOCc5ZDWuOOS1s8nrXHHJa2fT2xxx35PVEQkS3Q5LyISgSpREZEIYqtEzexwM3vHzBaY2bC4zlsqM9vGzKaZ2Rwzm21mF+bK25nZVDObn/vaNulYq0Uacqu8lk55LTKGOO6Jmlkz4F2gD7AImAEMcM7NqfeNCcityd3BOfemmbUC3gCOBQYDy5xzw3O/UG2dc5clGGpVSEtuldfSKK/Fi6sluj+wwDn3vnNuNTAO6BfTuUvinFvsnHszt70CmAt0pCbeMbnDxlCTKElJbpXXkimvRYpUiZbQ3O8IfBR6vShXVtXMrBOwF/A60N45tzi3awnQPqGwKq7Ey7jU5bap5hWy/TebVF4bXYnmmvt3AkcAXYEBZta1XIElzcxaAuOBoc655eF9ruYeSCb7himv2cwrZDu3iebVOdeof8CBwJTQ698Bv6vv2NwP0pT/fdbYzzuuf6XkNXR80p9r0v+qPq+N/JtN+nNN+l9ReY0yi1Ntzf0D1j7IzIYAQ4DdI5wrKxYmHUARSs2rpCOvUERuldcCReW14g+WnHMjXc1sKnUvJC+p4/PqUjjDj9RNeS1dlEr0Y2Cb0Outc2W1cs49HeFcEp+S8iqpotxWQJRKdAawk5l1NrMNgZOBSeUJSxKkvGaXclsBjb4n6pxbY2bnUfPAqBkwyjk3u2yRSSKU1+xSbisj1lmczCy+k1WnN7J4r0l5VV4zqqi8agISEZEIVImKiESgSlREJAJVoiIiEcS97nzVu/LKKwG45pprgrL11qv5v6ZXr14AvPDCC7HHJdJUtWrVKthu2bIlAEceeSQAW2yxBQC33nprcMyqVatijE4tURGRSFSJiohEoMv5nMGDBwNw2WU1k1//+OOP6xwTZ59akaaqU6dOQP5v8cADDwz27bbbbrW+p0OHDsH2BRdcULngaqGWqIhIBGqJ5my33XYAbLTRRglHIvU54ID8zG2DBg0C4OCDDwZg1113Xef4Sy65BIBPPvkEgB49egT7HnnkEQBef/31ygQrDerSpQsAQ4cODcoGDhwIQIsWLQAws2DfRx/VzOS3YsUKAHbZZRcA+vfvHxxz1113ATBv3rxKhV1ALVERkQiafEu0d+/eAJx//vkF5eH/xY466igAPv300/gCkwInnXQSACNGjAjKNt98cyDfUnn++eeDfb7ryy233FLwfcKtGn/MySefXP6ApVabbropADfddBOQz2u4G9Pa5s+fH2wfdthhAGywwQZA/u/U/y6svR0HtURFRCJosBI1s1FmttTM3g6VtTOzqWY2P/e1bWXDlHJTXrNLuY1XMZfzo4E7gIdCZcOAZ51zw3PLrg4DLit/eJURfrjw4IMPAvnLDC98GbhwYVqW0CnJaKo4r+uvX/Orue++NTOR3XfffQBsvPHGwTHTp08H4NprrwXgpZdeCvY1b94cgMceewyAQw89dJ1zzJw5s9xhV4vRVGlujzuuZpWg3/zmNw0e+9577wHQp0+foMw/WNpxxx0rEF3jNNgSdc5NB5atVdwPGJPbHgMcW+a4pMKU1+xSbuPV2AdL7Z1zi3PbS4D2ZYonFqeddlqwvdVWWxXs8w8nHnroIZqgqsmr7750//33F5RPnTo12PYPJZYvL1hmvGDf2i3QRYsWBdtjxoyhCamK3J544om1ln/wwQfB9owZM4B8Z3vf+gzzXZuqQeSn8845V98M2FqCNZ2U1+yqL7fKa+kaW4l+amYdnHOLzawDsLSuA51zI4GRkPxyA77rw69//eugzA/v/OqrrwC47rrr4g+seiSaV39vE+Dyyy/35wHyHaj9LFtQewvUu+KKK2otDw8J/OyzzxofbPoUldtK/72eeeaZAAwZUlNPP/PMMwAsWLAgOGbp0jp/7QLt21fPxW9juzhNAvw18WnAxPKEIwlTXrNLua2QBluiZvYXoBewuZktAn4PDAceM7MzgIVA/7q/Q/L8hAbjx4+v85jbb78dgGnTpsURUuKqKa9XXXUVkG99AqxevRqAKVOmAPn7Y99999067/dDdcP3P7fddlsg37neX2FMnJj9uqOacrs2P/z26quvjvR9wpOSJK3BStQ5N6COXYeUORaJkfKaXcptvDRiSUQkgiYxdv7www8HoFu3buvse/bZZ4HCMdkSjzZt2gBwzjnnAIXztfrL+GOPrbs7o+9wPXbsWAD22WefdY7529/+BsDNN99choglDv7h3yabbFLnMbvvvnvB61deeSXYfvXVVysTWB3UEhURiSCzLdFwC2b48OEF+8LDA33H+6+//jqewCSw4YYbArXPuuNbIz/5yU8AOP300wE45phjgmP8LOd+8bJwS9Zv+zlDV65cWdbYJRo/fLdr164A/P73vw/29e3bt+BYv1AkrLvihH9Q5X8/AH744YfyBtsAtURFRCLIXEu0mO5M77//frCtOUKT47sx+U7vfn5PgP/+979A/eta+VaI73QfXmfn888/B+DJJ58sY8TSGH7uT4C99toLyP99+pyFu675vPp7m/6ZBhROQAP5iWqOP/74oMw/3/C/X5WmlqiISASqREVEIsjc5Xx9Sx57az9okmT4+Qr8Q8Cnnnoq2NeuXTsgP6ekH2k0evTo4Jhly2pmexs3bhxQeDnvyyQ5/sFh+HL88ccfLzjmmmuuAeC5554Lyl5++WUg/zsQ3rf2ksn+FtCNN94YlH344YcATJgwAYBVq1ZF+CkappaoiEgEmWmJ7rnnnkDtM5h7vjXzzjvvxBKTFMcvWRx+sFSMnj17Avklk8NXH+GHhxIv/yDJtzIvvfTSdY6ZPHkykJ+zwl+VQP734OmnnwYKO9b7h0V+8IRvmfbr1y84xg+++Oc//wnkF8UD+PLLLwvimDVrVgk/We3UEhURiSAzLVE/L2Hbtuuuv/Xaa68BMHjw4DhDkgpr0aIFkG+BhrtD6Z5ovJo1axZs+3lhL7nkEqBwoMOwYcOAfH58C9SvpQVwxx13APnuUOElk88++2wgP9ta69atATjooIOCYwYOHAjkB2aEV0Pw/Gz5nTt3LvpnrItaoiIiERQzn+g21Kwa2B5wwEjn3Agzawc8CnQCPgD6O+e+rOv7VNpmm20G1P5U3s+K/s0338QaUzVLS17r4ycpkbyk8upnqod8C/Tbb78F4Kyzzgr2+SvG7t27A/nhmkcccURwjL/C+MMf/gDkV+SFdddb8gMt/vGPfwRlfnvAgJoZAX/1q1+tE+9FF11U5E/WsGJaomuAi51zXYHuwLlm1pX8Eqw7Ac/mXkt6KK/ZpLzGrJglkxc7597Mba8A5gId0RKsqaa8ZpPyGj+rb2zyOgebdQKmA7sBHzrn2uTKDfjSv67n/WVf+Mo39f1Do9ou57fffnsAFi5cWO7Tl+oN59y+DR8Wr2rMazEOO+wwIN8VJvy77Dvex7QYXZPP6+LFi4Nt30XJd3KfN29esM/PEerngq2NXzrEd6CPe1amkKLyWvTTeTNrCYwHhjrnlvu1a0BLsKaZ8ppNymt8iqpEzWwDahIy1jnnx20ltgSr71gP0Lt3byDfAvWdce+8887gGM3UVLtqy2up/BWGFEoir0uWLAm2fUu0efPmAOyxxx7rHO+vHqZPnw7kh2gCfPDBB0CiLdCSNHhPNNf0fwCY65y7NbRLS7CmmPKaTcpr/Bq8J2pmPYAXgbcAf8PxcuB14DFgW3JLsDrnljXwvcrSYunVq1ew7TvS+tmv/TyU9d1zSVDV3DurxryWyg/5e+utt4DC++Fbbrkl0PTuiSaV11atWgXbfkKZvffeG4ClS/ON3lGjRgH54ZdxzfnZSOW5J+qcewmwOnZrCdaUUl6zSXmNn0YsiYhEkJmx89L0vP3220B+bHX4QdMOO+wAxHY53+StWLEi2H744YcLvmadWqIiIhGksiUa7rz7yiuvANCjR4+kwpGE3XDDDQDcf//9Qdn1118PwPnnnw/AnDlz4g9MmgS1REVEIihp2GfkkyXUFaaKVE1XmHJKOq9+TsnHHnssKPODMPyaPn62oPDclmWkvGZTUXlVS1REJAK1ROOlFksF+RYp5O+J+pnQu3XrBlTs3qjymk1qiYqIVJoqURGRCHQ5Hy9d9mWT8ppNupwXEam0uDvbfw6szH1Nm82JHvd25QikCimv2aS8FiHWy3kAM5uZxkuftMYdl7R+PmmNOy5p/XzijFuX8yIiEagSFRGJIIlKdGQC5yyHtMYdl7R+PmmNOy5p/Xxiizv2e6IiIlmiy3kRkQhiq0TN7HAze8fMFpjZsLjOWyoz28bMppnZHDObbWYX5srbmdlUM5uf+9o26VirRRpyq7yWTnktMoY4LufNrBnwLtAHWATMAAY456puptzcmtwdnHNvmlkr4A3gWGAwsMw5Nzz3C9XWOXdZgqFWhbTkVnktjfJavLhaovsDC5xz7zvnVgPjgH4xnbskzrnFzrk3c9srgLlAR2riHZM7bAw1iZKU5FZ5LZnyWqRIlWgJzf2OwEeh14tyZVXNzDoBe1GzZnd759zi3K4lQPuEwqq4Ei/jUpfbpppXyPbfbFJ5bXQlmmvu3wkcAXQFBphZ13IFljQzawmMB4Y655aH97maeyCZ7NagvGYzr5Dt3CaZ1ygt0VKa+x8D24Reb50rq0pmtgE1CRnrnHs8V/xp7v6Lvw+zNKn4KqzUy7jU5LaJ5xUy+jebdF4b/WDJzH4JHO6c+03u9SnAAc6582o5dn1qblJ3jhBrFnzunNsi6SDqU0pec/vXB/4XY4jVqOrzCo36m1Vei8hrxR8smdkQ4DXgh0qfKwUWJh1AuZjZEDObSU1umzrlNZuKymuUSrSo5r5zbqRzbl/n3E4RziXxKTWvqZvhpwlrMLfKa+miVKIzgJ3MrLOZbQicDEwqT1iSIOU1u5TbCmj0pMzOuTVmdh4wBWgGjHLOzS5bZJII5TW7lNvK0BpL8dJaPNmkvGaT1lgSEak0VaIiIhGoEhURiSDu1T5jM2LEiGD7ggsuAODtt98G4Kijjgr2LVyYmS5+IpIAtURFRCLIXEu0U6dOAAwaNCgo+/HHHwHYZZddAOjSpUuwTy3RdNh5550B2GCDDYKynj17AnDXXXcB+TwXa+LEiQCcfPLJAKxevTpynNI44bwedNBBANxwww0A/OxnP0skpmKpJSoiEoEqURGRCDJ3Of/ZZ58BMH369KDsmGOOSSocaaRdd90VgMGDBwNw4oknArDeevn/97faaisgfxlf6sAR/3txzz33ADB06NBg3/Lly2t9j1TGpptuGmxPmzYNgCVLlgCw5ZZbBvt8WTVRS1REJILMtURXrlwJ6IFR2t14440A9O3bt+LnOvXUUwF44IEHgrKXX3654ueV+vkWqFqiIiIZlrmWaJs2bQDYY489Eo5Eopg6dSqwbkt06dL8Kg++5ejvk9bWxcl3lzn44IMrEqdUjpklHUJR1BIVEYmgwUrUzEaZ2VIzeztU1s7MpprZ/NzXtpUNU8pNec0u5TZexVzOjwbuAB4KlQ0DnnXODc+tXT0MuKz84ZVu4403BmDbbbet85j99tsv2J43bx7QJB9EjaaK83r33XcDMGHChILy//0vv3ZaMQ8ZWrduDeTnTfDdosL8OWbOnNm4YKvPaKo4t8XyXdY22mijhCOpX4MtUefcdGDZWsX9gDG57THAsWWOSypMec0u5TZejX2w1N45tzi3vQRoX6Z4Ivvkk08AGD16dFB29dVXFxwTfv3VV18BcMcdd1Q6tDSomryuWbMGgI8++ijS9znssMMAaNu27qvXRYsWAbBq1apI56pyVZPbUu27b35y+ddeq75FSCM/nXfOufqWEcgtmTwk6nkkXsprdtWXW+W1dI2tRD81sw7OucVm1gFYWteBzrmRwEiId82Wa6+9NtheuyUqdar6vBbLz8x05plnAtCiRYs6j73qqqtiiSlhReU2qbz6Kw+Ar7/+GsgPBd1hhx3iCqNRGtvFaRJwWm77NGBiecKRhCmv2aXcVkiDLVEz+wvQC9jczBYBvweGA4+Z2RnAQqB/JYOMqr7O2E1VFvLqDRw4EIBhw4YFZTvuuCNQOE/l2mbNmgUUPvHPgjTm1j+bAHjxxReBwhUoqlmDlahzbkAduw4pcywSI+U1u5TbeGnEkohIBJkbO1+bxs43Kcnxy7yccsopAPTu3bvOY3v06AHUn18/P2j4kv/pp58G4LvvvosUqzRtaomKiETQJFqikg677bZbsD1p0iSg/uG7pfAPK0aOHFmW7yfx2WyzzZIOoV5qiYqIRKCWqFQlP5dkMXNKFtOFzXeXOeKII4KyyZMnRwlRYlLta6SpJSoiEoEqURGRCJrE5Xx9l3s9e/YENItTNfBzfgL06tULgEGDBgEwZcoUAL7//vuivtcZZ5wBwPnnn1/GCCUOfsnktIxYUktURCQCi7MDelKz/fzwww9A/Z2xu3XrBsCcOXMqGcobzrl9Gz4sXapxFic/A9AXX3xRUH700UcH22V8sKS8ltEJJ5wAwF//+legcDBE165dgdhWoigqr2qJiohE0CTuid5zzz0AnHXWWXUeM2RIzTy0Q4cOjSUmqSw/o72kT3huUSjs5ta8efO4w2mQWqIiIhEUM5/oNtSsGtgecMBI59wIM2sHPAp0Aj4A+jvnvqxcqI3nV/SUvGrIq5/r89BDDwXgueeeC/Y1ZlKQ008/PdgeMWJExOjSqRryGtXEiTXzRfu/2y5dugT7/JXiOeecE39gdSimJboGuNg51xXoDpxrZl3JL8G6E/Bs7rWkh/KaTcprzIpZMnmxc+7N3PYKYC7QES3BmmrKazYpr/ErqYuTmXUCpgO7AR8659rkyg340r+u5/2JdoV59913gdoXvvId8v2yEu+9914lQqjKrjBx5tXP/QlwxRVzJQNAAAAEM0lEQVRXANCnTx8AOnfuHOwrZqnkdu3aAdC3b18Abr/99mBfq1atCo71twfC47B9p+4yaPJ5rYQ///nPQOFtmvbta1Z6LnbQRURF5bXop/Nm1hIYDwx1zi0PPzHTEqzppbxmk/Ian6IqUTPbgJqEjHXOPZ4rruolWGsze/ZsALbffvt19jXFReySyGt4eG14/lCA3/72t8H2ihUrGvxevgW79957+5jWOeb5558H4O677wbK2vqsWln5e/XCeV29enWCkdSuwXuiuab/A8Bc59ytoV1agjXFlNdsUl7jV0xL9GfAKcBbZjYrV3Y5Vb4Ea238rObhoX9NWNXl9eyzz470/qVL842rJ598EoALL7wQiO0eWjWourxG1bp162C7X79+ADzxxBNJhbOOYpZMfgmoa2ZcLcGaUsprNimv8dOIJRGRCJrE2HnPz9A0d+7coGyXXXZJKpwmafDgwcG2n+vztNNOq+PodYW7nn377bdA7YvQhecmlXTq37/mjsOqVauCsvDfbrVQS1REJIIm1RL1cxDuvvvuCUfSdM2aNSvY9uOf//WvfwFw3XXXBfvatm0LwIQJEwCYOnUqkB9XDbBkyZLKBiuJmj59OlB4tdiYORUqTS1REZEImsTM9lWkKocHRqW8Kq8ZpZntRUQqTZWoiEgEqkRFRCJQJSoiEoEqURGRCFSJiohEEHdn+8+BlbmvabM50ePerhyBVCHlNZuU1yLE2k8UwMxmprFPXVrjjktaP5+0xh2XtH4+ccaty3kRkQhUiYqIRJBEJTqy4UOqUlrjjktaP5+0xh2XtH4+scUd+z1REZEs0eW8iEgEsVWiZna4mb1jZgvMbFhc5y2VmW1jZtPMbI6ZzTazC3Pl7cxsqpnNz31tm3Ss1SINuVVeS6e8FhlDHJfzZtYMeBfoAywCZgADnHNzKn7yEuXW5O7gnHvTzFoBbwDHAoOBZc654blfqLbOucsSDLUqpCW3ymtplNfixdUS3R9Y4Jx73zm3GhgH9Ivp3CVxzi12zr2Z214BzAU6UhPvmNxhY6hJlKQkt8pryZTXIsVViXYEPgq9XpQrq2pm1gnYC3gdaO+cW5zbtQRon1BY1SZ1uVVei6K8FkkPlupgZi2B8cBQ59zy8D5Xcw9E3RpSSHnNpiTzGlcl+jGwTej11rmyqmRmG1CTkLHOucdzxZ/m7r/4+zBLk4qvyqQmt8prSZTXIsVVic4AdjKzzma2IXAyMCmmc5fEzAx4AJjrnLs1tGsS4BdIPw2YuPZ7m6hU5FZ5LZnyWmwMcXW2N7O+wJ+BZsAo59z1sZy4RGbWA3gReAv4MVd8OTX3WR4DtgUWAv2dc8sSCbLKpCG3ymvplNciY9CIJRGRxtODJRGRCFSJiohEoEpURCQCVaIiIhGoEhURiUCVqIhIBKpERUQiUCUqIhLB/wOA2s4TjzPBwgAAAABJRU5ErkJggg==\n", "text/plain": [ "
" ] }, "metadata": { "needs_background": "light" }, "output_type": "display_data" } ], "source": [ "import numpy as np\n", "import tensorflow as tf\n", "from tensorflow import keras\n", "\n", "import matplotlib.pyplot as plt\n", "%matplotlib inline\n", "import datetime as dt\n", "\n", "from pathlib import Path\n", "\n", "from sklearn.datasets import fetch_openml\n", "from sklearn import preprocessing\n", "\n", "data_home = '/tmp/scikit_learn_data/'\n", "datafile = '/tmp/mnist.npz'\n", "\n", "datapath = Path(datafile)\n", "if not(datapath.exists()):\n", " print(\"Data File not found... downloading it\")\n", " Xmnist, ymnist = fetch_openml('mnist_784',\n", " version=1,\n", " return_X_y=True,\n", " data_home=data_home)\n", " np.savez(datapath.as_posix(),\n", " X=np.array(Xmnist,dtype='u8'),\n", " y=np.array(ymnist,dtype='u8'))\n", " print(\"Data File downloaded and saved\")\n", " del Xmnist, ymnist\n", "\n", "print(\"Data File found... loading it into memory\")\n", "data = np.load(datapath.as_posix())\n", "Xmnist = data['X']\n", "ymnist = keras.utils.to_categorical(data['y'])\n", "print(\"Data File loaded\")\n", " \n", "Xtrain, Ytrain = Xmnist[:50000], ymnist[:50000]\n", "Xvalid, Yvalid = Xmnist[50000:60000], ymnist[50000:60000]\n", "Xtest, Ytest = Xmnist[-10000:], ymnist[-10000:]\n", "\n", "plt.figure()\n", "for i in range(9):\n", " plt.subplot(3,3,i+1)\n", " plt.imshow(Xtrain[i].reshape((28,28)),cmap='gray', vmin=0, vmax=255)\n", "plt.show()\n", "\n", "\n", "scaler = preprocessing.StandardScaler().fit(Xtrain)\n", "Xtrain = scaler.transform(Xtrain)\n", "Xvalid = scaler.transform(Xvalid)\n", "Xtest = scaler.transform(Xtest)\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Preventing vanishing gradient\n", "\n", "Vanishing gradient can be prevented by:\n", "- using weight regularization [doc](https://keras.io/regularizers/).\n", "- using non saturating activation functions [simple](https://keras.io/activations/) [advanced](https://keras.io/layers/advanced-activations/)\n", "- adding dropout [doc](https://keras.io/layers/core/)\n", "- adding batch normalization [doc](https://keras.io/layers/normalization/)\n", "\n", "You can also play on the batch size and the learning rate.\n", "\n", "1) Try to modify your code to include the aforementioned features.\n", "\n", "2) Observe the effect of the modification on the vanishing gradient on?" ] } ], "metadata": { "celltoolbar": "Tags", "kernelspec": { "display_name": "Python 3", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.7.3" } }, "nbformat": 4, "nbformat_minor": 2 }