# Logging the training progress in a CSV - DL
Data Science for Political and Social Phenomena
Author: Guillaume Redoulès

# Logging the training progress in a CSV

Let's see how we can log the progress and various metrics during the training process to a csv file. Let's first import some libraries

    import keras
    import numpy as np

In this example, we will be using the fashion MNIST dataset to do some basic computer vision, where we will train a Keras neural network to classify items of clothing.

In order to import the data we will be using the built in function in Keras :

    keras.datasets.fashion_mnist.load_data()

The model is a very simple neural network consisting in 2 fully connected layers. The model loss function is chosen in order to have a multiclass classifier : "sparse_categorical_crossentropy"

Let's define a simple feedforward network.

    ##get and preprocess the data
    fashion_mnist = keras.datasets.fashion_mnist
    (train_images, train_labels), (test_images, test_labels) = fashion_mnist.load_data()
    train_images = train_images / 255.0
    test_images = test_images / 255.0

    ## define the model

    model = keras.Sequential([
        keras.layers.Flatten(input_shape=(28,28)),
        keras.layers.Dense(128, activation="relu"),
        keras.layers.Dense(10, activation="softmax")
    ])

    model.compile(optimizer="adam",
                 loss = "sparse_categorical_crossentropy",
                 metrics = ["accuracy",'mae'])

In order to stream to a csv file the epoch results and metrics, we define a CSV logger. It is a callback located in

    keras.callbacks

Let's first import it

    from keras.callbacks import CSVLogger

We now need to define the callback by specifiying a file to be written to, the separator and whether to append to the file or erase it every time.

The callback has to be added to the callbacks list in the fit method.

    csv_logger = CSVLogger(filename="my_csv.csv", separator=';', append=False)
    model.fit(train_images, train_labels, epochs=5, callbacks=[csv_logger])

    Epoch 1/5
    60000/60000 [==============================] - 9s 148us/step - loss: 0.5020 - acc: 0.8234 - mean_absolute_error: 4.4200
    Epoch 2/5
    60000/60000 [==============================] - 8s 138us/step - loss: 0.3765 - acc: 0.8630 - mean_absolute_error: 4.4200
    Epoch 3/5
    60000/60000 [==============================] - 8s 129us/step - loss: 0.3371 - acc: 0.8789 - mean_absolute_error: 4.4200
    Epoch 4/5
    60000/60000 [==============================] - 8s 133us/step - loss: 0.3129 - acc: 0.8843 - mean_absolute_error: 4.4200
    Epoch 5/5
    60000/60000 [==============================] - 9s 151us/step - loss: 0.2952 - acc: 0.8916 - mean_absolute_error: 4.4200

    <keras.callbacks.History at 0x1582adc6780>

The results are stored in the my_csv.csv file and contain the epoch results

    import pandas as pd
    pd.read_csv("my_csv.csv", sep=";")

| epoch | acc | loss | mean_absolute_error
--- | --- | --- | --- | ---
0 | 0 | 0.823400 | 0.502013 | 4.42
1 | 1 | 0.863050 | 0.376516 | 4.42
2 | 2 | 0.878867 | 0.337097 | 4.42
3 | 3 | 0.884317 | 0.312893 | 4.42
4 | 4 | 0.891583 | 0.295157 | 4.42

### Find an error or bug? Have a suggestion?

Everything on this site is avaliable on GitHub. Head on over and submit an issue. You can also message me directly by email.