Tutorial

This section covers the fundamentals of developing with doce, including a package overview, basic and advanced usage. We will assume basic familiarity with Python and NumPy.

Overview

The doce package is structured as collection of submodules that are each responsible for the important parts of managing a computational experiment:

  • doce.cli

    Command-line interaction.

  • doce.experiment

    Specify every aspects of the experiments from naming, storage location, plan, etc…

  • doce.plan

    Generate a number of settings by selecting factors and modalities of a given plan.

  • doce.setting

    Manipulate the settings generated by the plan.

  • doce.metric

    Manipulate and retrieve the output data.

  • doce.util

    Utility functions.

Quickstart

The doce package is designed to require very few lines of code around your processing code to handle the task of evaluating its performance with respect to different parametrizations.

Define the experiment

In a .py file, ideally named after the name of your experiment, you have to implement a set function that contains the relevant definition of your experiment. The demonstrations discussed in this tutorial are available in the examples directory of the github repository of the project. In this first example, the demo.py is considered.

 1# define the experiment
 2experiment = doce.Experiment(
 3  name = 'demo',
 4  purpose = 'hello world of the doce package',
 5  author = 'mathieu Lagrange',
 6  address = 'mathieu.lagrange@ls2n.fr',
 7)
 8# set acces paths (here only storage is needed)
 9experiment.set_path('output', '/tmp/'+experiment.name+'/')
10# set some non varying parameters (here the number of cross validation folds)
11experiment.n_cross_validation_folds = 10

Define the plan

In doce, the parametrization of the processing code is called a setting. Each setting is a set of factors, each factor being uniquely instantiated by a modality, chosen among a pre-defined set of modalities.

1  # set the plan (factor : modalities)
2  experiment.add_plan('plan',
3    nn_type = ['cnn', 'lstm'],
4    n_layers = np.arange(2, 10, 3),
5    learning_rate = [0.001, 0.0001],
6    dropout = [0, 1]
7  )

Interact with your experiment

The doce package have a convenient way of interacting with experiments, through the command-line. For this to work, you need to add those lines to your python file:

1# invoke the command line management of the doce package
2if __name__ == "__main__":
3  doce.cli.main(experiment = experiment,
4          func = step
5          )

Now you can interact with your experiment. For example you can display the plan:

$ python demo.py -p
         Factors      0       1  2
0        nn_type    cnn    lstm
1       n_layers      2       5  8
2  learning_rate  0.001  0.0001  0.00001
3        dropout      0       1

You can also access to a reference list of each pre-defined argument:

$ python demo.py -h
usage: demo.py [-h] [-A [ARCHIVE]] [-C] [-d [DISPLAY]] [-e [EXPORT]] [-H HOST] [-i] [-K [KEEP]] [-l]
               [-M [MAIL]] [-p] [-P [PROGRESS]] [-r [RUN]] [-R [REMOVE]] [-s SELECT] [-S] [-u USERDATA]
               [-v] [-V]

optional arguments:
  -h, --help            show this help message and exit
...

Control the plan

You can list the different settings generated by the plan:

$ python demo.py -l
nn_type=cnn+n_layers=2+learning_rate=0dot001+dropout=0
nn_type=cnn+n_layers=2+learning_rate=0dot001+dropout=1
nn_type=cnn+n_layers=2+learning_rate=0dot0001+dropout=0
... (36 lines)
Most of the time you want to process or retrieve the output data of a selection of settings. Doce provides 3 selection formats for expressing that selection :

  1. the string format,

  2. the dictionary format,

  3. the numeric array format.

Suppose you want to select the settings with n_layers=2 and no dropout, you can do that easily with a string formatted selector:

python demo.py -l -s n_layers=2+dropout=0
nn_type=cnn+n_layers=2+learning_rate=0dot001+dropout=0
nn_type=cnn+n_layers=2+learning_rate=0dot0001+dropout=0
nn_type=cnn+n_layers=2+learning_rate=1edash05+dropout=0
nn_type=lstm+n_layers=2+learning_rate=0dot001+dropout=0
nn_type=lstm+n_layers=2+learning_rate=0dot0001+dropout=0
nn_type=lstm+n_layers=2+learning_rate=1edash05+dropout=0

Suppose you want to select the settings with nn_type=cnn, n_layers=2, n_layers=8 and no dropout with the string format, the only way is to chain selectors:

$ python demo.py -l -s nn_type=cnn+n_layers=2+dropout=0,nn_type=cnn+n_layers=5+dropout=0
nn_type=cnn+n_layers=2+learning_rate=0dot001+dropout=0
nn_type=cnn+n_layers=2+learning_rate=0dot0001+dropout=0
nn_type=cnn+n_layers=2+learning_rate=1edash05+dropout=0
nn_type=cnn+n_layers=5+learning_rate=0dot001+dropout=0
nn_type=cnn+n_layers=5+learning_rate=0dot0001+dropout=0
nn_type=cnn+n_layers=5+learning_rate=1edash05+dropout=0

This can get tedious when you want to select multiple modalities for multiple factors. For example, suppose you want to select the settings with nn_type=cnn, n_layers=[2, 4] and learning_rate= [0.001, 0.00001], you can do that conveniently with a dictionary formatted selector:

$ python demo.py -l -s '{"nn_type"="cnn", "n_layers":[2, 5],"learning_rate":[0.001,0.00001]}'
nn_type=cnn+n_layers=2+learning_rate=0dot001+dropout=0
nn_type=cnn+n_layers=2+learning_rate=0dot001+dropout=1
nn_type=cnn+n_layers=2+learning_rate=1edash05+dropout=0
nn_type=cnn+n_layers=2+learning_rate=1edash05+dropout=1
nn_type=cnn+n_layers=5+learning_rate=0dot001+dropout=0
nn_type=cnn+n_layers=5+learning_rate=0dot001+dropout=1
nn_type=cnn+n_layers=5+learning_rate=1edash05+dropout=0
nn_type=cnn+n_layers=5+learning_rate=1edash05+dropout=1

The ‘’ delimiters are required to avoid interpretetation of the selector by the shell. The ” inside the selector delimiters must not be replaced by ‘’ delimiters.

You can perform the same selection with a numeric array formatted selector:

$ python demo.py -l -s '[0,[0, 1],[0, 2]]'
nn_type=cnn+n_layers=2+learning_rate=0dot001+dropout=0
nn_type=cnn+n_layers=2+learning_rate=0dot001+dropout=1
nn_type=cnn+n_layers=2+learning_rate=1edash05+dropout=0
nn_type=cnn+n_layers=2+learning_rate=1edash05+dropout=1
nn_type=cnn+n_layers=5+learning_rate=0dot001+dropout=0
nn_type=cnn+n_layers=5+learning_rate=0dot001+dropout=1
nn_type=cnn+n_layers=5+learning_rate=1edash05+dropout=0
nn_type=cnn+n_layers=5+learning_rate=1edash05+dropout=1

As with the string selector, the dict and numeric array types of selector can be chained with a ,.

Define processing code

You must define which code shall be processed for any setting, given the computing environnent defined by the experiment by implementing a step function:

1def step(setting, experiment):
2  # the accuracy  is a function of cnn_type, and use of dropout
3  accuracy = (len(setting.nn_type)+setting.dropout+np.random.random_sample(experiment.n_cross_validation_folds))/6
4  # duration is a function of cnn_type, and n_layers
5  duration = len(setting.nn_type)+setting.n_layers+np.random.randn(experiment.n_cross_validation_folds)
6  # storage of outputs (the string between _ and .npy must be the name of the metric defined in the set function)
7  np.save(experiment.path.output+setting.id()+'_accuracy.npy', accuracy)
8  np.save(experiment.path.output+setting.id()+'_duration.npy', duration)

In this demo, the processing code simply stores some dummy outputs to the disk.

Perform computation

Now that we have set all this, performing the computation of some settings can simply be done by:

$ python demo.py -c -s '{"nn_type":"cnn", "n_layers":[2, 5],"learning_rate":[0.001,0.00001]}'

Adding a -P to the command line conveniently displays a per setting progress bar.

Removing the -s will require the computation of all the settings.

Some settings can fail, which will stop the entire loop. If you want to compute all the non failing settings, you can use the detached computation mode, available with -D.

If some settings have failed, a log file is available to provide guidance for debugging your code.

Once fixed, you can be interested in computing only the settings that have failed. For this, you can use the skipping computation mode, available with -S. In that mode, for each setting, doce will search for available metrics. If available, the setting is not computed.

Warning: do not consider skipping if some settings have been previously succesfully computed using an outdated version of your code.

Define metrics

Before inspecting the results of our computation, we have to define how the output stored on disc shall be reduced to metrics for interpretation purposes.

To do so, we have to use the set_metric() method.

 1# set the metrics
 2experiment.set_metric(
 3  name = 'accuracy',
 4  percent=True,
 5  higher_the_better= True,
 6  significance = True,
 7  precision = 10
 8  )
 9
10experiment.set_metric(
11  name = 'acc_std',
12  output = 'accuracy',
13  func = np.std,
14  percent=True
15  )
16
17# custom metric function shall input an np.nd_array and output a scalar
18def my_metric(data):
19  return np.sum(data)
20
21experiment.set_metric(
22  name = 'acc_my_metric',
23  output = 'accuracy',
24  func = my_metric,
25  percent=True
26  )

Display metrics

The reduced version of the metrics can be visualized in the command-line using -d :

$ python demo.py -d
Displayed data generated from Mon Mar 21 13:59:13 2022 to Mon Mar 21 13:59:13 2022
nn_type: cnn
   n_layers  learning_rate  dropout  accuracyMean%+  accuracyStd%  durationMean*-
0         2        0.00100        0            58.0           5.0            5.63
1         2        0.00100        1            74.0           5.0            5.21
2         2        0.00001        0            56.0           4.0            4.67
3         2        0.00001        1            78.0           3.0            4.81
4         5        0.00100        0            56.0           4.0            8.44
5         5        0.00100        1            76.0           5.0            8.20
6         5        0.00001        0            60.0           6.0            8.59
7         5        0.00001        1            75.0           4.0            7.90

Only the metrics available on disc are considered in the table.

You can select the metrics you want to display. To display one metric:

$ python demo.py -d 0
Displayed data generated from Mon May 16 15:56:16 2022 to Mon May 16 15:56:16 2022
nn_type: cnn
   n_layers  learning_rate  dropout  accuracyMean%+
0         2        0.00100        0              58
...

To display an arbitrary number of metrics, say first and third:

$ python demo.py -d '[0, 2]'
Displayed data generated from Mon May 16 15:56:16 2022 to Mon May 16 15:56:16 2022
nn_type: cnn
   n_layers  learning_rate  dropout  accuracyMean%+  durationMean*-
0         2        0.00100        0              58            4.31

doce allows you to analyse the impact of a given factor on a given metric. for example, let us study the impact of n_layers on durationMean:

$ python demo.py -d 2:n_layers  -s '{"nn_type":"cnn", "n_layers":[2, 5],"learning_rate":[0.001,0.00001]}'

Displayed data generated from Mon May 16 16:47:38 2022 to Mon May 16 16:47:38 2022
metric: durationMean*- for factor nn_type: cnn  n_layers
   learning_rate  dropout     2     5
0        0.00100        0  5.32  8.14
1        0.00100        1  4.85  7.69
2        0.00001        0  5.43  8.20
3        0.00001        1  5.54  7.98

Note that here you have to provide the selector for doce to infer the correct organization of the table. This command will fail if some of the needed settings are not available.

Export metrics

The table can exported in various format:

  • html

  • pdf

  • png

  • tex

  • csv

  • xls

To export the table in files called demo, please type : .. code-block:: console

$ python demo.py -d -e demo

To only generate the html output, please type : .. code-block:: console

$ python demo.py -d -e demo.html

For visualization purposes, the html output is perhaps the most interesting one, as it shows best values per metrics and statistical analysis :

_images/demo.png

The title specifies the factors with unique modality in the selection.

Please note that the page as an auto-reload javascript code snippet that conveniently reloads the page at each new focus.

The mean accuracy is defined as a higher-the-better metric; thus 78 is displayed in bold. the average duration is specified as a lower-the-better metric the 4.67 is displayed in bold. A statistical analysis as been requested (with the *), the several t-tests are operated to check whether the best setting can be assumed to be significantly better than the others. In our example, the other settings with n_layers=2 cannot be assumed to be slower than the most rapid setting.

Mine metrics

Reduced versions of the metrics are convenient to quickly analyse the data. For more refined purposes, such as designing a custom designed plot, one needs to have access to the raw data saved during the processing.

For this example, let us first compute the performance of the cnn and lstm system at a given number of layers and learning with or without dropout:

$ python demo.py -s '{"nn_type":["cnn", "lstm"],"n_layers":2,"learning_rate":0.001}' -c

Within a python file or a jupyer notebook, we can now retrieve the accuracy data:

 1# your experiment file shall be in the current directory or in the python path
 2import demo
 3
 4experiment = demo.set()
 5selector = {"nn_type":["cnn", "lstm"],"n_layers":2,"learning_rate":0.001}
 6
 7(data, settings, header) = experiment.get(
 8  metric = 'accuracy',
 9  selector = selector,
10  path = 'output'
11  )

The data is a list of np.arrays, the settings is a list of str and the header is a str describing the constant factors. data and settings are of the same size.

In our example, the data can be conveniently displayed using any horizontal bar plot:

 1import numpy as np
 2import matplotlib.pyplot as plt
 3
 4settingIds = np.arange(len(description))
 5
 6fig, ax = plt.subplots()
 7ax.barh(settingIds, np.mean(data, axis=1), xerr=np.std(data, axis=1), align='center')
 8ax.set_yticks(settingIds)
 9ax.set_yticklabels(settings)
10ax.invert_yaxis()  # labels read top-to-bottom
11ax.set_xlabel('Accuracy')
12ax.set_title(header)
13
14fig.tight_layout()
15plt.show()
_images/barh.png

Customizing the plan

The definite plan for a given experiment is only known when the experiment is over. It is therefore important to be able to fine tune the plan along with your exploration.

This is not trivial to achieve as it may lead to inconsistencies in stored metric naming conventions if not properly handled.

If you are looking for adding another whole new algorithm or processing step to your experiment, it may be worth considering multiple plans, as described in the dedicated section.

Adding a modality

The addition of a modality is simply done by adding a value to the array of a given factor.

Note that order of modalities matters as it will determine the order in which settings are computed. This is convenient, because you can assume that when requesting the computation of all steps, the output data of step1 will be available to step2, and so on.

Important, this assertion no longer holds if parallelization over settings is selected.

Removing a modality

The removal of a modality is simply done by removing the value to the array of a given factor.

If you want to discard the output data that is no longer accessible, you can do it manually by considering the rm command. Let us assume that we want to remove the modality 0.001 from the factor learning_rate. You can type:

$ rm *learning_rate=0dot001*.npy <insert_path>

You can also do before removing the modality in the array:

$ python demo.py -R output -s learning_rate=0dot001
INFORMATION: setting path.archive allows you to move the unwanted files to the archive path and not delete them.
List the 24 files ? [Y/n]
/tmp/demo/dropout=0+learning_rate=0dot001+n_layers=8+nn_type=lstm_accuracy.npy
...
/tmp/demo/dropout=0+learning_rate=0dot001+n_layers=8+nn_type=cnn_accuracy.npy
About to remove 24 files from /tmp/demo/
 Proceed ? [Y/n]

The selector can be more precise that just one modality.

Adding a factor

Let us say you are considering two classifiers in your experiment: as cnn based and a lstm (code is available in the example directory under file factor_manipulation.py). The plan would be:

1experiment.addPlan('plan',
2  nn_type = ['cnn', 'lstm'],
3  # dropout = [0, 1]
4)

Please note the dropout factor is commented for now. The step function simply saves a .npy file with a 0 value in it. Thus, the output directory contains:

$ ls -1 /tmp/factor_manipulation/
nn_type=cnn_accuracy.npy
nn_type=lstm_accuracy.npy

And the display command will show:

$ python factor_manipulation.py -d
Displayed data generated from Thu Mar 24 10:02:24 2022 to Thu Mar 24 10:02:24 2022

  nn_type  accuracyMean
0     cnn           0.0
1    lstm           0.0

Now, let’s add the dropout factor by uncommenting its line in the plan:

1experiment.addPlan('plan',
2  nn_type = ['cnn', 'lstm'],
3  dropout = [0, 1]
4)

Now, the problem is that the display command will show nothing:

$ python factor_manipulation.py -d

Why is that ? Well, now that we have added a new factor, the settings file list is:

$ python factor_manipulation.py -f
dropout=0+nn_type=cnn
dropout=1+nn_type=cnn
dropout=0+nn_type=lstm
dropout=1+nn_type=lstm

which do not match any of the stored files. In this example, we could simply recompute dropout=0+nn_type=cnn and dropout=0+nn_type=lstm, but in production, that could mean a loss of lengthy computations. The solution to this critical problem is to explicitly state a default value for the factor dropout:

1experiment.default(plan='plan', factor='dropout', modality=0)

Now the settings file list is:

$ python factor_manipulation.py -f
nn_type=cnn
dropout=1+nn_type=cnn
nn_type=lstm
dropout=1+nn_type=lstm

And the previously computed metrics can now be displayed as before:

$ python factor_manipulation.py -d
Displayed data generated from Thu Mar 24 10:02:24 2022 to Thu Mar 24 10:02:24 2022
dropout: 0
  nn_type  accuracyMean
0     cnn           0.0
1    lstm           0.0

Removing a factor

Important, this kind of manipulation may lead to output data loss. Be sure to make a backup before attempting to remove a factor.

Let us consider that you have tested whether dropout is useful or not and have decided that dropout is always useful and that you want to remove the dropout factor to avoid clutter in the plan.

Simply removing the factor will lead to the need to redo every computation. It is thus required to perform the following steps:

  1. keep only wanted settings (here settings with dropout=0)

  2. rename files by removing reference to the dropout setting.

Let us assume that we have computed every settings, the files are:

$ python factor_manipulation.py -c
$ ls -1 /tmp/factor_manipulation/
dropout=1+nn_type=cnn_accuracy.npy
dropout=1+nn_type=lstm_accuracy.npy
nn_type=cnn_accuracy.npy
nn_type=lstm_accuracy.npy

Keeping only the files of interest is done so:

$ python factor_manipulation.py -K output -s dropout=1
INFORMATION: setting path.archive allows you to move the unwanted files to the archive path and not delete them.
List the 2 files ? [Y/n]
/tmp/factor_manipulation/nn_type=cnn_accuracy.npy
/tmp/factor_manipulation/nn_type=lstm_accuracy.npy
About to remove 2 files from /tmp/factor_manipulation/
 Proceed ? [Y/n]

To rename files by removing reference to the dropout setting.

$ rename -n 's/(\+)?dropout=1(\+)?(_)?/$3/' /tmp/factor_manipulation/*npy
Use of uninitialized value $3 in substitution (s///) at (eval 2) line 1.
'/tmp/factor_manipulation/dropout=1+nn_type=cnn_accuracy.npy' would be renamed to '/tmp/factor_manipulation/nn_type=cnn_accuracy.npy'
Use of uninitialized value $3 in substitution (s///) at (eval 2) line 1.
'/tmp/factor_manipulation/dropout=1+nn_type=lstm_accuracy.npy' would be renamed to '/tmp/factor_manipulation/nn_type=lstm_accuracy.npy'

Check that the correct files are targeted and remove the -n in the command. Now you can safely remove the dropout factor from the plan.

Managing multiple plans

Most of the time, computational approaches have different needs in terms of parametrization, which add difficulties in managing plans of computations. The doce package handle this by allowing the definition of multiple plans that are then automatically merged is needed. In this first example, the demo_multiple_plan.py is considered.

Assume that we want to compare 3 classifiers : 1. an svm 2. a cnn 3. an lstm

The last two classifiers share the same factors, but the svm have only one factor, called c.

We start by defining the “svm” plan:

1# set the "svm" plan
2experiment.addPlan('svm',
3  classifier = ['svm'],
4  c = [0.001, 0.0001, 0.00001]
5)

We then define the “deep” plan:

1# set the "deep" plan
2experiment.addPlan('deep',
3  classifier = ['cnn', 'lstm'],
4  n_layers = [2, 4, 8],
5  dropout = [0, 1]
6)

Selecting a given plan is done using the selector:

$ python demo_multiple_plan.py  -s svm/ -l
Plan svm is selected
classifier=svm+c=0dot001
classifier=svm+c=0dot0001
classifier=svm+c=1edash05

Otherwise, the merged plan is considered:

$ python demo_multiple_plan.py  -p
Plan svm:
      Factors      0       1      2
0  classifier    svm
1           c  0.001  0.0001  1e-05
Plan deep:
      Factors    0     1  2
0  classifier  cnn  lstm
1    n_layers    2     4  8
2     dropout    0     1
Those plans can be selected using the selector parameter.
Otherwise the merged plan is considered:
      Factors      0      1       2      3
0  classifier    svm    cnn    lstm
1           c  *0.0*  0.001  0.0001  1e-05
2    n_layers    *0*      2       4      8
3     dropout    *0*      1

Computation can be done using the specified plans:

$ python demo_multiple_plan.py  -s svm/ -c
Plan svm is selected
$ python demo_multiple_plan.py  -s deep/ -c
Plan deep is selected

Display of metric is conveniently done using the merged plan:

$ python demo_multiple_plan.py  -d
Displayed data generated from Mon Mar 21 17:22:32 2022 to Mon Mar 21 17:26:22 2022

  classifier     c  n_layers  dropout  accuracyMean%
0        svm  1.00         0        0            8.0
1        svm  0.10         0        0            1.0
2        svm  0.01         0        0            0.0
3        cnn  0.00         2        1           76.0
4        cnn  0.00         4        1           74.0
5        cnn  0.00         8        1           77.0
6       lstm  0.00         2        1           94.0
7       lstm  0.00         4        1           91.0
8       lstm  0.00         8        1           91.0

Advanced usage

Tagging computations

During development, it is sometimes useful to differentiate between several runs of the experiment. For example, you might want to try out a new tweak or play around with some parameters that you do not want to add to the plan.

You can do so by tagging. The tag will add a level of hierarchy in your output paths. Let us assume that you have python file demo.py that defines a storage path named output pointing to /tmp/experiment/. When running python demo.py –tag my_tag, the storage path will now be /tmp/experiment/my_tag.

This gives you the freedom to switch easily to compare relative performance. For replication purposes, this tag can conveniently be defined as an id of your prefered code versioning system.

If you want tag outputs to become the default outputs, you simply have to move file from the tag directory to the root directory. In this example, mv /tmp/experiment/my_tag/* /tmp/experiment.

Storage within an hdf5 file

Remote computing