.. DO NOT EDIT. .. THIS FILE WAS AUTOMATICALLY GENERATED BY SPHINX-GALLERY. .. TO MAKE CHANGES, EDIT THE SOURCE PYTHON FILE: .. "_build/auto_examples/tutorial/02-Caching_benchmarks.py" .. LINE NUMBERS ARE GIVEN BELOW. .. only:: html .. note:: :class: sphx-glr-download-link-note :ref:`Go to the end ` to download the full example code. .. rst-class:: sphx-glr-example-title .. _sphx_glr__build_auto_examples_tutorial_02-Caching_benchmarks.py: Caching benchmarks ================== .. GENERATED FROM PYTHON SOURCE LINES 7-17 Saving benchmarks ***************** Performance benchmarks as run by pyquickbench are typically quite lengthy. While obvious for complex benchmarks, this is still true for simple benchmarks. Indeed, by default, pyquickbench uses a strategy derived from the standard :meth:`python:timeit.Timer.autorange` to assess the number of times a benchmark should be run to ensure reliable timings, which comes with significant overhead as any call to this function will require a (configurable) minimum execution time of 0.2 seconds. Caching benchmarks results is a great way to reduce this overhead. In pyquickbench, caching results in order to avoid a full re-run is as simple as providing a file name to :func:`pyquickbench.run_benchmark`. If this file does not exist, it will be created. Both ``*.npy`` and ``*.npz`` extensions are accepted. .. GENERATED FROM PYTHON SOURCE LINES 17-58 .. code-block:: Python import numpy as np import pyquickbench def comprehension(n): return ['' for _ in range(n)] def star_operator(n): return ['']*n def for_loop_append(n): l = [] for _ in range(n): l.append('') all_funs = [ comprehension , star_operator , for_loop_append , ] n_bench = 12 all_sizes = [2**n for n in range(n_bench)] timings_filename = "My_benchmark_file.npy" all_times = pyquickbench.run_benchmark( all_sizes , all_funs , filename = timings_filename , ) pyquickbench.plot_benchmark( all_times , all_sizes , all_funs , show = True , ) .. image-sg:: /_build/auto_examples/tutorial/images/sphx_glr_02-Caching_benchmarks_001.png :alt: 02 Caching benchmarks :srcset: /_build/auto_examples/tutorial/images/sphx_glr_02-Caching_benchmarks_001.png :class: sphx-glr-single-img .. GENERATED FROM PYTHON SOURCE LINES 89-93 Skipping benchmarks ******************* Another call to :func:`pyquickbench.run_benchmark` will detect that the file exists. The benchmark will not be run the a second time, and the contents of the file is used instead. .. GENERATED FROM PYTHON SOURCE LINES 93-102 .. code-block:: Python all_times_bis = pyquickbench.run_benchmark( all_sizes , all_funs , filename = timings_filename , ) np.all(all_times == all_times_bis) .. rst-class:: sphx-glr-script-out .. code-block:: none np.True_ .. GENERATED FROM PYTHON SOURCE LINES 103-116 Forcing benchmarks ****************** A full re-run can nonetheless be forced if the keyword ``ForceBenchmark`` is set to :data:`python:True`. The default value for ``ForceBenchmark`` is ``False``. .. code-block:: all_times_ter = pyquickbench.run_benchmark( all_sizes , all_funs , filename = timings_filename , ForceBenchmark = True , ) .. GENERATED FROM PYTHON SOURCE LINES 118-122 Detecting obselete benchmarks ***************************** If the file on disk corresponds to a benchmark with more or fewer runs, the whole benchmark is run again, and the contents of the file is updated. For instance, the following will run the benchmark again: .. GENERATED FROM PYTHON SOURCE LINES 122-126 .. code-block:: Python n_bench = 8 all_sizes_small = [2**n for n in range(n_bench)] .. GENERATED FROM PYTHON SOURCE LINES 127-142 .. code-block:: Python all_times_small = pyquickbench.run_benchmark( all_sizes_small , all_funs , filename = timings_filename , ) pyquickbench.plot_benchmark( all_times_small , all_sizes_small , all_funs , show = True , ) .. image-sg:: /_build/auto_examples/tutorial/images/sphx_glr_02-Caching_benchmarks_002.png :alt: 02 Caching benchmarks :srcset: /_build/auto_examples/tutorial/images/sphx_glr_02-Caching_benchmarks_002.png :class: sphx-glr-single-img .. GENERATED FROM PYTHON SOURCE LINES 150-151 We can check that the two benchmarks have non-matching shapes. .. GENERATED FROM PYTHON SOURCE LINES 151-155 .. code-block:: Python print(f'Initial benchmark shape {all_times.shape}') print(f'Current benchmark shape {all_times_small.shape}') .. rst-class:: sphx-glr-script-out .. code-block:: none Initial benchmark shape (12, 3, 1, 1) Current benchmark shape (8, 3, 1, 1) .. GENERATED FROM PYTHON SOURCE LINES 156-158 This mechanism can easily be tricked as it only checks for array dimensions and not content. Indeed, ``all_sizes`` is not stored in the ``*.npy`` benchmark file, and the test only relies on file size. .. GENERATED FROM PYTHON SOURCE LINES 158-163 .. code-block:: Python all_sizes_lin = [1+n for n in range(n_bench)] print(len(all_sizes_lin) == len(all_sizes_small)) print(all([size_lin == size_small for size_lin, size_small in zip(all_sizes_lin, all_sizes_small)])) .. rst-class:: sphx-glr-script-out .. code-block:: none True False .. GENERATED FROM PYTHON SOURCE LINES 164-165 Hence, the following will not trigger a new benchmark although it should, as the sizes are not stored in the ``*.npy`` benchmark file. **As a consequence, the following plot is inaccurate**. .. GENERATED FROM PYTHON SOURCE LINES 165-173 .. code-block:: Python pyquickbench.run_benchmark( all_sizes_lin , all_funs , filename = timings_filename , show = True , ) .. image-sg:: /_build/auto_examples/tutorial/images/sphx_glr_02-Caching_benchmarks_003.png :alt: 02 Caching benchmarks :srcset: /_build/auto_examples/tutorial/images/sphx_glr_02-Caching_benchmarks_003.png :class: sphx-glr-single-img .. GENERATED FROM PYTHON SOURCE LINES 174-175 Using a ``*.npz`` file as an output allows :func:`pyquickbench.run_benchmark` to detect this change. For instance, both outputs here are correct. .. GENERATED FROM PYTHON SOURCE LINES 175-186 .. code-block:: Python timings_filename = "My_benchmark_file.npz" pyquickbench.run_benchmark( all_sizes_small , all_funs , filename = timings_filename , show = True , ) .. image-sg:: /_build/auto_examples/tutorial/images/sphx_glr_02-Caching_benchmarks_004.png :alt: 02 Caching benchmarks :srcset: /_build/auto_examples/tutorial/images/sphx_glr_02-Caching_benchmarks_004.png :class: sphx-glr-single-img .. GENERATED FROM PYTHON SOURCE LINES 195-204 .. code-block:: Python pyquickbench.run_benchmark( all_sizes_lin , all_funs , filename = timings_filename , show = True , ) .. image-sg:: /_build/auto_examples/tutorial/images/sphx_glr_02-Caching_benchmarks_005.png :alt: 02 Caching benchmarks :srcset: /_build/auto_examples/tutorial/images/sphx_glr_02-Caching_benchmarks_005.png :class: sphx-glr-single-img .. _sphx_glr_download__build_auto_examples_tutorial_02-Caching_benchmarks.py: .. only:: html .. container:: sphx-glr-footer sphx-glr-footer-example .. container:: sphx-glr-download sphx-glr-download-jupyter :download:`Download Jupyter notebook: 02-Caching_benchmarks.ipynb <02-Caching_benchmarks.ipynb>` .. container:: sphx-glr-download sphx-glr-download-python :download:`Download Python source code: 02-Caching_benchmarks.py <02-Caching_benchmarks.py>` .. container:: sphx-glr-download sphx-glr-download-zip :download:`Download zipped: 02-Caching_benchmarks.zip <02-Caching_benchmarks.zip>` .. only:: html .. rst-class:: sphx-glr-signature `Gallery generated by Sphinx-Gallery `_