.. DO NOT EDIT. .. THIS FILE WAS AUTOMATICALLY GENERATED BY SPHINX-GALLERY. .. TO MAKE CHANGES, EDIT THE SOURCE PYTHON FILE: .. "_build/auto_examples/tutorial/07-Multidimensional_benchmarks.py" .. LINE NUMBERS ARE GIVEN BELOW. .. only:: html .. note:: :class: sphx-glr-download-link-note :ref:`Go to the end ` to download the full example code. .. rst-class:: sphx-glr-example-title .. _sphx_glr__build_auto_examples_tutorial_07-Multidimensional_benchmarks.py: Multidimensional benchmarks =========================== .. GENERATED FROM PYTHON SOURCE LINES 7-11 One of pyquickbench's strengths is its ability to run multidimensional benchmarks to test function behavior changes with respect to several different arguments, or to assess repeatability of a benchmark. For instance, let's run the following benchmark a thousand times. .. GENERATED FROM PYTHON SOURCE LINES 11-53 .. code-block:: Python import pyquickbench def comprehension(n): return ['' for _ in range(n)] def star_operator(n): return ['']*n def for_loop_append(n): l = [] for _ in range(n): l.append('') all_funs = [ comprehension , star_operator , for_loop_append , ] n_bench = 12 all_sizes = [2**n for n in range(n_bench)] n_repeat = 1000 time_per_test = 0.2 all_values = pyquickbench.run_benchmark( all_sizes , all_funs , n_repeat = n_repeat , time_per_test = time_per_test , filename = timings_filename , ) pyquickbench.plot_benchmark( all_values , all_sizes , all_funs , show = True , ) .. image-sg:: /_build/auto_examples/tutorial/images/sphx_glr_07-Multidimensional_benchmarks_001.png :alt: 07 Multidimensional benchmarks :srcset: /_build/auto_examples/tutorial/images/sphx_glr_07-Multidimensional_benchmarks_001.png :class: sphx-glr-single-img .. GENERATED FROM PYTHON SOURCE LINES 81-83 By default, only the minminum timing is reported on the plot as recommended by :meth:`python:timeit.Timer.repeat`. This being said, the array ``all_values`` does contain ``n_repeat`` timings. .. GENERATED FROM PYTHON SOURCE LINES 83-88 .. code-block:: Python print(all_values.shape[0] == len(all_sizes)) print(all_values.shape[1] == len(all_funs)) print(all_values.shape[2] == n_repeat) .. rst-class:: sphx-glr-script-out .. code-block:: none True True True .. GENERATED FROM PYTHON SOURCE LINES 89-90 All the different timings can be superimposed on the same plot with the following ``plot_intent`` argument: .. GENERATED FROM PYTHON SOURCE LINES 90-106 .. code-block:: Python plot_intent = { pyquickbench.default_ax_name : "points" , pyquickbench.fun_ax_name : "curve_color" , pyquickbench.repeat_ax_name : "same" , } pyquickbench.plot_benchmark( all_values , all_sizes , all_funs , show = True , plot_intent = plot_intent , ) .. image-sg:: /_build/auto_examples/tutorial/images/sphx_glr_07-Multidimensional_benchmarks_002.png :alt: 07 Multidimensional benchmarks :srcset: /_build/auto_examples/tutorial/images/sphx_glr_07-Multidimensional_benchmarks_002.png :class: sphx-glr-single-img .. GENERATED FROM PYTHON SOURCE LINES 107-108 The above plot is quite cluttered. For more concise information, let's use curve transparency: .. GENERATED FROM PYTHON SOURCE LINES 108-118 .. code-block:: Python pyquickbench.plot_benchmark( all_values , all_sizes , all_funs , show = True , plot_intent = plot_intent , alpha = 1./255 , ) .. image-sg:: /_build/auto_examples/tutorial/images/sphx_glr_07-Multidimensional_benchmarks_003.png :alt: 07 Multidimensional benchmarks :srcset: /_build/auto_examples/tutorial/images/sphx_glr_07-Multidimensional_benchmarks_003.png :class: sphx-glr-single-img .. GENERATED FROM PYTHON SOURCE LINES 119-121 The above plot gives a good idea of the concentration of data, but bounds on timing are not very clear. Using reductions in ``plot_intent``, we can for instance choose to plot minimal, median and maximal values. The list of all possible reductions is accessible in :data:`pyquickbench.all_reductions`. .. GENERATED FROM PYTHON SOURCE LINES 121-159 .. code-block:: Python fig, ax = pyquickbench.plot_benchmark( all_values , all_sizes , all_funs , return_empty_plot = True , ) all_reductions = ["reduction_min", "reduction_max", "reduction_median"] all_linestyles = ["dotted", "dashed", "solid"] for ( reduction , linestyle , )in zip( all_reductions , all_linestyles , ): plot_intent = { pyquickbench.default_ax_name : "points" , pyquickbench.fun_ax_name : "curve_color" , pyquickbench.repeat_ax_name : reduction , } pyquickbench.plot_benchmark( all_values , all_sizes , all_funs , plot_intent = plot_intent , linestyle_list = linestyle , fig = fig , ax = ax , ) plt.tight_layout() plt.show() .. image-sg:: /_build/auto_examples/tutorial/images/sphx_glr_07-Multidimensional_benchmarks_004.png :alt: 07 Multidimensional benchmarks :srcset: /_build/auto_examples/tutorial/images/sphx_glr_07-Multidimensional_benchmarks_004.png :class: sphx-glr-single-img .. GENERATED FROM PYTHON SOURCE LINES 160-161 Another way to render data dispersion is to use the ``"violin"`` plot intent. This type of plot shows a `Kernel Density Estimation `_ of the timings distribution as well as extreme measured values for every set of input parameters. .. GENERATED FROM PYTHON SOURCE LINES 161-176 .. code-block:: Python plot_intent = { pyquickbench.default_ax_name : "points" , pyquickbench.fun_ax_name : "curve_color" , pyquickbench.repeat_ax_name : "violin" , } pyquickbench.plot_benchmark( all_values , all_sizes , all_funs , plot_intent = plot_intent , show = True , ) .. image-sg:: /_build/auto_examples/tutorial/images/sphx_glr_07-Multidimensional_benchmarks_005.png :alt: 07 Multidimensional benchmarks :srcset: /_build/auto_examples/tutorial/images/sphx_glr_07-Multidimensional_benchmarks_005.png :class: sphx-glr-single-img .. GENERATED FROM PYTHON SOURCE LINES 177-178 More generally, the ``plot_intent`` argument controls what dimension of the array ``all_values`` is plotted, and in what way. For instance, as a way to better understand the statistics of the measured timings, we can plot the measured time of execution as a function of the index of the repeated benchmark for a single function. .. GENERATED FROM PYTHON SOURCE LINES 178-196 .. code-block:: Python plot_intent = { pyquickbench.default_ax_name : "curve_color" , pyquickbench.fun_ax_name : "single_value" , pyquickbench.repeat_ax_name : "points" , } single_values_val = {pyquickbench.fun_ax_name: "star_operator"} pyquickbench.plot_benchmark( all_values , all_sizes , all_funs , show = True , plot_intent = plot_intent , single_values_val = single_values_val , ) .. image-sg:: /_build/auto_examples/tutorial/images/sphx_glr_07-Multidimensional_benchmarks_006.png :alt: 07 Multidimensional benchmarks :srcset: /_build/auto_examples/tutorial/images/sphx_glr_07-Multidimensional_benchmarks_006.png :class: sphx-glr-single-img .. GENERATED FROM PYTHON SOURCE LINES 197-198 Or for all functions, but a single value of input size. .. GENERATED FROM PYTHON SOURCE LINES 198-214 .. code-block:: Python plot_intent = { pyquickbench.default_ax_name : "reduction_max" , pyquickbench.fun_ax_name : "curve_color" , pyquickbench.repeat_ax_name : "points" , } pyquickbench.plot_benchmark( all_values , all_sizes , all_funs , show = True , plot_intent = plot_intent , ) .. image-sg:: /_build/auto_examples/tutorial/images/sphx_glr_07-Multidimensional_benchmarks_007.png :alt: 07 Multidimensional benchmarks :srcset: /_build/auto_examples/tutorial/images/sphx_glr_07-Multidimensional_benchmarks_007.png :class: sphx-glr-single-img .. GENERATED FROM PYTHON SOURCE LINES 215-216 As can be seen in the above plots, the timings are automatically sorted along the :data:`pyquickbench.repeat_ax_name` axis. The list of all possible ``plot_intent`` values is available in :data:`pyquickbench.all_plot_intents`. .. _sphx_glr_download__build_auto_examples_tutorial_07-Multidimensional_benchmarks.py: .. only:: html .. container:: sphx-glr-footer sphx-glr-footer-example .. container:: sphx-glr-download sphx-glr-download-jupyter :download:`Download Jupyter notebook: 07-Multidimensional_benchmarks.ipynb <07-Multidimensional_benchmarks.ipynb>` .. container:: sphx-glr-download sphx-glr-download-python :download:`Download Python source code: 07-Multidimensional_benchmarks.py <07-Multidimensional_benchmarks.py>` .. container:: sphx-glr-download sphx-glr-download-zip :download:`Download zipped: 07-Multidimensional_benchmarks.zip <07-Multidimensional_benchmarks.zip>` .. only:: html .. rst-class:: sphx-glr-signature `Gallery generated by Sphinx-Gallery `_