.. DO NOT EDIT. .. THIS FILE WAS AUTOMATICALLY GENERATED BY SPHINX-GALLERY. .. TO MAKE CHANGES, EDIT THE SOURCE PYTHON FILE: .. "_build/auto_examples/tutorial/11-Manual_benchmarks.py" .. LINE NUMBERS ARE GIVEN BELOW. .. only:: html .. note:: :class: sphx-glr-download-link-note :ref:`Go to the end ` to download the full example code. .. rst-class:: sphx-glr-example-title .. _sphx_glr__build_auto_examples_tutorial_11-Manual_benchmarks.py: Running benchmarks manually =========================== .. GENERATED FROM PYTHON SOURCE LINES 7-10 While :func:`pyquickbench.run_benchmark` can be very useful to programatically run benchmarks, there are cases where it is either too difficult, or even impossible. This mostyl happens when the measurement involves some kind of manual action from the user. In this case, :mod:`pyquickbench` can still be used to organize and plot the results. In the the following benchmark, the user manually ran a particular AI workload for varying maximal power settings of their GPU from 100% down to 35% in increments of 5%. Each run was repeated 4 times, and the results were measured as follows: .. GENERATED FROM PYTHON SOURCE LINES 10-24 .. code-block:: Python all_args = { 'power_rate' : [1.00, 0.95, 0.90, 0.85, 0.80, 0.75, 0.70, 0.65, 0.60, 0.55, 0.50, 0.45, 0.40, 0.35] } timings_results_manual = [ [ 17.00, 16.04, 15.85, 16.09, 16.65, 16.79, 17.24, 17.47, 17.99, 19.98, 22.25, 25.33, 30.28, 35.96], [ 15.88, 15.59, 15.70, 15.84, 16.11, 16.43, 16.87, 17.18, 17.97, 19.88, 22.31, 25.38, 29.62, 36.12], [ 15.67, 15.57, 15.68, 15.83, 16.10, 16.41, 16.82, 17.35, 17.92, 19.65, 22.06, 25.56, 29.67, 35.18], [ 15.59, 15.57, 15.65, 15.84, 16.24, 16.55, 16.80, 17.33, 17.90, 19.64, 22.04, 25.31, 29.62, 35.67], ] .. GENERATED FROM PYTHON SOURCE LINES 52-53 In order to understand the format required by :mod:`pyquickbench` as its input, let's create a mockup benchmark with similar settings, and run it with the option :class:`return_array_descriptor = True`. .. GENERATED FROM PYTHON SOURCE LINES 53-73 .. code-block:: Python import numpy as np import pyquickbench def timer_fun(power_rate): pass n_repeat = 4 all_args = { 'power_rate' : [1.00, 0.95, 0.90, 0.85, 0.80, 0.75, 0.70, 0.65, 0.60, 0.55, 0.50, 0.45, 0.40, 0.35] } array_descriptor = pyquickbench.run_benchmark( all_args , [timer_fun] , n_repeat = n_repeat , return_array_descriptor = True , ) .. GENERATED FROM PYTHON SOURCE LINES 74-75 The result of this call is a :class:`python:dict` containing the description of the shape of the expected array of results. The order of entries is the same as the order of the axes (remember that :class:`python:dict` are ordered since `Python 3.7 `_), the keys are their names, and their values are their sizes. .. GENERATED FROM PYTHON SOURCE LINES 75-79 .. code-block:: Python print(f'{type(array_descriptor)=}') print(array_descriptor) .. rst-class:: sphx-glr-script-out .. code-block:: none type(array_descriptor)= {'power_rate': 14, 'function': 1, 'repeat': 4, 'out': 1} .. GENERATED FROM PYTHON SOURCE LINES 80-83 .. code-block:: Python print(f'{np.array(timings_results_manual).shape=}') .. rst-class:: sphx-glr-script-out .. code-block:: none np.array(timings_results_manual).shape=(4, 14) .. GENERATED FROM PYTHON SOURCE LINES 84-85 Formatting the user data as as simple as swapping axes and reshaping. .. GENERATED FROM PYTHON SOURCE LINES 85-88 .. code-block:: Python timings_results_pyquickbench = np.array(timings_results_manual).T.reshape([d for (k,d) in array_descriptor.items()]) .. GENERATED FROM PYTHON SOURCE LINES 89-90 The resulting array can then be used as an input to :func:`pyquickbench.plot_benchmark`. .. GENERATED FROM PYTHON SOURCE LINES 90-109 .. code-block:: Python plot_intent = { 'power_rate' : 'points' , 'repeat' : 'reduction_min', } pyquickbench.plot_benchmark( timings_results_pyquickbench , all_args , [timer_fun] , logx_plot = False , logy_plot = False , plot_intent = plot_intent , title = 'Time of computation compared to 100% power' , xlabel = "Power rate" , ylabel = "Time (s)" , show = True , ) .. image-sg:: /_build/auto_examples/tutorial/images/sphx_glr_11-Manual_benchmarks_001.png :alt: Time of computation compared to 100% power :srcset: /_build/auto_examples/tutorial/images/sphx_glr_11-Manual_benchmarks_001.png :class: sphx-glr-single-img .. GENERATED FROM PYTHON SOURCE LINES 110-131 .. code-block:: Python plot_intent = { 'power_rate' : 'points' , 'repeat' : 'reduction_max', } freq_results = 1./timings_results_pyquickbench pyquickbench.plot_benchmark( freq_results , all_args , [timer_fun] , logx_plot = False , logy_plot = False , plot_intent = plot_intent , title = 'Update frequency compared to 100% power' , xlabel = "Power rate" , ylabel = "Update frequency (it/s)" , show = True , ) .. image-sg:: /_build/auto_examples/tutorial/images/sphx_glr_11-Manual_benchmarks_002.png :alt: Update frequency compared to 100% power :srcset: /_build/auto_examples/tutorial/images/sphx_glr_11-Manual_benchmarks_002.png :class: sphx-glr-single-img .. GENERATED FROM PYTHON SOURCE LINES 132-137 The plots above show that : * Speed is an increasing function of power consumption as expected * Growth is quickest for low power rates, wit an inflexion point at ~60% and a threshold near 90%. From this information, and using an estimation of the baseline power of the rest of the system can be inferred a power rate setting for minimum energy consumption. This estimate lies around 60% max power. .. GENERATED FROM PYTHON SOURCE LINES 137-157 .. code-block:: Python max_pow = 450 baseline_pow = 60 energy = np.empty_like(timings_results_pyquickbench) for i, pwr_rate in enumerate(all_args['power_rate']): energy[i,...] = (pwr_rate * max_pow + baseline_pow) * timings_results_pyquickbench[i,...] pyquickbench.plot_benchmark( energy , all_args , [timer_fun] , logx_plot = False , logy_plot = False , plot_intent = plot_intent , title = 'Total energy consumption of the computation' , xlabel = "Power rate" , relative_to_val = {'power_rate': 1.}, show = True , ) .. image-sg:: /_build/auto_examples/tutorial/images/sphx_glr_11-Manual_benchmarks_003.png :alt: Total energy consumption of the computation :srcset: /_build/auto_examples/tutorial/images/sphx_glr_11-Manual_benchmarks_003.png :class: sphx-glr-single-img .. _sphx_glr_download__build_auto_examples_tutorial_11-Manual_benchmarks.py: .. only:: html .. container:: sphx-glr-footer sphx-glr-footer-example .. container:: sphx-glr-download sphx-glr-download-jupyter :download:`Download Jupyter notebook: 11-Manual_benchmarks.ipynb <11-Manual_benchmarks.ipynb>` .. container:: sphx-glr-download sphx-glr-download-python :download:`Download Python source code: 11-Manual_benchmarks.py <11-Manual_benchmarks.py>` .. container:: sphx-glr-download sphx-glr-download-zip :download:`Download zipped: 11-Manual_benchmarks.zip <11-Manual_benchmarks.zip>` .. only:: html .. rst-class:: sphx-glr-signature `Gallery generated by Sphinx-Gallery `_