Getting the GPU utilization of NVIDIA playing cards with the Linux dstat instrument

The dstat is an superior little instrument which lets you get useful resource statistics on your Linux field. It has a modular structure which lets you develop further plugins and it’s straightforward to make use of. Not too long ago I used to be profiling a Deep Studying pipeline developed with Keras and Tensorflow and I wanted detailed statistics in regards to the CPU, Arduous Disk and GPU utilization. The primary two can be found out-of-the-box by dstat, however so far as I do know there is no such thing as a plugin for monitoring GPU utilization for NVIDIA graphics playing cards.

Fortunately it’s tremendous straightforward to write down a python plugin for dstat. I’ve already despatched a pull-request on the official repo however since new variations are launched comparatively hardly ever listed below are some directions on methods to arrange the dstat NVIDIA GPU utilization plugin in your field.

Set up

The next instructions are examined on Ubuntu 16.04 and they’ll provide help to set up dstat, the Python NVIDIA Administration Library and my dstat nvidia plugin:

sudo apt-get set up dstat #set up dstat
sudo pip set up nvidia-ml-py #set up Python NVIDIA Administration Library
wget https://uncooked.githubusercontent.com/datumbox/dstat/grasp/plugins/dstat_nvidia_gpu.py
sudo mv dstat_nvidia_gpu.py /usr/share/dstat/ #transfer file to the plugins listing of dstat

To get all of the default statistics together with GPU utilization (proportion) kind the next command:

dstat -a --nvidia-gpu

----total-cpu-usage---- -dsk/total- -net/total- ---paging-- ---system-- gpu-u
usr sys idl wai hiq siq| learn  writ| recv  ship|  in   out | int   csw |complete
  2   1  96   0   0   0|5816k   15M|   0     0 |   0     0 |  45k   98k|   68
  0   1  98   0   0   0|  57M  128k| 104B  902B|   0     0 |  42k   85k|   50
  8   7  84   1   0   0| 152M    0 | 292B  448B|   0     0 |  52k   93k|   39
  1   1  97   1   0   0| 111M    0 |  52B  374B|   0     0 |  51k  116k|   62
  0   1  98   1   0   0| 129M    0 |  80B  416B|   0     0 |  43k   85k|   92
  0   2  98   0   0   0|   0     0 |  52B  374B|   0     0 |  41k   83k|   81

To get all of the utilization statistics for every GPU use the next command:

dstat --nvidia-gpu -f
-------------------------------------------gpu-usage-nvidia------------------------------------------
complete  gpu0  gpu1  gpu2  gpu3  gpu4  gpu5  gpu6  gpu7  gpu8  gpu9 gpu10 gpu11 gpu12 gpu13 gpu14 gpu15
   19    23    22    21    21    20    22    23    25    15    18    16    16    16    18    16    14
   18    21    20    18    22    21    21    22    21    15    15    14    14    14    15    16    13
   10    14     9    13     8     9    11     9    12     9     9    10    10     8     7     9     9
   18    20    22    19    21    20    21    21    22    14    15    14    15    14    15    15    15
   20    24    22    23    24    25    22    22    22    16    16    16    16    16    16    18    16
   15    21    18    19    18    17    17    16    18    14    13    13    14    13    12    11    11
   20    24    22    22    24    25    23    24    22    16    18    16    14    17    17    17    15
   19    29    18    23    21    22    21    20    21    18    16    16    18    14    14    17    17

The way it works

The plugin fetches the variety of accessible GPUs on the system and samples 10 instances the utilization metric for every GPU. Sampling a number of instances will hopefully return smoother metrics than getting a single measurement. After that it averages the utilization throughout all GPUs and returns the outcomes to the person. The supply code of the plugin is accessible right here.

Hope you get pleasure from it, completely satisfied GPU programming! 🙂