Since many PyTorch kernels are open-source (or even custom written by the user, as in CUDA Extensions), this provides the user with information that helps root cause performance issues and prioritize optimization work. Other useful information might include knowing that a particular kernel did not exploit much thread parallelism, as determined by the grid/block dimensions. In fact, PyProf comes with a flag that lets the user obtain information regarding whether Tensor Cores were used by the kernel. For instance, according to the Tensor Core Performance Guide, the M, N and K dimensions that result in Tensor Core usage need to be divisible by 8. For more details, see NVIDIA's Deep Learning Performance Guide.Īrmed with such information, the user can determine various issues to help them tune the network. Note that these numbers are based on the algorithm, not the actual performance of the specific kernel. For example, for matrices A MxK and B KxN, the FLOP count for a matrix multiplication is 2 * M * N * K, and bandwidth is M * K + N * K + M * N. Regarding FLOP and bandwidth implementations, these are usually quite straightforward. In addition, extra information from the profile is added for use by CUDA professionals, such as CUDA launch parameters (block/grid dimensions). Querying the record produced by the profiler to correlate the kernel name and duration with PyTorch API/layer name, tensor dimensions, tensor precision, as well as calculating FLOPs and bandwidth for common operations.This information is recorded at profile capture time, e.g. Instrumenting PyTorch operations to capture the tensor dimensions and precision using NVTX.PyProf addresses all of the issues above by:
NVPROF CUDALAUNCH CODE
Which line in the user's code resulted in launching this particular kernel (program trace)?.Forward-backward correlation: currently it's very hard to determine what the forward pass step was that resulted in the particular weight and data gradients (wgrad, dgrad), which makes it difficult to determine the tensor dimensions required by these backprop steps to assess their performance.Knowing the tensor dimensions and precision, we can figure out the FLOPs and bandwidth required by a layer, and then determine how close to maximum performance the kernel is for that operation. What the tensor dimensions and precision were: without knowing the tensor dimensions and precision, it's impossible to reason about whether the actual (silicon) kernel time is close to maximum performance of such a kernel on the GPU.the association of ComputeOffsetsKernel with a concrete PyTorch layer or API is not obvious. Getting kernels out of NVProf or NSight Compute provides some generic kernel names and execution times, but not detailed information regarding the following: PyProf2 - PyTorch Profiling tool What does this tool do?Īnalyzing the performance of deep neural networks is hard.
0 kommentar(er)
