# Profile Explorer

Profile Explorer is a script that makes analyzing profile outputs generated by ONNX Runtime a breeze. Simply point the script at a generated .json file and gain visibility into the aspects of the model that are taking the most time.

## Usage

### Basic statistics over operators and kernels

For the most up-to-date options, please run `python ./profile_explorer.py --help` to see a list of all options. Here are some of the common use cases covered:

Find the top-5 operators that are taking the most amount of time:
```bash
python ./profile_explorer.py --count 5 <JSON file containing profiling data>
```

Perform a shape sensitive version of the previous command:
```bash
python ./profile_explorer.py --count 5 --shape-sensitive <JSON file containing profiling data>
```

Be sensitive to GPU kernel dimensions while performing the top-5 analysis, in addition to kernel input shapes:
```bash
python ./profile_explorer.py --count 5 --shape-sensitive --dimension-sensitive <JSON file containing profiling data>
```

Use a custom C++ name demangler to produce readable outputs for GPU kernel names:
```bash
python ./profile_explorer.py --count 5 --demangler </path/to/your/favorite/demangler> <JSON file containing profiling data>
```

Save output to CSV files with names prefixes `profile_output`
```bash
python ./profile_explorer.py --count 5 --csv profile_output <JSON file containing profiling data>
```

An example run and output from the run:
```
python3 ./profile_explorer.py onnxruntime_profile__2022-10-23_07-01-30.json --filter '*Mul*'

------ Top CPU Kernel Times ------
       name  duration   pct  count  cumulative_pct  cumulative_dur
     MatMul    321596 98.16    924           98.16          321596
FusedMatMul      5670  1.73    132           99.89          327266
        Mul       369  0.11     11          100.00          327635

------ Top GPU Kernel Times ------
                                                                                                                                                                                                                                                                                                                                                                                                                                                              name  duration   pct  count  cumulative_pct  cumulative_dur
   Cijk_Ailk_Bljk_HHS_BH_MT256x128x16_MI32x32x8x1_SN_1LDSB0_APM1_ABV0_ACED0_AF0EM1_AF1EM1_AMAS3_ASGT_ASAE01_ASCE01_ASEM1_AAC0_BL1_DTL0_DTVA0_DVO0_ETSP_EPS0_FL0_GRVW8_GSU1_GSUASB_GLS0_ISA90a_IU1_K1_KLA_LBSPP128_LPA0_LPB8_LDL1_LRVW4_LWPMn1_LDW0_MAC_MIAV0_MDA2_NTC0_NTD0_NEPBS0_NLCA1_NLCB1_ONLL1_OPLV0_PK0_PAP0_PGR2_PLR1_RK0_SIA3_SS1_SU32_SUM3_SUS64_SCIUI1_SPO0_SRVW0_SSO0_SVW2_SNLL0_TT4_64_TLDS1_USFGROn1_VAW2_VSn1_VW2_WSGRA0_WSGRB0_WS64_WG64_4_1_WGM16    491799 32.53    660           32.53          491799
     Cijk_Ailk_Bljk_HHS_BH_MT128x256x32_MI32x32x8x1_SE_1LDSB1_APM1_ABV0_ACED0_AF0EM8_AF1EM1_AMAS3_ASGT_ASAE01_ASCE01_ASEM1_AAC0_BL1_DTL0_DTVA0_DVO0_ETSP_EPS0_FL0_GRVW8_GSU1_GSUAMB_GLS0_ISA90a_IU1_K1_KLA_LBSPP128_LPA0_LPB8_LDL1_LRVW8_LWPMn1_LDW0_MAC_MIAV0_MDA2_NTC0_NTD0_NEPBS0_NLCA1_NLCB1_ONLL1_OPLV0_PK0_PAP0_PGR2_PLR5_RK0_SIA3_SS0_SU0_SUM0_SUS0_SCIUI1_SPO0_SRVW4_SSO0_SVW4_SNLL0_TT2_128_TLDS1_USFGROn1_VAW2_VSn1_VW2_WSGRA1_WSGRB1_WS64_WG64_4_1_WGM1    230208 15.23    132           47.76          722007
                                                                                                                                                                                                                                                                                          void onnxruntime::contrib::rocm::BiasSoftmaxWarpForward<__half, __half, float, 9, true>(__half*, __half const*, __half const*, int, int, onnxruntime::rocm::fast_divmod)    218718 14.47    132           62.23          940725
                                                                                                                                                                                                                                                                                                                                              void onnxruntime::contrib::rocm::FastGeluKernelVec<__half, 256u, 1>(int, int, __half const*, __half const*, __half*)    137577  9.10    132           71.33         1078302
       Cijk_Ailk_Bljk_HHS_BH_MT64x128x32_MI32x32x8x1_SE_1LDSB0_APM1_ABV0_ACED0_AF0EM8_AF1EM1_AMAS3_ASGT_ASAE01_ASCE01_ASEM1_AAC0_BL1_DTL0_DTVA0_DVO0_ETSP_EPS1_FL0_GRVW8_GSU1_GSUAMB_GLS0_ISA90a_IU1_K1_KLA_LBSPP128_LPA0_LPB8_LDL1_LRVW8_LWPMn1_LDW0_MAC_MIAV0_MDA2_NTC0_NTD0_NEPBS0_NLCA1_NLCB1_ONLL1_OPLV0_PK0_PAP0_PGR1_PLR5_RK0_SIA3_SS0_SU0_SUM0_SUS0_SCIUI1_SPO0_SRVW4_SSO0_SVW4_SNLL0_TT1_64_TLDS1_USFGROn1_VAW2_VSn1_VW2_WSGRA1_WSGRB1_WS64_WG64_4_1_WGM1    103875  6.87    132           78.20         1182177
                                                                       void onnxruntime::rocm::_BinaryElementWise<__half, __half, __half, onnxruntime::rocm::OP_Add<__half, __half, __half>, true, false, 512, 2>(int, onnxruntime::rocm::TArray<long, 8>, __half const*, onnxruntime::rocm::TArray<long, 8>, __half const*, onnxruntime::rocm::TArray<onnxruntime::rocm::fast_divmod, 8>, __half*, onnxruntime::rocm::OP_Add<__half, __half, __half> const&, int)     93086  6.16    396           84.36         1275263
Cijk_Ailk_Bljk_HHS_BH_MT128x144x32_MI16x16x16x1_SE_1LDSB1_APM1_ABV0_ACED0_AF0EM8_AF1EM1_AMAS3_ASGT_ASAE01_ASCE01_ASEM1_AAC0_BL1_DTL0_DTVA0_DVO0_ETSP_EPS1_FL0_GRVW2_GSU1_GSUAMB_GLS0_ISA90a_IU1_K1_KLA_LBSPP128_LPA0_LPB16_LDL1_LRVW8_LWPMn1_LDW0_MAC_MIAV0_MDA2_NTC0_NTD0_NEPBS0_NLCA1_NLCB1_ONLL1_OPLV0_PK0_PAP0_PGR1_PLR2_RK0_SIA3_SS0_SU32_SUM0_SUS256_SCIUI1_SPO0_SRVW4_SSO0_SVW4_SNLL0_TT2_144_TLDS1_USFGROn1_VAW2_VSn1_VW2_WSGRA1_WSGRB1_WS64_WG64_4_1_WGM1     74188  4.91    132           89.27         1349451
                                                                                                                                                                                                                                                                                 void onnxruntime::contrib::rocm::SkipLayerNormKernelSmall<__half, 192u, 4>(int, __half const*, __half const*, __half const*, __half const*, __half const*, __half, __half*, bool)     62288  4.12    264           93.39         1411739
                                                                                                                                                                                                                                                             void onnxruntime::rocm::Transpose4DKernelParallelizeMultipleElementsPerThreadInInnermostDim<2>(onnxruntime::rocm::TArray<long, 8>, void const*, onnxruntime::rocm::TArray<long, 8>, void*, long, int)     47324  3.13    396           96.52         1459063
                                                                                                                                                                                                                                                                                                               void onnxruntime::rocm::Transpose3DKernel<short, 32u>(onnxruntime::rocm::TArray<long, 8>, onnxruntime::rocm::TArray<long, 8>, short const*, short*)     19311  1.28    132           97.80         1478374
                                                                                                                                                                                                                                                                                                                                                                                                                                                  CopyHostToDevice     14215  0.94    206           98.74         1492589
                                                                                                                                                                                                                                                                                   void onnxruntime::rocm::_GatherKernel<short>(long, long, onnxruntime::rocm::fast_divmod, onnxruntime::rocm::fast_divmod, void const*, unsigned long, short const*, short*, int)      6816  0.45     44           99.19         1499405
                                                                                                                                                                                                                                                                                                     void onnxruntime::rocm::cuApplyLayerNorm<__half, float, __half, false>(__half*, float*, float*, __half const*, int, int, float, __half const*, __half const*)      3477  0.23     11           99.42         1502882
                                                                       void onnxruntime::rocm::_BinaryElementWise<__half, __half, __half, onnxruntime::rocm::OP_Add<__half, __half, __half>, false, true, 512, 2>(int, onnxruntime::rocm::TArray<long, 8>, __half const*, onnxruntime::rocm::TArray<long, 8>, __half const*, onnxruntime::rocm::TArray<onnxruntime::rocm::fast_divmod, 8>, __half*, onnxruntime::rocm::OP_Add<__half, __half, __half> const&, int)      2599  0.17     11           99.59         1505481
                                                                                                                                                                                                                                                                                void onnxruntime::rocm::_UnaryElementWise<__half, float, onnxruntime::rocm::OP_Cast<__half, float>, 256, 4>(__half const*, float*, onnxruntime::rocm::OP_Cast<__half, float>, int)      2161  0.14     22           99.74         1507642
                                                                                                                                                                                                             void onnxruntime::rocm::_BinaryElementWiseSimple<true, true, __half, __half, __half, onnxruntime::rocm::OP_Add<__half, __half, __half>, 512, 2>(__half const*, __half const*, __half*, onnxruntime::rocm::OP_Add<__half, __half, __half> const&, int)      2037  0.13     11           99.87         1509679
                                                                                                                                                                                                                                                                                                                                                                                                                                                  CopyDeviceToHost      1329  0.09      2           99.96         1511008
    Cijk_Alik_Bljk_HHS_BH_MT64x32x64_MI16x16x16x1_SE_1LDSB0_APM1_ABV0_ACED0_AF0EM8_AF1EM1_AMAS3_ASGT_ASAE01_ASCE01_ASEM1_AAC0_BL1_DTL0_DTVA0_DVO0_ETSP_EPS0_FL0_GRVW8_GSU1_GSUAMB_GLS0_ISA90a_IU1_K1_KLA_LBSPP128_LPA16_LPB16_LDL1_LRVW8_LWPMn1_LDW0_MAC_MIAV0_MDA2_NTC0_NTD0_NEPBS0_NLCA1_NLCB1_ONLL1_OPLV0_PK0_PAP0_PGR2_PLR5_RK0_SIA3_SS0_SU0_SUM0_SUS0_SCIUI1_SPO0_SRVW4_SSO0_SVW4_SNLL0_TT2_16_TLDS1_USFGROn1_VAW2_VSn1_VW2_WSGRA1_WSGRB1_WS64_WG32_8_1_WGM15       152  0.01     11           99.97         1511160
                                                                                                                                                                                                   void onnxruntime::rocm::_SliceKernel<false, 2, 256, 4, long>(onnxruntime::rocm::TArray<long, 8>, onnxruntime::rocm::TArray<long, 8>, onnxruntime::rocm::TArray<long, 8>, onnxruntime::rocm::TArray<onnxruntime::rocm::fast_divmod, 8>, long const*, long*, int)       109  0.01     22           99.98         1511269
                                                                                                                                                                                                                                                                                                                                                   void onnxruntime::rocm::ExpandKernel2D<long>(int, long const*, long*, onnxruntime::rocm::fast_divmod, int, int)        89  0.01     11           99.98         1511358
     Cijk_Ailk_Bljk_HHS_BH_MT64x16x32_MI16x16x16x1_SE_1LDSB0_APM1_ABV0_ACED0_AF0EM8_AF1EM1_AMAS3_ASGT_ASAE01_ASCE01_ASEM1_AAC0_BL1_DTL0_DTVA0_DVO0_ETSP_EPS0_FL0_GRVW8_GSU1_GSUAMB_GLS0_ISA90a_IU1_K1_KLA_LBSPP128_LPA0_LPB16_LDL1_LRVW8_LWPMn1_LDW0_MAC_MIAV0_MDA2_NTC0_NTD0_NEPBS0_NLCA1_NLCB1_ONLL1_OPLV0_PK0_PAP0_PGR2_PLR2_RK0_SIA3_SS0_SU0_SUM0_SUS0_SCIUI1_SPO0_SRVW0_SSO0_SVW4_SNLL0_TT1_16_TLDS1_USFGROn1_VAW2_VSn1_VW2_WSGRA1_WSGRB1_WS64_WG64_4_1_WGM15        83  0.01     11           99.99         1511441
                                                                                                                                                                                                                                                                                                                                                                                                  void onnxruntime::rocm::_Fill<short, 256, 4>(short*, short, int)        52  0.00     11           99.99         1511493
                                                                                                                                                                                                                                                                                            void onnxruntime::rocm::_UnaryElementWise<__half, __half, onnxruntime::rocm::OP_Tanh<__half>, 256, 4>(__half const*, __half*, onnxruntime::rocm::OP_Tanh<__half>, int)        51  0.00     11           99.99         1511544
                                                                                                                                                                                                            void onnxruntime::rocm::_BinaryElementWiseSimple<true, false, __half, __half, __half, onnxruntime::rocm::OP_Mul<__half, __half, __half>, 512, 2>(__half const*, __half const*, __half*, onnxruntime::rocm::OP_Mul<__half, __half, __half> const&, int)        46  0.00     11          100.00         1511590
                                                                                                                                                                                                            void onnxruntime::rocm::_BinaryElementWiseSimple<false, true, __half, __half, __half, onnxruntime::rocm::OP_Sub<__half, __half, __half>, 512, 2>(__half const*, __half const*, __half*, onnxruntime::rocm::OP_Sub<__half, __half, __half> const&, int)        45  0.00     11          100.00         1511635
                                                                                                                                                                                                                                                                                                                                                                                               void onnxruntime::rocm::_Fill<__half, 256, 4>(__half*, __half, int)         4  0.00      1          100.00         1511639
```

### Operator-kernel correlation statistics

We provide an optional argument `--mapping`/`-m` to turn on operator-kernel correlation analysis.
This feature by default groups and counts operators by their names and input info (shape and type).

The output `.csv` file covers the occurrences of operators and kernels as well as kernel duration,
in an averaged model run, so note that in some corner cases the occurrences can be decimal.
Each operator with input info is expanded to several rows of kernels it calls if there are multiple kernels.
Numbers include and are not limited to count, duration, percentage.

Following is a simple example of the output file (some entries are omitted or simplified for brevity):

| op_name            | input_type_shape         | op_count | kernel_dims          | kernel_count | kernel_avg_dur | ... | op_pct | kernel_name    |
|--------------------|--------------------------|---------:|----------------------|-------------:|---------------:|-----|-------:|----------------|
| MultiHeadAttention | float16(2x4096x8x3x40)   | 5        | b256x1x1,g512x1x1    | 5            | 1734.85        | ... | 23.26  | some_ck_kernel |
| GroupNorm          | float16(2x64x64x320),... | 13       | b1024x1x1,g64x1x1    | 8            | 121.52         | ... | 4.28   | norm_kernel_1  |
| GroupNorm          | float16(2x64x64x320),... | 13       | b256x1x1,g64x1x1     | 5            | 124.49         | ... | 4.28   | norm_kernel_2  |
| NhwcConv           | float16(2x64x64x320),... | 8        | b256x1x1,g163840x1x1 | 7            | 178.51         | ... | 4.00   | igemm_kernel_1 |
| NhwcConv           | float16(2x64x64x320),... | 8        | b512x1x1,g2560x1x1   | 7            | 13.27          | ... | 4.00   | OpAdd_kernel_1 |
| NhwcConv           | float16(2x64x64x320),... | 8        | b256x1x1,g65536x1x1  | 8            | 9.88           | ... | 4.00   | SubTensorOp... |
| NhwcConv           | float16(2x64x64x320),... | 8        | b256x1x1,g40960x1x1  | 1            | 64.03          | ... | 4.00   | igemm_kernel_2 |
| NhwcConv           | float16(2x64x64x320),... | 8        | b512x1x1,g640x1x1    | 1            | 4.97           | ... | 4.00   | OpAdd_kernel_2 |
| ...                | ...                      | ...      | ...                  | ...          | ...            | ... | ...    | ...            |

From the above table, we see that the operator `MultiHeadAttention` with input info `float16(2x4096x8x3x40)`
has 5 occurrences in an model run (on average), and they all call the same kernel `some_ck_kernel`. Therefore,
the kernel `some_ck_kernel` has 5 occurrences in total.

There are cases where operators with the same input info call different kernels. For example, the operator `GroupNorm`
with input info `float16(2x64x64x320),...` has 13 occurrences in an model run, but 8 of them call the kernel
`norm_kernel_1` while the other 5 call the kernel `norm_kernel_2`.

Still, often there are cases where an operator calls multiple kernels. For example, the operator `NhwcConv` with
input info `float16(2x64x64x320),...` has 8 occurrences in an model run, but 7 of them call kernels `igemm_kernel_1`,
`SubTensorOp...` and `OpAdd_kernel_1` respectively, while the other one calls kernels `igemm_kernel_2`,
`SubTensorOp...` and `OpAdd_kernel_2`, resulting in the above five rows of this operator.
