Support Nvidia Hopper GPUs

[giordano]

This is an initial attempt to support Nvidia Hopper GPUs, opening as draft because lots of thing still don't work. For example, theoretical peakflops for tensor cores is wrong, it looks like the formula used for A100 doesn't apply to Hopper. I tried to adapt based on figures 10-11 of https://resources.nvidia.com/en-us-tensor-core/gtc22-whitepaper-hopper but with GH200 I get:

julia> for tensorcores in (false, true), dtype in (Float64, Float32, Float16, Int8)
           dtype in (Int8, Float16) && !tensorcores || theoretical_peakflops_gpu(; dtype, tensorcores)
       end
Theoretical Peakflops (TFLOP/s):
 ├ tensorcores: false
 ├ dtype: Float64
 └ max: 33.5
Theoretical Peakflops (TFLOP/s):
 ├ tensorcores: false
 ├ dtype: Float32
 └ max: 66.9
Theoretical Peakflops (TFLOP/s):
 ├ tensorcores: true
 ├ dtype: Float64
 └ max: 66.9
Theoretical Peakflops (TFLOP/s):
 ├ tensorcores: true
 ├ dtype: Float32
 └ max: 535.3
Theoretical Peakflops (TFLOP/s):
 ├ tensorcores: true
 ├ dtype: Float16
 └ max: 1070.5
Theoretical Peakflops (TOP/s):
 ├ tensorcores: true
 ├ dtype: Int8
 └ max: 4282.1

Values for Float64 (with and without tensorcores) and Float32 (without tensorcores) are good, but all other tensorcores peakflops are wrong according to column "H100 SXM5" table 2 of the document above, it should be 494.7 TFLOP/s for Float32, 989.4 for Float16, and 1978.9 for Int8 (also https://resources.nvidia.com/en-us-grace-cpu/grace-hopper-superchip which is specific to GH200 agrees with those numbers, but it has fewer significant digits, they rounded to integers).

[giordano]

Note: I replaced this link with the DOI because the link is now broken.

[giordano]

Trying to measure peakflops, I get

julia> for tensorcores in (false, true), dtype in (Float64, Float32, Float16, Int8)
           (dtype in (Int8, Float16) && !tensorcores) || (dtype in (Float64, Float32) && tensorcores) || GPUInspector.peakflops_gpu(; dtype, tensorcores)
       end
Peakflops (TFLOP/s):
 ├ tensorcores: false
 ├ dtype: Float64
 └ max: 22.3
Peakflops (TFLOP/s):
 ├ tensorcores: false
 ├ dtype: Float32
 └ max: 32.3
Peakflops (TFLOP/s):
 ├ tensorcores: true
 ├ dtype: Float16
 └ max: 633.2
Peakflops (TOP/s):
 ├ tensorcores: true
 ├ dtype: Int8
 └ max: 940.9

These results are quite far from the theoretical peaks, about 50% less, is there anything to tweak in the kernels for a new architecture?

[giordano]

Interestingly enough, all theoretical tensorcores peakflops for Float32, Float16, and Int8 are wrong by about 8%

julia> 535.3 / 494.7
1.0820699413786132

julia> 1070.5 / 989.4
1.0819688700222356

julia> (4282.1 / 2) / 1978.9
1.0819394613168933

but I have no clue of where this factor comes from.

[giordano]

Maybe there's an extra factor of 2 in this formula, but I based this on the Int8 calculation below