To anyone who wants to further increase streaming quality using NVENC, here's something you can try:
In FFmpeg options, you can add these options:
-qcomp=0.8 -trellis=2 -subq=10
-pre_dia_size=768 -dia_size=1024 -me_range=32
-precmp=sad -cmp=+chroma+rd -subcmp=+rd -ildctcmp=+rd -mbcmp=+chroma+rd -mbd=simple
-dct=int -idct=int
-flags=+loop -err_detect=+crccheck+bitstream+buffer -ec=+guess_mvs+deblock+favor_inter
These are
FFmpeg global encoder options that can apply on top of native encoder options.
-qcomp=0.8
Standard qcomp option. 0.8 to give more headroom for temporal AQ.
-trellis=2
Good old trellis. RDO for DCT rounding.
0: disabled
1: enabled only on the final encode of a MB
2: enabled on all mode decisions
-subq=10
Subpixel motion estimation refinement, same as in x264. Higher the value better the quality (and encoding time). 1 to 11.
-pre_dia_size=768 -dia_size=1024
motion estimation method:
‘(1024, INT_MAX)’
full motion estimation(slowest)
‘(768, 1024]’
umh motion estimation
‘(512, 768]’
hex motion estimation
‘(256, 512]’
l2s diamond motion estimation
‘[2,256]’
var diamond motion estimation
-me_range=32
Motion search range. 16-24 for 720P. 24-32 for 1080p. Not much point going beyond.
-precmp=sad -cmp=+chroma+rd -subcmp=+rd -ildctcmp=+rd -mbcmp=+chroma+rd -mbd=simple
What metrics to use for finding the best vector or prediction mode:
‘sad’
sum of absolute differences, fast (default)
‘sse’
sum of squared errors
‘satd’
sum of absolute Hadamard transformed differences
‘dct’
sum of absolute DCT transformed differences
‘psnr’
sum of squared quantization errors (avoid, low quality)
‘bit’
number of bits needed for the block
‘rd’
rate distortion optimal, slow
So basically enable RDO for all decisions (except pre-pass, unnecessary), because NVENC is that fast. Try that, x264.
+chroma for motion estimation on chroma planes.
-dct=int -idct=int
H264 already by default uses integer DCT, just put these two here in case encoder uses fast integer for speed.
-flags=+loop
Enable loop filter to increase motion estimation quality
-err_detect=+crccheck+bitstream+buffer -ec=+guess_mvs+deblock+favor_inter
Singal decoder to enable more error detection and correction functions. The cost is not on us so why not.