On top of the NVENC error, OBS got off to a bad start too. When you're testing make sure OBS gets a clean start. Was there a crash log?
09:14:16.769: [Safe Mode] Unclean shutdown detected!
09:14:30.289: [Safe Mode] User elected to launch normally.
There's still minor rendering lag but more improvement as it's now under 0.1%.
14:49:07.651: Output 'rtmp multitrack video': Number of lagged frames due to rendering lag/stalls: 170 (0.0%)
09:14:30.301: Webroot SecureAnywhere: enabled (AV)
09:14:30.290: Game Mode: Off > On
Stop both the Replay Buffer & Vertical Canvas for the next test. Let's see if there's less of these failures.
09:25:01.692: ==== Replay Buffer Start ===========================================
09:25:01.697: NV12 texture support enabled
09:25:01.697: P010 texture support not available
09:25:01.741: [obs-nvenc] init_session: nv.nvEncOpenEncodeSessionEx(¶ms, &enc->session) failed: 21 (NV_ENC_ERR_INCOMPATIBLE_CLIENT_KEY)
09:25:01.901: Output 'Replay Buffer': stopping
09:25:01.901: Output 'Replay Buffer': Total frames output: 3
09:25:01.901: Output 'Replay Buffer': Total drawn frames: 13
09:25:01.903: [obs-nvenc] init_session: nv.nvEncOpenEncodeSessionEx(¶ms, &enc->session) failed: 21 (NV_ENC_ERR_INCOMPATIBLE_CLIENT_KEY)
09:25:01.929: Already in fallback encoder, can't fall back further!
09:25:01.929: [Vertical Canvas] record stop error NVENC Error: init_session: nv.nvEncOpenEncodeSessionEx(¶ms, &enc->session) failed: 21 (NV_ENC_ERR_INCOMPATIBLE_CLIENT_KEY)
09:25:01.973: ==== Replay Buffer Stop ============================================