persistent GPU encoder lag, skips, and overload with 3080 Ti and 3950x CPU

[minto-bean]

Hi! I'm really hoping someone can help me, I've been troubleshooting something that started happening a couple months ago and still can't find an old topic that helps or quite fits. Sorry for how long this is, trying to be as thorough as I can!

My PC specs are:​

Windows 10 Pro (21H2)
motherboard: Gigabyte x570 Aorus Master
CPU: AMD Ryzen 9 3950x 16-Core
GPU: 3080 Ti FE (studio drivers)
RAM: 64 GB
2 monitors (main is 4k, the other is 1080p)

Video settings:
OBS color settings are: NV12, color format 709, and color range at partial.
Video base res is 3840 x 2160, scaled down to 1920 x 1080
rescaled to 720p for streaming output, left alone for recording output

Logs (recording only for 1 min) with different encoding formats, got the same results but progressively worse:​

CQP (20) : https://obsproject.com/logs/DXB-c_qLesNx3X1a (analyzer)
CBR (4500) : https://obsproject.com/logs/r30pM-AlqUTyMOMc (analyzer)
CQP (20, with different game used in game capture, and 2nd display turned off): https://obsproject.com/logs/XwvlV4eUTH64GQ8p (analyzer)
CQP (20, with both games used in game capture, GPU driver updated and 2nd display turned off): https://obsproject.com/logs/h4rcyr4P2vDHlU89 (analyzer)

Problem:​

I've been having persistent encoder lag and overload with my specs, even with just having a scene with only one image, or one game capture inside it up to encode. I'm only trying to stream and record at 720p/60 FPS with a 4500 bitrate. It's bad enough to where just sitting idle with a game open (FFXIV), or recording 1 min. of footage with either CBR or CQP, I get 7.5%-10% encoder lag and missed frames. In task manager at the same time, with FFXIV on max settings, OBS open and encoding, my CPU sits at 6%, RAM at 24% and GPU at 45%. I have no idea what overload is happening. I stream to twitch and record my streams at the same time, but right now I'm just trying to get things working without the lag while recording while I'm on a long break.

This wasn't an issue I was having at all until semi-recently, (no driver update, and windows had long already updated) I have no idea what might be happening. I never see stuttering in-game, only in OBS and the recordings. The recording tests themselves are a 50/50 chance between coming out super blurry and pixelated with the frame lag and stuttering, or looking fine, but with frame lag and stuttering absolutely everywhere. This GPU is brand new, everything looks normal in GPU-Z (temps, fan speeds, voltage, etc.), I just got it about a month ago (unfortunately to replace one that completely died on me). :(

I've tried (an exhaustive list):​

• Running OBS as admin
• Restarting OBS and my PC
• Using nvidia NVENC h.264 as the encoder
• Outputting the recording to mkv
• changing the encoding type from CBR to CQP + keeping replay buffer disabled
• Setting OBS' process priority to "above normal"
• Uninstalling and reinstalling OBS, and doing a complete OBS wipe clean. Doing the same but with OBS' steam edition (on my C drive).
• Disabling look ahead + psycho visual tuning
• switching the drive that I record to to an external USB 3.0 SSD
• moving all streaming overlay files to the same SSD
• lowering the file size of anything in my scenes and deleting unused sources
• adding sources back one by one
• checked for Missing Files in the scene collection menu, validating OBS program files on Steam
• moving the Steam games I'm playing from a HDD to a new library on my C drive (SSD)
• Having OBS open with a fresh scene collection with one scene with nothing else inside
• Changing the settings to both simple and advanced, and trying OBS' auto settings output
• Changing downscaling from bicubic to lanczos and back, 1080p to 720p and back
• Disabling hardware acceleration both in Windows and OBS, Xbox game bar and the Xbox DVR thing, and game mode (and vice versa)
• Allowing OBS in my anti-virus (malwarebytes and windows defender)
• Making sure all my monitors are set at the same refresh rate (60 hz)
• Always keeping game frame rates capped at 60 FPS
• Only 1 tab of google chrome is open
• Using xSplit with the the settings as close as I could get to compare (somehow had just a bit of stuttering, nothing as intense as this)
• using streamlabs to compare the same, had the same issues but even worse (12% frame skips in 1 min)

I'm pretty bummed, and I have no idea what to do next. I know I can run and stream FFXIV at max settings with no issues, because of both my specs and that I was able to for months before without any problems until sometime in April, and of course, was able to just have the program open without the encoder lag etc. happening.

Is there anything else at all I could try or do to hopefully get it to work now? Could it be another underlying hardware problem with my PC, a software conflict issue with OBS, or maybe a file I'm missing or something? I've updated my BIOS (march 2022), GPU (nvidia studio driver 512.96 from May 23rd, and in the last log 516.94 from August 9th) and I've never installed a driver for my CPU. Didn't even know that was a thing. I have iCloud installed and running, but it doesn't back up any of the areas where streaming stuff is/ends up.

Thank you to anyone looking in advance or who read this far!

 

[PaiSand]

Run the Auto-configuration Wizard and apply the recommended settings. Restart OBS. Test.
If the GPU is overclocked, undo it.
You can try doing a clean GPU driver install:

Performing a clean GPU driver installation

If you're experiencing strange issues or errors that might be related to your GPU drivers, one of the easiest potential fixes is a clean driver installation. Follow the instructions below. 1. Download the latest version of Display Driver...

obsproject.com

 

[minto-bean]

Posting this in case someone else with the same problem stumbles on my topic, but I figured it out! I uninstalled Malwarebytes, and everything fixed itself. I have 0 problems in my recordings or streams now. I have no idea what it was doing that was conflicting with OBS, (it was whitelisted) but it was doing something. I don’t overclock any of my hardware, and I do clean GPU driver installs with every new update. Hopefully this helps someone else! Windows Defender/firewall is still enabled.

 

[FullMe7alJacke7]

Posting this in case someone else with the same problem stumbles on my topic, but I figured it out! I uninstalled Malwarebytes, and everything fixed itself. I have 0 problems in my recordings or streams now. I have no idea what it was doing that was conflicting with OBS, (it was whitelisted) but it was doing something. I don’t overclock any of my hardware, and I do clean GPU driver installs with every new update. Hopefully this helps someone else! Windows Defender/firewall is still enabled.

Thank you!!! I was having this same exact problem!