If it's available from your hardware, choose one of the hardware encoders offering hevc. Hevc is the same as h.265. If you need more fine tuning, use advanced output mode where more tuning parameters are available. However, the crf rate control isn't available from hardware encoders. CQP is a mode that is similar, but it's not as space conserving. If you're on a Intel iGPU and have Quicksync hevc, try the icq rate control - it's more like crf.
Libx265 itself isn't offered by OBS, not even in custom ffmpeg output mode. It's probably due to its high CPU demand that makes it unfeasible for life video encoding.
