It has to mostly to do with with bitrate. Lower bitrate = worse looking output. If you're streaming while playing, and your upload speeds aren't great, you have to keep the bitrate manageable. If you're uploading to youtube, you can turn up the bitrate higher as well. Native recording software from AMD/NVidia tend to work well for keeping artifacting and crushing minimal, from my limited experience. To get the best bitrate you'll need fast upload speeds, and what most pros do is use a 2nd, more cpu-heavy machine to capture and upload the video as you play it on your less-powerful gaming rig. You can get around this necessity by using a heavily threaded multicore cpu to do both - which is why if you're planning to use one system for everything many recommend at minimum a R5 1600 (better to get a 1700 with prices so low rn). I would not suggest intel for streaming while playing on a single system currently, as the video comparisons I've seen seem to indicate that intel cpus (even the 8700K) seem to have weird stuttering issues, and lower min frame outputs than Ryzen. I just play at my 1440p res and drop my stream down to 720p, with slightly higher bitrate in OBS. Sure it's lower resolution, but the higher bitrate makes it look fairly clear consistently at least. Also make sure you're using the proper newest version of OBS -
I had some weird older version and it wasn't detecting newer updates and for months struggled until I realized it was way way out of date.
Good luck!