Performance difference between Ryzen 3700X and Ryzen 5950X

[JohannesGaessler]

I just upgraded my desktop CPU from a Ryzen 3700X to a Ryzen 5950X. I power limited the 5950X to 95W because otherwise my PC would sometimes crash under heavy load (possibly an issue with my motherboard since it's only a B450). The performance from the CPU change change alone in combination with 3200 MHz RAM and an RTX 3090 is as follows:

Model	LLAMA_CUBLAS	-ngl	test	t/s 3700X	t/s 5950X	Speedup
LLaMA 2 q4_0	No	0	pp512	24.65	41.15	1.67
LLaMA 2 q4_0	No	0	tg128	9.93	11.00	1.11
LLaMA 2 q4_0	Yes	0	pp512	351.06	379.81	1.08
LLaMA 2 q4_0	Yes	0	tg128	9.93	10.98	1.11
LLaMA 2 q4_0	Yes	33	pp512	4060.50	4123.53	1.02
LLaMA 2 q4_0	Yes	33	tg128	131.95	133.00	1.01

The biggest difference is for prompt processing without a GPU since this is compute bound. If a GPU is available the benefit is much smaller and if the entire model is on the GPU the difference is negligible.

[Technetium1]

Try to change your load line calibration settings to help with the power spikes. Thanks for posting this!

[JohannesGaessler]

Thanks for the tip but load line calibrations does not fix the issue. I think the fundamental problem is that the motherboard was designed at a time when the most powerful AM4 CPU was a 1800X.

[ikawrakow]

@JohannesGaessler How many threads did you use in these tests?

Here is what I get on a Ryzen 5975-WX (same cores as 5950X but 32 of them) running on the CPU:

model	size	params	backend	threads	test	t/s
llama 7B Q4_0	3.56 GiB	6.74 B	CPU	64	pp 512	93.79 ± 0.79
llama 7B Q4_0	3.56 GiB	6.74 B	CPU	32	pp 512	100.26 ± 1.63
llama 7B Q4_0	3.56 GiB	6.74 B	CPU	16	pp 512	55.46 ± 0.06
llama 7B Q4_0	3.56 GiB	6.74 B	CPU	16	tg 128	11.79 ± 0.00
llama 7B Q4_0	3.56 GiB	6.74 B	CPU	8	tg 128	11.91 ± 0.00
llama 7B Q4_0	3.56 GiB	6.74 B	CPU	4	tg 128	11.14 ± 0.02

[JohannesGaessler]

I used the default number of threads for both CPUs (number of cores, 8 for 3700X, 16 for 5950X). Using more threads seemed to be detrimental.

[ikawrakow]

Yes, using more threads than available cores is detrimental for performance, but for TG one typically gets better performance by using fewer threads. It is interesting that in your setup you get only 80% (compared to my PP with 32 threads) or 75% (compared to my PP with 16 threads) of the performance for PP. I used to have a 5950X and my experience has been that performance per core is equivalent to the 5975WX. Perhaps your motherboard is not supplying enough power to the CPU?

[JohannesGaessler]

As I said, I'm limiting the power to 95W because I'm having stability issues. I have a B450 motherboard which was presumably not designed for 16 core CPUs.

[drazdra]

try to undervolt it, fixing the voltage. that might help your mb to manage the load. and disable precision boost, instead set the freq to its top.

[EriIaz]

I have a 5900X. The cores can be safely undervolted. Even if you overdo that, the lions share of inference performance comes from memory bandwidth instead of CPU core compute anyway. I wouldn't recommend turning PB off, though. Undervolting the frequency curve literally addresses the issue of Precision Boost voltages being too aggressive.

In the meantime, IF and RAM overclock can help LLMs a little bit. It's not much, but if you are into manual overclocking, FCLK=UCLK=MCLK=1900MHz can actually yield minor but noticeable performance gain. Just watch your SoC voltage, some motherboards increase it needlessly when you do it. Also, some CPUs (or rather IMCs) aren't stable there, silicon lottery is a thing. So don't go overboard if that doesn't work.

In any case, even if that level OC isn't your thing, I'd recommend to reconfigure your RAM to 3600MT/s instead of 3200. It increases the memory bandwidth by 12%, doesn't take much effort and even improves L3 cache and core to core latency performance, so you'll benefit all across the board. Memory timings can be recalculated for this frequency, and most modern DDR4 chips can handle 3600MT/s no problem.

The power consumption might increase a tad, but only as the result of reduced data hunger. If your PSU or VRM can't keep up, adjust power limit accordingly, you'll still benefit from that. Chances are though, the effect of power limit on the performance won't be too significant - the more data the CPU gets the more energy efficient it becomes because it increases the likelihood it will do actual compute instead of speculative execution.

But honestly, you should also consider solving the power issue. If you stress PSU or VRM too much, the tiny little MOSFETs and capacitors won't like it. They're humble little components, but when they protest, they aint kidding - they don't write polite letters of resignation, they just send 12V straight to Vcore.