Performance of llama.cpp on Apple Silicon M-series

[ggerganov]

Summary

LLaMA 7B

	BW [GB/s]	GPU Cores	F16 PP [t/s]	F16 TG [t/s]	Q8_0 PP [t/s]	Q8_0 TG [t/s]	Q4_0 PP [t/s]	Q4_0 TG [t/s]
✅ M1 1	68	7			108.21	7.92	107.81	14.19
✅ M1 1	68	8			117.25	7.91	117.96	14.15
✅ M1 Pro 1	200	14	262.65	12.75	235.16	21.95	232.55	35.52
✅ M1 Pro 1	200	16	302.14	12.75	270.37	22.34	266.25	36.41
✅ M1 Max 1	400	24	453.03	22.55	405.87	37.81	400.26	54.61
✅ M1 Max 1	400	32	599.53	23.03	537.37	40.2	530.06	61.19
✅ M1 Ultra 1	800	48	875.81	33.92	783.45	55.69	772.24	74.93
✅ M1 Ultra 1	800	64	1168.89	37.01	1042.95	59.87	1030.04	83.73
✅ M2 2	100	8			147.27	12.18	145.91	21.7
✅ M2 2	100	10	201.34	6.72	181.4	12.21	179.57	21.91
✅ M2 Pro 2	200	16	312.65	12.47	288.46	22.7	294.24	37.87
✅ M2 Pro 2	200	19	384.38	13.06	344.5	23.01	341.19	38.86
✅ M2 Max 2	400	30	600.46	24.16	540.15	39.97	537.6	60.99
✅ M2 Max 2	400	38	755.67	24.65	677.91	41.83	671.31	65.95
✅ M2 Ultra 2	800	60	1128.59	39.86	1003.16	62.14	1013.81	88.64
✅ M2 Ultra 2	800	76	1401.85	41.02	1248.59	66.64	1238.48	94.27
🟥 M3 3	100	8
🟨 M3 3	100	10			187.52	12.27	186.75	21.34
🟨 M3 Pro 3	150	14			272.11	17.44	269.49	30.65
✅ M3 Pro 3	150	18	357.45	9.89	344.66	17.53	341.67	30.74
✅ M3 Max 3	300	30	589.41	19.54	566.4	34.3	567.59	56.58
✅ M3 Max 3	400	40	779.17	25.09	757.64	42.75	759.7	66.31
🟥 M3 Ultra	800	60
🟥 M3 Ultra	800	80

plot.py

# GPT-4 Generated Code

import pandas as pd
import numpy as np
import matplotlib.pyplot as plt

# Creating DataFrame from the provided data
data = {
    "Chip": ["M1", "M1", "M1 Pro", "M1 Pro", "M1 Max", "M1 Max", "M1 Ultra", "M2", "M2 Pro", "M2 Pro", "M2 Max", "M2 Max", "M2 Ultra", "M2 Ultra", "M3", "M3 Pro", "M3 Pro", "M3 Max"],
    "BW (GB/s)":     [68, 68, 200, 200, 400, 400, 800, 100, 200, 200, 400, 400, 800, 800, 100, 150, 150, 400],
    "GPU Cores":     [7, 8, 14, 16, 24, 32, 48, 10, 16, 19, 30, 38, 60, 76, 10, 14, 18, 40],
    "F16 PP (t/s)":  [None, None, None, 302.14, 453.03, 599.53, 875.81, 201.34, 312.65, 384.38, 600.46, 755.67, 1128.59, 1401.85, None, None, 357.45, 779.17],
    "F16 TG (t/s)":  [None, None, None, 12.75, 22.55, 23.03, 33.92, 6.72, 12.47, 13.06, 24.16, 24.65, 39.86, 41.02, None, None, 9.89, 25.09],
    "Q8_0 PP (t/s)": [108.21, 117.25, 235.16, 270.37, 405.87, 537.37, 783.45, 181.4, 288.46, 344.5, 540.15, 677.91, 1003.16, 1248.59, 187.52, 272.11, 344.66, 757.64],
    "Q8_0 TG (t/s)": [7.92, 7.91, 21.95, 22.34, 37.81, 40.2, 55.69, 12.21, 22.7, 23.01, 39.97, 41.83, 62.14, 66.64, 12.27, 17.44, 17.53, 42.75],
    "Q4_0 PP (t/s)": [107.81, 117.96, 232.55, 266.25, 400.26, 530.06, 772.24, 179.57, 294.24, 341.19, 537.6, 671.31, 1013.81, 1238.48, 186.75, 269.49, 341.67, 759.7],
    "Q4_0 TG (t/s)": [14.19, 14.15, 35.52, 36.41, 54.61, 61.19, 74.93, 21.91, 37.87, 38.86, 60.99, 65.95, 88.64, 94.27, 21.34, 30.65, 30.74, 66.31]
}
df = pd.DataFrame(data)

# Helper function to plot and annotate multiple data series in the same plot
def plot_multi_series(ax, x, y_series, labels, xlabel, ylabel, title, poly_power=1):
    colors = ['r', 'g', 'b']  # Colors for different series
    for i, y in enumerate(y_series):
        # Sorting data for regression
        sorted_indices = np.argsort(x)
        x_sorted = x[sorted_indices]
        y_sorted = y[sorted_indices]

        # Masking NaN values
        mask = ~np.isnan(y_sorted)
        x_sorted = x_sorted[mask]
        y_sorted = y_sorted[mask]

        # Fitting a polynomial regression model
        coefficients = np.polyfit(x_sorted, y_sorted, poly_power)
        polynomial = np.poly1d(coefficients)

        # Creating a range of x-values for a smoother trendline
        x_range = np.linspace(x_sorted.min(), x_sorted.max(), 500)
        trendline = polynomial(x_range)

        # Plotting
        ax.scatter(x, y, color=colors[i], label=labels[i], s=20)
        ax.plot(x_range, trendline, f"{colors[i]}-", linewidth=1)  # Trendline in the same color

    ax.set_xlabel(xlabel)
    ax.set_ylabel(ylabel)
    ax.set_title(title)
    ax.legend()

    # Annotating points with the number of GPU cores and Bandwidth
    for i, txt in enumerate(df["Chip"]):
        ax.annotate(f"{df['GPU Cores'][i]} Cores, {df['BW (GB/s)'][i]} GB/s", (x[i], y_series[0][i]))


# Creating plots for PP vs Cores and TG vs Bandwidth
fig, axs = plt.subplots(1, 2, figsize=(15, 6))
fig.suptitle('PP vs GPU Cores and TG vs Bandwidth for F16, Q8_0, and Q4_0')

# PP vs GPU Cores
y_series_cores_pp = [df["F16 PP (t/s)"], df["Q8_0 PP (t/s)"], df["Q4_0 PP (t/s)"]]
plot_multi_series(axs[0], df["GPU Cores"], y_series_cores_pp,
                  ['F16 PP', 'Q8_0 PP', 'Q4_0 PP'], 'GPU Cores', 'Performance (t/s)',
                  'PP Performance vs GPU Cores', 1)

# TG vs Bandwidth
y_series_bw_tg = [df["F16 TG (t/s)"], df["Q8_0 TG (t/s)"], df["Q4_0 TG (t/s)"]]
plot_multi_series(axs[1], df["BW (GB/s)"], y_series_bw_tg,
                  ['F16 TG', 'Q8_0 TG', 'Q4_0 TG'], 'Bandwidth (GB/s)', 'Performance (t/s)',
                  'TG Performance vs Bandwidth', 2)

plt.tight_layout(rect=[0, 0.03, 1, 0.95])
plt.show()

---

Description

This is a collection of short llama.cpp benchmarks on various Apple Silicon hardware. It can be useful to compare the performance that llama.cpp achieves across the M-series chips and hopefully answer questions of people wondering if they should upgrade or not. Collecting info here just for Apple Silicon for simplicity. Similar collection for A-series chips is available here: #4508

If you are a collaborator to the project and have an Apple Silicon device, please add your device, results and optionally username for the following command directly into this post (requires LLaMA 7B v2):

git checkout 8e672efe
make clean && make -j llama-bench && ./llama-bench \
  -m ./models/llama-7b-v2/ggml-model-f16.gguf  \
  -m ./models/llama-7b-v2/ggml-model-q8_0.gguf \
  -m ./models/llama-7b-v2/ggml-model-q4_0.gguf \
  -p 512 -n 128 -ngl 99 2> /dev/null

• Make sure to run the benchmark on commit 8e672ef
• Please also include the F16 model as shown, not just the quantum models
• Contributors can post the same results in the comments below
• If a device is already benchmarked and your results are comparable, there is no need to add it again
• PP means "prompt processing" (bs = 512), TG means "text-generation" (bs = 1), t/s means "tokens per second"
• ✅ means the data has been added to the summary

---

M1 Pro, 8+2 CPU, 16 GPU (@ggerganov) ✅

model	size	params	backend	ngl	test	t/s
llama 7B mostly F16	12.55 GiB	6.74 B	Metal	99	pp 512	302.14 ± 0.07
llama 7B mostly F16	12.55 GiB	6.74 B	Metal	99	tg 128	12.75 ± 0.00
llama 7B mostly Q8_0	6.67 GiB	6.74 B	Metal	99	pp 512	270.37 ± 0.02
llama 7B mostly Q8_0	6.67 GiB	6.74 B	Metal	99	tg 128	22.34 ± 0.00
llama 7B mostly Q4_0	3.56 GiB	6.74 B	Metal	99	pp 512	266.25 ± 0.07
llama 7B mostly Q4_0	3.56 GiB	6.74 B	Metal	99	tg 128	36.41 ± 0.01

build: 8e672ef (1550)

M2 Ultra, 16+8 CPU, 76 GPU (@ggerganov) ✅

model	size	params	backend	ngl	test	t/s
llama 7B mostly F16	12.55 GiB	6.74 B	Metal	99	pp 512	1401.85 ± 1.75
llama 7B mostly F16	12.55 GiB	6.74 B	Metal	99	tg 128	41.02 ± 0.02
llama 7B mostly Q8_0	6.67 GiB	6.74 B	Metal	99	pp 512	1248.59 ± 0.73
llama 7B mostly Q8_0	6.67 GiB	6.74 B	Metal	99	tg 128	66.64 ± 0.02
llama 7B mostly Q4_0	3.56 GiB	6.74 B	Metal	99	pp 512	1238.48 ± 0.76
llama 7B mostly Q4_0	3.56 GiB	6.74 B	Metal	99	tg 128	94.27 ± 0.05

build: 8e672ef (1550)

M3 Max (MBP 14), 12+4 CPU, 40 GPU (@slaren) ✅

model	size	params	backend	ngl	test	t/s
llama 7B mostly F16	12.55 GiB	6.74 B	Metal	99	pp 512	794.26 ± 3.16
llama 7B mostly F16	12.55 GiB	6.74 B	Metal	99	tg 128	25.27 ± 0.07
llama 7B mostly Q8_0	6.67 GiB	6.74 B	Metal	99	pp 512	749.37 ± 8.35
llama 7B mostly Q8_0	6.67 GiB	6.74 B	Metal	99	tg 128	43.00 ± 0.12
llama 7B mostly Q4_0	3.56 GiB	6.74 B	Metal	99	pp 512	690.99 ± 33.76
llama 7B mostly Q4_0	3.56 GiB	6.74 B	Metal	99	tg 128	65.85 ± 0.22

build: d103d93 (1553)

Footnotes

1. https://en.wikipedia.org/wiki/Apple_M1#Variants ↩ ↩² ↩³ ↩⁴ ↩⁵ ↩⁶ ↩⁷ ↩⁸

2. https://en.wikipedia.org/wiki/Apple_M2#Variants ↩ ↩² ↩³ ↩⁴ ↩⁵ ↩⁶ ↩⁷ ↩⁸

3. https://en.wikipedia.org/wiki/Apple_M3#Variants ↩ ↩² ↩³ ↩⁴ ↩⁵ ↩⁶

[QueryType]

M2 Mac Mini, 4+4 CPU, 10 GPU, 24 GB Memory (@QueryType) ✅

---

model	size	params	backend	ngl	test	t/s
llama 7B mostly F16	12.55 GiB	6.74 B	Metal	99	pp 512	201.34 ± 0.21
llama 7B mostly F16	12.55 GiB	6.74 B	Metal	99	tg 128	6.72 ± 0.01
llama 7B mostly Q8_0	6.67 GiB	6.74 B	Metal	99	pp 512	181.40 ± 0.05
llama 7B mostly Q8_0	6.67 GiB	6.74 B	Metal	99	tg 128	12.21 ± 0.04
llama 7B mostly Q4_0	3.56 GiB	6.74 B	Metal	99	pp 512	179.57 ± 0.04
llama 7B mostly Q4_0	3.56 GiB	6.74 B	Metal	99	tg 128	21.91 ± 0.02

build: 8e672ef (1550)

[brozkrut]

M2 Max Studio, 8+4 CPU, 38 GPU ✅

model	size	params	backend	ngl	test	t/s
llama 7B mostly F16	12.55 GiB	6.74 B	Metal	99	pp 512	755.67 ± 0.11
llama 7B mostly F16	12.55 GiB	6.74 B	Metal	99	tg 128	24.65 ± 0.02
llama 7B mostly Q8_0	6.67 GiB	6.74 B	Metal	99	pp 512	677.91 ± 0.26
llama 7B mostly Q8_0	6.67 GiB	6.74 B	Metal	99	tg 128	41.83 ± 0.03
llama 7B mostly Q4_0	3.56 GiB	6.74 B	Metal	99	pp 512	671.31 ± 0.20
llama 7B mostly Q4_0	3.56 GiB	6.74 B	Metal	99	tg 128	65.95 ± 0.08

build: 8e672ef (1550)

[marcingomulkiewicz]

5800X3D, Linux Mint, kernel 6.2.0-37-generic, NV driver 535.129.03:

build: 8e672ef (1550):

model	size	params	backend	ngl	test	t/s
llama 7B mostly F16	12.55 GiB	6.74 B	CUDA	99	pp 512	9905.28 ± 614.92
llama 7B mostly F16	12.55 GiB	6.74 B	CUDA	99	tg 128	60.91 ± 0.17
llama 7B mostly Q8_0	6.67 GiB	6.74 B	CUDA	99	pp 512	6602.12 ± 299.45
llama 7B mostly Q8_0	6.67 GiB	6.74 B	CUDA	99	tg 128	99.90 ± 0.21
llama 7B mostly Q4_0	3.56 GiB	6.74 B	CUDA	99	pp 512	6927.76 ± 78.51
llama 7B mostly Q4_0	3.56 GiB	6.74 B	CUDA	99	tg 128	156.47 ± 0.24

newer build basically the same:

build: 33e171d (1604):

model	size	params	backend	ngl	test	t/s
llama 7B mostly F16	12.55 GiB	6.74 B	CUDA	99	pp 512	9918.85 ± 597.74
llama 7B mostly F16	12.55 GiB	6.74 B	CUDA	99	tg 128	60.66 ± 0.88
llama 7B mostly Q8_0	6.67 GiB	6.74 B	CUDA	99	pp 512	6616.48 ± 287.31
llama 7B mostly Q8_0	6.67 GiB	6.74 B	CUDA	99	tg 128	100.23 ± 0.42
llama 7B mostly Q4_0	3.56 GiB	6.74 B	CUDA	99	pp 512	6970.15 ± 38.83
llama 7B mostly Q4_0	3.56 GiB	6.74 B	CUDA	99	tg 128	157.91 ± 0.36

for comparison, CPU only (1604):

model	size	params	backend	threads	test	t/s
llama 7B mostly F16	12.55 GiB	6.74 B	CPU	8	pp 512	30.69 ± 0.14
llama 7B mostly F16	12.55 GiB	6.74 B	CPU	8	tg 128	2.92 ± 0.00
llama 7B mostly Q8_0	6.67 GiB	6.74 B	CPU	8	pp 512	33.72 ± 0.13
llama 7B mostly Q8_0	6.67 GiB	6.74 B	CPU	8	tg 128	5.40 ± 0.18
llama 7B mostly Q4_0	3.56 GiB	6.74 B	CPU	8	pp 512	28.29 ± 0.11
llama 7B mostly Q4_0	3.56 GiB	6.74 B	CPU	8	tg 128	10.24 ± 0.03

[AndreasKunar]

Thanks a lot. very interesting data and to me in-line with Apple silicon. If you add a GPU FP32 TFLOPS column (pure GPUs is not comparable cross architecture), the PP F16 scales with TFLOPS (FP16 with FP32 accumulate = 165.2 TFLOPS for the 4090), the TG F16 scales with memory-bandwidth (1008 GB/s for 4090). And quantization adds speed due to reduced memory, but requires more compute. What's amazing to me, is that this might enable a good predictor for model-performance based on easily available GPU FP32 TFLOPS + memory-bandwidth information.

[maver1ck]

Wow. I wasn't aware that 4090 is so fast.

[vitali-fridman]

This is from one/two generations old hardware but it's for 70B model which might be of interest.

CPU: AMD 3995WX, GPU: 2x Nvidia 3090, Ubuntu 23.10, Kernel 6.5.0-14, NV Driver: 545.23.08, CUDA: 12.3.1

model	size	params	backend	ngl	test	t/s
llama 70B Q4_0	36.20 GiB	68.98 B	CUDA	99	pp 512	179.29 ± 2.83
llama 70B Q4_0	36.20 GiB	68.98 B	CUDA	99	tg 128	21.17 ± 0.04

For comparison, 7B model on the same hardware

model	size	params	backend	ngl	test	t/s
llama 7B Q4_0	3.56 GiB	6.74 B	CUDA	99	pp 512	1178.60 ± 88.08
llama 7B Q4_0	3.56 GiB	6.74 B	CUDA	99	tg 128	87.34 ± 0.89

[zotona]

could you try at 7b model for correct comparation? Thanks!

[crasm]

M2 Ultra, 16+8 CPU, 60 GPU (@crasm) ✅

model	size	params	backend	ngl	test	t/s
llama 7B mostly F16	12.55 GiB	6.74 B	Metal	99	pp 512	1128.59 ± 0.82
llama 7B mostly F16	12.55 GiB	6.74 B	Metal	99	tg 128	39.86 ± 0.01
llama 7B mostly Q8_0	6.67 GiB	6.74 B	Metal	99	pp 512	1003.16 ± 0.39
llama 7B mostly Q8_0	6.67 GiB	6.74 B	Metal	99	tg 128	62.14 ± 0.03
llama 7B mostly Q4_0	3.56 GiB	6.74 B	Metal	99	pp 512	1013.81 ± 0.92
llama 7B mostly Q4_0	3.56 GiB	6.74 B	Metal	99	tg 128	88.64 ± 0.06

build: 8e672ef (1550)

[ymcui]

M3 Max (MBP 16), 12+4 CPU, 40 GPU (@ymcui) ✅

model	size	params	backend	ngl	test	t/s
llama 7B mostly F16	12.55 GiB	6.74 B	Metal	99	pp 512	779.17 ± 0.49
llama 7B mostly F16	12.55 GiB	6.74 B	Metal	99	tg 128	25.09 ± 0.01
llama 7B mostly Q8_0	6.67 GiB	6.74 B	Metal	99	pp 512	757.64 ± 1.03
llama 7B mostly Q8_0	6.67 GiB	6.74 B	Metal	99	tg 128	42.75 ± 0.06
llama 7B mostly Q4_0	3.56 GiB	6.74 B	Metal	99	pp 512	759.70 ± 2.26
llama 7B mostly Q4_0	3.56 GiB	6.74 B	Metal	99	tg 128	66.31 ± 0.12

build: 55978ce (1555)

---

Short Note: mostly similar to the one reported by @slaren . But for Q4_0 pp 512, my result is 759.70 ± 2.26, while the one in the main post is 690.99 ± 33.76. Not sure about the source of the difference.

[slaren]

I am not sure why, but the results that I get are not very consistent. I suspect that it may due to the cooling limitations of the smaller laptop. I repeated the test now and the results are very similar to yours.

model	size	params	backend	ngl	test	t/s
llama 7B mostly F16	12.55 GiB	6.74 B	Metal	99	pp 512	787.24 ± 0.84
llama 7B mostly F16	12.55 GiB	6.74 B	Metal	99	tg 128	25.15 ± 0.02
llama 7B mostly Q8_0	6.67 GiB	6.74 B	Metal	99	pp 512	755.88 ± 1.56
llama 7B mostly Q8_0	6.67 GiB	6.74 B	Metal	99	tg 128	42.64 ± 0.04
llama 7B mostly Q4_0	3.56 GiB	6.74 B	Metal	99	pp 512	760.65 ± 0.77
llama 7B mostly Q4_0	3.56 GiB	6.74 B	Metal	99	tg 128	66.35 ± 0.24

[Azirine]

In the graph, why is PP t/s plotted against bandwidth and TG t/s plotted against GPU cores? Seems like GPU cores have more effect on PP t/s.

[marko-radojcic]

This graph is excellent extremely strong linear correlation.

[Azirine]

How about also sharing the largest model sizes and context lengths people can run with their amount of RAM? It's important to get the amount of RAM right when buying Apple computers because you can't upgrade later.

[ggerganov]

You can compute these. By default, you can use ~75% of the total RAM with the GPU. You can use more if you do some tricks

[minosvasilias]

M2 Pro, 6+4 CPU, 16 GPU (@minosvasilias) ✅

model	size	params	backend	ngl	test	t/s
llama 7B mostly F16	12.55 GiB	6.74 B	Metal	99	pp 512	312.65 ± 15.75
llama 7B mostly F16	12.55 GiB	6.74 B	Metal	99	tg 128	12.47 ± 0.71
llama 7B mostly Q8_0	6.67 GiB	6.74 B	Metal	99	pp 512	288.46 ± 0.06
llama 7B mostly Q8_0	6.67 GiB	6.74 B	Metal	99	tg 128	22.70 ± 0.12
llama 7B mostly Q4_0	3.56 GiB	6.74 B	Metal	99	pp 512	294.24 ± 0.10
llama 7B mostly Q4_0	3.56 GiB	6.74 B	Metal	99	tg 128	37.87 ± 0.10

build: e9c13ff (1560)

[to3d]

Would love to see how M1 Max and M1 Ultra fare given their high memory bandwidth.

[MrSparc]

M2 MAX (MBP 16) 8+4 CPU, 38 GPU, 96 GB RAM (@MrSparc) ✅

model	size	params	backend	ngl	test	t/s
llama 7B mostly Q8_0	6.67 GiB	6.74 B	Metal	99	pp 512	674.50 ± 0.58
llama 7B mostly Q8_0	6.67 GiB	6.74 B	Metal	99	tg 128	41.79 ± 0.04
llama 7B mostly Q4_0	3.56 GiB	6.74 B	Metal	99	pp 512	669.51 ± 1.17
llama 7B mostly Q4_0	3.56 GiB	6.74 B	Metal	99	tg 128	64.55 ± 1.36

build: e9c13ff (1560)

[rlippmann]

I'm also using a MBP16 M2Max with the same CPU/GPU specs, but only 32 gb ram and my results are roughly the same:

M2 MAX (MBP 16) 8+4 CPU, 38 GPU, 32 GB RAM ✅

model	size	params	backend	ngl	test	t/s
llama 7B mostly F16	12.55 GiB	6.74 B	Metal	99	pp 512	747.99 ± 0.28
llama 7B mostly F16	12.55 GiB	6.74 B	Metal	99	tg 128	24.54 ± 0.22
llama 7B mostly Q8_0	6.67 GiB	6.74 B	Metal	99	pp 512	674.37 ± 0.63
llama 7B mostly Q8_0	6.67 GiB	6.74 B	Metal	99	tg 128	40.67 ± 0.05
llama 7B mostly Q4_0	3.56 GiB	6.74 B	Metal	99	pp 512	668.28 ± 0.24
llama 7B mostly Q4_0	3.56 GiB	6.74 B	Metal	99	tg 128	62.98 ± 0.06

build: 22da055 (1566)

[MrSparc]

Yes, it is expected that the same cpu/gpu spec will have similar performance values for same models to be compared regardless of RAM, as long as the size of the model to be used can be loaded into memory.
The amount of RAM is a limiting factor in the size of the model that can be loaded, as only 75% (by default) of the unified memory can be used as VRAM on the GPU
https://github.com/ggerganov/llama.cpp#memorydisk-requirements

[CedricYauLBD]

M1 Max (MBP 16) 8+2 CPU, 32 GPU, 64GB RAM (@CedricYauLBD) ✅

model	size	params	backend	ngl	test	t/s
llama 7B mostly F16	12.55 GiB	6.74 B	Metal	99	pp 512	599.53 ± 0.86
llama 7B mostly F16	12.55 GiB	6.74 B	Metal	99	tg 128	23.03 ± 0.09
llama 7B mostly Q8_0	6.67 GiB	6.74 B	Metal	99	pp 512	537.37 ± 0.19
llama 7B mostly Q8_0	6.67 GiB	6.74 B	Metal	99	tg 128	40.20 ± 0.03
llama 7B mostly Q4_0	3.56 GiB	6.74 B	Metal	99	pp 512	530.06 ± 0.17
llama 7B mostly Q4_0	3.56 GiB	6.74 B	Metal	99	tg 128	61.19 ± 0.15

build: e9c13ff (1560)

Note: M1 Max RAM Bandwidth is 400GB/s

[philipturner]

Look at what I started

[yxzwayne]

off topic, but your benchmark output is my desktop rn :D

[paramaggarwal]

M3 Pro (MBP 14), 5+6 CPU, 14 GPU (@paramaggarwal) ✅

model	size	params	backend	ngl	test	t/s
llama 7B mostly Q8_0	6.67 GiB	6.74 B	Metal	99	pp 512	272.11 ± 1.40
llama 7B mostly Q8_0	6.67 GiB	6.74 B	Metal	99	tg 128	17.44 ± 0.42
llama 7B mostly Q4_0	3.56 GiB	6.74 B	Metal	99	pp 512	269.49 ± 1.14
llama 7B mostly Q4_0	3.56 GiB	6.74 B	Metal	99	tg 128	30.65 ± 0.20

build: e9c13ff (1560)

[ggerganov]

This one has 150 GB/s memory bandwidth, correct?

[paramaggarwal]

Yes, that's correct. (source)

[brozkrut]

Chip (vs. Predecessor)	F16 PP	F16 TG	Q8_0 PP	Q8_0 TG	Q4_0 PP	Q4_0 TG
M2 Pro (16) vs. M1 Pro (16)	312.65 302.14	12.47 12.75	288.46 270.37	22.7 22.34	294.24 266.25	37.87 36.41
	+3.48%	-2.20%	+6.69%	+1.61%	+10.51%	+4.01%
M2 Max (38) vs. M1 Max (32)	755.67 599.53	24.65 23.03	677.91 537.37	41.83 40.2	671.31 530.06	65.95 61.19
	+26.04%	+7.03%	+26.15%	+4.05%	+26.65%	+7.78%
M2 Ultra (60) vs. M2 Max (38)	1128.59 755.67	39.86 24.65	1003.16 677.91	62.14 41.83	1013.81 671.31	88.64 65.95
	+49.34%	+61.90%	+48.04%	+48.48%	+51.03%	+34.41%
M2 Ultra (76) vs. M2 Max (38)	1401.85 755.67	41.02 24.65	1248.59 677.91	66.64 41.83	1238.48 671.31	94.27 65.95
	+85.67%	+66.45%	+84.24%	+59.47%	+84.53%	+43.06%
M2 Ultra (76) vs. M2 Ultra (60)	1401.85 1128.59	41.02 39.86	1248.59 1003.16	66.64 62.14	1238.48 1013.81	94.27 88.64
	+24.25%	+2.91%	+24.43%	+7.23%	+22.19%	+6.33%
M3 Pro (14) vs. M2 Pro (16)			272.11 288.46	17.44 22.7	269.49 294.24	30.65 37.87
			-5.67%	-23.17%	-8.41%	-19.07%
M3 Max (40) vs. M2 Max (38)	779.17 755.67	25.09 24.65	757.64 677.91	42.75 41.83	759.7 671.31	66.31 65.95
	+3.11%	+1.78%	+11.76%	+2.20%	+13.17%	+0.55%

[pudepiedj]

### M2 MAX (MBP 16) 38 Core 32GB ✅

model	size	params	backend	ngl	test	t/s
llama 7B mostly F16	12.55 GiB	6.74 B	Metal	99	pp 512	754.39 ± 0.36
llama 7B mostly F16	12.55 GiB	6.74 B	Metal	99	tg 128	24.31 ± 0.38
llama 7B mostly Q8_0	6.67 GiB	6.74 B	Metal	99	pp 512	671.33 ± 2.65
llama 7B mostly Q8_0	6.67 GiB	6.74 B	Metal	99	tg 128	40.85 ± 0.32
llama 7B mostly Q4_0	3.56 GiB	6.74 B	Metal	99	pp 512	664.07 ± 9.11
llama 7B mostly Q4_0	3.56 GiB	6.74 B	Metal	99	tg 128	63.29 ± 0.15

build: 795cd5a (1493)

[MrSparc]

I'm looking at the summary plot about "PP performance vs GPU cores" and evidence that original unquantised fp16 model always delivers more performance than quantized models.
Sorry if my question is silly, I'm new to this area, but can someone explain to me why original model delivers more performance than quantized models? Thanks

[ggerganov]

The question is not silly - the observation is expected. At large batch size (PP means batch size of 512) the computation is compute bound. I.e. the speed depends on how many FLOPS you can utilize. For quantum models, the existing kernels require extra compute to dequantize the data compared to F16 models where the data is already in F16 format.

[soleblaze]

M3 Max 10+4 CPU, 30 GPU (MacBook Pro 14" 36GB ram) ✅

model	size	params	backend	ngl	test	t/s
llama 7B mostly F16	12.55 GiB	6.74 B	Metal	99	pp 512	589.41 ± 0.31
llama 7B mostly F16	12.55 GiB	6.74 B	Metal	99	tg 128	19.54 ± 0.12
llama 7B mostly Q8_0	6.67 GiB	6.74 B	Metal	99	pp 512	566.40 ± 3.32
llama 7B mostly Q8_0	6.67 GiB	6.74 B	Metal	99	tg 128	34.30 ± 0.24
llama 7B mostly Q4_0	3.56 GiB	6.74 B	Metal	99	pp 512	567.59 ± 0.53
llama 7B mostly Q4_0	3.56 GiB	6.74 B	Metal	99	tg 128	56.85 ± 0.37

[LB426]

server, Dual Intel Xeon Platinum 8380, 160 cores, RAM 2000GB, Ubuntu 22.04.3, Kernel: 5.15.0-91-generic

model	size	params	backend	threads	test	t/s
llama 7B mostly F16	12.55 GiB	6.74 B	CPU	80	pp 512	69.67 ± 12.55
llama 7B mostly F16	12.55 GiB	6.74 B	CPU	80	tg 128	4.04 ± 0.02
llama 7B mostly Q8_0	6.86 GiB	6.93 B	CPU	80	pp 512	81.57 ± 6.07
llama 7B mostly Q8_0	6.86 GiB	6.93 B	CPU	80	tg 128	7.21 ± 0.02
llama 7B mostly Q4_0	3.69 GiB	6.93 B	CPU	80	pp 512	69.78 ± 0.75
llama 7B mostly Q4_0	3.69 GiB	6.93 B	CPU	80	tg 128	12.40 ± 0.11

@ggerganov could you add decrypt of t/s abbreviation to description, please.

[ggerganov]

It means "tokens per second". Added to the description

[LB426]

Thank you!

[alimaslax]

M2 Pro, 12 CPU, 19 GPU, 32GB RAM ✅

model	size	params	backend	ngl	test	t/s
llama 7B mostly F16	12.55 GiB	6.74 B	Metal	99	pp 512	373.97 ± 0.15
llama 7B mostly F16	12.55 GiB	6.74 B	Metal	99	tg 128	12.97 ± 0.03
llama 7B mostly Q8_0	6.67 GiB	6.74 B	Metal	99	pp 512	335.90 ± 0.09
llama 7B mostly Q8_0	6.67 GiB	6.74 B	Metal	99	tg 128	23.24 ± 0.07
llama 7B mostly Q4_0	3.56 GiB	6.74 B	Metal	99	pp 512	342.61 ± 0.06
llama 7B mostly Q4_0	3.56 GiB	6.74 B	Metal	99	tg 128	39.71 ± 0.02

build: 8e672ef (1550)

[AiQura]

Hi Alimaslax,

Wanted to know what do you recommend me?, M2 pro same as yours or should I go for M2 max 12‑Core CPU and 38‑Core GPU, what do you feel about your machine, does it satisfy the ML needs or you feel like you want to upgrade ?

[pudepiedj]

I bought my M2 max 12-core CPU 38-core GPU last May (2023) and, knowing what I know now, if I could go back, I would buy something bigger. The 32GB unified memory limit, which seemed enormous at the time, is quite restrictive even though you can force the system to allow you to use up to about 28GB VRAM. I can just run mixtral-8x7b-instruct-v0.1.Q4_K_M.gguf but I'd love to be able to run Q5 or Q8 etc. or Gemma-7B-it.gguf and obviously a 70B model is way beyond it. Buy as much memory as you can afford would be my bottom line!

[chrisbraddock]

M3 Max, 16 CPU, 40 GPU, 128GB RAM ✅

model	size	params	backend	ngl	test	t/s
llama 7B all F32	25.10 GiB	6.74 B	Metal	99	pp 512	644.56 ± 1.70
llama 7B all F32	25.10 GiB	6.74 B	Metal	99	tg 128	13.43 ± 0.05
llama 7B mostly F16	12.55 GiB	6.74 B	Metal	99	pp 512	784.83 ± 2.11
llama 7B mostly F16	12.55 GiB	6.74 B	Metal	99	tg 128	25.14 ± 0.03
llama 7B mostly Q8_0	6.67 GiB	6.74 B	Metal	99	pp 512	707.07 ± 27.03
llama 7B mostly Q8_0	6.67 GiB	6.74 B	Metal	99	tg 128	42.37 ± 0.08

8e672ef

[zcbenz]

@chrisbraddock Do you mind sharing how useful is 128GB RAM for inferencing models? I'm hesitating about getting 64GB or 128GB for M3 Max, money is not very big problem but I want to avoid buying something I don't need.

[beebopkim]

I think it would be �better to have threads that collect and share such benchmark data for AMD, Intel, NVIDIA GPUs as well.

[andrewginns]

M1 Pro, 8 CPU, 14 GPU, 16 GB RAM. ✅

I think the limited ram is why I am getting lower confidence on the pp 512 test for the FP16 model, this is despite me setting sudo sysctl iogpu.wired_limit_mb=15360

model	size	params	backend	ngl	test	t/s
llama 7B mostly F16	12.55 GiB	6.74 B	Metal	99	pp 512	262.65 ± 0.16
llama 7B mostly F16	12.55 GiB	6.74 B	Metal	99	tg 128	12.75 ± 0.02
llama 7B mostly Q8_0	6.67 GiB	6.74 B	Metal	99	pp 512	237.00 ± 0.05
llama 7B mostly Q8_0	6.67 GiB	6.74 B	Metal	99	tg 128	22.51 ± 0.01
llama 7B mostly Q4_0	3.56 GiB	6.74 B	Metal	99	pp 512	233.49 ± 0.04
llama 7B mostly Q4_0	3.56 GiB	6.74 B	Metal	99	tg 128	36.34 ± 0.02

build: 8e672ef (1550)

[ggerganov]

The uncertainty of the pp 512 is quite big (± 69.71) - maybe try to re-run it. You can also run it 2 times: -p 512,512 and pick the one with lower uncertainty

[andrewginns]

I ran it a number of times and updated the table with the lowest uncertainty. Results are all over the place on my 16GB machine.

model	size	params	backend	ngl	test	t/s
llama 7B mostly F16	12.55 GiB	6.74 B	Metal	99	pp 512	149.71 ± 102.20
llama 7B mostly F16	12.55 GiB	6.74 B	Metal	99	pp 512	218.53 ± 69.53
llama 7B mostly F16	12.55 GiB	6.74 B	Metal	99	pp 512	207.19 ± 83.42
llama 7B mostly F16	12.55 GiB	6.74 B	Metal	99	pp 512	263.38 ± 0.60
llama 7B mostly F16	12.55 GiB	6.74 B	Metal	99	pp 512	263.49 ± 0.27
llama 7B mostly F16	12.55 GiB	6.74 B	Metal	99	pp 512	262.87 ± 1.67
llama 7B mostly F16	12.55 GiB	6.74 B	Metal	99	pp 512	263.19 ± 0.23
llama 7B mostly F16	12.55 GiB	6.74 B	Metal	99	pp 512	262.50 ± 1.42
llama 7B mostly F16	12.55 GiB	6.74 B	Metal	99	pp 512	263.23 ± 0.42
llama 7B mostly F16	12.55 GiB	6.74 B	Metal	99	pp 512	262.65 ± 0.16

[ggerganov]

Thanks, added the data to the table. I suppose it is pushing the machine to it's limits, so we can expect some variation, though it seems to settle up at the expected value after a few runs

[andrewginns]

Yep, my assumption is the OS scheduler realises it needs to evict other processes from memory after the first few runs.

[DavidCA916]

M2 MacBook Air, 8‑core CPU, 8‑core GPU, 16GB RAM ✅

model	size	params	backend	ngl	test	t/s
llama 7B mostly Q8_0	6.67 GiB	6.74 B	Metal	99	pp 512	147.27 ± 0.53
llama 7B mostly Q8_0	6.67 GiB	6.74 B	Metal	99	tg 128	12.18 ± 0.02
llama 7B mostly Q4_0	3.56 GiB	6.74 B	Metal	99	pp 512	145.91 ± 0.05
llama 7B mostly Q4_0	3.56 GiB	6.74 B	Metal	99	tg 128	21.70 ± 0.01

build: 8e672ef (1550)

[gelim]

Would it be possible for MBP owners to add performance stats on Mixtral 8x7B quantized models?
Thanks!

[ptimmons]

M1 Ultra, 16+4 core CPU, 64 core GPU, 128GB RAM ✅

model	size	params	backend	ngl	test	t/s
llama 7B mostly F16	12.55 GiB	6.74 B	Metal	99	pp 512	1158.95 ± 1.67
llama 7B mostly F16	12.55 GiB	6.74 B	Metal	99	tg 128	36.31 ± 0.18

build: 8e672efe (1550)

I'll work on the quantized models soon, I'm having some challenges getting them working.

[ptimmons]

model	size	params	backend	ngl	test	t/s
llama 7B mostly F16	12.55 GiB	6.74 B	Metal	99	pp 512	1168.89 ± 0.49
llama 7B mostly F16	12.55 GiB	6.74 B	Metal	99	tg 128	37.01 ± 0.01
llama 7B mostly Q8_0	6.67 GiB	6.74 B	Metal	99	pp 512	1042.95 ± 1.01
llama 7B mostly Q8_0	6.67 GiB	6.74 B	Metal	99	tg 128	59.87 ± 0.09
llama 7B mostly Q4_0	3.56 GiB	6.74 B	Metal	99	pp 512	1030.04 ± 0.72
llama 7B mostly Q4_0	3.56 GiB	6.74 B	Metal	99	tg 128	83.73 ± 0.13

[ai-bits]

Major edit!
With this bench I wanted to share the results of my X86 + 2x Nvidia compared to my
M3 Max 10+4 CPU, 30 GPU (MacBook Pro 14 36GB RAM); here results from @soleblaze

Model	size	params	backend	ngl	test	t/s
llama 7B mostly Q8_0	6.67 GiB	6.74 B	Metal	99	pp 512	566.40 ± 3.32
llama 7B mostly Q8_0	6.67 GiB	6.74 B	Metal	99	tg 128	34.30 ± 0.24

Xeon 3435X w/ 256GB RAM and 2x 20GB Nvidia RTX 4000, compiled w/ CUDA and tooling 12.3:
Ubuntu 22.04 kernel 6.5.0-14-generic driver 545.23.08
Looks much better than on WSL, especially tg! Does anyone have an educated guess re the reason?
Still, if one 4090 is sooo much better I need to adjust my HW plans. Or is it optimization?
Cudos MacBook Pro M3!

model	size	params	backend	ngl	test	t/s
llama 7B mostly F16	12.55 GiB	6.74 B	CUDA	99	pp 512	962.94 ± 10.27
llama 7B mostly F16	12.55 GiB	6.74 B	CUDA	99	tg 128	35.28 ± 0.08
llama 7B mostly Q8_0	6.67 GiB	6.74 B	CUDA	99	pp 512	1049.96 ± 26.63
llama 7B mostly Q8_0	6.67 GiB	6.74 B	CUDA	99	tg 128	50.47 ± 0.16
llama 7B mostly Q4_0	3.56 GiB	6.74 B	CUDA	99	pp 512	1087.59 ± 32.54
llama 7B mostly Q4_0	3.56 GiB	6.74 B	CUDA	99	tg 128	69.49 ± 0.29

On WSL2 Ubuntu 22.04 kernel 5.15.133.1 driver 546.01 (on Windows).

model	size	params	backend	ngl	test	t/s
llama 7B mostly F16	12.55 GiB	6.74 B	CUDA	99	pp 512	648.07 ± 2.41
llama 7B mostly F16	12.55 GiB	6.74 B	CUDA	99	tg 128	18.46 ± 0.67
llama 7B mostly Q8_0	6.67 GiB	6.74 B	CUDA	99	pp 512	637.51 ± 102.02
llama 7B mostly Q8_0	6.67 GiB	6.74 B	CUDA	99	tg 128	19.11 ± 0.53
llama 7B mostly Q4_0	3.56 GiB	6.74 B	CUDA	99	pp 512	671.10 ± 94.51
llama 7B mostly Q4_0	3.56 GiB	6.74 B	CUDA	99	tg 128	19.23 ± 1.21

There is one bench output that worries me:
ggml_init_cublas: GGML_CUDA_FORCE_MMQ: no
Optimization problem? Impact?

I use the low-cores 20GB VRAM RTX 4000 because I thought the combined VRAM would be important.

Wonder why a 5800X3D w/ RTX 4090 @marcingomulkiewicz shows roughly 10x the speed, albeit on Linux directly...

@vitali-fridman Glad that my 2x RTX 4000 seem to be roughly on par with 2x 3090 when run directly on Ubuntu and not via WSL!

Thanks for any insights
G.

[slaren]

There was a recent change that should have improved multi GPU performance in some cases, at worst it should only be slightly slower than with a single GPU. You can also set the environment variable CUDA_VISIBLE_DEVICES=0 to use a single GPU.

[marcingomulkiewicz]

Felt like my results were significantly off, so I re-run this (in the meantime I upgraded kernel to 6.5.0-14, NV drivers to 545.23.08, CUDA tools to 12.3 (was 11.x before), and llama.cpp to 1930, but results still remain similar:

model	size	params	backend	ngl	test	t/s
llama 7B F16	12.55 GiB	6.74 B	CUDA	99	pp 512	10384.14 ± 829.23
llama 7B F16	12.55 GiB	6.74 B	CUDA	99	tg 128	61.99 ± 0.07
llama 7B Q8_0	6.67 GiB	6.74 B	CUDA	99	pp 512	7786.36 ± 5.41
llama 7B Q8_0	6.67 GiB	6.74 B	CUDA	99	tg 128	101.58 ± 0.25
llama 7B Q4_0	3.56 GiB	6.74 B	CUDA	99	pp 512	8040.05 ± 9.72
llama 7B Q4_0	3.56 GiB	6.74 B	CUDA	99	tg 128	156.48 ± 0.48

I have no direct access to 'original' llamas, but I took the ones by TheBloke - but I guess it shouldn't matter...?

[ai-bits]

Thanks guys!
@marcingomulkiewicz

I have no direct access to 'original' llamas, but I took the ones by TheBloke...

Same here. And the F16 one is from here.

@slaren
Please excuse the naive q from a 71y old hobby-programmer: export LLAMA_CUBLAS=1 and compile.
What did I overlook that I got
ggml_init_cublas: GGML_CUDA_FORCE_MMQ: no

MTIA
G.

[slaren]

You didn't overlook anything, it is just printing the setting used to compile llama.cpp. MMQ disabled is the default, as generally it is faster for newer GPUs with good FP16 performance, but you can try MMQ by compiling with LLAMA_CUDA_FORCE_MMQ defined.

[ai-bits]

Hey, @slaren, love weekend night support! ;-) Wishing / Having a VERY good Sunday morning!
export LLAMA_CUDA_FORCE_MMQ=1
compile & run:
ggml_init_cublas: GGML_CUDA_FORCE_MMQ: yes
ggml_init_cublas: CUDA_USE_TENSOR_CORES: no
ggml_init_cublas: found 2 CUDA devices:
Device 0: NVIDIA RTX 4000 Ada Generation, compute capability 8.9
Device 1: NVIDIA RTX 4000 Ada Generation, compute capability 8.9
ggml_cuda_set_main_device: using device 0 (NVIDIA RTX 4000 Ada Generation) as main device

F16 didn't change, but look at Q8 & Q4:

model	size	params	backend	ngl	test	t/s
llama 7B mostly F16	12.55 GiB	6.74 B	CUDA	99	pp 512	925.62 ± 23.50
llama 7B mostly F16	12.55 GiB	6.74 B	CUDA	99	tg 128	35.42 ± 0.01
llama 7B mostly Q8_0	6.67 GiB	6.74 B	CUDA	99	pp 512	1937.77 ± 109.23
llama 7B mostly Q8_0	6.67 GiB	6.74 B	CUDA	99	tg 128	50.75 ± 0.02
llama 7B mostly Q4_0	3.56 GiB	6.74 B	CUDA	99	pp 512	1956.45 ± 134.41
llama 7B mostly Q4_0	3.56 GiB	6.74 B	CUDA	99	tg 128	69.83 ± 0.04

Oops, now we get there. <vbg> (Does anyone still know these abbreviations like <very broad grin> - with all the icons out there?)
(Variance is very bad, but we took the values be4, take it now too and second runs showed very similar values.)

export CUDA_VISIBLE_DEVICES=0 #use only 1 GPU

ggml_init_cublas: GGML_CUDA_FORCE_MMQ: yes
ggml_init_cublas: CUDA_USE_TENSOR_CORES: no
ggml_init_cublas: found 1 CUDA devices:
Device 0: NVIDIA RTX 4000 Ada Generation, compute capability 8.9

model	size	params	backend	ngl	test	t/s
llama 7B mostly F16	12.55 GiB	6.74 B	CUDA	99	pp 512	3022.37 ± 94.50
llama 7B mostly F16	12.55 GiB	6.74 B	CUDA	99	tg 128	23.93 ± 0.00
llama 7B mostly Q8_0	6.67 GiB	6.74 B	CUDA	99	pp 512	1688.37 ± 51.25
llama 7B mostly Q8_0	6.67 GiB	6.74 B	CUDA	99	tg 128	42.43 ± 0.02
llama 7B mostly Q4_0	3.56 GiB	6.74 B	CUDA	99	pp 512	1816.45 ± 68.24
llama 7B mostly Q4_0	3.56 GiB	6.74 B	CUDA	99	tg 128	73.38 ± 0.03

Now look at F16 PP!
1 GPU 3x faster than 2 GPUs with FORCE_MMQ instead of USE_TENSOR_CORES!!!
Guess this needs some major investigation / profiling!?
OMG!
G.

[dariox1337]

Since people already posted some non-apple results. Here is one more just for comparison.

CPU: AMD 7840u, GPU: 780m, RAM: LPDDR5-7500

model	size	params	backend	ngl	mmq	test	t/s
llama 7B Q4_0	3.56 GiB	6.74 B	ROCm	99	0	pp 512	310.95 ± 2.10
llama 7B Q4_0	3.56 GiB	6.74 B	ROCm	99	0	tg 128	21.47 ± 0.01

build 213d143

ROCm compiled without shared memory because the latter affects generation very badly.
So, this system has the same bandwidth as Apple M3, which results in almost the same tg, but pp is better.

[dariox1337]

Where did you get the formula? With /3.56 you mean scanning the whole model? Are you sure the algo is that primitive?

I may be wrong, but that's my understanding about how transformers work, gathered from many sources that I've read over months.

If that is correct, why do we need the bench?

Because the theoretical limit is unreachable. It implies that actual computation is instant, but it's not. The computation speed depends on the GPU's processing power and the specific implementation of the transformer algo.

And why does one of my RTX 4000 score 3022 pp w/ F16 and two only 925?

I guess the overhead of transferring the model's internal state between two gpus slows things down. It's happening over pcie lanes after all.

And why don't 8 v 4 threads change very much in LM Studio?

I think the accepted explanation is that on most systems, when running in cpu-only mode, 4 threads are enough to saturate available-for-cpu memory bandwidth.

[ai-bits]

And why does one of my RTX 4000 score 3022 pp w/ F16 and two only 925?

I guess the overhead of transferring the model's internal state between two gpus slows things down. It's happening over pcie lanes after all.

So, does the optimization using two GPUs when the model easily fits into one confirm my suspicion that the optimization on amd64 leaves sth to be desired or is your assumption wrong that it uses two GPUs?

[ai-bits]

Good morning, without an undertone:
I think we can agree the op is really highly memory-(throughput-)bound and the simple formula may not be too far off.

Hope we can agree that inference (search for the next, most probable token) is highly serial (not parallizable), so using both GPUs would rather hinder than help!?

Neither had the time nor the capacity to check if the algo uses the 2nd GPU where I think it's detrimental to perf, but my numbers suggest it is so.

Are you in the position to investigate this?
Thanks
G.

[dariox1337]

my suspicion that the optimization on amd64 leaves sth to be desired

I don't know the internals of llama.cpp, but amd64 isn't related to your issue. It's mostly CUDA. If apple systems supported discrete nvida gpus in pcie slots, you'd still have the same issue.

both GPUs would rather hinder than help!?

Using a single gpu is almost universally better. It's just harder to properly utilize multiple gpus. The only reason why you'd want multiple gpus is to increase available vram pool.

Are you in the position to investigate this?

Not me.

[ai-bits]

Using a single gpu is almost universally better.

The situation changed by the month over the last year. First it looked like you need to shell out sth like 20 or 50k, primarily to get 80GB VRAM to run sth locally. NvLink on consumer RTX died with the 2000 series. With 3.25-slot 4090s you could get max 2x 24GB into a box. There are water-cooled 1-slot ones, but you need room for the coolers, so same outcome for affordable.
Etc.
I went for a Xeon w/ 112 PCIe 5 lanes so I could stick in 6x 20GB RTX 4000, alas only PCIe 4, if necessary. In hindsight 2 RTX 4090 would have been better, cuz model sizes went down by the week with quantization.

We'll see how my hw does with OLMo fine-tuning - just this week's development.

I don't know the internals of llama.cpp, but amd64 isn't related to your issue. It's mostly CUDA.

AMD is extremely lucky with the development in this project. They couldn't have closed the >10-year-dev-gap otherwise.
I guess I'll need to look for other quantizations and better Nvidia-specific optimizations.

Regards
G.

[Borobo]

I have the same performance on an M3 Max as OP when running the benchmark, but there seems to be a big gap between llama and Llava performance. As the Llama and Llava 1.5 architectures don't differ much (Llava 1.5 = ViT + Projection Layer + Llama), I'm quite surprised. I'm sorry I can't give you any figures, as I don't think there's a benchmark script for multimodal in llama.cpp. Does anyone have the same felling?

[Borobo]

It seems that the issue is rather linked to metal than Llava, here's my result after running batched-bench with TheBloke/Mistral-7B-v0.1-GGUF:

First of all, is "default.metallib not found" normal?
Then, I think there is a problem concerning S_PP S_TG t/s as their performances are even worse when I'm using llama.cpp batches. How is it possible that my S_TG t/s is divided by two when using two batches?
Is it linked to my configuration?

[ggerganov]

First of all, is "default.metallib not found" normal?

normal

How is it possible that my S_TG t/s is divided by two when using two batches?

This is a known limitation - I've written more about it here: #3524

Unfortunately, I haven't found a universal solution yet to achieve optimal performance for all batches for all Apple devices and models. The best option you have atm is to enable this #if and adjust the values for your use case:

llama.cpp/ggml-metal.m

Lines 1244 to 1270 in e00d2a6

	// find the break-even point where the matrix-matrix kernel becomes more efficient compared
	// to the matrix-vector kernel
	int ne11_mm_min = 1;
	#if 0
	// the numbers below are measured on M2 Ultra for 7B and 13B models
	// these numbers do not translate to other devices or model sizes
	// TODO: need to find a better approach
	if ([ctx->device.name isEqualToString:@"Apple M2 Ultra"]) {
	switch (src0t) {
	case GGML_TYPE_F16: ne11_mm_min = 2; break;
	case GGML_TYPE_Q8_0: ne11_mm_min = 7; break;
	case GGML_TYPE_Q2_K: ne11_mm_min = 15; break;
	case GGML_TYPE_Q3_K: ne11_mm_min = 7; break;
	case GGML_TYPE_Q4_0:
	case GGML_TYPE_Q4_1: ne11_mm_min = 15; break;
	case GGML_TYPE_Q4_K: ne11_mm_min = 11; break;
	case GGML_TYPE_Q5_0: // not tested yet
	case GGML_TYPE_Q5_1: ne11_mm_min = 13; break; // not tested yet
	case GGML_TYPE_Q5_K: ne11_mm_min = 7; break;
	case GGML_TYPE_Q6_K: ne11_mm_min = 7; break;
	default: ne11_mm_min = 1; break;
	}
	}
	#endif

It can lead to significant improvements for bs > 1

Additionally, in this branch I'm currently writing new matrix multiplication kernels aiming to resolve this problem: https://github.com/ggerganov/llama.cpp/tree/gg/metal-batched
However, the results are not looking great so far

[agnosticlines]

Hey I know this is focused on a very specific benchmark but wanted to draw attention to this as it is performance related albeit only affecting the new IQ quant types #5617 and didn't want it being lost in discussions forever

[grigio]

can somebody run the benchmark on Linux with Apple M1 or M2 and Asahi Linux ?

[aneeshmb02]

What data/prompts are used for this?

[beebopkim]

llama-bench generates prompts by itself. So there is no given data or prompts.