Disable benchmark on forked repo #7034

CISC · 2024-05-02T03:14:10Z

It's a little annoying to get spammed by failed runs on your forked repo... :)

ngxson · 2024-05-04T08:56:16Z

Good idea, but seems like this change also disable benchmark for all PRs, not just forked repo

CISC · 2024-05-04T09:12:27Z

Oh, hmmm, would have thought owner applied to the target repo, any ideas how to fix?

ngxson · 2024-05-04T09:23:09Z

Maybe leave all pull request related fields outside of owner check:

github.event.pull_request
github.event.push.ref
github.head_ref ==> See https://docs.github.com/en/actions/learn-github-actions/contexts

CISC · 2024-05-04T10:05:55Z

Actually, just checking owner on schedule event should be enough, the others are not likely to be triggered in most forks.

phymbert · 2024-05-04T10:20:42Z

It's pull_target_event for security reason, if you find a better one:
https://docs.github.com/en/actions/using-workflows/events-that-trigger-workflows

CISC · 2024-05-04T10:26:42Z

Anyone knows if it's possible to set an if condition on schedule itself? That would probably be even nicer.

CISC · 2024-05-04T11:45:43Z

Sigh, syncing fork counts as a push, so triggers also. The only check usable here seems to be the full repository in github.event.push.repository, not sure if I want to do that?

~~I guess getting a failed run every time you sync master is not that bad. :)~~

Or, it just occurred to me that the push event is only for the merge, not updating the PR, in which case, checking owner should be fine.

CISC · 2024-05-04T12:43:20Z

@ngxson / @phymbert I think it should be good now?

.github/workflows/bench.yml

ngxson · 2024-05-04T18:25:01Z

@CISC Can you try adding something to llama.cpp file, then remove it, just to see if that still triggers benchmark workflow in PR?

The condition I'm talking about:

  pull_request_target:
    types: [opened, synchronize, reopened]
    paths: ['llama.cpp', 'ggml.c', 'ggml-backend.c', 'ggml-quants.c', '**/*.cu', 'examples/server/*.h*', 'examples/server/*.cpp']

CISC · 2024-05-04T18:48:33Z

@CISC Can you try adding something to llama.cpp file, then remove it, just to see if that still triggers benchmark workflow in PR?

I'm not sure I understand? You mean to this PR? Couldn't you just submit a PR to my fork for the same effect (without the (even messier) commit history :) )?

ngxson · 2024-05-04T19:25:44Z

I'm not very confident to merge without doing any test, so it's up to you to test it out. The way to test that I'm talking about is to trigger benchmark workflow.

We don't really care about commit history because PR are squashed anw.

CISC · 2024-05-04T19:50:23Z

@ngxson Looks good for PR, but it also triggered benchmark on my fork, not sure why it did that, only thing I can think of triggering it is github.head_ref, but that should only exist for pull_request according to docs?

github-actions · 2024-05-04T20:13:15Z

📈 llama.cpp server for bench-server-baseline on Standard_NC4as_T4_v3 for phi-2-q4_0: 435 iterations 🚀

Expand details for performance related PR only

Concurrent users: 8, duration: 10m
HTTP request : avg=10863.25ms p(95)=27476.04ms fails=, finish reason: stop=386 truncated=49
Prompt processing (pp): avg=122.76tk/s p(95)=536.29tk/s
Token generation (tg): avg=25.89tk/s p(95)=36.27tk/s
ggml-org/models/phi-2/ggml-model-q4_0.gguf parallel=8 ctx-size=16384 ngl=33 batch-size=2048 ubatch-size=256 pp=1024 pp+tg=2048 branch=master commit=94ebaae0222dff891555a32f5739f4268d684717

More

---
config:
    xyChart:
        titleFontSize: 12
        width: 900
        height: 600
    themeVariables:
        xyChart:
            titleColor: "#000000"
---
xychart-beta
    title "llama.cpp bench-server-baseline on Standard_NC4as_T4_v3
 duration=10m 435 iterations"
    y-axis "llamacpp:prompt_tokens_seconds"
    x-axis "llamacpp:prompt_tokens_seconds" 1714861391 --> 1714862023
    line [0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 390.95, 390.95, 390.95, 390.95, 390.95, 775.25, 775.25, 775.25, 775.25, 775.25, 563.78, 563.78, 563.78, 563.78, 563.78, 587.18, 587.18, 587.18, 587.18, 587.18, 610.91, 610.91, 610.91, 610.91, 610.91, 645.04, 645.04, 645.04, 645.04, 645.04, 659.85, 659.85, 659.85, 659.85, 659.85, 661.29, 661.29, 661.29, 661.29, 661.29, 674.81, 674.81, 674.81, 674.81, 674.81, 675.7, 675.7, 675.7, 675.7, 675.7, 676.92, 676.92, 676.92, 676.92, 676.92, 682.14, 682.14, 682.14, 682.14, 682.14, 683.23, 683.23, 683.23, 683.23, 683.23, 697.72, 697.72, 697.72, 697.72, 697.72, 716.96, 716.96, 716.96, 716.96, 716.96, 730.65, 730.65, 730.65, 730.65, 730.65, 748.16, 748.16, 748.16, 748.16, 748.16, 674.46, 674.46, 674.46, 674.46, 674.46, 681.89, 681.89, 681.89, 681.89, 681.89, 682.89, 682.89, 682.89, 682.89, 682.89, 686.86, 686.86, 686.86, 686.86, 686.86, 693.09, 693.09, 693.09, 693.09, 693.09, 694.44, 694.44, 694.44, 694.44, 694.44, 692.84, 692.84, 692.84, 692.84, 692.84, 697.66, 697.66, 697.66, 697.66, 697.66, 699.65, 699.65, 699.65, 699.65, 699.65, 701.67, 701.67, 701.67, 701.67, 701.67, 704.96, 704.96, 704.96, 704.96, 704.96, 705.55, 705.55, 705.55, 705.55, 705.55, 707.83, 707.83, 707.83, 707.83, 707.83, 714.84, 714.84, 714.84, 714.84, 714.84, 716.62, 716.62, 716.62, 716.62, 716.62, 714.12, 714.12, 714.12, 714.12, 714.12, 713.42, 713.42, 713.42, 713.42, 713.42, 714.0, 714.0, 714.0, 714.0, 714.0, 718.61, 718.61, 718.61, 718.61, 718.61, 719.54, 719.54, 719.54, 719.54, 719.54, 718.88, 718.88, 718.88, 718.88, 718.88, 721.52, 721.52, 721.52, 721.52, 721.52, 727.24, 727.24, 727.24, 727.24, 727.24, 734.52, 734.52, 734.52, 734.52, 734.52, 734.61, 734.61, 734.61, 734.61, 734.61, 733.88, 733.88, 733.88, 733.88, 733.88, 731.41, 731.41, 731.41, 731.41, 731.41, 726.21, 726.21, 726.21, 726.21, 726.21, 725.2, 725.2, 725.2, 725.2, 725.2, 724.89, 724.89, 724.89, 724.89, 724.89, 727.92, 727.92, 727.92, 727.92, 727.92, 729.76, 729.76, 729.76, 729.76, 729.76, 732.52, 732.52, 732.52, 732.52, 732.52, 720.98, 720.98, 720.98, 720.98, 720.98, 693.91, 693.91, 693.91, 693.91, 693.91, 679.68, 679.68, 679.68, 679.68, 679.68, 677.94, 677.94, 677.94, 677.94, 677.94, 677.45, 677.45, 677.45, 677.45, 677.45, 674.78, 674.78, 674.78, 674.78, 674.78, 673.64, 673.64, 673.64, 673.64, 673.64, 673.63, 673.63, 673.63, 673.63, 673.63, 676.19, 676.19, 676.19, 676.19, 676.19, 678.06, 678.06, 678.06, 678.06, 678.06, 678.3, 678.3, 678.3, 678.3, 678.3, 678.21, 678.21]

More

---
config:
    xyChart:
        titleFontSize: 12
        width: 900
        height: 600
    themeVariables:
        xyChart:
            titleColor: "#000000"
---
xychart-beta
    title "llama.cpp bench-server-baseline on Standard_NC4as_T4_v3
 duration=10m 435 iterations"
    y-axis "llamacpp:predicted_tokens_seconds"
    x-axis "llamacpp:predicted_tokens_seconds" 1714861391 --> 1714862023
    line [0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 29.64, 29.64, 29.64, 29.64, 29.64, 20.36, 20.36, 20.36, 20.36, 20.36, 21.09, 21.09, 21.09, 21.09, 21.09, 21.76, 21.76, 21.76, 21.76, 21.76, 22.08, 22.08, 22.08, 22.08, 22.08, 22.71, 22.71, 22.71, 22.71, 22.71, 23.8, 23.8, 23.8, 23.8, 23.8, 24.7, 24.7, 24.7, 24.7, 24.7, 24.94, 24.94, 24.94, 24.94, 24.94, 24.96, 24.96, 24.96, 24.96, 24.96, 24.55, 24.55, 24.55, 24.55, 24.55, 24.41, 24.41, 24.41, 24.41, 24.41, 24.59, 24.59, 24.59, 24.59, 24.59, 24.63, 24.63, 24.63, 24.63, 24.63, 23.96, 23.96, 23.96, 23.96, 23.96, 23.77, 23.77, 23.77, 23.77, 23.77, 23.24, 23.24, 23.24, 23.24, 23.24, 22.94, 22.94, 22.94, 22.94, 22.94, 23.03, 23.03, 23.03, 23.03, 23.03, 23.13, 23.13, 23.13, 23.13, 23.13, 22.94, 22.94, 22.94, 22.94, 22.94, 22.68, 22.68, 22.68, 22.68, 22.68, 22.68, 22.68, 22.68, 22.68, 22.68, 22.65, 22.65, 22.65, 22.65, 22.65, 22.64, 22.64, 22.64, 22.64, 22.64, 22.75, 22.75, 22.75, 22.75, 22.75, 22.79, 22.79, 22.79, 22.79, 22.79, 22.81, 22.81, 22.81, 22.81, 22.81, 22.82, 22.82, 22.82, 22.82, 22.82, 23.0, 23.0, 23.0, 23.0, 23.0, 22.84, 22.84, 22.84, 22.84, 22.84, 22.63, 22.63, 22.63, 22.63, 22.63, 22.29, 22.29, 22.29, 22.29, 22.29, 22.35, 22.35, 22.35, 22.35, 22.35, 22.45, 22.45, 22.45, 22.45, 22.45, 22.61, 22.61, 22.61, 22.61, 22.61, 22.73, 22.73, 22.73, 22.73, 22.73, 22.84, 22.84, 22.84, 22.84, 22.84, 22.81, 22.81, 22.81, 22.81, 22.81, 22.8, 22.8, 22.8, 22.8, 22.8, 22.7, 22.7, 22.7, 22.7, 22.7, 22.65, 22.65, 22.65, 22.65, 22.65, 22.03, 22.03, 22.03, 22.03, 22.03, 21.87, 21.87, 21.87, 21.87, 21.87, 21.91, 21.91, 21.91, 21.91, 21.91, 22.03, 22.03, 22.03, 22.03, 22.03, 22.09, 22.09, 22.09, 22.09, 22.09, 22.23, 22.23, 22.23, 22.23, 22.23, 22.31, 22.31, 22.31, 22.31, 22.31, 22.13, 22.13, 22.13, 22.13, 22.13, 22.01, 22.01, 22.01, 22.01, 22.01, 22.01, 22.01, 22.01, 22.01, 22.01, 21.77, 21.77, 21.77, 21.77, 21.77, 21.59, 21.59, 21.59, 21.59, 21.59, 21.5, 21.5, 21.5, 21.5, 21.5, 20.88, 20.88, 20.88, 20.88, 20.88, 20.36, 20.36, 20.36, 20.36, 20.36, 20.23, 20.23, 20.23, 20.23, 20.23, 20.24, 20.24, 20.24, 20.24, 20.24, 20.35, 20.35, 20.35, 20.35, 20.35, 20.43, 20.43]

Details

More

---
config:
    xyChart:
        titleFontSize: 12
        width: 900
        height: 600
    themeVariables:
        xyChart:
            titleColor: "#000000"
---
xychart-beta
    title "llama.cpp bench-server-baseline on Standard_NC4as_T4_v3
 duration=10m 435 iterations"
    y-axis "llamacpp:kv_cache_usage_ratio"
    x-axis "llamacpp:kv_cache_usage_ratio" 1714861391 --> 1714862023
    line [0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.06, 0.06, 0.06, 0.06, 0.06, 0.37, 0.37, 0.37, 0.37, 0.37, 0.35, 0.35, 0.35, 0.35, 0.35, 0.21, 0.21, 0.21, 0.21, 0.21, 0.16, 0.16, 0.16, 0.16, 0.16, 0.16, 0.16, 0.16, 0.16, 0.16, 0.11, 0.11, 0.11, 0.11, 0.11, 0.11, 0.11, 0.11, 0.11, 0.11, 0.13, 0.13, 0.13, 0.13, 0.13, 0.12, 0.12, 0.12, 0.12, 0.12, 0.19, 0.19, 0.19, 0.19, 0.19, 0.23, 0.23, 0.23, 0.23, 0.23, 0.1, 0.1, 0.1, 0.1, 0.1, 0.1, 0.1, 0.1, 0.1, 0.1, 0.26, 0.26, 0.26, 0.26, 0.26, 0.16, 0.16, 0.16, 0.16, 0.16, 0.3, 0.3, 0.3, 0.3, 0.3, 0.12, 0.12, 0.12, 0.12, 0.12, 0.17, 0.17, 0.17, 0.17, 0.17, 0.18, 0.18, 0.18, 0.18, 0.18, 0.21, 0.21, 0.21, 0.21, 0.21, 0.22, 0.22, 0.22, 0.22, 0.22, 0.21, 0.21, 0.21, 0.21, 0.21, 0.21, 0.21, 0.21, 0.21, 0.21, 0.17, 0.17, 0.17, 0.17, 0.17, 0.16, 0.16, 0.16, 0.16, 0.16, 0.16, 0.16, 0.16, 0.16, 0.16, 0.18, 0.18, 0.18, 0.18, 0.18, 0.12, 0.12, 0.12, 0.12, 0.12, 0.1, 0.1, 0.1, 0.1, 0.1, 0.22, 0.22, 0.22, 0.22, 0.22, 0.32, 0.32, 0.32, 0.32, 0.32, 0.32, 0.32, 0.32, 0.32, 0.32, 0.23, 0.23, 0.23, 0.23, 0.23, 0.14, 0.14, 0.14, 0.14, 0.14, 0.12, 0.12, 0.12, 0.12, 0.12, 0.09, 0.09, 0.09, 0.09, 0.09, 0.14, 0.14, 0.14, 0.14, 0.14, 0.14, 0.14, 0.14, 0.14, 0.14, 0.19, 0.19, 0.19, 0.19, 0.19, 0.17, 0.17, 0.17, 0.17, 0.17, 0.26, 0.26, 0.26, 0.26, 0.26, 0.32, 0.32, 0.32, 0.32, 0.32, 0.32, 0.32, 0.32, 0.32, 0.32, 0.11, 0.11, 0.11, 0.11, 0.11, 0.16, 0.16, 0.16, 0.16, 0.16, 0.09, 0.09, 0.09, 0.09, 0.09, 0.1, 0.1, 0.1, 0.1, 0.1, 0.14, 0.14, 0.14, 0.14, 0.14, 0.23, 0.23, 0.23, 0.23, 0.23, 0.39, 0.39, 0.39, 0.39, 0.39, 0.47, 0.47, 0.47, 0.47, 0.47, 0.54, 0.54, 0.54, 0.54, 0.54, 0.59, 0.59, 0.59, 0.59, 0.59, 0.58, 0.58, 0.58, 0.58, 0.58, 0.57, 0.57, 0.57, 0.57, 0.57, 0.52, 0.52, 0.52, 0.52, 0.52, 0.13, 0.13, 0.13, 0.13, 0.13, 0.14, 0.14, 0.14, 0.14, 0.14, 0.16, 0.16, 0.16, 0.16, 0.16, 0.14, 0.14, 0.14, 0.14, 0.14, 0.18, 0.18]

More

---
config:
    xyChart:
        titleFontSize: 12
        width: 900
        height: 600
    themeVariables:
        xyChart:
            titleColor: "#000000"
---
xychart-beta
    title "llama.cpp bench-server-baseline on Standard_NC4as_T4_v3
 duration=10m 435 iterations"
    y-axis "llamacpp:requests_processing"
    x-axis "llamacpp:requests_processing" 1714861391 --> 1714862023
    line [0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 7.0, 7.0, 7.0, 7.0, 7.0, 6.0, 6.0, 6.0, 6.0, 6.0, 2.0, 2.0, 2.0, 2.0, 2.0, 3.0, 3.0, 3.0, 3.0, 3.0, 4.0, 4.0, 4.0, 4.0, 4.0, 5.0, 5.0, 5.0, 5.0, 5.0, 6.0, 6.0, 6.0, 6.0, 6.0, 7.0, 7.0, 7.0, 7.0, 7.0, 3.0, 3.0, 3.0, 3.0, 3.0, 6.0, 6.0, 6.0, 6.0, 6.0, 8.0, 8.0, 8.0, 8.0, 8.0, 4.0, 4.0, 4.0, 4.0, 4.0, 6.0, 6.0, 6.0, 6.0, 6.0, 8.0, 8.0, 8.0, 8.0, 8.0, 7.0, 7.0, 7.0, 7.0, 7.0, 6.0, 6.0, 6.0, 6.0, 6.0, 3.0, 3.0, 3.0, 3.0, 3.0, 7.0, 7.0, 7.0, 7.0, 7.0, 6.0, 6.0, 6.0, 6.0, 6.0, 6.0, 6.0, 6.0, 6.0, 6.0, 6.0, 6.0, 6.0, 6.0, 6.0, 6.0, 6.0, 6.0, 6.0, 6.0, 5.0, 5.0, 5.0, 5.0, 5.0, 6.0, 6.0, 6.0, 6.0, 6.0, 7.0, 7.0, 7.0, 7.0, 7.0, 6.0, 6.0, 6.0, 6.0, 6.0, 3.0, 3.0, 3.0, 3.0, 3.0, 7.0, 7.0, 7.0, 7.0, 7.0, 2.0, 2.0, 2.0, 2.0, 2.0, 8.0, 8.0, 8.0, 8.0, 8.0, 7.0, 7.0, 7.0, 7.0, 7.0, 6.0, 6.0, 6.0, 6.0, 6.0, 7.0, 7.0, 7.0, 7.0, 7.0, 5.0, 5.0, 5.0, 5.0, 5.0, 1.0, 1.0, 1.0, 1.0, 1.0, 6.0, 6.0, 6.0, 6.0, 6.0, 6.0, 6.0, 6.0, 6.0, 6.0, 6.0, 6.0, 6.0, 6.0, 6.0, 3.0, 3.0, 3.0, 3.0, 3.0, 2.0, 2.0, 2.0, 2.0, 2.0, 8.0, 8.0, 8.0, 8.0, 8.0, 7.0, 7.0, 7.0, 7.0, 7.0, 8.0, 8.0, 8.0, 8.0, 8.0, 4.0, 4.0, 4.0, 4.0, 4.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 2.0, 2.0, 2.0, 2.0, 2.0, 3.0, 3.0, 3.0, 3.0, 3.0, 6.0, 6.0, 6.0, 6.0, 6.0, 5.0, 5.0, 5.0, 5.0, 5.0, 6.0, 6.0, 6.0, 6.0, 6.0, 7.0, 7.0, 7.0, 7.0, 7.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 5.0, 5.0, 5.0, 5.0, 5.0, 8.0, 8.0, 8.0, 8.0, 8.0, 6.0, 6.0, 6.0, 6.0, 6.0, 8.0, 8.0, 8.0, 8.0, 8.0, 3.0, 3.0, 3.0, 3.0, 3.0, 2.0, 2.0]

phymbert

Hum, let's see if it works :)

CISC · 2024-05-04T20:32:04Z

Hum, let's see if it works :)

I'm trying to debug the if...

CISC · 2024-05-04T20:39:05Z

Sigh, I can't get any debug logs from this action as it just hangs waiting for the runner.

CISC · 2024-05-04T21:32:50Z

This looks a lot better:

##[debug]Evaluating: (success() && ((inputs.gpu-series == 'Standard_NC4as_T4_v3') || ((github.event_name == 'schedule') && (github.repository_owner == 'ggerganov')) || (github.event_name == 'pull_request') || (github.head_ref == 'master') || ((github.event_name == 'push') && (github.event.ref == 'refs/heads/master') && (github.repository_owner == 'ggerganov')) || ((github.event_name != 'push') && (github.ref_name == 'master'))))
##[debug]Evaluating And:
##[debug]..Evaluating success:
##[debug]..=> true
##[debug]..Evaluating Or:
##[debug]....Evaluating Equal:
##[debug]......Evaluating Index:
##[debug]........Evaluating inputs:
##[debug]........=> Object
##[debug]........Evaluating String:
##[debug]........=> 'gpu-series'
##[debug]......=> null
##[debug]......Evaluating String:
##[debug]......=> 'Standard_NC4as_T4_v3'
##[debug]....=> false
##[debug]....Evaluating And:
##[debug]......Evaluating Equal:
##[debug]........Evaluating Index:
##[debug]..........Evaluating github:
##[debug]..........=> Object
##[debug]..........Evaluating String:
##[debug]..........=> 'event_name'
##[debug]........=> 'push'
##[debug]........Evaluating String:
##[debug]........=> 'schedule'
##[debug]......=> false
##[debug]....=> false
##[debug]....Evaluating Equal:
##[debug]......Evaluating Index:
##[debug]........Evaluating github:
##[debug]........=> Object
##[debug]........Evaluating String:
##[debug]........=> 'event_name'
##[debug]......=> 'push'
##[debug]......Evaluating String:
##[debug]......=> 'pull_request'
##[debug]....=> false
##[debug]....Evaluating Equal:
##[debug]......Evaluating Index:
##[debug]........Evaluating github:
##[debug]........=> Object
##[debug]........Evaluating String:
##[debug]........=> 'head_ref'
##[debug]......=> ''
##[debug]......Evaluating String:
##[debug]......=> 'master'
##[debug]....=> false
##[debug]....Evaluating And:
##[debug]......Evaluating Equal:
##[debug]........Evaluating Index:
##[debug]..........Evaluating github:
##[debug]..........=> Object
##[debug]..........Evaluating String:
##[debug]..........=> 'event_name'
##[debug]........=> 'push'
##[debug]........Evaluating String:
##[debug]........=> 'push'
##[debug]......=> true
##[debug]......Evaluating Equal:
##[debug]........Evaluating Index:
##[debug]..........Evaluating Index:
##[debug]............Evaluating github:
##[debug]............=> Object
##[debug]............Evaluating String:
##[debug]............=> 'event'
##[debug]..........=> Object
##[debug]..........Evaluating String:
##[debug]..........=> 'ref'
##[debug]........=> 'refs/heads/master'
##[debug]........Evaluating String:
##[debug]........=> 'refs/heads/master'
##[debug]......=> true
##[debug]......Evaluating Equal:
##[debug]........Evaluating Index:
##[debug]..........Evaluating github:
##[debug]..........=> Object
##[debug]..........Evaluating String:
##[debug]..........=> 'repository_owner'
##[debug]........=> 'CISC'
##[debug]........Evaluating String:
##[debug]........=> 'ggerganov'
##[debug]......=> false
##[debug]....=> false
##[debug]....Evaluating And:
##[debug]......Evaluating NotEqual:
##[debug]........Evaluating Index:
##[debug]..........Evaluating github:
##[debug]..........=> Object
##[debug]..........Evaluating String:
##[debug]..........=> 'event_name'
##[debug]........=> 'push'
##[debug]........Evaluating String:
##[debug]........=> 'push'
##[debug]......=> false
##[debug]....=> false
##[debug]..=> false
##[debug]=> false
##[debug]Expanded: (true && ((null == 'Standard_NC4as_T4_v3') || (('push' == 'schedule') && (github['repository_owner'] == 'ggerganov')) || ('push' == 'pull_request') || ('' == 'master') || (('push' == 'push') && ('refs/heads/master' == 'refs/heads/master') && ('CISC' == 'ggerganov')) || (('push' != 'push') && (github['ref_name'] == 'master'))))
##[debug]Result: false

CISC · 2024-05-04T21:40:28Z

Yay, benchmark skipped on fork, queued here!

CISC · 2024-05-04T21:43:46Z

All done, looks like it's working now, and should have fixed some previous issues with this condition as well! :)

phymbert · 2024-05-04T21:44:39Z

Wait, it's a pull request target event. The master file is in used...

CISC · 2024-05-04T21:45:29Z

Wait, it's a pull request target event. The master file is in used...

Ah, you mean the event name? Right.

CISC · 2024-05-04T22:12:24Z

Alright, this time benchmark is actually running, all good? ~~Huh, and suddenly it's skipped, weird, I just saw it running...~~ And it shows as successful again, must have been a glitch. :)

CISC · 2024-05-04T22:25:06Z

Looking more closely at bench.yml there are more uses of github.event that are incorrect, strange...

phymbert · 2024-05-04T22:34:39Z

https://docs.github.com/en/actions/using-workflows/events-that-trigger-workflows#pull_request_target

CISC · 2024-05-04T22:57:31Z

@phymbert Yeah, ok? I'm confused, do I need to do anything special, seems to work now?

CISC · 2024-05-05T06:35:22Z

Ok, benchmark is still scheduled on fork, so not quite working.

I think it's the github.head_ref and github.ref_name checks, what are they for?

According to docs github.head_ref only exists for pull requests, but in that case it was short-circuited by the github.event.pull_request, so it looks like this can be removed? Or was it meant to be &&?

Finally, I guess github.ref_name should only be checked together with schedule event?

.github/workflows/bench.yml

ngxson · 2024-05-05T11:38:40Z

Small note that EditorConfig check failed because it's taking code from master branch. This is not due to this PR, so I'm merging now.

Thanks @CISC for the work

* Disable benchmark on forked repo * only check owner on schedule event * check owner on push also * more readable as multi-line * ternary won't work * style++ * test++ * enable actions debug * test-- * remove debug * test++ * do debug where we can get logs * test-- * this is driving me crazy * correct github.event usage * remove test condition * correct github.event usage * test++ * test-- * event_name is pull_request_target * test++ * test-- * update ref checks

Discovered this while debugging ggerganov#7034 **Warning**: This may change current behavior as all these evaluated to null before!

* Disable benchmark on forked repo * only check owner on schedule event * check owner on push also * more readable as multi-line * ternary won't work * style++ * test++ * enable actions debug * test-- * remove debug * test++ * do debug where we can get logs * test-- * this is driving me crazy * correct github.event usage * remove test condition * correct github.event usage * test++ * test-- * event_name is pull_request_target * test++ * test-- * update ref checks

Disable benchmark on forked repo

abf0ff0

only check owner on schedule event

d686299

check owner on push also

b259634

ngxson reviewed May 4, 2024

View reviewed changes

.github/workflows/bench.yml Outdated Show resolved Hide resolved

CISC added 3 commits May 4, 2024 19:28

more readable as multi-line

f006b5c

ternary won't work

c4ae6c1

style++

2d4be61

CISC requested a review from ngxson May 4, 2024 18:09

test++

bde2fc7

phymbert approved these changes May 4, 2024

View reviewed changes

CISC added 4 commits May 4, 2024 22:24

enable actions debug

86394bf

test--

f15f9d9

remove debug

29c30fa

test++

8122c94

do debug where we can get logs

fb7304b

correct github.event usage

dfb33e8

CISC added 3 commits May 4, 2024 23:34

remove test condition

5e8ca26

correct github.event usage

c3d9d70

test++

ab57af8

test--

06e9307

CISC requested review from phymbert and ngxson May 4, 2024 21:42

CISC added 3 commits May 5, 2024 00:01

event_name is pull_request_target

840986b

test++

94ebaae

test--

4bcd1d8

update ref checks

589c74d

phymbert requested changes May 5, 2024

View reviewed changes

.github/workflows/bench.yml Show resolved Hide resolved

phymbert approved these changes May 5, 2024

View reviewed changes

ngxson approved these changes May 5, 2024

View reviewed changes

ngxson merged commit 8f8acc8 into ggerganov:master May 5, 2024
20 of 21 checks passed

CISC added a commit to CISC/llama.cpp that referenced this pull request May 6, 2024

Fix incorrect usage of github.event

cbb1583

Discovered this while debugging ggerganov#7034 **Warning**: This may change current behavior as all these evaluated to null before!

CISC mentioned this pull request May 6, 2024

Fix incorrect usage of github.event #7103

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Disable benchmark on forked repo #7034

Disable benchmark on forked repo #7034

CISC commented May 2, 2024

ngxson commented May 4, 2024

CISC commented May 4, 2024

ngxson commented May 4, 2024

CISC commented May 4, 2024

phymbert commented May 4, 2024

CISC commented May 4, 2024 •

edited

CISC commented May 4, 2024 •

edited

CISC commented May 4, 2024

ngxson commented May 4, 2024

CISC commented May 4, 2024

ngxson commented May 4, 2024

CISC commented May 4, 2024 •

edited

github-actions bot commented May 4, 2024 •

edited

phymbert left a comment

CISC commented May 4, 2024

CISC commented May 4, 2024

CISC commented May 4, 2024

CISC commented May 4, 2024

CISC commented May 4, 2024

phymbert commented May 4, 2024

CISC commented May 4, 2024 •

edited

CISC commented May 4, 2024 •

edited

CISC commented May 4, 2024

phymbert commented May 4, 2024

CISC commented May 4, 2024

CISC commented May 5, 2024

ngxson commented May 5, 2024 •

edited

Disable benchmark on forked repo #7034

Disable benchmark on forked repo #7034

Conversation

CISC commented May 2, 2024

ngxson commented May 4, 2024

CISC commented May 4, 2024

ngxson commented May 4, 2024

CISC commented May 4, 2024

phymbert commented May 4, 2024

CISC commented May 4, 2024 • edited

CISC commented May 4, 2024 • edited

CISC commented May 4, 2024

ngxson commented May 4, 2024

CISC commented May 4, 2024

ngxson commented May 4, 2024

CISC commented May 4, 2024 • edited

github-actions bot commented May 4, 2024 • edited

phymbert left a comment

Choose a reason for hiding this comment

CISC commented May 4, 2024

CISC commented May 4, 2024

CISC commented May 4, 2024

CISC commented May 4, 2024

CISC commented May 4, 2024

phymbert commented May 4, 2024

CISC commented May 4, 2024 • edited

CISC commented May 4, 2024 • edited

CISC commented May 4, 2024

phymbert commented May 4, 2024

CISC commented May 4, 2024

CISC commented May 5, 2024

ngxson commented May 5, 2024 • edited

CISC commented May 4, 2024 •

edited

CISC commented May 4, 2024 •

edited

CISC commented May 4, 2024 •

edited

github-actions bot commented May 4, 2024 •

edited

CISC commented May 4, 2024 •

edited

CISC commented May 4, 2024 •

edited

ngxson commented May 5, 2024 •

edited