loadtest from local CLI works normally, but when run on Github actions self-hosted runner it returns errors.ETIMEDOUT

[Timur-Lenk]

I'm experiencing an issue with my CI/CD pipeline where my Artillery load tests are failing due to timeouts when run on a self-hosted GitHub Actions runner. The setup works perfectly on my local machine, but not in the CI/CD environment. I'm running the loadtest on my kubernetes deployment. Here are the details:

Workflow Configuration
GitHub Actions Workflow:

jobs:
  artillery:
    runs-on: self-hosted
    steps:
      - name: Checkout repository
        uses: actions/checkout@v2

      - name: Set up Node.js
        uses: actions/setup-node@v2
        with:
          node-version: '14'  # specify the node version you need

      - name: Install Artillery
        run: npm install -g artillery

      - name: Execute load tests
        run: artillery run ./Loadtest/gateway.yaml --record --output ./artillery-report.json --verbose
        env:
          ARTILLERY_CLOUD_API_KEY: ${{ secrets.ARTILLERY_CLOUD_KEY }}
      - name: Upload Artillery Report
        uses: actions/upload-artifact@v2
        with:
          name: artillery-report
          path: ./artillery-report.json

Artillery Configuration:

config:
  target: 'http://192.168.144.10:8080/api'
  phases:
    - duration: 60
      arrivalRate: 30
  ensure:
    maxErrorRate: 1
    max: 500
  defaults:
    headers:
      Content-Type: application/json
    http:
      timeout: 10000  # Increase timeout to 10 seconds
scenarios:
  - name: 'Get a list of posts'
    flow:
      - get:
          url: '/posts'
          expect:
            - statusCode: 200

Issue Description

When running the Artillery load tests on my local machine, all requests return a status code 200 as expected. However, when the same tests are executed in the CI/CD pipeline using a self-hosted GitHub Actions runner, I encounter numerous ETIMEDOUT errors.

errors.ETIMEDOUT: .............................................................. 196
http.request_rate: ............................................................. 30/sec
http.requests: ................................................................. 196
vusers.created: ................................................................ 196
vusers.failed: ................................................................. 196
...

Despite the self-hosted runner being on the same network as the target server (a Linux VM), all virtual users (VUs) fail with timeouts. Notably, running curl http://192.168.144.10:8080/api/posts from the self-hosted runner returns the expected data, indicating that the target server is accessible.

In order to connect to the VM one does need to connect to a VPN, but the self-hosted VM is already in the same network. So if running it on my pc works, I don't see why running it in the VM wouldn't

[hassy]

Very odd! ETIMEDOUT would imply that the connection is made successfully, but the server does not respond quickly enough. If Artillery could not connect you'd see ECONNREFUSED errors in the report. Timeouts can happen when the service is overloaded but you're only creating 30 VUs/second, so unless the service runs on a toaster that should not be the cause of the issue. You could try reducing the arrivalRate down to 1 just to rule that out though.

[Timur-Lenk]

It's not an issue of the VMs it runs on, because I had run some test loadtests from the CLI on my own PC which is connected to the VPN so it had direct access. And those tests performed perfectly, that's how I knew there wasn't an issue in my k8s deployment or the VMs themselves, but rather a connection issue.

Although I think I fixed, by simply running the self-hosted runner on the same vm, rather a different vm on the same network. I can even bump up the arrivalRate too 100 and I get these as the results:

--------------------------------
Summary report @ 11:42:41(+0200)
--------------------------------

http.codes.200: ................................................................ 3000
http.downloaded_bytes: ......................................................... 19080000
http.request_rate: ............................................................. 89/sec
http.requests: ................................................................. 3000
http.response_time:
  min: ......................................................................... 0
  max: ......................................................................... 19
  mean: ........................................................................ 1.4
  median: ...................................................................... 1
  p95: ......................................................................... 3
  p99: ......................................................................... 5
http.responses: ................................................................ 3000
vusers.completed: .............................................................. 3000
vusers.created: ................................................................ 3000
vusers.created_by_name.Get a list of posts: .................................... 3000
vusers.failed: ................................................................. 0
vusers.session_length:
  min: ......................................................................... 2.1
  max: ......................................................................... 29.3
  mean: ........................................................................ 3
  median: ...................................................................... 2.7
  p95: ......................................................................... 4.4
  p99: ......................................................................... 7.5

[70mm1]

I experience the same observation for running Artillery with Playwright in Jenkins (via an image from the Dockerfile). A curl is able to reach the target but Artillery times out.
This is even reproducible for arrivalRates of 1.

[Timur-Lenk]

I fixed it by running the self-hosted runner on the same vm, not sure if this is applicable to you though.