perf: Benchmark `nodes-base` #9355

ivov · 2024-05-10T10:13:46Z

WIP

codspeed-hq · 2024-05-10T10:29:16Z

CodSpeed Performance Report

Congrats! CodSpeed is installed 🎉

🆕 93 new benchmarks were detected.

You will start to see performance impacts in the reports once the benchmarks are run from your default branch.

Detected benchmarks

nodes/Aws/S3/test/V1/AwsS3.file.upload.V1.workflow (39.9 ms)
nodes/Aws/S3/test/V2/AwsS3.file.upload.V2.workflow (39.9 ms)
nodes/Code/test/Code.workflow (245.5 ms)
nodes/Crypto/test/CryptoTest.workflow (31.5 ms)
nodes/Discord/test/v2/node/channel/create.workflow (43.7 ms)
nodes/Discord/test/v2/node/channel/deleteChannel.workflow (43.4 ms)
nodes/Discord/test/v2/node/channel/get.workflow (42.5 ms)
nodes/Discord/test/v2/node/channel/getAll.workflow (61.3 ms)
nodes/Discord/test/v2/node/channel/update.workflow (44.6 ms)
nodes/Discord/test/v2/node/member/getAll.workflow (44.3 ms)
nodes/Discord/test/v2/node/member/roleAdd.workflow (43.7 ms)
nodes/Discord/test/v2/node/member/roleRemove.workflow (72 ms)
nodes/Discord/test/v2/node/message/deleteMessage.workflow (43.7 ms)
nodes/Discord/test/v2/node/message/get.workflow (44.1 ms)
nodes/Discord/test/v2/node/message/getAll.workflow (44 ms)
nodes/Discord/test/v2/node/message/react.workflow (56.5 ms)
nodes/Discord/test/v2/node/message/send.workflow (46.2 ms)
nodes/Discord/test/v2/node/webhook/sendLegacy.workflow (15.8 ms)
nodes/Files/ConvertToFile/test/toText.workflow (42.4 ms)
nodes/Files/ReadWriteFile/test/ReadWriteFile.workflow (4 ms)
nodes/Files/test/convert_extract.workflow (30.7 ms)
nodes/Google/BigQuery/test/v2/node/executeQuery.workflow (5.7 ms)
nodes/Google/BigQuery/test/v2/node/insert.autoMapMode.workflow (42.7 ms)
nodes/Google/BigQuery/test/v2/node/insert.manualMode.workflow (5.8 ms)
nodes/Html/test/extractHtmlContent.workflow (282.9 ms)
nodes/Jwt/test/jwt.workflow (106.4 ms)
nodes/Microsoft/Excel/test/v2/node/table/addTable.workflow (10.3 ms)
nodes/Microsoft/Excel/test/v2/node/table/append.workflow (12.1 ms)
nodes/Microsoft/Excel/test/v2/node/table/convertToRange.workflow (10.4 ms)
nodes/Microsoft/Excel/test/v2/node/table/deleteTable.workflow (10.5 ms)
nodes/Microsoft/Excel/test/v2/node/table/getColumns.workflow (11.1 ms)
nodes/Microsoft/Excel/test/v2/node/table/getRows.workflow (11.7 ms)
nodes/Microsoft/Excel/test/v2/node/table/lookup.workflow (11.7 ms)
nodes/Microsoft/Excel/test/v2/node/workbook/addWorksheet.workflow (18.6 ms)
nodes/Microsoft/Excel/test/v2/node/workbook/deleteWorkbook.workflow (10.2 ms)
nodes/Microsoft/Excel/test/v2/node/workbook/getAll.workflow (9.7 ms)
nodes/Microsoft/Excel/test/v2/node/worksheet/append.workflow (18.9 ms)
nodes/Microsoft/Excel/test/v2/node/worksheet/clear.workflow (10.9 ms)
nodes/Microsoft/Excel/test/v2/node/worksheet/deleteWorksheet.workflow (10.8 ms)
nodes/Microsoft/Excel/test/v2/node/worksheet/getAll.workflow (10.9 ms)
nodes/Microsoft/Excel/test/v2/node/worksheet/readRows.workflow (15.2 ms)
nodes/Microsoft/Excel/test/v2/node/worksheet/update.workflow (19.7 ms)
nodes/Microsoft/Excel/test/v2/node/worksheet/upsert.workflow (20.1 ms)
nodes/Microsoft/Outlook/test/v2/node/calendar/create.workflow (120.7 ms)
nodes/Microsoft/Outlook/test/v2/node/calendar/delete.workflow (104.3 ms)
nodes/Microsoft/Outlook/test/v2/node/calendar/get.workflow (104.7 ms)
nodes/Microsoft/Outlook/test/v2/node/calendar/getAll.workflow (104.7 ms)
nodes/Microsoft/Outlook/test/v2/node/calendar/update.workflow (536.4 ms)
nodes/Microsoft/Outlook/test/v2/node/contact/create.workflow (104.4 ms)
nodes/Microsoft/Outlook/test/v2/node/contact/update.workflow (105.3 ms)
nodes/Microsoft/Outlook/test/v2/node/draft/create.workflow (104.7 ms)
nodes/Microsoft/Outlook/test/v2/node/draft/send.workflow (105 ms)
nodes/Microsoft/Outlook/test/v2/node/event/create.workflow (106.2 ms)
nodes/Microsoft/Outlook/test/v2/node/folder/create.workflow (105 ms)
nodes/Microsoft/Outlook/test/v2/node/folderMessage/getAll.workflow (106.9 ms)
nodes/Microsoft/Outlook/test/v2/node/message/move.workflow (118.2 ms)
nodes/Microsoft/Outlook/test/v2/node/message/reply.workflow (106.5 ms)
nodes/Microsoft/Outlook/test/v2/node/message/send.workflow (105.3 ms)
nodes/Microsoft/Teams/test/v2/node/channel/create.workflow (5.5 ms)
nodes/Microsoft/Teams/test/v2/node/channel/deleteChannel.workflow (5.5 ms)
nodes/Microsoft/Teams/test/v2/node/channel/get.workflow (5.5 ms)
nodes/Microsoft/Teams/test/v2/node/channel/getAll.workflow (5.6 ms)
nodes/Microsoft/Teams/test/v2/node/channel/update.workflow (5.5 ms)
nodes/Microsoft/Teams/test/v2/node/channelMessage/create.workflow (5.5 ms)
nodes/Microsoft/Teams/test/v2/node/channelMessage/getAll.workflow (5.5 ms)
nodes/Microsoft/Teams/test/v2/node/chatMessage/create.workflow (5.5 ms)
nodes/Microsoft/Teams/test/v2/node/chatMessage/get.workflow (5.5 ms)
nodes/Microsoft/Teams/test/v2/node/chatMessage/getAll.workflow (5.5 ms)
nodes/Microsoft/Teams/test/v2/node/task/create.workflow (5.7 ms)
nodes/Microsoft/Teams/test/v2/node/task/deleteTask.workflow (5.5 ms)
nodes/Microsoft/Teams/test/v2/node/task/get.workflow (5.5 ms)
nodes/Microsoft/Teams/test/v2/node/task/getAll.workflow (5.6 ms)
nodes/Microsoft/Teams/test/v2/node/task/update.workflow (5.5 ms)
nodes/MoveBinaryData/test/MoveBinaryData.workflow (7.1 ms)
nodes/MySql/test/v1/executeQuery.workflow (11.6 ms)
nodes/QuickChart/test/QuickChart.workflow (147.9 ms)
nodes/ReadBinaryFile/test/ReadBinaryFile.workflow (3.7 ms)
nodes/ReadBinaryFiles/test/ReadBinaryFiles.workflow (18 ms)
nodes/ReadPdf/test/ReadPDF-encrypted.workflow (3.8 ms)
nodes/ReadPdf/test/ReadPDF.workflow (3.9 ms)
nodes/RenameKeys/test/RenameKeys.workflow (18.9 ms)
nodes/Set/test/Set.v3.workflow (267.2 ms)
nodes/Set/test/Set.v3_3.workflow (237 ms)
nodes/Set/test/Set.workflow (47.2 ms)
nodes/Switch/V1/test/switch.expression.workflow (46.2 ms)
nodes/Switch/V1/test/switch.rules.workflow (23.9 ms)
nodes/Switch/V2/test/switch.expression.workflow (71.2 ms)
nodes/Switch/V2/test/switch.rules.workflow (56.3 ms)
nodes/Switch/V3/test/switch.expression.workflow (73 ms)
nodes/Switch/V3/test/switch.regex.workflow (33.1 ms)
nodes/Switch/V3/test/switch.rules.workflow (96.3 ms)
nodes/Wait/test/Wait.workflow (21.8 ms)
nodes/WriteBinaryFile/test/WriteBinaryFile.workflow (3.8 ms)

netroy

Looking pretty good. It's unfortunate that we have "only" 93 tests that can be used here. but, this is a great start 🎉 👏🏽

.github/workflows/benchmark-nodes-base.yml

netroy · 2024-05-10T11:21:43Z

packages/nodes-base/package.json

@@ -14,6 +14,8 @@
    "url": "git+https://github.com/n8n-io/n8n.git"
  },
  "scripts": {
+    "benchmark:build": "tsc -p tsconfig.benchmark.json > /dev/null; tsc-alias -p tsconfig.benchmark.json; pnpm build:metadata",


why the > /dev/null ?

To silence type errors when transpiling test helpers. Did not fix those here to avoid a larger PR.

netroy · 2024-05-10T11:22:59Z

packages/nodes-base/test/nodes/Helpers.ts

@@ -373,6 +386,9 @@ export const workflowToTests = (workflowFiles: string[]) => {
 				);
 			}
 		});
+
+		if (process.env.NODE_ENV === 'benchmark') workflowData.pinData = {};


we use pinData for assertions. does this change mean that we disable those assertions when running nodes tests for benchmarking?

Benchmarks use executeWorkflow but do not use equalityTest so they sidestep all Jest assertions.

This line specifically I added because a handful of workflows (MoveBinaryData, QuickChart) were getting caught by the pindata check right after this line, because they have their own implementation of testWorkflows.

netroy · 2024-05-10T11:24:59Z

packages/nodes-base/tsconfig.benchmark.json

@@ -0,0 +1,9 @@
+{


Could we do a on-the-fly compilation of TS code (like ts-jest does), instead of needing yet another tsconfig?

cypress · 2024-05-10T14:29:19Z

1 flaky test on run #4925 ↗︎

0	359	12	0	1

Details:

🌳 🖥️ browsers:node18.12.0-chrome107 🤖 ivov 🗃️ e2e/*
Project: n8n	Commit: `f581e40fe4`
Status: Passed	Duration: 04:23 💡
Started: May 10, 2024 2:24 PM	Ended: May 10, 2024 2:29 PM

cypress/e2e/5-ndv.cy.ts • 1 flaky test

View Output Video

Test		Artifacts
NDV > Stop listening for trigger event from NDV		`Screenshots` `Video`

Review all test suite changes for PR #9355 ↗︎

github-actions · 2024-05-10T14:29:19Z

✅ All Cypress E2E specs passed

ivov added 2 commits May 10, 2024 12:02

Initial setup

94f7fbe

Remove temp restriction

98c8938

ivov mentioned this pull request May 10, 2024

perf: Benchmark nodes-base #9317

Closed

Remove logging

4c8f903

netroy reviewed May 10, 2024

View reviewed changes

Better naming

34efe10

n8n-assistant bot added n8n team Authored by the n8n team node/improvement New feature or request labels May 10, 2024

netroy reviewed May 10, 2024

View reviewed changes

Remove cli path from GH action trigger

f581e40

netroy previously approved these changes May 10, 2024

View reviewed changes

perf: Reduce variance in nodes-base benchmarks (#9363)

4ccef05

ivov dismissed netroy’s stale review via 4ccef05 May 10, 2024 17:12

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

perf: Benchmark `nodes-base` #9355

perf: Benchmark `nodes-base` #9355

ivov commented May 10, 2024

codspeed-hq bot commented May 10, 2024 •

edited

Detected benchmarks

netroy left a comment

netroy May 10, 2024

ivov May 10, 2024

netroy May 10, 2024

ivov May 10, 2024

netroy May 10, 2024

cypress bot commented May 10, 2024

github-actions bot commented May 10, 2024

perf: Benchmark nodes-base #9355

Are you sure you want to change the base?

perf: Benchmark nodes-base #9355

Conversation

ivov commented May 10, 2024

codspeed-hq bot commented May 10, 2024 • edited

CodSpeed Performance Report

Congrats! CodSpeed is installed 🎉

Detected benchmarks

netroy left a comment

Choose a reason for hiding this comment

netroy May 10, 2024

Choose a reason for hiding this comment

ivov May 10, 2024

Choose a reason for hiding this comment

netroy May 10, 2024

Choose a reason for hiding this comment

ivov May 10, 2024

Choose a reason for hiding this comment

netroy May 10, 2024

Choose a reason for hiding this comment

cypress bot commented May 10, 2024

1 flaky test on run #4925 ↗︎

Review all test suite changes for PR #9355 ↗︎

github-actions bot commented May 10, 2024

perf: Benchmark `nodes-base` #9355

perf: Benchmark `nodes-base` #9355

codspeed-hq bot commented May 10, 2024 •

edited