feat: Test connection using dedicated endpoint #17993

disq · 2024-05-16T16:47:23Z

Opening for comments.

A run with 2 plugin test connections (one success and one fail) and one plugin failure (not related to TestConnection)
(The postgresql plugin is an older public release with TestConnection not implemented, falling back on Init)

$ ./cli test-connection ./testsrc.yml ./gandi.yml 
Loading spec(s) from ./testsrc.yml, ./gandi.yml
Test Connection Results
source       gandi (cloudquery/gandi@v3.2.1)          Success
destination  pg@github (cloudquery/postgresql@v7.0.1) Failure: OTHER      rpc error: code = Internal desc = failed to init plugin: failed to initialize client: failed to get current database: failed to connect to `host=localhost user=postgres database=cq`: dial error (dial tcp 127.0.0.1:15432: connect: connection refused)
Error: at least one test connection failed
(exit code 1) $

With postgres implementing the test connector and a configuration with host=badhost

❯ ./cli test-connection test-connection.yaml
Loading spec(s) from test-connection.yaml
Test Connection Results
source       hackernews (cloudquery/hackernews@v3.1.8)  Success
destination  postgresql (grpc@localhost:7777)           Failure: INVALID_CREDENTIALS 	 no such host "badhost"
Error: at least one test connection failed

disq · 2024-05-17T06:57:49Z

cli/go.mod

+
+replace github.com/cloudquery/plugin-sdk/v4 => github.com/disq/plugin-sdk/v4 v4.0.0-20240516151118-4958828f53c9
+
+replace github.com/cloudquery/plugin-pb-go => github.com/cloudquery/plugin-pb-go v1.19.13-0.20240516105108-3fdf6ac15ea2


To be removed before merge when plugin-sdk and plugin-pb-go are released.

erezrokah · 2024-05-17T08:43:40Z

Can you share the specs you used to test? How can the test connection fail for the test plugin (failed to test source test)?
For the second spec not sure host= badhost is invalid credentials (credentials usually refer to user/pass), but that's a plugin issue

disq · 2024-05-17T09:04:09Z

# testsrc.yml
kind: source
spec:
  name: test
  version: v4.1.0
  path: cloudquery/test
  tables: ["*"]
  destinations: ["pg"]
  spec:
    num_clients: 1
    num_rows: 10
    num_sub_rows: 20000
    num_sub_cols: 200

# gandi.yml
kind: source
spec:
  name: gandi
  path: cloudquery/gandi
  registry: cloudquery
  version: "v3.2.1"
  tables:
    - "*"
  destinations:
    - pg
  spec:
    api_key: my-valid-key
    endpoint_url: https://api.sandbox.gandi.net/
---
kind: destination
spec:
  name: "pg"
  path: "cloudquery/postgresql"
  registry: "github"
  version: "v7.0.1"
  write_mode: "overwrite-delete-stale"
  spec:
    connection_string: "postgresql://postgres:pass@localhost:15432/cq"

# singlepg.yml
kind: destination
spec:
  name: "pg"
  path: "cloudquery/postgresql"
  registry: "github"
  version: "v7.0.1"
  write_mode: "overwrite-delete-stale"
  spec:
    connection_string: "postgresql://postgres:pass@localhost:15432/cq"

$ ./cli test-connection ./singlepg.yml 
Loading spec(s) from ./singlepg.yml
Test Connection Results
destination  pg@github (cloudquery/postgresql@v7.0.1)  Success
(exit code 0) $

disq · 2024-05-17T09:09:01Z

@erezrokah The "failed to test source test" issue was fixed in 4477eb2 and the next cleanup commit. I just updated the PR description to reflect. It can still generate failed to test source|destination %v: %w type errors if for example it cannot communicate via gRPC with the plugin etc.

cli/cmd/test_connection_test.go

cli/cmd/test_connection.go

erezrokah

A few more comments

cli/cmd/test_connection.go

erezrokah · 2024-05-17T14:13:47Z

cli/cmd/test_connection.go

+}
+
+func (*testConnectionFailures) Error() string {
+	// The errs are already shown in the console, so we don't need to show them again here


Are the errors shown when you do --log-console too?

Yes, now better, with proper plugin_ref etc.

2024-05-17T14:33:56Z INF Loading spec(s) args=["./pgrpc.yml"] invocation-id=dafde1e3-5e32-4fad-8fb5-c7ad7a885284 module=cli 2024-05-17T14:33:56Z INF Testing destination destination="pg (grpc@localhost:7778)" invocation-id=dafde1e3-5e32-4fad-8fb5-c7ad7a885284 module=cli 2024-05-17T14:33:56Z INF Test connection completed invocation-id=dafde1e3-5e32-4fad-8fb5-c7ad7a885284 module=cli testresults=[{"failure_code":"OTHER","failure_description":"rpc error: code = Internal desc = failed to init plugin: failed to initialize client: failed to get current database: failed to connect to `host=localhost user=postgres database=cq`: dial error (dial tcp 127.0.0.1:15432: connect: connection refused)","plugin_kind":"destination","plugin_ref":"pg (grpc@localhost:7778)","success":false}] Error: at least one test connection failed 2024-05-17T14:33:56Z ERR exiting with error error="at least one test connection failed" invocation-id=dafde1e3-5e32-4fad-8fb5-c7ad7a885284 module=cli

erezrokah · 2024-05-17T14:15:07Z

cli/cmd/test_connection.go

+//
+// The function returns any failed test results, or nil if all tests passed. The hackernews plugin is excluded from the
+// failed test results since it was previously used as a test case for the test connection command.
+func filterFailedTestResults(results []testConnectionResult) (*testConnectionResult, error) {


Wait what's the reason we need to filter these plugins? I don't think their connection test can ever fail

Those should be there until we update the cloud side to use the new cli.

Ah because with existing Cloud the new CLI will report failures for them?

I don't think it matters much this feature is only enabled for the cloudquery team

If we want the cloud api side to work, this filtering needs to be there at least for now.

🤖 I have created a release *beep* *boop* --- ## [5.20.0](cli-v5.19.0...cli-v5.20.0) (2024-05-20) ### Features * Test connection using dedicated endpoint ([#17993](#17993)) ([0baae95](0baae95)) ### Bug Fixes * **deps:** Update module github.com/cloudquery/cloudquery-api-go to v1.11.2 ([#17995](#17995)) ([9bc2353](9bc2353)) * **deps:** Update module github.com/cloudquery/plugin-pb-go to v1.19.13 ([#18001](#18001)) ([042f3b8](042f3b8)) * **deps:** Update module github.com/cloudquery/plugin-sdk/v4 to v4.42.2 ([#18000](#18000)) ([5fc0f46](5fc0f46)) * **deps:** Update module github.com/cloudquery/plugin-sdk/v4 to v4.43.0 ([#18014](#18014)) ([20592c8](20592c8)) --- This PR was generated with [Release Please](https://github.com/googleapis/release-please). See [documentation](https://github.com/googleapis/release-please#release-please).

mnorbury and others added 5 commits May 16, 2024 15:03

WIP - testing the gRPC TestConnection endpoint

553cb81

Allow single plugin for test connection

0879755

tiny cleanup

e667078

Fallback to Init

4f0d353

Add some console UI

fd763b6

disq requested a review from a team as a code owner May 16, 2024 16:47

disq requested review from savme and removed request for a team May 16, 2024 16:47

cq-bot added area/cli area/website labels May 16, 2024

disq requested a review from erezrokah May 16, 2024 16:48

disq added 6 commits May 16, 2024 17:55

clean up initErrors

c3c9251

remove unused param

2da1694

Print plugin kind

ef87b58

left align for more readable results

f3cfcc3

Fix test, expose errors without exposing them in the console

4477eb2

clean up

1427a6a

disq commented May 17, 2024

View reviewed changes

erezrokah reviewed May 17, 2024

View reviewed changes

cli/cmd/test_connection_test.go Outdated Show resolved Hide resolved

disq and others added 3 commits May 17, 2024 10:42

Simplify test

7991515

Merge branch 'main' into feat/adding-test-connection-cli

ead8f44

Send the failed test results to the backend API

bafa042

disq commented May 17, 2024

View reviewed changes

cli/cmd/test_connection.go Outdated Show resolved Hide resolved

disq commented May 17, 2024

View reviewed changes

cli/cmd/test_connection.go Outdated Show resolved Hide resolved

Exclude specfic versions of hackernews and file

8f13c0a

disq commented May 17, 2024

View reviewed changes

cli/cmd/test_connection.go Outdated Show resolved Hide resolved

Update cli/cmd/test_connection.go

a4946aa

disq and others added 4 commits May 17, 2024 14:48

Don't import sdk, update go.mod

3b35762

Use OTHER as reason if we're going through the Init flow

7292e9a

Merge branch 'main' into feat/adding-test-connection-cli

6df2aa7

Use context.Background() on status update

a51b83e

erezrokah reviewed May 17, 2024

View reviewed changes

CR

04c3793

erezrokah approved these changes May 17, 2024

View reviewed changes

disq and others added 2 commits May 20, 2024 09:47

Use tagged SDK

2712223

Merge branch 'main' into feat/adding-test-connection-cli

403164f

disq added the automerge Automatically merge once required checks pass label May 20, 2024

Merge branch 'main' into feat/adding-test-connection-cli

75632dd

kodiakhq bot merged commit 0baae95 into main May 20, 2024
20 checks passed

kodiakhq bot deleted the feat/adding-test-connection-cli branch May 20, 2024 09:41

cq-bot mentioned this pull request May 20, 2024

chore(main): Release cli v5.20.0 #17998

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: Test connection using dedicated endpoint #17993

feat: Test connection using dedicated endpoint #17993

disq commented May 16, 2024 •

edited

disq May 17, 2024

erezrokah commented May 17, 2024 •

edited

disq commented May 17, 2024 •

edited

disq commented May 17, 2024

erezrokah left a comment

erezrokah May 17, 2024

disq May 17, 2024

erezrokah May 17, 2024

disq May 17, 2024

erezrokah May 17, 2024 •

edited

erezrokah May 17, 2024

disq May 17, 2024


		replace github.com/cloudquery/plugin-sdk/v4 => github.com/disq/plugin-sdk/v4 v4.0.0-20240516151118-4958828f53c9

		replace github.com/cloudquery/plugin-pb-go => github.com/cloudquery/plugin-pb-go v1.19.13-0.20240516105108-3fdf6ac15ea2

feat: Test connection using dedicated endpoint #17993

feat: Test connection using dedicated endpoint #17993

Conversation

disq commented May 16, 2024 • edited

disq May 17, 2024

Choose a reason for hiding this comment

erezrokah commented May 17, 2024 • edited

disq commented May 17, 2024 • edited

disq commented May 17, 2024

erezrokah left a comment

Choose a reason for hiding this comment

erezrokah May 17, 2024

Choose a reason for hiding this comment

disq May 17, 2024

Choose a reason for hiding this comment

erezrokah May 17, 2024

Choose a reason for hiding this comment

disq May 17, 2024

Choose a reason for hiding this comment

erezrokah May 17, 2024 • edited

Choose a reason for hiding this comment

erezrokah May 17, 2024

Choose a reason for hiding this comment

disq May 17, 2024

Choose a reason for hiding this comment

disq commented May 16, 2024 •

edited

erezrokah commented May 17, 2024 •

edited

disq commented May 17, 2024 •

edited

erezrokah May 17, 2024 •

edited