[Bug] The registry uses etcd, but javaagent skywalking fails to start #14198

glynnwei · 2024-05-16T03:16:25Z

Pre-check

I am sure that all the content I provide is in English.

Search before asking

I had searched in the issues and found no similar issues.

Apache Dubbo Component

Java SPI Extensions (apache/dubbo-spi-extensions)

Dubbo Version

dubbo: 3.2.11
jdk: 1.8
skywalking agent: 9.2.0

Steps to reproduce this issue

1、maven import

...
<maven.compiler.source>1.8</maven.compiler.source>
<maven.compiler.target>1.8</maven.compiler.target>
<dubbo.version>3.2.11</dubbo.version>
<spring-boot.version>2.6.4</spring-boot.version>
...
<dependency>
    <groupId>org.apache.dubbo.extensions</groupId>
    <artifactId>dubbo-registry-etcd3</artifactId>
    <version>3.2.0</version>
</dependency>
<dependency>
    <groupId>org.apache.dubbo.extensions</groupId>
    <artifactId>dubbo-metadata-report-etcd</artifactId>
    <version>3.2.0</version>
</dependency>
<dependency>
    <groupId>org.apache.dubbo</groupId>
    <artifactId>dubbo-spring-boot-observability-starter</artifactId>
    <version>3.2.11</version>
</dependency>
<dependency>
    <groupId>org.apache.dubbo</groupId>
    <artifactId>dubbo-metrics-api</artifactId>
    <version>3.2.11</version>
</dependency>
...

2、start javaagent

-javaagent:/skywalking-agent-9.2.0/skywalking-agent.jar
-Dskywalking.agent.service_name=test-project
-Dskywalking.collector.backend_service=127.0.0.1:11800

3、error message

java.lang.IllegalStateException: Failed to unregister grpc://10.63.9.0:8082/proto.helloworld.Greeter?application=test-project&deprecated=false&dubbo=2.0.2&prefer.serialization=fastjson2,hessian2&release=3.2.11&side=provider to registry 127.0.0.1:2379, cause: Failed to unregister grpc://10.63.9.0:8082/proto.helloworld.Greeter?application=test-project&deprecated=false&dubbo=2.0.2&prefer.serialization=fastjson2,hessian2&release=3.2.11&side=provider to etcd etcd3://127.0.0.1:2379/org.apache.dubbo.registry.RegistryService?application=test-project&dubbo=2.0.2&executor-management-mode=isolation&file-cache=true&interface=org.apache.dubbo.registry.RegistryService&logger=slf4j&pid=90951&proxy=nativestub&register=false&release=3.2.11&simplified=true, cause: java.lang.IllegalStateException: Exit span doesn't include meaningful peer information.
	at org.apache.dubbo.registry.support.FailbackRegistry.unregister(FailbackRegistry.java:299)
	at org.apache.dubbo.registry.support.AbstractRegistry.destroy(AbstractRegistry.java:626)
	at org.apache.dubbo.registry.support.FailbackRegistry.destroy(FailbackRegistry.java:473)
	at org.apache.dubbo.registry.etcd.EtcdRegistry.destroy(EtcdRegistry.java:275)
	at org.apache.dubbo.registry.support.RegistryManager.destroyAll(RegistryManager.java:107)
	at org.apache.dubbo.config.deploy.DefaultApplicationDeployer.destroyRegistries(DefaultApplicationDeployer.java:1372)
	at org.apache.dubbo.config.deploy.DefaultApplicationDeployer.postDestroy(DefaultApplicationDeployer.java:1106)
	at org.apache.dubbo.rpc.model.ApplicationModel.onDestroy(ApplicationModel.java:172)
	at org.apache.dubbo.rpc.model.ScopeModel.destroy(ScopeModel.java:122)
	at org.apache.dubbo.rpc.model.ApplicationModel.tryDestroy(ApplicationModel.java:276)
	at org.apache.dubbo.rpc.model.ModuleModel.onDestroy(ModuleModel.java:133)
	at org.apache.dubbo.rpc.model.ScopeModel.destroy(ScopeModel.java:122)
	at org.apache.dubbo.config.spring.context.DubboDeployApplicationListener.onContextClosedEvent(DubboDeployApplicationListener.java:189)
	at org.apache.dubbo.config.spring.context.DubboDeployApplicationListener.onApplicationEvent(DubboDeployApplicationListener.java:145)
	at org.apache.dubbo.config.spring.context.DubboDeployApplicationListener.onApplicationEvent(DubboDeployApplicationListener.java:52)
	at org.springframework.context.event.SimpleApplicationEventMulticaster.doInvokeListener(SimpleApplicationEventMulticaster.java:176)
	at org.springframework.context.event.SimpleApplicationEventMulticaster.invokeListener(SimpleApplicationEventMulticaster.java:169)
	at org.springframework.context.event.SimpleApplicationEventMulticaster.multicastEvent(SimpleApplicationEventMulticaster.java:143)
	at org.springframework.context.support.AbstractApplicationContext.publishEvent(AbstractApplicationContext.java:421)
	at org.springframework.context.support.AbstractApplicationContext.publishEvent(AbstractApplicationContext.java:378)
	at org.springframework.context.support.AbstractApplicationContext.doClose(AbstractApplicationContext.java:1058)
	at org.springframework.boot.web.servlet.context.ServletWebServerApplicationContext.doClose(ServletWebServerApplicationContext.java:172)
	at org.springframework.context.support.AbstractApplicationContext.close(AbstractApplicationContext.java:1021)
	at org.springframework.boot.SpringApplication.handleRunFailure(SpringApplication.java:796)
	at org.springframework.boot.SpringApplication.run(SpringApplication.java:313)
	at org.springframework.boot.SpringApplication.run(SpringApplication.java:1312)
	at org.springframework.boot.SpringApplication.run(SpringApplication.java:1301)
	at com.test.server.TestApplication.main(TestApplication.java:25)
Caused by: org.apache.dubbo.rpc.RpcException: Failed to unregister grpc://10.63.9.0:8082/proto.helloworld.Greeter?application=test-project&deprecated=false&dubbo=2.0.2&prefer.serialization=fastjson2,hessian2&release=3.2.11&side=provider to etcd etcd3://127.0.0.1:2379/org.apache.dubbo.registry.RegistryService?application=test-project&dubbo=2.0.2&executor-management-mode=isolation&file-cache=true&interface=org.apache.dubbo.registry.RegistryService&logger=slf4j&pid=90951&proxy=nativestub&register=false&release=3.2.11&simplified=true, cause: java.lang.IllegalStateException: Exit span doesn't include meaningful peer information.
	at org.apache.dubbo.registry.etcd.EtcdRegistry.doUnregister(EtcdRegistry.java:130)
	at org.apache.dubbo.registry.support.FailbackRegistry.unregister(FailbackRegistry.java:284)
	... 27 common frames omitted
Caused by: java.lang.IllegalStateException: java.lang.IllegalStateException: Exit span doesn't include meaningful peer information.
	at org.apache.dubbo.remoting.etcd.jetcd.JEtcdClientWrapper.delete(JEtcdClientWrapper.java:481)
	at org.apache.dubbo.remoting.etcd.jetcd.JEtcdClient.delete(JEtcdClient.java:167)
	at org.apache.dubbo.registry.etcd.EtcdRegistry.doUnregister(EtcdRegistry.java:128)
	... 28 common frames omitted
Caused by: java.util.concurrent.ExecutionException: java.lang.IllegalStateException: Exit span doesn't include meaningful peer information.
	at java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:357)
	at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1928)
	at org.apache.dubbo.remoting.etcd.jetcd.JEtcdClientWrapper.lambda$delete$15(JEtcdClientWrapper.java:476)
	at org.apache.dubbo.remoting.etcd.jetcd.RetryLoops.invokeWithRetry(RetryLoops.java:57)
	at org.apache.dubbo.remoting.etcd.jetcd.JEtcdClientWrapper.delete(JEtcdClientWrapper.java:471)
	... 30 common frames omitted
Caused by: java.lang.IllegalStateException: Exit span doesn't include meaningful peer information.
	at org.apache.skywalking.apm.agent.core.context.TracingContext.inject(TracingContext.java:169)
	at org.apache.skywalking.apm.agent.core.context.TracingContext.inject(TracingContext.java:149)
	at org.apache.skywalking.apm.agent.core.context.ContextManager.inject(ContextManager.java:145)
	at org.apache.skywalking.apm.plugin.grpc.v1.client.TracingClientCall.start(TracingClientCall.java:84)
	at io.grpc.stub.ClientCalls.startCall(ClientCalls.java:341)
	at io.grpc.stub.ClientCalls.asyncUnaryRequestCall(ClientCalls.java:315)
	at io.grpc.stub.ClientCalls.$sw$original$futureUnaryCall$q891cd3(ClientCalls.java:227)
	at io.grpc.stub.ClientCalls.$sw$original$futureUnaryCall$q891cd3$accessor$$sw$i3l8ir2(ClientCalls.java)
	at io.grpc.stub.ClientCalls$$sw$auxiliary$286ppc0.call(Unknown Source)
	at org.apache.skywalking.apm.agent.core.plugin.interceptor.enhance.StaticMethodsInter.intercept(StaticMethodsInter.java:83)
	at io.grpc.stub.ClientCalls.futureUnaryCall(ClientCalls.java)
	at io.etcd.jetcd.api.KVGrpc$KVFutureStub.deleteRange(KVGrpc.java:520)
	at io.etcd.jetcd.KVImpl.lambda$delete$6(KVImpl.java:138)
	at io.etcd.jetcd.ClientConnectionManager.lambda$execute$4(ClientConnectionManager.java:349)
	at net.jodah.failsafe.Functions$1.get(Functions.java:134)
	at net.jodah.failsafe.Functions$1.get(Functions.java:129)
	at net.jodah.failsafe.Functions$2.get(Functions.java:226)
	at net.jodah.failsafe.Functions$2.get(Functions.java:216)
	at net.jodah.failsafe.internal.util.DelegatingScheduler.lambda$schedule$0(DelegatingScheduler.java:141)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:750)

What you expected to happen

Other registries (nacos, zookeeper) can be started successfully, and skywalking data can be collected

Anything else

The /skywalk-agent-9.2.0/plugins directory is loaded properly

Are you willing to submit a pull request to fix on your own?

Yes I am willing to submit a pull request on my own!

Code of Conduct

I agree to follow this project's Code of Conduct

The text was updated successfully, but these errors were encountered:

chickenlj · 2024-05-16T03:58:11Z

...
<maven.compiler.source>1.8</maven.compiler.source>
<maven.compiler.target>1.8</maven.compiler.target>
<dubbo.version>3.2.11</dubbo.version>
<spring-boot.version>2.6.4</spring-boot.version>
...

org.apache.dubbo.extensions
dubbo-registry-etcd3
3.2.0

org.apache.dubbo.extensions
dubbo-metadata-report-etcd
3.2.0

org.apache.dubbo
dubbo-spring-boot-observability-starter
3.2.11

org.apache.dubbo
dubbo-metrics-api
3.2.11

...

Please try to add dubbo-spring-boot-starter dependency and necessary registry starter dependency. dubbo-metrics-api is not necessary once dubbo-spring-boot-starter is added.

chickenlj · 2024-05-16T06:18:12Z

Why the url in registry is grpc://10.63.9.0:8082/proto.helloworld.Greeter? Do you set dubbo.protocol.name=tri?

glynnwei · 2024-05-16T06:26:14Z

...
<maven.compiler.source>1.8</maven.compiler.source>
<maven.compiler.target>1.8</maven.compiler.target>
<dubbo.version>3.2.11</dubbo.version>
< spring-boot.version>2.6.4</spring-boot.version>
...

org.apache.dubbo.extensions
dubbo-registry-etcd3
3.2.0

org.apache.dubbo.extensions
dubbo-metadata-report-etcd
3.2 .0

org.apache.dubbo
dubbo-spring-boot-observability-starter
3.2.11

org.apache.dubbo
dubbo-metrics-api
3.2.11

...

请尝试添加dubbo-spring-boot-starter依赖项和必要的注册表启动器依赖项。一旦添加dubbo-metrics-api就不需要了。dubbo-spring-boot-starter

<dependency>
            <groupId>org.apache.dubbo</groupId>
            <artifactId>dubbo-spring-boot-starter</artifactId>
            <version>3.2.11</version>
        </dependency>
        <dependency>
            <groupId>org.apache.dubbo</groupId>
            <artifactId>dubbo-spring-boot-observability-starter</artifactId>
            <version>3.2.11</version>
        </dependency>
        <dependency>
            <groupId>org.apache.dubbo.extensions</groupId>
            <artifactId>dubbo-registry-etcd3</artifactId>
            <version>3.2.0</version>
        </dependency>
        <dependency>
            <groupId>org.apache.dubbo.extensions</groupId>
            <artifactId>dubbo-metadata-report-etcd</artifactId>
            <version>3.2.0</version>
        </dependency>

Thanks. It's still the same problem after modification

glynnwei · 2024-05-16T06:27:36Z

Why the url in registry is grpc://10.63.9.0:8082/proto.helloworld.Greeter? Do you set dubbo.protocol.name=tri?
application.yml

  protocol:
    name: grpc
    port: 8082
  registry:
    address: etcd3://127.0.0.1:2379

chickenlj · 2024-05-16T06:31:38Z

More likely to be an issue of skywalking integrating with etcd.

glynnwei · 2024-05-16T07:29:33Z

More likely to be an issue of skywalking integrating with etcd.

Yes, I think so too.

I hope someone can follow up and fix it.

glynnwei added component/need-triage Need maintainers to triage type/need-triage Need maintainers to triage labels May 16, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Bug] The registry uses etcd, but javaagent skywalking fails to start #14198

[Bug] The registry uses etcd, but javaagent skywalking fails to start #14198

glynnwei commented May 16, 2024

chickenlj commented May 16, 2024

chickenlj commented May 16, 2024 •

edited

glynnwei commented May 16, 2024

glynnwei commented May 16, 2024

chickenlj commented May 16, 2024

glynnwei commented May 16, 2024 •

edited

[Bug] The registry uses etcd, but javaagent skywalking fails to start #14198

[Bug] The registry uses etcd, but javaagent skywalking fails to start #14198

Comments

glynnwei commented May 16, 2024

Pre-check

Search before asking

Apache Dubbo Component

Dubbo Version

Steps to reproduce this issue

What you expected to happen

Anything else

Are you willing to submit a pull request to fix on your own?

Code of Conduct

chickenlj commented May 16, 2024

chickenlj commented May 16, 2024 • edited

glynnwei commented May 16, 2024

glynnwei commented May 16, 2024

chickenlj commented May 16, 2024

glynnwei commented May 16, 2024 • edited

chickenlj commented May 16, 2024 •

edited

glynnwei commented May 16, 2024 •

edited