-
Notifications
You must be signed in to change notification settings - Fork 2.2k
Replies: 1 comment · 9 replies
-
Hello, It looks like the terminate signal took place at 2024-03-13 01:26:45.077, but the operator logs end at 2024-03-13 01:07:07.453. Do you have the logs covering the same time period? |
Beta Was this translation helpful? Give feedback.
All reactions
-
It looks like there is a 20-second timeout waiting for the dashboard: |
Beta Was this translation helpful? Give feedback.
All reactions
-
@arcarmona Cloud you please show the |
Beta Was this translation helpful? Give feedback.
All reactions
-
Hello @Rory-Z, thanks for the reply. I am a bit confused as in It seems to me that with the current information I have available this is pretty difficult to debug, I will try setting
emqx-core-7bf8f698c-0
emqx-core-7bf8f698c-1
|
Beta Was this translation helpful? Give feedback.
All reactions
-
So strange, if EMQX has been restart, the lastState of containerStatuses should not be empty.
In theory, upgrading EMQX operator will not interrupt EMQX services, because the latest EMQX operator is version 2.2.0 patch, but I still recommend running test, because 2.2.0 is too old, and its test case just checked EMQX 5.0 and EMQX 5.1 |
Beta Was this translation helpful? Give feedback.
All reactions
-
Hello @Rory-Z, reopening this discussion as I encountered a similar issue again after close to one month of no issues, this time with some more logs before the shutdown signal, however I believe it does not provide much insight neither but want to share just in case I am missing something. Besides that, there is little more to share, execept for similar volatile levels of cpu afterthe shutdown (see images of the original post) and the permanent stoppage of os_mon and jq. 1st crash emqx-core-0 logs
1st crash emqx-core-1 logs
2nd crash emqx-core-1 logs
emqx-operator-controller-manager logs
|
Beta Was this translation helpful? Give feedback.
-
Hello, I am currently running in GKE a two node emqx community edition (5.5.0) deployment using emqx operator (2.2.0). Resource wise I am assigning 3.5cpu and 3.5Gi RAM per node. So far this deployment has been working wonders for me, the only issue is I unexpectedly see nodes restart with the following log
Received terminate signal, shutting down now
. Most recent case happened today at 00:58 for the first node and subsequently at 01:26 for the second node. As of right now the cluster is back up working and I believe there was not significant downtime, however I would like to find out what could have caused this.I am having trouble debugging the issue and identifying the root cause of the shutdown signal since I believe its not a resource problem (see attached graph below). I will attach the logs of emqx-core and emqx-operator as well. I can also share the .yaml used for the deployment if required.
emqx-core logs
emqx-operator-controller-manager logs
emqx-core-0
emqx-core-1
Any help, insights or ideas for further investigating this issue is greatly appreciated, thanks.
Beta Was this translation helpful? Give feedback.
All reactions