Worker exits without much information

[stnguyen90]

Please answer these questions before submitting your issue.

1. What did you do? If possible, provide a simple script for reproducing the error.

git clone https://github.com/appwrite/appwrite.git
cd appwrite
git submodule update --init
# update _APP_OPTIONS_ABUSE in .env and set it to enabled
docker compose build
docker compose up -d
k6 run tests/benchmarks/ws.js
docker compose logs appwrite-realtime

3. What did you expect to see?

No errors

4. What did you see instead?

Errors like

appwrite-realtime  | Worker 59 started successfully
appwrite-realtime  | Pub/sub connection established (worker: 59)
appwrite-realtime  | [Error] Type: Exception
appwrite-realtime  | [Error] Type: Exception
appwrite-realtime  | [Error] Message: Pool 'console' is empty (size 10)
appwrite-realtime  | [Error] Message: Pool 'console' is empty (size 10)
appwrite-realtime  | [Error] File: /usr/src/code/vendor/utopia-php/pools/src/Pools/Pool.php
appwrite-realtime  | [Error] File: /usr/src/code/vendor/utopia-php/pools/src/Pools/Pool.php
appwrite-realtime  | [Error] Line: 174
appwrite-realtime  | [Error] Line: 174
appwrite-realtime  | Connection close: 26494
appwrite-realtime  | [Error] Connection Error
appwrite-realtime  | [Error] Connection Error
appwrite-realtime  | [Error] Type: Exception
appwrite-realtime  | [Error] Type: Exception
appwrite-realtime  | Connection close: 26517
appwrite-realtime  | [Error] Type: Exception
appwrite-realtime  | [Error] Code: 0
appwrite-realtime  | [Error] Code: 0
appwrite-realtime  | [Error] Message: Pool 'console' is empty (size 10)
appwrite-realtime  | [Error] Message: Pool 'console' is empty (size 10)
appwrite-realtime  | [Error] Message: Pool 'console' is empty (size 10)
appwrite-realtime  | [Error] Message: Pool 'console' is empty (size 10)
appwrite-realtime  | [Error] Message: Pool 'console' is empty (size 10)
appwrite-realtime  | [Error] File: /usr/src/code/vendor/utopia-php/pools/src/Pools/Pool.php
appwrite-realtime  | [Error] File: /usr/src/code/vendor/utopia-php/pools/src/Pools/Pool.php
appwrite-realtime  | [Error] File: /usr/src/code/vendor/utopia-php/pools/src/Pools/Pool.php
appwrite-realtime  | [Error] Line: 174
appwrite-realtime  | [Error] Line: 174
appwrite-realtime  | [Error] Line: 174
appwrite-realtime  | Connection close: 26520
appwrite-realtime  | [Error] Connection Error
appwrite-realtime  | Connection close: 26519
appwrite-realtime  | [Error] Connection Error
appwrite-realtime  | Connection close: 26521
appwrite-realtime  | [Error] Connection Error
appwrite-realtime  | [Error] Code: 0
appwrite-realtime  | [Error] Code: 0
appwrite-realtime  | [Error] Message: Pool 'console' is empty (size 10)
appwrite-realtime  | [Error] Code: 0
appwrite-realtime  | [Error] Message: Pool 'console' is empty (size 10)
appwrite-realtime  | [Error] Message: Pool 'console' is empty (size 10)
appwrite-realtime  | [2024-05-07 22:54:26 $116.0]       WARNING Server::check_worker_exit_status(): worker(pid=51535, id=33) abnormal exit, status=0, signal=11
appwrite-realtime  | A bug occurred in Swoole-v5.1.2, please report it.
appwrite-realtime  | The Swoole developers probably don't know about it,
appwrite-realtime  | and unless you report it, chances are it won't be fixed.
appwrite-realtime  | You can read How to report a bug doc before submitting any bug reports:
appwrite-realtime  | >> https://github.com/swoole/swoole-src/blob/master/.github/ISSUE.md 
appwrite-realtime  | Please do not send bug reports in the mailing list or personal letters.
appwrite-realtime  | The issue page is also suitable to submit feature requests.
appwrite-realtime  | 
appwrite-realtime  | OS: Linux 6.1.75-060175-generic #202402010024 SMP PREEMPT_DYNAMIC Thu Feb  1 01:44:23 UTC 2024 x86_64
appwrite-realtime  | GCC_VERSION: 13.2.1 20231014
appwrite-realtime  | OPENSSL_VERSION: OpenSSL 3.1.4 24 Oct 2023
appwrite-realtime  | PHP_VERSION : 8.3.3
appwrite-realtime  | 
appwrite-realtime  | Worker 33 started successfully
appwrite-realtime  | Pub/sub connection established (worker: 33)

5. What version of Swoole are you using (show your php --ri swoole)?

$ docker compose exec appwrite-realtime php --ri swoole

swoole

Swoole => enabled
Author => Swoole Team <team@swoole.com>
Version => 5.1.2
Built => Feb 23 2024 10:25:58
coroutine => enabled with boost asm context
epoll => enabled
eventfd => enabled
signalfd => enabled
spinlock => enabled
rwlock => enabled
sockets => enabled
openssl => OpenSSL 3.1.4 24 Oct 2023
dtls => enabled
http2 => enabled
json => enabled
curl-native => enabled
zlib => 1.3.1
brotli => E16781312/D16781312
mutex_timedlock => enabled
pthread_barrier => enabled
futex => enabled
async_redis => enabled

Directive => Local Value => Master Value
swoole.enable_coroutine => On => On
swoole.enable_library => On => On
swoole.enable_fiber_mock => Off => Off
swoole.enable_preemptive_scheduler => Off => Off
swoole.display_errors => On => On
swoole.use_shortname => On => On
swoole.unixsock_buffer_size => 8388608 => 8388608

6. What is your machine environment used (show your uname -a & php -v & gcc -v) ?

$ docker compose exec appwrite-realtime uname -a
Linux 24abc2ae63d0 6.1.75-060175-generic #202402010024 SMP PREEMPT_DYNAMIC Thu Feb  1 01:44:23 UTC 2024 x86_64 Linux

$ docker compose exec appwrite-realtime php -v
PHP 8.3.3 (cli) (built: Feb 16 2024 21:25:21) (NTS)
Copyright (c) The PHP Group
Zend Engine v4.3.3, Copyright (c) Zend Technologies

$ docker compose exec appwrite-realtime gcc -v
Using built-in specs.
COLLECT_GCC=gcc
COLLECT_LTO_WRAPPER=/usr/libexec/gcc/x86_64-alpine-linux-musl/13.2.1/lto-wrapper
Target: x86_64-alpine-linux-musl
Configured with: /home/buildozer/aports/main/gcc/src/gcc-13-20231014/configure --prefix=/usr --mandir=/usr/share/man --infodir=/usr/share/info --build=x86_64-alpine-linux-musl --host=x86_64-alpine-linux-musl --target=x86_64-alpine-linux-musl --enable-checking=release --disable-cet --disable-fixed-point --disable-libstdcxx-pch --disable-multilib --disable-nls --disable-werror --disable-symvers --enable-__cxa_atexit --enable-default-pie --enable-default-ssp --enable-languages=c,c++,d,objc,go,fortran,ada --enable-link-serialization=2 --enable-linker-build-id --disable-libssp --disable-libsanitizer --enable-shared --enable-threads --enable-tls --with-bugurl=https://gitlab.alpinelinux.org/alpine/aports/-/issues --with-system-zlib --with-linker-hash-style=gnu --with-pkgversion='Alpine 13.2.1_git20231014'
Thread model: posix
Supported LTO compression algorithms: zlib
gcc version 13.2.1 20231014 (Alpine 13.2.1_git20231014) 

Would you be able to help or suggest what we can do to troubleshoot why the swoole workers crash?

[NathanFreeman]

Please show me your php extension list.
php -m

[stnguyen90]

Here you go!

$ docker compose exec appwrite-realtime php -m
[PHP Modules]
brotli
Core
ctype
curl
date
dom
fileinfo
filter
hash
iconv
imagick
json
libxml
lz4
maxminddb
mbstring
mysqlnd
openssl
pcre
PDO
pdo_mysql
pdo_pgsql
pdo_sqlite
Phar
posix
random
readline
redis
Reflection
scrypt
session
SimpleXML
snappy
sockets
sodium
SPL
sqlite3
standard
swoole
tokenizer
xdebug
xml
xmlreader
xmlwriter
yaml
zlib
zstd

[Zend Modules]
Xdebug

[NathanFreeman]

Try to debug without using xdebug and see how it goes.