containerd crash - containerd[501]: [signal SIGSEGV: segmentation violation code=0x1 addr=0x129a0 pc=0x563a4cdd0d6f]

[sg893052]

Description

Jan 15 05:40:45.093251+00:00 2024 ax1-spine103 INFO containerd[501]: fatal error: unexpected signal during runtime execution
Jan 15 05:40:45.095785+00:00 2024 ax1-spine103 INFO containerd[501]: [signal SIGSEGV: segmentation violation code=0x1 addr=0x129a0 pc=0x563a4cdd0d6f]
Jan 15 05:40:45.095807+00:00 2024 ax1-spine103 INFO containerd[501]: runtime stack:
Jan 15 05:40:45.095900+00:00 2024 ax1-spine103 INFO containerd[501]: runtime.throw({0x563a4e0938e8, 0x563a4cde1213})
Jan 15 05:40:45.095918+00:00 2024 ax1-spine103 INFO containerd[501]: #011/usr/local/go/src/runtime/panic.go:1198 +0x71
Jan 15 05:40:45.095934+00:00 2024 ax1-spine103 INFO containerd[501]: runtime.sigpanic()
Jan 15 05:40:45.095952+00:00 2024 ax1-spine103 INFO containerd[501]: #011/usr/local/go/src/runtime/signal_unix.go:719 +0x389
Jan 15 05:40:45.095969+00:00 2024 ax1-spine103 INFO containerd[501]: runtime.findObject(0xc0000a9540, 0x6f99b3a6b07f42, 0x0)
Jan 15 05:40:45.095986+00:00 2024 ax1-spine103 INFO containerd[501]: #011/usr/local/go/src/runtime/mbitmap.go:401 +0x6f
Jan 15 05:40:45.096004+00:00 2024 ax1-spine103 INFO containerd[501]: runtime.scanblock(0x563a4f45b5c0, 0x40000, 0x7fa7b6210a49, 0x7fa785ffad60, 0x0)
Jan 15 05:40:45.096021+00:00 2024 ax1-spine103 INFO containerd[501]: #011/usr/local/go/src/runtime/mgcmark.go:1185 +0xb8
Jan 15 05:40:45.096177+00:00 2024 ax1-spine103 INFO containerd[501]: runtime.markrootBlock(0xc000717000, 0x7fa785ffad80, 0x0, 0xc000c5a4e0, 0x7fa785ffad90)
Jan 15 05:40:45.096195+00:00 2024 ax1-spine103 INFO containerd[501]: #011/usr/local/go/src/runtime/mgcmark.go:268 +0x5d
Jan 15 05:40:45.096212+00:00 2024 ax1-spine103 INFO containerd[501]: runtime.markroot(0xc00004de98, 0x3)
Jan 15 05:40:45.096229+00:00 2024 ax1-spine103 INFO containerd[501]: #011/usr/local/go/src/runtime/mgcmark.go:166 +0x246
Jan 15 05:40:45.096246+00:00 2024 ax1-spine103 INFO containerd[501]: runtime.gcDrain(0xc00004de98, 0x3)
Jan 15 05:40:45.096263+00:00 2024 ax1-spine103 INFO containerd[501]: #011/usr/local/go/src/runtime/mgcmark.go:1013 +0x379
Jan 15 05:40:45.096282+00:00 2024 ax1-spine103 INFO containerd[501]: runtime.gcBgMarkWorker.func2()
Jan 15 05:40:45.096299+00:00 2024 ax1-spine103 INFO containerd[501]: #011/usr/local/go/src/runtime/mgc.go:1269 +0xa5
Jan 15 05:40:45.096315+00:00 2024 ax1-spine103 INFO containerd[501]: runtime.systemstack()
Jan 15 05:40:45.096332+00:00 2024 ax1-spine103 INFO containerd[501]: #011/usr/local/go/src/runtime/asm_amd64.s:383 +0x46
Jan 15 05:40:45.096349+00:00 2024 ax1-spine103 INFO containerd[501]: goroutine 6 [GC worker (idle)]:
Jan 15 05:40:45.096366+00:00 2024 ax1-spine103 INFO containerd[501]: runtime.systemstack_switch()
Jan 15 05:40:45.096383+00:00 2024 ax1-spine103 INFO containerd[501]: #011/usr/local/go/src/runtime/asm_amd64.s:350 fp=0xc00005a760 sp=0xc00005a758 pc=0x563a4ce265c0
Jan 15 05:40:45.096399+00:00 2024 ax1-spine103 INFO containerd[501]: runtime.gcBgMarkWorker()
Jan 15 05:40:45.096417+00:00 2024 ax1-spine103 INFO containerd[501]: #011/usr/local/go/src/runtime/mgc.go:1256 +0x1b3 fp=0xc00005a7e0 sp=0xc00005a760 pc=0x563a4cdd7f53
Jan 15 05:40:45.096434+00:00 2024 ax1-spine103 INFO containerd[501]: runtime.goexit()
Jan 15 05:40:45.096451+00:00 2024 ax1-spine103 INFO containerd[501]: #011/usr/local/go/src/runtime/asm_amd64.s:1581 +0x1 fp=0xc00005a7e8 sp=0xc00005a7e0 pc=0x563a4ce28861
Jan 15 05:40:45.096468+00:00 2024 ax1-spine103 INFO containerd[501]: created by runtime.gcBgMarkStartWorkers
Jan 15 05:40:45.096485+00:00 2024 ax1-spine103 INFO containerd[501]: #011/usr/local/go/src/runtime/mgc.go:1124 +0x25
Jan 15 05:40:45.096502+00:00 2024 ax1-spine103 INFO containerd[501]: goroutine 1 [chan receive, 523545 minutes]:

Steps to reproduce the issue

1. Normal Operations

Describe the results you received and expected

Jan 15 05:40:45.093251+00:00 2024 ax1-spine103 INFO containerd[501]: fatal error: unexpected signal during runtime execution
Jan 15 05:40:45.095785+00:00 2024 ax1-spine103 INFO containerd[501]: [signal SIGSEGV: segmentation violation code=0x1 addr=0x129a0 pc=0x563a4cdd0d6f]
Jan 15 05:40:45.095807+00:00 2024 ax1-spine103 INFO containerd[501]: runtime stack:
Jan 15 05:40:45.095900+00:00 2024 ax1-spine103 INFO containerd[501]: runtime.throw({0x563a4e0938e8, 0x563a4cde1213})
Jan 15 05:40:45.095918+00:00 2024 ax1-spine103 INFO containerd[501]: #011/usr/local/go/src/runtime/panic.go:1198 +0x71
Jan 15 05:40:45.095934+00:00 2024 ax1-spine103 INFO containerd[501]: runtime.sigpanic()
Jan 15 05:40:45.095952+00:00 2024 ax1-spine103 INFO containerd[501]: #011/usr/local/go/src/runtime/signal_unix.go:719 +0x389
Jan 15 05:40:45.095969+00:00 2024 ax1-spine103 INFO containerd[501]: runtime.findObject(0xc0000a9540, 0x6f99b3a6b07f42, 0x0)
Jan 15 05:40:45.095986+00:00 2024 ax1-spine103 INFO containerd[501]: #011/usr/local/go/src/runtime/mbitmap.go:401 +0x6f
Jan 15 05:40:45.096004+00:00 2024 ax1-spine103 INFO containerd[501]: runtime.scanblock(0x563a4f45b5c0, 0x40000, 0x7fa7b6210a49, 0x7fa785ffad60, 0x0)
Jan 15 05:40:45.096021+00:00 2024 ax1-spine103 INFO containerd[501]: #011/usr/local/go/src/runtime/mgcmark.go:1185 +0xb8
Jan 15 05:40:45.096177+00:00 2024 ax1-spine103 INFO containerd[501]: runtime.markrootBlock(0xc000717000, 0x7fa785ffad80, 0x0, 0xc000c5a4e0, 0x7fa785ffad90)
Jan 15 05:40:45.096195+00:00 2024 ax1-spine103 INFO containerd[501]: #011/usr/local/go/src/runtime/mgcmark.go:268 +0x5d
Jan 15 05:40:45.096212+00:00 2024 ax1-spine103 INFO containerd[501]: runtime.markroot(0xc00004de98, 0x3)
Jan 15 05:40:45.096229+00:00 2024 ax1-spine103 INFO containerd[501]: #011/usr/local/go/src/runtime/mgcmark.go:166 +0x246
Jan 15 05:40:45.096246+00:00 2024 ax1-spine103 INFO containerd[501]: runtime.gcDrain(0xc00004de98, 0x3)
Jan 15 05:40:45.096263+00:00 2024 ax1-spine103 INFO containerd[501]: #011/usr/local/go/src/runtime/mgcmark.go:1013 +0x379
Jan 15 05:40:45.096282+00:00 2024 ax1-spine103 INFO containerd[501]: runtime.gcBgMarkWorker.func2()
Jan 15 05:40:45.096299+00:00 2024 ax1-spine103 INFO containerd[501]: #011/usr/local/go/src/runtime/mgc.go:1269 +0xa5
Jan 15 05:40:45.096315+00:00 2024 ax1-spine103 INFO containerd[501]: runtime.systemstack()
Jan 15 05:40:45.096332+00:00 2024 ax1-spine103 INFO containerd[501]: #011/usr/local/go/src/runtime/asm_amd64.s:383 +0x46
Jan 15 05:40:45.096349+00:00 2024 ax1-spine103 INFO containerd[501]: goroutine 6 [GC worker (idle)]:
Jan 15 05:40:45.096366+00:00 2024 ax1-spine103 INFO containerd[501]: runtime.systemstack_switch()
Jan 15 05:40:45.096383+00:00 2024 ax1-spine103 INFO containerd[501]: #011/usr/local/go/src/runtime/asm_amd64.s:350 fp=0xc00005a760 sp=0xc00005a758 pc=0x563a4ce265c0
Jan 15 05:40:45.096399+00:00 2024 ax1-spine103 INFO containerd[501]: runtime.gcBgMarkWorker()
Jan 15 05:40:45.096417+00:00 2024 ax1-spine103 INFO containerd[501]: #011/usr/local/go/src/runtime/mgc.go:1256 +0x1b3 fp=0xc00005a7e0 sp=0xc00005a760 pc=0x563a4cdd7f53
Jan 15 05:40:45.096434+00:00 2024 ax1-spine103 INFO containerd[501]: runtime.goexit()
Jan 15 05:40:45.096451+00:00 2024 ax1-spine103 INFO containerd[501]: #011/usr/local/go/src/runtime/asm_amd64.s:1581 +0x1 fp=0xc00005a7e8 sp=0xc00005a7e0 pc=0x563a4ce28861
Jan 15 05:40:45.096468+00:00 2024 ax1-spine103 INFO containerd[501]: created by runtime.gcBgMarkStartWorkers
Jan 15 05:40:45.096485+00:00 2024 ax1-spine103 INFO containerd[501]: #011/usr/local/go/src/runtime/mgc.go:1124 +0x25
Jan 15 05:40:45.096502+00:00 2024 ax1-spine103 INFO containerd[501]: goroutine 1 [chan receive, 523545 minutes]:

What version of containerd are you using?

containerdcontainerd.io 1.6.8 9cd3357

Any other relevant information

uname -a
Linux sonic 4.19.0-9-2-amd64 #1 SMP Debian 4.19.118-2+deb10u1 (2020-06-07) x86_64 GNU/Linux

runc --version
runc version 1.1.4
commit: v1.1.4-0-g5fd4c4d
spec: 1.0.2-dev
go: go1.17.13
libseccomp: 2.3.3

[#]docker --version
Docker version 20.10.8, build 3967b7d

Show configuration if it is related to CRI plugin.

No response

[thaJeztah]

Looks like you're running a very old version of containerd 1.6, and a version of docker that reached EOL.

Are you able to reproduce this on a current version of containerd 1.6?