Memory leak in 0.13.2-6476f8a #10649
Replies: 4 comments
-
please fill out the support template, there is not much information provided here |
Beta Was this translation helpful? Give feedback.
-
This is not Frigate's issue. Follow this tutorial to fix the issue: |
Beta Was this translation helpful? Give feedback.
-
Is Proxmox aware? |
Beta Was this translation helpful? Give feedback.
-
of possibily an issue with NFS ?! https://forum.proxmox.com/threads/memory-leak-on-6-8-4-2-3-pve-8-2.146649/ here , i roll back to kernel 6.5 and it's ok for now. |
Beta Was this translation helpful? Give feedback.
-
After upgrading to 0.13.2-6476f8a I have frequent errors in all my frigate servers.
Frigate in Proxmox containers (LXC) with 2Gb RAM each.
Increasing the RAM doesn't help.
Call Trace:
[46862.703189]
[46862.703194] dump_stack_lvl+0x4a/0x63
[46862.703203] dump_stack+0x10/0x16
[46862.703207] dump_header+0x53/0x225
[46862.703214] oom_kill_process.cold+0xb/0x10
[46862.703219] out_of_memory+0x1dc/0x530
[46862.703232] mem_cgroup_out_of_memory+0x145/0x160
[46862.703238] try_charge_memcg+0x69d/0x760
[46862.703245] charge_memcg+0x45/0xb0
[46862.703249] __mem_cgroup_charge+0x2d/0x90
[46862.703253] __add_to_page_cache_locked+0x2e1/0x360
[46862.703258] ? scan_shadow_nodes+0x40/0x40
[46862.703265] add_to_page_cache_lru+0x4d/0xd0
[46862.703269] pagecache_get_page+0x1c2/0x5c0
[46862.703274] filemap_fault+0x488/0xb10
[46862.703277] ? filemap_map_pages+0x139/0x660
[46862.703282] __do_fault+0x3c/0x120
[46862.703287] __handle_mm_fault+0xde8/0x1610
[46862.703293] handle_mm_fault+0xd8/0x2c0
[46862.703297] do_user_addr_fault+0x1c2/0x660
[46862.703304] ? exit_to_user_mode_prepare+0x90/0x1b0
[46862.703311] exc_page_fault+0x77/0x170
[46862.703318] asm_exc_page_fault+0x27/0x30
[46862.703325] RIP: 0033:0x7fd9d3a1f9f0
[46862.703334] Code: Unable to access opcode bytes at RIP 0x7fd9d3a1f9c6.
[46862.703336] RSP: 002b:00007ffec21c80e8 EFLAGS: 00010246
[46862.703340] RAX: 0000000000000000 RBX: 0000560af8de2f48 RCX: 0000000000000000
[46862.703343] RDX: ffffffffffffff80 RSI: 000000000000000b RDI: 0000560af8de2f10
[46862.703346] RBP: 00007fd9d3bae240 R08: 0000000000000000 R09: 0000000000000000
[46862.703348] R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000000
[46862.703350] R13: 0000560af8de3430 R14: 0000560af8de2f10 R15: 0000560af8de3a40
[46862.703356]
[46862.703371] memory: usage 2097252kB, limit 2097152kB, failcnt 30836
[46862.703375] swap: usage 0kB, limit 0kB, failcnt 0
[46862.703377] Memory cgroup stats for /lxc/2203:
[46862.703399] anon 2009395200
[46862.703399] file 51322880
[46862.703399] kernel_stack 14172160
[46862.703399] pagetables 22544384
[46862.703399] percpu 2007424
[46862.703399] sock 23662592
[46862.703399] shmem 51138560
[46862.703399] file_mapped 14696448
[46862.703399] file_dirty 0
[46862.703399] file_writeback 45056
[46862.703399] swapcached 0
[46862.703399] anon_thp 127926272
[46862.703399] file_thp 0
[46862.703399] shmem_thp 0
[46862.703399] inactive_anon 2043564032
[46862.703399] active_anon 16969728
[46862.703399] inactive_file 159744
[46862.703399] active_file 24576
[46862.703399] unevictable 0
[46862.703399] slab_reclaimable 1844824
[46862.703399] slab_unreclaimable 19508376
[46862.703399] slab 21353200
[46862.703399] workingset_refault_anon 0
[46862.703399] workingset_refault_file 1548838
[46862.703399] workingset_activate_anon 0
[46862.703399] workingset_activate_file 106220
[46862.703399] workingset_restore_anon 0
[46862.703406] Tasks state (memory values in pages):
[46862.703407] [ pid ] uid tgid total_vm rss pgtables_bytes swapents oom_score_adj name
[46862.703411] [ 824978] 0 824978 41397 1631 94208 0 0 systemd
[46862.703425] [ 825778] 0 825778 1842 282 53248 0 0 agetty
[46862.703430] [ 839725] 0 839725 3006 578 65536 0 0 login
[46862.703435] [3918613] 0 3918613 2511 678 53248 0 0 bash
[46862.703457] [3918651] 0 3918651 3012 585 57344 0 0 top
[46862.703463] [ 825604] 0 825604 27477 2292 241664 0 -250 systemd-journal
[46862.703469] [ 825626] 104 825626 4012 841 77824 0 0 systemd-network
[46862.703474] [ 825650] 0 825650 24970 902 69632 0 0 dhclient
[46862.703479] [ 825744] 107 825744 1975 517 57344 0 0 rpcbind
[46862.703484] [ 825745] 105 825745 6025 1869 90112 0 0 systemd-resolve
[46862.703516] [ 825753] 0 825753 2152 408 53248 0 0 cron
[46862.703522] [ 825754] 106 825754 2062 605 57344 0 -900 dbus-daemon
[46862.703527] [ 825756] 0 825756 55199 495 77824 0 0 rsyslogd
[46862.703532] [ 825757] 0 825757 3350 1012 61440 0 0 systemd-logind
[46862.703537] [ 825771] 0 825771 339365 4650 282624 0 -999 containerd
[46862.703542] [ 827736] 0 827736 180082 1315 110592 0 -998 containerd-shim
[46862.703547] [3918798] 0 3918798 290946 1463 167936 0 -998 runc
[46862.703552] [ 826044] 110 826044 4651 526 81920 0 0 exim4
[46862.703557] [ 825768] 0 825768 90050 5532 237568 0 0 ruby
[46862.703562] [ 825795] 112 825795 5335 473 69632 0 0 zabbix_agentd
[46862.703566] [ 825805] 112 825805 5335 509 69632 0 0 zabbix_agentd
[46862.703571] [ 825806] 112 825806 5390 1036 81920 0 0 zabbix_agentd
[46862.703575] [ 825807] 112 825807 5390 1055 81920 0 0 zabbix_agentd
[46862.703579] [ 825808] 112 825808 5390 1052 81920 0 0 zabbix_agentd
[46862.703584] [ 825809] 112 825809 5392 1026 81920 0 0 zabbix_agentd
[46862.703598] [ 825775] 0 825775 1842 280 53248 0 0 agetty
[46862.703603] [ 826119] 0 826119 628963 12518 585728 0 -500 dockerd
[46862.703609] [ 827648] 0 827648 417930 482 147456 0 -500 docker-proxy
[46862.703614] [ 827655] 0 827655 399497 514 139264 0 -500 docker-proxy
[46862.703618] [ 827670] 0 827670 417930 504 143360 0 -500 docker-proxy
[46862.703623] [ 827676] 0 827676 399497 492 135168 0 -500 docker-proxy
[46862.703628] [ 827691] 0 827691 417994 493 143360 0 -500 docker-proxy
[46862.703633] [ 827697] 0 827697 417930 498 139264 0 -500 docker-proxy
[46862.703638] [ 827713] 0 827713 399561 579 143360 0 -500 docker-proxy
[46862.703643] [ 827718] 0 827718 399497 512 135168 0 -500 docker-proxy
[46862.703648] [ 827756] 0 827756 52 5 28672 0 0 s6-svscan
[46862.703652] [ 827796] 0 827796 53 5 24576 0 0 s6-supervise
[46862.703657] [ 827798] 0 827798 50 1 24576 0 0 s6-linux-init-s
[46862.703661] [ 827811] 0 827811 53 4 28672 0 0 s6-supervise
[46862.703665] [ 827812] 0 827812 53 5 28672 0 0 s6-supervise
[46862.703669] [ 827813] 0 827813 53 5 28672 0 0 s6-supervise
[46862.703673] [ 827814] 0 827814 53 5 28672 0 0 s6-supervise
[46862.703678] [ 827815] 0 827815 53 6 28672 0 0 s6-supervise
[46862.703682] [ 827816] 0 827816 53 5 28672 0 0 s6-supervise
[46862.703686] [ 827817] 0 827817 53 5 28672 0 0 s6-supervise
[46862.703690] [ 827818] 0 827818 53 5 28672 0 0 s6-supervise
[46862.703694] [ 827819] 0 827819 53 5 28672 0 0 s6-supervise
[46862.703699] [ 827831] 0 827831 131 16 28672 0 0 s6-fdholderd
[46862.703713] [ 827833] 0 827833 47 1 24576 0 0 s6-ipcserverd
[46862.703718] [ 828029] 65534 828029 69 26 36864 0 0 s6-log
[46862.703722] [ 828032] 65534 828032 69 1 40960 0 0 s6-log
[46862.703727] [ 828036] 65534 828036 77 32 40960 0 0 s6-log
[46862.703731] [ 828040] 0 828040 310467 11535 233472 0 0 go2rtc
[46862.703736] [ 828058] 0 828058 973 436 53248 0 0 bash
[46862.703740] [ 828061] 0 828061 433874 105637 1572864 0 0 python3
[46862.703744] [ 828073] 0 828073 148012 2102 110592 0 0 nginx
[46862.703749] [ 828116] 0 828116 213774 1162 229376 0 0 nginx
[46862.703753] [ 828117] 0 828117 213774 1164 229376 0 0 nginx
[46862.703757] [ 828118] 0 828118 213774 1096 229376 0 0 nginx
[46862.703762] [ 828119] 0 828119 213774 1109 229376 0 0 nginx
[46862.703766] [ 828121] 0 828121 213774 1182 229376 0 0 nginx
[46862.703770] [ 828122] 0 828122 213774 1094 229376 0 0 nginx
[46862.703774] [ 828123] 0 828123 213774 1201 229376 0 0 nginx
[46862.703779] [ 828124] 0 828124 213774 1169 229376 0 0 nginx
[46862.703783] [ 828125] 0 828125 213774 1083 229376 0 0 nginx
[46862.703787] [ 828126] 0 828126 213774 1209 229376 0 0 nginx
[46862.703792] [ 828127] 0 828127 213774 1221 229376 0 0 nginx
[46862.703796] [ 828128] 0 828128 213774 1196 229376 0 0 nginx
[46862.703800] [ 828129] 0 828129 213774 1214 229376 0 0 nginx
[46862.703805] [ 828130] 0 828130 213774 1196 229376 0 0 nginx
[46862.703809] [ 828131] 0 828131 213774 1186 229376 0 0 nginx
[46862.703822] [ 828132] 0 828132 213774 1209 229376 0 0 nginx
[46862.703849] [ 828133] 0 828133 148062 876 94208 0 0 nginx
[46862.703853] [ 829778] 0 829778 36119 2883 184320 0 0 ffmpeg
[46862.703858] [ 830210] 0 830210 36119 2902 192512 0 0 ffmpeg
[46862.703862] [ 830436] 0 830436 124174 24589 458752 0 0 frigate.logger
[46862.703867] [ 830838] 0 830838 244103 48820 843776 0 0 frigate.recordi
[46862.703871] [ 830847] 0 830847 3781 1849 69632 0 0 python3
[46862.703875] [ 830848] 0 830848 253906 33639 626688 0 0 frigate.detecto
[46862.703880] [ 830851] 0 830851 351842 32173 684032 0 0 frigate.output
[46862.703885] [ 830859] 0 830859 31705 2322 163840 0 0 ffmpeg
[46862.703889] [ 830871] 0 830871 31705 2291 167936 0 0 ffmpeg
[46862.703893] [ 830872] 0 830872 31705 2288 167936 0 0 ffmpeg
[46862.703898] [ 830873] 0 830873 31705 2297 167936 0 0 ffmpeg
[46862.703903] [ 830874] 0 830874 31705 2273 167936 0 0 ffmpeg
[46862.703907] [ 830917] 0 830917 287727 43797 745472 0 0 frigate.process
[46862.703912] [ 830921] 0 830921 287809 43700 749568 0 0 frigate.process
[46862.703916] [ 830922] 0 830922 287873 43846 749568 0 0 frigate.process
[46862.703920] [ 830924] 0 830924 280641 38990 765952 0 0 frigate.capture
[46862.703935] [ 830925] 0 830925 294844 41118 679936 0 0 frigate.capture
[46862.703940] [ 830928] 0 830928 284017 36768 634880 0 0 frigate.capture
[46862.703945] [ 830933] 0 830933 280641 37883 659456 0 0 frigate.capture
[46862.703950] [ 830936] 0 830936 281815 37687 688128 0 0 frigate.audio_m
[46862.703956] [ 830943] 0 830943 122583 14126 319488 0 0 ffmpeg
[46862.703961] [1514787] 0 1514787 36119 2904 196608 0 0 ffmpeg
[46862.703966] [3053514] 0 3053514 121186 13462 299008 0 0 ffmpeg
[46862.703970] [3542320] 0 3542320 36349 3144 192512 0 0 ffmpeg
[46862.703975] [3651746] 0 3651746 245059 147719 1392640 0 0 ffmpeg
[46862.703980] [3831390] 0 3831390 123183 37964 491520 0 0 ffmpeg
[46862.703985] [3831408] 0 3831408 37121 3822 204800 0 0 ffmpeg
[46862.703989] [3914504] 0 3914504 622 115 40960 0 0 sleep
[46862.703994] [3919309] 0 3919309 272513 1118 151552 0 0 runc:[2:INIT]
[46862.703999] [ 833560] 0 833560 3338 1178 65536 0 -1000 sshd
[46862.704004] [ 841965] 0 841965 43593 11895 315392 0 0 puppet
[46862.704010] oom-kill:constraint=CONSTRAINT_MEMCG,nodemask=(null),cpuset=mcollective.service,mems_allowed=0-1,oom_memcg=/lxc/2203,task_memcg=/lxc/2203/ns/system.slice/docker-8c770fc842fc6939e0d2afd9ed8baaac97a81b38bebaa1856a5da4ae383e7316.scope,task=ffmpeg,pid=3651746,uid=0
[46862.704080] Memory cgroup out of memory: Killed process 3651746 (ffmpeg) total-vm:980236kB, anon-rss:583108kB, file-rss:7768kB, shmem-rss:0kB, UID:0 pgtables:1360kB oom_score_adj:0
Beta Was this translation helpful? Give feedback.
All reactions