Falco on k3s on OpenStack

[raminrabbani]

Falco not starting and staying in Init:CrashLoopBackOff state when installed on k3s on OpenStack.
Same if driver.kind is set to "kmod", "ebpf", or ""
Describe the bug

How to reproduce it
Install OpenStack - https://docs.openstack.org/install-guide/
Install k3s - https://github.com/CyVerse-Ansible/ansible-jupyterhub
Install falco - https://falco.org/docs/getting-started/falco-kubernetes-quickstart/

Expected behaviour
Falco to be in running state as below:

NAME READY STATUS RESTARTS AGE
falco-falcosidekick-59c5d6cc45-l2qjz 1/1 Running 0 3m7s
falco-falcosidekick-59c5d6cc45-kpcws 1/1 Running 0 3m7s
falco-vdsc8 2/2 Running 0 3m7s

Screenshots

kubectl get pods -n falco
NAME READY STATUS RESTARTS AGE
falco-falcosidekick-64755fc647-8g8kd 1/1 Running 0 2m46s
falco-falcosidekick-64755fc647-9pzr6 1/1 Running 0 2m46s
falco-nf222 1/2 CrashLoopBackOff 5 (23s ago) 2m46s

Environment

• Falco version:

latest from https://falco.org/docs/getting-started/falco-kubernetes-quickstart/

• System info:

• Cloud provider or hardware configuration:
• OS:

PRETTY_NAME="Ubuntu 22.04.1 LTS"
NAME="Ubuntu"
VERSION_ID="22.04"
VERSION="22.04.1 LTS (Jammy Jellyfish)"
VERSION_CODENAME=jammy
ID=ubuntu
ID_LIKE=debian
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
UBUNTU_CODENAME=jammy

• Kernel:

Linux osm 5.15.0-57-generic #63-Ubuntu SMP Thu Nov 24 13:43:17 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux

• Installation method:

Kubernetes helm chart
Additional context

Here is the result of running command "kubectl logs daemonset/falco -n falco -c falco-driver-loader"

• Setting up /usr/src links from host
  2024-05-16 10:42:54 INFO Running falcoctl driver install
  ├ driver version: 7.0.0+driver
  ├ driver type: kmod
  ├ driver name: falco
  ├ compile: true
  ├ download: true
  ├ arch: x86_64
  ├ kernel release: 5.15.0-57-generic
  └ kernel version: Reduce rule FPs based on more complex environments #63-Ubuntu SMP Thu Nov 24 13:43:17 UTC 2022
  2024-05-16 10:42:54 INFO Found distro target: ubuntu-generic
  2024-05-16 10:42:54 INFO Check if kernel module is still loaded.
  2024-05-16 10:42:54 INFO Kernel module is still loaded.
  2024-05-16 10:42:54 INFO Trying to unload it with 'rmmod'.
  2024-05-16 10:42:55 INFO OK! Unloading module succeeded.
  2024-05-16 10:42:55 INFO Check all versions of kernel module in dkms.
  2024-05-16 10:42:55 INFO OK! There are no module versions in dkms.
  2024-05-16 10:42:55 INFO Trying to download a driver.
  └ url: https://download.falco.org/driver/7.0.0%2Bdriver/x86_64/falco_ubuntu-generic_5.15.0-57-generic_63.ko
  2024-05-16 10:42:55 WARN Non-200 response from url. code: 404
  2024-05-16 10:42:55 WARN unable to find a prebuilt driver
  2024-05-16 10:42:55 INFO Trying to dkms install module. gcc: /usr/bin/gcc
  2024-05-16 10:43:14 INFO Module installed in dkms.
  └ file: /var/lib/dkms/falco/7.0.0+driver/5.15.0-57-generic/x86_64/module/falco.ko
  2024-05-16 10:43:14 INFO Copying built driver to its destination.
  ├ src: /var/lib/dkms/falco/7.0.0+driver/5.15.0-57-generic/x86_64/module/falco.ko
  └ dst: /root/.falco/7.0.0+driver/x86_64/falco_ubuntu-generic_5.15.0-57-generic_63.ko
  2024-05-16 10:43:14 INFO Driver built.
  └ path: /root/.falco/7.0.0+driver/x86_64/falco_ubuntu-generic_5.15.0-57-generic_63.ko
  2024-05-16 10:43:14 INFO Success: module found and loaded in dkms.
  └ driver: /root/.falco/7.0.0+driver/x86_64/falco_ubuntu-generic_5.15.0-57-generic_63.ko

[FedeDP]

Hi! Thanks for opening this issue!
The log you posted shows that falco-driver-loader was able to build the driver; can you share the falco container logs?
kubectl logs daemonset/falco -n falco -c falco

[raminrabbani]

Hi,

Here are the logs:

$ kubectl logs daemonset/falco -n falco -c falco
Fri May 17 09:28:51 2024: Falco version: 0.37.1 (x86_64)
Fri May 17 09:28:51 2024: Falco initialized with configuration file: /etc/falco/falco.yaml
Fri May 17 09:28:51 2024: System info: Linux version 5.15.0-57-generic (buildd@lcy02-amd64-110) (gcc (Ubuntu 11.3.0-1ubuntu1~22.04) 11.3.0, GNU ld (GNU Binutils for Ubuntu) 2.38) #63-Ubuntu SMP Thu Nov 24 13:43:17 UTC 2022
Fri May 17 09:28:51 2024: Loading rules from file /etc/falco/falco_rules.yaml
Fri May 17 09:28:51 2024: Hostname value has been overridden via environment variable to: osm
Error: could not initialize inotify handler

Based on the error, I researched potential solutions and found this:

Check the inotify limits on your host and increase them if necessary. You can do this by editing /etc/sysctl.conf or creating a file in /etc/sysctl.d/, then reloading the sysctl settings.

Run the following commands:
$ echo "fs.inotify.max_user_instances=8192" | sudo tee -a /etc/sysctl.conf
$ echo "fs.inotify.max_user_watches=1048576" | sudo tee -a /etc/sysctl.conf
$ sudo sysctl -p

This helped me solve the issue. The inotify limits were increased, and Falco is now running without errors.

[FedeDP]

Yay!
/milestone 0.38.0