Discussion: Align traceloop and tracer gadgets

[mauriciovasquezbernal]

The traceloop gadget is unique. It uses an over-writable ring buffer to store the last syscalls executed by a process / container, they are only dumped when requested by the user. We usually describe it as a flight recorder as it saves the last syscalls the process / container executed before crashing / exiting.

This "flight recorder" behavior can be useful for other tracers, hence the my proposal on this issue is to align the traceloop and tracer gadgets. Specifically:

• All tracers should implement a --loop option that tells them to run in "flight recorder" mode.
• Traceloop should be converted in a "trace syscall" gadget, also providing the --loop option for sure.

This issue covers only image-based gadgets, we shouldn't make such change to built-in gadgets as the goal is to deprecate and remove them later on.

Any thoughts on this one?

@alban do you think this is feasible from the technical point of view?

[eiffel-fl]

Any thoughts on this one?

This seems a good idea, but --loop would only be compatible with perf buffers, as ring ones cannot be written backward:
https://lore.kernel.org/lkml/20220906195656.33021-2-flaniel@linux.microsoft.com/T/#mce0a6e920165e5b55d4abdc83e522363364ef229

[alban]

All tracers should implement a --loop option that tells them to run in "flight recorder" mode.

In traceloop, there is one perf ring buffer for each container, so one verbose container does not fill the ring buffer for other containers, and users can decide which container's ring buffer to dump at the end.

In order to do the same in a generic way in image-based gadget, we would need to automatically install a generic ebpf program on tracepoints/kprobes and do a tail call to the gadget ebpf program instantiated for each container with its own ring buffer.

It is doable for tracepoints/kprobes. But it is more difficult to do for socket filters and uprobes because they already use some kind of ebpf program duplication (networktracer and uprobetracer).

[mauriciovasquezbernal]

In order to do the same in a generic way in image-based gadget, we would need to automatically install a generic ebpf program on tracepoints/kprobes and do a tail call to the gadget ebpf program instantiated for each container with its own ring buffer.

Could you please extend a bit more on this idea? Why do we need the tail call? Can't the same eBPF program handle the logic to put the event in the right perf ring buffer?

[alban]

Could you please extend a bit more on this idea? Why do we need the tail call? Can't the same eBPF program handle the logic to put the event in the right perf ring buffer?

If it is a single eBPF program, then it is the one coming from the gadget. So the gadget author will have to use bpf_perf_event_output with the correct struct bpf_map *map in parameter:

long bpf_perf_event_output(void *ctx, struct bpf_map *map, u64 flags, void *data, u64 size)

To get the correct map reference for a specific container, the gadget author would have to use a map of maps (BPF_MAP_TYPE_HASH_OF_MAPS). The reason why traceloop was not implemented with a map of maps was that it required Linux 4.12. That reason is obsolete now. But for image-based gadgets, the program would have a different code path in the --loop=true case and in the --loop=false case. So we add a burden on the (potentially third-party) gadget author.