mpv: Runners seems to be stuck

[kasper93]

Hi,

Initially I though it was due to excessive timeouts, but they have been fixed now. Some of testcases are stuck, all I see is pending status and progression started that never ends.

oss-fuzz-linux-zone8-host-scn6-11: Progression task started.

Sure enough after searching similar issues, I found #11490 that was related to disk space issues on runners. And now is my fault, because we were leaking files in /tmp... oops, sorry, though it would be one per process, not that much data. It now has been fixed and rewritten to use memfd_create. mpv-player/mpv@6ede789

I'm creating this issue, because there is not much visibility into runners. Currently I don't see many of fuzz binaries running, stats and logs are missing, coverage build is failing. So I presume /tmp is persistent and it is failing?

Could you take a look and see if runners rebuild is needed similar to #11490?

EDIT:

One more generic question, what are the limits of concurrent jobs? FAQ says

Fuzzing machines only have a single core and fuzz targets should not use more than 2.5GB of RAM.

Say we have N fuzzing targets multiplied by sanitizers and fuzzing engines, each target is allowed one fuzz runner or they are queued and what's the limit?

EDIT2: I think I found the root cause #11965 (will close this issue if this helps after merge)

EDIT3: Nothing changed, still there is no progression.

EDIT4: Example of completely stuck testcase https://oss-fuzz.com/testcase-detail/4875501058457600

Thanks,
Kacper

[kasper93]

Sorry to bother you again. Is there anything I can do to help resolve this situation? Currently there seem to be no jobs running at all. So far only clue I have is that disk quota is exceeded and this makes runners stuck somehow. Is /tmp storage persistent? In libfuzzer fork mode (which seems to be used) it would indeed leak some files there previously, but I have no way to validate that this is the problem. I don't think fuzzers itself are that big to cause the problem.

Everything is working fine locally and with cifuzz workflow, only clusterfuzz (oss-fuzz) seems to be stuck completely.

[oliverchang]

Sorry for the delay. It doesn't appear to be a disk space issue, and I'm not sure why they're stuck. I'll kick off a restart of all the machines to see if that resolves it.

[kasper93]

Thank you. Unfortunately nothing moved. On fuzzer statistic I get Got error with status: 404, on testcase(s) [2024-05-24 13:08:05 UTC] oss-fuzz-linux-zone8-host-lt79-0: Progression task started. and Pending status.

In fairness, it never fully worked, since the initial integration we got some crash reports and some of them were detected as fixed. So far so good, but we never got corpus saved, coverage build since the beginning is failing with

Step #5: Failed to unpack the corpus for fuzzer_load_config_file. This usually means that corpus backup for a particular fuzz target does not exist. If a fuzz target was added in the last 24 hours, please wait one more day. Otherwise, something is wrong with the fuzz target or the infrastructure, and corpus pruning task does not finish successfully.

I thought it needs to stabilize, but now it doesn't seem to give any sign of life, no logs, reports.

I've tested locally full infra/helper.py pipeline and I can generate coverage report without issue, so build and fuzzers seems to be ok. I'd appreciate any help on this matter. I had plans to improve things, add initial corpus, but first we need to stabilize things. There is no rush, but if you need anything on my side to change/update, let me know.

[kasper93]

It is probably by this typo 9ad0f4d Would be nice if we had some feedback on ClasterFuzz about possible errors during operation. Else things just don't work and there is no really way to see why.