Mamba end-to-end perf hanging in CI pipelines

[esmalTT]

Failed pipelines:

• https://github.com/tenstorrent/tt-metal/actions/runs/9099763560/job/25055147766

[tt-rkim]

Is there update to this? Are we just running it manually to get perf numbers?

[esmalTT]

Is there update to this? Are we just running it manually to get perf numbers?

We were never able to replicate this issue locally. There was an FD hang that was fixed this week, it may have been related.

I would suggest we re-enable this for a few days to see if the issue is still occurring.

[tt-rkim]

Looks like it hung: https://github.com/tenstorrent/tt-metal/actions/runs/9407769341/job/25914428732

It seems that tt_cache_path is still set to None though unless I'm reading the code wrong... so it can't be a cache problem?

[esmalTT]

Is it possible we are just timing out?

…

________________________________ From: Raymond Kim ***@***.***> Sent: Thursday, June 6, 2024 7:28:59 PM To: tenstorrent/tt-metal ***@***.***> Cc: Evan Smal ***@***.***>; Author ***@***.***> Subject: [CAUTION - External Sender] Re: [tenstorrent/tt-metal] Mamba end-to-end perf hanging in CI pipelines (Issue #8606) Looks like it hung: https://github.com/tenstorrent/tt-metal/actions/runs/9407769341/job/25914428732<https://www.google.com/url?q=https://github.com/tenstorrent/tt-metal/actions/runs/9407769341/job/25914428732&source=gmail-imap&ust=1718321340000000&usg=AOvVaw23IAmbvovRU0O7WHTwD7XQ> It seems that tt_cache_path is still set to None though unless I'm reading the code wrong... so it can't be a cache problem? — Reply to this email directly, view it on GitHub<https://www.google.com/url?q=https://github.com/tenstorrent/tt-metal/issues/8606%23issuecomment-2153558871&source=gmail-imap&ust=1718321340000000&usg=AOvVaw2y-2C2Rt-yeb8Eb8yg51Ky>, or unsubscribe<https://www.google.com/url?q=https://github.com/notifications/unsubscribe-auth/BF6SXPVMVWRFAVSVZV2ZE2DZGDWDXAVCNFSM6AAAAABH4LIARSVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDCNJTGU2TQOBXGE&source=gmail-imap&ust=1718321340000000&usg=AOvVaw0G1oHJpRd2mnfMDmeDy5Nc>. You are receiving this because you authored the thread.Message ID: ***@***.***>

[tt-rkim]

Maybe, but is this supposed to take over 15 mins on such a powerful BM like this?

[tt-rkim]

bumped up to 40 min to see if we need more time https://github.com/tenstorrent/tt-metal/actions/runs/9419004292

[tt-rkim]

@esmalTT Looks like it took 4 mins on the Single-card BM.
Running a couple more times, but my initial guess is that it's ND hanging.