Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ttnn.sinh operation fails with low PCC in most situations #8598

Open
nemanjagrujic opened this issue May 17, 2024 · 6 comments
Open

ttnn.sinh operation fails with low PCC in most situations #8598

nemanjagrujic opened this issue May 17, 2024 · 6 comments
Labels
bug Something isn't working GS op_cat: eltwise WH

Comments

@nemanjagrujic
Copy link
Contributor

nemanjagrujic commented May 17, 2024

ttnn.sinh operation fails with low PCC in all tested situations.

Problem is observed on both GS and WH cards.

To Reproduce
Steps to reproduce the behavior:

  1. Checkout branch ngrujic/op_bug_unit_tests (soon to be merged into main).
  2. Run unit test test_eltwise_sinh.py using this command:
    pytest tests/ttnn/python_api_testing/non_working_unit_tests/grayskull/test_eltwise_sinh.py

Expected behavior
There are multiple test cases presented in the unit test, which are failing with low PCC.

Getting Additional info for the operation under test and its behavior
To get additional information and results for different combinations of input shapes, types, layouts and memory configs for which this operation was tested you can also run locally sweeps and check the results. To do this you should:

  1. Run non working sweep by using pytest tests/ttnn/python_api_testing/sweep_tests/run_sweep_test.py --input-path tests/ttnn/python_api_testing/sweep_tests/test_configs/ci_sweep_tests_broken/grayskull/ttnn_eltwise_sinh_test.yaml --input-method cli --cli-input results_ttnn_sinh
  2. After the run is completed all test sweeps results should be available inside specified output directory (in this case ./results_ttnn_sinh).
@tt-aho
Copy link
Contributor

tt-aho commented May 17, 2024

I added mixed precision support to bcast on latest, so it should no longer crash.

@nemanjagrujic
Copy link
Contributor Author

nemanjagrujic commented May 20, 2024

Hello, @tt-aho, I tried with rebasing to latest main today and its still failing.

@tt-aho
Copy link
Contributor

tt-aho commented May 20, 2024

Hey, that was my mistake. I added support for it in the op itself but forgot to remove the assert. Should be resolved on latest main a1b9781

@nemanjagrujic
Copy link
Contributor Author

Hello @tt-aho, just rebased with latest main and rebuilt, but its still the same issue. Not sure what is happening. You can always test your self by running:

pytest tests/ttnn/python_api_testing/non_working_unit_tests/grayskull/test_eltwise_sinh.py

To see if its fixed.

@tt-aho
Copy link
Contributor

tt-aho commented May 21, 2024

Hey, the crash/error should finally be resolved (d5c7fef). I've verified locally that it now only mismatches

@nemanjagrujic
Copy link
Contributor Author

Confirmed. Now only PCC is failing.

@nemanjagrujic nemanjagrujic changed the title ttnn.sinh operation fails with low PCC in most situations and bfloat8_b crashes ttnn.sinh operation fails with low PCC in most situations May 22, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working GS op_cat: eltwise WH
Projects
None yet
Development

No branches or pull requests

2 participants