v0.28.1 Version Mismatch with TF 2.12.0. Works with v0.28.0

[liamaltarac]

Environment:

1. Framework: TensorFlow, Keras
2. Framework version: 2.12.0
3. Horovod version: 0.28.1
4. MPI version:
5. CUDA version:
6. NCCL version:
7. Python version: 3.10
8. Spark / PySpark version:
9. Ray version:
10. OS and version:
11. GCC version:
12. CMake version:

Bug report:

1. pip install keras==2.12.0
2. pip install tensorflow==2.12.0
3. HOROVOD_WITH_TENSORFLOW=1 pip install horovod
4. (in python) : import horovod.keras as hvd

I get this error:

horovod.common.exceptions.HorovodVersionMismatchError: Framework tensorflow installed with version 2.14.0 but found version 2.12.0.
This can result in unexpected behavior including runtime errors.
Reinstall Horovod using pip install --no-cache-dir to build with the new version.

I tried reinstalling Horovod with --no-cache-dir, but it still gave the same error.

It works however when I install with Horovod v0.28.0