Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support of Python 3.12 #2098

Draft
wants to merge 5 commits into
base: staging
Choose a base branch
from
Draft

Support of Python 3.12 #2098

wants to merge 5 commits into from

Conversation

miguelgfierro
Copy link
Collaborator

@miguelgfierro miguelgfierro commented May 15, 2024

Description

Related Issues

#2097

References

Checklist:

  • I have followed the contribution guidelines and code style for this project.
  • I have added tests covering my contributions.
  • I have updated the documentation accordingly.
  • I have signed the commits, e.g. git commit -s -m "your commit message".
  • This PR is being made to staging branch AND NOT TO main branch.

Signed-off-by: miguelgfierro <miguelgfierro@users.noreply.github.com>
Signed-off-by: miguelgfierro <miguelgfierro@users.noreply.github.com>
Signed-off-by: miguelgfierro <miguelgfierro@users.noreply.github.com>
@miguelgfierro
Copy link
Collaborator Author

let's try, this can be as simple as changing a few lines of code, or we might need to fix many dependencies.

Error in Cornac dependency:

2024-05-15T05:14:26: Collecting cornac<2,>=1.15.2 (from recommenders[dev,gpu]@ git+https://github.com/recommenders-team/recommenders.git@809a1d39c3e4bdab72aa56ee80d9eb79413ac29d->-r /azureml-environment-setup/condaenv.8fr4a5nk.requirements.txt (line 3))
2024-05-15T05:14:26:   Downloading cornac-1.18.0.tar.gz (5.8 MB)
2024-05-15T05:14:26:      ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 5.8/5.8 MB 76.2 MB/s eta 0:00:00
2024-05-15T05:14:26:   Preparing metadata (setup.py): started
2024-05-15T05:14:26:   Preparing metadata (setup.py): finished with status 'error'

2024-05-15T05:14:26: failed
2024-05-15T05:14:26: Pip subprocess error:
2024-05-15T05:14:26:   Running command git clone --filter=blob:none --quiet https://github.com/recommenders-team/recommenders.git /tmp/pip-install-va67dnkb/recommenders_830bb823f0204ab9ae042e7d5057b430
2024-05-15T05:14:26:   Running command git rev-parse -q --verify 'sha^809a1d39c3e4bdab72aa56ee80d9eb79413ac29d'
2024-05-15T05:14:26:   Running command git fetch -q https://github.com/recommenders-team/recommenders.git 809a1d39c3e4bdab72aa56ee80d9eb79413ac29d
2024-05-15T05:14:26:   Running command git checkout -q 809a1d39c3e4bdab72aa56ee80d9eb79413ac29d
2024-05-15T05:14:26:   error: subprocess-exited-with-error
2024-05-15T05:14:26:   
2024-05-15T05:14:26:   √ó python setup.py egg_info did not run successfully.
2024-05-15T05:14:26:   │ exit code: 1
2024-05-15T05:14:26:   ╰─> [2 lines of output]
2024-05-15T05:14:26:       We need some dependencies to build Cornac.
2024-05-15T05:14:26:       Run: pip3 install Cython numpy scipy
2024-05-15T05:14:26:       [end of output]
2024-05-15T05:14:26:   
2024-05-15T05:14:26:   note: This error originates from a subprocess, and is likely not a problem with pip.
2024-05-15T05:14:26: error: metadata-generation-failed

2024-05-15T05:14:26: √ó Encountered error while generating package metadata.
2024-05-15T05:14:26: ╰─> See above for output.

2024-05-15T05:14:26: note: This is an issue with the package mentioned above, not pip.
2024-05-15T05:14:26: hint: See above for details.
2024-05-15T05:14:26: CondaEnvException: Pip failed

@miguelgfierro
Copy link
Collaborator Author

Other error, it seems it's coming from dependencies:

"error": ***
        "message": "Activity Failed:\n***\n    \"error\": ***\n        \"code\": \"UserError\",\n        \"message\": \"Image build failed. For more details, check log file azureml-logs/20_image_build_log.txt.\",\n        \"messageFormat\": \"Image build failed. For more details, check log file ***ArtifactPath***.\",\n        \"messageParameters\": ***\n            \"ArtifactPath\": \"azureml-logs/20_image_build_log.txt\"\n        ***,\n        \"details\": [],\n        \"innerError\": ***\n            \"code\": \"BadArgument\",\n            \"innerError\": ***\n                \"code\": \"ImageBuildFailure\"\n            ***\n        ***\n    ***,\n    \"correlation\": ***\n        \"operation\": \"5f6a3d4a1da7af81649b6ef9e4ebee05\",\n        \"request\": \"b3c45b90ff1ad97d\"\n    ***,\n    \"environment\": \"eastus\",\n    \"location\": \"eastus\",\n    \"time\": \"2024-05-15T06:37:34.302265Z\",\n    \"componentName\": \"RunHistory\"\n***"
    ***
***
2024-05-15T06:37:11:   Running command git fetch -q https://github.com/recommenders-team/recommenders.git 496b22fca0faf861ff98739c8deec46b031e8d44
2024-05-15T06:37:11:   Running command git checkout -q 496b22fca0faf861ff98739c8deec46b031e8d44
2024-05-15T06:37:11:   error: subprocess-exited-with-error
2024-05-15T06:37:11:   
2024-05-15T06:37:11:   × Getting requirements to build wheel did not run successfully.
2024-05-15T06:37:11:   │ exit code: 1
2024-05-15T06:37:11:   ╰─> [33 lines of output]
2024-05-15T06:37:11:       Traceback (most recent call last):
2024-05-15T06:37:11:         File "/azureml-envs/azureml_06f0f6a5b197cd9c88[281](https://github.com/recommenders-team/recommenders/actions/runs/9090676506/job/24984293755#step:3:288)4406eaed054/lib/python3.12/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 353, in <module>
2024-05-15T06:37:11:           main()
2024-05-15T06:37:11:         File "/azureml-envs/azureml_06f0f6a5b197cd9c882814406eaed054/lib/python3.12/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 335, in main
2024-05-15T06:37:11:           json_out['return_val'] = hook(**hook_input['kwargs'])
2024-05-15T06:37:11:                                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-05-15T06:37:11:         File "/azureml-envs/azureml_06f0f6a5b197cd9c882814406eaed054/lib/python3.12/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 112, in get_requires_for_build_wheel
2024-05-15T06:37:11:           backend = _build_backend()
2024-05-15T06:37:11:                     ^^^^^^^^^^^^^^^^
2024-05-15T06:37:11:         File "/azureml-envs/azureml_06f0f6a5b197cd9c882814406eaed054/lib/python3.12/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 77, in _build_backend
2024-05-15T06:37:11:           obj = import_module(mod_path)
2024-05-15T06:37:11:                 ^^^^^^^^^^^^^^^^^^^^^^^
2024-05-15T06:37:11:         File "/azureml-envs/azureml_06f0f6a5b197cd9c882814406eaed054/lib/python3.12/importlib/__init__.py", line 90, in import_module
2024-05-15T06:37:11:           return _bootstrap._gcd_import(name[level:], package, level)
2024-05-15T06:37:11:                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-05-15T06:37:11:         File "<frozen importlib._bootstrap>", line 1381, in _gcd_import
2024-05-15T06:37:11:         File "<frozen importlib._bootstrap>", line 1354, in _find_and_load
2024-05-15T06:37:11:         File "<frozen importlib._bootstrap>", line 1304, in _find_and_load_unlocked
2024-05-15T06:37:11:         File "<frozen importlib._bootstrap>", line 488, in _call_with_frames_removed
2024-05-15T06:37:11:         File "<frozen importlib._bootstrap>", line 1381, in _gcd_import
2024-05-15T06:37:11:         File "<frozen importlib._bootstrap>", line 1354, in _find_and_load
2024-05-15T06:37:11:         File "<frozen importlib._bootstrap>", line 1325, in _find_and_load_unlocked
2024-05-15T06:37:11:         File "<frozen importlib._bootstrap>", line 929, in _load_unlocked
2024-05-15T06:37:11:         File "<frozen importlib._bootstrap_external>", line 994, in exec_module
2024-05-15T06:37:11:         File "<frozen importlib._bootstrap>", line 488, in _call_with_frames_removed
2024-05-15T06:37:11:         File "/tmp/pip-build-env-boxwps2r/overlay/lib/python3.12/site-packages/setuptools/__init__.py", line 16, in <module>
2024-05-15T06:37:11:           import setuptools.version
2024-05-15T06:37:11:         File "/tmp/pip-build-env-boxwps2r/overlay/lib/python3.12/site-packages/setuptools/version.py", line 1, in <module>
2024-05-15T06:37:11:           import pkg_resources
2024-05-15T06:37:11:         File "/tmp/pip-build-env-boxwps2r/overlay/lib/python3.12/site-packages/pkg_resources/__init__.py", line 2172, in <module>
2024-05-15T06:37:11:           register_finder(pkgutil.ImpImporter, find_on_path)
2024-05-15T06:37:11:                           ^^^^^^^^^^^^^^^^^^^
2024-05-15T06:37:11:       AttributeError: module 'pkgutil' has no attribute 'ImpImporter'. Did you mean: 'zipimporter'?
2024-05-15T06:37:11:       [end of output]
2024-05-15T06:37:11:   
2024-05-15T06:37:11:   note: This error originates from a subprocess, and is likely not a problem with pip.
2024-05-15T06:37:11: error: subprocess-exited-with-error

2024-05-15T06:37:11: × Getting requirements to build wheel did not run successfully.
2024-05-15T06:37:11: │ exit code: 1
2024-05-15T06:37:11: ╰─> See above for output.

source: https://github.com/recommenders-team/recommenders/actions/runs/9090676506/job/24984293755

@daviddavo
Copy link
Collaborator

pkgutil.ImpImporter and ImpLoader have been deprecated since python 3.3, and they removed it in 3.12

As the problem is with setuptools, perhaps upgrading it or forcing a more recent version should suffice. I think cornac fails just because it is the first thing to be installed with that method.

According to that issue, changing the dependency to setuptools>=67 should work, but this might create a cascade of having to upgrade numpy, among other things.

Note: the distutils package was removed, but now is provided by setuptools. setuptools is no longer automatically installed by venv, but that's no problem because it's already specified in the pyproject

@daviddavo
Copy link
Collaborator

Now the requirements are fine, but it errors after installing build dependencies and I can't see why

2024-05-15T06:37:11:   Installing build dependencies: started
2024-05-15T06:37:11:   Installing build dependencies: finished with status 'done'
2024-05-15T06:37:11:   Getting requirements to build wheel: started
2024-05-15T06:37:11:   Getting requirements to build wheel: finished with status 'error'

2024-05-15T06:37:11: failed
2024-05-15T06:37:11: 
2024-05-15T06:37:11: CondaEnvException: Pip failed

Source: https://github.com/recommenders-team/recommenders/actions/runs/9090676506/job/24984293755

@miguelgfierro
Copy link
Collaborator Author

It seems the error is the same:

ErrorResponse 
***
    "error": ***
        "message": "Activity Failed:\n***\n    \"error\": ***\n        \"code\": \"UserError\",\n        \"message\": \"Image build failed. For more details, check log file azureml-logs/20_image_build_log.txt.\",\n        \"messageFormat\": \"Image build failed. For more details, check log file ***ArtifactPath***.\",\n        \"messageParameters\": ***\n            \"ArtifactPath\": \"azureml-logs/20_image_build_log.txt\"\n        ***,\n        \"details\": [],\n        \"innerError\": ***\n            \"code\": \"BadArgument\",\n            \"innerError\": ***\n                \"code\": \"ImageBuildFailure\"\n            ***\n        ***\n    ***,\n    \"correlation\": ***\n        \"operation\": \"0eb40f8a3a5a1b8bdc7eeb65cbe1a34b\",\n        \"request\": \"c0470de765320b12\"\n    ***,\n    \"environment\": \"eastus\",\n    \"location\": \"eastus\",\n    \"time\": \"2024-05-15T08:53:48.984678Z\",\n    \"componentName\": \"RunHistory\"\n***"
    ***
***
2024-05-15T08:53:17:   error: subprocess-exited-with-error
2024-05-15T08:53:17:   
2024-05-15T08:53:17:   × Getting requirements to build wheel did not run successfully.
2024-05-15T08:53:17:   │ exit code: 1
2024-05-15T08:53:17:   ╰─> [33 lines of output]
2024-05-15T08:53:17:       Traceback (most recent call last):
2024-05-15T08:53:17:         File "/azureml-envs/azureml_5e1c99080a501af5b78a0863f3002a98/lib/python3.12/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 353, in <module>
2024-05-15T08:53:17:           main()
2024-05-15T08:53:17:         File "/azureml-envs/azureml_5e1c99080a501af5b78a0863f3002a98/lib/python3.12/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 335, in main
2024-05-15T08:53:17:           json_out['return_val'] = hook(**hook_input['kwargs'])
2024-05-15T08:53:17:                                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-05-15T08:53:17:         File "/azureml-envs/azureml_5e1c99080a501af5b78a0863f3002a98/lib/python3.12/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 112, in get_requires_for_build_wheel
2024-05-15T08:53:17:           backend = _build_backend()
2024-05-15T08:53:17:                     ^^^^^^^^^^^^^^^^
2024-05-15T08:53:17:         File "/azureml-envs/azureml_5e1c99080a501af5b78a0863f3002a98/lib/python3.12/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 77, in _build_backend
2024-05-15T08:53:17:           obj = import_module(mod_path)
2024-05-15T08:53:17:                 ^^^^^^^^^^^^^^^^^^^^^^^
2024-05-15T08:53:17:         File "/azureml-envs/azureml_5e1c99080a501af5b78a0863f[300](https://github.com/recommenders-team/recommenders/actions/runs/9092635078/job/24989789632#step:3:307)2a98/lib/python3.12/importlib/__init__.py", line 90, in import_module
2024-05-15T08:53:17:           return _bootstrap._gcd_import(name[level:], package, level)
2024-05-15T08:53:17:                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-05-15T08:53:17:         File "<frozen importlib._bootstrap>", line 1381, in _gcd_import
2024-05-15T08:53:17:         File "<frozen importlib._bootstrap>", line 1354, in _find_and_load
2024-05-15T08:53:17:         File "<frozen importlib._bootstrap>", line 1[304](https://github.com/recommenders-team/recommenders/actions/runs/9092635078/job/24989789632#step:3:311), in _find_and_load_unlocked
2024-05-15T08:53:17:         File "<frozen importlib._bootstrap>", line 488, in _call_with_frames_removed
2024-05-15T08:53:17:         File "<frozen importlib._bootstrap>", line 1381, in _gcd_import
2024-05-15T08:53:17:         File "<frozen importlib._bootstrap>", line 1354, in _find_and_load
2024-05-15T08:53:17:         File "<frozen importlib._bootstrap>", line 1325, in _find_and_load_unlocked
2024-05-15T08:53:17:         File "<frozen importlib._bootstrap>", line 929, in _load_unlocked
2024-05-15T08:53:17:         File "<frozen importlib._bootstrap_external>", line 994, in exec_module
2024-05-15T08:53:17:         File "<frozen importlib._bootstrap>", line 488, in _call_with_frames_removed
2024-05-15T08:53:17:         File "/tmp/pip-build-env-yq5nkn2j/overlay/lib/python3.12/site-packages/setuptools/__init__.py", line 16, in <module>
2024-05-15T08:53:17:           import setuptools.version
2024-05-15T08:53:17:         File "/tmp/pip-build-env-yq5nkn2j/overlay/lib/python3.12/site-packages/setuptools/version.py", line 1, in <module>
2024-05-15T08:53:17:           import pkg_resources
2024-05-15T08:53:17:         File "/tmp/pip-build-env-yq5nkn2j/overlay/lib/python3.12/site-packages/pkg_resources/__init__.py", line 2172, in <module>
2024-05-15T08:53:17:           register_finder(pkgutil.ImpImporter, find_on_path)
2024-05-15T08:53:17:                           ^^^^^^^^^^^^^^^^^^^
2024-05-15T08:53:17:       AttributeError: module 'pkgutil' has no attribute 'ImpImporter'. Did you mean: 'zipimporter'?
2024-05-15T08:53:17:       [end of output]
2024-05-15T08:53:17:   
2024-05-15T08:53:17:   note: This error originates from a subprocess, and is likely not a problem with pip.
2024-05-15T08:53:17: error: subprocess-exited-with-error

This seems like a long problem to debug. In my experience, it is better to debug it first in local. @daviddavo would you have time to take a look?

@daviddavo
Copy link
Collaborator

Yeah, I can try and give it some time next week

@daviddavo
Copy link
Collaborator

I can't install lightfm on Python 3.12, this affects recommenders too

@daviddavo
Copy link
Collaborator

Forked the repository and made a PR

The package can be installed from my repository:

pip install git+https://github.com/daviddavo/lightfm

If the pull requests wasn't approved, I can transfer ownership of this repository to recommenders-team, and then that github repository could be used in the requirements.

Signed-off-by: David Davó <david@ddavo.me>
@daviddavo
Copy link
Collaborator

By changing temporarily changing the requirements, it works on my machine. Let's see if it works on the test groups

@daviddavo
Copy link
Collaborator

daviddavo commented May 21, 2024

The following command works on my machine:

python -m venv venv
source venv/bin/activate
pip install git+https://github.com/recommenders-team/recommenders@python312

But perhaps my system comes with some build dependencies that are not installed in the test groups...

Tomorrow I'll try and use conda to install the package on my machine and see how it goes

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants