How to get a python venv working?

[MeisterP]

I have GoldenCheetah compiled myself with system python enabled.
Starting with system python works as expected:

$ GoldenCheetah
Python loaded [3.10.10 (main, Mar 15 2023, 17:39:16) [GCC 12.2.1 20230121]]

Creating a venv with:

$ python3.10 -m venv gc-test
$ source gc-test/bin/activate

Checking for the path according to https://github.com/GoldenCheetah/GoldenCheetah/wiki/UG_Special-Topics_Working-with-Python#using-the-python-chart

$ which python
/home/poncho/Tmp/gc-test/bin/python

$ python
Python 3.10.10 (main, Mar 15 2023, 17:39:16) [GCC 12.2.1 20230121] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import sys
>>> sys.exec_prefix
'/home/poncho/Tmp/gc-test'

But when launching GoldenCheetah with "Python home" set to /home/poncho/Tmp/gc-test, I get:

$ GoldenCheetah
Didn't get output: /home/poncho/Tmp/gc-test/bin/python3.10

While a popup window shows:

Python v3.10 is required for Python embedding.
Python disabled in preferences.

Some help would be appreciated.

[liversedge]

I would enable python debug in PythonEmbed.h (the default is false) and see what it is doing

[MeisterP]

Looks like the binary is found and run. But something fails afterwards:

$ GoldenCheetah --debug
[Python/PythonEmbed.cpp:219 PythonEmbed] PYTHONHOME setting used: /home/poncho/Tmp/gc-test
[Python/PythonEmbed.cpp:101 pythonInstalled] Binary found
[Python/PythonEmbed.cpp:116 pythonInstalled] Running: /home/poncho/Tmp/gc-test/bin/python3.10
Didn't get output: /home/poncho/Tmp/gc-test/bin/python3.10
[Python/PythonEmbed.cpp:316 PythonEmbed] Embedding failed

Seems that the waitForReadyRead call fails in https://github.com/GoldenCheetah/GoldenCheetah/blob/master/src/Python/PythonEmbed.cpp#L138
I did increase the timeout just to be sure that it's not a timing issue. That didn't help though.

[MeisterP]

When I change the code to just run python --version instead of python -c "import sys ...." I actually get to the point where the output is parsed.

[Python/PythonEmbed.cpp:214 PythonEmbed] PYTHONHOME setting used: /home/poncho/Tmp/gc-test
[Python/PythonEmbed.cpp:101 pythonInstalled] Binary found
[Python/PythonEmbed.cpp:116 pythonInstalled] Running: /home/poncho/Tmp/gc-test/bin/python3.10
[Python/PythonEmbed.cpp:174 pythonInstalled] Python output doesn't parse: Python 3.10.10

I've tried to add more debug output to see why python -c "import sys ...." is failing. But so far didn't succeed.

[amtriathlon]

Likely an error is printed to stderr and nothing to stdout on import error.

[liversedge]

Might be worth adding a read of stderr before terminating on timeout using readAllStandardError()

[MeisterP]

I've added py.setProcessChannelMode(QProcess::ForwardedErrorChannel); and got the following output:

[Python/PythonEmbed.cpp:220 PythonEmbed] PYTHONHOME setting used: /home/poncho/Tmp/gc-test
[Python/PythonEmbed.cpp:101 pythonInstalled] Binary found
[Python/PythonEmbed.cpp:116 pythonInstalled] Running: /home/poncho/Tmp/gc-test/bin/python3.10
Python path configuration:
  PYTHONHOME = '/home/poncho/Tmp/gc-test'
  PYTHONPATH = (not set)
  program name = '/home/poncho/Tmp/gc-test/bin/python3.10'
  isolated = 0
  environment = 1
  user site = 1
  import site = 1
  sys._base_executable = '/home/poncho/Tmp/gc-test/bin/python3.10'
  sys.base_prefix = '/home/poncho/Tmp/gc-test'
  sys.base_exec_prefix = '/home/poncho/Tmp/gc-test'
  sys.platlibdir = 'lib'
  sys.executable = '/home/poncho/Tmp/gc-test/bin/python3.10'
  sys.prefix = '/home/poncho/Tmp/gc-test'
  sys.exec_prefix = '/home/poncho/Tmp/gc-test'
  sys.path = [
    '/home/poncho/Tmp/gc-test/lib/python310.zip',
    '/home/poncho/Tmp/gc-test/lib/python3.10',
    '/home/poncho/Tmp/gc-test/lib/python3.10/lib-dynload',
  ]
Fatal Python error: init_fs_encoding: failed to get the Python codec of the filesystem encoding
Python runtime state: core initialized
ModuleNotFoundError: No module named 'encodings'

Current thread 0x00007ff6ee7ae740 (most recent call first):
  <no Python frame>
Didn't get output: /home/poncho/Tmp/gc-test/bin/python3.10
[Python/PythonEmbed.cpp:317 PythonEmbed] Embedding failed

[MeisterP]

Python venv needs PYTHONHOME unset to work properly. I actually got to the point where the embedded python is loaded with PyInitializeEx() by adding:

+    QProcessEnvironment env = QProcessEnvironment::systemEnvironment();
+    env.remove( QStringLiteral("PYTHONHOME") );
+    py.setProcessEnvironment(env);

But then I hit all the roadblocks related to embedded python.

from https://pl.python.org/docs/api/embedding.html
An application that requires total control has to provide its own implementation of Py_GetPath() Py_GetPrefix() Py_GetExecPrefix() and Py_GetProgramFullPath() (all defined in Modules/getpath.c).

[MeisterP]

To answer my own question: python venv is currently not supported in GoldenCheetah

[liversedge]

It looks like the venv embedding API is broken
https://bugs.python.org/issue22213

There is also a fundamental disagreement at play with the Py developers; embedding app should be in control and setup Python vs the embedding app should operate in the same manner as running python from the command line.

Obvs here at GC we are for the latter, but some developers are for the former.

[amtriathlon]

IIRC from v3.5 tests, virtualenv worked with GC, and so did Conda environments.

[liversedge]

I tried using venv with the latest from master and had the same problem as Poncho

[amtriathlon]

I think virtualenv should work (https://pypi.org/project/virtualenv/), I am not able to test now, but the last article you linked provides a rationale for the difference.

[MeisterP]

Here, it doesn't make a difference whether I use the internal venv or the external virtualenv module.

A comment in the python bug mentions:

Since virtualenv 20.0.0 (or there about) switched to the python -m venv style virtual environment structure...

[amtriathlon]

Not strictly related to venv but, if what you are looking for is to use a dedicated Python installation for GoldenCheetah embedding, an alternative is to use an AppImage distribution of Python (https://github.com/niess/python-appimage) like we do for our Linux builds.

[MeisterP]

My distribution switched to python 3.11 which unfortunately breaks GoldenCheetah. (see #4362). So yeah, I'm desperately trying to keep GoldenCheetah including python support running on my system. I will look into the python appimages, thanks for the hint.