Skip to content

Force mise to compile Python

Stan Hu requested to merge sh-mise-force-compile-python into main

What does this merge request do and why?

Currently poetry.toml configures virtual environments with always-copy = true, but this doesn't work with the precompiled Python packages installed by mise because the python3 binary is symlinked to a shared library with a relative path.

To fix this just compile Python from scratch to ensure the binary is symlinked to an absolute path.

Alternatively we could:

  1. Drop the always-copy setting.
  2. Fix Poetry so that it also copies over the shared library.

Relates to https://github.com/mise-plugins/mise-poetry/issues/16

How to set up and validate locally

  1. Make sure you have the latest mise to ensure https://github.com/jdx/mise/issues/2480 is present.
  2. Ensure that mise install has installed all dependencies.
  3. Wipe .venv via rm -rf .venv
  4. Run poetry env use 3.11.10
  5. You should see something like:
Creating virtualenv ai-gateway in /Users/stanhu/gdk-ee/gitlab-ai-gateway/.venv

Command ['/Users/stanhu/gdk-ee/gitlab-ai-gateway/.venv/bin/python', '-I', '-W', 'ignore', '-c', 'import sys\n\nif hasattr(sys, "real_prefix"):\n    print(sys.real_prefix)\nelif hasattr(sys, "base_prefix"):\n    print(sys.base_prefix)\nelse:\n    print(sys.prefix)\n'] errored with the following return code -6

Error output:
dyld[77480]: Library not loaded: @executable_path/../lib/libpython3.11.dylib
  Referenced from: <4C4C4450-5555-3144-A178-FB8F18DA55E1> /Users/stanhu/gdk-ee/gitlab-ai-gateway/.venv/bin/python
  Reason: tried: '/Users/stanhu/gdk-ee/gitlab-ai-gateway/.venv/lib/libpython3.11.dylib' (no such file)
  1. This is because of this relative path. In macOS, @executable_path is a special path:
% otool -L /Users/stanhu/.local/share/mise/installs/python/3.11.10/bin/python
/Users/stanhu/.local/share/mise/installs/python/3.11.10/bin/python:
        /System/Library/Frameworks/CoreFoundation.framework/Versions/A/CoreFoundation (compatibility version 150.0.0, current version 1858.112.0)
        @executable_path/../lib/libpython3.11.dylib (compatibility version 3.11.0, current version 3.11.0)
        /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1311.100.3)
  1. You may need to mise uninstall python 3.11.10 before running mise install to see the absolute paths:
%  otool -L /Users/stanhu/.local/share/mise/installs/python/3.11.10/bin/python
/Users/stanhu/.local/share/mise/installs/python/3.11.10/bin/python:
	/System/Library/Frameworks/CoreFoundation.framework/Versions/A/CoreFoundation (compatibility version 150.0.0, current version 2420.0.0)
	/Users/stanhu/.local/share/mise/installs/python/3.11.10/lib/libpython3.11.dylib (compatibility version 3.11.0, current version 3.11.0)
	/opt/homebrew/opt/gettext/lib/libintl.8.dylib (compatibility version 13.0.0, current version 13.0.0)
	/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1345.100.2)

Merge request checklist

  • Tests added for new functionality. If not, please raise an issue to follow up.
  • Documentation added/updated, if needed.
Edited by Stan Hu

Merge request reports

Loading