Skip to content

Add MLproject

Anatoli Babenia requested to merge abitrolly/tanuki-stan:mlproject into master

This doesn't work yet, just a wishlist to share.

https://mlflow.org/docs/latest/projects.html

$ mlflow run .
2023/06/20 14:05:59 ERROR mlflow.cli: === Could not find main among entry points ['train', 'train.gpu', 'classify'] or interpret main as a runnable script. Supported script file extensions: ['.py', '.sh'] ===

$ mlflow run . -h
Usage: mlflow run [OPTIONS] URI
Try 'mlflow run --help' for help.

Error: No such option: -h

Quite a few UX things that need to be improved above. Change error message to be "'main' entrypoint is not defined. Specify entrypoint explicitly with -e <train|train.gpu|classify>".

$ mlflow run . -e train
2023/06/20 14:08:24 INFO mlflow.utils.virtualenv: Installing python 3.11.1 if it does not exist
Downloading Python-3.11.1.tar.xz...
-> https://www.python.org/ftp/python/3.11.1/Python-3.11.1.tar.xz
Installing Python-3.11.1...

This doesn't show which environment file it is going to use (should be notebooks/requirements.txt), and support for python_env: notebooks/requirements.txt is probably not there yet.

Installing Python-3.11.1...
Installed Python-3.11.1 to /workspace/.pyenv_mirror/fakeroot/versions/3.11.1
2023/06/20 14:10:43 INFO mlflow.utils.virtualenv: Creating a new environment in /home/gitpod/.mlflow/envs/mlflow-d56ed199c280db9ff9f252f944dcfc7dc54ea2f6 with /home/gitpod/.pyenv/versions/3.11.1/bin/python
/home/gitpod/.pyenv/versions/3.11.1/bin/python3: No module named virtualenv
2023/06/20 14:10:43 WARNING mlflow.utils.virtualenv: Encountered an unexpected error: ShellCommandException("Non-zero exit code: 1\nCommand: ['/home/gitpod/.pyenv/versions/3.11.1/bin/python3', '-m', 'virtualenv', '--python', '/home/gitpod/.pyenv/versions/3.11.1/bin/python', '/home/gitpod/.mlflow/envs/mlflow-d56ed199c280db9ff9f252f944dcfc7dc54ea2f6']") while creating a virtualenv environment in /home/gitpod/.mlflow/envs/mlflow-d56ed199c280db9ff9f252f944dcfc7dc54ea2f6, removing the environment directory...
Traceback (most recent call last):
  File "/workspace/.pyenv_mirror/user/3.11.1/bin/mlflow", line 8, in <module>
    sys.exit(cli())
             ^^^^^
  File "/workspace/.pyenv_mirror/user/current/lib/python3.11/site-packages/click/core.py", line 1130, in __call__
    return self.main(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/workspace/.pyenv_mirror/user/current/lib/python3.11/site-packages/click/core.py", line 1055, in main
    rv = self.invoke(ctx)
         ^^^^^^^^^^^^^^^^
  File "/workspace/.pyenv_mirror/user/current/lib/python3.11/site-packages/click/core.py", line 1657, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/workspace/.pyenv_mirror/user/current/lib/python3.11/site-packages/click/core.py", line 1404, in invoke
    return ctx.invoke(self.callback, **ctx.params)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/workspace/.pyenv_mirror/user/current/lib/python3.11/site-packages/click/core.py", line 760, in invoke
    return __callback(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/workspace/.pyenv_mirror/user/current/lib/python3.11/site-packages/mlflow/cli.py", line 204, in run
    projects.run(
  File "/workspace/.pyenv_mirror/user/current/lib/python3.11/site-packages/mlflow/projects/__init__.py", line 338, in run
    submitted_run_obj = _run(
                        ^^^^^
  File "/workspace/.pyenv_mirror/user/current/lib/python3.11/site-packages/mlflow/projects/__init__.py", line 105, in _run
    submitted_run = backend.run(
                    ^^^^^^^^^^^^
  File "/workspace/.pyenv_mirror/user/current/lib/python3.11/site-packages/mlflow/projects/backend/local.py", line 167, in run
    activate_cmd = _create_virtualenv(
                   ^^^^^^^^^^^^^^^^^^^
  File "/workspace/.pyenv_mirror/user/current/lib/python3.11/site-packages/mlflow/utils/virtualenv.py", line 251, in _create_virtualenv
    _exec_cmd(
  File "/workspace/.pyenv_mirror/user/current/lib/python3.11/site-packages/mlflow/utils/process.py", line 117, in _exec_cmd
    raise ShellCommandException.from_completed_process(comp_process)
mlflow.utils.process.ShellCommandException: Non-zero exit code: 1
Command: ['/home/gitpod/.pyenv/versions/3.11.1/bin/python3', '-m', 'virtualenv', '--python', '/home/gitpod/.pyenv/versions/3.11.1/bin/python', '/home/gitpod/.mlflow/envs/mlflow-d56ed199c280db9ff9f252f944dcfc7dc54ea2f6']

Eventually it fails.

$ mlflow version
Traceback (most recent call last):
  File "/workspace/.pyenv_mirror/user/3.11.1/bin/mlflow", line 5, in <module>
    from mlflow.cli import cli
  File "/workspace/.pyenv_mirror/user/current/lib/python3.11/site-packages/mlflow/__init__.py", line 41, in <module>
    from mlflow import projects  # pylint: disable=unused-import
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/workspace/.pyenv_mirror/user/current/lib/python3.11/site-packages/mlflow/projects/__init__.py", line 5, in <module>
    import yaml
ModuleNotFoundError: No module named 'yaml'

Might be Gitpod problem. Need to try this MR from scratch without workspace restart.

Merge request reports