Automate release and publishing logic for python

What does this MR do and why?

This MR introduces the following enhancements and automations for managing the Python package and deployment pipeline using a Makefile and GitLab CI/CD:

Makefile Enhancements:

• Package Building:
  • The build target is responsible for building the Python package, copying assets like config, LICENSE, and CONTRIBUTING.md, and cleaning up post-build.
• Asset Management:
  • The copy-assets target handles copying configuration files and assets to the package directory.
• Cleaning:
  • A clean target is provided to clean up the copied assets from the package directory.

CI/CD Pipeline Enhancements:

• Shared Environments:
  • .shared-build-env: Prepares the build environment by installing necessary packages like jq and fetching release notes and commit information from GitLab’s API. The script gathers versioning information and writes it to an environment file (variables.env). The same script will be used for building the Ruby gem.
• Python Release Process:
  • Environment Setup:
    • .install-poetry: Installs poetry for managing Python dependencies and sets up the virtual environment. It caches the environment to improve CI speed.
  • Build and Package:
    • build:python:env: Parses the Python package version from pyproject.toml and stores it in the environment.
    • build:python:package: Builds the Python package using make build. (it copies config/** files, LICENSE and CONTRIBUTING.md file to be included in the package)
  • Smoke Testing:
    • build:python:package:smoke-test: Installs the built package and performs a basic import test to ensure that the package was built correctly.
  • Release Creation:
    • release:python:creation: Creates a release in GitLab based on the gathered release notes and commit information (latest Configuration change - latest commit sha and commit date)
  • Publishing to PyPi:
    • release:python:pypi:publication: Publishes the built Python package to Gitlab PyPi package repository using twine if the package artifacts are present.

Pipeline Stages:

• Test Stage:
  • Linting, formatting, and testing steps are included, ensuring code quality before building the package.
• Deploy Stage:
  • Builds the package, tests it, and prepares for release.
• Release Stage:
  • Creates a GitLab release and publishes the package to PyPi.

NOTE: The Python deploy and release jobs are only triggered if the version has changed in pyproject.toml, and the change is merged to the main branch.

---

Testing locally

To install the package from GitLab's PyPI package registry, you'll need to configure poetry to use the GitLab registry.

1. Create a Personal Access Token:

Navigate to GitLab > User Settings > Access Tokens. Generate a new token with the necessary scopes (:read_api).

2. Configure Poetry

Edit your pyproject.toml file to include the gitlab_cloud_connector package and configure the source repository.

[tool.poetry.dependencies]
python = "^3.9"
gitlab_cloud_connector = { version = "==0.1.1", source = "gitlab-cloud-connector" }

[[tool.poetry.source]]
name = "gitlab-cloud-connector"
url = "https://gitlab.com/api/v4/projects/58733651/packages/pypi/simple"
default = false

2. Set Environment Variables:

Set the following environment variables, adjusting for your source name and credentials.

export POETRY_HTTP_BASIC_GITLAB_CLOUD_CONNECTOR_USERNAME="your_username_or_deploy_token_username"
export POETRY_HTTP_BASIC_GITLAB_CLOUD_CONNECTOR_PASSWORD="your_personal_access_token_or_deploy_token"

Source Name in Environment Variables: Convert the source name to uppercase and replace hyphens - with underscores _.

Note Make sure that credentials aren't stored in version-controlled files. If project is deployed to Runway, you can add these secrets to the Secrets Management Vault

3. Install the package using poetry:

    $ poetry install

4. Check Installed Version:

    $ poetry show gitlab-cloud-connector

5. Include gitlab_cloud_connector library in your code

    poetry run python

    import gitlab_cloud_connector

    print(gitlab_cloud_connector.__version__)

Run test

make test

Build a package

make build