Skip to content

Artifacts upload with v13 and split storage

Summary

So I just migrated gitlab from v12 to v13 (and therefore updated the chart too), the migration went reasonably well (although I ran into some issue with the migration script but that's for a different issue).

The issue at hand here is that the jobs' logs cannot be accessed from within gitlab. Checking the logs I can see the following (below), which should be on the cloud storage but it's not.

Errno::ENOENT (No such file or directory @ rb_sysopen - /srv/gitlab/shared/artifacts/b1/7e/b17ef6d19c7a5b1ee83b907c595526dcb1eb06db8227d650d5dda0a9f4ce8cd9/2020_08_31/49737/33152/job.log):

After looking at minio I can see that no artifacts have been created since the migration (yesterday) although plenty of jobs have succeeded.

Up to now (in v12) I have used a split storage configuration where the critical data (uploads and backups) are stored in OVH cloud storage (through S3 compatibility) and the things i don't care too much about are stored in minIO. Did the behaviour of this change with v13?

Configuration used

global:
  edition: ce
  ingress:
    configureCertmanager: false
    tls:
      secretName: [...]
    class: "nginx"
  hosts:
    domain: [...]
    externalIP: [...]
    gitlab:
      name: [...]
      https: true
    registry:
      name: [...]
      https: true
    minio:
      name: [...]
      https: true
  minio:
    enabled: true
  appConfig:
    uploads:
      bucket: gitlab-uploads
      proxy_download: false
      connection:
        secret: gitlab-rails-storage
        key: connection
    lfs:
      bucket: gitlab-lfs
      proxy_download: false
      connection:
        secret: gitlab-rails-storage
        key: connection
    backups:
      bucket: gitlab-backups

[...]

Current behavior

The current behaviour is that the artifacts aren't uploaded to minio.

Expected behavior

Following the previous behaviour I would have excepted the artifacts to be pushed to minIO but it doesn't look to be the case. My chart configuration doesn't explicitly state to use minio for the artifacts but the old behaviour was to default to minio if not explicitly defined. Did this behaviour change by any chance?

Note: I doubt only artifacts is impacted but it's the only one I can reproduce and share at the moment so I'd rather be clear that it's where i'm encountering the issue. (uploads and lfs are not uploaded to minio and the registry which uses minio is working fine)

Versions

  • Chart: 4.3.2
  • Platform:
    • Self-hosted: bare metal
  • Kubernetes: (kubectl version)
    • Client: v1.18.8
    • Server: v1.18.8
  • Helm: (helm version)
    • Client: v3.3.0

EDIT:

I just checked gitlab.yml and it looks to be correct? With artifacts enabled and no overrides it should use the config from object_store is that correct?

production: &base
  gitlab:
    host: [...]
    https: true
    max_request_duration_seconds: 57
    impersonation_enabled: 
    usage_ping_enabled: true
    seat_link_enabled: true
    default_can_create_group: true
    username_changing_enabled: true
    issue_closing_pattern: 
    default_theme: 
    default_projects_features:
      issues: true
      merge_requests: true
      wiki: true
      snippets: true
      builds: true
      container_registry: true
    webhook_timeout: 
    trusted_proxies:
    time_zone: "UTC"        
    email_from: [...]
    email_display_name: [...]
    email_reply_to: [...]
    email_subject_suffix: ""
  

  # Consolidated object storage configuration
  ## property local configuration will override object_store
  object_store:
    enabled: true
    direct_upload: true
    background_upload: false
    proxy_download: true
    connection:
      provider: AWS
      region: us-east-1
      host: minio.[...]
      endpoint: http://gitlab-minio-svc:9000
      path_style: true
      aws_access_key_id: [...]
      aws_secret_access_key: [...]
    objects:
      artifacts:
        bucket: gitlab-artifacts
      lfs:
        bucket: gitlab-lfs
      uploads:
        bucket: gitlab-uploads
      packages:
        bucket: gitlab-packages
      external_diffs:
        bucket: gitlab-mr-diffs
      terraform_state:
        bucket: gitlab-terraform-state
      dependency_proxy:
        bucket: gitlab-dependency-proxy
  # Individual object storage backed feature properties configuration
  artifacts:
    enabled: true
  lfs:
    enabled: true
    object_store:
      enabled: true
      remote_directory: gitlab-lfs
      direct_upload: true
      background_upload: false
      proxy_download: false
      connection: [...]
  uploads:
    enabled: true
    object_store:
      enabled: true
      remote_directory: gitlab-uploads
      direct_upload: true
      background_upload: false
      proxy_download: false
      connection: [...]
Edited by Simon Jouet