Rake task: Do not generate too long file names for uploads
Summary
After merging !88872 (merged) we should provide a rake task that centralizes and automates the given steps to unblock users who try to make a backup procedure. This MR provides this automation for backup and restore.
Steps to reproduce
One or more uploaded files with filenames longer than 255 characters. UNIX-based machines allow the creation of filenames with 255 bytes maximum, so this issue seems to be replicable only in machines using Windows as OS.
What is the current bug behavior?
The file name generated for a file uploaded via Gitlab UI is too long, and the backup script can't proceed with the backup as expected.
What is the expected correct behavior?
Backup and restore procedures should move forward when this error happens. In addition, the rake task should provide fixes for all the upload
records, references, and filesystem.
Relevant logs and/or screenshots
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
An unexpected error has occurred.
Please try reproducing the error using
the latest s3cmd code from the git master
branch found at:
https://github.com/s3tools/s3cmd
and have a look at the known issues list:
https://github.com/s3tools/s3cmd/wiki/Common-known-issues-and-their-solutions-(FAQ)
If the error persists, please report the
following lines (removing any private
info as necessary) to:
s3tools-bugs@lists.sourceforge.net
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
Invoked as: /usr/local/bin/s3cmd --stop-on-error --delete-removed sync s3://dkb-gitlab-gitlab-uploads-eu-central-1/ /srv/gitlab/tmp/uploads/
Problem: <class 'OSError: [Errno 36] File name too long: '/srv/gitlab/tmp/uploads/@hashed/71/ee/71ee45a3c0db9a9865f7313dd3372cf60dca6479d46261f3542eb9346e4a04d6/028acbe35bbe90de09dc21d3d4159916/.s3cmd.9lj_ex2h.tmp' -> b'/srv/gitlab/tmp/uploads/@hashed/71/ee/71ee45a3c0db9a9865f7313dd3372cf60dca6479d46261f3542eb9346e4a04d6/028acbe35bbe90de09dc21d3d4159916/url_sa_i_url_https_3A_2F_2Fwww.latintimes.com_2Fburn-notice-series-finale-spoilers-how-did-it-all-end-and-which-characters-survived-explosive-finale_psig_AOvVaw2d2_5UBy4dGKayGeDAmyhT_ust_1622228720698000_source_images_cd_vfe_ved_0CAIQjRxqFwoTCODh84vH6vACFQAAAAAdAAAAABAD'
S3cmd: 2.2.0
python: 3.8.9 (default, Apr 6 2021, 18:45:51)
[GCC 8.3.0]
environment LANG=C.UTF-8
Traceback (most recent call last):
File "/usr/local/bin/s3cmd", line 3209, in <module>
rc = main()
File "/usr/local/bin/s3cmd", line 3106, in main
rc = cmd_func(args)
File "/usr/local/bin/s3cmd", line 1967, in cmd_sync
return cmd_sync_remote2local(args)
File "/usr/local/bin/s3cmd", line 1533, in cmd_sync_remote2local
ret, seq, size_transferred = _download(remote_list, seq, remote_count + update_count, size_transferred, dir_cache)
File "/usr/local/bin/s3cmd", line 1387, in _download
os.rename(chkptfname_b, deunicodise(dst_file))
OSError: [Errno 36] File name too long: '/srv/gitlab/tmp/uploads/@hashed/71/ee/71ee45a3c0db9a9865f7313dd3372cf60dca6479d46261f3542eb9346e4a04d6/028acbe35bbe90de09dc21d3d4159916/.s3cmd.9lj_ex2h.tmp' -> b'/srv/gitlab/tmp/uploads/@hashed/71/ee/71ee45a3c0db9a9865f7313dd3372cf60dca6479d46261f3542eb9346e4a04d6/028acbe35bbe90de09dc21d3d4159916/url_sa_i_url_https_3A_2F_2Fwww.latintimes.com_2Fburn-notice-series-finale-spoilers-how-did-it-all-end-and-which-characters-survived-explosive-finale_psig_AOvVaw2d2_5UBy4dGKayGeDAmyhT_ust_1622228720698000_source_images_cd_vfe_ved_0CAIQjRxqFwoTCODh84vH6vACFQAAAAAdAAAAABAD'
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
An unexpected error has occurred.
Please try reproducing the error using
the latest s3cmd code from the git master
branch found at:
https://github.com/s3tools/s3cmd
and have a look at the known issues list:
https://github.com/s3tools/s3cmd/wiki/Common-known-issues-and-their-solutions-(FAQ)
If the error persists, please report the
above lines (removing any private
info as necessary) to:
s3tools-bugs@lists.sourceforge.net
Results of GitLab environment info
System information
System:
Proxy: GITLAB_GITLAB_PAGES_SERVICE_PORT_HTTP_GITLAB_PAGES_PROXY: 8090
Current User: git
Using RVM: no
Ruby Version: 2.7.5p203
Gem Version: 3.2.19
Bundler Version:2.2.19
Rake Version: 13.0.6
Redis Version: unknown
Git Version: unknown
Sidekiq Version:6.3.1
Go Version: unknown
GitLab information
Version: 14.7.2-ee
Revision: 39a169b2f25
Directory: /srv/gitlab
DB Adapter: PostgreSQL
DB Version: 12.4
URL: https://gitlab.dev.codefactory.sh
HTTP Clone URL: https://gitlab.dev.codefactory.sh/some-group/some-project.git
SSH Clone URL: git@gitlab.dev.codefactory.sh:some-group/some-project.git
Elasticsearch: yes
Geo: no
Using LDAP: yes
Using Omniauth: yes
Omniauth Providers: saml, saml_2
GitLab Shell
Version: 13.22.2
Repository storage paths:
- default: /var/opt/gitlab/repo
GitLab Shell path: /home/git/gitlab-shell
Git: /usr/bin/git
git@gitlab-toolbox-backup-manual-z9fu3-vnt4x:/$
Possible fixes
Manual fix documented: !88872 (merged)