Skip to content

Resolve "Document workaround to fix too long file names for uploads"

What does this MR do and why?

This MR adds the Backup fails with file name too long error section to the Troubleshooting section on Backup and Restore documentation.

This MR takes #334374 (comment 699375957) as a reference to unblock users who try backup and restore procedures and gets this error when uploaded files have filenames too long. We have already prevented this problem using !69500 (merged) solution. However, this problem persists when trying to backup old uploads using filenames too long in their paths, references, and filesystem.

In summary, the documentation changes include the following instructions:

Truncate the filenames (a maximum of 246 characters might work) in the following places:

  • In the uploads table.
  • In the references found.
  • On the filesystem.

For the first step, an SQL query is suggested for truncating the upload records detected with long filenames to 246 characters maximum. Invalid entries for references and filesystem will need action from the user at this point. Using the grep command to check the references works after dumping the database, it's a manual solution, but it might help to recognize where those invalid entries are.

There will be a separate issue for handling the rake task. The idea is to centralize most of those steps and automate the process.

Regex implemented:

  • [^\\/:*?"<>|\r\n]+$ : To get filenames from uploads paths.
  • ^.*[\\\/] : To get paths without filenames.
  • \.(?:.(?!\.))+$: To get filename extensions

Query plan:

                                              QUERY PLAN
-------------------------------------------------------------------------------------------------------
 Seq Scan on uploads u  (cost=0.00..8.96 rows=22 width=381) (actual time=0.152..2.378 rows=65 loops=1)
   Filter: (length((regexp_match((path)::text, '[^\\/:*?"<>|\r\n]+$'::text))[1]) > 250)
 Planning Time: 0.162 ms
 Execution Time: 2.478 ms
(4 rows)

Output example when searching for long filenames:

-[ RECORD 1 ]----+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
id               | 34
path             | public/@hashed/loremipsumdolorsitametconsecteturadipiscingelitseddoeiusmodtemporincididuntutlaboreetdoloremagnaaliquaauctorelitsedvulputatemisitloremipsumdolorsitametconsecteturadipiscingelitseddoeiusmodtemporincididuntutlaboreetdoloremagnaaliquaauctorelitsedvulputatemisit.txt
current_filename | loremipsumdolorsitametconsecteturadipiscingelitseddoeiusmodtemporincididuntutlaboreetdoloremagnaaliquaauctorelitsedvulputatemisitloremipsumdolorsitametconsecteturadipiscingelitseddoeiusmodtemporincididuntutlaboreetdoloremagnaaliquaauctorelitsedvulputatemisit.txt
new_filename     | loremipsumdolorsitametconsecteturadipiscingelitseddoeiusmodtemporincididuntutlaboreetdoloremagnaaliquaauctorelitsedvulputatemisitloremipsumdolorsitametconsecteturadipiscingelitseddoeiusmodtemporincididuntutlaboreetdoloremagnaaliquaauctorelits.txt
new_path         | public/@hashed/loremipsumdolorsitametconsecteturadipiscingelitseddoeiusmodtemporincididuntutlaboreetdoloremagnaaliquaauctorelitsedvulputatemisitloremipsumdolorsitametconsecteturadipiscingelitseddoeiusmodtemporincididuntutlaboreetdoloremagnaaliquaauctorelits.txt
-[ RECORD 2 ]----+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
id               | 35
path             | public/@hashed/loremipsumdolorsitametconsecteturadipiscingelitseddoeiusmodtemporincididuntutlaboreetdoloremagnaaliquaauctorelitsedvulputatemisitloremipsumdolorsitametconsecteturadipiscingelitseddoeiusmodtemporincididuntutlaboreetdoloremagnaaliquaauctorelitsedvulputatemisit.txt
current_filename | loremipsumdolorsitametconsecteturadipiscingelitseddoeiusmodtemporincididuntutlaboreetdoloremagnaaliquaauctorelitsedvulputatemisitloremipsumdolorsitametconsecteturadipiscingelitseddoeiusmodtemporincididuntutlaboreetdoloremagnaaliquaauctorelitsedvulputatemisit.txt
new_filename     | loremipsumdolorsitametconsecteturadipiscingelitseddoeiusmodtemporincididuntutlaboreetdoloremagnaaliquaauctorelitsedvulputatemisitloremipsumdolorsitametconsecteturadipiscingelitseddoeiusmodtemporincididuntutlaboreetdoloremagnaaliquaauctorelits.txt
new_path         | public/@hashed/loremipsumdolorsitametconsecteturadipiscingelitseddoeiusmodtemporincididuntutlaboreetdoloremagnaaliquaauctorelitsedvulputatemisitloremipsumdolorsitametconsecteturadipiscingelitseddoeiusmodtemporincididuntutlaboreetdoloremagnaaliquaauctorelits.txt

Output example using the query with rollback update:

-[ RECORD 1 ]---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
id   | 34
path | public/@hashed/loremipsumdolorsitametconsecteturadipiscingelitseddoeiusmodtemporincididuntutlaboreetdoloremagnaaliquaauctorelitsedvulputatemisitloremipsumdolorsitametconsecteturadipiscingelitseddoeiusmodtemporincididuntutlaboreetdoloremagnaaliquaauctorelits.txt
-[ RECORD 2 ]---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
id   | 35
path | public/@hashed/loremipsumdolorsitametconsecteturadipiscingelitseddoeiusmodtemporincididuntutlaboreetdoloremagnaaliquaauctorelitsedvulputatemisitloremipsumdolorsitametconsecteturadipiscingelitseddoeiusmodtemporincididuntutlaboreetdoloremagnaaliquaauctorelits.txt

How to set up and validate locally

  1. Update uploads using paths with long filenames:
    Upload.all.update(path: "public/@hashed/loremipsumdolorsitametconsecteturadipiscingelitseddoeiusmodtemporincididuntutlaboreetdoloremagnaaliquaauctorelitsedvulputatemisitloremipsumdolorsitametconsecteturadipiscingelitseddoeiusmodtemporincididuntutlaboreetdoloremagnaaliquaauctorelitsedvulputatemisit.txt")
  2. Follow the steps given in the documentation created to see if uploads paths get truncated.

MR acceptance checklist

This checklist encourages us to confirm any changes have been analyzed to reduce risks in quality, performance, reliability, security, and maintainability.

Related to #354984 (closed)

Edited by Javiera Tapia

Merge request reports

Loading