Introduce new DB columns for storing PyPi metadata
What does this MR do and why?
This adds metadata columns for PyPi packages.
Reasoning behind the column length constraints
DB Column | Metadata field | Limit | Why? |
---|---|---|---|
metadata_version | Metadata-Version | 16 | This is a version number string. We use 16 chars for ci_runners.semver , let's do the same here. |
summary | Summary | 255 | According to the specification, this is a one-line description |
author_email | Author-email | 2048 | This is an array of email address strings, e.g. "Rad Batnag <rbatnag@gitlab.com>" . I erred on the side of making this bigger than what we'll need for the average use case because we get occasional errors with NPM around packages with a large contributors field in package.json
|
description | Description | 4000 | This is the contents of the README file. We used 4000 chars for packages_nuget_metadata.description ; let's do the same here. |
description_content_type | Description-Content-Type | 128 | The specification allows for only a few specific values, and the longest example value with all the options added is 41 characters long. |
keywords | Keywords | 255 | It's a comma-separated array of words |
Database Review
rake db:migrate:up
main: == [advisory_lock_connection] object_id: 228020, pg_backend_pid: 23874
main: == 20230906100001 AddMetadataColumnsToPackagesPyPiMetadata: migrating =========
main: -- transaction_open?()
main: -> 0.0000s
main: -- add_column(:packages_pypi_metadata, :metadata_version, :text, {:null=>true, :if_not_exists=>true})
main: -> 0.0785s
main: -- add_column(:packages_pypi_metadata, :summary, :text, {:null=>true, :if_not_exists=>true})
main: -> 0.0013s
main: -- add_column(:packages_pypi_metadata, :keywords, :text, {:null=>true, :if_not_exists=>true})
main: -> 0.0009s
main: -- add_column(:packages_pypi_metadata, :author_email, :text, {:null=>true, :if_not_exists=>true})
main: -> 0.0012s
main: -- add_column(:packages_pypi_metadata, :description, :text, {:null=>true, :if_not_exists=>true})
main: -> 0.0013s
main: -- add_column(:packages_pypi_metadata, :description_content_type, :text, {:null=>true, :if_not_exists=>true})
main: -> 0.0014s
main: -- transaction_open?()
main: -> 0.0000s
main: -- transaction_open?()
main: -> 0.0000s
main: -- execute("ALTER TABLE packages_pypi_metadata\nADD CONSTRAINT check_2d3ed32225\nCHECK ( char_length(metadata_version) <= 16 )\nNOT VALID;\n")
main: -> 0.0006s
main: -- execute("SET statement_timeout TO 0")
main: -> 0.0001s
main: -- execute("ALTER TABLE packages_pypi_metadata VALIDATE CONSTRAINT check_2d3ed32225;")
main: -> 0.0008s
main: -- execute("RESET statement_timeout")
main: -> 0.0003s
main: -- transaction_open?()
main: -> 0.0000s
main: -- transaction_open?()
main: -> 0.0000s
main: -- execute("ALTER TABLE packages_pypi_metadata\nADD CONSTRAINT check_76afb6d4f3\nCHECK ( char_length(summary) <= 255 )\nNOT VALID;\n")
main: -> 0.0006s
main: -- execute("ALTER TABLE packages_pypi_metadata VALIDATE CONSTRAINT check_76afb6d4f3;")
main: -> 0.0004s
main: -- transaction_open?()
main: -> 0.0000s
main: -- transaction_open?()
main: -> 0.0000s
main: -- execute("ALTER TABLE packages_pypi_metadata\nADD CONSTRAINT check_02be2c39af\nCHECK ( char_length(keywords) <= 255 )\nNOT VALID;\n")
main: -> 0.0003s
main: -- execute("ALTER TABLE packages_pypi_metadata VALIDATE CONSTRAINT check_02be2c39af;")
main: -> 0.0004s
main: -- transaction_open?()
main: -> 0.0000s
main: -- transaction_open?()
main: -> 0.0000s
main: -- execute("ALTER TABLE packages_pypi_metadata\nADD CONSTRAINT check_65d8dbbd9f\nCHECK ( char_length(author_email) <= 2048 )\nNOT VALID;\n")
main: -> 0.0003s
main: -- execute("ALTER TABLE packages_pypi_metadata VALIDATE CONSTRAINT check_65d8dbbd9f;")
main: -> 0.0003s
main: -- transaction_open?()
main: -> 0.0000s
main: -- transaction_open?()
main: -> 0.0000s
main: -- execute("ALTER TABLE packages_pypi_metadata\nADD CONSTRAINT check_80308aa9bd\nCHECK ( char_length(description) <= 4000 )\nNOT VALID;\n")
main: -> 0.0003s
main: -- execute("ALTER TABLE packages_pypi_metadata VALIDATE CONSTRAINT check_80308aa9bd;")
main: -> 0.0003s
main: -- transaction_open?()
main: -> 0.0000s
main: -- transaction_open?()
main: -> 0.0000s
main: -- execute("ALTER TABLE packages_pypi_metadata\nADD CONSTRAINT check_b1f32be96c\nCHECK ( char_length(description_content_type) <= 128 )\nNOT VALID;\n")
main: -> 0.0004s
main: -- execute("ALTER TABLE packages_pypi_metadata VALIDATE CONSTRAINT check_b1f32be96c;")
main: -> 0.0004s
main: == 20230906100001 AddMetadataColumnsToPackagesPyPiMetadata: migrated (0.1183s)
main: == [advisory_lock_connection] object_id: 228020, pg_backend_pid: 23874
ci: == [advisory_lock_connection] object_id: 228560, pg_backend_pid: 23876
ci: == 20230906100001 AddMetadataColumnsToPackagesPyPiMetadata: migrating =========
ci: -- transaction_open?()
ci: -> 0.0000s
ci: -- add_column(:packages_pypi_metadata, :metadata_version, :text, {:null=>true, :if_not_exists=>true})
ci: -> 0.0022s
ci: -- add_column(:packages_pypi_metadata, :summary, :text, {:null=>true, :if_not_exists=>true})
ci: -> 0.0011s
ci: -- add_column(:packages_pypi_metadata, :keywords, :text, {:null=>true, :if_not_exists=>true})
ci: -> 0.0009s
ci: -- add_column(:packages_pypi_metadata, :author_email, :text, {:null=>true, :if_not_exists=>true})
ci: -> 0.0010s
ci: -- add_column(:packages_pypi_metadata, :description, :text, {:null=>true, :if_not_exists=>true})
ci: -> 0.0010s
ci: -- add_column(:packages_pypi_metadata, :description_content_type, :text, {:null=>true, :if_not_exists=>true})
ci: -> 0.0015s
ci: -- transaction_open?()
ci: -> 0.0000s
ci: -- transaction_open?()
ci: -> 0.0000s
ci: -- execute("ALTER TABLE packages_pypi_metadata\nADD CONSTRAINT check_2d3ed32225\nCHECK ( char_length(metadata_version) <= 16 )\nNOT VALID;\n")
ci: -> 0.0006s
ci: -- execute("SET statement_timeout TO 0")
ci: -> 0.0001s
ci: -- execute("ALTER TABLE packages_pypi_metadata VALIDATE CONSTRAINT check_2d3ed32225;")
ci: -> 0.0035s
ci: -- execute("RESET statement_timeout")
ci: -> 0.0003s
ci: -- transaction_open?()
ci: -> 0.0000s
ci: -- transaction_open?()
ci: -> 0.0000s
ci: -- execute("ALTER TABLE packages_pypi_metadata\nADD CONSTRAINT check_76afb6d4f3\nCHECK ( char_length(summary) <= 255 )\nNOT VALID;\n")
ci: -> 0.0003s
ci: -- execute("ALTER TABLE packages_pypi_metadata VALIDATE CONSTRAINT check_76afb6d4f3;")
ci: -> 0.0003s
ci: -- transaction_open?()
ci: -> 0.0000s
ci: -- transaction_open?()
ci: -> 0.0000s
ci: -- execute("ALTER TABLE packages_pypi_metadata\nADD CONSTRAINT check_02be2c39af\nCHECK ( char_length(keywords) <= 255 )\nNOT VALID;\n")
ci: -> 0.0003s
ci: -- execute("ALTER TABLE packages_pypi_metadata VALIDATE CONSTRAINT check_02be2c39af;")
ci: -> 0.0004s
ci: -- transaction_open?()
ci: -> 0.0000s
ci: -- transaction_open?()
ci: -> 0.0000s
ci: -- execute("ALTER TABLE packages_pypi_metadata\nADD CONSTRAINT check_65d8dbbd9f\nCHECK ( char_length(author_email) <= 2048 )\nNOT VALID;\n")
ci: -> 0.0003s
ci: -- execute("ALTER TABLE packages_pypi_metadata VALIDATE CONSTRAINT check_65d8dbbd9f;")
ci: -> 0.0003s
ci: -- transaction_open?()
ci: -> 0.0000s
ci: -- transaction_open?()
ci: -> 0.0000s
ci: -- execute("ALTER TABLE packages_pypi_metadata\nADD CONSTRAINT check_80308aa9bd\nCHECK ( char_length(description) <= 4000 )\nNOT VALID;\n")
ci: -> 0.0004s
ci: -- execute("ALTER TABLE packages_pypi_metadata VALIDATE CONSTRAINT check_80308aa9bd;")
ci: -> 0.0004s
ci: -- transaction_open?()
ci: -> 0.0000s
ci: -- transaction_open?()
ci: -> 0.0000s
ci: -- execute("ALTER TABLE packages_pypi_metadata\nADD CONSTRAINT check_b1f32be96c\nCHECK ( char_length(description_content_type) <= 128 )\nNOT VALID;\n")
ci: -> 0.0005s
ci: -- execute("ALTER TABLE packages_pypi_metadata VALIDATE CONSTRAINT check_b1f32be96c;")
ci: -> 0.0003s
ci: == 20230906100001 AddMetadataColumnsToPackagesPyPiMetadata: migrated (0.0425s)
ci: == [advisory_lock_connection] object_id: 228560, pg_backend_pid: 23876
rake db:migrate:down
main: == [advisory_lock_connection] object_id: 225860, pg_backend_pid: 24289
main: == 20230906100001 AddMetadataColumnsToPackagesPyPiMetadata: reverting =========
main: -- transaction_open?()
main: -> 0.0000s
main: -- remove_column(:packages_pypi_metadata, :metadata_version, {:if_exists=>true})
main: -> 0.0710s
main: -- remove_column(:packages_pypi_metadata, :summary, {:if_exists=>true})
main: -> 0.0014s
main: -- remove_column(:packages_pypi_metadata, :keywords, {:if_exists=>true})
main: -> 0.0011s
main: -- remove_column(:packages_pypi_metadata, :author_email, {:if_exists=>true})
main: -> 0.0011s
main: -- remove_column(:packages_pypi_metadata, :description, {:if_exists=>true})
main: -> 0.0012s
main: -- remove_column(:packages_pypi_metadata, :description_content_type, {:if_exists=>true})
main: -> 0.0012s
main: == 20230906100001 AddMetadataColumnsToPackagesPyPiMetadata: reverted (0.0881s)
main: == [advisory_lock_connection] object_id: 225860, pg_backend_pid: 24289
ci: == [advisory_lock_connection] object_id: 227020, pg_backend_pid: 24677
ci: == 20230906100001 AddMetadataColumnsToPackagesPyPiMetadata: reverting =========
ci: -- transaction_open?()
ci: -> 0.0000s
ci: -- remove_column(:packages_pypi_metadata, :metadata_version, {:if_exists=>true})
ci: -> 0.0668s
ci: -- remove_column(:packages_pypi_metadata, :summary, {:if_exists=>true})
ci: -> 0.0017s
ci: -- remove_column(:packages_pypi_metadata, :keywords, {:if_exists=>true})
ci: -> 0.0010s
ci: -- remove_column(:packages_pypi_metadata, :author_email, {:if_exists=>true})
ci: -> 0.0011s
ci: -- remove_column(:packages_pypi_metadata, :description, {:if_exists=>true})
ci: -> 0.0009s
ci: -- remove_column(:packages_pypi_metadata, :description_content_type, {:if_exists=>true})
ci: -> 0.0010s
ci: == 20230906100001 AddMetadataColumnsToPackagesPyPiMetadata: reverted (0.0878s)
ci: == [advisory_lock_connection] object_id: 227020, pg_backend_pid: 24677
MR acceptance checklist
This checklist encourages us to confirm any changes have been analyzed to reduce risks in quality, performance, reliability, security, and maintainability.
-
I have evaluated the MR acceptance checklist for this MR.
Related to #423529 (closed)
Edited by Radamanthus Batnag