Maven Virtual Registry: Database models
🔭 Issues plan
-
Maven Virtual Registry: Database models (#467972 - closed).
👈 You're here. - Maven Virtual Registry: Permissions policy (#467977 - closed).
- Maven Virtual Registry: Registry models API (#467979 - closed).
- Maven Virtual Registry: Maven API endpoint (#467982 - closed).
- Maven Virtual Registry: Cache logic (#467983 - closed).
- Maven Registry: Cached responses destruction (#468113 - closed).
- Improve workhorse dependencyproxy logic (#461561 - closed).
- Maven Virtual Registry: Documentation (#468115).
- Maven Virtual Registry: Performance review (#468116).
- Maven Virtual Registry: feature flag cleanup (#468117).
🗒 Description
This issue will deal with all the new database tables and models that we need for the Maven Virtual Registry. Being the first package format to be implemented, we have a few tables to introduce:
classDiagram
class R["VirtualRegistries::Parent"]
R : Pkey id
R : Fkey group_id
class Reg["VirtualRegistries::Packages::Maven::Registry"]
Reg : Pkey id
Reg : Fkey parent_id
Reg : Integer cache_validity_period_hours
class RegU["VirtualRegistries::Packages::Maven::RegistryUpstream"]
RegU: Fkey registry_id
RegU: Fkey upstream_id
class U["VirtualRegistries::Packages::Maven::Upstream"]
U : Pkey id
U : String url
U : EncryptedJson credentials
class CR["VirtualRegistries::Packages::Maven::CachedResponse"]
CR : Pkey id
CR : Fkey upstream_id
CR : String relative_path
CR : object storage related fields
CR : String upstream_etag
CR : Timestamp upstream_etag_checked_at
CR : String content_type
CR : SmallInt status
CR : Timestamp last_downloaded_at
CR : Integer downloaded_count
R "1" --> "1" Reg
Reg "1" --> "1" RegU
RegU "1" --> "1" U
U "1" --> "0..*" CR
Details
VirtualRegistries::Parent
: holds the reference of the parent object. In this case and first iteration, a root Group
:
-
id
: Primary key as usual. -
group_id
: Foreign key on a Group. Model validation: this must be a rootGroup
. - Notes:
- Not very useful in the first iteration but required so that the entire feature can be "anchored" at a different existing level (such as
Project
orOrganization
).
- Not very useful in the first iteration but required so that the entire feature can be "anchored" at a different existing level (such as
VirtualRegistries::Packages::Maven::Registry
: the registry object for the Maven package format:
-
id
: Primary key as usual. -
parent_id
: Foreign key on aVirtualRegistries::Parent
. Restriction for the first iteration: unique . -
cache_validity_period_hours
: Integer. Period (in hours) that the backend should consider cache objects as valid (and thus, not check them against the related upstream). - Model validations:
- Uniqueness on
parent_id
(first iteration only: only one maven registry per parent).
- Uniqueness on
VirtualRegistries::Packages::Maven::RegistryUpstream
: the bridge between a Maven Registry and a Maven upstream.
-
registry_id
: Foreign key on aVirtualRegistries::Packages::Maven::Registry
. -
upstream_id
: Foreign key on aVirtualRegistries::Packages::Maven::Upstream
. - Model validations:
- Uniqueness on
upstream_id
(first iteration only: only one maven upstream per registry).
- Uniqueness on
- Notes:
- Not very useful in the first iteration but required when we go multiple upstreams + shareable upstreams between registries.
VirtualRegistries::Packages::Maven::Upstream
: holds the data on how to contact the upstream, mainly the url and credentials if any to use.
-
id
: Primary key as usual. -
url
: String. Url format. Required. -
credentials
: Encrypted field storing a json structure. The model should expose the fields that are available:username
,password
. - Model validations:
-
username
andpassword
need both to be set or empty. -
url
is required and should be validated to have an URL like format.
-
VirtualRegistries::Packages::Maven::CachedResponse
: holds a response from an upstream. Main fields are the relative path and the columns dedicated to the object storage file.
-
id
: Primary key as usual. -
upstream_id
: Foreign key on aVirtualRegistries::Packages::Maven::Upstream
. -
relative_path
: Text. Required. -
file
: text. Object storage related field. Required. -
file_store
integer. Object storage related field. Required. -
size
: integer. Object storage related field. Required. -
object_storage_key
: Object storage related field. Required. -
upstream_etag
: text. The etag sent by the upstream (could be blank). -
upstream_checked_at
: timestamp. Last time that we checked upstream. Required. -
content_type
: Text. Content type value as sent by the upstream. Required. -
downloaded_at
: Timestamp. Last time that this cached response was downloaded. -
downloads_count
: (positive) Integer. Amount of times that this cached response was downloaded. - Model validations:
- We have a few fields required (see above).
- Uniqueness on
[upstream_id, relative_path]
(only one relative_path per upstream)
Notes:
-
VirtualRegistries
is a new bounded context under categoryvirtual_registry
. - We have some tables that are not useful but these indirections will be required in follow ups.
⏫ Update
During the implementation, we discovered this requirement:
All newly created cell-local tables are required to have a sharding_key defined in the corresponding db/docs/ file for that table.
and this challenged the existence of the Parent
table. Thus, we are not implementing it for the time being:
classDiagram
class Reg["VirtualRegistries::Packages::Maven::Registry"]
Reg : Pkey id
Reg : Fkey parent_id
Reg : Integer cache_validity_period_hours
Reg : Fkey group_id
class RegU["VirtualRegistries::Packages::Maven::RegistryUpstream"]
RegU: Fkey registry_id
RegU: Fkey upstream_id
RegU: Fkey group_id
class U["VirtualRegistries::Packages::Maven::Upstream"]
U : Pkey id
U : String url
U : EncryptedJson credentials
U : Fkey group_id
class CR["VirtualRegistries::Packages::Maven::CachedResponse"]
CR : Pkey id
CR : Fkey upstream_id
CR : String relative_path
CR : object storage related fields
CR : String upstream_etag
CR : Timestamp upstream_etag_checked_at
CR : String content_type
CR : SmallInt status
CR : Timestamp last_downloaded_at
CR : Integer downloaded_count
CR : Fkey group_id
Reg "1" --> "1" RegU
RegU "1" --> "1" U
U "1" --> "0..*" CR
Same constraints apply.