Discuss options to implement instance level integration inheritance
Discuss options to solve #199388 (closed)
As part of &2430 we are going to introduce a new column inherit on the services table to track what are the project services that are going to inherit settings from instance level services.
Option A (Propagation)
When we save or update an instance-level service, we have to propagate the change to the project services that are marked as inherit. We can implement this with an asynchronous worker that runs when the instance level is created or updated.
We have something similar for service templates that can create a project service if it doesn't exist. (This functionality is currently not working for services using external tables to store settings. For example jira_tracker_data, fix !29805 (merged))
Option B (Override getters)
MR: !29044 (closed)
When we retrieve any field from a project service, we need to find the value with the right precedence. If the project service is inherit = true we need to read the field from the instance level service. Otherwise, we will pick from the project level.
def api_url
inherit? instance_level.api_url : read_attribute(:api_url)
endOur idea is to use meta-programming for this instead of writing this method for every field.
But we are not sure if this going to work with data_fields
Here is a proof of concept: !29044 (diffs)
Option C (Fix find service/s)
Review the codebase to check all the places where we are using project services and try to find the right service based on the precedence described in option B.
We think this is not a valid option because it is a very complex task and error-prone, we have to find all the places in the codebase where services are retrieved, and we also note that some places are also updating the service attributes (we should not update the instance level service).
Option D (Cache table)
Performance and complexity are the main concerns with option B. It is very complex and potentially slow to query for services when not all the data is in the same record. To avoid this, we can generate a cache table. This table would hold all the service data in a way that it is easy to query. The source of truth though, is another table that holds the settings and takes care of the inheritance. This enables us to query in a performant way while we are able to keep the source data somewhere else. Generating the cache table wouldn't be very performant though.
Cache table database example:
| Projects | Groups | Services | ServicesCache | ||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| id | group_id | id | id | instance | group_id | project_id | type | url | username | password | id | project_id | type | url | username | password | |||
| 1 | NULL | 1 | 1 | TRUE | NULL | NULL | JiraService | jira.com | admin | 123 | 1 | 1 | JiraService | jira.com | admin | 123 | |||
| 2 | NULL | 2 | FALSE | 2 | JiraService | selfhosted.com | NULL | 456 | 2 | 2 | JiraService | selfhosted.com | admin | 456 | |||||
| 3 | NULL | 3 | FALSE | 4 | JiraService | NULL | test | 321 | 3 | 3 | JiraService | jira.com | admin | 123 | |||||
| 4 | NULL | 4 | FALSE | 1 | NULL | JIraService | NULL | group_1 | sdfsdf | 4 | 4 | JiraService | jira.com | admin | 123 | ||||
| 5 | NULL | 5 | 5 | JiraService | jira.com | test | 321 | ||||||||||||
| 6 | 1 | 6 | 6 | JiraService | jira.com | group_1 | sdfsdf | ||||||||||||
| 7 | 1 | 7 | 7 | JiraService | jira.com | group_1 | sdfsdf | ||||||||||||
| 8 | 1 | 8 | 8 | JiraService | jira.com | group_1 | sdfsdf |
Option E (JSONB fields)
Each service type requires different settings. In order to keep the database schema simple, we can use a JSONB field to store the settings dynamically. With JSONB we can also build the inheritance on the database layer. When we select the services in the right hierarchical order and union (merge) the JSON fields together, we will get the right set of settings. See this example and watch https://www.youtube.com/watch?v=gIhHFSm0uaA&feature=youtu.be.
There are some things we need to consider before we can start following this approach:
- How can we do encryption in JSON fields
- &1443 (closed) is about moving settings out of json fields into its own table. With this option we would basically revert it back to json fields.
-
#26243 is about moving away from
attr_encryptedwhich we need to do anyway if we go with json fields. - Do we want to introduce this as a new field on the services table, use the properties field or even create a whole new table?
Keep in mind that we are also planning to introduce group level services in the future adding an extra layer of complexity to any of the options