Potential n+1 query for DAST site profile with policies
The following discussion from !55704 (merged) should be addressed:
-
@aturinske started a discussion: (+13 comments) @djadmin could you give this a review for direction before I start writing tests/clean this up?
Issue
The following errors were encountered on DAST configuration page while testing !55704 (merged). Please note that
- The feature flag
security_orchestration_policies_configuration
was enabled - Steps mentioned at https://gitlab.com/-/snippets/2088929 were followed prior testing
- Project had
32 site profiles
and10 scanner profiles
in the account (local gdk)
Error details
Internal server error: GitalyClient#call called 31 times from single request. Potential n+1?↵The following call site called into Gitaly 10 times:↵lib/gitlab/gitaly_client.rb:171:in `execute'↵lib/gitlab/gitaly_client/call.rb:18:in `block in call'↵lib/gitlab/gitaly_client/call.rb:55:in `recording_request'↵lib/gitlab/gitaly_client/call.rb:17:in `call'↵lib/gitlab/gitaly_client.rb:167:in `call'↵lib/gitlab/gitaly_client/commit_service.rb:19:in `ls_files'↵lib/gitlab/git/repository.rb:717:in `ls_files'↵app/models/repository.rb:983:in `ls_files'↵ee/app/models/security/orchestration_policy_configuration.rb:28:in `active_policies'↵ee/app/models/security/orchestration_policy_configuration.rb:55:in `block in active_policy_names_with_dast_profiles'↵lib/gitlab/utils/strong_memoize.rb:30:in `strong_memoize'↵ee/app/models/security/orchestration_policy_configuration.rb:52:in `active_policy_names_with_dast_profiles'↵ee/app/models/security/orchestration_policy_configuration.rb:42:in `active_policy_names_with_dast_site_profile'↵ee/app/models/dast_site_profile.rb:27:in `referenced_in_security_policies'↵lib/gitlab/graphql/present/field_extension.rb:17:in `resolve'↵lib/gitlab/graphql/calls_gitaly/field_extension.rb:15:in `resolve'↵lib/gitlab/graphql/generic_tracing.rb:48:in `with_labkit_tracing'↵lib/gitlab/graphql/generic_tracing.rb:30:in `platform_trace'↵lib/gitlab/graphql/generic_tracing.rb:48:in `with_labkit_tracing'↵lib/gitlab/graphql/generic_tracing.rb:30:in `platform_trace'↵app/graphql/gitlab_schema.rb:41:in `multiplex'↵app/controllers/graphql_controller.rb:85:in `execute_multiplex'↵app/controllers/graphql_controller.rb:36:in `execute'↵ee/lib/gitlab/ip_address_state.rb:10:in `with'↵ee/app/controllers/ee/application_controller.rb:44:in `set_current_ip_address'↵app/controllers/application_controller.rb:486:in `set_current_admin'↵lib/gitlab/session.rb:11:in `with_session'↵app/controllers/application_controller.rb:477:in `set_session_storage'↵app/controllers/application_controller.rb:471:in `set_locale'↵app/controllers/application_controller.rb:464:in `block in set_current_context'↵lib/gitlab/application_context.rb:60:in `block in use'↵lib/gitlab/application_context.rb:60:in `use'↵lib/gitlab/application_context.rb:22:in `with_context'↵app/controllers/application_controller.rb:455:in `set_current_context'↵lib/gitlab/jira/middleware.rb:19:in `call'\nlib/gitlab/gitaly_client.rb:171:in `execute'↵lib/gitlab/gitaly_client/call.rb:18:in `block in call'↵lib/gitlab/gitaly_client/call.rb:55:in `recording_request'↵lib/gitlab/gitaly_client/call.rb:17:in `call'↵lib/gitlab/gitaly_client.rb:167:in `call'↵lib/gitlab/gitaly_client/commit_service.rb:94:in `tree_entry'↵lib/gitlab/git/blob.rb:66:in `tree_entry'↵lib/gitlab/git/rugged_impl/blob.rb:21:in `tree_entry'↵lib/gitlab/git/blob.rb:49:in `find'↵lib/gitlab/git/repository.rb:828:in `blob_at'↵app/models/repository.rb:501:in `blob_at'↵app/models/repository.rb:1090:in `blob_data_at'↵ee/app/models/security/orchestration_policy_configuration.rb:71:in `policy_at'↵ee/app/models/security/orchestration_policy_configuration.rb:30:in `block in active_policies'↵ee/app/models/security/orchestration_policy_configuration.rb:30:in `map'↵ee/app/models/security/orchestration_policy_configuration.rb:30:in `active_policies'↵ee/app/models/security/orchestration_policy_configuration.rb:55:in `block in active_policy_names_with_dast_profiles'↵lib/gitlab/utils/strong_memoize.rb:30:in `strong_memoize'↵ee/app/models/security/orchestration_policy_configuration.rb:52:in `active_policy_names_with_dast_profiles'↵ee/app/models/security/orchestration_policy_configuration.rb:42:in `active_policy_names_with_dast_site_profile'↵ee/app/models/dast_site_profile.rb:27:in `referenced_in_security_policies'↵lib/gitlab/graphql/present/field_extension.rb:17:in `resolve'↵lib/gitlab/graphql/calls_gitaly/field_extension.rb:15:in `resolve'↵lib/gitlab/graphql/generic_tracing.rb:48:in `with_labkit_tracing'↵lib/gitlab/graphql/generic_tracing.rb:30:in `platform_trace'↵lib/gitlab/graphql/generic_tracing.rb:48:in `with_labkit_tracing'↵lib/gitlab/graphql/generic_tracing.rb:30:in `platform_trace'↵app/graphql/gitlab_schema.rb:41:in `multiplex'↵app/controllers/graphql_controller.rb:85:in `execute_multiplex'↵app/controllers/graphql_controller.rb:36:in `execute'↵ee/lib/gitlab/ip_address_state.rb:10:in `with'↵ee/app/controllers/ee/application_controller.rb:44:in `set_current_ip_address'↵app/controllers/application_controller.rb:486:in `set_current_admin'↵lib/gitlab/session.rb:11:in `with_session'↵app/controllers/application_controller.rb:477:in `set_session_storage'↵app/controllers/application_controller.rb:471:in `set_locale'↵app/controllers/application_controller.rb:464:in `block in set_current_context'↵lib/gitlab/application_context.rb:60:in `block in use'↵lib/gitlab/application_context.rb:60:in `use'↵lib/gitlab/application_context.rb:22:in `with_context'↵app/controllers/application_controller.rb:455:in `set_current_context'↵lib/gitlab/jira/middleware.rb:19:in `call'↵"
Implementation plan
-
backend add GraphQL aggregate to reduce gitaly calls to only 1 per project