Git clone over HTTPS on Geo secondary triggers attempted project statistics write to database

Summary

A Git clone over HTTPS from a Geo secondary kicks off a Sidekiq ProjectDailyStatisticsWorker which attempts to write to the read-only Geo secondary database.

This was reported (Zendesk, internal use only) by a 13,000-seat premium customer. They discovered this while investigating sproadic replication lag to a busy Geo secondary.

Steps to reproduce

1. Do a Git clone over HTTPS from a Geo secondary
2. Watch the PostgreSQL and Sidekiq logs

Example Project

N/A

What is the current bug behavior?

A Git clone over HTTPS kicks off a Sidekiq worker which attempts to write to the read-only Geo secondary database.

What is the expected correct behavior?

A Git clone over HTTPS kick off a Sidekiq worker which attempts to write to the primary Geo secondary database. Or perhaps not try at all?

Relevant logs and/or screenshots

sc-git-mirror-builders-75 sidekiq: {"severity":"WARN","time":"2020-01-08T17:59:43.190Z","error_class":"ActiveRecord::StatementInvalid","error_message":"PG::ReadOnlySqlTransaction: ERROR: cannot execute INSERT in a read-only transaction\n: INSERT INTO project_daily_statistics (project_id, date, fetch_count)\nVALUES (1797, '2020-01-08', 1)\nON CONFLICT (project_id, date) DO UPDATE SET fetch_count = project_daily_statistics.fetch_count + 1\n","context":"Job raised exception","jobstr":"{\"class\":\"ProjectDailyStatisticsWorker\",\"args\":[1797],\"retry\":3,\"queue\":\"project_daily_statistics\",\"jid\":\"960ba819cb33233b08a573ea\",\"created_at\":1578506383.181156,\"correlation_id\":\"8bdd5f43-c0a7-46be-aeac-3de479779865\",\"enqueued_at\":1578506383.1820388}","class":"ProjectDailyStatisticsWorker","args":[1797],"retry":3,"queue":"project_daily_statistics","jid":"960ba819cb33233b08a573ea","created_at":1578506383.181156,"correlation_id":"8bdd5f43-c0a7-46be-aeac-3de479779865","enqueued_at":1578506383.1820388,"error_backtrace":["app/services/projects/fetch_statistics_increment_service.rb:18:in `execute'","app/workers/project_daily_statistics_worker.rb:13:in `perform'","lib/gitlab/sidekiq_daemon/monitor.rb:49:in `within_job'"]}

2020-01-06_21:34:35.71636 sc-git-mirror-builders-75 postgresql: ERROR: cannot execute INSERT in a read-only transaction

2020-01-06_21:34:35.71637 sc-git-mirror-builders-75 postgresql: STATEMENT: INSERT INTO project_daily_statistics (project_id, date, fetch_count)

2020-01-06_21:34:35.71637 sc-git-mirror-builders-75 postgresql: VALUES (15247, '2020-01-06', 1)

2020-01-06_21:34:35.71637 sc-git-mirror-builders-75 postgresql: ON CONFLICT (project_id, date) DO UPDATE SET fetch_count = project_daily_statistics.fetch_count + 1

Output of checks

This bug was reported on GitLab v12.5.5-ee.

Possible fixes

The customer identified this as the source of the problem:

https://gitlab.com/gitlab-org/gitlab/blob/v12.5.5-ee/app/controllers/projects/git_http_controller.rb#L77-82

And they resolved it for themselves by adding a guard clause:

return if Gitlab::Geo.secondary?

Consider return if Gitlab::Database.read_only?. We don't have to override it in EE, and it should work with a future maintenance mode.