Elasticsearch: Id is too long, must be no longer than 512 bytes

Summary

Some project fail during ElasticSearch indexing.

What is the current bug behavior?

See bug trace.

There is a constraint violation on the generated document Ids (See https://discuss.elastic.co/t/id-is-too-long-must-be-no-longer-than-512/80643 as well) that are too long for a hardcoded limit in ES 5.x.

On https://gitlab.com/gitlab-org/gitlab-elasticsearch-indexer/blob/master/indexer/blob.go#L58 the ID is formed with a filename which can exceed the expected length.

The same behaviour appears on https://gitlab.com/gitlab-org/gitlab-ee/blob/master/ee/lib/elasticsearch/git/repository.rb#L135

Bug trace

W, [2018-04-24T02:13:37.303124 #1]  WARN -- : /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/elasticsearch-transport-5.0.3/lib/elasticsearch/transport/transport/base.rb:201:in `__raise_transport_error': [400] 
{"error":{"root_cause":[{"type":"action_request_validation_exception","reason":"Validation Failed: 1: id is too long, must be no longer than 512 bytes but was: 557;2: id is too long, must be no longer than 512 bytes but was: 
549;3: id is too long, must be no longer than 512 bytes but was: 560;4: id is too long, must be no longer than 512 bytes but was: 572;5: id is too long, must be no longer than 512 bytes but was: 574;6: id is too long, must be no 
longer than 512 bytes but was: 572;7: id is too long, must be no longer than 512 bytes but was: 574;8: id is too long, must be no longer than 512 bytes but was: 574;9: id is too long, must be no longer than 512 bytes but was: 
576;10: id is too long, must be no longer than 512 bytes but was: 573;11: id is too long, must be no longer than 512 bytes but was: 575;12: id is too long, must be no longer than 512 bytes but was: 571;13: id is too long, must 
be no longer than 512 bytes but was: 573;14: id is too long, must be no longer than 512 bytes but was: 580;15: id is too long, must be no longer than 512 bytes but was: 582;16: id is too long, must be no longer than 512 bytes 
but was: 577;17: id is too long, must be no longer than 512 bytes but was: 579;18: id is too long, must be no longer than 512 bytes but was: 578;19: id is too long, must be no longer than 512 bytes but was: 580;20: id is too 
long, must be no longer than 512 bytes but was: 589;21: id is too long, must be no longer than 512 bytes but was: 591;22: id is too long, must be no longer than 512 bytes but was: 573;23: id is too long, must be no longer than 
512 bytes but was: 575;24: id is too long, must be no longer than 512 bytes but was: 584;25: id is too long, must be no longer than 512 bytes but was: 586;26: id is too long, must be no longer than 512 bytes but was: 586;27: id 
is too long, must be no longer than 512 bytes but was: 588;28: id is too long, must be no longer than 512 bytes but was: 594;29: id is too long, must be no longer than 512 bytes but was: 596;30: id is too long, must be no longer 
than 512 bytes but was: 587;31: id is too long, must be no longer than 512 bytes but was: 589;32: id is too long, must be no longer than 512 bytes but was: 584;33: id is too long, must be no longer than 512 bytes but was: 
586;34: id is too long, must be no longer than 512 bytes but was: 573;35: id is too long, must be no longer than 512 bytes but was: 575;36: id is too long, must be no longer than 512 bytes but was: 580;37: id is too long, must 
be no longer than 512 bytes but was: 582;38: id is too long, must be no longer than 512 bytes but was: 577;39: id is too long, must be no longer than 512 bytes but was: 579;40: id is too long, must be no longer than 512 bytes 
but was: 572;41: id is too long, must be no longer than 512 bytes but was: 574;42: id is too long, must be no longer than 512 bytes but was: 583;43: id is too long, must be no longer than 512 bytes but was: 585;44: id is too 
long, must be no longer than 512 bytes but was: 590;45: id is too long, must be no longer than 512 bytes but was: 592;46: id is too long, must be no longer than 512 bytes but was: 581;47: id is too long, must be no longer than 
512 bytes but was: 583;48: id is too long, must be no longer than 512 bytes but was: 582;49: id is too long, must be no longer than 512 bytes but was: 584;50: id is too long, must be no longer than 512 bytes but was: 593;51: id 
is too long, must be no longer than 512 bytes but was: 595;52: id is too long, must be no longer than 512 bytes but was: 587;53: id is too long, must be no longer than 512 bytes but was: 589;54: id is too long, must be no longer 
than 512 bytes but was: 580;55: id is too long, must be no longer than 512 bytes but was: 582;56: id is too long, must be no longer than 512 bytes but was: 583;57: id is too long, must be no longer than 512 bytes but was: 
585;58: id is too long, must be no longer than 512 bytes but was: 581;59: id is too long, must be no longer than 512 bytes but was: 583;60: id is too long, must be no longer than 512 bytes but was: 576;61: id is too long, must 
be no longer than 512 bytes but was: 578;62: id is too long, must be no longer than 512 bytes but was: 575;63: id is too long, must be no longer than 512 bytes but was: 577;64: id is too long, must be no longer than 512 bytes 
but was: 572;65: id is too long, must be no longer than 512 bytes but was: 574;66: id is too long, must be no longer than 512 bytes but was: 573;67: id is too long, must be no longer than 512 bytes but was: 575;68: id is too 
long, must be no longer than 512 bytes but was: 575;69: id is too long, must be no longer than 512 bytes but was: 577;70: id is too long, must be no longer than 512 bytes but was: 575;71: id is too long, must be no longer than 
512 bytes but was: 577;72: id is too long, must be no longer than 512 bytes but was: 573;73: id is too long, must be no longer than 512 bytes but was: 575;74: id is too long, must be no longer than 512 bytes but was: 580;75: id 
is too long, must be no longer than 512 bytes but was: 582;76: id is too long, must be no longer than 512 bytes but was: 580;77: id is too long, must be no longer than 512 bytes but was: 582;78: id is too long, must be no longer 
than 512 bytes but was: 579;79: id is too long, must be no longer than 512 bytes but was: 581;80: id is too long, must be no longer than 512 bytes but was: 583;81: id is too long, must be no longer than 512 bytes but was: 
585;82: id is too long, must be no longer than 512 bytes but was: 576;83: id is too long, must be no longer than 512 bytes but was: 578;84: id is too long, must be no longer than 512 bytes but was: 573;85: id is too long, must 
be no longer than 512 bytes but was: 575;86: id is too long, must be no longer than 512 bytes but was: 578;87: id is too long, must be no longer than 512 bytes but was: 580;88: id is too long, must be no longer than 512 bytes 
but was: 574;"}],"type":"action_request_validation_exception","reason":"Validation Failed: 1: id is too long, must be no longer than 512 bytes but was: 557;2: id is too long, must be no longer than 512 bytes but was: 549;3: id 
is too long, must be no longer than 512 bytes but was: 560;4: id is too long, must be no longer than 512 bytes but was: 572;5: id is too long, must be no longer than 512 bytes but was: 574;6: id is too long, must be no longer 
than 512 bytes but was: 572;7: id is too long, must be no longer than 512 bytes but was: 574;8: id is too long, must be no longer than 512 bytes but was: 574;9: id is too long, must be no longer than 512 bytes but was: 576;10: 
id is too long, must be no longer than 512 bytes but was: 573;11: id is too long, must be no longer than 512 bytes but was: 575;12: id is too long, must be no longer than 512 bytes but was: 571;13: id is too long, must be no 
longer than 512 bytes but was: 573;14: id is too long, must be no longer than 512 bytes but was: 580;15: id is too long, must be no longer than 512 bytes but was: 582;16: id is too long, must be no longer than 512 bytes but was: 
577;17: id is too long, must be no longer than 512 bytes but was: 579;18: id is too long, must be no longer than 512 bytes but was: 578;19: id is too long, must be no longer than 512 bytes but was: 580;20: id is too long, must 
be no longer than 512 bytes but was: 589;21: id is too long, must be no longer than 512 bytes but was: 591;22: id is too long, must be no longer than 512 bytes but was: 573;23: id is too long, must be no longer than 512 bytes 
but was: 575;24: id is too long, must be no longer than 512 bytes but was: 584;25: id is too long, must be no longer than 512 bytes but was: 586;26: id is too long, must be no longer than 512 bytes but was: 586;27: id is too 
long, must be no longer than 512 bytes but was: 588;28: id is too long, must be no longer than 512 bytes but was: 594;29: id is too long, must be no longer than 512 bytes but was: 596;30: id is too long, must be no longer than 
512 bytes but was: 587;31: id is too long, must be no longer than 512 bytes but was: 589;32: id is too long, must be no longer than 512 bytes but was: 584;33: id is too long, must be no longer than 512 bytes but was: 586;34: id 
is too long, must be no longer than 512 bytes but was: 573;35: id is too long, must be no longer than 512 bytes but was: 575;36: id is too long, must be no longer than 512 bytes but was: 580;37: id is too long, must be no longer 
than 512 bytes but was: 582;38: id is too long, must be no longer than 512 bytes but was: 577;39: id is too long, must be no longer than 512 bytes but was: 579;40: id is too long, must be no longer than 512 bytes but was: 
572;41: id is too long, must be no longer than 512 bytes but was: 574;42: id is too long, must be no longer than 512 bytes but was: 583;43: id is too long, must be no longer than 512 bytes but was: 585;44: id is too long, must 
be no longer than 512 bytes but was: 590;45: id is too long, must be no longer than 512 bytes but was: 592;46: id is too long, must be no longer than 512 bytes but was: 581;47: id is too long, must be no longer than 512 bytes 
but was: 583;48: id is too long, must be no longer than 512 bytes but was: 582;49: id is too long, must be no longer than 512 bytes but was: 584;50: id is too long, must be no longer than 512 bytes but was: 593;51: id is too 
long, must be no longer than 512 bytes but was: 595;52: id is too long, must be no longer than 512 bytes but was: 587;53: id is too long, must be no longer than 512 bytes but was: 589;54: id is too long, must be no longer than 
512 bytes but was: 580;55: id is too long, must be no longer than 512 bytes but was: 582;56: id is too long, must be no longer than 512 bytes but was: 583;57: id is too long, must be no longer than 512 bytes but was: 585;58: id 
is too long, must be no longer than 512 bytes but was: 581;59: id is too long, must be no longer than 512 bytes but was: 583;60: id is too long, must be no longer than 512 bytes but was: 576;61: id is too long, must be no longer 
than 512 bytes but was: 578;62: id is too long, must be no longer than 512 bytes but was: 575;63: id is too long, must be no longer than 512 bytes but was: 577;64: id is too long, must be no longer than 512 bytes but was: 
572;65: id is too long, must be no longer than 512 bytes but was: 574;66: id is too long, must be no longer than 512 bytes but was: 573;67: id is too long, must be no longer than 512 bytes but was: 575;68: id is too long, must 
be no longer than 512 bytes but was: 575;69: id is too long, must be no longer than 512 bytes but was: 577;70: id is too long, must be no longer than 512 bytes but was: 575;71: id is too long, must be no longer than 512 bytes 
but was: 577;72: id is too long, must be no longer than 512 bytes but was: 573;73: id is too long, must be no longer than 512 bytes but was: 575;74: id is too long, must be no longer than 512 bytes but was: 580;75: id is too 
long, must be no longer than 512 bytes but was: 582;76: id is too long, must be no longer than 512 bytes but was: 580;77: id is too long, must be no longer than 512 bytes but was: 582;78: id is too long, must be no longer than 
512 bytes but was: 579;79: id is too long, must be no longer than 512 bytes but was: 581;80: id is too long, must be no longer than 512 bytes but was: 583;81: id is too long, must be no longer than 512 bytes but was: 585;82: id 
is too long, must be no longer than 512 bytes but was: 576;83: id is too long, must be no longer than 512 bytes but was: 578;84: id is too long, must be no longer than 512 bytes but was: 573;85: id is too long, must be no longer 
than 512 bytes but was: 575;86: id is too long, must be no longer than 512 bytes but was: 578;87: id is too long, must be no longer than 512 bytes but was: 580;88: id is too long, must be no longer than 512 bytes but was: 
574;"},"status":400} (Elasticsearch::Transport::Transport::Errors::BadRequest)
        from /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/elasticsearch-transport-5.0.3/lib/elasticsearch/transport/transport/base.rb:318:in `perform_request'
        from /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/elasticsearch-transport-5.0.3/lib/elasticsearch/transport/transport/http/faraday.rb:20:in `perform_request'
        from /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/elasticsearch-transport-5.0.3/lib/elasticsearch/transport/client.rb:131:in `perform_request'
        from /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/elasticsearch-api-5.0.3/lib/elasticsearch/api/actions/bulk.rb:95:in `bulk'
        from /opt/gitlab/embedded/service/gitlab-rails/ee/lib/elasticsearch/git/repository.rb:112:in `perform_bulk'
        from /opt/gitlab/embedded/service/gitlab-rails/ee/lib/elasticsearch/git/repository.rb:99:in `block in index_blobs'
        from /opt/gitlab/embedded/service/gitlab-rails/ee/lib/elasticsearch/git/repository.rb:84:in `each'
        from /opt/gitlab/embedded/service/gitlab-rails/ee/lib/elasticsearch/git/repository.rb:84:in `each_slice'
        from /opt/gitlab/embedded/service/gitlab-rails/ee/lib/elasticsearch/git/repository.rb:84:in `index_blobs'
        from /opt/gitlab/embedded/service/gitlab-rails/bin/elastic_repo_indexer:91:in `block in '
        from /opt/gitlab/embedded/lib/ruby/2.3.0/benchmark.rb:293:in `measure'
        from /opt/gitlab/embedded/service/gitlab-rails/bin/elastic_repo_indexer:89:in `'
 indexing XXX / YYY (ID=18100), trace - ["/opt/gitlab/embedded/service/gitlab-rails/ee/lib/gitlab/elastic/indexer.rb:64:in `run_indexer!'", 
"/opt/gitlab/embedded/service/gitlab-rails/ee/lib/gitlab/elastic/indexer.rb:38:in `run'", "/opt/gitlab/embedded/service/gitlab-rails/ee/app/workers/elastic_batch_project_indexer_worker.rb:23:in `run_indexer'", 
"/opt/gitlab/embedded/service/gitlab-rails/ee/app/workers/elastic_batch_project_indexer_worker.rb:14:in `block in perform'", 
"/opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/activerecord-4.2.10/lib/active_record/relation/batches.rb:51:in `block (2 levels) in find_each'", 
"/opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/activerecord-4.2.10/lib/active_record/relation/batches.rb:51:in `each'", 
"/opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/activerecord-4.2.10/lib/active_record/relation/batches.rb:51:in `block in find_each'", 
"/opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/activerecord-4.2.10/lib/active_record/relation/batches.rb:124:in `find_in_batches'", 
"/opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/activerecord-4.2.10/lib/active_record/relation/batches.rb:50:in `find_each'", 
"/opt/gitlab/embedded/service/gitlab-rails/ee/app/workers/elastic_batch_project_indexer_worker.rb:14:in `perform'", "/opt/gitlab/embedded/service/gitlab-rails/ee/lib/tasks/gitlab/elastic.rake:39:in `block (4 levels) in '", "/opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/storage.rake:72:in `block in project_id_batches'", "/opt/gitlab/embedded/service/gitlab-rails/config/initializers/ar5_batching.rb:34:in `block in 
in_batches'", "/opt/gitlab/embedded/service/gitlab-rails/config/initializers/ar5_batching.rb:18:in `loop'", "/opt/gitlab/embedded/service/gitlab-rails/config/initializers/ar5_batching.rb:18:in `in_batches'", 
"/opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/storage.rake:69:in `project_id_batches'", "/opt/gitlab/embedded/service/gitlab-rails/ee/lib/tasks/gitlab/elastic.rake:38:in `block (3 levels) in '", 
"/opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/rake-12.3.0/lib/rake/task.rb:251:in `block in execute'", "/opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/rake-12.3.0/lib/rake/task.rb:251:in `each'", 
"/opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/rake-12.3.0/lib/rake/task.rb:251:in `execute'", "/opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/rake-12.3.0/lib/rake/task.rb:195:in `block in invoke_with_call_chain'", 
"/opt/gitlab/embedded/lib/ruby/2.3.0/monitor.rb:214:in `mon_synchronize'", "/opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/rake-12.3.0/lib/rake/task.rb:188:in `invoke_with_call_chain'", 
"/opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/rake-12.3.0/lib/rake/task.rb:181:in `invoke'", "/opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/rake-12.3.0/lib/rake/application.rb:160:in `invoke_task'", 
"/opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/rake-12.3.0/lib/rake/application.rb:116:in `block (2 levels) in top_level'", "/opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/rake-12.3.0/lib/rake/application.rb:116:in `each'", 
"/opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/rake-12.3.0/lib/rake/application.rb:116:in `block in top_level'", "/opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/rake-12.3.0/lib/rake/application.rb:125:in `run_with_threads'", 
"/opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/rake-12.3.0/lib/rake/application.rb:110:in `top_level'", "/opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/rake-12.3.0/lib/rake/application.rb:83:in `block in run'", 
"/opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/rake-12.3.0/lib/rake/application.rb:186:in `standard_exception_handling'", "/opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/rake-12.3.0/lib/rake/application.rb:80:in `run'", 
"/opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/rake-12.3.0/exe/rake:27:in `'", "/opt/gitlab/embedded/bin/rake:23:in `load'", "/opt/gitlab/embedded/bin/rake:23:in `'", 
"/opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/bundler-1.13.7/lib/bundler/cli/exec.rb:74:in `load'", "/opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/bundler-1.13.7/lib/bundler/cli/exec.rb:74:in `kernel_load'", 
"/opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/bundler-1.13.7/lib/bundler/cli/exec.rb:27:in `run'", "/opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/bundler-1.13.7/lib/bundler/cli.rb:332:in `exec'", 
"/opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/bundler-1.13.7/lib/bundler/vendor/thor/lib/thor/command.rb:27:in `run'", 
"/opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/bundler-1.13.7/lib/bundler/vendor/thor/lib/thor/invocation.rb:126:in `invoke_command'", 
"/opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/bundler-1.13.7/lib/bundler/vendor/thor/lib/thor.rb:359:in `dispatch'", "/opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/bundler-1.13.7/lib/bundler/cli.rb:20:in `dispatch'", 
"/opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/bundler-1.13.7/lib/bundler/vendor/thor/lib/thor/base.rb:440:in `start'", "/opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/bundler-1.13.7/lib/bundler/cli.rb:11:in `start'", 
"/opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/bundler-1.13.7/exe/bundle:34:in `block in '", "/opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/bundler-1.13.7/lib/bundler/friendly_errors.rb:100:in 
`with_friendly_errors'", "/opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/bundler-1.13.7/exe/bundle:26:in `'", "/opt/gitlab/embedded/bin/bundle:23:in `load'", "/opt/gitlab/embedded/bin/bundle:23:in `'"]
time="2018-04-24T02:13:37Z" level=info msg="finished unary call" grpc.code=OK grpc.meta.call_site=repository_exists grpc.meta.client_name=gitlab-web grpc.method=RepositoryExists 
grpc.request.repoPath=atlas-dcs-common-software/ScaSoftwarePerformanceSuite.git grpc.request.repoStorage=default grpc.request.topLevelGroup=atlas-dcs-common-software grpc.service=gitaly.RepositoryService grpc.time_ms=125 
peer.address=@ span.kind=server system=grpc