FIPS packages will not work with OpenSSL 3

Upon testing FIPS package for EL9 in #8569 (closed), which ships OpenSSL 3, the following issues were found

Reconfigure fails to start with the following message
```
$ sudo gitlab-ctl reconfigure
Not applying net/http monkey patch needed for ruby 3.1
/opt/gitlab/embedded/bin/ruby: symbol lookup error: /opt/gitlab/embedded/lib/ruby/3.2.0/x86_64-linux/openssl.so: undefined symbol: FIPS_mode
```
This is because OpenSSL 3 dropped the methods FIPS_mode and FIPS_mode_set, but Ruby only dropped them in version 3.3.0 in https://github.com/ruby/ruby/commit/678d41bc51f. We should look include that commit as a patch.

On applying the above patch, reconfigure fails with the following message

RuntimeError                                                                                                                                                                                                                                                  
------------                                                                                                                                                                                                                                                  
ci_jwt_signing_key: The provided key is not valid RSA key                                                                                                                                                                                                     
                                                                                                                                                                                                                                                              
Cookbook Trace: (most recent call first)                                                                                                                                                                                                                      
----------------------------------------                                                                                                                                                                                                                      
  /opt/gitlab/embedded/cookbooks/cache/cookbooks/gitlab/libraries/gitlab_rails.rb:113:in `rescue in validate_secrets'                                                                                                                                         
  /opt/gitlab/embedded/cookbooks/cache/cookbooks/gitlab/libraries/gitlab_rails.rb:109:in `validate_secrets'                                                                                                                                                   
  /opt/gitlab/embedded/cookbooks/cache/cookbooks/package/libraries/settings_dsl.rb:184:in `block in generate_secrets'                                                                                                                                         
  /opt/gitlab/embedded/cookbooks/cache/cookbooks/package/libraries/settings_dsl.rb:181:in `each'                                                                                                                                                              
  /opt/gitlab/embedded/cookbooks/cache/cookbooks/package/libraries/settings_dsl.rb:181:in `generate_secrets'                                                                                                                                                  
  /opt/gitlab/embedded/cookbooks/cache/cookbooks/package/libraries/settings_dsl.rb:203:in `generate_config'                                                                                                                                                   
  /opt/gitlab/embedded/cookbooks/cache/cookbooks/gitlab/recipes/config.rb:35:in `from_file'                                                                                                                                                                   
  /opt/gitlab/embedded/cookbooks/cache/cookbooks/gitlab/recipes/default.rb:27:in `from_file'
  /opt/gitlab/embedded/cookbooks/cache/cookbooks/gitlab-ee/recipes/default.rb:22:in `from_file'

Relevant File Content:
----------------------
/opt/gitlab/embedded/cookbooks/cache/cookbooks/gitlab/libraries/gitlab_rails.rb:

106:  
107:        return unless Gitlab['gitlab_rails']['ci_jwt_signing_key']
108:  
109:        begin
110:          key = OpenSSL::PKey::RSA.new(Gitlab['gitlab_rails']['ci_jwt_signing_key'])
111:          raise 'ci_jwt_signing_key: The provided key is not private RSA key' unless key.private?
112:        rescue OpenSSL::PKey::RSAError
113>>         raise 'ci_jwt_signing_key: The provided key is not valid RSA key'
114:        end
115:      end
116:  
117:      def parse_external_url
118:        return unless Gitlab['external_url']
119:  
120:        uri = URI(Gitlab['external_url'].to_s)
121:  
122:        raise "GitLab external URL must include a schema and FQDN, e.g. http://gitlab.example.com/" unless uri.host

On inspecting code, we see that https://gitlab.com/gitlab-org/omnibus-gitlab/-/blob/1a68ecb14a895fc479602544defd769593daabf4/files/gitlab-cookbooks/gitlab/libraries/gitlab_rails.rb#L80 creates a key using OpenSSL::PKey::RSA.new, converts it to PEM format, and then in https://gitlab.com/gitlab-org/omnibus-gitlab/-/blob/1a68ecb14a895fc479602544defd769593daabf4/files/gitlab-cookbooks/gitlab/libraries/gitlab_rails.rb#L110 attempts to read it back and fail.

Seems related to https://github.com/ruby/openssl/issues/603, but few of the fixes should already be in 3.2.0.

Edited Sep 03, 2024 by Balasankar 'Balu' C