Skip to content

Do not escape double quote msgid in po file during compile po to json

Hi, there

I found one issue related i18n, when compile po to json in gitlab:assets:compile, it will escape double quote msgid in po file, which will cause app.js NOT match with original msgid. So all msgid contains double quote ARE not working currently.

Behavior

let's say we have a i18n template

    considerFlexibleRollout: s__(
      'FeatureFlags|Consider using the more flexible "Percent rollout" strategy instead.',
    ),

after extract task, po file will be like below, msgid is in double quote, so \" means " , the msgid is same with js file, which is correct

msgid "FeatureFlags|Consider using the more flexible \"Percent rollout\" strategy instead."
msgstr ""

however convert po to json, the app.js is

"FeatureFlags|Consider using the more flexible \\\"Percent rollout\\\" strategy instead.":["请考虑使用更灵活的“百分比上线”策略。"]

please pay attention, the string in app.js is FeatureFlags|Consider using the more flexible \"Percent rollout\" strategy instead. , which add extra \ in msgid, mismatch with js file, break the i18n functionality.

Analyze

there are two ways to translate po to json, one is po_to_json gem, another is gitlab/scripts/frontend/po_to_json.js written by nodejs.

At first, the bug was introduce in po_to_json gem, line 2 in method push_buffer , value.gsub(/\"/, """) will try to replace \" to " , which should not escape double quote, however gsub method didn't change value variable, so it should be value = value.gsub or value.gsub! (I will create pr to fix this gem)

      def push_buffer(value, key = nil)
        value = ::Regexp.last_match(1) if value =~ /^"(.*)"/
        value.gsub(/\\"/, "\"")

        if key.nil?
          buffer[lastkey] = [
            buffer[lastkey],
            value
          ].join("")
        else
          buffer[key] = value
          @lastkey = key
        end
      end

Currently, gitlab/scripts/frontend/po_to_json.js is introduced at 2023.03, to replace po_to_json task for better performance, and funny thing is, we reproduce the bug logic in the task, just to make sure app.js generated by old way and new way are identical, and we know it may be a bug. read the following code in po_to_json.js, we escape msgid which change " to \", which is wrong.

  /**
   * TODO: This replacer might be unnecessary _or_ even cause bugs.
   *   due to potential unnecessary double escaping.
   *   But for now it is here to ensure that the old and new output
   *   are equivalent.
   * @param str
   * @returns {string}
   */
  function escapeMsgid(str) {
    return `${str}`.replace(/([\\"])/g, '\\$1');
  }

Action

  1. since we use po_to_json.js instead of po_to_json gem, so currently we can only focus on po_to_json.js , we should remove the function escapeMsgid, then msgid will be correct.
  2. I will create PR to po_to_json gem
Edited by MAO Chao