Skip to content

Force convert invalid ASCII 8bit diffs and use a replacement ASCII character

Gary Holtz requested to merge 207124-encoding-undefinedconversionerror-in-highlight_cache-rb into master

What does this MR do and why?

As described in #207124 (closed), invalid characters in an ASCII-8bit encoded diff were causing the entire diff to not load. In order to at least mitigate that portion of it, this MR will determine if there is an invalid character in the diff and replace it with a generic ASCII snowman in the text.

Since the actual diff highlighting uses the file blob, the ascii snowman is really only seen in the text field in diffs_batch.json

"highlighted_diff_lines": [

{
  "line_code": "f9abdc73eba4e926645a8761ea178b0ca630a28b_1_1",
    "type": null,
    "old_line": 1,
    "new_line": 1,
    "text": " ☃☃#language fr-FR  \"☃\"\r",
    "meta_data": null,
    "rich_text": " <span id=\"LC1\" class=\"line\" lang=\"plaintext\"><feff>#language fr-FR  \"è\"</span>\n",
    "can_receive_suggestion": true
},

The text field doesn't seem to be used in this situation at all, so I think it's a definite improvement over the current functionality of a complete diff failure :)

I'm introducing a feature flag so we can cautiously enable it and check any other unforseen consequences.

Screenshots or screen recordings

These are strongly recommended to assist reviewers and reduce the time to merge your change.

How to set up and validate locally

Numbered steps to set up and validate the change are strongly suggested.

MR acceptance checklist

This checklist encourages us to confirm any changes have been analyzed to reduce risks in quality, performance, reliability, security, and maintainability.

Related to #207124 (closed)

Edited by Gary Holtz

Merge request reports