Formatter doesn't trim trailing whitespace in heredoc

[straight-shoota]

The formatter usually trims all whitespace from a line that only contains whitespace.

But it does not do that in heredocs.
For example, the formatter would remove the two white spaces between 1 and 2, but not between 3 and 4.

1
  
2

<<-TXT
  3
  
  4
  TXT

Removing the spaces in a heredoc is entirely fine, as long as its fewer characters than the heredoc indent of the end delimiter.
Many editors would automatically trim this anyway on save.

https://github.com/crystal-lang/crystal/blob/c16d805b3f06c723d3e521d73c326207f34d2b31/spec/std/spec/list_tags_spec.cr

Between 5 and 6 there are 4 spaces, but the indent is only 4 characters. So maybe should not be changed because the whitespace is part of the heredoc content.

<<-TXT
  5
    
  6
  TXT

Editors' autotrim-on-save feature is usually not aware of special heredoc semantics. So in practice, this exception might be hard to use because developer tooling isn't inclined to maintain a file with an explicit whitespace-only line in a heredoc.
So it's probably fine to ignore this special case and always trim all whitespace even in heredocs.

[straight-shoota]

In 2652279 we explicitly addressed to exclude heredocs from rstrip, fixing #3183.

I suppose we can enhance this feature to take the heredoc indent into account, and strip whitespace if the line length is shorter than the indent.

However, I wouldn't mind simply reverting 2652279. That would help simplify the formatter code instead of adding even more complexity.
This special rule seems pretty useless because trailing whitespace wouldn't survive an editor with "trim trailing whitespace" feature (which I believe many editor configurations have enabled by default; we have it in our .editorconfig as well).

[straight-shoota]

A way to preserve trailing whitespace in a heredoc is to put some none-whitespace character at the end of the line. This should however not affect the heredoc content, only the literal representation.

We should be able to do that by combining an escaped newline character with line continuation.
However, for some reason, the parser chokes on that and fails to recognize the indent after the line continuation.

<<-TXT # Syntax error: heredoc line must have an indent greater than or equal to 2 (in line "   6")
  5
    \n\
  6
  TXT

Another option would be combining line continuation with the unescaped newline, which adds another line to the heredoc literal. But it seems line continuation in heredocs is entirely broken. The following example ignores the empty line before "6" and indents the number "6" instead.

<<-TXT # => "5\n  6"
  5
    \

  6
  TXT

So it looks like the only option to reliably represent a string with trailing whitespace in a heredoc is to use explicit newlines. This is not ideal because it merges lines of the literal and thus they don't directly map to lines in the content. But at least it works consistently.

<<-TXT # => "5\n  \n6"
  5
    \n6
  TXT

Another alternative of course is not to a different kind of string literal.

[straight-shoota]

There's actually another option: Escape the last whitespace on the line.

<<-TXT
  5
   \u0020
  6
  TXT