Make sure we convert the source to a native UTF-8 string before parsing

[ahoppen]

It appears like the compiler is now optimizing the self += "" away in release builds. Furthermore, we didn't catch the issue since the assertion checking for a native UTF-8 string also gets optimized away in release builds.

Fix the issue in two ways:

1. Make the assertion a precondition that also fails in release builds
2. Use the newly (as of Swift 5.1) provided standard library APIs to make the String a contiguous UTF-8 string

rdar://71863757

[ahoppen]

@swift-ci Please test

[akyrtzi]

@milseman do you maybe have another recommendation for forcing a native UTF8 string?
We also use this:

  var isNativeUTF8: Bool {
    return utf8.withContiguousStorageIfAvailable { _ in 0 } != nil
  }

[milseman]

Are you able to use Swift 5.1? SE-0247 Contiguous Strings provides the preferred API on String: var isContiguousUTF8: Bool and mutating func makeContiguousUTF8().

[milseman]

If you cannot use Swift 5.1, let's come up with a solution that doesn't assemble and then disassemble grapheme clusters. Have you tried something like self = String(decoding: self.utf16, as: UTF16.self) if you know it's a bridged fully-UTF-16 Cocoa string? You can compare that performance to self = String(decoding: self.utf8, as: UTF8.self).

[akyrtzi]

We can use Swift 5.1, thank you!

[akyrtzi]

Let's use the String APIs that are provided for this kind of functionality.

[ahoppen]

Thanks for the pointer @milseman, that’s a much cleaner solution.

@swift-ci Please test.

[swift-ci]

Build failed
Swift Test Linux Platform
Git Sha - b3b1f79

[swift-ci]

Build failed
Swift Test OS X Platform
Git Sha - b3b1f79