Skip to content

[stdlib] Rewrite UTF8 decoding #1525

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 15 commits into from
Mar 22, 2016
Merged

[stdlib] Rewrite UTF8 decoding #1525

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
15 commits
Select commit Hold shift + click to select a range
49741cf
[stdlib] Rewrite UTF8 decoding
PatrickPijnappel Mar 3, 2016
25b5028
[stdlib] Improve UTF8._decodeOne() doc comment
PatrickPijnappel Mar 3, 2016
f5d7bd2
[stdlib] Improve comments in UTF8
PatrickPijnappel Mar 3, 2016
17124c8
[stdlib] Clarify variable name in UTF8
PatrickPijnappel Mar 3, 2016
500dc9d
[stdlib] Fix bitshift operand mask in UTF8.decode()
PatrickPijnappel Mar 4, 2016
c109bb2
[stdlib] Copy warning to docs of all UnicodeCodecTypes
PatrickPijnappel Mar 4, 2016
636423c
[stdlib] Minor refactor for clarity in UTF8.decode()
PatrickPijnappel Mar 5, 2016
4d05dc6
[stdlib] Resolve merge conflict in Unicode.swift
PatrickPijnappel Mar 13, 2016
2ec5710
[stdlib] Format and clarify documentation in Unicode.swift
PatrickPijnappel Mar 13, 2016
549f7e2
[stdlib] Fix incorrect merge 4d05dc69
PatrickPijnappel Mar 13, 2016
43bca07
[stdlib] Fix doc comment for UnicodeScalar.encode()
PatrickPijnappel Mar 14, 2016
4d3d723
[stdlib] Replace usages of generator with iterator
PatrickPijnappel Mar 16, 2016
7e0945e
[stdlib] Use explicit internal in Unicode.swift
PatrickPijnappel Mar 16, 2016
771a815
[stdlib] Fix indentation in Unicode.swift
PatrickPijnappel Mar 16, 2016
ffe8646
[stdlib] Clarify buffer comments in UTF8._decodeOne()
PatrickPijnappel Mar 22, 2016
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 3 additions & 1 deletion stdlib/public/core/StringUTF8.swift
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -159,7 +159,9 @@ extension String {
/// True iff the index is at the end of its view or if the next
/// byte begins a new UnicodeScalar.
internal var _isOnUnicodeScalarBoundary : Bool {
return UTF8._isValidUTF8(UInt32(truncatingBitPattern: _buffer)) || _isAtEnd
let buffer = UInt32(truncatingBitPattern: _buffer)
let (codePoint, _) = UTF8._decodeOne(buffer)
return codePoint != nil || _isAtEnd
}
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@PatrickPijnappel
Just out of curiosity, why return !UTF8.isContinuation(UTF8.CodeUnit(truncatingBitPattern: _buffer)) is insufficient?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@rintaro That would change the behavior slightly w.r.t. the original version, as not all non-continuation code units are valid starting code units (e.g. 0xff). Though the difference is somewhat inconsequential as _buffer should be well-formed, and in general it's not clear from the documentation what the handling of mal-formed sequences should be.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for clarifying :)


/// True iff the index is at the end of its view
Expand Down
Loading