[SR-2956] CharacterSet union and formUnion not working properly in Swift 3 with unicode

[swift-ci]

Previous ID	SR-2956
Radar	None
Original Reporter	thuss (JIRA User)
Type	Bug
Status	Reopened
Resolution

Environment

Mac OS Sierra 10.12
Xcode 8.0 (8a218a)
Apple Swift version 3.0 (swiftlang-800.0.46.2 clang-800.0.38)
Target: x86_64-apple-macosx10.9

Additional Detail from JIRA

Votes	2
Component/s	Foundation
Labels	Bug, SDKOverlay
Assignee	None
Priority	Medium

md5: eea36f50a39958f934a3b2a2adf3ded2

Issue Description:

You can reproduce with the following code: https://gist.github.com/twobitlabs/5ba150aed3c159d215ef049f0c5739de

import Foundation
var charset = CharacterSet(charactersIn: "a")
charset.formUnion(CharacterSet(charactersIn: "\u{1F600}"))
print(charset.contains("\u{1F600}")) // prints false but should print true

[jepers]

The issue is still present in Xcode 9.3 beta 2.

Also, note that .insert can be used to demonstrate the same problem (maybe union uses insert):

var charset = CharacterSet(charactersIn: "a")
charset.insert(charactersIn: "\u{1F600}")
print(charset.contains("\u{1F600}")) // false

And this:

var charset = CharacterSet()
charset.insert(charactersIn: "\u{1F600}")
print(charset.contains("\u{1F600}")) // true

And this:

var charset = CharacterSet()
charset.insert(charactersIn: "a")
charset.insert(charactersIn: "\u{1F600}")
print(charset.contains("\u{1F600}")) // false

And this:

var charset = CharacterSet()
charset.insert(charactersIn: "\u{1F600}")
charset.insert(charactersIn: "a")
print(charset.contains("\u{1F600}")) // true

Since the following is working as expected no matter the order of a and b, maybe the problem is in how the ...(charactersIn: String) separates the String into unicode scalars?

let a = Unicode.Scalar("a")!
let b = Unicode.Scalar("\u{1F600}")!
var cs = CharacterSet()
cs.insert(a)
cs.insert(b)
print(cs.contains("\u{1F600}")) // true, and
// also true when swapping the order in which
// a and b are inserted.

Seems like it has problems handling supplemental code points:

var charset = CharacterSet()
charset.insert(charactersIn: "a")
charset.insert(charactersIn: "\u{1F600}")
print(charset.contains("\u{1F600}")) // false
print(charset.debugDescription)
// <CFCharacterSet Items(U+0061 U+F600)>
// Note missing upper 16 bits from supplemental
// code point U+F600 instead of U+1F600.

var charset = CharacterSet()
//charset.insert(charactersIn: "a") <-- Commented out
charset.insert(charactersIn: "\u{1F600}")
print(charset.contains("\u{1F600}")) // false
print(charset.debugDescription)
// <CFCharacterSet Range(128512, 1)>
// (128512 == 0x1F600)

[milseman]

@itaiferber, @parkera @phausler

What is the semantics of CharacterSet exactly? The description: "A set of Unicode character values for use in search operations." implies that it can at least store unicode scalar values, but this behavior makes it look like it can only store UTF-16 code units. Is this a bug and can it be fixed, or is this the (undocumented) semantics of CharacterSet?

[phausler]

this is undocumented semantics where we just don't trap on invalid characters.

[milseman]

Could you define character? Is that restricted to BMP scalar?

[itaiferber]

@milseman CharacterSet and co work primarily in UTF-16 code units. CFString/NSString have APIs which work at higher levels, but the core of all of this is set historically in UTF-16 code units.

[milseman]

Can we at least document the fact that CharacterSet is only for use with BMP scalars?