Validity of strings

[RalfJung]

Discussing the validity invariant of strings: Do they just have to be like byte slices, or do we require in the validity invariant that they be valid UTF-8?

Certainly, their safety invariant requires UTF-8, but that is not what we are discussing here.

[nikomatsakis]

I think they should just be byte-slices, analogously with how char's validity invariant is independent from unicode.

EDIT: That statement was incorrect

[RalfJung]

Note that the Nomicon calls non-UTF-8 str a form of insta-UB, making it a validity invariant: https://doc.rust-lang.org/nightly/nomicon/what-unsafe-does.html.

[Lokathor]

I think that in this case the Nomicon simply isn't quite correct. The Validity Invariant of a type, if any, is known to the compiler. In the case of str, the utf8 invariant is known only to actual Rust code. As far as I know, LLVM has absolutely no concept that there's a byte encoding involved.

[hanna-kruppe]

I don't think the current rustc-LLVM compilation pipeline exploits knowledge of str being UTF-8 anywhere, but since str is a built-in type, the compiler absolutely could know about this invariant if we defined it so, and of course rustc is under no obligation to actually exploit that knowledge. So what the Nomicon says is perfectly compatible with the current state of the implementation.

That said, I don't think there's anything useful the compiler can do with strs being well-formed UTF-8, since you usually won't store or copy them by value. Even if reference validity includes validity of the pointee, that still will be hard to make use of -- niches behind references are useless for layout optimizations, and the invariant is rather complicated (since the format is variable-length) so I doubt it could be used to automatically optimize code that processes UTF-8 on a byte by byte basis. For these reasons, I don't object to downgrading this to the safety invariant.

[RalfJung]

I opened a PR against the reference to decide that strings do not have UTF-8 in their validity invariant: rust-lang/reference#792.

[scottmcm]

I saw this comment above:

I think they should just be byte-slices, analogously with how char's validity invariant is independent from unicode.

Is that true? It seems in opposition to the following statement from rust-lang/rust#71033 (comment)

And anyway Rust also limits char to U+10FFFF max.

Trying out matching on char demonstrates that, at least as implemented today, even in unsafe code you can't look for things beyond the unicode-defined limit: https://rust.godbolt.org/z/gLBW08

So I don't think that "analogously [...] independent from unicode" is good justification here.

(Which isn't a statement that removing the invariant would bad, just that I'd like to strike that particular piece of evidence.)

[nikomatsakis]

I agree I was incorrect.