> In the above example, on every loop, the += operator causes a new string to be allocated, and the content to be copied, which gets exponentially more expensive as the string grows.
But that’s only the theoretical behaviour. In practice, languages tend to end up optimising it in various ways. As noted a paragraph later, the Java compiler is able to detect and “fix” this, by rewriting the code to use a mutable string.
Another solution is to put concatenation into your string type as another possible representation. I believe at least some (no idea if it’s all) JavaScript engines do this. You end up with something like this (expressed in Rust syntax, and much simpler than the real ones are):
Then, when you try to access the string, if it’s Concatenated it’ll flatten it into one of the other representations.
Thus, the += itself becomes cheap, and in typical patterns you only incur the cost of allocating a new string once, when you next try to read from it (including things like JSON.stringify(object_containing_this_string) or element.setAttribute(name, this_string)).
masklinn 33 days ago [-]
> As noted a paragraph later, the Java compiler is able to detect and “fix” this, by rewriting the code to use a mutable string.
Does it actually do that nowadays? Back in my days it was incapable of lifting the builder out of loops, so for each iteration it would instantiate a builder with the accumulator string, append the concatenation, then stringify and reset the accumulator.
The linked docs don’t say anything about loops.
emil-lp 33 days ago [-]
I agree. If you append to a string in a loop in Java, you will see quadratic behavior.
masklinn 32 days ago [-]
An option you did not mention, although not generally available to languages with advanced runtime, is that even if you have immutable strings you can realloc them if you know you're the only owner of that string (e.g. because they're refcounted, or can't be shared).
CPython does that, so a trivial concatenation loop is (amortised) linear (causing issues for alternate implementations when they have to run that). Swift might also be able to via COW optimisation.
Rust's string concatenation is a variant of this (though it has mutable strings anyway): `String::add` takes an owned value on the LHS, and the implementation just appends to the LHS before returning it:
so repeated concatenation will realloc and amortize as if you were just `push_str`-ing in a loop (which maps directly to appending to the underlying buffer).
chrismorgan 32 days ago [-]
For more technical precision in Rust (not because it actually changes anything), += will use AddAssign rather than Add, if it’s implemented, which mutates in-place, whereas `a = a + b` would move twice (though in a way that will always optimise to the same thing). This means you’re actually invoking
which the doc comment notes “has the same behavior as the `push_str` method”.
And this shows yet another option: add a separate overload for +=. Python does actually have that in the form of the __iadd__ magic method, and I presume CPython’s optimisation could be implemented that way, though it doesn’t look to be (and this might not have quite the same semantics, I’m not sure):
class str:
def __iadd__(self, other):
if sys.getrefcount(self) == probably 2, it’s fiddly:
mutate self
raise NotImplemented
bruce343434 32 days ago [-]
Maybe I'm missing something, but I thought that when you "grow" an array or a string, eventually something under the hood needs to call realloc()? which allocates a bigger chunk of memory and copies the content O(n)? Why would it matter if you manually do alloc() and then memcpy() (replacing an immutable string with its concatenation) vs letting the runtime do it for you with realloc()?
chrismorgan 32 days ago [-]
There are two differences.
① A growable string type overallocates, so you only eventually need to reallocate. An immutable string type has an exact-size allocation, so you must make a new allocation every time.
② An immutable string type can’t use realloc() anyway unless you can prove nothing else holds a reference to it, it needs to use a new malloc().
shawn_w 32 days ago [-]
You're basically describing the Rope. Fancier versions use self balancing trees, allowing other string operations to be fairly efficient too.
Wow, if that is true then that‘s the most succinct explanation of Ropes I have seen. Super helpful if you are trying to learn about them, like I am.
Dylan16807 32 days ago [-]
It's partway to being a rope, I would say some balancing and the ability to replace substrings are crucial to a real rope.
miki123211 33 days ago [-]
> Another solution is to put concatenation into your string type
Aah, the Erlang way of handling strings.
On Beam (Erlang's VM), that goes as deep as IO. It's perfectly fine to pass a (possibly nested) list of strings (whether charlists or binaries) to an IO function, and the system just knows how to deal with that.
ramchip 33 days ago [-]
An iolist isn't a string, you can't pass it to the uppercase function for instance. It's really meant for I/O as the name implies. Regular string concatenation is optimized to avoid copying when possible: https://www.erlang.org/doc/system/binaryhandling.html#constr...
capitainenemo 33 days ago [-]
Article claims python 3 uses UTF-8.
https://stackoverflow.com/questions/1838170/
"In Python 3.3 and above, the internal representation of the string will depend on the string, and can be any of latin-1, UCS-2 or UCS-4, as described in PEP 393."
Article also says PHP has immutable strings. They are mutable, although often copied.
Article also claims majority of popular languages have immutable strings.
As well as the ones listed there is also PHP and Rust (and C, but they did say C++ - and obviously Ruby since that's the subject of the article).
I'm also a bit surprised by the last sentence.
"However, if you do measure a negative performance impact, there is no doubt you are measuring incorrectly."
There must surely be programs doing a lot of string building or in-place modification that would benefit from non-frozen.
byroot 33 days ago [-]
> There must surely be programs doing a lot of string building or in-place modification that would benefit from non-frozen.
The point is that the magic comment (or the --enable-frozen-string-literal) only applies to literals. If you have some code using mutable strings to iteratively append to it, flipping that switch doesn't change that. It just means you'll have to explicitly create a mutable string. So it doesn't change the performance profile.
byroot 33 days ago [-]
> can be any of latin-1, UCS-2 or UCS-4, as described in PEP 393
My bad, I haven't seriously used Python for over 15 years now, so I stand corrected (and will clarify the post).
My main point stands though, Python strings have an internal representation, but it's not exposed to the user like Ruby strings.
> Article also says PHP has immutable strings. They are mutable, although often copied.
Same. Thank you for the correction, I'll update the post.
capitainenemo 32 days ago [-]
Cool, although I feel if on one side you have Java, JavaScript, Python, Go and on the other Perl, PHP, C/C++, Ruby, Rust it's hard to say overwhelming majority in either direction.
Also someone below claims python byte arrays can be considered mutable strings, although I have no idea of the stringy ergonomics of that and whether it would be convenient to do - I try to avoid python too.
capitainenemo 32 days ago [-]
... and honestly, since java has both stringbuffer and string I feel it's really in the "has mutable" camp too
chrismorgan 33 days ago [-]
Python strings aren’t even proper Unicode strings. They’re sequences of code points rather than scalar values, meaning they can contain surrogates. This is incompatible with basically everything: UTF-* as used by sensible things, and unvalidated UTF-16 as used in the likes of JavaScript, Windows wide strings and Qt.
nilslindemann 33 days ago [-]
But isn't 'surrogateescape' supposed to address this? (no expert)
surrogateescape is something else altogether. It’s a hack to allow non-Unicode file names/environment variables/command line arguments in an otherwise-Unicode environment, by smuggling them through a part of the surrogate range (0x80 to 0xFF → U+DC80 to U+DCFF) which otherwise can’t occur (since it’s invalid Unicode). It’s a cunning hack that makes a lot of sense: they used a design error in one place (Python string representation) to cancel out a design error in another place (POSIX being late to the game on Unicode)!
Dylan16807 32 days ago [-]
It's not taking advantage of the weird way python strings work. You can put that hack on top of any string format that converts back and forth with unicode.
chrismorgan 31 days ago [-]
No you can’t: it only works if your string type is something other than a sequence of Unicode scalar values. In Rust, for example, strings must be valid UTF-8, so this hack is not possible.
Dylan16807 31 days ago [-]
Python strings are normally Unicode, but they are augmented with this mechanism to to smuggle other data as invalid surrogates.
Rust strings are normally Unicode, but Windows OSStrings are augmented with a similar mechanism to smuggle other data as invalid surrogates. (Rust smuggles 16 bit values as WTF-8 but it could do 8 bit smuggling instead with barely any change.)
Rust chooses not to make that the main string type, but it could. Any system based on Unicode can implement a hack like this to become a superset of Unicode.
Why do you think it can't? Rust would have to admit that the type is no longer exactly Unicode, just like python did. That's the opposite of a disqualifier, it's a pattern to follow.
Maybe you're unaware that [generalized] UTF-8 has a way to encode lone surrogates? They encode into 3 bytes just fine, either ED A_ __ or ED B_ __
capitainenemo 30 days ago [-]
With regards to what rust team is admitting or not...
https://wtf-8.codeberg.page/#the-wtf-8-encoding
"It is identical to generalized UTF-8, with the additional well-formedness constraint that a surrogate pair byte sequence is ill-formed. It is a strict subset of generalized UTF-8 and a strict superset of UTF-8."
https://wtf-8.codeberg.page/#intended-audience
"WTF-8 is a hack intended to be used internally in self-contained systems with components that need to support potentially ill-formed UTF-16 for legacy reasons.
Any WTF-8 data must be converted to a Unicode encoding at the system’s boundary before being emitted. UTF-8 is recommended. WTF-8 must not be used to represent text in a file format or for transmission over the Internet."
They seem very transparent, and certainly are not proposing it as a general type.
Dylan16807 30 days ago [-]
> With regards to what rust team is admitting or not...
That wasn't an accusation. They admit things just fine. It was a hypothetical about using it as the main string type.
> and certainly are not proposing it as a general type.
1. Python's hack isn't used in file formats or transmissions either, as far as I know. It's also internal-only.
2. What they propose it for has zero relevance to my argument. It's merely proof that a hack like this can be added to ordinary Unicode representations. Python's goofy string representation is not enabling its surrogate hack.
ameliaquining 33 days ago [-]
In C, C++, and Rust, the question of "are strings in this language mutable or immutable?" isn't applicable, because those languages have transitive mutability qualifiers. So they only need a single string type, and whether you can mutate it or not depends on context. (C++ and Rust have multiple string types, but the differences among them aren't about mutability.) In languages without this feature, a given value is either always mutable or never mutable, and so it's necessary to pick one or the other for string literals.
capitainenemo 33 days ago [-]
Sure, that doesn't change the point that mutable strings are a thing in those languages. And I don't think C's const is really a "mutability qualifier" - certainly not a very effective one at any rate.
Important information omitted from title: this is for the Ruby language.
kazinator 33 days ago [-]
It's perfectly fine to have mutable strings in a hash table; just document that the behavior becomes unspecified if keys are mutated while they are in the table.
Make sure the behavior is safe: it won't crash or be exploitable by a remote attacker.
It works especially well in a language that doesn't emphasize mutation; i.e. you don't reach for string mutation as your go-to tool for manipulation.
Explicit "freeze" stuff is an awful thing to foist onto the programmer.
lmm 33 days ago [-]
> just document that the behavior becomes unspecified if keys are mutated while they are in the table.
> Make sure the behavior is safe: it won't crash or be exploitable by a remote attacker.
There is no such thing as unspecified but safe behaviour. Developers who can't predict what will happen will make invalid assumptions which will lead to security vulnerabilities when they are violated.
kazinator 33 days ago [-]
You can predict unspecified behavior: it gives a range of possibilities which do not include failures like termination, or data corruption.
The order of evaluation of function arguments in C is unspecified, so every time any function whatsoever is called which has two or more arguments, there is unspecified behavior.
Same in Scheme!
A security flaw can be caused by a bug that is built on nothing but 100% specified constructs.
The construct with unspecified behavior won't in and of itself cause a security problem. The programmer believing that a particular behavior will occur, whereas a different one occurs, can cause a bug.
The unspecified behaviors of a hash table in the face of modified keys can be spelled out in some detail.
Example requirements:
"If a key present in a hash table is modified to an unequal value, it is unspecified whether the entry can be found using the new key; in any case, the entry cannot be found using the old key. If a key present in a hash table is modified to be equal to another key also present in the same hash table, it is unspecified which entry is found using that key. Modification of a key doesn't prevent that key's entry from being visited during a traversal of the hash."
lmm 32 days ago [-]
> The order of evaluation of function arguments in C is unspecified, so every time any function whatsoever is called which has two or more arguments, there is unspecified behavior.
Yes, and that's bad! Subsequent languages like Java learned from this mistake.
> A security flaw can be caused by a bug that is built on nothing but 100% specified constructs.
Of course. But it's less common.
> The programmer believing that a particular behavior will occur, whereas a different one occurs, can cause a bug.
And unspecified behaviour is a major cause of this! Something like Hyrum's Law applies; programmers often believe that a thing will behave the way it did when they tested it.
> The unspecified behaviors of a hash table in the face of modified keys can be spelled out in some detail.
That is to say, specified :P. The more you narrow the scope of what is unspecified, the better, yes; and narrowing it to nothing at all is best.
31 days ago [-]
kazinator 32 days ago [-]
I'm a big opponent of unspecified argument evaluation order, but my point was more that the sky doesn't fall because of that.
Though it's pretty ridiculous that a mainstream Lisp dialect is that way, of all things.
lmm 32 days ago [-]
> the sky doesn't fall because of that.
No, especially in a language like C that has much bigger problems (I don't think there's ever been a nontrivial C program that has defined behaviour according to the standard), but it's one more papercut.
> Though it's pretty ridiculous that a mainstream Lisp dialect is that way, of all things.
I don't think I'd call any Lisp dialect "mainstream".
kazinator 31 days ago [-]
Regarding that last bit; yes, that's a kind of contextual phraseology used by Lisp people sometimes in reference to the major Lisps. Among the "streams" in the Lisp landscape, those are the main ones.
The copy-and-freeze behavior is a special case that applies only to strings, presumably because the alternative was too much of a footgun since programmers usually think of strings in terms of value semantics.
I don't think anyone likes the explicit .freeze calls everywhere; I think the case for frozen strings in Ruby is primarily based on performance rather than correctness (which is why it wasn't obvious earlier in the language's history that it was the right call), and the reason it's hard to make the default is because of compatibility.
kazinator 33 days ago [-]
> since programmers usually think of strings in terms of value semantics.
Can you blame them, when you out of your way to immerse strings in the stateful OOP paradigm, with idioms like "foo".upcase!
If you give programmers mainly a functional library for string manipulations that returns new values, then that's what they will use.
jasonrgorby 32 days ago [-]
[dead]
Rendered at 21:42:34 GMT+0000 (Coordinated Universal Time) with Vercel.
But that’s only the theoretical behaviour. In practice, languages tend to end up optimising it in various ways. As noted a paragraph later, the Java compiler is able to detect and “fix” this, by rewriting the code to use a mutable string.
Another solution is to put concatenation into your string type as another possible representation. I believe at least some (no idea if it’s all) JavaScript engines do this. You end up with something like this (expressed in Rust syntax, and much simpler than the real ones are):
Then, when you try to access the string, if it’s Concatenated it’ll flatten it into one of the other representations.Thus, the += itself becomes cheap, and in typical patterns you only incur the cost of allocating a new string once, when you next try to read from it (including things like JSON.stringify(object_containing_this_string) or element.setAttribute(name, this_string)).
Does it actually do that nowadays? Back in my days it was incapable of lifting the builder out of loops, so for each iteration it would instantiate a builder with the accumulator string, append the concatenation, then stringify and reset the accumulator.
The linked docs don’t say anything about loops.
CPython does that, so a trivial concatenation loop is (amortised) linear (causing issues for alternate implementations when they have to run that). Swift might also be able to via COW optimisation.
Rust's string concatenation is a variant of this (though it has mutable strings anyway): `String::add` takes an owned value on the LHS, and the implementation just appends to the LHS before returning it:
so repeated concatenation will realloc and amortize as if you were just `push_str`-ing in a loop (which maps directly to appending to the underlying buffer).And this shows yet another option: add a separate overload for +=. Python does actually have that in the form of the __iadd__ magic method, and I presume CPython’s optimisation could be implemented that way, though it doesn’t look to be (and this might not have quite the same semantics, I’m not sure):
① A growable string type overallocates, so you only eventually need to reallocate. An immutable string type has an exact-size allocation, so you must make a new allocation every time.
② An immutable string type can’t use realloc() anyway unless you can prove nothing else holds a reference to it, it needs to use a new malloc().
https://en.wikipedia.org/wiki/Rope_(data_structure)
Aah, the Erlang way of handling strings.
On Beam (Erlang's VM), that goes as deep as IO. It's perfectly fine to pass a (possibly nested) list of strings (whether charlists or binaries) to an IO function, and the system just knows how to deal with that.
https://stackoverflow.com/questions/1838170/ "In Python 3.3 and above, the internal representation of the string will depend on the string, and can be any of latin-1, UCS-2 or UCS-4, as described in PEP 393."
Article also says PHP has immutable strings. They are mutable, although often copied.
Article also claims majority of popular languages have immutable strings. As well as the ones listed there is also PHP and Rust (and C, but they did say C++ - and obviously Ruby since that's the subject of the article).
I'm also a bit surprised by the last sentence. "However, if you do measure a negative performance impact, there is no doubt you are measuring incorrectly." There must surely be programs doing a lot of string building or in-place modification that would benefit from non-frozen.
The point is that the magic comment (or the --enable-frozen-string-literal) only applies to literals. If you have some code using mutable strings to iteratively append to it, flipping that switch doesn't change that. It just means you'll have to explicitly create a mutable string. So it doesn't change the performance profile.
My bad, I haven't seriously used Python for over 15 years now, so I stand corrected (and will clarify the post).
My main point stands though, Python strings have an internal representation, but it's not exposed to the user like Ruby strings.
> Article also says PHP has immutable strings. They are mutable, although often copied.
Same. Thank you for the correction, I'll update the post.
Also someone below claims python byte arrays can be considered mutable strings, although I have no idea of the stringy ergonomics of that and whether it would be convenient to do - I try to avoid python too.
https://vstinner.github.io/pep-383.html
Rust strings are normally Unicode, but Windows OSStrings are augmented with a similar mechanism to smuggle other data as invalid surrogates. (Rust smuggles 16 bit values as WTF-8 but it could do 8 bit smuggling instead with barely any change.)
Rust chooses not to make that the main string type, but it could. Any system based on Unicode can implement a hack like this to become a superset of Unicode.
Why do you think it can't? Rust would have to admit that the type is no longer exactly Unicode, just like python did. That's the opposite of a disqualifier, it's a pattern to follow.
Maybe you're unaware that [generalized] UTF-8 has a way to encode lone surrogates? They encode into 3 bytes just fine, either ED A_ __ or ED B_ __
https://wtf-8.codeberg.page/#intended-audience "WTF-8 is a hack intended to be used internally in self-contained systems with components that need to support potentially ill-formed UTF-16 for legacy reasons.
Any WTF-8 data must be converted to a Unicode encoding at the system’s boundary before being emitted. UTF-8 is recommended. WTF-8 must not be used to represent text in a file format or for transmission over the Internet."
They seem very transparent, and certainly are not proposing it as a general type.
That wasn't an accusation. They admit things just fine. It was a hypothetical about using it as the main string type.
> and certainly are not proposing it as a general type.
1. Python's hack isn't used in file formats or transmissions either, as far as I know. It's also internal-only.
2. What they propose it for has zero relevance to my argument. It's merely proof that a hack like this can be added to ordinary Unicode representations. Python's goofy string representation is not enabling its surrogate hack.
Make sure the behavior is safe: it won't crash or be exploitable by a remote attacker.
It works especially well in a language that doesn't emphasize mutation; i.e. you don't reach for string mutation as your go-to tool for manipulation.
Explicit "freeze" stuff is an awful thing to foist onto the programmer.
> Make sure the behavior is safe: it won't crash or be exploitable by a remote attacker.
There is no such thing as unspecified but safe behaviour. Developers who can't predict what will happen will make invalid assumptions which will lead to security vulnerabilities when they are violated.
The order of evaluation of function arguments in C is unspecified, so every time any function whatsoever is called which has two or more arguments, there is unspecified behavior.
Same in Scheme!
A security flaw can be caused by a bug that is built on nothing but 100% specified constructs.
The construct with unspecified behavior won't in and of itself cause a security problem. The programmer believing that a particular behavior will occur, whereas a different one occurs, can cause a bug.
The unspecified behaviors of a hash table in the face of modified keys can be spelled out in some detail.
Example requirements:
"If a key present in a hash table is modified to an unequal value, it is unspecified whether the entry can be found using the new key; in any case, the entry cannot be found using the old key. If a key present in a hash table is modified to be equal to another key also present in the same hash table, it is unspecified which entry is found using that key. Modification of a key doesn't prevent that key's entry from being visited during a traversal of the hash."
Yes, and that's bad! Subsequent languages like Java learned from this mistake.
> A security flaw can be caused by a bug that is built on nothing but 100% specified constructs.
Of course. But it's less common.
> The programmer believing that a particular behavior will occur, whereas a different one occurs, can cause a bug.
And unspecified behaviour is a major cause of this! Something like Hyrum's Law applies; programmers often believe that a thing will behave the way it did when they tested it.
> The unspecified behaviors of a hash table in the face of modified keys can be spelled out in some detail.
That is to say, specified :P. The more you narrow the scope of what is unspecified, the better, yes; and narrowing it to nothing at all is best.
Though it's pretty ridiculous that a mainstream Lisp dialect is that way, of all things.
No, especially in a language like C that has much bigger problems (I don't think there's ever been a nontrivial C program that has defined behaviour according to the standard), but it's one more papercut.
> Though it's pretty ridiculous that a mainstream Lisp dialect is that way, of all things.
I don't think I'd call any Lisp dialect "mainstream".
The copy-and-freeze behavior is a special case that applies only to strings, presumably because the alternative was too much of a footgun since programmers usually think of strings in terms of value semantics.
I don't think anyone likes the explicit .freeze calls everywhere; I think the case for frozen strings in Ruby is primarily based on performance rather than correctness (which is why it wasn't obvious earlier in the language's history that it was the right call), and the reason it's hard to make the default is because of compatibility.
Can you blame them, when you out of your way to immerse strings in the stateful OOP paradigm, with idioms like "foo".upcase!
If you give programmers mainly a functional library for string manipulations that returns new values, then that's what they will use.