Comments on: Ruby 1.9 made me remember how I hate the concept of encodings

By: mark

mark — Fri, 11 Nov 2011 20:23:10 +0000

You are absolutely right Frank.

Ruby 1.9.x forces everyone to deal with it.

The Encoding shit is still the number one reason why I can’t upgrade. I solved all the other problems with syntax change.

Ruby now forces me to deal with Encodings. I was able to avoid that.

Now I have to read up shit about Encoding and I KNOW THAT I WILL NEVER NEED THEM BECAUSE IN THE TEN YEARS SO FAR I NEVER HAD TO CATER TOWARDS ENCODINGS I DID NOT CARE ABOUT.

Now this choice has been removed from Ruby users.

By: Frank

Frank — Wed, 30 Jun 2010 13:26:50 +0000

Comments are now moderated. Please respect this public space and be nice in your comments if you want them to be published. Thank you!

By: May 14: Moving To Beta 3 « Rails Test Prescriptions Blog

May 14: Moving To Beta 3 « Rails Test Prescriptions Blog — Fri, 14 May 2010 15:09:11 +0000

[...] A brief rant on Ruby 1.9 and encodings. [...]

By: Frank

Frank — Fri, 14 May 2010 13:57:45 +0000

@Yehuda,

Thanks for the nice explanation. I’m not throwing the blame at anyone in fact. I know Unicode is supposed to be the “ultimate encoding table” that I am talking about but it just isn’t the case yet.

In a perfect world we would not have to worry about character encoding at all. Machines should do what has to be done behind the scene to make it all work without our intervention. I’m sure one day we’ll get there.

By: Frank

Frank — Fri, 14 May 2010 13:32:45 +0000

@Amadan, but what kind of disaster could happen if the website in question doesn’t need internationalization? These “local” websites/applications are not waiting for a disaster to happen… they would have continue to work for eternity. Now they don’t and I think it is stretching a bit too far to put all the blame on the developer.

Call me crazy but I prefer the silent “failing” of 1.8 over the unforgiving exception that is triggered with 1.9.1

By: Amadan

Amadan — Fri, 14 May 2010 08:34:02 +0000

@Frank: Well, if the Rails app is presenting a UTF-8 web page and using a Latin-1 table for storage, it is a broken app – conceptually if not in practice. It’s a disaster that was just waiting to happen. Well, it happened – not Ruby’s fault people don’t configure their DBs properly.

By: Frank

Frank — Fri, 14 May 2010 00:47:48 +0000

Jorg, that’s the most exaggerated comment I saw so far. With your standards every rails applications backed by mysql databases encoded in Latin-1 (often the default encoding with mysql) are broken. That’s a whole lot of broken apps!

By: Jörg W Mittag

Jörg W Mittag — Fri, 14 May 2010 00:20:04 +0000

> but it breaks working applications

Actually, it doesn’t. It makes *already broken* applications raise an exception instead of silently corrupting data.

By: nona

nona — Thu, 13 May 2010 17:59:49 +0000

Every string has an implicit encoding, even if it’s just ASCII (be happy it’s not EBCDIC). Ruby 1.9 just makes this explicit, and allows you to convert strings to specific encodings, or not (remember, some encoding conversions are lossy).

By: Yehuda Katz

Yehuda Katz — Thu, 13 May 2010 16:04:38 +0000

> Why on earth do we have all those different encodings in 2010?
> Why not making a huge encoding table UTF-16384 containing
> every single character in the universe so we can forget about this
> crazy concept of different encodings and pretend that it never existed?

We do have that. It’s called Unicode. Unicode has a list of every character the unicode committee has gotten to with a number. UTF-8, UTF-16, and UTF-32 are simply different ways to encode those numbers in a way that can be read.

Unfortunately, what you propose is an inherently human activity, and a hard one at that. They’ve been at it for quite some time, and there are still encodings that cannot be losslessly converted to Unicode and back. This is the same for UTF-8, UTF-16, and UTF-32, because UTF-8 can already encode the entire Unicode space.

One issue is that in multiple encodings, there might be characters that represent the same unicode character. For instance, we don’t need a separate “A” for the UTF-8 variant and the Latin-1 variant (Latin-1 is a common encoding that has the ASCII characters plus a bunch of stuff for latin countries). They attempted to do this with the myriad encodings in Japan, in a process called “Han Unification”. Unfortunately, the Japanese weren’t always happy with the results.

This process could sometimes lead to it being impossible to represent your name in Unicode. Imagine if your name was José, and the Unicode committee decided that é and e were the same character, so a program which read in José spit back out Jose. You would not be happy with it.

In short, it’s possible that this problem will eventually be solved, but it’s complicated and messy, and probably has many years to go before it’s completed.