Force ASCII_8BIT encoding in POST request bodies (UDCSL #671)

[blms]

In this PR

Per https://github.com/performant-software/urban-design-csl/issues/136:

• Force ASCII_8BIT encoding of all POST request payloads
• Solution from this HTTParty issue: incompatible character encodings: UTF-8 and ASCII-8BIT jnunemaker/httparty#671

[blms]

@NickLaiacona @camdendotlol Let me know if this seems like a reasonable way to handle Unicode filenames.

If so, once this is merged I'll go ahead and make a new release version, so that you can update your projects with the new version, and get us closer to merging and releasing performant-software/iiif-cloud#25.

[camdendotlol]

Interesting, I haven't run into this with NBU but the fix seems reasonable.

[NickLaiacona]

Would this drop accent characters on resourceable.name ? Looks like most of content would stay in UTF-8 except the original filename, is that right?

[blms]

@NickLaiacona

Would this drop accent characters on resourceable.name ?

I believe it wouldn't drop them, but force encoding, so it would look like this in the request body:

Medellín.pdf --> Medelli\xCC\x81n.pdf

It seems to be converted back to UTF-8 in IIIF Cloud and UDCSL; I'm not really sure how/when that happens, but it appears correctly both in the frontend and in the Postgres DB.

Looks like most of content would stay in UTF-8 except the original filename, is that right?

Digging into it a bit further, the problem seems to be that a different part of the content is in ASCII-8BIT. The error occurs here, when the request body is initialized:

https://github.com/jnunemaker/httparty/blob/c27adec455bc6f3e8556b51d711d99ab37c49468/lib/httparty/request/body.rb#L38-L49

The issue is that the multipart body starts getting written as UTF-8 because of the Unicode in the filename, but content_body(value) always returns an ASCII_8BIT string when the actual ActionDispatch::Http::UploadedFile is passed as the value param (when it gets .read and then .to_s called on it).

I can't exactly tell if this is a bug with HTTParty, a problem with the file itself, or a bug with our code. If I modify HTTParty the following way (replacing line 47 in the above snippet), I can get it to work fine:

if memo.encoding == content_body(value).encoding
  memo << content_body(value)
elsif !value.nil?
  memo << content_body(value).force_encoding(memo.encoding)
end

Alternatively, in our code, changing the body we pass along to HTTParty as I did in this branch to force ASCII_8BIT seems to work just as well. And FWIW, the change I made here is also what's suggested in the HTTParty issue I linked above. Maybe it makes sense to keep this change for now but post my findings in the HTTParty issue and hope for a patch release?

[NickLaiacona]

It seems to be converted back to UTF-8 in IIIF Cloud and UDCSL; I'm not really sure how/when that happens, but it appears correctly both in the frontend and in the Postgres DB.

@blms OK, so the data rests as UTF-8, it just gets transported as ASCII8. 👍 I think hacking it on our side is better than forking HTTParty.