(10:00:59 PM) racke: ding dong, anyone around for Interchange meeting
(10:01:10 PM) __justin__: racke: present
(10:01:26 PM) racke: hello __justin__
(10:04:31 PM) blueturtle [n=Lyn@genkt-048-002.t-mobile.co.uk] entered the room.
(10:06:28 PM) endpoint_david: racke: here, but mainly distracted; say my nick if there's something that needs my attention :-)
(10:06:46 PM) racke: ok
(10:07:10 PM) racke: endpoint_david: you will be responsible for 5.7.4 release right?
(10:07:15 PM) racke: hello blueturtle
(10:07:26 PM) endpoint_david: yes, jon's riding shotgun
(10:07:30 PM) blueturtle: Hi Rack, hthis is Lyn
(10:08:02 PM) blueturtle: finally getting stuff together again
(10:08:23 PM) racke: oh hello Lyn
(10:08:28 PM) racke: nice to meet you :-)
(10:08:31 PM) k2b3: here
(10:08:56 PM) blueturtle: Yay :)
(10:10:15 PM) blueturtle: I have a list of payment and shipping modules in various stages of completion that I want to get out the door soon
(10:10:38 PM) racke: ok that sounds cool
(10:11:02 PM) racke: where are you at with Chronopay?
(10:11:08 PM) blueturtle: Chronopay, Moneybookers, Iridium, Royalmail, Parcelforce, and I should have more time soon to do these
(10:11:24 PM) racke: ok
(10:11:35 PM) blueturtle: Crhonopay - I 'm getting a full a/c, which is needed to go past the basic stage and do the 3DS bit
(10:12:16 PM) blueturtle: They will probably take 4 weeks they say to do that
(10:12:36 PM) racke: 4 weeks for a full account? 
(10:13:22 PM) blueturtle: So they say, this is to process the merchant a/c which is part of the deal with them
(10:13:41 PM) racke: ok
(10:15:11 PM) racke: did you apply this patch to your version of PayPalExpress:
(10:15:14 PM) racke: http://git.icdevgroup.org/?p=interchange.git;a=commitdiff;h=50745037ce75d35cd246e5451cd348674fbe3c80
(10:15:31 PM) blueturtle: Not yet, but I will apply both that Josh did
(10:15:50 PM) blueturtle: and then I've still got the subscription billing and the Giropay to do 
(10:15:57 PM) snewpy [n=Mark@2001:44b8:745b:aed0:e56c:8332:f1b8:6289] entered the room.
(10:16:06 PM) racke: hello Mark
(10:16:09 PM) snewpy: hi!
(10:16:12 PM) racke: long time no see
(10:16:22 PM) snewpy: finally noticed the email about the meeting before the end of it ;)
(10:16:40 PM) racke: it's quite early now, right
(10:16:46 PM) snewpy: 8am
(10:16:55 PM) snewpy: your email arrived at 1:33am :)
(10:17:05 PM) snewpy: how goes it?
(10:17:23 PM) racke: lots of things to do
(10:17:44 PM) racke: resurrect an Interchange site today from a broken server which had two RAID disks failure
(10:17:56 PM) snewpy: ouch!
(10:22:17 PM) racke: yes
(10:22:30 PM) racke: there was a backup :-D
(10:23:02 PM) racke: and I could recover by copying the binary MySQL files even without exact match of the binaries
(10:24:44 PM) racke: blueturtle: if i submit patches inside Interchange to modules written by you, should I send the patches to you so you can integrate them?
(10:25:27 PM) blueturtle: Please, as it usually turns out that little things need adding or updating (and big things adding)
(10:26:15 PM) blueturtle: so I'm forever fiddling around with things ..
(10:27:27 PM) racke: ok
(10:28:53 PM) racke: if you put the modules on kiwi, can I assume that I can include them in Interchange unless stated otherwise?
(10:29:47 PM) blueturtle: Definitely - though I doubt you want the OS/2 ones :)
(10:30:21 PM) racke: does still work - loooooooooooooong time ago that I worked on OS/2 machines 
(10:30:36 PM) snewpy: what's the status of the google checkout one... I know there's two modules floating around at the moment, mine and yours, Lyn
(10:31:10 PM) racke: yeah good question
(10:31:45 PM) blueturtle: Ah, at the end of the day the Google supplied libs are doomed, so using XML::Simple is the only way forward. I do intend adding in the vital missing 
(10:31:55 PM) blueturtle: feature, ie dynamic lookups for tax and shipping when a customer changes delivery address once at Google.
(10:32:08 PM) snewpy: i wrote mine mainly because the way Lyn's is disregards their (reasonably strictly enforced) policies about checkout flow
(10:32:26 PM) snewpy: mine is feature complete and compliant, but is using Google::Checkout
(10:32:42 PM) snewpy: but does mechant callback calculations for shipping & tax, etc
(10:32:44 PM) blueturtle: Fair comment, though there is an option to comply if you want to
(10:33:05 PM) blueturtle: Don't suppose we could join efforts and get one that does it all??
(10:33:15 PM) snewpy: of course :)
(10:33:31 PM) blueturtle: Seems a good option to me:)
(10:33:53 PM) snewpy: possibly a quick way is to just convert my module to use XML::Simple... it's otherwise feature complete
(10:34:22 PM) blueturtle: I haven't looked at either for quite a while, so I'm open to suggestions
(10:35:14 PM) blueturtle: Mine originally used the Google libs, but then I converted it, so I suppose it's really just the dynamic tax/shipping that needs doing
(10:35:35 PM) snewpy_ [n=Mark@2001:44b8:745b:aed0:e56c:8332:f1b8:6289] entered the room.
(10:35:43 PM) snewpy_: stupid ipv6 freenode server is so flakey
(10:37:33 PM) snewpy_: there's also a bug in IC that needs to be fixed for both modules
(10:37:46 PM) snewpy_: the workaround of setting MV_HTTP_CHARSET sucks
(10:37:50 PM) blueturtle: UTF8?
(10:38:05 PM) blueturtle: yes ..
(10:38:06 PM) snewpy_: no, the fact that it tries to split xml posts
(10:38:09 PM) snewpy_: let me fine the bit of code
(10:38:38 PM) snewpy_: if ($CGI::content_type =~ m{^(?:multipart/form-data|application/x-www-form-urlencoded|application/xml)\b}i) {
(10:38:45 PM) snewpy_:         parse_post(\$CGI::query_string)
(10:38:56 PM) snewpy_: parse_post shouldn't be called on application/xml content type
(10:39:22 PM) snewpy_: sorry, it's the next bit that's the prob
(10:39:26 PM) snewpy_: parse_post($h->{entity});
(10:39:36 PM) snewpy_: the query string should be parsed, but the post contents should not
(10:40:18 PM) blueturtle: I don't recall it splitting xml posts ? though it's a long time now ..
(10:40:24 PM) snewpy_: it looks for things that can legally exist in the xml document and throws errors 
(10:40:49 PM) snewpy_: take a look in Server.pm:map_cgi
(10:41:11 PM) matjones [n=chatzill@c-69-181-63-55.hsd1.ca.comcast.net] entered the room.
(10:41:20 PM) blueturtle: Ah, perhaps i recall having to counter something like that now ..
(10:41:34 PM) snewpy_: it makes google checkout responses break in subtle and hard to diagnose ways
(10:41:47 PM) racke: it is now in parse_cgi, snewpy_
(10:41:51 PM) rsiddall [i=189705da@gateway/web/freenode/x-pfzksfnktwlwjfil] entered the room.
(10:42:00 PM) snewpy_: but either way, IC should not be parsing application/xml post content like that
(10:42:05 PM) racke: hello matjones, hello rsiddall
(10:42:10 PM) matjones: hey stefan
(10:42:35 PM) snewpy left the room (quit: Read error: 60 (Operation timed out)).
(10:42:37 PM) snewpy_: racke: oops you're right
(10:42:39 PM) snewpy_ is now known as snewpy
(10:42:55 PM) racke: snewpy: so we are just doing an elsif for application/xml 
(10:42:57 PM) blueturtle: ah, i see that i put code into the violation.html page to counter that stuff ..
(10:43:14 PM) snewpy: racke: yes
(10:43:32 PM) snewpy: we could still parse the query string if Global::TolerateGet is set, but we shouldn't parse the post entity
(10:43:58 PM) racke: why don't you submit a patch?
(10:44:45 PM) snewpy: happy to.. have just been so busy :)
(10:47:06 PM) racke: ok
(10:48:17 PM) racke: so any opinion to the Encode::Alias mess? on the last meeting I suggested to put a local copy inside Interchange without the require
(10:49:45 PM) k2b3 left the room.
(10:57:41 PM) snewpy: I missed the original discussion... is there a link to it somewhere?
(10:59:21 PM) racke: http://ftp.icdevgroup.org/interchange/meeting/irc_meeting_20091124.txt
(11:00:05 PM) racke: search for #331
(11:02:07 PM) snewpy: we're fighting a perl bug with this and it goes away if we distribute a replacement ourselves?
(11:02:11 PM) snewpy: if so, sounds like a winner
(11:04:11 PM) racke: if goes away - though endpoint_david spread some rumours about it :-)
(11:04:36 PM) racke: it fixed my issues with the bug on different machines 
(11:06:05 PM) racke: so IMHO it is good enough to put it into 5.7 and it won't hurt other Perl scripts on the machine
(11:07:59 PM) snewpy: yeah, biting the bullet and fixing this safe mess isn't going to help us for 5.7 or 5.8
(11:08:34 PM) steamguy left the room ("No matter how dark the night, somehow the Sun rises once again").
(11:10:58 PM) snewpy: all these problems seem to come back to Safe, right?
(11:11:19 PM) racke: yes
(11:12:31 PM) snewpy: if we were to decide that we hated safe, didn't need the security compartmentalization any more, and ditched it... most of our problems disappear?
(11:13:36 PM) racke: that would be great :-/
(11:13:47 PM) docelic: MIke would not like it ;-/
(11:13:49 PM) racke: it would fix some problems
(11:13:53 PM) racke: hello docelic
(11:13:55 PM) snewpy: i know it's controversial ;)
(11:14:07 PM) racke: but it wouldn't fix the mail headers :-)
(11:14:14 PM) snewpy: mail headers?
(11:14:36 PM) racke: encoding of mail headers as per RFC
(11:15:31 PM) snewpy: umm isn't that a relatively simple proposition of just making sure everything uses mime encoded words?
(11:16:49 PM) racke: I thought so too, but couldn't get a patch out of it
(11:17:26 PM) racke: David and I searched for a solution: http://ftp.icdevgroup.org/interchange/meeting/irc_meeting_20091117.txt
(11:17:47 PM) snewpy: isn't there an Encode module that does it, too?
(11:18:00 PM) racke: yeah, but it didn't work for me
(11:18:14 PM) racke: somehow it produced double encoding
(11:18:28 PM) racke: of course that could be all my fault :-)
(11:20:20 PM) snewpy: using encode('MIME-Q', $subject) works as long as we know the actual encoding of the string, right?
(11:20:37 PM) snewpy: i tihnk there are more places than that where we need to know what kind of encoding it is anyway, right?
(11:21:27 PM) racke: well it didn't work for me
(11:23:54 PM) snewpy: if you paste text into a perl script to test something, you need to make sure that perl is reading the literals as UTF8
(11:23:54 PM) snewpy: maybe that's why your test didn't work?
(11:24:28 PM) snewpy: like a 'use utf8' up the top
(11:24:49 PM) racke: I put the code into Interchange, I didn't use a test script
(11:24:57 PM) snewpy: Note that if you have bytes with the eighth bit on in your script (for example embedded Latin-1 in your string literals), use utf8 will be unhappy since the bytes are most probably not well-formed UTF-X. If you want to have such bytes under use utf8 , you can disable this pragma until the end the block (or file, if at top level) by no utf8; .
(11:25:22 PM) snewpy: hmmm... this is the problem with the disaster that is UTF8.. next question is did your browser send it at utf8 ;)
(11:25:44 PM) racke: the browser isn't in the game with email headers
(11:26:01 PM) snewpy: how did you get that supposedly utf8 string into IC tho?
(11:26:11 PM) snewpy: put it in a page, which was possibly not read as utf8?
(11:27:02 PM) snewpy: maybe put a utf8::is_utf8 somewhere before the encoding to see if perl thinks so?
(11:27:37 PM) racke: (11:18:52 PM) racke: 127.0.1.1 esh7EqHi:127.0.1.1 - [17/November/2009:23:16:49 +0100] ulisses www.ulisses.exp:443/cgi-bin/ic/ulisses/checkout Subject: Bestellbestätigung F-Shop, U8: 1 (11:18:52 PM) racke: 127.0.1.1 esh7EqHi:127.0.1.1 - [17/November/2009:23:16:49 +0100] ulisses www.ulisses.exp:443/cgi-bin/ic/ulisses/checkout Subject: =?UTF-8?Q?Bestellbest=C3=83=C2=A4tigung?==?UTF-8?Q?=20F=2DShop?=. 
(11:27:58 PM) racke: so Perl thinks it is utf8, but the Subject is wrong
(11:30:22 PM) endpoint_david: classic case of the raw octets
(11:30:24 PM) endpoint_david: is that from the log files?
(11:30:55 PM) racke: yes
(11:31:04 PM) endpoint_david: maybe the log is just writing out utf8, but your viewer (less, grep, whatever) is trying to display as ascii (latin1, whatever)
(11:31:41 PM) endpoint_david: although the MIME-Q data is encoded wrong
(11:31:56 PM) endpoint_david: i.e,. it's encoded the already-encoded octets
(11:32:27 PM) snewpy: btw, what's wrong with that encoding?  it decodes correctly
(11:32:31 PM) endpoint_david: remind me, were you doing an encode('utf8') or fiddling with the encoding of the subject/header lines?  if so, that may be your problem, as you wouldn't need that
(11:32:48 PM) snewpy: use Encode qw(decode);
(11:32:49 PM) snewpy: my $decoded = decode("MIME-Header", "=?UTF-8?Q?Bestellbest=C3=83=C2=A4tigung?==?UTF-8?Q?=20F=2DShop?=.");
(11:32:49 PM) snewpy: print $decoded;
(11:33:05 PM) snewpy: seems to print something reasonable?
(11:33:14 PM) racke: this is the original log:
(11:33:18 PM) racke: 127.0.1.1 esh7EqHi:127.0.1.1 - [17/November/2009:23:16:49 +0100] ulisses www.ulisses.exp:443/cgi-bin/ic/ulisses/checkout
(11:33:18 PM) racke:  Subject: Bestellbestätigung F-Shop, U8: 1
(11:33:18 PM) racke: 127.0.1.1 esh7EqHi:127.0.1.1 - [17/November/2009:23:16:49 +0100] ulisses www.ulisses.exp:443/cgi-bin/ic/ulisses/checkout
(11:33:18 PM) racke:  Subject: =?UTF-8?Q?Bestellbest=C3=83=C2=A4tigung?==?UTF-8?Q?=20F=2DShop?=.
(11:33:29 PM) racke: so it is supposed to be UTF8
(11:33:40 PM) endpoint_david: yeah, but binmode(STDOUT, ':utf8'); it just happens to output the valid octets for the utf8 sequence
(11:34:00 PM) snewpy: but =?UTF-8?Q?Bestellbest=C3=83=C2=A4tigung?==?UTF-8?Q?=20F=2DShop?=. is valid MIME Q by the looks of it
(11:34:04 PM) racke: ??
(11:34:04 PM) endpoint_david: as an example, decode just the quoted chars, and check the length
(11:34:24 PM) snewpy: it might not be the most compact representation (i dont know.. haven't read the rfc), but it's valid and decodes
(11:34:32 PM) racke: snewpy: but ä is only two characters in UTF-8, not four
(11:34:42 PM) endpoint_david: my $decoded = decode("MIME-Header", "=?UTF-8?Q?=C3=83=C2=A4?=");
(11:34:47 PM) racke: and it is still crap
(11:34:54 PM) snewpy: racke: on my terminal it outputs two characters
(11:35:08 PM) snewpy: Bestellbestätigung F-Shop.
(11:35:10 PM) endpoint_david: length of the one character should be 1, not 2, but you'll get two, because it's doubly-encoded
(11:36:05 PM) endpoint_david: yeah, it's unfortunate, because the leading byte sequences of utf8 are all A + various accents
(11:36:12 PM) endpoint_david: (viewed as latin1)
(11:36:28 PM) snewpy: so it does, or does not, decode into the right string?
(11:36:33 PM) snewpy: is Bestellbestätigung F-Shop. correct?
(11:36:36 PM) endpoint_david: does not
(11:36:37 PM) racke: yes
(11:36:50 PM) snewpy: Bestellbestätigung F-Shop. is what it decodes into here
(11:36:52 PM) endpoint_david: it outputs on your term right, because you've got a utf8 term
(11:37:15 PM) snewpy: isn't that the point? ;)
(11:37:20 PM) ***racke head burns
(11:37:58 PM) endpoint_david: no, 'cause you're viewing in the email client
(11:38:19 PM) snewpy: which supports utf8, or is always going to have trouble displaying utf8 right?
(11:38:20 PM) endpoint_david: it "works" in the term, but if you check the length in perl, you're looking at teh characters for the raw octets
(11:38:58 PM) snewpy: I'm thoughroughly confused... isn't all that matters for e-mail headers that we create a correct representation of the UTF8 string as MIME Q?
(11:38:59 PM) endpoint_david: it's valid utf8, it just encodes the multi-byte sequence for the character you intended to display
(11:39:11 PM) endpoint_david: nstead of the character itself
(11:40:09 PM) snewpy: right.. it declares the charset as UTF-8 with =?UTF-8?Q? and then proceeds to encode multi-byte utf8 sequences.. what's wrong with that?
(11:40:22 PM) endpoint_david: the wrong thing is encoded
(11:40:40 PM) endpoint_david: it's akin to declaring the charset latin 1 and outputting utf8 octets
(11:41:31 PM) snewpy: what should it be encoded as?
(11:41:47 PM) endpoint_david: the Q encoding is right
(11:41:59 PM) endpoint_david: as in, it needs to be encoded via Q
(11:42:35 PM) endpoint_david: but the data going in should be a perl string, not an already-encoded perl string, as it appears the Q encoding encodes the output as utf8 already
(11:43:09 PM) snewpy: thunderbird decodes that string correctly
(11:43:58 PM) endpoint_david: encode_utf8(encode_utf8(encode_utf8('doofenschmïrtz'))) is a valid utf8 string, but probably not what you want to output to the user
(11:45:14 PM) endpoint_david: snewpy: that may be due to the prevalence of non-conforming mail output programs :-).  IE renders a lot of crap html, too... :-)
(11:46:00 PM) snewpy: I think I must be missing the point.. which of my assumptions are wrong... the source string is valid UTF8, it's correctly represented by a valid utf8 byte sequence for the characters in question, mime-q encoding has encoded the input faithfully, and everything seems to decode it faithfully
(11:46:03 PM) endpoint_david: racke: do you have the original code you were noodling with
(11:47:00 PM) endpoint_david: 0) the source string represents the data you originally intended
(11:47:26 PM) racke:    $subject = encode('MIME-Q', $subject);
(11:47:29 PM) racke: that's all
(11:47:38 PM) endpoint_david: where was $subject from?
(11:48:00 PM) endpoint_david: at this point, it looks like $subject is a decoded utf8 string, not a perl string
(11:48:20 PM) endpoint_david: we should be encoding the perl-internal string, not the encoded representation
(11:48:42 PM) racke: now you lost me :-)
(11:49:04 PM) endpoint_david: is there a code sample somewhere, or can you point me to the file in question?
(11:49:18 PM) snewpy: so your problem is that you want ä stored not as double byte chars?
(11:51:10 PM) snewpy: i just picked those chars out from the windows 7 character map and saved them in a utf8 encoded document with windows notepad
(11:51:39 PM) snewpy: and as hex in the file, they are c383 and c2a4
(11:51:41 PM) endpoint_david: racke: sorry, prior to the encode, it looks like $subject is an encoded utf8 string, not a perl string
(11:52:01 PM) racke: what is the difference ?
(11:52:11 PM) snewpy: which would match what we're seeing
(11:52:30 PM) racke: ok but shouldn't is_utf8 return 0 in that caseß
(11:52:54 PM) snewpy: why? the bytes c3 83 c2 a4 are UTF-8 multibytes for those two characters
(11:52:54 PM) racke: btw: I think in my case the string came from locales.txt
(11:53:03 PM) racke: which is flagged as UTF-8
(11:53:06 PM) jeff_b left the room (quit: "Leaving.").
(11:53:12 PM) snewpy: so it is utf-8, so is_utf8 should be true
(11:53:18 PM) endpoint_david: no
(11:53:28 PM) endpoint_david: is_utf8 shows if the SV_UTF8 flag is set
(11:53:33 PM) endpoint_david: which means that it's in perl's internal format
(11:53:51 PM) endpoint_david: (which happens to be utf8, but that's besides the point)
(11:54:17 PM) snewpy: perl's internal format is undefined (but utf8 on most platforms)
(11:54:25 PM) endpoint_david: ok, hand-waving details
(11:54:26 PM) snewpy: that's why we mess around with encodes and decodes, right?
(11:54:39 PM) snewpy: it's not utf8 on all platforms tho.. that was my point in hand-waving :)
(11:54:53 PM) endpoint_david: I hadn't heard that specifically, but for everything not EDCBIC (or whatever) I'd expect it to be utf8
(11:55:10 PM) snewpy: is_utf8 is " Test whether STRING is in UTF-8 internally. Functionally the same as Encode::is_utf8()."
(11:55:18 PM) snewpy: since perl 5.8.1
(11:55:21 PM) endpoint_david: in most cases, decode_utf8 will essentially flip the bit
(11:55:36 PM) endpoint_david: and leave the byte array along, it jsut changes the sematics of character interpretation
(11:56:13 PM) snewpy: you said above that the problem is that this string is not correctly represented.. how do you want it to be represented when it's fed into that function?
(11:57:28 PM) endpoint_david: it should be in perl's internal format (i.e., is_utf8 set, or ascii), as the MIME-Q encoding handles the encoding to utf8 itself
(11:57:53 PM) snewpy: which would represent those characters as C3 83 C2 A4, right?
(11:58:15 PM) endpoint_david: so I'd say either this source string got read in raw, or got read in and successfully converted into perl's internal encoding and somewhere along the line got encoded back into utf8
(11:58:19 PM) endpoint_david: no, it's a single character
(11:58:26 PM) endpoint_david: you've encoded the encoding
(11:58:33 PM) snewpy: for which =C3=83=C2=A4 would be a sensible output, just the same as =20F is for a space?
(11:58:48 PM) snewpy: it's a single character which cannot be represented in a single UTF-8 byte
(11:58:49 PM) racke: so how do i get back from into Perl's internal format?
(11:59:03 PM) snewpy: you're wanting it to be stored in iso8859-1 or something?
(11:59:20 PM) snewpy: which single byte in utf8 represents either of those characters?
(12:00:23 AM) endpoint_david: in this case, the single character is represented as two bytes in utf8, which will output as =CX=XX in the MIME-Q
(12:01:05 AM) snewpy: yes... and that is a problem?
(12:01:20 AM) endpoint_david: no, but when one char in -> 4 bytes out, it is
(12:01:30 AM) endpoint_david: when the 1 char has  2-byte representation
(12:01:54 AM) snewpy: ä is not what should be output?
(12:02:28 AM) endpoint_david: snewpy: are you testing in an email client?
(12:02:33 AM) endpoint_david: or jsut via the shell?
(12:02:44 AM) snewpy: both.. but i'm trying to understand the parameters of the problem
(12:03:47 AM) snewpy: in both i see Bestellbestätigung F-Shop. as the end result... is that correct?
(12:04:30 AM) snewpy: the two funny chracters being an upper case A with an accent over it, and a clittle circle with a 45 degree angled little line coming from all four corners
(12:04:37 AM) snewpy: (just in case irc is munging it)
(12:04:41 AM) endpoint_david: snewpy: perl -MEncode -e 'binmode(STDOUT, q{:utf8}); print decode("MIME-Header", "=?UTF-8?Q?=C3=83=C2=A4?=")'
(12:04:48 AM) endpoint_david: illustrates the issue
(12:04:54 AM) endpoint_david: even in a utf8 term
(12:06:42 AM) endpoint_david: that's not the output you want for the 'ä' char
(12:07:30 AM) endpoint_david: so I reiterate the above: so I'd say either this source string got read in raw, or got read in and successfully converted into perl's internal encoding and somewhere along the line got encoded back into utf8
(12:07:40 AM) endpoint_david: this is what we need to fix; the rest of the pipeline is ok
(12:07:47 AM) snewpy: or there is an output encoding problem?
(12:07:58 AM) endpoint_david: not in the case of MIME-Q
(12:08:14 AM) endpoint_david: it returns ASCII
(12:08:31 AM) ***endpoint_david needs to start dinner
(12:08:35 AM) snewpy: but hang on.. the example you showed me just now...
(12:08:43 AM) snewpy: that's the decode function being screwy
(12:08:54 AM) snewpy: the MIME Q content is plainly correct on it's face
(12:08:54 AM) endpoint_david: no
(12:09:15 AM) snewpy: =C3=83=C2=A4 is the right byte sequence for those two characters
(12:09:24 AM) endpoint_david: that's perl showing you the characters you're outputting, not the bytes (getting interpreted as characters by your term)
(12:09:39 AM) endpoint_david: snewpy: you don't need to encode it twice
(12:09:47 AM) snewpy: what are we encoding twice?
(12:09:50 AM) endpoint_david: mime-q takes care of the encoding
(12:10:24 AM) snewpy: MIME Q works by printing ASCII printable characters for more complex encodings.. for utf-8 it does this by outputing the bytes, right?
(12:10:51 AM) racke: no
(12:10:53 AM) snewpy: just the same as =20 is a space, =C3=83 is ?
(12:11:08 AM) racke: oh wait
(12:14:45 AM) snewpy: the problem the example above illustrates is that perl does something stupid when *decoding* that string.. like it doesn't set perl's internal representation of the string right, or forgets to flip a flag
(12:14:45 AM) snewpy: the actual MIME Q being decodes is valid and correct
(12:14:45 AM) endpoint_david: racke: is this email.tag, or where is this?
(12:14:45 AM) endpoint_david: I can probably come up with a patch that'll work
(12:14:58 AM) racke: send_email in Util.pm
(12:15:22 AM) racke: that would be great :-)
(12:16:25 AM) racke: ok, I'll leave in a few minutes ... I need to get up quite early
(12:16:27 AM) snewpy: I don't understand how representing ? as anything other than =C3=83 could be right?
(12:16:30 AM) endpoint_david: snewpy: this is the actual encoding you want:
(12:16:31 AM) endpoint_david: perl -MEncode -Mutf8 -e 'print encode(q{MIME-Q}, q{ä})'
(12:16:39 AM) endpoint_david: =?UTF-8?Q?=C3=A4?=
(12:16:58 AM) endpoint_david: anything else is wrong
(12:17:20 AM) endpoint_david: (sans a different encoding in the =?UTF-8?Q? section...)</pedantic>
(12:18:15 AM) snewpy: why is that different to the unicode bytes for that string?
(12:19:12 AM) endpoint_david: it's a straight quoted-printable of the encoding
(12:19:31 AM) racke: only ASCII in the header!
(12:19:37 AM) endpoint_david: right
(12:19:56 AM) endpoint_david: but it's a way of specifying the encoding that is being represented
(12:20:28 AM) snewpy:  perl -MEncode -Mutf8 -e 'print decode(q{MIME-Q}, q{=?UTF-8?Q?=C3=A4?=})'
(12:20:42 AM) snewpy: gives the wrong characters back
(12:20:56 AM) endpoint_david: no, you're still in perl's internal format
(12:21:27 AM) endpoint_david: you'd need an encode_utf8 around that to get output that your term can read, or a binmode(*STDOUT, q{:utf8});
(12:22:12 AM) endpoint_david: -Mutf8 only influences the interpretation of the program text
(12:22:34 AM) endpoint_david: it doesn't change anything out *STDOUT or *STDIN (I think, might be wrong)
(12:23:25 AM) endpoint_david: yeah, it's literally only the program text
(12:23:47 AM) endpoint_david: -e lines get turned into program text automagically for you, which is why this does what it does
(12:24:05 AM) snewpy: thunderbird and outlook both display it same as my terminal
(12:24:18 AM) snewpy: as ?
(12:24:25 AM) snewpy: not ä
(12:24:52 AM) endpoint_david: that's becuase you don't have the right output yet
(12:24:59 AM) endpoint_david: it's supposed to be ä
(12:25:00 AM) endpoint_david: :-)
(12:25:06 AM) racke: you can't trust the MUAs
(12:25:16 AM) snewpy: right, so your encoding is wrong, at very least in practice
(12:25:37 AM) racke: ok, good night see you
(12:25:41 AM) snewpy: i opened a telnet to port 25, and crafted an email with Subject: =?UTF-8?Q?=C3=A4?=
(12:25:49 AM) endpoint_david: see 0); the encoding is right, but what you're actually encoding and what you think you're encoding is wrong
(12:25:59 AM) snewpy: in both thunderbird and outlook on windows, it displays ?
(12:25:59 AM) endpoint_david: •differ