NNTP Unicode messages getting mangled

• Post UTF-8 message with several emoji on via NNTP (3.18)
• Message transmitted to VERT via QWK/FTP
• Message received on second system via QWK/FTP (3.17)
• Viewing message on second system, emoji are mangled/lost

This message is in question is under DoveNet Network Sysops.. the original message didn't seem to make it through, but my reply did... which may be a different bug/issue.

changed title from Unicode messages getting mangled to NNTP Unicode messages getting mangled

added bug net-QWK nntp-service labels

Synchronet v3.17 didn't support UTF-8.

It's also mangled in a local terminal (3.18) that is detecting as utf-8 for the terminal.

Is your news client including the MIME "Charset" message header to specify that the character set is UTF-8?

It's header that would looks like this: Content-Type: text/plain; charset=UTF-8

If this is present, then Synchronet knows it's a UTF-8 encoded message.

Also, this code in newsutil.js automatically detects posted NNTP message text as valid UTF-8 and sets a header field accordingly:

	if((!hdr.extra_headers || !hdr.extra_headers["content-type"]) && !str_is_ascii(body) && str_is_utf8(body))
		hdr.ftn_charset = "UTF-8 4";

This should be working to auto-detect a valid NNTP-posted UTF-8 message if if your news reader is not specifying the charset in the message headers.

I don't see any evidence of any "mangling" going on.

Headers below...

From: "Tracker1" <tracker1@roughneckbbs.com>
To: Tracker1
Subject: Re: Unicode test
Message-ID: <5FA33EC5.703.dove-ops@roughneckbbs.com>
Date: Wed, 4 Nov 2020 16:52:37 -0700
X-Comment-To: Tracker1
Path: roughneckbbs.com!not-for-mail
Organization: Roughneck BBS
Newsgroups: DOVE-Net.DOVE-Net_Sysops
In-Reply-To: <5F9B78F8.696.dove-ops@roughneckbbs.com>
References: <5F9B78F8.696.dove-ops@roughneckbbs.com>
X-FTN-PID: Synchronet 3.18b-Win32  Sep 20 2020 MSC 1927
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:78.0) Gecko/20100101Thunderbird/78.4.0
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8; format=flowed
Content-Language: en-GB
Content-Transfer-Encoding: 8bit

And when you view the headers of the corresponding message on your BBS, can you paste that here as well?

A bit of follow-up: I think the ; format=flowed following the charset is throwing-off one checker for UTF-8 content in SMBLIB.

When I use a utf-8 detected terminal to connect (synchronet detects/sets utf-8 auto), reading the message it's mangled... I do have a terminal font that will do the emoji and other glyphs (fira code).

When you say "reading the message", I would need more details there. There are at least 3 ways to read a message via the Terminal server:

1. The built-in message reading interface
2. Third-party message reader (e.g. DDMessageReader)
3. My msglist.js module

The msglist.js module will tell you what the detected character set/encoding is and has useful debug options, like a hex dump of the message body text.

Or you could use smbutil to read the message headers, body text or pipe through a hex dump. All of these could be helpful in root-causing the issue.

Pretty sure I see the issue, you don't need to respond to this comment.

mentioned in commit a852cced

mentioned in commit ad79c591

Should be now fixed, in a couple of ways.

closed