smb: Header field contains control characters issue

I doubt the control characters were in the header field(s) as a part of import. But if they were, that'd be a bug.

Instead, more likely, the control characters are a result of some other corruption to the message base that occurred after the message was imported. Running chksmb with the '-e' option and reporting the results would help me to know.

As per the request "Can we have fixsmb strip out invalid control characters from the header" - it could, but that's likely only a symptom of the problem. Looking at the header fields of the message(s) in question would be a good idea to determine if the header is even recoverable. Simply stripping control character isn't likely what we actually want to do in this case.

As requested

Checking dove-syncops Headers

93%  #14922 (3DA220) SYS64738                  Header field contains control characters

message base     dove-syncops
To               Digital Man
Subject          Re: Mail Server Issue
Sender           SYS64738
SenderNetAddr    REPLY: <66A92984.55864.SYNC_SYS@VERT.SYNCHRO.NET>
@TZ: C168
> IF IT'S EXACTLY THE SAME PROBLEM, THEN YOUR PASSWORD IS STILL WRONG. I
> SUSPECT IT IS NOT THE "SAME PROBLEM", SO POST THE NEW LOG
SenderNetType    Internet
Message-ID       <66A9C774.764.998internetdiscussi@bbs.tangonine.net>
X-FTN-MSGID      55877.sync_sys@1:103/705 2b102c17
X-FTN-PID        Synchronet 3.20a-Linux master/d3c114993 Feb  1 2024 GCC 12.2.0
X-FTN-CHRS       CP437 2
OtherHeader      WhenImported: 20240731041114-0700  c1e0
OtherHeader      WhenExported: 20240731041558-0700  c1e0
OtherHeader      ExportedFrom: VERT sync_sys 55877
OtherHeader      @REPLY: <66A92984.55864.SYNC_SYS@VERT.SYNCHRO.NET>
OtherHeader      @TZ: C168
OtherHeader      > IF IT'S EXACTLY THE SAME PROBLEM, THEN YOUR PASSWORD IS STILL WRONG. I
OtherHeader      > SUSPECT IT IS NOT THE "SAME PROBLEM", SO POST THE NEW LOG
OtherHeader      WhenImported: 20240731141116+0900  121c
OtherHeader      WhenExported: 20240731201115+0900  121c
OtherHeader      ExportedFrom: TANGO99 998internetdiscussi 764
OtherHeader      @REPLY: <66A92984.55864.SYNC_SYS@VERT.SYNCHRO.NET>
OtherHeader      @TZ: C168
OtherHeader      > IF IT'S EXACTLY THE SAME PROBLEM, THEN YOUR PASSWORD IS STILL WRONG. I
OtherHeader      > SUSPECT IT IS NOT THE "SAME PROBLEM", SO POST THE NEW LOG
SenderNetAddr    REPLY: <66A92984.55864.SYNC_SYS@VERT.SYNCHRO.NET>
@TZ: C168
> IF IT'S EXACTLY THE SAME PROBLEM, THEN YOUR PASSWORD IS STILL WRONG. I
> SUSPECT IT IS NOT THE "SAME PROBLEM", SO POST THE NEW LOG
SenderNetType    QWKnet
when_written     66A983A8 0000 Tue Jul 30 19:22:00 2024 UTC
when_imported    66AA1CFB C168 Wed Jul 31 06:16:11 2024 CDT
type             0000h
version          0300h
attr             0000h ()
auxattr          00000000h ()
netattr          00000000h ()
header_offset    3DA220h
header_fields    25
header_length    1200 (calc: 1200)
number           14922
thread_id        14922
data_offset      48B600h
data_field[0]    TEXT_BODY, offset 0, length 67

100%

Checking dove-syncops Index

100%

Checking dove-syncops Hashes

100%

Status Total                        (=): 5165
Total Indexes                       (=): 5165
Active Indexes                      (=): 5165
Active Headers                      (=): 5165
Active Header Blocks                ( ): 15514        3,971,584 bytes used
Active Data Blocks                  ( ): 18174        4,652,544 bytes used
Header Records                      ( ): 5636
Deleted Indexes                     ( ): 0
Deleted Headers                     ( ): 471
Deleted Header Blocks               ( ): 1446           370,176 bytes used
Deleted Data Blocks                 ( ): 1755           449,280 bytes used
Oldest Message (import)             ( ): 965 days (0 max)
Largest Message (data)              ( ): 72125 bytes (#13716)
Control Characters in Header Fields (!): 1

dove-syncops Message Base has Errors!

Total Deleted Messages                 : 471            819,456 bytes used

That message is well corrupt

I recommend deleting the message and then let me know if this happens again (is reproducible).

Now it seems even more screwed up.

92%  #14922 (3DA220) SYS64738                  Header field contains control characters

So if I use smbutil to try and find 14922

4756/#14921 Errol Casey               phigan                    Cannot see all message boards
4757/#14923 Denn                      Errol Casey               Cannot see all message boards
4758/#14924 Errol Casey               Denn                      Cannot see all message boards
4759/#14925 SYS64738

It's not there.

So let's run fixsmb

smb_getmsghdr corrupt message header ID (04 00 00 00) at offset 3987232
92%  #14922 (3DA220) SYS64738                  Not indexing deleted message

Says it's deleted. So let's make sure

bbs@bbs:/sbbs/data/subs/dovenet$ smbutil mp1000 dove-syncops.shd

SMBUTIL v3.20-Linux master/d232ecdc6 SMBLIB 3.00 - Synchronet Message Base Utility

Opening dove-syncops
Maintaining dove-syncops
Maintaining dove-syncops hash file
Loading index...

Done.

Scanning for pre-flagged messages...
100% (0 pre-flagged for deletion)
Scanning for read messages to be killed...
100% (0 flagged for deletion due to read status)
dove-syncops locked successfully
Packing dove-syncops
Analyzing dove-syncops
820992 less than 1024000 compressible bytes.

dove-syncops unlocked successfully

Nothing flagged for deletions so let's double check with another chksmb

Checking dove-syncops Headers

92%  #14922 (3DA220) SYS64738                  Header field contains control characters

Still there. I don't seem to be able to get rid of it.

Use 'smbutil p' (pack) the message base to remove deleted messages.

Might need to use 'smbutil -a p' to force the pack.

Yes, the -a p did it. Thank you. I'll maybe run an smbchk every hour and have it flag me if there's corruption. Try and catch it quick.

I'll make a change to chksmb to not check header fields of deleted messages for ctrl chars

Cool, while you're diddling with smb stuff, could you have smbutil l show the type of message if it's not a netmail, for example and UPVOTE or DOWNVOTE instead of who it's too. The blanks confused me for a bit. Also a date would be good, either date written or date imported.

I can open a separate issue for that.

mentioned in commit 0fd380f8

mentioned in commit d84b8e3d

I think we can close this issue now.

chksmb -e is no longer displaying extended information.

$ chksmb dove-techtalk -e -p -b

CHKSMB v3.20-Linux master/d84b8e3d6 SMBLIB 3.00 - Check Synchronet Message/File Base

Checking dove-techtalk Headers

35%  #86    (02F120) TLDR                      Header field contains control characters
100%

Checking dove-techtalk Index

100%

Checking dove-techtalk Hashes

100%

Status Total                        (=): 230
Total Indexes                       (=): 230
Active Indexes                      (=): 230
Active Headers                      (=): 230
Active Header Blocks                ( ): 2104           538,624 bytes used
Active Data Blocks                  ( ): 66765       17,091,840 bytes used
Header Records                      ( ): 230
Deleted Indexes                     ( ): 0
Deleted Headers                     ( ): 0
Deleted Header Blocks               ( ): 0                    0 bytes used
Deleted Data Blocks                 ( ): 0                    0 bytes used
Oldest Message (import)             ( ): 249 days (0 max)
Largest Message (data)              ( ): 6308005 bytes (#88)
Control Characters in Header Fields (!): 1

dove-techtalk Message Base has Errors!

'fixsmb' can be used to repair many message/file base problems.

@rswindell prod?

I'm not sure what you're asking here. The '-e' option never had any impact on the output of the detected header fields with control characters. "no longer" does not apply here. Are you asking for chksmb to report more information for those particular detected msg header problems when using the '-e' option?

Okay, I think I see now what you're reporting: any/all problematic message headers are indeed dumped when using the '-e' option (even those with unexpected ctrl chars in them), but you have to specify the '-e' option before the message base(s) on the chksmb command-line.

smb: Header field contains control characters issue

Designs

Child items ...

Activity