<host>/error
.../spam
.../hack

Requires libmosquitto (e.g. install mosquitto-dev pkg) and set
in ctrl/main.ini:
[mqtt]
Enabled=true
broker_addr=127.0.0.1
broker_port=1883
qos=0
keepalive=10

Next up: Windows support

When the systemd dev package (e.g. libsystemd-dev) is installed, we can use
sd_notify() to inform systemd of Synchronet server state/status. The state
values (READY, STOPPING, RELOADING) are "well known" while the status
value is a free-form text string (the last lputs output).

Eliminated the free-form status() callback from *startup_t.
Eliminated the now-redundant started() callback from *startup_t.

Eliminated the use of sbbs_status.c/h as nothing is using that interface
and while I do plan on using a remote control/monitor interface, I'm
considering use of a standard pub/sub lib and protocol.

This commit is going to break the Windows sbbsctrl build for sure.
Probably break the sbbsNTsvcs build.

Upload and Download ARS must be matched *and* the directory's upload and download ARS.

Operator and Exemption ARS must be matched *or* the directory's operator and exemption ARS.

Also resolved some 32 vs 64-bit 'long' issues/ambiguities that have long-remained. :-)

This commit also removes logon.lst file support.

There's a TODO block remaining in js_user.c for setting portions of a user's birthdate (e.g. just the year or month or day).

1. Was not setting f->dir to the correct directory number, so only ftp-uploads to the *first* directory (dirnum = 0) would extract DIZ files of uploaded files.

Removing the 'dirnum' parameter to addfile() since that implied that you did not have to initialize the 'dir' element of the passed file_t, but you do: to get the correct file path for file size/date detection and the DIZ extraction.

2. Was getting heap-corruption when freeing the imported/formatted DIZ text on Windows once the above problem was fixed: can't free() in one DLL memory that was allocated in another DLL. Created and now using free_diz() to free the memory allocated in read_diz().

format_diz() handles a NULL 'lines' argument correctly/gracefully, so no need for the NULL lines check in sbbs_t::uploadfile().

Added FTP server log messages for successful file upload or update by user.

*/dsts.dab (daily statistics and running totals) -> */dsts.ini
*/csts.dab (cumulative statistics / log) -> */csts.tab

* dsts.ini now has both daily and total stats for all fields (not just timeon and logons).
* dsts.ini is now an easily modifiable text file - no longer need dstsedit (here-by deprecated and soon to be deleted)
* dsts.ini and csts.tab support 64-bit upload/download byte stats and are very extensible for future fields to be added or extended > 32-bit (this was the main inspiration for this overhaul, but it was overdue and already designed for v4, pretty much)
* csts.tab is a tab-delimited fixed length record format suitable for easy import to a spreadsheet program or parsing with scripts. Each day is a 128-character LF-delimited record with tab-delimited fields of plain ASCII text.
* All fields except timeon in dsts.ini files are updated immediately and by more non-terminal servers (e.g. post statistics from web UI scripts).
* New user stats are tracked more than just for "today".

The upgrade of these files is automatic and built-into SBBS.

Still to do: overhaul the slog utility to support the new csts.tab file format.

Credits and daily free credits are accurate to the byte up to (a maximum) of 18446744073709551615 (that's 18 Exbibytes - 1).

User's upload and download byte stats are now similarly extended in maximum range, but the accuracy is only "to the byte" for values less than 10,000,000,000. Beyond that value, the accuracy declines, but is generally pretty damn accurate (to 4 decimal places beyond the nearest multiple of a power of 1024), so I don't expect that to be an issue. This method of storing upload/download byte stats allowed me to use the same 10-character user record fields in the user.dat file.

As a side-effect of this enhancements:
* User and file credit values are now expressed in multiples of powers of 1024 (e.g. 4.0G rather than 4,294,967,296).
* Free credits per day per security level has now been extended from 32 to 64-bits (to accommodate values >= 4GB).
* adjustuserrec() now longer takes the record length since we can easily determine that automatically and don't need more "sources of truth" that can be out-of-sync (e.g. the U_CDT field length going from 10 to 20 chars with this change).
* setting the stage for locale-dependent thousands-separators (e.g. space instead of comma) - currently still hard-coded to comma
* more/better support for files > 4GB in size (e.g. in the batch download queue)
* user_t ulong fields changed to either uint32_t or uint64_t - I didn't realize how many long/ulong's remained in the code (which are sometmies 32-bit, sometimes 64-bit) - ugh
* Steve's ultoac() function renamed to u32toac() and created a C++ wrapper that still uses the old name, for homage

To fully support files > 4GB in size in file bases, credit values larger than 32-bits must be supported too.

There's a couple of todo comments/items included in this commit, but that's mainly to do with messages (which don't really have costs anyway).

The main thing to deal with now is the fact that users can't have more than 4GB in credits in the first place! That's got to be fixed next.

By setting sbbs.ini [web] FileIndexScript to an SSJS or XJS script filename, that script (by default, from your exec directory) will be executed when a file area/base listing has been http[s]-requested. File area/base requests are of the form <vpath_prefix> (for the list of libraries), <vpath_prefix>/<lib-name>/ (for list of directories of a library) or <vpath_prefix>/<lib-name>/<dir-code-suffix>/ (for a list of files in a directory). The new http_request "lib" and "dir" properties indicate that a library or directory listing was requested (if neither are defined, that's a request for the root / list of libs). The same configured script is executed to handle all 3 types of index/list requests.

A sample script (webfileindex.ssj) will be committed soon.

Authentication (via HTTP-AUTH) will be required if user #0 does not have access to all libraries or all directories within a required library.

file_area.lib[].link has been changed from "/<vdir>/" to just "<vdir>" (no slashes) and renamed to "vdir".

file_area.dir[].link has been changed from "/<vpath>/" to "<vpath>/" (no leading slash) and renamed to "vpath".

Added file_area.dir[].vdir property that contains just the directory's virtual directory name.

I don't think anyone was using these "link" properties since the dynamic FTP HTML index scripting feature is no longer supported.

Added can_user_access_lib() to insure that the user has access to at least one directory of a library before allowing access to the library (e.g. via JS). Something similar should be created for message groups.

Tired of being reminded that you were "awarded 0 credits"  for free downloads? I know I am, so I created another text.dat string (reusing Unused300), for notification of free-download files where no credits are awarded to the uploader. Also updated the default colors of the existing DownloadUserMsg string to have a little more variety.

Not sure why ftpsrvr.c isn't using user_downloaded_file(). That copy/pasta should be eliminated later.

As Andre pointed out while documenting this setting on the wiki, the option seemed confusing: if a sysop could not login with "system operator access", how could they login at all? Answer: they could not.

This setting used to be called "Allow Remote Sysop Logins", back when there was the concept of a "local login", so setting this option to "No" would mean that user accounts with sysop access could only be used for *local* login. But in Synchronet v3, there's really no such concept as a "local login", so it was changed to just "Allow Sysop Logins" (period) and not a lot of thought given to how/why a sysop would actually set to this "No" or what the implications would be (presumably, nobody ever sets this to "No").

So rather than just get rid if the option altogether, I changed it to mean: an account with sysop access (i.e. level 90+) can still login, but any action that normally requires the system password will not be allowed. This includes the sysop-actions available in the FTP server when authenticating with <user-pass>:<system-pass> as the password. The sysop-user can still authenticate (and login), but none of those sysop-actions will be available to them.

Make it more accurate/clear:
"attempted to upload invalid path/filename"

The logged error "!attempted to upload to invalid directory" did log the actual path that was attempted to be uploaded.

Added some quotes around other logged paths.

This change is just for internal consistency and convenience right now: the lib_t.vdir is a "sanitized" copy of the lib's short name (spaces are converted to dots or underscores based on the logic that the FTP server used in dotname()) and the dir_t.vdir is just a pointer to the dir's code_suffix. No other permutations are made (e.g. lower-casing the strings). Although the virtual directory names of libraries will now appear in mixed case in the FTP server (previously, they were all lowercase), the directory names are actually treated case-insensitively, so it should not make any difference. If forced-lowercase is preferred for some reason, please speak up.

This change leads the way to eventually, possibly, making these virtual path elements sysop-configurable. For now, it's just better to have a *copy* of the lib's short name that is appropriately modified to make a suitable directory name and have that vpath element available globally (to all servers and services) in a consistent manner.

So Nelgin asked (about filebase access via http), what if the library short name has a space in it? The answer now is, the spaces are replaced with a '.' or '_' (if there's already dots in the name).

By setting SCFG->File Options->Web File Virtual Path Prefix to something (e.g. "/files/"), all HTTP or HTTPS requests to the Synchronet Web Server with request paths beginning with this prefix will be interpreted as filebase access requests (with full access control enforcement). This is configured here (in SCFG) rather than, say, the [web] section of sbbs.ini, because I have plans for the terminal server to use this prefix to generate Web-URLs for files to display or email to users.

Currently, only requests to *files* (for download) are supported (no index generation, file information, etc. and definitely no upload support). Full access control (using HTTP auth, not cookies) is used for libraries and directories with controlled access. Credits are deducted and awarded and uploaders are notified of downloads, as one would expect. Requests to any dynamic-web-content files (e.g. .SSJS, .XJS, etc.) will be treated as static file download requests (no script will be executed).

I'm reusing the same virtual path parsing logic from the FTP server (moved to the userdat lib), so the virtual path to a file for download would be, for example, http://yourdomain/files/lib/dir-code/filename.ext

The main motivation for this feature is: FTP-links in email and web pages are just not useful to many users these days and I don't think that sysops should have to rely on a SSJS web UI (e.g. ecWebv4, cool as it is), to provide web-access to the filebases. Using this feature, you can share simpler/shorter web links to your files that will be more enduring.

Prefixed or trailing white-space characters would be hard to discern without this.

Don't treat CRYPT_ERROR_COMPLETE (-24) as a socket error during upload since it's an indication that the remote closed the connection and is the normal "end of file/transfer" indicator, not an error. 'rd' is already 0 in this case, so no need to set at all (since recv() returns 0 upon disconnect and that's what we're emulating here).

Fixes issue #309 reported by Jas Hud.

Midnight Commander (mc) apparently sends requests like this for files
(e.g. aliases) in the virtual root directory.

Fixes another part of the reported issue #288.

ftpalias() can return true even when the directory is not set to a valid
directory index (i.e. set to -1), so using as an array index would definitely
segfault. Part of commit 8ad30b6c by Deuce 3 years ago.

I didn't test this as I'm not sure exactly the combination of ftpalias.cfg
content and FTP command received that would trigger this, but it's most
definitely a bug.

So should fix the segfault reported in issue #288.

It's anticipated that this will be used for JS-populated file metadata in JSON format in the future (and not just "archive contents" in .ini format).

Also, fix the double-free issue that was occurring when moving files with extended file descriptions (sbbs_t::movefile()). This was actually the primary problem I was fixing here, but noticed the metadata issue: metadata would not have been moved along with the other file info between bases.

We can't (apparently) only rely on the return value of start_tls(), we have to check the value of the crypt session too.
This fix the possibility of this happening:
Jun  7 18:07:26 sbbs synchronet: ftp  0058 TLS ERROR 'No permiss.to perform this operation' (-21) opening keyset
Jun  7 18:07:26 sbbs synchronet: ftp  0058 <192.168.1.25> initialized TLS successfully

Instead, we'll detect the failure and disable FTPS support, logging "failed to initialize TLS successfully".

5 options:
- Safest Subset
- Most ASCII, Excluding Spaces (the default)
- Most ASCII, Including Spaces
- Most CP437, Excluding Spaces
- Most CP437, Including Spaces

sbbs_t::checkfname() now checks the file.can too.
new filedat.c functions:
- safest_filename() - not currently used
- illegal_filename() - returns true for a highly-suspicious (e.g. hack attempt) filename
- allowed_filename() - returns true if the filename is good for upload (assumed to be already checked to be legal as well).

Importantly, filenames beginning or ending in a '.' are now unallowed:
- 'dot files' are hidden (by default) on *nix
- files ending in a '.' are problematic on Windows

Inspired by Blocktronics (and other ANSI art group) packs' FILE_ID.DIZ/ANS files:
* Support (and prioritize) FILE_ID.ANS
* Convert ANSI color/attribute sequences in DIZ files to Ctrl-A equivalent (uses SAUCE width and ICE color, if specified)
* Don't treat DIZ as a series of lines, they're not always nowadays.
* New putmsg() mode: P_INDENT to print files indented by current column
* Display full (up to 64-char) filenames in lists when using 132+ column terminal.
* Use the Author, Group, and Title fields from the SAUCE if present/non-blank
* 2 new text.dat strings: 301 (FiAuthor) and 302 (FiGroup)
* Also fix bug with repeated Cost header field on bulk-uploaded files.

I know this'll break the *nix build (sauce.c dependency), but I'll fix that next.

Increase total extended description length from 1024 to 4000 characters. Perhaps this should be configurable?

There was a bug with reloading the configuration files in sbbsctrl where the sound button no longer reflected the truth and the sysop's previous click-state of the button was lost. Rather than going through writing the OPT_MUTE flag to the Options fields of all the sections of the sbbs.ini and then re-loading that file as a result, just do like we did with the sysop chat availability: use a semfile. So much simpler.

If anyone ever needs instance-specific muting, we can create/check instance/host-specific mute semfiles then. Doubt that'll happen though.

Also, removed the old sysavail control methods of ntsvcs too.

A "hack attempt" sound file is now supported in the Terminal Server, Mail Server,  and Services.

"login" and "logoff" sound files are now supported in the Terminal Server, FTP Server, Web Server, Mail Server, and Services.
This enhancement fixes Issue #157

The following sound files may now be configured in the [Global] section of the ctrl/sbbs.ini file, if desired to set the default sound files for all servers/services in on place:
- AnswerSound
- LoginSound
- LogoutSound
- HangupSound
- HackAttemptSound

This macro has expanded to nothing for a while now and even before, the usage was misguided and unnecessary as explained in this video: https://www.youtube.com/watch?v=cjotPqQxxAY

This won't impact Synchronet as it has a separate signal handling
thread, but we still need to behave properly for processes that
don't.  I'm also saying that ENOMEM does not indicate a disconnection,
though it may be better to pretend it was disconnected...

Still needs updates in services_thread(), CGI stuff in websrvr.c,
and sbbs_t::external()

See if this resolves rjwboys reported error:
threadwrap.h:204:42: error: expected expression before ‘do’
 #define protected_uint32_init(pval, val) atomic_init(pval, val)

Nobody's checking the return values anyway.

protected_*_adjust() only adjusts now.

Deal with the resulting warnings (using (void)).
Deal with the incorrect integer to protected_int* assignment in services.c
(just don't support server.clients property reading in service scripts).

Also, the strcpy()->SAFECOPY() change in ftpsrvr.c was wrong, caught
by GCC warning - oops.

The Synchronet FTP server has (since 2001) disallowed PORT/EPRT/LPRT commands with a "reserved" port number (i.e. < 1024) as recommended by RFC2577 and when attempted, would log a "SUSPECTED FTP BOUNCE HACK ATTEMPT" in the data/hack.log file.

However, as Karloch (HISPAMSX) pointed out recently, an FTP Bounce Attack to other TCP ports was still possible (and detected/reported by some security scans as a potential vulnerability).

So, reject all PORT/EPRT/LPRT commands that specify an IP address other than that used for the control TCP connection unless the sysop specifically enables the new "ALLOW_BOUNCE" option flag (in the [ftp] section of sbbs.ini) and the user is an authenticated non-guest/anonymous user. And as before, log the attempt as a suspected hack attempt.

This change also removes the "Directory File Access" checkbox from the Synchronet Control Panel for Windows as that feature is "going away" soon (or at least, it won't be an FTP-specific option/feature if it remains).