Or even weirder, u_long?

And dereffing a ulong/u_long pointer where you expect to find an IPv4 address?
Yet even weirder still.

Fix that spit: It appears in_addr_t is defined on all platforms (?), so use
that type instead.

No known sightings of these sites actually being the location of a segfault,
but as we learned from the segfaults in rblchk(), the first entry in the
h_addr_list can be NULL in some cases.

This bug only impacted non-passive FTP connections. Using an FTP client
with active (not passive) data connections over an IPv6 connection would
false-trigger the "bounce attack" detection and the FTP server responded with
"504 Bad port number" and logged a hack attempt in data/hack.log.

The issue was that we were comparing the socket address structure (which
contains other fields besides the address itself) between the control and
proposed-data connections. While this logic worked okay for IPv4,
it did not for IPv6 (the 2 structs contained some non-address differences).
Rather than modify the socket address structures to match where needed, I'm
just comparing the string representation of the addresses, since that's
what we really care about anyway.

Thank to "mark i" of Truck Stop BBS for alerting me to this issue

The lower of the configured maximum file size (for the FTP server) and
the available disk space minus the configured minimum free disk space
is used as the maximum file size to allow upload. Appended files
are accounted for too.

Disallow uploads when free disk space falls below minimum configured
minimum free disk space.

This fixes issue #535

Handles integer overflow in the summing of user's credits and remaining daily
free credits.

Mostly [s]printf format fixups

Publish all login-failures to this new topic.

This commit also includes a few 64-bit free disk space fixes in the FTP server.

... not before. This explains why FTP-uploaded actions had a (null) filename.

Since v3.19 (the new filebases), when a user FTP-downloaded a file, we failed
to properly find/load that file's record from the filebase (searching for the
file's full path, rather than just the filename), so the code the increments
the file's download counter, notifies the uploader, awards credits, etc. did
not ever execute. This means that FTP-downloads for all files downloaded via
FTP were effectively "free" (and nobody noticed). No error was logged either.

I discovered this while debugging the case of "(null)" filenames in the
action/download MQTT topic messages being published by the FTP server. So
that issue is fixed as part of this commit as well.

Oh, and if this code had executed before, it would have memory-leaked the
file information, so that's fixed too (added call to smb_freefilemem). Ugh.

- Most published messages (besides log entries) have a timestamp (in ISO8601 format) prepended and tab-separated
- The order and number of elements in client messages (list and activities) has been updated, now includes user number
- Server client lists are now published to .../SERVER/client/list
- Server client activities (connect, disconnect, update) are now published to .../SERVER/client/action/#
- Server client count is now published to .../SERVER/client (with the maximum client count, if applicable)
- Server states are now just represented by name (e.g. initializing, ready, stopping, stopped) and not number
- BBS errors are logged to sbbs/BBS/action/error/LEVEL (where LEVEL is the log level name, e.g. "critical" or "error')
- All server hack-attempts, SPAM attempts, logins, logouts, uploads, downloads, are published to sbbs/BBS/action/ACTION/*
- Chat pages are published to sbbs/BBS/action/page/node/#
- New users (on the terminal server) are published to sbbs/BBS/action/newuser
- Posted messages and executed external programs (on the terminal server) are published to sbbs/BBS/action/ACTION/CODE topic
- The event thread started/stopped status is published to .../SERVER/event

Yeah, the wiki will get updated soon to reflect/document all these changes

This fixes issue #495.

This is more consistent with how these events are logged in a BBS-common log file in data/*.log.

This change also restores the server abbreviation to the error log entries that used to be there until recently.

Each Synchronet server is now its own MQTT client. This means there's no
longer any MQTT logic in the Synchronet "hosts" (e.g. sbbscon.c, ctrl/*.cpp)
and none needed for SBBS NT services (they'll "just work" with MQTT).

This also means that just about everything (except for nodes, spam and hack)
is now published per-server (in the sbbs/BBS-ID/hostname/server/ topic branch)
and if you want aggregated totals or client lists, you'll have to do that in
your own MQTT client or dashboard.

I also removed the publishing of thread_count and socket_count topics as
they weren't universally supported across all servers and are of questionable
value. They can be added back later if determined to be useful.

Now support subscriptions (e.g. recycle topics, node input topics) in SBBSCTRL.
This required a lot of search/replace and fun with circular struct pointers.

The only difference in the data/scheme is that the "error" topic (error log) is now under each server rather than each host. I don't *think* there are any other changes from the MQTT consumer side.

Still not done: subscribing (e.g. support for recycle or node-spy-input via MQTT) and NT services support.

This change also includes a cool feature that will prompt the sysop if there's a timeout (30 seconds) while waiting for servers to shutdown gracefully and giving the sysop the option to abort (Cancel) the wait (and shutdown ungracefully) or continue the wait (OK).

Previously, many servers and services didn't support login by real name
(e.g. issue #469) even if the sysop had that option enabled in SCFG.

Move login control settings from node.ini to system (main.ini -> login)

The 3 node toggle options:
- Allow Login by User Number
- Allow Login by Real Name
- Always Prompt for Password

... have been now moved from SCFG->Nodes->Node x->Toggle Options to
SCFG-System->Toggle Options.

If you upgraded to v3.20a before now, you'll want to double-check these
settings to make sure they're how you want them set. New upgraders that run
upgrade_to_v320.js (e.g. via 'jsexec update') will get these settings migrated
automatically.

Added some error detection/logging to upgrade_to_v320.js when failing to open
.cnf files.

Constified some more user/login related function args and return types.

<host>/error
.../spam
.../hack

Requires libmosquitto (e.g. install mosquitto-dev pkg) and set
in ctrl/main.ini:
[mqtt]
Enabled=true
broker_addr=127.0.0.1
broker_port=1883
qos=0
keepalive=10

Next up: Windows support

When the systemd dev package (e.g. libsystemd-dev) is installed, we can use
sd_notify() to inform systemd of Synchronet server state/status. The state
values (READY, STOPPING, RELOADING) are "well known" while the status
value is a free-form text string (the last lputs output).

Eliminated the free-form status() callback from *startup_t.
Eliminated the now-redundant started() callback from *startup_t.

Eliminated the use of sbbs_status.c/h as nothing is using that interface
and while I do plan on using a remote control/monitor interface, I'm
considering use of a standard pub/sub lib and protocol.

This commit is going to break the Windows sbbsctrl build for sure.
Probably break the sbbsNTsvcs build.

Upload and Download ARS must be matched *and* the directory's upload and download ARS.

Operator and Exemption ARS must be matched *or* the directory's operator and exemption ARS.

Also resolved some 32 vs 64-bit 'long' issues/ambiguities that have long-remained. :-)

This commit also removes logon.lst file support.

There's a TODO block remaining in js_user.c for setting portions of a user's birthdate (e.g. just the year or month or day).

1. Was not setting f->dir to the correct directory number, so only ftp-uploads to the *first* directory (dirnum = 0) would extract DIZ files of uploaded files.

Removing the 'dirnum' parameter to addfile() since that implied that you did not have to initialize the 'dir' element of the passed file_t, but you do: to get the correct file path for file size/date detection and the DIZ extraction.

2. Was getting heap-corruption when freeing the imported/formatted DIZ text on Windows once the above problem was fixed: can't free() in one DLL memory that was allocated in another DLL. Created and now using free_diz() to free the memory allocated in read_diz().

format_diz() handles a NULL 'lines' argument correctly/gracefully, so no need for the NULL lines check in sbbs_t::uploadfile().

Added FTP server log messages for successful file upload or update by user.

*/dsts.dab (daily statistics and running totals) -> */dsts.ini
*/csts.dab (cumulative statistics / log) -> */csts.tab

* dsts.ini now has both daily and total stats for all fields (not just timeon and logons).
* dsts.ini is now an easily modifiable text file - no longer need dstsedit (here-by deprecated and soon to be deleted)
* dsts.ini and csts.tab support 64-bit upload/download byte stats and are very extensible for future fields to be added or extended > 32-bit (this was the main inspiration for this overhaul, but it was overdue and already designed for v4, pretty much)
* csts.tab is a tab-delimited fixed length record format suitable for easy import to a spreadsheet program or parsing with scripts. Each day is a 128-character LF-delimited record with tab-delimited fields of plain ASCII text.
* All fields except timeon in dsts.ini files are updated immediately and by more non-terminal servers (e.g. post statistics from web UI scripts).
* New user stats are tracked more than just for "today".

The upgrade of these files is automatic and built-into SBBS.

Still to do: overhaul the slog utility to support the new csts.tab file format.

Credits and daily free credits are accurate to the byte up to (a maximum) of 18446744073709551615 (that's 18 Exbibytes - 1).

User's upload and download byte stats are now similarly extended in maximum range, but the accuracy is only "to the byte" for values less than 10,000,000,000. Beyond that value, the accuracy declines, but is generally pretty damn accurate (to 4 decimal places beyond the nearest multiple of a power of 1024), so I don't expect that to be an issue. This method of storing upload/download byte stats allowed me to use the same 10-character user record fields in the user.dat file.

As a side-effect of this enhancements:
* User and file credit values are now expressed in multiples of powers of 1024 (e.g. 4.0G rather than 4,294,967,296).
* Free credits per day per security level has now been extended from 32 to 64-bits (to accommodate values >= 4GB).
* adjustuserrec() now longer takes the record length since we can easily determine that automatically and don't need more "sources of truth" that can be out-of-sync (e.g. the U_CDT field length going from 10 to 20 chars with this change).
* setting the stage for locale-dependent thousands-separators (e.g. space instead of comma) - currently still hard-coded to comma
* more/better support for files > 4GB in size (e.g. in the batch download queue)
* user_t ulong fields changed to either uint32_t or uint64_t - I didn't realize how many long/ulong's remained in the code (which are sometmies 32-bit, sometimes 64-bit) - ugh
* Steve's ultoac() function renamed to u32toac() and created a C++ wrapper that still uses the old name, for homage

To fully support files > 4GB in size in file bases, credit values larger than 32-bits must be supported too.

There's a couple of todo comments/items included in this commit, but that's mainly to do with messages (which don't really have costs anyway).

The main thing to deal with now is the fact that users can't have more than 4GB in credits in the first place! That's got to be fixed next.

By setting sbbs.ini [web] FileIndexScript to an SSJS or XJS script filename, that script (by default, from your exec directory) will be executed when a file area/base listing has been http[s]-requested. File area/base requests are of the form <vpath_prefix> (for the list of libraries), <vpath_prefix>/<lib-name>/ (for list of directories of a library) or <vpath_prefix>/<lib-name>/<dir-code-suffix>/ (for a list of files in a directory). The new http_request "lib" and "dir" properties indicate that a library or directory listing was requested (if neither are defined, that's a request for the root / list of libs). The same configured script is executed to handle all 3 types of index/list requests.

A sample script (webfileindex.ssj) will be committed soon.

Authentication (via HTTP-AUTH) will be required if user #0 does not have access to all libraries or all directories within a required library.

file_area.lib[].link has been changed from "/<vdir>/" to just "<vdir>" (no slashes) and renamed to "vdir".

file_area.dir[].link has been changed from "/<vpath>/" to "<vpath>/" (no leading slash) and renamed to "vpath".

Added file_area.dir[].vdir property that contains just the directory's virtual directory name.

I don't think anyone was using these "link" properties since the dynamic FTP HTML index scripting feature is no longer supported.

Added can_user_access_lib() to insure that the user has access to at least one directory of a library before allowing access to the library (e.g. via JS). Something similar should be created for message groups.

Tired of being reminded that you were "awarded 0 credits"  for free downloads? I know I am, so I created another text.dat string (reusing Unused300), for notification of free-download files where no credits are awarded to the uploader. Also updated the default colors of the existing DownloadUserMsg string to have a little more variety.

Not sure why ftpsrvr.c isn't using user_downloaded_file(). That copy/pasta should be eliminated later.

As Andre pointed out while documenting this setting on the wiki, the option seemed confusing: if a sysop could not login with "system operator access", how could they login at all? Answer: they could not.

This setting used to be called "Allow Remote Sysop Logins", back when there was the concept of a "local login", so setting this option to "No" would mean that user accounts with sysop access could only be used for *local* login. But in Synchronet v3, there's really no such concept as a "local login", so it was changed to just "Allow Sysop Logins" (period) and not a lot of thought given to how/why a sysop would actually set to this "No" or what the implications would be (presumably, nobody ever sets this to "No").

So rather than just get rid if the option altogether, I changed it to mean: an account with sysop access (i.e. level 90+) can still login, but any action that normally requires the system password will not be allowed. This includes the sysop-actions available in the FTP server when authenticating with <user-pass>:<system-pass> as the password. The sysop-user can still authenticate (and login), but none of those sysop-actions will be available to them.

Make it more accurate/clear:
"attempted to upload invalid path/filename"

The logged error "!attempted to upload to invalid directory" did log the actual path that was attempted to be uploaded.

Added some quotes around other logged paths.

This change is just for internal consistency and convenience right now: the lib_t.vdir is a "sanitized" copy of the lib's short name (spaces are converted to dots or underscores based on the logic that the FTP server used in dotname()) and the dir_t.vdir is just a pointer to the dir's code_suffix. No other permutations are made (e.g. lower-casing the strings). Although the virtual directory names of libraries will now appear in mixed case in the FTP server (previously, they were all lowercase), the directory names are actually treated case-insensitively, so it should not make any difference. If forced-lowercase is preferred for some reason, please speak up.

This change leads the way to eventually, possibly, making these virtual path elements sysop-configurable. For now, it's just better to have a *copy* of the lib's short name that is appropriately modified to make a suitable directory name and have that vpath element available globally (to all servers and services) in a consistent manner.

So Nelgin asked (about filebase access via http), what if the library short name has a space in it? The answer now is, the spaces are replaced with a '.' or '_' (if there's already dots in the name).

By setting SCFG->File Options->Web File Virtual Path Prefix to something (e.g. "/files/"), all HTTP or HTTPS requests to the Synchronet Web Server with request paths beginning with this prefix will be interpreted as filebase access requests (with full access control enforcement). This is configured here (in SCFG) rather than, say, the [web] section of sbbs.ini, because I have plans for the terminal server to use this prefix to generate Web-URLs for files to display or email to users.

Currently, only requests to *files* (for download) are supported (no index generation, file information, etc. and definitely no upload support). Full access control (using HTTP auth, not cookies) is used for libraries and directories with controlled access. Credits are deducted and awarded and uploaders are notified of downloads, as one would expect. Requests to any dynamic-web-content files (e.g. .SSJS, .XJS, etc.) will be treated as static file download requests (no script will be executed).

I'm reusing the same virtual path parsing logic from the FTP server (moved to the userdat lib), so the virtual path to a file for download would be, for example, http://yourdomain/files/lib/dir-code/filename.ext

The main motivation for this feature is: FTP-links in email and web pages are just not useful to many users these days and I don't think that sysops should have to rely on a SSJS web UI (e.g. ecWebv4, cool as it is), to provide web-access to the filebases. Using this feature, you can share simpler/shorter web links to your files that will be more enduring.

Prefixed or trailing white-space characters would be hard to discern without this.

Don't treat CRYPT_ERROR_COMPLETE (-24) as a socket error during upload since it's an indication that the remote closed the connection and is the normal "end of file/transfer" indicator, not an error. 'rd' is already 0 in this case, so no need to set at all (since recv() returns 0 upon disconnect and that's what we're emulating here).

Fixes issue #309 reported by Jas Hud.