e.g. 20230130T171211-480	0	<unknown user>	76.89.231.66	<no name>

the user number name actually *are* known at this stage

- Most published messages (besides log entries) have a timestamp (in ISO8601 format) prepended and tab-separated
- The order and number of elements in client messages (list and activities) has been updated, now includes user number
- Server client lists are now published to .../SERVER/client/list
- Server client activities (connect, disconnect, update) are now published to .../SERVER/client/action/#
- Server client count is now published to .../SERVER/client (with the maximum client count, if applicable)
- Server states are now just represented by name (e.g. initializing, ready, stopping, stopped) and not number
- BBS errors are logged to sbbs/BBS/action/error/LEVEL (where LEVEL is the log level name, e.g. "critical" or "error')
- All server hack-attempts, SPAM attempts, logins, logouts, uploads, downloads, are published to sbbs/BBS/action/ACTION/*
- Chat pages are published to sbbs/BBS/action/page/node/#
- New users (on the terminal server) are published to sbbs/BBS/action/newuser
- Posted messages and executed external programs (on the terminal server) are published to sbbs/BBS/action/ACTION/CODE topic
- The event thread started/stopped status is published to .../SERVER/event

Yeah, the wiki will get updated soon to reflect/document all these changes

This fixes issue #495.

Value is "http" or "https" depending on if TLS is in use.

This is more consistent with how these events are logged in a BBS-common log file in data/*.log.

This change also restores the server abbreviation to the error log entries that used to be there until recently.

Each Synchronet server is now its own MQTT client. This means there's no
longer any MQTT logic in the Synchronet "hosts" (e.g. sbbscon.c, ctrl/*.cpp)
and none needed for SBBS NT services (they'll "just work" with MQTT).

This also means that just about everything (except for nodes, spam and hack)
is now published per-server (in the sbbs/BBS-ID/hostname/server/ topic branch)
and if you want aggregated totals or client lists, you'll have to do that in
your own MQTT client or dashboard.

I also removed the publishing of thread_count and socket_count topics as
they weren't universally supported across all servers and are of questionable
value. They can be added back later if determined to be useful.

Now support subscriptions (e.g. recycle topics, node input topics) in SBBSCTRL.
This required a lot of search/replace and fun with circular struct pointers.

The only difference in the data/scheme is that the "error" topic (error log) is now under each server rather than each host. I don't *think* there are any other changes from the MQTT consumer side.

Still not done: subscribing (e.g. support for recycle or node-spy-input via MQTT) and NT services support.

This change also includes a cool feature that will prompt the sysop if there's a timeout (30 seconds) while waiting for servers to shutdown gracefully and giving the sysop the option to abort (Cancel) the wait (and shutdown ungracefully) or continue the wait (OK).

Previously, many servers and services didn't support login by real name
(e.g. issue #469) even if the sysop had that option enabled in SCFG.

Move login control settings from node.ini to system (main.ini -> login)

The 3 node toggle options:
- Allow Login by User Number
- Allow Login by Real Name
- Always Prompt for Password

... have been now moved from SCFG->Nodes->Node x->Toggle Options to
SCFG-System->Toggle Options.

If you upgraded to v3.20a before now, you'll want to double-check these
settings to make sure they're how you want them set. New upgraders that run
upgrade_to_v320.js (e.g. via 'jsexec update') will get these settings migrated
automatically.

Added some error detection/logging to upgrade_to_v320.js when failing to open
.cnf files.

Constified some more user/login related function args and return types.

The MSS detection will set this to the best value on a per-connection
basis.  This will allow larger packets to be sent from the BBS.

We've never really cared how many times the ring buffer has become
empty, how many times data was added to it, or how many times we've
been over the highwater mark.  These have effectively always been
event signalling, with extra hackiness to clear "extra" semaphore
posts.

This commit removes RINGBUF_SEM entirely, and uses events for
everything.  There's an empty event (set when the ring buffer is empty),
a data event (set when the ring buffer is not empty), and a highwater
event (set when the ring buffer has at least highwater mark bytes).

A RingBufWrite() will set data and highwater events if applicable,
and clear the empty event.

A RingBufRead() will set empty event and clear data and highwater
events if applicable.

RingBufReInit() will now set the empty event, and clear the data
and highwater events.

These are the only actions the RingBuf API will perform on the
events, it does *not* clear the highwater event on a RingBufWrite()
if the buffer does not have enough bytes in it for example, this
will *only* be done by RingBufRead() or RingBufReInit().  This
allows consumers to force specific behaviours (such as forcing
the highwater event on shutdown to prevent waiting for it).  Since
the current code was able to deal with the semaphores having
arbitrarily high counts, the existing code shouldn't have any
issue with this.

For things that want to shut something down, we're now setting both
the data and highwater events to ensure it doesn't wait for a
timeout (even for things that don't use highwater like the inbuf).
The RingBuffer API should grow a thing to do this (RingBufferFlush
perhaps?) rather than the consumers poking into the innards
arbitrarily.

Tested on the webserver and RLogin server and seems to be working
great.  sexyz builds.  It seems there's some Windows thing that
uses the ringbuffer sem for something, but I couldn't figure out
what... updated the build file, but no clue if that's broken or not
now... I'm sure DigitalMan will be happy to fix it if CI doesn't
catch it. ;)

This should allow the console object to grow a "flush" method that
will set the highwater and data events.

<host>/error
.../spam
.../hack

Requires libmosquitto (e.g. install mosquitto-dev pkg) and set
in ctrl/main.ini:
[mqtt]
Enabled=true
broker_addr=127.0.0.1
broker_port=1883
qos=0
keepalive=10

Next up: Windows support

When the systemd dev package (e.g. libsystemd-dev) is installed, we can use
sd_notify() to inform systemd of Synchronet server state/status. The state
values (READY, STOPPING, RELOADING) are "well known" while the status
value is a free-form text string (the last lputs output).

Eliminated the free-form status() callback from *startup_t.
Eliminated the now-redundant started() callback from *startup_t.

Eliminated the use of sbbs_status.c/h as nothing is using that interface
and while I do plan on using a remote control/monitor interface, I'm
considering use of a standard pub/sub lib and protocol.

This commit is going to break the Windows sbbsctrl build for sure.
Probably break the sbbsNTsvcs build.

Now, if the BOOL argument value is FALSE, then the .ini file is opened read-only (and thus only read permissions are needed, resolving issue #455).

If the BOOL argument value is TRUE, create-if-not-exist is implied. There are no use cases where we would want to open an .ini file for writing only if it already existed.

To be more consistent in syntax and include more details (e.g. the command
being invoked).

A reused HTTP session would never (apparently) perform garbage collection.
The evidence of this was the collection of concurrent user.dat file opens
that would never close until the HTTP sessions were closed. Hundreds or
even thousands of open user.dat's have been seen. After this change,
active web server (webv4 UI) users have not caused these spikes in open
user.dat files, at least in my testing.

If no garbage collection was being performed, then likely a lot of JS
heap was being needlessly wasted, which could eventually result in a JS
"out of memory" error. But that's just a theory.

Investigation is needed into why the js_CommonOperationCallback()'s calls to
JS_MaybeGC() were not sufficient to actually perform garbage collection
in this case.

Attempt to address 2 GCC version 11.2.0 2 warnings reported by Nelgin
‘%s’ directive writing up to 3 bytes into a region of size between 1 and 4097

I knew I created these is_valid_*num() functions for a reason!

By setting sbbs.ini [web] FileIndexScript to an SSJS or XJS script filename, that script (by default, from your exec directory) will be executed when a file area/base listing has been http[s]-requested. File area/base requests are of the form <vpath_prefix> (for the list of libraries), <vpath_prefix>/<lib-name>/ (for list of directories of a library) or <vpath_prefix>/<lib-name>/<dir-code-suffix>/ (for a list of files in a directory). The new http_request "lib" and "dir" properties indicate that a library or directory listing was requested (if neither are defined, that's a request for the root / list of libs). The same configured script is executed to handle all 3 types of index/list requests.

A sample script (webfileindex.ssj) will be committed soon.

Authentication (via HTTP-AUTH) will be required if user #0 does not have access to all libraries or all directories within a required library.

file_area.lib[].link has been changed from "/<vdir>/" to just "<vdir>" (no slashes) and renamed to "vdir".

file_area.dir[].link has been changed from "/<vpath>/" to "<vpath>/" (no leading slash) and renamed to "vpath".

Added file_area.dir[].vdir property that contains just the directory's virtual directory name.

I don't think anyone was using these "link" properties since the dynamic FTP HTML index scripting feature is no longer supported.

Added can_user_access_lib() to insure that the user has access to at least one directory of a library before allowing access to the library (e.g. via JS). Something similar should be created for message groups.

Weird early failure return on TLS sends.  Most noticible on local
(ie: fast) connections, but clearly stupid all around when you
look at the code.

Fixed, but some day, I should go back and look how we eneded up in
this mess.

CID 174223
CID 174227
CID 174461
CID 174519
CID 229602
CID 319040
CID 319137
CID 319159

If the request is to a virtual web host, don't recognize the filebase vpath prefix (FileVPathPrefix setting) unless FileVPathForVHosts is set to "true" (in the [web] section of sbbs.ini).

This addresses another of Deuce's concerns about this feature.

An int is 32-bits on all supported platforms, so this has always been broken. The actual file size/request-length sent would depend on fun 2's complement math (a 32GB file was being truncated to 433MB).

Also fixed some wrong uses of PRIuOFF: off_t is a signed integer, so technically the maximum file size you can request now is 2^63 bytes, which is "big enough".

It bothered Deuce having a web server setting in scfg_t/SCFG->File Options, so I moved this setting to [web_]startup_t and the sbbs.ini file.

The downside is that file_area.web_file_prefix is no longer available to JS environments outside of the web server and terminal server, but meh, probably not going to use it elsewhere anyway? I can imagine use cases for JSexec scripts to want to generate URLs to filebase files. If that ends up being a need, they'll have to find and parse the "right" sbbs.ini file to determine the vpath prefix. 

"Request for x is outside of the web root" was already logged (with a "NOTICE" log level), but would not sound the hack attempt alarm (on Windows) or log to the hack.log. Now it does.

Similar to the ctrl/ftpalias.cfg file, the new ctrl/web_alias.ini file (optional) can be used to map a portion (the first portion, only) of a web request path to a different physical or virtual path. For example, I'm using it to map:
/Synchronet/ = /files/main/sbbs/
for filebase access to my main->sbbs directory of Vertrauen's filebase using a /Synchronet/* web request (i.e. for slightly prettier or shorter custom URLs, if desired).

By setting SCFG->File Options->Web File Virtual Path Prefix to something (e.g. "/files/"), all HTTP or HTTPS requests to the Synchronet Web Server with request paths beginning with this prefix will be interpreted as filebase access requests (with full access control enforcement). This is configured here (in SCFG) rather than, say, the [web] section of sbbs.ini, because I have plans for the terminal server to use this prefix to generate Web-URLs for files to display or email to users.

Currently, only requests to *files* (for download) are supported (no index generation, file information, etc. and definitely no upload support). Full access control (using HTTP auth, not cookies) is used for libraries and directories with controlled access. Credits are deducted and awarded and uploaders are notified of downloads, as one would expect. Requests to any dynamic-web-content files (e.g. .SSJS, .XJS, etc.) will be treated as static file download requests (no script will be executed).

I'm reusing the same virtual path parsing logic from the FTP server (moved to the userdat lib), so the virtual path to a file for download would be, for example, http://yourdomain/files/lib/dir-code/filename.ext

The main motivation for this feature is: FTP-links in email and web pages are just not useful to many users these days and I don't think that sysops should have to rely on a SSJS web UI (e.g. ecWebv4, cool as it is), to provide web-access to the filebases. Using this feature, you can share simpler/shorter web links to your files that will be more enduring.

What took down cvs/sbbs yesterday:
Program terminated with signal SIGSEGV, Segmentation fault.
6203                    session->req.post_data[session->req.post_len]=0;
[Current thread is 1 (Thread 0x7f2b989ff700 (LWP 17031))]
(gdb) print post_len
No symbol "post_len" in current context.
(gdb) print session->req.post_len
$1 = 0
(gdb) print session->req.post_data
$2 = 0x0

This fixes issue #269 (NTFS Alternate Data Stream vulnerability) and other
potential pathname issues on Windows involving colons.

There are other illegal filename characters on Windows (e.g. <>|"?*), but
filenames with these characters aren't expected to pass the later stat() test,
so should fail with a 404 error.

The requested path was being URL-decoded with unescape(), but that function does a '+' to ' ' (space) conversion.
The '+' to space decoding is supposed to occur for the "query" portion of the URL only, not the "path" portion.
Use the new hex_decode() function instead of unescape() for "path" decoding.

There was a bug with reloading the configuration files in sbbsctrl where the sound button no longer reflected the truth and the sysop's previous click-state of the button was lost. Rather than going through writing the OPT_MUTE flag to the Options fields of all the sections of the sbbs.ini and then re-loading that file as a result, just do like we did with the sysop chat availability: use a semfile. So much simpler.

If anyone ever needs instance-specific muting, we can create/check instance/host-specific mute semfiles then. Doubt that'll happen though.

Also, removed the old sysavail control methods of ntsvcs too.