Merge remote-tracking branch 'origin/v320a_dev'

v3.20a has been in development for several months now and appears stable enough
to merge to master.

Sysops running v3.19 or earlier *must* run 'jsexec update' to get their config
files (ctrl/*.cnf) converted to *.ini and their user base (data/user/user.dat)
converted to user.tab.

docs/newuserbase.txt

+_New User Database Format_
+
+Synchronet v3.20 introduces a new primary user database file format and
+naming:
+
+data/user/user.dat -> user.tab
+
+The secondary user data files (data/user/####.ini) remain as-is.
+
+_Why the change?_
+
+The Synchronet user.dat file format has not changed since the inception of
+Synchronet BBS software back in 1991. Each user's record within the user.dat
+file had a fixed length (834 bytes) at a fixed offset within the file and
+each field (e.g. user name, alias, password) is also at a fixed offset within
+each record, with a fixed maximum length. A portion of each user record
+was reserved with unused or "padding" bytes that have been consumed over
+the years to add new fields or relocate fields to extend their maximum length.
+While this record padding has enabled extensibility over the years, each
+additional field or extended field (e.g. the extension of maximum password
+length from 8 to 40 characters) has incurred special upgrade logic and
+consumed the available padding bytes, leaving little room for adding or
+moving/extending user fields in 2022.
+
+Simply redefining the user.dat record format to use longer/wider fields would
+only serve to reset expectations based on the current predictions of future
+needs which are inevitably flawed. So the fundamental improvement needed
+for future-proofing was the switch from fixed-length to variable-length
+fields.
+
+Variable-length records would make fast random access (i.e. direct-seeking to
+a specific user's record) and record locking (insuring exclusive access,
+e.g. for read/modify/write operations) more difficult. Fixed-length records
+are a good solution for fast random access and locking, so the new user data
+format uses the combination of:
+* Fixed-length records (1000 bytes) at fixed offsets
+* Variable-length fields
+
+This change in format will allow:
+* new fields to be added easily
+* existing fields to be extended (e.g. in maximum string length) easily
+
+What happens when the total length of all variable-length fields exceeds the
+record's fixed length? Dynamic run-time detection of the record length would
+be trivial and upgrading to add padding bytes to each user.tab record if/when
+necessary, without changing the underlying format, is very realistic. The
+current estimate is that 1000-byte records will be sufficient for a long time.
+
+_What are the details of the user.tab format?_
+
+The legacy user.dat was primarily a text file that used the ASCII ETX (0x03)
+character to right-pad fields and records to their fixed lengths. The user.dat
+was fairly easily readable in an ASCII/plain-text viewer/editor, but not
+universally so (e.g. the ASCII ETX characters were not rendered consistently
+between different programs, sometimes a heart glyph, sometimes ^C). Being
+mostly-text allowed the user.dat file to be viewed or even manually-edited
+fairly easily, but the data format was not easily imported into other data
+formats/programs without Synchronet-specific conversion scripts or programs.
+
+The new user.tab format is *entirely* a plain-text file containing ASCII LF
+(0x0a) terminated records of fixed length: 1000 characters. Each user record
+is a line of text.
+
+Each field within each record is separated with an ASCII Horizontal Tab (0x09)
+character, hence the ".tab" file extension. No other control characters (ASCII
+0x00-0x1F) should be present in the file. Other non-ASCII characters are
+currently interpreted as CP437 characters (so-called IBM extended ASCII
+characters, not UTF-8 sequences).
+
+This file format is easily viewed with plain-text viewers/editors or imported
+into other programs (e.g. Microsoft Excel, Google Sheets). It is critical that
+when making a change to the file with any non-Synchronet software, that the
+1000 character line/record lengths (including the LF-terminator) are
+maintained. The ordering of the fields within each record is significant
+(don't insert new fields). Unused/padding bytes at the end of each record are
+filled with ASCII TAB characters.
+
+In addition to the record/field format changes, the content format of some of
+specific user fields has changed as well:
+- Date/time stamps are represented as ISO-8601 strings (e.g. "20221006T1453Z")
+  rather than Unix time_t (seconds since Jan-1-1970 UTC) integer values
+- Security flags, including exemptions and restrictions, are now expressed as
+  a list of alphabetic characters (A-Z) rather than a hexadecimal integer
+- Upload/download byte counts are always stored as an exact count rather than
+  an estimation (e.g. "10.1G") when the number of required digits exceeded 10
+
+_What is the impact of this change?_
+
+Initially, there should be no observable impact with this change: users should
+experience no difference in BBS behavior or performance. Sysops may notice
+that some configurable fields (e.g. internal codes) will now accept longer
+strings than previously allowed and eventually other user-visible strings may
+be extended to allow a greater total number of characters. Many user
+statistical values that were previously limited to 16-bit integers (0-65535)
+and 5 decimal digits will be increased to modern/32-bit ranges of value.
+
+User date/time values that were previously expected to encounter issues in the
+year 2038 (Y2K38, the Epochalypse) or best case, the year 2106, should be
+resolved by storing the date/time values in ISO-8601 format and using 64-bit
+time_t's for internal storage of user data in all Synchronet programs.
\ No newline at end of file

exec/archive.js

@@ -109,19 +109,19 @@ function list(filename, verbose)
 
 function install()
 {
-	var cnflib = load({}, "cnflib.js");
-	var file_cnf = cnflib.read("file.cnf");
-	if(!file_cnf) {
-		alert("Failed to read file.cnf");
+	var f = new File(system.ctrl_dir + "file.ini");
+	if(!f.open(f.exists ? 'r+':'w+')) {
+		alert("Failed to open " + f.name);
 		exit(-1);
 	}
-	file_cnf.fview.push({
+
+	if(!f.iniSetObject("viewer:new", {
 		extension: "*",
 		cmd: '?archive list %f'
-		});
-	if(!cnflib.write("file.cnf", undefined, file_cnf)) {
-		alert("Failed to write file.cnf");
+		})) {
+		alert("Failed to write " + f.name);
 		exit(-1);
 	}
+	f.close();
 	exit(0);
 }

exec/avatars.js

@@ -471,20 +471,17 @@ function install()
 {
 	if(!file_exists(lib.local_library() + "*.bin"))
 		return "No avatars collections (.bin files) found in " + lib.local_library();
-	var cnflib = load({}, "cnflib.js");
-	var xtrn_cnf = cnflib.read("xtrn.cnf");
-	if(!xtrn_cnf)
-		return "Failed to read xtrn.cnf";
+	var f = new File(system.ctrl_dir + "xtrn.ini");
+	if(!f.open(f.exists ? 'r+':'w+'))
+		return "Failed to open " + f.name;
 
-	var changed = false;
-	if(!xtrn_area.prog["avatchoo"]) {
+	var section = "prog:MAIN:AVATCHOO";
+	if(!f.iniGetValue(section, "cmd")) {
 		printf("Adding external program: Avatar Chooser\r\n");
-		xtrn_cnf.xtrn.push( {
-				"sec": 0,
+		f.iniSetObject(section, {
 				"name": "Avatar Chooser",
-				"code": "AVATCHOO",
 				"ars": "",
-				"run_ars": "ANSI & !GUEST & REST ! Q",
+				"execution_ars": "ANSI & !GUEST & REST ! Q",
 				"type": 0,
 				"settings": 1,
 				"event": 3,
@@ -495,13 +492,12 @@ function install()
 				"textra": 0,
 				"max_time": 0
 				});
-		changed = true;
 	}
 
-	if(!xtrn_area.event["avat-in"]) {
+	section = "event:AVAT-IN";
+	if(!f.iniGetValue(section, "cmd")) {
 		printf("Adding timed event: AVAT-IN\r\n");
-		xtrn_cnf.event.push( {
-				"code": "AVAT-IN",
+		f.iniSetObject(section, {
 				"cmd": "?avatars import",
 				"days": 255,
 				"time": 0,
@@ -512,13 +508,12 @@ function install()
 				"mdays": 0,
 				"months": 0
 				});
-		changed = true;
 	}
 
-	if(!xtrn_area.event["avat-out"]) {
+	section = "event:AVAT-OUT";
+	if(!f.iniGetValue(section, "cmd")) {
 		printf("Adding timed event: AVAT-OUT\r\n");
-		xtrn_cnf.event.push( {
-				"code": "AVAT-OUT",
+		f.iniSetObject(section, {
 				"cmd": "?avatars export",
 				"days": 255,
 				"time": 0,
@@ -529,11 +524,8 @@ function install()
 				"mdays": 0,
 				"months": 0
 				});
-		changed = true;
 	}
-
-	if(changed && !cnflib.write("xtrn.cnf", undefined, xtrn_cnf))
-		return "Failed to write xtrn.cnf";
+	f.close();
 
 	var ini = new File(file_cfgname(system.ctrl_dir, "modopts.ini"));
 	if(!ini.open(file_exists(ini.name) ? 'r+':'w+'))

exec/emailfiles.js

@@ -51,25 +51,23 @@ if(!options.maxpending)
 	options.maxpending = 100 * 1024 * 1024;
 
 if(argv[0] === '-install') {
-	var cnflib = load({}, "cnflib.js");
-	var file_cnf = cnflib.read("file.cnf");
-	if(!file_cnf) {
-		alert("Failed to read file.cnf");
+	var f = new File(system.ctrl_dir + "file.ini");
+	if(!f.open(f.exists ? 'r+':'w+')) {
+		alert("Failed to open " + f.name);
 		exit(-1);
 	}
-	file_cnf.prot.push({
+	if(!f.iniSetObject("protocol:new", {
 		  key: 'E'
 		, name: 'E-mail Attachment'
 		, dlcmd: '?emailfiles %f'
 		, batdlcmd: '?emailfiles +%f'
 		, ars: 'REST NOT M'
 		, settings: PROT_NATIVE | PROT_DSZLOG
-		});
-
-	if(!cnflib.write("file.cnf", undefined, file_cnf)) {
-		alert("Failed to write file.cnf");
+		})) {
+		alert("Failed to write " + f.name);
 		exit(-1);
 	}
+	f.close();
 	exit(0);
 }
 

exec/init-fidonet.js

-// $Id: init-fidonet.js,v 1.29 2020/05/12 17:23:30 rswindell Exp $
-
-// Initial FidoNet setup script - interactive, run via JSexec or ;exec
+// $Id: init-fidonet.tup script - interactive, run via JSexec or ;exec
 
 // usage: init-fidonet.js [zone | othernet-name] [http://path/to/echolist.na]
 
@@ -22,7 +20,7 @@
 
 "use strict";
 
-const REVISION = "$Revision: 1.30 $".split(' ')[1];
+const REVISION = "2.0;
 require('sbbsdefs.js', 'SUB_NAME');
 const temp_node = 9999;
 var netname;
@@ -472,10 +470,9 @@ if(netname) {
 } else
 	alert("Unrecognized network zone: " + netzone);
 
-print("Reading Message Area configuration file: msgs.cnf");
-var cnflib = load({}, "cnflib.js");
-var msgs_cnf = cnflib.read("msgs.cnf");
-if(!msgs_cnf) {
+var msgs_ini = new File(system.ctrl_dir + "msgs.ini");
+print("Reading Message Area configuration file: " + msgs_ini.name);
+if(!f.open(f.exists ? 'r+':'w+')) {
 	alert("Failed to read msgs.cnf");
 	exit(1);
 }
@@ -602,37 +599,33 @@ if(your.node === temp_node && network.email && network.email.indexOf('@') > 0
 }
 
 if(!find_sys_addr(fidoaddr.to_str(your))
-	&& confirm("Add node address " + fidoaddr.to_str(your) + " to your configuration"))
-	msgs_cnf.fido_addr_list.push(your);
+	&& confirm("Add node address " + fidoaddr.to_str(your) + " to your configuration")) {
+	var fido_addr_list = msgs_ini.iniGetValue("fidonet", "addr_list", []);
+	fido_addr_list.push(your);
+	msgs_ini.iniSetValue("fidonet", "addr_list", fido_addr_list);
+}
 
-if(!msgs_cnf.fido_default_origin)
-	msgs_cnf.fido_default_origin = system.name + " - " + system.inet_addr;
-while((!msgs_cnf.fido_default_origin
-	|| !confirm("Your origin line is '" +	msgs_cnf.fido_default_origin + "'")) && !aborted()) {
-	msgs_cnf.fido_default_origin = prompt("Your origin line");
+var fido_default_origin = msgs_ini.iniGetValue("fidonet", "default_origin");
+if(!fido_default_origin)
+	fido_default_origin = system.name + " - " + system.inet_addr;
+while((!fido_default_origin
+	|| !confirm("Your origin line is '" +	fido_default_origin + "'")) && !aborted()) {
+	fido_default_origin = prompt("Your origin line");
 }
+msgs_ini.iniSetValue("fidonet", "default_origin", fido_default_origin);
 
 /*******************/
-/* UPDATE MSGS.CNF */
+/* UPDATE MSGS.INI */
 /*******************/
-if(!msg_area.grp[netname]
+if(!msgs_ini.iniGetObject("grp:" + netname)
 	&& confirm("Create " + netname + " message group in SCFG->Message Areas")) {
 	print("Adding Message Group: " + netname);
-	msgs_cnf.grp.push( {
-		"name": netname,
+	msgs_ini.iniSetObject("grp:" + netname, {
 		"description": netname,
-		"ars": "",
 		"code_prefix": network.areatag_prefix === undefined
 			? (netname.toUpperCase() + "_") : network.areatag_prefix
 	});
 }
-if(confirm("Save Changes to Message Area configuration file: msgs.cnf")) {
-	if(!cnflib.write("msgs.cnf", undefined, msgs_cnf)) {
-		alert("Failed to write msgs.cnf");
-		exit(1);
-	}
-	print("msgs.cnf updated successfully.");
-}
 
 /*********************/
 /* DOWNLOAD ECHOLIST */