Synchronet now requires the libarchive development package (e.g. libarchive-dev on Debian-based Linux distros, libarchive.org for more info) to build successfully.

newfilebase.txt 11 KB
Newer Older
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66
_New FileBases_

The new FileBase files are stored in the same database location as before
(e.g. data/dirs/), but the file extensions are different:

^ Purpose                  ^   Old    ^   New    ^
| Index (e.g. filenames)   | code.ixb | code.sid |
| Data (e.g. descriptions) | code.dat | code.shd |
| Extended Descriptions    | code.exb | code.sdt |
| Metadata                 | code.dab | code.ini |
| Allocation Tables        | N/A      | code.sda and code.sha |

If these new filebase file extensions look familiar to you, that's because
we're using the Synchronet Message Base format/library (v3.0 now) for the
underlying database. This means that the SMB tools you may be familiar with
(e.g. CHKSMB, FIXSMB, SMBUTIL) also work on the new filebases.

The conversion of the filebases to the new format should occur automatically
when you run 'jsexec update' which in turn will execute the program
'upgrade_to_v319' when appropriate (just one time). Once converted, you can
delete the old files or leave them in place in case you need to revert back to
Synchronet v3.18 for any reason. The old filebase files won't harm anything if
left.

The creation of each new filebase will automatically calculate and store the
hashes of the contents of the actual files available for download. These
hashes are useful for duplicate file detection and data integrity assurance.
If you wish to opt-out of the file hashing (which consumes the majority of the
time during the upgrade process), you can turn off file hashing in the
per-directory Toggle Options in SCFG->File Areas. You would have to perform
this opt-out for the directories of choice *before* you run 'jsexec update' /
'upgrade_to_v319'. You should not normally need to run 'upgrade_to_v319'.

_Long Filenames_

While filenames stored in the filebases used to be limited to MS-DOS
compatible 8.3 formatted names, longer filenames are now supported on all
platforms. Additionally, some previously invalid filename characters (e.g.
spaces) are now allowed and files without extensions (i.e. no '.' in their
filename are now supported).

Although Synchronet for Windows previously used Win32 API functions for short
<-> long filename conversions in the Windows builds of Synchronet, resulting
in the unfortunate Micros~1 shorted filenames stored and sometimes seen, that
is no longer the case. Except for the %~ command-line specifier, those
short/long filename conversion functions are no longer in use anywhere within
Synchronet for Windows - it's native filenames through-out. The filebase
conversion process (upgrade_to_v319 / 'jsexec update') on Windows will attempt
to automatically resolve the native/long filenames and store those names and
only those names in the new filebases.

Note: abbreviated versions of long filenames are displayed in some situations
to accommodate the limited width of a traditional BBS user terminal. An effort
is made to always display the full file extension/suffix however
(e.g. "longfilename.jpeg" may be *displayed* as "longfil.jpeg").

Note: only 64 characters of each filename (always including any extension) are
indexed for searches and duplicate checking, but the entire filename, up to
64K characters in length, is stored intact in the filebase.

Filenames with /extensions/ longer than 3 characters, e.g. ".jpeg", ".tar.gz",
can be added to the filebases, but the configurable compressible, extractable,
and viewable file types/extensions remain limited to 3 characters in SCFG.
Similarly, a maximum length of 3 character archive "types" are stored per BBS
user record (for each user's QWK packet format and temp archive preference).

67 68 69 70 71
Since support for the viewing and extraction of several common archive formats
is now built into Synchronet, files uploaded with unrecognized or unsupported
file extensions (e.g. ".tar.gz" instead of ".tgz") may still be successfully
extracted or viewed.

72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97
_Large Files_

Files greater in size than 2GB or 4GB (depending) were previously a problem.
Though there are still some 32-bit file length limitations (e.g. only files
smaller than 4GB in size will be hashed), there is better and increasing
support for larger files in general.

Note: the ZMODEM transfer protocol as designed by Chuck Forsberg only supports
files up to 4GB in size and in many cases, files greater than or equal to 2GB
in size will prove difficult or impossible to transfer between some ZMODEM
implementations. In general, it is recommended to use an alternate transfer
protocol (e.g. YMODEM[-G], FTP, HTTP) for files >= 2GB in length.

_Large Directories_

Due to the new filebase design, directories with more than 10,000 files are
now supported (though, not encouraged).

_Descriptions_

The file "summary" or single-line "short description" remains limited to 58
characters in length for practical purposes. Though longer file summaries
(up to 64KB) can be stored in the filebase, they are not recommended.

Extended (multi-line) descriptions may now span more than the previous limit
of 10 lines or 512 total characters. There is no technical limit to the length
98 99
of extended file descriptions, though a limit of 4000 characters imported from
description files embedded in archives (e.g. FILE_ID.DIZ) is being imposed.
100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120

_Batches_

File upload and download batch queues used to be maintained in memory (though
they were written to disk files to be retained between user logons), they are
now entirely maintained in disk files (data/user/*.upload and *.dnload in .ini
file format). This means that custom batch management can now be performed
easily by modules or non-Terminal Server scripts.

_Hashes_

Files are now hashed, by default, using multiple hashing algorithms (CRC16,
CRC32, MD5, and SHA1) for duplicate file detection and for reporting to users
(e.g. to insure data integrity). For a file to be considered a duplicate
(i.e. and rejected for upload) it must have the same size and hash values as
another file in a filebase already. Each directory is configurable as to
whether or not to hash its files or use it for duplicate file detection
(by name or hash).

_Sorting_

121 122 123
While the old filebase indexes were re-ordered/sorted whenever a new file was
added, the new filebase indexes are sorted dynamically, when appropriate
(e.g. when viewed).
124 125 126

As a result, the "RESORT" file transfer operator command has been removed.

127 128 129 130 131 132 133 134 135 136
The new filebases are naturally indexed in the order in which the files are
imported into the database. Sorting of the files for display purposes in the
terminal and FTP servers is optional and configured by the sysop:
  Name Ascending (case-insensitive)
  Name Descending (case-insensitive)
  Name Ascending (case-sensitive)
  Name Descending (case-sensitive)
  Date Ascending
  Date Descending

137 138 139 140 141 142 143 144 145
_Tags_

Individual files can now be tagged for easy searching/grouping. This feature
will be utilized/enhanced more in the future.

_JavaScript_

The new "FileBase" class is used (similar to the existing MsgBase class) to
open and access filebases from JavaScript modules. Using this class, the
146
default methods of listing and transferring files can be replaced with
147 148 149 150 151 152 153 154 155 156 157 158
custom modules.

_TickIT_

The new FileBase JS class is now used to import files directly from FidoNet
-style .TIC files (via tickit.js) so no dependency or invocation of any
external utilities (e.g. addfiles) is required.

_Utilities_

The native utilities ADDFILES, FILELIST, DELFILES, and DUPEFIND have been
replaced with similarly named and purposed JavaScript utility scripts to be
159
invoked with JSexec or via timed event:
160 161 162 163
- addfiles.js for importing lists of files into filebases
- postfile.js for importing a single file into a filebase
- filelist.js for generating file listings from filebases
- delfiles.js for removing files from filebases
164
- dupefind.js for detecting/listing/removing duplicate files in the filebases
165 166 167 168 169

_Performance_

Due to the nature and use of the new filebase API, file listings are much
faster (e.g. large file listings from the Synchronet FTP server) as well as
170 171
improved performance when filename/pattern, description text, and duplicate
file searching.
172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197

_FTP Server_

Due to the removal of support for rendering FTP-downloaded content (e.g.
HTML files) in modern web browsers, the FTP Server no longer supports dynamic
HTML index file generation (e.g. 00index.html). Instead, we will focus on
better support for filebase browsing and file transfers via HTTP and HTTPS in
addition to the traditional FTP and FTPS uses. The dynamic generation of
the ASCII file listings via FTP (e.g. 00index) is still supported by the FTP
server, though now much faster than before.

_libarchive_

The libarchive library (http://libarchive.org/) has now been integrated into
Synchronet (and exposed via the new "Archive" JavaScript class) and integrated
into SBBSecho so that the creation, listing/viewing, and extraction of
archived files can now be performed "in-process" without the invocation of or
dependency on external programs (e.g. Info-Zip unzip or PKUNZIP).

Formats fully supported:
- zip
- 7zip
- gzipped-tar
- bzipped-tar

Formats supported for viewing and extraction only:
198
- rar (original and RAR v5 format)
199 200 201 202 203 204 205 206 207 208 209 210
- lha/lzh
- iso
- xar
- cab

This means that for most BBSes, no "Compressible" or "Extractable" file types
need to be configured in SCFG->File Options. Additionally, by setting
"Archive Format" to "ZIP" for SCFG->Networks->QWK->Hubs, no "pack" or "unpack"
command-line need be configured.

For listing the contents of archives, the new archive.js utility script may be
installed as a "Viewable File Type" handler for the commonly supported file
211 212
extensions by running 'jsexec archive.js install'. This installs archive.js as
a final "fall-through" viewer for "*" files (all file types).
213 214 215 216

_DIZ_

Description files embedded in archives (e.g. FILE_ID.DIZ) are now supported
217 218 219 220
more uniformly and seamlessly. Supported ANSI sequences (e.g. attribute/color
changes) are now converted to Ctrl-A equivalents upon import. Metadata
(e.g. declaring the file's author or group) is now imported from SAUCE records
of DIZ files, when it exists.
221 222 223 224 225 226 227 228 229 230 231 232 233 234 235 236 237 238 239 240 241 242 243 244 245 246 247 248 249 250

_File Echoes_

Each file transfer directory configured in SCFG->File Areas may now have an
"Area Tag" explicitly set for FidoNet-style file distribution networks. If
an Area Tag is not explicitly set, then the directory's short name is used
(with spaces replaced with underscores) automatically. tickit.js now uses
this new "area_tag" file_area.dir[] JS property for its "AutoAreas" feature.

_Opened Files_

The "open" (reference) counter for files is now gone. If you want to remove
a file from the filebase while a user has it in their batch download queue or
is actively downloading it, nothing is preventing you from doing so.

As a result, the "CLOSE" file transfer operator command has been removed.

_Alternate File Paths_

The support for "Alternate File Paths" has been removed. There are better
modern operating/file system solutions to the original problem solved with
this feature.

As a result, the "ALTUL" file transfer operator command has been removed.

_Bi-directional File Transfers_

The protocol drivers that supported bi-directional file transfers (Bi-Modem,
HS/Link) are now long unsupported DOS/OS2 programs with no equivalent in the
modern world. Bye bye Bi-modem. :-(