FSC-0014 A Bundle Proposal Wynn Wagner III January, 1988 UPFRONT ------- What follows is a proposal for a new structure of message bundles that are transmitted between Fidonet systems. Currently we deal with "packet version 2." This is a proposed format for packet version three. The version number should be recognizable by TYPE-2 programs, but the older programs will not be able to do anything more than report an error. In other words, there is no direct upwards compatibility except for the offset in the _BundleHeader (see below) of the TYPE-3 signal. Because of this, any conversion should be slow... possibly a year or more. Interim systems would have to be able to pack and unpack both kinds of bundles. It would be required for the packer to know whether its target system knows about TYPE-3 bundles or not... either by some node list flag or by using a control file. I think it is important that a new structure be seriously considered, but it is also vital that we approach such a change with a mind to keeping it an evolutionary process rather than an overnight revolution. It is important that TYPE-2 systems be retired using attrition instead of compulsion. This proposal is described in detail... possibly too much detail. The design looks scarier than it really is. Code to process this kind of a bundle is almost trivial. One design feature is that putting a message bundle together is somewhat more involved than taking one apart. The theory is that we will be getting more and more tiny installations operating as points. Collecting and unpacking such things as echomail will be easier even on small/slow computers. Heavy volume traffic with lots of packing and unpacking is usually carried on by computers more capable of heavy work. Please note that the added work on the shoulders of the packer is almost microscopic, but it exists. This uneven distribution of the work is intentional. ------------------------------------------------------------------------------ PRELIMARIES: METHODS -------------------- Messages are transmitted in "bundles." A bundle is a sequence of "packets." Every bundle has at least two packets: a header and a footer. This document describes the layout and use of those packets as well as the general format of a bundle. ------------------------------------------------------------------------------ PRELIMINARIES: DATA ------------------- #define BUNDLEVER 3 #define EIDSIZ (sizeof(struct _Address)+sizeof(unsigned long)) struct _Address { unsigned int Zone; unsigned int Net; unsigned int Node; unsigned int Point; }; DATA NOTES: "WORD" is a two byte unsigned integer with the most significant byte first. This storage arrangement is backwards from the way INTEL chips store numbers, but it is more in-line with the way the rest of the universe does it. The conversion for MsDOS and other INTEL programmers is quite trivial. A routine called SWAP can be used, where... unsigned int pascal SWAP(unsigned int VALUE); Here's the code... SWAP Proc Near mov bx, sp mov ax, [bx+2] xchg ah, al ret 2 SWAP EndP "UNSIGNED LONG" is a four byte unsigned integer with the most significant word first. Again, this storage arrangement conflicts with the method used by INTEL, but the trans- formation to "MSW first" is quite simple and it really makes the non-MsDOS programmers feel more comfortable. "UNSIGNED CHAR" is an 8-bit datum that can have any value between 0 and 0xff. CHARACTER ARRAYS are null-padded unless otherwise noted. There is a difference between "null-terminated" (common to C-language programming) and "null-padded" found here. Unless there is a conflicting note, insignificant bytes of a character array must be set to zero. UNUSED DATA are always set to zero. ------------------------------------------------------------------------------ BUNDLE HEADER ------------- struct _BundleHeader { struct _Address B_Destination; /* Address of target system */ struct _Address B_Origination; /* Address where bndl started */ unsigned int B_Local1; /* Used by bundler, etc */ unsigned int B_Version; /* Always 3 */ unsigned long B_CreationTime; /* Unix-type stamp (1970-base) */ unsigned int B_BundlerMajor; /* Major version of bundler */ unsigned int B_BundlerMinor; /* Minor version of bundler */ unsigned char B_Password[9]; /* NULL-padded array */ unsigned char B_Local2; /* Local use only */ unsigned char B_Product[4]; /* Meaningful to same product */ unsigned char B_FTSC[4]; /* Reserved by FTSC */ }; BUNDLE HEADER NOTES: The offset of B_Version coincides with the location of the `ver' field in type two bundle headers. The B_Local1 and B_Local2 fields have no meaning during the actual transfer. It is to hold such information as COST and/or BAUD rate of use to the sending system. B_Password is a NULL-padded field that can contain uppercase alpha bytes or ASCII digits. It should not contain lowercase characters, punctuation, control characters, etc. This is a NULL-padded field... not just NULL-terminated. A maximum of 8 bytes are significant. Note that the BUNDLER is identified by product code. This does NOT necessarily have anything to do with the software that actually transmits the bundle. This structure deals with message bundles, and the product identification shows which computer program was responsible for that layer of a netmail session. The software providing transportation is more properly identified during a session-level negotiation (eg WaZOO's YooHoo) or in a dynamically generated structure (eg TeLink's block zero). TYPE-3 tries to keep the various layers of the system separate and easily identifiable. This document describes data, not the method by which they are passed from one system to another. BUNDLE BODY NOTES: The bundle header packet is followed by one or more of the following packet types. Each of these packets begin with two bytes that identify the packet version and the packet type. In all cases, the version is three (0x03). Packet types include END_SIGNAL 0 AREA_HEADER 1 MESSAGE_HEADER 2 TEXT 3 ECHOMAIL_INFO 4 MISCINFO 128-256 One message is built using at least two packets (MESSAGE_HEADER and TEXT). Optionally, a message might also have ECHOMAIL_INFO and MISCINFO packets. Packets associated with a message must be bundled in numerical order (by packet type)... the header comes first, followed by the text, possibly followed by echomail information, and possibly ending with some miscellaneous packets. This arrangement of bundles allows the development of state machine type programs which effect efficient message processing even on slow or small computers. Here's a quick coding example: for(;;) switch(fgetc(BUNDLE)) { default: printf("Not Type-3 message"); return -1; case 3: switch(fgetc(BUNDLE)) { case -1: printf("EOF?"); return -1; case 0: printf("Done\n"); return 0; case 1: GetMsgArea(); break; case 2: GetMessage(); break; default: printf("Pkt?"); break; }; }; NOTE: For those re-reading and spotting what looks like a mistake... the `GetMessage()' routine would also take care of TEXT and any ECHOMAIL or MISCINFO packets. Also, for a little added robustness, the default item that prints "Pkt?" should look for a value of 0x03 (or and end of the file) before returning to the processing loop. ------------------------------------------------------------------------------ BUNDLE FOOTER ------------- struct _BundleEnd { unsigned char M_Version; /* Always 3 */ unsigned char M_PacketType; /* Always 0 */ }; BUNDLE END NOTES: All bundles end with this packet. It is not optional. ------------------------------------------------------------------------------ MESSAGE AREA ------------ struct _AreaHeader { unsigned char E_Version; /* Always 3 */ unsigned char E_PacketType; /* Always 1 */ unsigned byte E_NameLength; /* Actual bytes in E_Name */ unsigned byte E_Name[1]; /* VARIABLE-length field */ }; AREA HEADER NOTES: The area header packet marks the start of a sequence of messages destined for the same message area. Often E_Name will contain the name of an echomail area. If the legnth and first byte of E_Name are zero, it means that the following messages are inter-system traffic (ie regular netmail). The maximum value for E_NameLength is 63. E_Name can contain uppercase characters, digits, and the following punctuation: $ . - _ & # @ ! Note that E_NameLength combined with E_Name make up what is often considered a "Pascal style string." E_Name is NOT a null-terminated array (aka "ASCIIZ"). ------------------------------------------------------------------------------ MESSAGE HEADER -------------- struct _MessageHeader { unsigned char M_Version; /* Always 3 */ unsigned char M_PacketType; /* Always 2 */ struct _Address M_Destination; /* FINAL Destination */ struct _Address M_Origination; /* Where message was entered */ unsigned long M_CreationTime; /* Unix-type stamp (1970-base) */ unsigned int M_Attributes; /* Standard Fidonet bitweights */ unsigned char M_FromLength; /* Number of bytes in FROM */ unsigned char M_ToLength; /* Number of bytes in TO */ unsigned char M_SubjectLength; /* Number of bytes in SUBJECT */ }; MESSAGE HEADER NOTES: Every message begins with a message header packet. This structure is created by the system where the message originated. If there are any intermediate stops before it reaches its destination, it is the responsibility of intermediate systems to maintain all of this information without any modification. Following this header come three char-type data: FROM, TO, and SUBJECT. Using the final three fields of the header, a program can quickly read and process/store the the message. None of the character items is null-terminated. Their lengths are determined by values in the message header. ------------------------------------------------------------------------------ MESSAGE TEXT/BODY ----------------- struct _Text { unsigned char T_Version; /* Always 3 */ unsigned char T_PacketType; /* Always 3 */ unsigned int T_ByteCount; /* Number of bytes ( <0x1000) */ unsigned int T_Data[1]; /* VARIABLE-length field */ }; TEXT NOTES: The body of a message is contained in one or more _Text packets. No _Text packet is ever more than 1000H bytes long. That's 4096 bytes to the terminally base-10 folks. Of course there can be an infinite number of text packets, but you are absolutely positively guaranteed that with the TYPE-3 method, you will never need a buffer larger than 1000H. In addition to ASCII values 20h through 7Eh (inclusive), the following control codes are legal for TEXT data. Note that and are NOT in this list. 02H ... material from here to next is a quote from the parent message 0Ah ... forced 10h ... replicate Other control characters and values 7fH and above are symptoms of a damaged message. REPLICATE is a three byte sequence: . For example, if a message contains the bytes 10h, 20h, 09h ... it would mean that message display programs should replicate the character nine times. There is no minimum or maximum line length. If there is no before the display program needs one, it is the display program's responsibility to provide the needed "line wrap." One "line" can cross from one _Text packet to another. ------------------------------------------------------------------------------ ECHOMAIL -------- struct _EchomailInfo { unsigned char EI_Version; /* Always 3 */ unsigned char EI_PacketType /* Always 4 */ unsigned char EI_Parent[EIDSIZ];/* "up" message thread */ unsigned char EI_Child[EIDSIZ]; /* "down" message thread */ unsigned int EI_SeenbyCount; /* Number of SEENBY items */ struct _Address EI_Seenby[1]; /* VARIABLE-length field */ }; ECHOMAIL INFO NOTES: The EI_Parent and EI_Child fields contain some kind of identification of the parent and child messages. The size of the fields corresponds to the size of an _Address structure plus the size of a Unix-type time stamp. ------------------------------------------------------------------------------ A KLUDGE, BY ANY OTHER NAME... ------------------------------ struct _MiscInfo { unsigned char MI_Version; /* Always 3 */ unsigned char MI_PacketType; /* 0x80-0xff, assigned by FTSC */ unsigned char MI_ByteCount; /* Size of miscinfo data */ unsigned char MI_WhoKnows; /* Miscellaneous stuff */ }; MISCELLANEOUS INFO NOTES: This is the catch-all packet type that replaces "The Dreaded IFNA Kludge." If present, they are the last packets associated with a message. _MiscInfo items are bound to a message, and it is the responsibility of any packer to maintain any _MiscInfo packets exactly as they arrived on any message that will be retransmitted (ie netmail and echomail). Values above 0xf0 have a special meaning. They are reserved for the severe case that FTSC needs to make some kind of change that isn't backwards compatible. In most cases, unrecognized _MiscInfo packets should be preserved but otherwise ignored. If the packet type is 0xf0 through 0xff, it should be considered an error condition not to understand the packet. ###