Are most of the ASCII control characters obsolete?

Question

Most of the ASCII codes under \x20 appear to be entirely obsolete. Are they used at all today? Can they be considered "up for grabs", or is it best to avoid them?

I need a delimiter for grouping "lines" together and it would sure be nice to co-opt one of these for that purpose.

From man ascii:

    Oct   Dec   Hex   Char                        
    ----------------------------------------------
    000   0     00    NUL '\0'                    
    001   1     01    SOH (start of heading)      
    002   2     02    STX (start of text)         
    003   3     03    ETX (end of text)           
    004   4     04    EOT (end of transmission)   
    005   5     05    ENQ (enquiry)               
    006   6     06    ACK (acknowledge)           
    007   7     07    BEL '\a' (bell)             
    010   8     08    BS  '\b' (backspace)        
    011   9     09    HT  '\t' (horizontal tab)   
    012   10    0A    LF  '\n' (new line)         
    013   11    0B    VT  '\v' (vertical tab)     
    014   12    0C    FF  '\f' (form feed)        
    015   13    0D    CR  '\r' (carriage ret)     
    016   14    0E    SO  (shift out)             
    017   15    0F    SI  (shift in)              
    020   16    10    DLE (data link escape)      
    021   17    11    DC1 (device control 1)      
    022   18    12    DC2 (device control 2)      
    023   19    13    DC3 (device control 3)      
    024   20    14    DC4 (device control 4)      
    025   21    15    NAK (negative ack.)         
    026   22    16    SYN (synchronous idle)      
    027   23    17    ETB (end of trans. blk)     
    030   24    18    CAN (cancel)                
    031   25    19    EM  (end of medium)         
    032   26    1A    SUB (substitute)            
    033   27    1B    ESC (escape)                
    034   28    1C    FS  (file separator)        
    035   29    1D    GS  (group separator)       
    036   30    1E    RS  (record separator)      
    037   31    1F    US  (unit separator)        
    040   32    20    SPACE                       

Answer 1

The ASCII control codes aren't obsolete. They are not used as much these days because the technologies that made them so useful aren't mainstream technologies anymore as technology improvements in communication technologies (USB, Ethernet, WiFi, cellular at 3G and greater, etc.) as well as improvements in integrated circuit manufacturing (increases in components per square millimeter, CPU architecture improvements, more miniaturization of components such as System on a Chip) as well as improvements in protocols.

However in the world of Internet of Things, the same technology considerations that influenced the design of these codes still applies:

• small processors with limited RAM and storage
• low bandwidth communication over slow speed paths

There are several ASCII control codes that are designed to be used to structure text. The Wikipedia topic C0 and C1 control codes, Basic ASCII control codes describes the separator control codes, FS (File Separator), GS (Group Separator), RS (Record Separator), and US Unit Separator.

Can be used as delimiters to mark fields of data structures. If used for hierarchical levels, US is the lowest level (dividing plain-text data items), while RS, GS, and FS are of increasing level to divide groups made up of items of the level beneath it. The Unix info format uses US, followed by an optional form-feed and a line break, to mark the beginning of a node.[14]

MARC 21 uses US as a subfield delimiter, RS as a field terminator and GS as a record terminator.[15]

In the current edition of IPTC 7901, if they are not used for other purposes, US is recommended for use as a column separator in tables, FS as a "Central Field Separator" in tables, and GS and RS respectively for marking a following space or hyphen-minus as non-breaking or soft respectively (in character sets not supplying explicit NBSP and SHY characters).2

See as well the description in RFC20, ASCII format for Network Interchange, which describes FS, GS, RS, and US as:

FS (File Separator), GS (Group Separator), RS (Record Separator), and US (Unit Separator): These information separators may be used within data in optional fashion, except that their hierarchical relationship shall be: FS is the most inclusive, then GS, then RS, and US is least inclusive. (The content and length of a File, Group, Record, or Unit are not specified.)

The Wikipedia topic IPTC 7901 describes the use of control characters with news service messages beginning with the formal approval of the protocol in 1979 which sounds to be similar to an RSS feed protocol. The actual specification is available from the IPTC web site as The IPTC Recommended Message Format, 1995.

Answer 2

Bit patterns -- that is, digitized numeric values -- do not become obsolete. The labels of the ASCII control codes reflect suggested uses in a wide variety of contexts -- serial comms, text display and printing, command-line editing, etc. The better word processors and text editors have used all of those codes in their keyboard command sets, and allowed all of them to be inserted into files, since the 1970s, maybe even earlier. Such programs are careful not to send these codes directly to the screen; they interpret newlines and tabs and sometimes others, and show everything else symbolically, in caret notation ("^A" for SOH, for example) or as underlined or bracketed characters. Certainly avoid ESC and a few others mentioned above if you are afraid users will cat your files to the screen. Otherwise, use them freely.

Long ago I patched WordStar to make it put my dot-matrix printer into graphics mode when desired. Using WordStar, any seven-bit code at all could be put into the graphics data. Worked like a charm.

Answer 3

First the easy part: There are no network transmission concerns in most modern systems. Current protocols handle almost any data - whether 7-bit ASCII, 8-bit ASCII, Unicode characters, image data or compiled programs - as binary data. That has not always been the case. Many older systems had issues transferring control codes and other "unprintable" characters and especially problems with 8-bit data. But those days are, fortunately, behind us. The one big exception is if you want to be able to copy/paste data via an HTML form - for that you want to leave out all control codes and other funny stuff.

You can, of course, make the format anything you like. However, some characters are still used pretty frequently:

000   0     00    NUL '\0' - does "nothing" but is hard for some text editors to handle
003   3     03    ETX (end of text) - Control-C - "break" in a lot of systems
007   7     07    BEL '\a' (bell) - Still makes a bell sound.
011   9     09    HT  '\t' (horizontal tab) - A lot of text editors and file formats use this to set a fixed number of spaces
012   10    0A    LF  '\n' (new line) - like it says
015   13    0D    CR  '\r' (carriage ret) - used instead of, or together with \n on many systems
021   17    11    DC1 (device control 1) - Control-Q - Resume transmission - XON
023   19    13    DC3 (device control 3) - Control-S - Pause transmission - XOFF
033   27    1B    ESC (escape) - Used for PCL and other printer control codes and plenty of other things 

Everything else is pretty much up for grabs. I would especially avoid NUL and XON/XOFF - they are sometimes hard to enter into a file - and BEL because typing a file with BEL can be noisy.

If you have a truly binary format then you can do anything you want. But if you want to have a mostly-human-readable format then limiting the control codes is a good idea.