What are tokens used for in this example?

Question

Can someone please tell me what the tokens are for in this code? I've just recently discovered tokens in Java and trying to get an understanding of why they are used. I thought tokens were used as break points, but here it almost looks like it is used to limit

private String printResult()
{
    StringBuilder result = new StringBuilder();
    String swimType=null;
    String length=null;
    String width=null;
    String depth=null;
    String volume=null;
    String radius=null;
    String pnlName = null;
    try {
        File resultFile = new File("Report.txt");
        Scanner resultScanner = new Scanner(resultFile);

        while(resultScanner.hasNext())
        {
            StringTokenizer strToken = new StringTokenizer(resultScanner.nextLine(), ":");
            if(strToken.hasMoreTokens())
            {
                pnlName = strToken.nextToken();
                swimType = strToken.nextToken();

                if("Box".equalsIgnoreCase(swimType))
                {
                    length = strToken.nextToken();
                    width = strToken.nextToken();
                    depth = strToken.nextToken();
                    volume = strToken.nextToken();
                    result.append(createResultStr(swimType, length, width, depth, volume));
                }
                else
                {
                    radius = strToken.nextToken();
                    depth = strToken.nextToken();
                    volume = strToken.nextToken();                      
                    result.append(createResultStr(swimType, radius, depth, volume));
                }

            }               
        }   

Answer 1

Tokenizing is the process were you split up a string based on a character. You may want to split on the comma. You pass the string through the tokenizer giving ',' as the separator. Then as you read each item out of the tokenizer they are split up based on the separator. A common approach I have seen is reading a live stream. A tokenizer splits the stream on linefeed and then each line is split on commas or tabs using another tokenizer.

The processing of text often consists of parsing a formatted input string. Parsing is the division of text into a set of discrete parts, or tokens, which in a certain sequence can convey a semantic meaning. The StringTokenizer class provides the first step in this parsing process, often called the lexer (lexical analyzer) or scanner. StringTokenizer implements the Enumeration interface. Therefore, given an input string, you can enumerate the individual tokens contained in it using StringTokenizer.

To use StringTokenizer, you specify an input string and a string that contains delimiters. Delimiters are characters that separate tokens. Each character in the delimiters string is considered a valid delimiter—for example, ",;:" sets the delimiters to a comma, semicolon, and colon. The default set of delimiters consists of the whitespace characters: space, tab, newline, and carriage return.

The StringTokenizer constructors are shown here:

StringTokenizer(String str) 
StringTokenizer(String str, String delimiters) 
StringTokenizer(String str, String delimiters, boolean delimAsToken)

In all versions, str is the string that will be tokenized. In the first version, the default delimiters are used. In the second and third versions, delimiters is a string that specifies the delimiters. In the third version, if delimAsToken is true, then the delimiters are also returned as tokens when the string is parsed. Otherwise, the delimiters are not returned. 

Delimiters are not returned as tokens by the first two forms. Once you have created a StringTokenizer object, the nextToken( ) method is used to extract consecutive tokens. The hasMoreTokens( ) method returns true while there are more tokens to be extracted. Since StringTokenizer implements Enumeration, the hasMoreElements( ) and nextElement( ) methods are also implemented, and they act the same as hasMoreTokens( ) and nextToken( ), respectively.

Here is an example that creates a StringTokenizer to parse "key=value" pairs. Consecutive sets of "key=value" pairs are separated by a semicolon.

// Demonstrate StringTokenizer.
import java.util.StringTokenizer;

class STDemo {
    static String in = "title=Java-Samples;" +
        "author=Emiley J;" +
        "publisher=java-samples.com;" +
        "copyright=2007;";

    public static void main(String args[]) {
        StringTokenizer st = new StringTokenizer(in, "=;");
        while (st.hasMoreTokens()) {
            String key = st.nextToken();
            String val = st.nextToken();
            System.out.println(key + "\t" + val);
        }
    }
}

The output from this program is shown here:

title Java-samples 
author Emiley J 
publisher java-samples.com 
copyright 2007

Note
if delimiter is null, this constructor does not throw an exception. However, trying to invoke other methods on the resulting StringTokenizer may result in a NullPointerException.

Answer 2

seems like an input file contains data in certain structure and the : is a delimiter, so it reads data token by token and assigns values to the variables with the further append to some result string