file reading into array

Question

I am trying to read contents of a file using string tokenizer and store all the tokens in an array but i keep getting exception in main error. I need advise on how to do this.Below is the code am using for that;

import java.io.BufferedReader;
import java.io.File;
import java.io.FileReader;
import java.io.IOException;
import java.io.Reader;
import java.util.ArrayList;
import java.util.List;
import java.util.Scanner;
import java.util.StringTokenizer;

public class FileTokenizer
{
    private static final String DEFAULT_DELIMITERS = "< , { } >";
    private static final String DEFAULT_TEST_FILE = "trans1.txt";


    public List<String> tokenize(Reader reader) throws IOException
    {
        List<String> tokens = new ArrayList<String>();

        BufferedReader br = null;

        try
        {
            int i = 0;
            br = new BufferedReader(reader);
            Scanner scanner = new Scanner(br);
            while (scanner.hasNext())
            {

                StringTokenizer st = new StringTokenizer(scanner.next(), DEFAULT_DELIMITERS, true);
                while (st.hasMoreElements())
                {
                    String[] t = new String[200];

            tokens.add(st.nextToken());

                    t[i] = st.nextToken(); 

                    System.out.println(t[i]);

                    i++;                                      
                }
            }
        }
        finally
        {
            close(br);
        }

        return tokens;
    }

    public static void close(Reader r)
    {
        try
        {
            if (r != null)
            {
                r.close();
            }
        }
        catch (IOException e)
        {
            e.printStackTrace();
        }
    }

   public static void main(String[] args)
    {
        try
        {
            String fileName = ((args.length > 0) ?  args[0] : DEFAULT_TEST_FILE);
            FileReader fileReader = new FileReader(new File(fileName));
            FileTokenizer fileTokenizer = new FileTokenizer();
            List<String> tokens = fileTokenizer.tokenize(fileReader);
            //System.out.println(tokens);
        }
        catch (IOException e)
        {
            e.printStackTrace();
        }
    }
}

My file looks like;

PDA = (
{ q1, q2, q3, q4},
{ 0, 1 },
{ 0, $ },
{ (q1, @, @) -> { (q2, $) }, (q2, 0, @) -> { (q2, 0) },
(q2, 1, 0) -> { (q3, @) }, (q3, 1, 0) -> { (q3, @) },
(q3, @, $) -> { (q4, @) } },
q1,
{ q1, q4}
)

Answer 1

Looking at your input file, I should point out that its hierarchical and irregular structure makes it more suited to be parsed by an actual parser. You may have to learn how to use a parser generator and write a lexer and grammar for it etc, but in the end you'll end up with a much more maintainable code. Doing this yourself is rather painstaking and error-prone.

I recommend ANTLR. It's quite mature, and it has a wide enough user base that I'm sure you can get help easily.

Answer 2

Delimiters shouldn't be separated by spaces:

private static final String DEFAULT_DELIMITERS = "<,{}>";

Also, keep the following in mind (from the Javadoc):

StringTokenizer is a legacy class that is retained for compatibility reasons although its use is discouraged in new code. It is recommended that anyone seeking this functionality use the split method of String or the java.util.regex package instead.

String.split() was introduced in JDK 1.4.

That said:

• Using a Scanner to tokenize a stream together with a StringTokenizer looks a bit weird to me;
• You call st.nextToken() twice in the inner loop;
• t is useless. You re-create it each time in your inner loop and use only one element of it.

It seems that what you are trying to build is a lexical analyzer. Maybe you should look up some documentation on the subject.

Answer 3

You will get the java.util.NoSuchElementException since you are calling st.nextToken() twice within the loop while (st.hasMoreElements())

Modifying harigm's example, you can then add t[i] to tokens as you require

String[] t = new String[200];
System.out.println(t[i]);
tokens.add(t[i]);

Answer 4

HI,

I have modified your code and Now works perfectly fine, check this

package org.sample;
import java.io.BufferedReader; 
import java.io.File; 
import java.io.FileReader; 
import java.io.IOException; 
import java.io.Reader; 
import java.util.ArrayList; 
import java.util.List; 
import java.util.Scanner; 
import java.util.StringTokenizer; 

public class FileTokenizer 
{ 
    private static final String DEFAULT_DELIMITERS = "< , { } >"; 
 // private static final String DEFAULT_TEST_FILE = "trans1.txt"; 

    public List<String> tokenize(Reader reader) throws IOException 
    { 
        List<String> tokens = new ArrayList<String>(); 

        BufferedReader br = null; 

        try 
        { 
            int i = 0; 
            br = new BufferedReader(reader); 
            Scanner scanner = new Scanner(br); 
            while (scanner.hasNext()) 
            { 

                StringTokenizer st = new StringTokenizer(scanner.next(), DEFAULT_DELIMITERS, true); 
                while (st.hasMoreElements()) 
                { 
                    String[] t = new String[200]; 
                    // tokens.add(st.nextToken()); 
                    //    t[i] = st.nextToken();  

                    System.out.println(t[i]); 

                    i++;                                       
                } 
            } 
        } 
        finally 
        { 
            close(br); 
        } 

        return tokens; 
    } 

    public static void close(Reader r) 
    { 
        try 
        { 
            if (r != null) 
            { 
                r.close(); 
            } 
        } 
        catch (IOException e) 
        { 
            e.printStackTrace(); 
        } 
    } 

   public static void main(String[] args) 
    { 
        try 
        { 
          //  String fileName = ((args.length > 0) ?  args[0] : DEFAULT_TEST_FILE); 
            FileReader fileReader = new FileReader(new File("c:\\DevTest\\1.txt")); 
            FileTokenizer fileTokenizer = new FileTokenizer(); 
            List<String> tokens = fileTokenizer.tokenize(fileReader); 
            //System.out.println(tokens); 
        } 
        catch (IOException e) 
        { 
            e.printStackTrace(); 
        } 
    } 
}