CODEWITHSUNDEEP
javaeclipsecounttoken
abooo
abooo

Reputation: 15

tokenizing and count token

I have TextView with text that changed dynamically. i want tokenizing this text with delimiter space " " and count token next send to another textview

this is my code

public void onClick(View v) {
    // TODO Auto-generated method stub
    if (v.getId()==R.id.button5){
        Intent i = new Intent(this, Tokenizing.class);

        String test = ((TextView)findViewById(R.id.textView6)).getText().toString();
        test = test.toLowerCase();
        test = test.replaceAll("\\W", " ");
        StringBuilder result = new StringBuilder();
        StringTokenizer st2 = new StringTokenizer(test);
            while (st2.hasMoreTokens()) {
                String st3 = st2.nextToken();
                System.out.println(st3 + st2.countTokens());
            //  System.out.println("Count Token" + st2.countTokens());
                result.append(st3+'\n');
        }

        i.putExtra("result", result.toString());
        startActivity(i);
        //Log.i("Test Klik Next", result);
    }

result

       stopwords 
       are 
       commonly 
       occurring 
       words

tokenizing process went well but I did not get the result count tokens, is there something wrong with my coding?

i want expected output like this

       (number of tokens)
        stopwords 
        are 
        commonly 
        occurring 
        words

Upvotes: 0

Views: 1386

Answers (2)

boxed__l
boxed__l

Reputation: 1336

StringTokenizer has a method countTokens() --> Calculates the number of times that this tokenizer's nextToken method can be called before it generates an exception.

Therefore, for the desired output it has to be called before the loop:

public void onClick(View v) {
    if (v.getId()==R.id.button5){
        Intent i = new Intent(this, Tokenizing.class);

        String test = ((TextView)findViewById(R.id.textView6)).getText().toString();
        test = test.toLowerCase();
        test = test.replaceAll("\\W", " ");
        StringBuilder result = new StringBuilder();
        StringTokenizer st2 = new StringTokenizer(test);
        int len=st2.countTokens();
        System.out.println(len);
        result.append(len+"\n");
            while (st2.hasMoreTokens()) {
                String st3 = st2.nextToken();
                System.out.println(st3);
                result.append(st3+'\n');
        }

        i.putExtra("result", result.toString());
        startActivity(i);
        //Log.i("Test Klik Next", result);
    }

Upvotes: 0

StoopidDonut
StoopidDonut

Reputation: 8617

You might want to add a line to your code just before you loop through your generated tokens:

result.append(st2.countTokens() + "\n");
while (st2.hasMoreTokens()) {

You might want to comment out the sops in your while loop to avoid confusion.

Alternatively, you can achieve this without iterating over tokens created by using a regular String split:

    String test = "This is a test String proving the concept";
    StringBuilder result = new StringBuilder();

    String[] tokens = test.split("\\s");
    result.append(tokens.length + "\n");

    for (String str:tokens) {
             result.append(str+'\n');
            }
     System.out.println(result);

Output:

8
This
is
a
test
String
proving
the
concept

Upvotes: 1

Related Questions