How can I remove the stop words with a non-word boundary?

Question

I'm trying to remove a stop word in a txt file using from my stop word list. Some of the stop words are removed bot some are not.

Example this sentence: "it taste nice, doesn’t it?" should have an output like "taste nice" but my code output: "taste nice doesnt it "

my stop word list is from: https://www.ranks.nl/stopwords (Long stop word list).

Here is my code:

    public static void main(String[] args) {

    ArrayList sw = new ArrayList<>();

    try{
        FileInputStream fis = new FileInputStream("/Users/Dan/Desktop/DATA/stopwords.txt");

        byte b[] = new byte[fis.available()];
        fis.read(b);
            fis.close();

            String data[] = new String(b).split("\n");

        for(int i = 0; i < data.length; i++)
        {
            sw.add(data[i].trim());
        }
         FileInputStream fis2 = new FileInputStream("/Users/Dan/Desktop/DATA/cleandata.txt");

        byte bb[] = new byte[fis2.available()];
        fis2.read(bb);
            fis2.close();

            String data2[] = new String(bb).split("\n");



            for(int i = 0; i < data2.length; i++)

        {
            String file = "";
            String s[] = data2[i].split("\\s");
            for(int j = 0; j < s.length; j++)
            {
                if(!(sw.contains(s[j].trim().toLowerCase())))
                {
                    file=file + s[j] + " ";
                }

            }
            file = file.replaceAll("[^a-zA-Z\\s+]", "");

            System.out.println(file.replaceAll("\\s+", " ").toLowerCase() + "\n");

        }

    } catch(Exception a){
        a.printStackTrace();
    }

}

What should I do? I think I have a problem in printing

file = file.replaceAll("[^a-zA-Z\\s+]", "");

System.out.println(file.replaceAll("\\s+", " ").toLowerCase() + "\n");

Answer 1

There are two different quote characters being used. The stopwords file contains doesn't and your input contains doesn’t.

Because the quotes are different, the words don't match.

EDIT: Here's a slightly refactored solution which generates the correct output (if you don't use weird quotes in the input that is).

import java.util.Arrays;
import java.util.HashSet;
import java.util.Scanner;
import java.util.Set;
import java.util.stream.Collectors;

public class StopWordsApp {

    // the platform-specific end of line token
    private static final String EOL = String.format("%n");

    private final Set<String> stopWords = new HashSet<>(Arrays.asList(readLines("stopwords.txt")));

    public static void main(String[] args) {
        StopWordsApp stopWordsApp = new StopWordsApp();
        String[] lines = readLines("cleandata.txt");
        printLines(stopWordsApp.removeStopWords(lines));
    }

    private String[] removeStopWords(String[] inputLines) {
        return Arrays.stream(inputLines)
                // map the String array to a Line object
                .map(Line::new)
                // map the Line to a String without stop words
                .map(this::removeStopWords)
                // convert the stream to an array
                .toArray(String[]::new);
    }

    private String removeStopWords(Line line) {
        return line.words().stream()
                // map the word to its normalized version
                .map(Word::normalized)
                // remove stop words
                .filter(n -> !stopWords.contains(n))
                // join into a String separated by spaces
                .collect(Collectors.joining(" "));
    }

    private static String[] readLines(String fileName) {
        return readFile(fileName).split(EOL);
    }

    private static String readFile(String fileName) {
        return new Scanner(StopWordsApp.class.getResourceAsStream(fileName), "UTF-8").useDelimiter("\\A").next();
    }

    private static void printLines(String[] lines) {
        for (String line : lines) {
            System.out.println(line);
        }
    }
}

I extracted separate classes for a Line:

import java.util.Arrays;
import java.util.List;
import java.util.stream.Collectors;

public class Line {

    private final List<Word> words;

    public Line(String input) {
        String[] wordInputs = input.split("\\s+");
        words = Arrays.stream(wordInputs)
                // remove empty Strings
                .filter(v -> !v.isEmpty())
                // map String to a Word object
                .map(Word::new)
                // collect into a List
                .collect(Collectors.toList());
    }

    public List<Word> words() {
        return words;
    }

}

..and for a Word:

public class Word {

    private final String normalized;

    public Word(String input) {
        normalized = input
                // convert to lower case
                .toLowerCase()
                // remove everything that's not a lower case letter or a quote
                // (the stopwords file only contains lower case letters and quotes)
                .replaceAll("[^a-z']", "")
                // replace consecutive white space with a single space
                .replaceAll("\\s+", " ")
                // trim any white space at the edges
                .trim();
    }

    public String normalized() {
        return normalized;
    }

}

... and a custom (runtime) exception:

public class StopWordsException extends RuntimeException {
    public StopWordsException(Exception e) {
        super(e);
    }
}

I used Java 8 streams everywhere and added comments to explain what's going on.

With the input:

it Taste nice, doesn't it?

The output is:

taste nice

P.S. The files 'stopwords.txt' and 'cleandata.txt' need to be in the same package as the StopWordsApp class.