CODEWITHSUNDEEP
c#regex
Dean
Dean

Reputation: 1550

How do I split a phrase into words using Regex in C#

I am trying to split a sentence/phrase in to words using Regex.

var phrase = "This isn't a test.";
var words = Regex.Split(phrase, @"\W+").ToList();

words contains "This", "isn", "t", "a", "test"

Obviously it's picking up the apostrophe and splitting on that. Can I change this behavior? It also needs to be multilingual supporting a variety of languages (Spanish, French, Russian, Korean, etc...).

I need to pass the words in to a spellchecker. Specifically Nhunspell.

return (from word in words let correct = _engine[langId].Spell(word) where !correct select word).ToList();

Upvotes: 8

Views: 13460

Answers (8)

jessehouwing
jessehouwing

Reputation: 114641

Due to the fact that a number of languages use very complex rules to string words together into phrases and sentences, you can't rely on a simple Regular Expression to get all the words from a piece of text. Even for a language as 'simple' as English you'll run in a number of corner cases such as:

  • How to handle words like you're, isn't where there's two words combined and a number of characters replaces with '.
  • How to handle abbreviations such as Mr. Mrs. i.e.
  • combined words using '-'
  • hyphenated words at the end of a sentence.
  • Names like O'Brian and O'Connel.

Chinese and Japanese (among others) are notoriously hard to parse this way, as these languages do not use spaces between words, only between sentences.

You might want to read up on Text Segmentation and if the segmentation is important to you invest in a Spell Checker that can parse a whole text or a Text Segmentation engine which can split your sentences up into words according to the rules of the language.

I couldn't find a .NET based multi-lingual segmentation engine with a quick google search though. Sorry.

Upvotes: 4

maiconmm
maiconmm

Reputation: 311

This worked for me: [^(\d|\s|\W)]*

Upvotes: 0

user1344280
user1344280

Reputation:

If you want to split into words for spell checking purposes, this is a good solution:

new Regex(@"[^\p{L}]*\p{Z}[^\p{L}]*")

Basically you can use Regex.Split using the previous regex. It uses unicode syntax so it would work in several languages (not for most asian though). And it won't break words with apostrophes ot hyphens.

Upvotes: 11

user557597
user557597

Reputation:

I'm not a java person but you could try to exclude punctuation while splitting on
spaces at the same time. Something like this maybe.

These are raw and expanded regexes, the words are in capture group 1.
Do a global search.

Unicode (doesen't account for grapheme's) 

[\s\pP]* ([\pL\pN_-] (?: [\pL\pN_-] | \pP(?=[\pL\pN\pP_-]) )* )

Ascii 

[\s[:punct:]]* (\w (?: \w | [[:punct:]](?=[\w[:punct:]]) )* )

Upvotes: 0

David Z.
David Z.

Reputation: 5701

You can try if you're trying to split based on spaces only.

var words = Regex.Split(phrase, @"[^ ]+").ToList();

The other approach is to add the apostrophe by adding that to your character class.

var words = Regex.Split(phrase, @"(\W|')+").ToList();

Otherwise, is there a specific reason that you cannot use string.Split()? This would seem much more straightforward. Also, you would also be able to pass in other punctuation characters (i.e. split on . as well as spaces).

var words = phrase.Split(' ');
var words = phrase.Split(new char[] {' ', '.'});

Upvotes: 1

Jack
Jack

Reputation: 5768

Use Split().

words = phrase.Split(' ');

Without punctuation.

words = phrase.Split(new Char [] {' ', ',', '.', ':', , ';', '!', '?', '\t'});

Upvotes: 3

Jim Mischel
Jim Mischel

Reputation: 133995

What do you want to split on? Spaces? Punctuation? You have to decide what the stop characters are. A simple regex that uses space and a few punctuation characters would be "[^.?!\s]+". That would split on period, question mark, exclamation, and any whitespace characters.

Upvotes: 1

Michael Frederick
Michael Frederick

Reputation: 16714

It doesn't really seem like you need a regex. You could just do:

phrase.Split(" ");

Upvotes: 0

Related Questions