CODEWITHSUNDEEP
pythonparsingtext-processing
demos
demos

Reputation: 2630

Splitting words in running text using Python?

I am writing a piece of code which will extract words from running text. This text can contain delimiters like \r,\n etc. which might be there in text.

I want to discard all these delimiters and only extract full words. How can I do this with Python? any library available for crunching text in python?

Upvotes: 1

Views: 366

Answers (2)

Stephen
Stephen

Reputation: 49166

Assuming your delimiters are whitespace characters (like space, \r and \n), then basic str.split() does what you want:

>>> "asdf\nfoo\r\nbar too\tbaz".split()
['asdf', 'foo', 'bar', 'too', 'baz']

Upvotes: 1

Alex Martelli
Alex Martelli

Reputation: 881705

Assuming your definition of "word" agrees with that of the regular expression module (re), that is, letters, digits and underscores, it's easy:

import re
fullwords = re.findall(r'\w+', thetext)

where thetext is the string in question (e.g., coming from an f.read() of a file object f open for reading, if that's where you get your text from).

If you define words differently (e.g. you want to include apostrophes so for example "it's" will be considered "one word"), it isn't much harder -- just use as the first argument of findall the appropriate pattern, e.g. r"[\w']+" for the apostrophe case.

If you need to be very, very sophisticated (e.g., deal with languages that use no breaks between words), then the problem suddenly becomes much harder and you'll need some third-party package like nltk.

Upvotes: 5

Related Questions