CODEWITHSUNDEEP
rubyregexmongodbhash
Domness
Domness

Reputation: 7605

Ruby & Mongodb with an inverted index brings up some funny results

For my program I am creating an inverted index using data from the Twitter feed, however, when parsing and putting them into the mongodb, some funny problems occur.

The usual type of entry should be like this:

{"ax"=>1, "easyjet"=>1, "from"=>2}

However, when parsing some of the tweets they end up like this in the db:

{""=>{""=>{""=>{""=>{""=>{"giants"=>{"dhem"=>1, "giants"=>1, "giantss"=>1}}}}

I have got these lines that split a tweet up and increments the values in the db:

def pull_hash_tags(tweet, lang)
    hash_tags = tweet.split.find_all { |word| /^#.+/.match word }
    t = tweet.gsub(/https?:\/\/[\S]+/,"") # removing urls
    t = t.gsub(/#\w+/,"") # removing hash tags
    t = t.gsub(/[^0-9a-z ]/i, '') # removing non-alphanumerics and keeping spaces
    t = t.gsub(/\r/," ")
    t = t.gsub(/\n/," ")
    hash_tags.each { |tag| add_to_hash(lang, tag, t) }
end

def add_to_hash(lang, tag, t)
    t.gsub(/\W+/, ' ').split.each { |word| @db.collection.update({"_id" => lang}, {"$inc" => {"#{tag}.#{word}" => 1}}, { :upsert => true }) }
end

I'm trying to get normal words (with only alphanumeric characters) and no double spaces, and no carriage returns etc.

Upvotes: 3

Views: 253

Answers (1)

Michael Frederick
Michael Frederick

Reputation: 16714

You should add t.strip! as it seems the problem might be leading/trailing whitespace.

Upvotes: 2

Related Questions