CODEWITHSUNDEEP
pythonhtmltwitterwhile-loopweb-crawler
snehoozle
snehoozle

Reputation: 273

While Loops Problem with Python Twitter Crawler

I'm continuing writing my twitter crawler and am running into more problems. Take a look at the code below: 

from BeautifulSoup import BeautifulSoup
import re
import urllib2

url = 'http://mobile.twitter.com/NYTimesKrugman'

def gettweets(soup):
    tags = soup.findAll('div', {'class' : "list-tweet"})#to obtain tweet of a follower
    for tag in tags: 
        print tag.renderContents()
        print ('\n\n')

def are_more_tweets(soup):#to check whether there is more than one page on mobile 
    links = soup.findAll('a', {'href': True}, {id: 'more_link'})
    for link in links:
        b = link.renderContents()
        test_b = str(b)
        if test_b.find('more'):
            return True
        else:
            return False

def getnewlink(soup): #to get the link to go to the next page of tweets on twitter 
    links = soup.findAll('a', {'href': True}, {id : 'more_link'})
    for link in links:
        b = link.renderContents()
        if str(b) == 'more':
            c = link['href']
            d = 'http://mobile.twitter.com' +c
            return d

def checkforstamp(soup): # the parser scans a webpage to check if any of the tweets are   
    times = soup.findAll('a', {'href': True}, {'class': 'status_link'})
    for time in times:
        stamp = time.renderContents()
        test_stamp = str(stamp)
        if test_stamp.find('month'): 
            return True
        else:
            return False


response = urllib2.urlopen(url)
html = response.read()
soup = BeautifulSoup(html)
gettweets(soup)
stamp = checkforstamp(soup)
tweets = are_more_tweets(soup)
print 'stamp' + str(stamp)
print 'tweets' +str (tweets)
while (not stamp) and tweets: 
    b = getnewlink(soup)
    print b
    red = urllib2.urlopen(b)
    html = red.read()
    soup = BeautifulSoup(html)
    gettweets(soup)
    stamp = checkforstamp(soup)
    tweets = are_more_tweets(soup)
print 'done'

The code works in the following way: For a single user NYTimesKrugman -I obtain all tweets on a single page(gettweets) -provided more tweets exist(are more tweets) and that I haven't obtained a month of tweets yet(checkforstamp), I get the link for the next page of tweets -I go to the next page of tweets (entering the while loop) and continue the process until one of the above conditions is violated

However, I have done extensive testing and determined that I am not actually able to enter the while loop. Rather, the program is not doing so. This is strange, because my code is written such that tweets are true and stamp should yield false. However, I'm getting the below results: I am truly baffled! 

<div>
<span>
<strong><a href="http://mobile.twitter.com/nytimeskrugman">NYTimeskrugman</a></strong>
<span class="status">What Would I Have Done? <a rel="nofollow"   href="http://nyti.ms/nHxb8L" target="_blank" class="twitter_external_link">http://nyti.ms/nHxb8L</a></span>
</span>
<div class="list-tweet-status">
<a href="/nytimeskrugman/status/98046724089716739" class="status_link">3 days ago</a>
</div>
<div class="list-tweet-actions">
</div>
</div>




stampTrue
tweetsTrue
done
>>>

If someone could help that'd be great. Why can I not get more than 1 page of tweets? Is my parsing in checkstamp being done incorrectly? Thanx.

Upvotes: 0

Views: 372

Answers (2)

Mat
Mat

Reputation: 206679

Your checkforstamp function returns non-empty, defined strings:

return 'True'

So (not stamp) will always be false.

Change it to return booleans like are_more_tweets does:

return True

and it should be fine.

For reference, see the boolean operations documentation:

In the context of Boolean operations, and also when expressions are used by control flow statements, the following values are interpreted as false: False, None, numeric zero of all types, and empty strings and containers (including strings, tuples, lists, dictionaries, sets and frozensets). All other values are interpreted as true.

...

The operator not yields True if its argument is false, False otherwise.

Edit:

Same problem with the if test in checkforstamp. Since find('substr') returns -1 when the substring is not found, str.find('substr') in boolean context will be True if there is no match according to the rules above.

That is not the only place in your code where this problem appears. Please review all your tests.

Upvotes: 1

agf
agf

Reputation: 176730

if test_stamp.find('month'):

will evaluate to True if it doesn't find month, because it returns -1 when it doesn't find the substring. It would only evaluate to False here if month was at the beginning of the string, so its position was 0.

You need

if test_stamp.find('month') != -1:

or just

return test_stamp.find('month') != -1

Upvotes: 1

Related Questions