CODEWITHSUNDEEP
pythonweb-scrapinglxml
lela_98
lela_98

Reputation: 61

Determining most common name from web scraped birth name data

I have the task to do web scraping from this page https://www.ssa.gov/cgi-bin/popularnames.cgi. There you can find a list of the most common birth names. Now I have to find the most common name that both girls and boys have for a given year (in other words, the exact same name is used in both genders), but I don't know how I am able to do that. With the code below I solved the previous task to output the list for a given year but I have no clue how I can modify my code so I get the most common name that both girls and boys have. 

import requests
import lxml.html as lh


url = 'https://www.ssa.gov/cgi-bin/popularnames.cgi'
string = input("Year: ")
r = requests.post(url, data=dict(year=string, top="1000", number="n" ))



doc = lh.fromstring(r.content)
tr_elements = doc.xpath('//table[2]//td[2]//tr')
cols = []


for col in tr_elements[0]:
    name = col.text_content()
    number = col.text_content()
    cols.append((number, []))


count=0
for row in tr_elements[1:]:
    i = 0
    for col in row:
        val = col.text_content()
        cols[i][1].append(val)
        i += 1
        if(count<4):
            print(val, end = '  ')
            count += 1
        else:
            count=0
            print(val)

Upvotes: 1

Views: 129

Answers (1)

ggorlen
ggorlen

Reputation: 56865

Here's one approach. The first step is to group the data by name and record how many genders have used the name and their aggregate total. After that, we can filter the structure by names with more than one gender using it. Finally, we sort this multi-gender list by counts and take the 0-th element. This is our most popular multi-gender name for the year.

import requests
import lxml.html as lh

url = "https://www.ssa.gov/cgi-bin/popularnames.cgi"
year = input("Year: ")
response = requests.post(url, data=dict(year=year, top="1000", number="n"))
doc = lh.fromstring(response.content)
tr_elements = doc.xpath("//table[2]//td[2]//tr")
column_names = [col.text_content() for col in tr_elements[0]]
names = {}
most_common_shared_names_by_year = {}

for row in tr_elements[1:-1]:
    row = [cell.text_content() for cell in row]

    for i, gender in ((1, "male"), (3, "female")):
        if row[i] not in names:
            names[row[i]] = {"count": 0, "genders": set()}

        names[row[i]]["count"] += int(row[i+1].replace(",", "")) 
        names[row[i]]["genders"].add(gender)

shared_names = [
    (name, data) for name, data in names.items() if len(data["genders"]) > 1
]
most_common_shared_names = sorted(shared_names, key=lambda x: -x[1]["count"])
print("%s => %s" % most_common_shared_names[0])

If you're curious, here are the results since 2000:

2000 => Tyler, 22187
2001 => Tyler, 19842
2002 => Tyler, 18788
2003 => Ryan, 20171
2004 => Madison, 20829
2005 => Ryan, 18661
2006 => Ryan, 17116
2007 => Jayden, 17287
2008 => Jayden, 19040
2009 => Jayden, 19053
2010 => Jayden, 18641
2011 => Jayden, 18064
2012 => Jayden, 16952
2013 => Jayden, 15462
2014 => Logan, 14478
2015 => Logan, 13753
2016 => Logan, 12099
2017 => Logan, 15117

Upvotes: 1

Related Questions