CODEWITHSUNDEEP
pythonregexweburllib2
Jérémz
Jérémz

Reputation: 393

Python: extract string on webtext from a list of URL

I have a list of web text URL that I need to extract information from and then store these information in a list. the string I need to extract always start with (P: OR C: OR F:) and always end by a ";". I have trouble to make that work all together, any help would be greatly appreciated.

An example of webtext from one of the URL:

DR   Proteomes; UP000005640; Chromosome 3.
DR   Bgee; C9J872; -.
DR   ExpressionAtlas; C9J872; baseline and differential.
DR   GO; GO:0005634; C:nucleus; IBA:GO_Central.
DR   GO; GO:0005667; C:transcription factor complex; IEA:InterPro.
DR   GO; GO:0003677; F:DNA binding; IEA:UniProtKB-KW.
DR   GO; GO:0000981; F:sequence-specific DNA binding RNA polymerase II transcription factor activity; IBA:GO_Central.
DR   GO; GO:0003712; F:transcription cofactor activity; IEA:InterPro.
DR   GO; GO:0000278; P:mitotic cell cycle; IEA:InterPro.

Here the result expected for searching after C:

['nucleus', 'transcription factor complex']

But it also need to go through the different URL and append in the same list

An example of what I have tried so far without success:

import urllib2
import sys
import re
IDlist = ['C9JVZ1', 'C9JLN0', 'C9J872']

URLlist = ["http://www.uniprot.org/uniprot/"+x+".txt" for x in IDlist]
function_list = []
for item in URLlist:
    textfile = urllib2.urlopen(item)
    myfile = textfile.read()
    for line in myfile:
        function = re.search('P:(.+?);', line).group(1)
        function_list.append(function)

Upvotes: 1

Views: 211

Answers (1)

Prune
Prune

Reputation: 77860

Here's an updated file that includes your dictionary. Note that I changed the loop control to key on the file ID: that ID serves as the dictionary key.

import urllib2
import re

IDlist = ['C9JVZ1', 'C9JLN0', 'C9J872']
function_dict = {}

# Cycle through the data files, keyed by ID
for id in IDlist:

    # Start a new list of functions for this file.
    # Open the file and read line by line.
    function_list = []
    textfile = urllib2.urlopen("http://www.uniprot.org/uniprot/"+id+".txt")
    myfile = textfile.readlines()

    for line in myfile:

        # When you find a function tag, extract the function and add it to the list.
        found = re.search(' [PCF]:(.+?);', line)
        if found:
            function = found.group(1)
            function_list.append(function)

    # At end of file, insert the list into the dictionary.
    function_dict[id] = function_list

print function_dict

The output I get from your data is

{'C9JVZ1': [], 'C9J872': ['nucleus', 'transcription factor complex', 'DNA binding', 'sequence-specific DNA binding RNA polymerase II transcription factor activity', 'transcription cofactor activity', 'mitotic cell cycle', 'regulation of transcription from RNA polymerase II promoter', 'transcription, DNA-templated'], 'C9JLN0': ['cytosol']}

Upvotes: 1

Related Questions