Python: Parsing between two lines with the same key word

Question

I know how to parse between two lines, when the start "target word" and the end "target word" are different

e.g. if I want to parse between X and Y:

parse = False
for line in open(sys.argv[1]):
 if Y in line:
    parse = False
if parse:
   print line
if X in line:
   parse = True

I'm stuck on a slightly different question, where the word I want to parse between is the same word. i.e. in this example, there are 4 different homolog groups, and i want to extract the human/mouse pairs in each homolog group, so I want to turn this file:

1:_HomoloGene:_141209.Gene_conserved_in_Mammals 
LOC102724657                            Homo_sapiens
Gm12569                                 Mus_musculus
2:_HomoloGene:_141208.Gene_conserved_in_Euarchontoglires    
LOC102724737                            Homo_sapiens
LOC102636216                            Mus_musculus
3:_HomoloGene:_141152.Gene_conserved_in_Euarchontoglires    
LOC728763                               Homo_sapiens
E030010N07Rik                           Mus_musculus
E030010N09Rik                           Mus_musculus
E030010N010Rik                          Mus_musculus
E030010N08Rik                           Mus_musculus
LOC102551034                            Rattus_norvegicus
4:_HomoloGene:_141054.Gene_conserved_in_Boreoeutheria       
LOC102723572                            Homo_sapiens
LOC102157295                            Canis_lupus_familiaris
LOC102633228                            Mus_musculus

Into a Homo_sapiens/Mus musculus comparison like this:

Homo_sapiens    Mus_musculus
LOC102724657    Gm12569 
LOC102724737    LOC102636216
LOC728763       E030010N07Rik
LOC728763       E030010N09Rik
LOC728763       E030010N010Rik
LOC728763       E030010N08Rik   
LOC102723572    LOC102633228

I don't have almost-successful code to show, this is an example of what I've tried (and I've also tried regular expressions and splitting the line on the word "HomoloGene"):

import sys
ListOfLines = open(sys.argv[1])
for line in ListOfLines:
        if "HomoloGene" in line:
                if "HomoloGene" in ListOfLines.next():
                        print line
                        print "**"
                else:
                        print ListOfLines.next()

Thanks

Answer 1

Its a two part problem. First you extact the homology groups into a dictionary, then you iterate through the groups and print the pairs.

#!/bin/python
import re
# Opens the text file
with open("genes.txt","r") as f:
    data = {}
    # reads the lines
    for line in f.readlines():
        # When there is a : at the line start -> new group
        match = re.search("^([0-9]+):",line)
        if match:
            # extracts the group number and puts it to the dict
            group = match.group(1)
            # adds the species as entries with empty lists as values
            data[str(group)] = { "Homo_sapiens":[] , "Mus_musculus":[]}
        else:
            # splits the line (also removes the \n)
            text = line.replace("\n","").split()
            # if the species is in the group, add the gene name to the list
            if text[1] in data[group].keys():
                data[group][text[1]].append(text[0])
# Here you go with your parsed data
print data
# Now we feed it into the text format you want
print "Homo_sapiens\t\tMus_musculus"
# go through groups
for gr in data:
    # go through the Hs genes
    for hs_gene in data[gr]["Homo_sapiens"]:
        # get all the associated Ms genes
        for ms_gene in data[gr]["Mus_musculus"]:
            # print the pairs
            print hs_gene+"\t\t"+ms_gene

Hope this helps.

Answer 2

The commented code below produces the result in your example. To understand it you may want to read the following:

• Regular expressions and tutorial
• Unpacking argument lists
• defaultdict
• itertools.product

The Code:

import sys
import re
from collections import defaultdict
import itertools

#define the pairs of words we want to compare
compare = ['Homo_sapiens', 'Mus_musculus']

#define some regular expressions to split up the input data file
#this searches for a digit, a colon, and matches the rest of the line
group_re = re.compile("\n?\d+:.*\n")
#this matches non-whitespace, followed by whitespace, and then non-whitespace, returning the two non-whitespace sections
line_re = re.compile("(\S+)\s+(\S+)")

#to store our resulting comparisons
comparison = []

#open and read in the datafile
datafile = open(sys.argv[1]).read()
#use our regular expression to split the datafile into homolog groups
for dataset in group_re.split(datafile):
    #ignore empty matches
    if dataset.strip()=='': continue
    #split our group into lines
    dataset = dataset.split('\n')
    #use our regular expression to match each line, pulling out the two bits of data
    dataset = [line_re.match(line).groups() for line in dataset if line.strip()!='']
    #build a dictionary to store our words
    words = defaultdict(list)
    #loop through our group dataset, grouping each line by its word
    for v, k in dataset: words[k].append(v)
    #add the results to our output list. Note here we are unpacking an argument list
    comparison+=itertools.product(*[words[w] for w in compare])

#print out the words we wanted to compare
print('\t'.join(compare))
#loop through our output dataset
for combination in comparison:
    #print each comparison, spaced with a tab character
    print('\t'.join(combination))