CODEWITHSUNDEEP
pythonparsingpython-2.7bioinformaticsphylogeny
Bulworth
Bulworth

Reputation: 49

Parsing through sequence output- Python

I have this data from sequencing a bacterial community. I know some basic Python and am in the midst of completing the codecademy tutorial. For practical purposes, please think of OTU as another word for "species"

Here is an example of the raw data:

OTU ID   OTU Sum Lineage
591820   1083    k__Bacteria; p__Fusobacteria; c__Fusobacteria; o__Fusobacteriales; f__Fusobacteriaceae; g__u114; s__
532752   517     k__Bacteria; p__Fusobacteria; c__Fusobacteria; o__Fusobacteriales; f__Fusobacteriaceae; g__u114; s__
218456   346     k__Bacteria; p__Proteobacteria; c__Betaproteobacteria; o__Burkholderiales; f__Alcaligenaceae; g__Bordetella; s__
590248   330     k__Bacteria; p__Proteobacteria; c__Betaproteobacteria; o__Burkholderiales; f__Alcaligenaceae; g__; s__
343284   321     k__Bacteria; p__Proteobacteria; c__Betaproteobacteria; o__Burkholderiales; f__Comamonadaceae; g__Limnohabitans; s__

The data includes three things: a reference number for the species, how many times that species is in the sample, and the taxonomy of said species.

What I'm trying to do is add up all the times a sequence is found for a taxonomic family (designated as f_x in the data).

Here is an example of the desired output:

f__Fusobacteriaceae 1600
f__Alcaligenaceae  676
f__Comamonadaceae  321

This isn't for a class. I started learning python a few months ago, so I'm at least capable of looking up any suggestions. I know how it works out from doing it the slow way (copy & paste in excel), so this is for future reference.

Upvotes: 2

Views: 144

Answers (4)

Nathan Stocks
Nathan Stocks

Reputation: 2164

That's very doable with basic python. Keep the Library Reference under your pillow, as you'll want to refer to it often.

You'll likely end up doing something similar to this (I'll write it the longer-more-readable way -- there's ways to compress the code and do this quicker).

# Open up a file handle
file_handle = open('myfile.txt')
# Discard the header line
file_handle.readline()

# Make a dictionary to store sums
sums = {}

# Loop through the rest of the lines
for line in file_handle.readlines():
    # Strip off the pesky newline at the end of each line.
    line = line.strip()

    # Put each white-space delimited ... whatever ... into items of a list.
    line_parts = line.split()

    # Get the first column
    reference_number = line_parts[0]

    # Get the second column, convert it to an integer
    sum = int(line_parts[1])

    # Loop through the taxonomies (the rest of the 'columns' separated by whitespace)
    for taxonomy in line_parts[2:]:
        # skip it if it doesn't start with 'f_'
        if not taxonomy.startswith('f_'):
            continue
        # remove the pesky semi-colon
        taxonomy = taxonomy.strip(';')
        if sums.has_key(taxonomy):
            sums[taxonomy] += int(sum)
        else:
            sums[taxonomy] = int(sum)

# All done, do some fancy reporting.  We'll leave sorting as an exercise to the reader.
for taxonomy in sums.keys():
    print("%s %d" % (taxonomy, sums[taxonomy]))

Upvotes: 0

Lukas Graf
Lukas Graf

Reputation: 32590

Yet another solution, using collections.Counter:

from collections import Counter

counter = Counter()

with open('data.txt') as f:
    # skip header line
    next(f)
    for line in f:
        # Strip line of extraneous whitespace
        line = line.strip()

        # Only process non-empty lines
        if line:
            # Split by consecutive whitespace, into 3 chunks (2 splits)
            otu_id, otu_sum, lineage = line.split(None, 2)

            # Split the lineage tree into a list of nodes
            lineage = [node.strip() for node in lineage.split(';')]

            # Extract family node (assuming there's only one)
            family = [node for node in lineage if node.startswith('f__')][0]

            # Increase count for this family by `otu_sum`
            counter[family] += int(otu_sum)

for family, count in counter.items():
    print "%s %s" % (family, count)

See the docs for str.split() for the details of the None argument (matching consecutive whitespace).

Upvotes: 1

Tim Pietzcker
Tim Pietzcker

Reputation: 336128

If the lines in your file really look like this, you can do

from collections import defaultdict
import re
nums = defaultdict(int)
with open("file.txt") as f:
    for line in f:
        items =  line.split(None, 2)  # Split twice on any whitespace
        if items[0].isdigit():
            key = re.search(r"f__\w+", items[2]).group(0)
            nums[key] += int(items[1])

Result:

>>> nums
defaultdict(<type 'int'>, {'f__Comamonadaceae': 321, 'f__Fusobacteriaceae': 1600, 
'f__Alcaligenaceae': 676})

Upvotes: 1

war8lock
war8lock

Reputation: 1

Get all your raw data and process it first, I mean structure it and then use the structured data to do any sort of operations you desire. In case if you have GB's of data you can use elasticsearch. Feed your raw data and query with your required string in this case f_* and get all entries and add them

Upvotes: 0

Related Questions