CODEWITHSUNDEEP
pythonfilesplit
Reks bot
Reks bot

Reputation: 25

Python Split files into multiple smaller files

I have this problem statement

Write a function named file_split(filename, number_of_files) that will split an input file into a number of output files. The files should be split as evenly as possible. When the file length is evenly divisible by the number of files to create (a 10-line file, split into 2 files, each output file should have 5 lines. When the length is not evenly divisible all output files’ length must not have a difference greater than 1. For example, a file of 10 lines, split in 3, would have output files of length 3, 3 and 4.

I have written my code but I can not figure out how to do the difference greater than 1 part, I need help modifying my code to include that part. (The code I have creates a new file for the last line if it is not an even multiple)

def get_line_counts(filename, number_of_files):
    try:
        my_file = open(filename, 'r')
    except IOError:
        print("File does not exist")
        return    
    input = my_file.read().split('\n')
    outputBase = 'lel'    
    total_lines = 0
    with open('myfile.txt') as infp:
        for line in infp:
            if line.strip():  
                total_lines +=1    
    base_size = total_lines // number_of_files    
    at = 1
    for lines in range(0, len(input), base_size):
        outputData = input[lines:lines+base_size]
        output = open(outputBase + str(at) + '.txt', 'w')
        output.write('\n'.join(outputData))
        output.close()
        at += 1

Upvotes: 2

Views: 15057

Answers (4)

roshan ok
roshan ok

Reputation: 383

Check this https://github.com/roshanok/SplitAndCompile

--------------------------------------------------------------
Usage: SplitAndCombine.py [-h] [-i INPUT] [-s] [-n CHUNK] [-m]\n
optional arguments:
-h, --help            show this help message and exit
-i INPUT, --input INPUT
                    Provide the File that needs to be Split
-s, --split           To Split the File
-n CHUNK, --chunk CHUNK
                    [n]: No. of files to be created
                    [n]kb : Split the file in nKB size
                    [n]b : Split the file in nb size
                    [n]mb : Split the file in nmb size
                    [n]gb : Split the file in ngb size

-m, --merge           Merge the Files

Upvotes: 0

Abdull
Abdull

Reputation: 11

from future import print_function

import boto3
import shutil
import os
import os.path
import urllib
import json
import urllib2
import subprocess
import linecache
import sys

s3client = boto3.client('s3')
s3 = boto3.resource('s3')
def lambda_handler(event, context):
    try:
        for record in event['Records']:
            bucket = record['s3']['bucket']['name']
            key = record['s3']['object']['key']
            print(key)
            p = key.rsplit('/',1)
            keyfile =p[1]
            print("S Object: " + keyfile + " is a FILE")
            inpfilename = keyfile
            ou = inpfilename.split('.',1)
            outfilename = ou[0]
            print("inpfilename :" + inpfilename)
            body = s3client.get_object(
                                        Bucket=bucket,
                                        Key=key
                                        )["Body"].read().split('\n')

            lines_per_file = 3  # Lines on each small file
            created_files = 0  # Counting how many small files have been created
            op_rec=''     # Stores lines not yet written on a small file
            lines_counter = 0  # Same as len(lines)
            for line in body:  # Go throught the whole big file
                op_rec = op_rec + '\n' + line
                lines_counter += 1
                if lines_counter == lines_per_file:
                    idx = lines_per_file * (created_files + 1)
                    body_contents = str(op_rec)
                    file_name = "%s_%s.txt" %(outfilename, idx)
                    target_file = "folder-name/" + file_name
                    print(target_file)
                    s3client.put_object(ACL='public-read',ServerSideEncryption='AES256', Bucket='bucket-name',Key= target_file, Body=body_contents )
                    op_rec =''  # Reset variables
                    lines_counter = 0
                    created_files += 1  # One more small file has been created
            # After for-loop has finished
            if lines_counter:  # There are still some lines not written on a file?
                idx = lines_per_file * (created_files + 1)
                body_contents = str(op_rec)
                file_name = "%s_%s.txt" %(outfilename, idx)
                target_file = "folder-name/" + file_name
                print(target_file)
                s3client.put_object(ACL='public-read',ServerSideEncryption='AES256', Bucket='bucket-name',Key= target_file, Body=body_contents )
                created_files += 1

            print ('%s small files (with %s lines each) were created.' % (created_files,lines_per_file))


    except Exception as e:
           print(e)

Upvotes: 0

Stefan Pochmann
Stefan Pochmann

Reputation: 28596

Round-robin works and is easy:

with open('myfile.txt') as infp:
    files = [open('%d.txt' % i, 'w') for i in range(number_of_files)]
    for i, line in enumerate(infp):
        files[i % number_of_files].write(line)
    for f in files:
        f.close()

Upvotes: 5

fiacre
fiacre

Reputation: 1180

Untested:

I'd use modular arithmetic 

res = len(lines) % number_of_files
for lines in range(0, len(input), base_size):
    if at == len(input)+res+1:
        outputData = input[lines:-1]
    else: 
       ...

That is, just dump the remainder of the lines into the last file.

Upvotes: 0

Related Questions