CODEWITHSUNDEEP
amazon-web-servicesamazon-s3aws-cloudformationaws-cloud9aws-data-wrangler
fervifervi
fervifervi

Reputation: 1

How to upload periodically a .txt file to S3?

I need to convert a Dataframe into a .txt file and upload it to S3 by using CloudFormation and Cloud9 periodically. I'm used to do it with excel file and no problems found in it as wrangler brings a 'to_excel' function.

This is how I use to do it.

import awswrangler as wr
import boto3
import os
import pandas as pd

def add_partition_cols_to_path(today_date, path):

    path += 'year='+today_date.strftime('%Y')+'/month='+today_date.strftime('%m')+'/day='+today_date.strftime('%d')+'/ITGFSCEP'+ today_date.strftime('%Y') + today_date.strftime('%m') + today_date.strftime('%d') + '.xlsx'
    
    return path

def write_file(df, path):
    try:
        if df.empty:
            raise TypeError("Empty Dataframe")
        logger.info('Writing files in:  %s', path) 
        wr.s3.to_excel(df, path, index=False)
    except Exception as e:
        logger.error("Error writing files: %s", e)
        raise e

But I need it to convert the dataframe to .txt file format.

Now It would be good for me a function called to_txt doing parallel than to_excel but that doesn't exist (as far as I know).

I also tried using df.to_csv function:

 texto = df.to_csv('data.txt', sep='',index=False)

But It force me to introduce a 1-character string delimiter which I really don't want as I need altogether rows (without delimiter).

Like this:

00566XX202304261400000161500000000000000041385EUR

What can I do?

Upvotes: 0

Views: 120

Answers (0)

Related Questions