CODEWITHSUNDEEP
amazon-web-servicesgoamazon-s3
Daffodil
Daffodil

Reputation: 69

Reading last modified file name from Amazon S3 bucket in GO

I need to read the filenames in an amazon s3 bucket using Golang. The bucket contains csv files mainly with 2 types of name formats.

1. uploaded/2022-03-21-18:31:06.608058.csv
2. overwritten/2022-03-22-18:31:06.608058.csv

I need to find out the name of the last modified file with uploaded prefix. (The bucket in question contains 1000s of files.)

Any help is much appreciated.

Upvotes: 1

Views: 2663

Answers (1)

reda la
reda la

Reputation: 840

As already mentioned in the comments:

You have to list all keys started with the given prefix, then (in your code) sort and find the wanted object key.

    ...

    objs := []types.Object{}
    params := &s3.ListObjectsV2Input{
        Bucket:     aws.String(s.bucket),
        Prefix:     aws.String(prefix), //uploaded
    }
    p := s3.NewListObjectsV2Paginator(svc, params)

    for p.HasMorePages() {
        out, err := p.NextPage(ctx)
        if err != nil {
            return nil, err
        }
        objs = append(objs, out.Contents...)
    }

    if l :=len(objs); l > 0 {
        sort.Slice(objs, func(a, b int) bool {
            return objs[a].LastModified.Before(*objs[b].LastModified)
        })

        return objs[l-1].Key, nil
    }

    return "", nil

If you are trying to find the object key with the most recent DateTime (within the key's name), then you can use the StartAfter parameter to avoid listing all available keys.

    ...

    lastModifiedKey := ""

    params := &s3.ListObjectsV2Input{
        Bucket:     aws.String(s.bucket),
        Prefix:     aws.String(prefix), //uploaded
        StartAfter: aws.String(previousSearchKey),
    }
    p := s3.NewListObjectsV2Paginator(svc, params)

    for p.HasMorePages() {
        out, err := p.NextPage(ctx)
        if err != nil {
            return nil, err
        }
        // ListObjectV2 sorts key by ascending order & your name format is sortable
        lastModifiedKey =  out.Contents[len(out.Contents) -1]
    }

    return lastModifiedKey, nil

Note that StartAfter key doesn't require to be an existing key in the bucket.

It may be the result of a previous search, or a guessed value that you can assert is behind (and preferably close to the wanted result).

Upvotes: 1

Related Questions