AWS S3 Presigned Request Cache

Question

I want to store user profile pictures in an S3 bucket, but keep these images private. In order to do this, I am creating a presigned url whenever the image is required. However, this creates a unique url each time which means the image will never be cached by the browser and I'll end up paying a lot more in GET requests.

Here's an example of my code to generate the url, I'm using Laravel:

$s3 = \Storage::disk('s3');
$client = $s3->getDriver()->getAdapter()->getClient();
$expiry = new \DateTime('2017-07-25');

$command = $client->getCommand('GetObject', [
    'Bucket' => \Config::get('filesystems.disks.s3.bucket'),
    'Key'    => $key
]);

$request = $client->createPresignedRequest($command, $expiry);

return (string) $request->getUri();

I thought that by specifying a datetime rather a unit of time that it would create the same url but it actually adds the number of seconds remaining to the url, here's an example:

xxxx.s3.eu-west-2.amazonaws.com/profile-pics/92323.png?X-Amz-Content-Sha256=UNSIGNED-PAYLOAD&X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AXXXXXXXXXXX%2Feu-west-2%2Fs3%2Faws4_request&X-Amz-Date=20170720T112123Z&X-Amz-SignedHeaders=host&X-Amz-Expires=391117&X-Amz-Signature=XXXXXXXXX

Is it possible to generate a repeatable presigned request url so that an image may be cached by the users browser?

Answer 1

Extending the solution from Francisco Escher to golang v2, something along the lines of this may work:

import (
    "context"
    "log"
    "net/http"
    "time"

    "github.com/aws/aws-sdk-go-v2/aws"
    v4 "github.com/aws/aws-sdk-go-v2/aws/signer/v4"
    "github.com/aws/aws-sdk-go-v2/service/s3"
)

// Lifted from https://stackoverflow.com/a/70249671

// Will create same url if in the same 15 minutes time bucket
const presignPeriod = 15 * time.Minute

// getSignTime function returns the signing time
// (initial time) for the time bucket
func getSignTime() time.Time {
    now := time.Now().UTC()
    signTime := now.Round(presignPeriod)
    if signTime.After(now) {
        signTime = signTime.Add(-presignPeriod)
    }
    return signTime
}

// HTTPPresignerV4 is a wrapper around the v4.Signer that implements the
// PresignHTTP method.
type HTTPPresignerV4 struct {
    signer v4.Signer
}

func (client *HTTPPresignerV4) PresignHTTP(ctx context.Context, credentials aws.Credentials, r *http.Request,
    payloadHash string, service string, region string, signingTime time.Time,
    optFns ...func(*v4.SignerOptions),
) (url string, signedHeader http.Header, err error) {
    // Get the signing time
    signTime := getSignTime()

    // Sign the request
    signedUrl, signedHeaders, err := client.signer.PresignHTTP(context.Background(), credentials, r, payloadHash, service, region, signTime)

    if err != nil {
        log.Printf("Failed to sign URL: %v", err)
        return "", nil, err
    }

    // Return the URL
    return signedUrl, signedHeaders, nil
}

type S3PresignClient struct {
    PresignClient *s3.PresignClient
    Bucket        string
}

func (client *S3PresignClient) GetPresignedUrl(key string) (string, error) {
    // Create the request
    signedRequest, err := client.PresignClient.PresignGetObject(context.TODO(), &s3.GetObjectInput{
        Bucket: aws.String(client.Bucket),
        Key:    aws.String(key),
    })

    if err != nil {
        log.Printf("Failed to create presigned URL request: %v", err)
        return "", err
    }

    if err != nil {
        log.Printf("Failed to sign URL: %v", err)
        return "", err
    }

    // Return the URL
    return signedRequest.URL, nil
}

// Create a new PresignClient
func NewPresignClient(config aws.Config, bucket string) *S3PresignClient {
    // Here we create a presigning client that has the signingTime set to the
    // beginning of the time bucket. This is so that we can reuse the same
    // presigned URL for the same time bucket. This is useful to being able to
    // generate the same url for the same time bucket and allow some caching of
    // the presigned URL.

    s3Client := s3.NewFromConfig(config)
    presignClient := s3.NewPresignClient(s3Client, func(o *s3.PresignOptions) {
        o.Presigner = &HTTPPresignerV4{
            signer: *v4.NewSigner(),
        }
    })

    return &S3PresignClient{
        PresignClient: presignClient,
        Bucket:        bucket,
    }
}

Answer 2

In case anyone is struggling using golang for presigning urls with cache possibilities, you can create a custom sign handler and swap the named handler with your own to change the sign time and make the urls the same for a time bucket:

import (
    "time"

    "github.com/aws/aws-sdk-go/aws/request"
    v4 "github.com/aws/aws-sdk-go/aws/signer/v4"
)

// Will create same url if in the same 15 minutes time bucket
const presignPeriod = 15 * time.Minute

// TimeInterface implements an interface that
// has the a time variable (Now) and a function
// to retrieve the time variable
type TimeInterface struct {
    Now time.Time
}

func (t *TimeInterface) NowFunc() time.Time {
    return t.Now
}

// getSignTime function returns the signing time
// (initial time) for the time bucket
func getSignTime() time.Time {
    now := time.Now().UTC()
    signTime := now.Round(presignPeriod)
    if signTime.After(now) {
        signTime.Add(-presignPeriod)
    }

    return signTime
}

// CustomSignSDKRequest Implements a custom aws signing
// handler that sets signing time on buckets of 
// <presignPeriod> minutes.
// It is used so browsers can cache the result of the
// url for get requests, instead of downloading the resource everytime.
func CustomSignSDKRequest(req *request.Request) {
    t := TimeInterface{
        Now: getSignTime(),
    }
    v4.SignSDKRequestWithCurrentTime(req, t.NowFunc)
}

Answer 3

Here's a solution in python I came up with after following this post. It uses the freezegun library to manipulate the time to make the signature the same over a given period.

import time
import datetime

import boto3
from freezegun import freezetime


S3_CLIENT = boto3.client("s3")

SEVEN_DAYS_IN_SECONDS = 604800
MAX_EXPIRES_SECONDS = SEVEN_DAYS_IN_SECONDS



def get_presigned_get_url(bucket: str, key: str, expires_in_seconds: int = MAX_EXPIRES_SECONDS) -> str:
        current_timestamp = int(time.time())
        truncated_timestamp = current_timestamp - (current_timestamp % expires_in_seconds)
        with freeze_time(datetime.datetime.fromtimestamp(truncated_timestamp)):
            presigned_url = S3_CLIENT.generate_presigned_url(
                ClientMethod="get_object",
                Params={
                    "Bucket": bucket,
                    "Key": key,
                    "ResponseCacheControl": f"private, max-age={expires_in_seconds}, immutable",
                },
                ExpiresIn=expires_in_seconds,
                HttpMethod="GET",
            )
        return presigned_url

Answer 4

Similar to the concept of @Aragorn but this is more complete code. This is Java again though. Also since my app is multi-region I had to put in the region properties.

import lombok.SneakyThrows;
import lombok.extern.slf4j.Slf4j;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.beans.factory.annotation.Value;
import org.springframework.stereotype.Component;
import software.amazon.awssdk.core.client.config.ClientOverrideConfiguration;
import software.amazon.awssdk.core.signer.Signer;
import software.amazon.awssdk.services.s3.S3AsyncClient;
import software.amazon.awssdk.services.s3.S3Client;
import software.amazon.awssdk.services.s3.model.GetObjectRequest;
import software.amazon.awssdk.services.s3.model.PutObjectRequest;
import software.amazon.awssdk.services.s3.presigner.S3Presigner;
import software.amazon.awssdk.services.s3.presigner.model.GetObjectPresignRequest;

import javax.annotation.PostConstruct;
import javax.validation.constraints.NotNull;
import java.net.URI;
import java.net.URISyntaxException;
import java.nio.file.Path;
import java.time.Duration;
import java.util.UUID;
import java.util.concurrent.CompletableFuture;

@Component
@Slf4j
public class S3Operations {

    @Autowired
    private Signer awsSigner;

    private final Map<Region, S3Presigner> presignerMap = new ConcurrentHashMap<>();

    private S3Presigner buildPresignerForRegion(
      AwsCredentialsProvider credentialsProvider,
      Region region) {

        return S3Presigner.builder()
            .credentialsProvider(credentialsProvider)
            .region(region)
            .build();

    }


    /**
     * Convert an S3 URI to a normal HTTPS URI that expires.
     *
     * @param s3Uri S3 URI (e.g. s3://bucketname/ArchieTest/フェニックス.jpg)
     * @return https URI
     */
    @SneakyThrows
    public URI getExpiringUri(final URI s3Uri) {

        final GetObjectRequest getObjectRequest =
            GetObjectRequest.builder()
                .bucket(s3Uri.getHost())
                .key(s3Uri.getPath().substring(1))
                .overrideConfiguration(builder -> builder.signer(awsSigner))
                .build();

        final Region bucketRegion = bucketRegionMap.computeIfAbsent(s3Uri.getHost(),
            bucketName -> {
                final GetBucketLocationRequest getBucketLocationRequest = GetBucketLocationRequest.builder()
                    .bucket(bucketName)
                    .build();

                return Region.of(s3Client.getBucketLocation(getBucketLocationRequest).locationConstraint().toString());
            });

        final GetObjectPresignRequest getObjectPresignRequest = GetObjectPresignRequest.builder()
            .signatureDuration(Duration.ofSeconds(0)) // required, but ignored
            .getObjectRequest(getObjectRequest)
            .build();

        return presignerMap.computeIfAbsent(bucketRegion, this::buildPresignerForRegion).presignGetObject(getObjectPresignRequest).url().toURI();

    }

For CustomAwsSigner which is injected above. The key difference being I throw an unsupported operation exception.

import org.jetbrains.annotations.TestOnly;
import org.springframework.beans.factory.annotation.Value;
import org.springframework.stereotype.Component;
import software.amazon.awssdk.auth.signer.AwsS3V4Signer;
import software.amazon.awssdk.auth.signer.AwsSignerExecutionAttribute;
import software.amazon.awssdk.auth.signer.params.Aws4PresignerParams;
import software.amazon.awssdk.core.interceptor.ExecutionAttributes;
import software.amazon.awssdk.core.signer.Presigner;
import software.amazon.awssdk.core.signer.Signer;
import software.amazon.awssdk.http.SdkHttpFullRequest;

import java.time.Clock;
import java.time.Instant;
import java.time.ZoneId;
import java.time.ZonedDateTime;
import java.time.temporal.ChronoField;
import java.time.temporal.ChronoUnit;

/**
 * This is a custom signer where the expiration is preset to a 5 minute block within an hour.
 * This must only be used for presigning.
 */
@Component
public class CustomAwsSigner implements Signer, Presigner {
    private final AwsS3V4Signer theSigner = AwsS3V4Signer.create();

    /**
     * This is the clip time for the expiration.  This should be divisible into 60.
     */
    @Value("${aws.s3.clipTimeInMinutes:5}")
    private long clipTimeInMinutes;

    @Value("${aws.s3.expirationInSeconds:3600}")
    private long expirationInSeconds;

    /**
     * Computes the base time as the processing time to the floor of nearest clip block.
     *
     * @param processingDateTime processing date time
     * @return base time
     */
    @TestOnly
    public ZonedDateTime computeBaseTime(final ZonedDateTime processingDateTime) {

        return processingDateTime
            .truncatedTo(ChronoUnit.MINUTES)
            .with(temporal -> temporal.with(ChronoField.MINUTE_OF_HOUR, temporal.get(ChronoField.MINUTE_OF_HOUR) / clipTimeInMinutes * clipTimeInMinutes));

    }

    @Override
    public SdkHttpFullRequest presign(final SdkHttpFullRequest request, final ExecutionAttributes executionAttributes) {

        final Instant baselineInstant = computeBaseTime(ZonedDateTime.now()).toInstant();

        final Aws4PresignerParams signingParams = Aws4PresignerParams.builder()
            .awsCredentials(executionAttributes.getAttribute(AwsSignerExecutionAttribute.AWS_CREDENTIALS))
            .signingName(executionAttributes.getAttribute(AwsSignerExecutionAttribute.SERVICE_SIGNING_NAME))
            .signingRegion(executionAttributes.getAttribute(AwsSignerExecutionAttribute.SIGNING_REGION))
            .signingClockOverride(Clock.fixed(baselineInstant, ZoneId.of("UTC")))
            .expirationTime(baselineInstant.plus(expirationInSeconds, ChronoUnit.SECONDS))
            .build();
        return theSigner.presign(request, signingParams);

    }

    @Override
    public SdkHttpFullRequest sign(final SdkHttpFullRequest request, final ExecutionAttributes executionAttributes) {

        throw new UnsupportedOperationException("this class is only used for presigning");

    }
}

Answer 5

Maybe a late reply, but I'll add my approach for the benefit of people reading this in future.

To force the browser cache to kick in, it's important to generate same exact url every time until you specifically want the browser to reload content from the server. Unfortunately the presigner provided in the sdk, relies on current timestamp leading to a new url every time.

This example is in Java but it can easily be extended to other languages

The GetObjectRequest builder(used to create the presigned url) allows overriding configuration. We can supply a custom signer to modify its behaviour

AwsRequestOverrideConfiguration.builder()
    .signer(new CustomAwsS3V4Signer())
    .credentialsProvider(<You may need to provide a custom credential provider 
here>)))
.build())

GetObjectRequest getObjectRequest =
    GetObjectRequest.builder()
            .bucket(getUserBucket())
            .key(key)
            .responseCacheControl("max-age="+(TimeUnit.DAYS.toSeconds(7)+ defaultIfNull(version,0L)))
            .overrideConfiguration(overrideConfig)
            .build();

public class CustomAwsS3V4Signer implements Presigner, Signer
{
    private final AwsS3V4Signer awsSigner;

    public CustomAwsS3V4Signer()
    {
        awsSigner = AwsS3V4Signer.create();
    }

@Override
public SdkHttpFullRequest presign(SdkHttpFullRequest request, ExecutionAttributes executionAttributes)
{
    Instant baselineInstant = Instant.now().truncatedTo(ChronoUnit.DAYS);

    executionAttributes.putAttribute(AwsSignerExecutionAttribute.PRESIGNER_EXPIRATION,
            baselineInstant.plus(3, ChronoUnit.DAYS));

Here we override the signing clock to simulate a fixed time which ultimately results in consistent expiry and signature in the url until a certain date in future:

    Aws4PresignerParams.Builder builder = Aws4PresignerParams.builder()
            .signingClockOverride(Clock.fixed(baselineInstant, ZoneId.of("UTC")));

    Aws4PresignerParams signingParams =
            extractPresignerParams(builder, executionAttributes).build();

    return awsSigner.presign(request, signingParams);
    }
}

More details are available here:

https://murf.ai/resources/creating-cache-friendly-presigned-s3-urls-using-v4signer-q1bbqgk

Answer 6

Rather than using the presigned URL mechanism perhaps you could add an authenticated endpoint to your application and within said endpoint retrieve the image? Using this URL in your img tags and such. This endpoint could cache the image and provide the appropriate response headers for the browser to cache the image too.