CODEWITHSUNDEEP
javacloudgoogle-cloud-storagebucket
user3352488
user3352488

Reputation: 281

Google Cloud Endpoint Bucket Downloader

I am very new to the GC platform and am trying to create an API in Java with two methods: one which returns a list of all of the files in a specific bucket and another which retrieves a specified file from that bucket. The goal is to be able to iterate the file list in order to download every file from the bucket. Essentially, I am wanting to mirror the contents of the bucket on an Android device, so the API will be called from a generated client library in an Android app. My getFileList() method returns a ListResult object. How do I extract a list of files from this?

@ApiMethod(name = "getFileList", path = "getFileList", httpMethod = ApiMethod.HttpMethod.GET)
public ListResult getFileList(@Named("bucketName") String bucketName) {
    GcsService gcsService = GcsServiceFactory.createGcsService(RetryParams.getDefaultInstance());
    ListResult result = null;
    try {
        result = gcsService.list(bucketName, ListOptions.DEFAULT);
        return result;
    } catch (IOException e) {
        return null; // Handle this properly.
    }
}

Also, I am struggling to determine what the return type of my getFile() API method should be. I can’t use a byte array since return types cannot be simple types as I understand it. This is where I am with it:

@ApiMethod(name = "getFile", path = "getFile", httpMethod = ApiMethod.HttpMethod.GET)
public byte[] getFile(@Named("bucketName") String bucketName, ListItem file) {
    GcsService gcsService = GcsServiceFactory.createGcsService();
    GcsFilename gcsFilename = new GcsFilename(bucketName, file.getName());
    ByteBuffer byteBuffer;
    try {
        int fileSize = (int) gcsService.getMetadata(gcsFilename).getLength();
        byteBuffer = ByteBuffer.allocate(fileSize);
        GcsInputChannel gcsInputChannel = gcsService.openReadChannel(gcsFilename, 0);
        gcsInputChannel.read(byteBuffer);
        return byteBuffer.array();
    } catch (IOException e) {
        return null; // Handle this properly.
    }
}

I am lost in the Google documentation for this stuff and am concerned that I am coming at it from completely the wrong direction since all I am trying to do is securely download a bunch of files!

Upvotes: 0

Views: 244

Answers (1)

Abacus
Abacus

Reputation: 19421

I can't give you a complete solution because, this is code I wrote for my company, but I can show you some basics. I use the google-cloud-java API.

First you need to create an API key and download this in JSON format. More details can be found here.

I have - amongst other - these two fields in my class:

protected final Object storageInitLock = new Object();
protected Storage storage;

First you will need a method to initialize a com.google.cloud.storage.Storage object, something like (set your project-id and path to json api key):

protected final Storage getStorage() {
    synchronized (storageInitLock) {
        if (null == storage) {
            try {
                storage = StorageOptions.newBuilder()
                        .setProjectId(PROJECTID)
                        .setCredentials(ServiceAccountCredentials.fromStream(new FileInputStream(pathToJsonKey)))
                        .build()
                        .getService();
            } catch (IOException e) {
                throw new MyCustomException("Error reading auth file " + pathToJsonKey, e);
            } catch (StorageException e) {
                throw new MyCustomException("Error initializing storage", e);
            }
        }

        return storage;
    }
}

to get all entries you could use something like:

protected final Iterator<Blob> getAllEntries() {
    try {
        return getStorage().list(bucketName).iterateAll();
    } catch (StorageException e) {
        throw new MyCustomException("error retrieving entries", e);
    }
}

list files in a directory:

public final Optional<Page<Blob>> listFilesInDirectory(@NotNull String directory) {
    try {
        return Optional.ofNullable(getStorage().list(getBucketName(), Storage.BlobListOption.currentDirectory(),
                Storage.BlobListOption.prefix(directory)));
    } catch (Exception e) {
        return Optional.empty();
    }
}

get info about a file:

public final Optional<Blob> getFileInfo(@NotNull String bucketFilename) {
    try {
        return Optional.ofNullable(getStorage().get(BlobId.of(getBucketName(), bucketFilename)));
    } catch (Exception e) {
        return Optional.empty();
    }
}

adding a file:

public final void addFile(@NotNull String localFilename, @NotNull String bucketFilename,
                               @Nullable ContentType contentType) {
    final BlobInfo.Builder builder = BlobInfo.newBuilder(BlobId.of(bucketName, bucketFilename));
    if (null != contentType) {
        builder.setContentType(contentType.getsValue());
    }
    final BlobInfo blobInfo = builder.build();

    try (final RandomAccessFile raf = new RandomAccessFile(localFilename, "r");
         final FileChannel channel = raf.getChannel();
         final WriteChannel writer = getStorage().writer(blobInfo)) {

        writer.write(channel.map(FileChannel.MapMode.READ_ONLY, 0, channel.size()));
    } catch (Exception e) {
        throw new MyCustomException(MessageFormat.format("Error storing {0} to {1}", localFilename,
                bucketFilename), e);
    }
}

I hope these code snippets and the referenced documentation will get you going, actulally it's not too hard.

Upvotes: 1

Related Questions