Reputation: 13
I am using the Javascript SDK v3 to submit several API commands in parallel. The general flow is
const client = new SomeAWSClient(config) const response: Promise<SomeAWSClientCommandOutput> = client.send(new SomeAWSClientCommand(input))
I am looking to batch together a group of compatible clients/commands/outputs and wrap them in Promise.allSettled(). I found I can't just invoke allSettled() with a arbitrarily long Promises[]. Once it gets above length=10 or so they will fail. So I would like to write a generic function that can take a valid client and (potentially) lengthy command[] and batch the commands into groups no larger than 10 until they are all fulfilled and then return the results. Here is an example of the logic for a single client/command combination.
export async function s3GetObjects(client: S3Client, commands: GetObjectCommand[]) {
const commandsClone = [...commands];
const retArr: PromiseSettledResult<GetObjectCommandOutput>[] = [];
const batchSize = 10;
let count = 0;
const promises: Promise<GetObjectCommandOutput>[] = [];
let command = commandsClone.shift();
while (command) {
count += 1;
if (count <= batchSize) {
promises.push(client.send(command));
} else {
const result = await Promise.allSettled(promises);
retArr.push(...result);
promises.length = 0;
promises.push(client.send(command));
count = 1;
}
command = commandsClone.shift();
}
if (promises.length > 0) {
const result = await Promise.allSettled(promises);
retArr.push(...result);
}
return retArr;
}
To make it generic, I've tried coming up with three type definitions for valid client/command/output combinations and then a generic function to perform allSettled() call. Here it is without the batchSize=10 logic yet.
type ValidClient = S3Client | CloudFormationClient | LambdaClient;
type ValidCommand<T extends ValidClient> = T extends S3Client
? GetObjectCommand | PutObjectCommand
: T extends CloudFormationClient
? DescribeStacksCommand
: T extends LambdaClient
? GetFunctionCommand
: never;
type ValidCommandOutput<T extends ValidCommand<ValidClient>> = T extends GetObjectCommand
? GetObjectCommandOutput
: T extends PutObjectCommand
? PutObjectCommandOutput
: T extends DescribeStacksCommand
? DescribeStacksCommandOutput
: T extends GetFunctionCommand
? GetFunctionCommandOutput : never;
export async function promiseAllSettled<
TClient extends ValidClient,
TCommand extends ValidCommand<TClient>,
TOutput extends ValidCommandOutput<ValidCommand<TClient>>
>(client: TClient, commands: TCommand[]): Promise<PromiseSettledResult<Awaited<TOutput>>[]> {
const promises: Promise<TOutput>[] = [];
for (const command of commands) {
promises.push(client.send(command));
}
return Promise.allSettled(promises);
}
It seems to work well, preventing invalid input combinations, but I get an error on the send() call that I can't figure out.
This expression is not callable.
Each member of the union type '{ \<InputType extends ServiceInputTypes, OutputType extends
ServiceOutputTypes\>(command: Command\<ServiceInputTypes, InputType, ServiceOutputTypes, OutputType,
SmithyResolvedConfiguration\<...\>\>, options?: HttpHandlerOptions | undefined): Promise\<...\>; \<InputType
extends ServiceInputTypes, OutputType extends ServiceOut...' has signatures, but none of those signatures
are compatible with each other.ts(2349)
Any ideas of good ways to resolve this?
Upvotes: 1
Views: 90