How to chain requests correctly with RxJS

Question

I'm just trying to learn Angular and RxJS (Observables) and I'm having some problems chaining requests the right way.

The scenario is the following:

1. I run request A and get a JSON object that contains an array of objects.
2. For each of these objects I need to execute request B which in turn returns a JSON object with an array of objects.
3. For each object in the response from request B, I need to execute request C which returns a simple JSON object.
4. Now I have to store all the objects I received from request C and extend them with information (information comes from the answer to request D, E and F).
5. Only when all requests are finished I want to show the UI of my application, before that only a loading spinner should be visible.

My current solution gives me all the necessary information but I can't find a way to wait until all the requests are finished. It looks something like this:

private fetchData(): void {
  this.backendServie.getObjectA().pipe(
    flatMap((objects: IObjectA[]) => {
      // return objects the get them one by one
      return objects
    }),
    flatMap((object: IObjectA) => {
      return this.backendService.getObjectB(object.id)
    }),
    flatMap((objects: IObjectB[]) => {
      // return objects the get them one by one
      return objects
    }),
    flatMap((object: IObjectB) => {
      return this.backendService.getObjectC(object.id)
    }),
    flatMap((object: IObjectC) => {
      // convert object to make additional fields available
      const extendedObject: IObjectCExtended = object as IObjectCExtended;
      this.backendService.getAdditionalInformationD(object.id).subscribe((info: string) >= {
        extendedObject.additionalInfoD = info;
      });
      this.backendService.getAdditionalInformationE(object.id).subscribe((info: string) >= {
        extendedObject.additionalInfoE = info;
      });
      this.backendService.getAdditionalInformationF(object.id).subscribe((info: string) >= {
        extendedObject.additionalInfoF = info;
      })
      return extendedObject;
    })
  ).subscribe((extendedObject: IObejectCExtended) => {
    this.objects.push(extendedObject);
  })
}

Unfortunately, this way of chaining the requests feels wrong and, as I said, I can't find a way to check if all requests are finished and the UI fills up only little by little.

I hope my question is formulated understandably and thanks in advance for your help.

Best Regards

Answer 1

This design is pretty inefficient, as already mentioned. So it would be preferable to change your backend - as already mentioned in a comment.

If you can't change your backend, you could group your requests using a combination of switchMap and forkJoin.

Streamlining all entries at once

You could streamline all of the entries at once for each API you want to call. So basically:

1. Get all objectAs
2. Get all objectBs for each objectA
3. Get all objectCs for each objectB
4. Get all additions for each objectC
5. Extend all objectCs with each addition

private fetchData(): void {
  // get your initial object
  this.backendServie.getObjectA().pipe(

    // switch to forkJoin, which waits until all inner observables complete and returns them as an array
    switchMap((objects: IObjectA[]) => forkJoin(
       // map all your objects to an array of observables which will then be waited on by forkJoin
       objects.map(obj => this.backendService.getObjectB(obj.id))
    )),

    // Again, switch to another forkJoin
    switchMap((objects: IObjectB[]) => forkJoin(
      // Again, map all your objects to observables that forkJoin will collect and wait on
       objects.map(obj => this.backendService.getObjectC(obj.id))
    )),

    // switch to another forkJoin that retrieves all your extensions for every object
    switchMap((objects: IObjectC[]) => forkJoin(
       // map each object to a forkJoin that retrieves the extensions for that object and map it to the extended object
       objects.map(obj => forkJoin([
         this.backendService.getAdditionalInformationD(obj.id),
         this.backendService.getAdditionalInformationE(obj.id),
         this.backendService.getAdditionalInformationF(obj.id)
       ]).pipe(
         map(([infoD, infoE, infoF]) => ({
            ...obj,
            additionalInfoD: infoD,
            additionalInfoE: infoE,
            additionalInfoF: infoF
         }))
       )

      /* Alternatively, you can use the dictionary syntax to shorten this
       objects.map(obj => forkJoin({
         additionalInfoD: this.backendService.getAdditionalInformationD(obj.id),
         additionalInfoE: this.backendService.getAdditionalInformationE(obj.id),
         additionalInfoF: this.backendService.getAdditionalInformationF(obj.id)
       }).pipe(map(additionalInfo => ({...obj, ...additionalInfo })))
      */
    ))

  // You're now getting a list of the extended objects
  ).subscribe((extendedObjects: IObejectCExtended[]) => {
    this.objects = extendedObjects;
  });
}

To make the code a bit clearer you could group the different parts into different functions. This could look something like this.

private fetchData(): void {
  this.fetchExtendedObjects()
    .subscribe(extendedObjects => this.objects = extendedObjects);
}

private fetchExtendedObjects():Observable<IObjectCExtended[]> {
  return this.backendServie.getObjectA().pipe(
    switchMap(objectsA => this.getAllObjectsB(objectsA)),
    switchMap(objectsB => this.getAllObjectsC(objectsB)),
    switchMap(objectsC => this.extendAllObjectsC(objectsC))
  )
}

private getAllObjectsB(objects: IObjectA[]):Observable<IObjectB[]> {
  return forkJoin(objects.map(obj => this.backendService.getObjectB(obj.id)));
}

private getAllObjectsC(objects: IObjectB[]):Observable<IObjectC[]> {
  return forkJoin(objects.map(obj => this.backendService.getObjectC(obj.id)));
}

private extendAllObjectsC(objects: IObjectC[]):Observable<IObjectCExtended[]> {
  return forkJoin(objects.map(obj => this.extendObjectC(obj)));
}

private extendObjectC(object: IObjectC):Observable<IObejectCExtended> {
  objects.map(obj => forkJoin({
    additionalInfoD: this.backendService.getAdditionalInformationD(obj.id),
    additionalInfoE: this.backendService.getAdditionalInformationE(obj.id),
    additionalInfoF: this.backendService.getAdditionalInformationF(obj.id)
  }).pipe(map(additionalInfo => ({...obj, ...additionalInfo })))
}

Streamlining each entry separately

As an optimization to the above, you could streamline each object seperately, which will give you a small performance boost. Overall, this should not be that much of an impact since you still have to wait on all requests to finish, but this might help out if some of your API's are slower for some of the objects.

This basically means:

1. Get all objectAs
   1. For each objectA get objectB in parallel
   2. For each objectB get objectC in parallel
   3. For each objectC get all extensions (for each objectC in parallel)
   4. For each objectC extend objectC with all extensions (for each objectC in parallel)
2. Combine all extended objectCs into a single array

private fetchData(): void {
  // get your initial object
  this.backendServie.getObjectA().pipe(

    switchMap((objects: IObjectA[]) => forkJoin(
      // Switch each objectA to an observable that retrieves objectB
      // In contrast to the first version, this is done for each objectA separately

      objects.map(obj => this.backendService.getObjectB(obj.id).pipe(

         // Switch each objectB to an observable that retrieves objectC
         switchMap((object: IObjectB) => this.backendService.getObjectC(obj.id)),

         // Switch each objectC to an observable that retrieves all of the
         // extensions an combines them with objectC
         switchMap((object: IObjectC) => 
           // Retrieves all extensions and provides them as a dictionary
           forkJoin({
             additionalInfoD: this.backendService.getAdditionalInformationD(obj.id),
             additionalInfoE: this.backendService.getAdditionalInformationE(obj.id),
             additionalInfoF: this.backendService.getAdditionalInformationF(obj.id)
           }).pipe(
             // combine the original objectC with all the extensions
             map(additionalInfo => ({...object, ...additionalInfo}))
           )
         )
      ))
    ))
  // You're now getting a list of the extended objects
  ).subscribe((extendedObjects: IObejectCExtended[]) => {
    this.objects = extendedObjects;
  });
}

Again, to make it more readable, you could separate the parts into functions:

private fetchData(): void {
  this.fetchExtendedObjects().subscribe(objects => this.objects = objects);
}

private fetchExtendedObjects():Observable<IObjectCExtended[]> {
  return this.backendServie.getObjectA().pipe(
    switchMap((objects: IObjectA[]) => this.getExtendedObjectsC(objects))
  );
}

private getExtendedObjectsC(objects: IObjectA[]):Observable<IObjectCExtended[]> {
  return forkJoin(objects.map(obj => this.getExtendedObjectC(obj)));
}

private getExtendedObjectC(objectA: IObjectA):Observable<IObjectCExtended> {
  return this.backendService.getObjectB(objectA.id).pipe(
    switchMap((object: IObjectB) => this.backendService.getObjectC(obj.id)),
    switchMap((object: IObjectC) => this.extendObjectC(object))
  );
}

private extendObjectC(object: IObjectC):Observable<IObejectCExtended> {
  objects.map(obj => forkJoin({
    additionalInfoD: this.backendService.getAdditionalInformationD(obj.id),
    additionalInfoE: this.backendService.getAdditionalInformationE(obj.id),
    additionalInfoF: this.backendService.getAdditionalInformationF(obj.id)
  }).pipe(map(additionalInfo => ({...obj, ...additionalInfo })))
}