Wait for all observables to finish (in sequence + parent children relationship)

Question

I have a problem with Angular and Observables, and I reproduced it in this Stackblizt: https://stackblitz.com/edit/angular-ivy-dl1y3y

To put some context:

• I need to call a 1st webservice.
• With the data from the 1st call, I need to call a 2nd web service.
• With the data from the 2nd call, I need to call a 3rd groups of webservices ("parent with children" relationship)
• All this information will be displayed in a table (using Angular Datatables), and I don't want to display the table with partial data (otherwise the remaining data are popping up as long as all the remaining observables complete. Also I could face some other issues such as user clicking on a button while part of the data is not available yet...). So: I want ALL the observables to complete, THEN I pass the full object to the datatable and everything is displayed at once !
• I want to do it with Observables, not Promises

Ok, let's me show you step by step.

Step 1

The first step is to call a first URL (assets/step-1-getAccountReference.json) to retrieve the account reference ID :

{
  "accountIdRef": "/assets/step-2.getAccount.json"
}

Step 2

With this accountIdRef, I can call another URL ("/assets/step-2.getAccount.json") to retrieve the account information:

{
   "accountId": "123",
   "details": [
      {
         "nameRef": "/assets/step-3-pet-1-name.json",
         "genderRef": "/assets/step-3-pet-1-gender.json"
      },
      {
         "nameRef": "/assets/step-3-pet-2-name.json",
         "genderRef": "/assets/step-3-pet-2-gender.json"
      }
   ]
}

Step 3

The last step is to retrieve all the details for each pets, by calling some other urls (nameRef and genderRef).

If you open the console, you should see that if I subscribe and log the account directly, this is displayed (Observables from step 1 and step 2 have completed) :

{
    "accountId": "123",
    "details": [
        {
            "nameRef": "/assets/step-3-pet-1-name.json",
            "genderRef": "/assets/step-3-pet-1-gender.json"
        },
        {
            "nameRef": "/assets/step-3-pet-2-name.json",
            "genderRef": "/assets/step-3-pet-2-gender.json"
        }
    ]
}

If I log the account again 3 seconds later, this is displayed (all the observables from step 3 have completed):

{
    "accountId": "123",
    "details": [
        {
            "nameRef": "/assets/step-3-pet-1-name.json",
            "genderRef": "/assets/step-3-pet-1-gender.json",
            "name": "Santa's Little Helper",
            "gender": "Male"
        },
        {
            "nameRef": "/assets/step-3-pet-2-name.json",
            "genderRef": "/assets/step-3-pet-2-gender.json",
            "name": "Snowball II",
            "gender": "Female"
        }
    ]
}

I would like to wait for all the observables to complete (Step 3 included), but of course dynamically, not by using a fixed timeout.

Here is what I have for now:

export class HttpService {
  constructor(private http: HttpClient) {}

  getAccount(): Observable<Account> {
    return this.http.get("assets/step-1-getAccountReference.json").pipe( // Step 1
      mergeMap((accountReference: AccountReference) => {
        return this.http.get("" + accountReference.accountIdRef);        // Step 2
      }),
      delay(500),
      map((account: Account) => {
        account.details.forEach((details: AccountDetails) => {           // Step 3
          let name$ = this.http.get("" + details.nameRef);
          let gender$ = this.http.get("" + details.genderRef);
          forkJoin([name$, gender$]).subscribe(results => {
            details.name = results[0]["name"];
            details.gender = results[1]["gender"];
          });
        });

        return account;
      })
    );
  }
}

So, how can I adapt this code, so that Step 3 is synchronous ? Which operator should I use to replace this forEach ?

Thank you for your help !

Answer 1

This seemed to work:

  getAccount() {
    return this.http.get("assets/step-1-getAccountReference.json").pipe(
      mergeMap((accountReference: AccountReference) =>
        this.http.get("" + accountReference.accountIdRef).pipe(
          mergeMap((account: Account) => 
            forkJoin(account.details.map(detail => this.getDetails(detail))).pipe(
              map(_ => account)
            )
          )
        )
      )
    );
  }

  private getDetails(detail: AccountDetails): Observable<AccountDetails> {
    let name$ = this.http.get("" + detail.nameRef);
    let gender$ = this.http.get("" + detail.genderRef);
    return forkJoin([name$, gender$]).pipe(
      map(([nameObj, genderObj]: [{name: string}, {gender: string}]) => {
        detail.name = nameObj.name;
        detail.gender = genderObj.gender;
        return detail;
      })
    );
  }

It seemed a bit easier to grok as two separate methods.

The getDetails method uses the info from the detail to set up the two get operations. It then uses a forkJoin to execute both of them. NOTICE, there is no need then for a subscribe here! The forkJoin then uses a map to map the name and gender and return the resulting detail as an Observable.

The getAccount method uses a mergeMap to get the first set of child data (account reference) and another mergeMap to work with the account details. The forkJoin uses a map (instead of a foreach) to process each set of details. For each detail, it calls the getDetails method to set the appropriate values into the details object.

It then maps the result to the account to return the resulting account info.

The resulting StackBlitz is here: https://stackblitz.com/edit/angular-ivy-etwwas?file=src/app/http.service.ts

Answer 2

What you could do is to map through your details and build an array of observables that will populate the missing attributes.
You then pass that array to a forkJoin that will pull the missing data into your details. Finally, you update your account details and return the account.

export class HttpService {
  constructor(private http: HttpClient) {}

  getAccount(): Observable<Account> {
    return this.http.get("assets/step-1-getAccountReference.json").pipe( // Step 1
      mergeMap((accountReference: AccountReference) => {
        return this.http.get("" + accountReference.accountIdRef);        // Step 2
      }),
      delay(500), // why this delay ?
      mergeMap((account: Account) => {
        const populatedDetailsObservableArray = account.details.map((details: AccountDetails) => {
            return forkJoin([name$, gender$]).pipe(
                map(results => {
                    details.name = results[0]["name"];
                    details.gender = results[1]["gender"];
                    return details;
                })
            );
        });
        return forkJoin(populatedDetailsObservableArray).pipe(
            map((newDetails: AccountDetails[]) => {
                account.details = newDetails;
                return account;
            })
        );
      })
    );
  }
}

Answer 3

You can use RxJs library for that purpose. Here is a link of one util function that will work for your case. It's called concatMap and its role is to merge requests and saving their order.

Below is an example from the official documentation:

// RxJS v6+
import { of } from 'rxjs';
import { concatMap, delay, mergeMap } from 'rxjs/operators';

//emit delay value
const source = of(2000, 1000);
// map value from source into inner observable, when complete emit result and move to next
const example = source.pipe(
  concatMap(val => of(`Delayed by: ${val}ms`).pipe(delay(val)))
);
//output: With concatMap: Delayed by: 2000ms, With concatMap: Delayed by: 1000ms
const subscribe = example.subscribe(val =>
  console.log(`With concatMap: ${val}`)
);

// showing the difference between concatMap and mergeMap
const mergeMapExample = source
  .pipe(
    // just so we can log this after the first example has run
    delay(5000),
    mergeMap(val => of(`Delayed by: ${val}ms`).pipe(delay(val)))
  )
  .subscribe(val => console.log(`With mergeMap: ${val}`));