Background service that spawns other services/tasks

Question

i'm trying to wrap my head around Service/Task in javafx and how to handle nesting them together. My app is a simple rss downloader. It downloads multiple feeds, and it also downloads the html inside each feed item's <link>. I want the entire download process to be async (to keep the GUI from freezing), as well as each rss feed download and each html download. I want the process to look something like this.

Application thread.
|
|
|--------Download process start(Service)
|       |
|       |
|       |----RSS download start(Service)
|       |    |(30+ Tasks that each download an individual feed.
|       |----RSS download end
|       |
|       |
|       |----HTML download start(Service)
|       |    |(100+ Tasks that each download an individual HTML page.
|       |----HTML download end
|       |
|       |
|--------Download process end.
|
|

My Code. downloadStart() kicks off the Downloader Service.

    @Override 
    public void downloadStart(List<Channel> channels) {
        Downloader downloader = new Downloader(channels);
        downloader.setOnSucceeded(new EventHandler<WorkerStateEvent>() {
            @Override
            public void handle(WorkerStateEvent t) {
                List<Article> result = (List<Article>)t.getSource().getValue();
                display.printToOutput("Completed download process : " + result.size());
            }
        });
        downloader.start();
    }

The Downloader class.

public class Downloader extends Service<List<Article>> {

List<Channel> channels;

public Downloader(List<Channel> channels){
    this.channels = channels;
}

public void downloadRSS() {
    for(Channel channel : channels){
        RSSDownloadService<List<Article>> downloader = new RSSDownloadService<List<Article>>(channel);
        downloader.setOnSucceeded(new EventHandler<WorkerStateEvent>() {
            @Override
            public void handle(WorkerStateEvent t) {
                List<Article> result = (List<Article>)t.getSource().getValue();
                downloadHTML(result);
            }
        });
        downloader.start();
    }
}

private void downloadHTML(List<Article> articles){
    HTMLDownloadService<List<Article>> downloader = new HTMLDownloadService<List<Article>>(articles);
    downloader.setOnSucceeded(new EventHandler<WorkerStateEvent>() {
        @Override
        public void handle(WorkerStateEvent t) {
            List<Article> result = (List<Article>)t.getSource().getValue();
            //how do i tell the Downloader service to return this result?
        }
    });
    downloader.start();
}

@Override
protected Task<List<Article>> createTask() {
    return new Task<List<Article>>() {
        protected List<Article> call() {
            downloadRSS();
            //i can't return anything until downloadHTML() finishes!!
        }
    };
}

}

The Problem: After starting the Downloader service, it's createTask() method calls downloadRSS() and expects a return value. However, downloadRSS() method doesn't return anything, it starts RSSDownloadService. When RSSDownloadService succeeds, it calls downloadHTML(), which starts HTMLDownloadService. Finally, when that succeeds, i want to end the entire Downloader service and return the List of articles. I'm not sure how to proceed.

The RSSDownloadService and HTMLDownloadService work just fine. They were simple for me to implement because they call one method with a return value. However the `DownloaderService' somehow needs to wait for the 2 services to complete, and return the 2nd services succeed value.

Answer 1

If I understand correctly, you want downloadRSS() to return a List<Article> which contains all the Articles from all the lists returned by the HTMLDownloadServices it spawns.

I think the following does what you want:

public List<Article> downloadRSS() {

    List<Article> mainList = Collections.synchronizedList(new ArrayList<>());
    CountDownLatch latch = new CountDownLatch(channels.size());

    for(Channel channel : channels){
        RSSDownloadService<List<Article>> downloader = new RSSDownloadService<List<Article>>(channel);
        downloader.setOnSucceeded(new EventHandler<WorkerStateEvent>() {
            @Override
            public void handle(WorkerStateEvent t) {
                List<Article> result = (List<Article>)t.getSource().getValue();
                downloadHTML(result, mainList, latch);
            }
        });
        downloader.setOnFailed(t -> {
            // handle error if neccessary...
            latch.countDown();
        });
        downloader.start();
    }
    latch.await();
    // return a regular list, don't need the overhead of synchronization any more:
    return new ArrayList<>(mainList);
}

private void downloadHTML(List<Article> articles, List<Article> mainList, CountDownLatch latch){
    HTMLDownloadService<List<Article>> downloader = new HTMLDownloadService<List<Article>>(articles);
    downloader.setOnSucceeded(new EventHandler<WorkerStateEvent>() {
        @Override
        public void handle(WorkerStateEvent t) {
            List<Article> result = (List<Article>)t.getSource().getValue();
            mainList.addAll(result);
            latch.countDown();
        }
    });

    downloader.setOnFailed(t -> {
        // handle error if needed...
        latch.countDown();
    });
    downloader.start();
}

@Override
protected Task<List<Article>> createTask() {
    return new Task<List<Article>>() {
        protected List<Article> call() {
            return downloadRSS();
        }
    };
}

Here's perhaps a better approach, using Java 8 streams and built-in parallelization to manage most of the threading:

public class Downloader extends Task<List<Article>> {

    private final List<Channel> channels ;

    public Downloader(List<Channel> channels) {
        this.channels = channels ;
    }

    @Override
    public List<Article> call() throws Exception {
        return channels.parallelStream()
            .flatMap(channel -> getRssList(channel).parallelStream())
            .flatMap(rss -> getHtmlList(rss).stream())
            .collect(Collectors.toList());
    }

    private List<Article> getRssList(Channel channel) {
        // this runs in its own thread, return List<Article> for given channel
    }

    private List<Article> getHtmlList(Article rss) {
        // this runs in its own thread, return List<Article> for given rss
    }
}

And then all you need in the ui is:

List<Channel> channels = ... ;
Downloader downloader = new Downloader(channels);
downloader.setOnSucceeded(e -> {
    List<Article> articles = downloader.getValue();
    // update UI with articles...
});
Thread t = new Thread(downloader);
t.setDaemon(true) ; // will not prevent application exit...
t.start();