Categories
Development

Do until loop in Spring reactor

Reactor 3.1.0 added a brand new and shiny operator expand, which solved by long term dilemma about how to implement downloading of paginated REST endpoints without recursion.

Take Github for example, if you request list of contributors, it will return first 100 contributors for a repo. In header in links, it will provide links to next 100 contributors (it also provides link to last page which can be used to calculate links for all pages, but lets leave that for now). So in order to download next page, you need the result of previous page. That could be easily implemented with recursion:


private <T> Flux<T> performPageableRequestRecursive(URI uri, Class<T> clazz, int pagesLimit) {
        return getRequestWrapped(uri, clazz)
                .flatMapMany(responseWrapper -> responseWrapper.getData()
                        .concatWith(pagesLimit == 0 ? Flux.empty() : responseWrapper.getNextLink()
                                .concatMap(link -> performPageableRequestRecursive(
                                        link.getUri(), clazz, pagesLimit - 1))));
}

The problem with recursion in reactor is that it leads quickly to stack overflow. Try downloading 1000 pages … Which is not that much. Luckily now we have elegant solution: expand operator.


private <T> Flux<T> performPageableRequestExpand(URI uri, Class<T> clazz) {
return getRequestWrapped(uri, clazz)
.expand(responseWrapper -> responseWrapper
.getNextLink()
.flatMap(link -> getRequestWrapped(link.getUri(), clazz)))
.flatMap(GithubResponseWrapper::getData);
}

Javadoc says: Recursively expand elements into a graph and emit all the resulting element using a breadth-first traversal strategy. Well that does not say much does it. Luckily the seccond line is better: That is: emit the values from this Flux first, then expand each at a first level of recursion and emit all of the resulting values, then expand all of these at a second level and so on. Yaay, just what we need. We emit one value at each “level” the result page and use the next link from that page to “expand” next level.

I have updated the code samples in:
https://github.com/MavoCz/spring-webclient-reactor-test/tree/master