Delegating File downloads sends incomplete chunks [SPR-15949]

[spring-projects-issues]

Alexander Wilhelmer opened SPR-15949 and commented

I noticed problems when u using this construct:

public Flux<DataBuffer> getFileContent(String url) {
     Mono<ClientResponse> exchange = webClientService.getWebClient().get().uri(url).accept(MediaType.APPLICATION_OCTET_STREAM).exchange();
     return exchange.flatMapMany(clientResponse -> {
        return clientResponse.bodyToFlux(DataBuffer.class);
     });
   }

You have to use a HTTP source with transcoding: chunked header without a content-length header. The problem of incomplete chunks only appears when you deploy the spring application on a windows system. I can't reproduce it on linux e.g. ubuntu. I'm using Windows Server 2008 R2 and Windows Server 2003, both with java 1.8 update 144. Requests against localhost works too, you have to stream over network!

Example: I deployed the app on windows. Just call http://217.79.182.40:8091/download/stream/102400 a few times. You will get a Network Error sometimes (1-3 errors on ten attempts). I'll keep that server online for max 7 days.

Source see at reference URL.

---

Affects: 5.0 RC3

Reference URL: https://github.com/awilhelmer/streambug

Attachments:

• errors.png (4.25 kB)
• screenshot-1.png (3.53 kB)

Issue Links:

• WebClient: DELETE request should not send transfer-encoding: chunked header [SPR-15947] #20500 WebClient: DELETE request should not send transfer-encoding: chunked header
• Multipart File upload: Sometimes receiving empty files! [SPR-15955] #20507 Multipart File upload: Sometimes receiving empty files!

Referenced from: commits ec2218c

[spring-projects-issues]

Brian Clozel commented

Hello Alexander Wilhelmer,

Thanks for all your efforts; thanks to your detailed description, we've managed to narrow things down to a (possibly fixed) bug in reactor-netty.

Could you try upgrading your application to Spring Boot 2.0.0.BUILD-SNAPSHOT and see if the issue is gone?

[spring-projects-issues]

Alexander Wilhelmer commented

I tried with 2.0.0.BUILD-SNAPSHOT.
The Bug still exists:

!screenshot-1.png!

I found a file, which nearly always produce the error. But it contained personal data, so i can't make it public .... It has 221.182 Bytes and on disc 221.184 Bytes. Maybe it helps....

[spring-projects-issues]

Brian Clozel commented

The test you're using doesn't look quite right to me. You're asking for a Mono when the server will give you a Flux<DataBuffer>, and the DataBufferDecoder doesn't support reducing to a Mono. If you'd like to deal with low-level DataBuffer instances, it's your job to manage those pooled resources.

Rewriting that test to reduce it to a Mono<DataBuffer>, things work properly with httpbin.org, but there's indeed an issue with your server instance. Is your server currently running the snapshot version?

[spring-projects-issues]

Alexander Wilhelmer commented

The test isn't active and don't makes sense, cause requests against localhost are working fine.

The Server is running with RC3. I tested with SNAPSHOT on a local server from my employer. I can deploy the app with SNAPSHOT on the server in the evening today.

[spring-projects-issues]

Alexander Wilhelmer commented

I updated the reference project to SNAPSHOT and added a remote test and corrected the local test.

This is the error on the RemoteTest:

java.lang.AssertionError: expected:<102400> but was:<98304>

> GET http://217.79.182.40:8091/download/stream/102400
> WebTestClient-Request-Id: [6]
> Accept: [application/octet-stream]

No content

< 200 OK
< Last-Modified: [Wed, 31 Dec 1969 23:59:59 GMT]
< Content-Type: [application/octet-stream]
< Transfer-Encoding: [chunked]
< Date: [Mon, 11 Sep 2017 15:56:03 GMT]

Unknown charset (98304 bytes)

So you can see that always the last chunk is missing, i saw the chunk size is 4096 bytes.

[spring-projects-issues]

Alexander Wilhelmer commented

The remote server is already deployed with SNAPSHOT

[spring-projects-issues]

Brian Clozel commented

Yes, Violeta Georgieva managed to reproduce the issue on her gear. I couldn't reproduce it on win8 + in a local VM. You're right, the network probably comes into play.
I've streamlined things a bit in my fork: https://github.com/bclozel/streambug

We're still looking into this. This is not related to reactor-netty as a server, since this repro project is using Tomcat. So this might be the client, or a subtle combination of several things.

Thanks for your efforts!

[spring-projects-issues]

Alexander Wilhelmer commented

I tested on a remote linux instance. No errors. Localhost no errors too.

[spring-projects-issues]

Alexander Wilhelmer commented

i will test it with jetty and undertow too now. so the server might be offline for some moments.

[spring-projects-issues]

Alexander Wilhelmer commented

With Jetty:
Caused by: java.lang.AssertionError: expected:<102400> but was:<100862>
at org.junit.Assert.fail(Assert.java:88)

With Undertow:
No errors!

[spring-projects-issues]

Alexander Wilhelmer commented

BTW. Today i saw this error on a JSON only endpoint with a array with a lot of data (30 elements with ~20 attributes, very unusual!).It can be the same cause, so this problem can maybe more critical for windows hosts.

[spring-projects-issues]

Violeta Georgieva commented

Hi,

I can reproduce the issue only with Tomcat/Jetty as runtimes.
My analysis showed that sometimes the last flush operation is not performed as expected.

I prepared here a PR with a change that ensures the last flush.
#1526

Do you think you can rebuild on your side and test the change?

Regards,
Violeta

[spring-projects-issues]

Alexander Wilhelmer commented

Hi,

thanks for your great work. I updated my server and run this test five times. Can't reproduce now with your PR.

[spring-projects-issues]

Brian Clozel commented

Thanks for the hard work Violeta Georgieva and Alexander Wilhelmer - this is now in master!