CreateBlob fails on files greater than 5 gigs

[halfpeaw]

Verified using the 64bit version of LibGit2sharp

Created a file stream that refers to a large 5 gig pst file.
Stream readStream = new FileStream(filePath, FileMode.Open);
Blob newBlob = repo.ObjectDatabase.CreateBlob(readStream, readStream.Length);

Got this error
LibGit2Sharp.LibGit2SharpException: failed to finish zlib inflation; stream aborted prematurely

[ethomson]

In a cruel twist, this isn't failing to create a blob in the repository, it's failing to load the blob after it's been created. Hence the failed to finish zlib inflation error (not deflate).

1. This API is not ideal for you, unless you really do need the resultant Blob object (with all 5 GB loaded into memory). You want an API that writes to the object database and just gives you back the oid instead of trying to reread it.
2. Unfortunately, the only API that does that expects you to hand it the entire byte[] to write. Ugh.
3. I don't know why this particular call is failing (why we can't Blob.Lookup a 5 GB object) but unless that's actually important to you, I'm going to worry about that part later.

[ethomson]

I created #1518 to provide a mechanism to write a stream to the object database. I hope that this resolves your problems, @halfpeaw. However, the Blob.Lookup issue remains.

[carlosmn]

There is a rather persistent idea across the bindings that objects are lightweight, which makes some things a bit complicated.

IIRC we used to just keep the id of the blob in the Blob object until you tried to load it. This caused different issues so we went with actually loading stuff instead, and I suspect a lot of these uses are a left-over from that time (for compat with types and all that).

[ethomson]

I've created #1518 and #1519 which should help you here, @halfpeaw. However, I'm still seeing some issues with large files going into the ODB, so I'm still digging.

[ethomson]

So it seems that Git for Windows also has an issue decompressing one of the large files that I created ((the file was basically uncompressible, so the zip file itself was >5GB).

So as far as a situation report goes: it looks like LibGit2Sharp should be able to add very large files to the object database properly, and add them to a tree. I'll dig in to why we fail reading next, but I think that this should unblock your use case @halfpeaw and I'll create a prerelease nuget package.