Reading document did not exhaust input buffer

[felipsmartins]

Description

While I know this is not a bug/issue at all, I couldn't to understand the reason this exception:

PHP Fatal error: Uncaught exception 'MongoDB\Driver\Exception\UnexpectedValueException' with message 'Reading document did not exhaust input buffer' in /home/martins/www/projects/cadunico-agent/main.php:69
Stack trace:
#0 /home/martins/www/projects/cadunico-agent/main.php(69): MongoDB\BSON\toJSON('\xDE\x00\x00\x00\x07_id\x00X\x17\x81\xE4"9...')
#1 {main}
thrown in /home/martins/www/projects/cadunico-agent/main.php on line 69

Fatal error: Uncaught exception 'MongoDB\Driver\Exception\UnexpectedValueException' with message 'Reading document did not exhaust input buffer' in /home/martins/www/projects/cadunico-agent/main.php on line 69

MongoDB\Driver\Exception\UnexpectedValueException: Reading document did not exhaust input buffer in /home/martins/www/projects/cadunico-agent/main.php on line 69

BSON files from mongodump:

Note reading non large files like 307bytes it going okay. But reading bson files bigger than 2 kbytes the above exception is throwed.

Environment

$ php -i | grep -E 'mongodb|libmongoc|libbson'

/etc/php5/cli/conf.d/20-mongodb.ini,
mongodb
libbson bundled version => 1.5.4
libmongoc bundled version => 1.5.4
libmongoc SSL => enabled
libmongoc SSL library => OpenSSL
libmongoc crypto => enabled
libmongoc crypto library => libcrypto
libmongoc crypto system profile => disabled
libmongoc SASL => enabled
mongodb.debug => no value => no value
PATH => /opt/mongodb/3.0/bin:/home/martins/bin:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games:/home/martins/.config/composer/vendor/bin:/opt/java/bin:/opt/anaconda/bin
_SERVER["PATH"] => /opt/mongodb/3.0/bin:/home/martins/bin:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games:/home/martins/.config/composer/vendor/bin:/opt/java/bin:/opt/anaconda/bin

Test Script

If possible, attach a complete PHP script that can be excuted on its own to
reproduce the issue. Clarify whether this script can be run from the CLI or if
it can only be reproduced through a web SAPI.

<?php
$bsonFilename = '/path/to/mongodump/my_collection.bson';
$binaryContent = file_get_contents($bsonFilename);
$extendedJson = MongoDB\BSON\toJSON($binaryContent);

Well, there's any limitation for reading bson files via mongodb-php-driver? It's supposed to read only non-large bson files? If so what is the max bson file size?

I've attached bson files in this current issue.
Also, reading the exactly same bson files via bsondump works well.

Thanks in advance!

BSON-files.zip

[jmikola]

The bsondump utility does support processing a sequence of BSON documents in a single file, but the PHP driver's BSON functions only deal with single documents at a time.

MongoDB\BSON\toJSON() is documented as throwing an UnexpectedValueException if "the input did not contain exactly one BSON document". The function converts one BSON document to a single JSON document; it does not support converting a sequence of BSON documents (which I expect my_collection.bson may be) to a JSON array of documents.

Well, there's any limitation for reading bson files via mongodb-php-driver? It's supposed to read only non-large bson files? If so what is the max bson file size?

To recap, the PHP driver does not have a concept of "BSON files", but rather single BSON documents. The limits for BSON documents are documented in the MongoDB manual. Standard memory limits would apply when working with large files in PHP. For instance, you may need to use stream functions to read through large files instead of loading them into a single in-memory string with file_get_contents().

If iterating through a large stream of BSON documents, you can differentiate documents in the stream by the length field present in the header of each document (see: BSON specification for more info). Each BSON document will then consist of the bytes between the leading 32-bit integer and the trailing null byte (inclusive). I applied that process when implementing BsonIterator class I wrote for another project in order to iterate through multiple BSON documents in a binary string.

[felipsmartins]

Thanks for replying, @jmikola !

So I switched to PyMongo as it ships with built-in bson file reader:

import bson

bson_filename = "/path/to/my_file.bson"
docs = bson.decode_file_iter(open(bson_filename)) 
# or bson.decode_all(open(bson_filename).read())

It would be nice php-mongodb-driver could implementing this feature anyway.
So I guess this issue can be closed.

Thanks again, @jmikola!

[jmikola]

I've opened PHPLIB-262 as a tracking ticket for the feature, as it sounds like something we could implement in the userland library.

[felipsmartins]

WOW! Thanks a lot!