How to Increase Buffered Data Limit?
Within Ubuntu I've Created a Mongodb Db Collection Containing 1 Million Records, and When Trying to Run a Sort Command Against It, I'm Getting the Following...
Within Ubuntu I've created a MongoDB Db collection containing 1 million records, and when trying to run a sort command against it, I'm getting the following error:
"$err" : "Runner error: Overflow sort stage buffered data usage of 33555002 bytes exceeds internal limit of 33554432 bytes",
"code" : 17144
Please can anyone describe how to increase the internal limit so that I can execute this and some other large scale commands against the data?
2 Answers
This can happen because of a bug like SERVER-13611 (so make sure you are on the latest version), or because you are trying to sort on a sparse index in 2.6, but more usually it is because you are simply attempting to sort too many records in memory without an index.
The specific limit you are hitting is intentional and is documented here - it cannot be changed, so you need to reduce the set of results, or use an index etc. to perform the sort.
Update (November 2014): The upcoming 2.8 release (2.8.0-rc0 as of writing this) does now allow this setting to be tweaked, as follows:
db.adminCommand({setParameter: 1, internalQueryExecMaxBlockingSortBytes: <limit in bytes>})
The default value is 32MiB (33554432 bytes) and should be adjusted with care - large in-memory sorts can cause your database to grind to a halt (which is why there was a limit in the first place).
I ran into the issue too when sorting and paginating 200K+ records. The easiest solution seems to be to add an index ( for the attributes you are sorting on.
Sarah Jenkins is a veteran tech journalist with over 12 years of experience covering artificial intelligence, mobile innovations, and digital ethics. Her insights have appeared in leading technology publications worldwide.
