pymongo


Can't replace mongo document


I am attempting to save documents to a mongoDB cluster (sharded replica sets) and am having a strange issue. I am using pymongo 2.7.2 and TokuMX 1.5 mongodb 2.4.10.
When I attempt to save (overwrite) existing documents I am getting an exception that looks like the document I am saving is too large:
doc = db.collection.find_one()
db.collection.save(doc)
pymongo.errors.OperationFailure: BSONObj size: 18798961 (0x71D91E01) is invalid. Size must be between 0 and 16793600(16MB) First element: op: "u"
However this works fine:
doc = db.collection.find_one()
db.collection.remove({'_id': doc['_id']})
db.collection.save(doc)
The document in question is about 9mb, so it looks like when I attempt to replace the document it is somehow doubling the size of the document, exceeding the 16mb limit.
Any ideas as to what could cause this behavior?
Apparently this is a known issue with TokuMX. Oplog entries are twice the size of the document, so replacing a 9mb document will result in a 18mb oplog entry- which raises the exception.
The solution would be to limit document writes to less than 8mb so that oplog entries never exceed 16mb.
I think this is a side effect of how save is implemented in PyMongo.
Under the hood if the document has a _id then the save(doc) is turned into an update(doc, doc). That is where the doubling is coming into play since the query+update is 18MB.
When you removed the _id you changed the save(doc) into a insert(doc) of a new document with a new _id. I don't think that is what you wanted.
Rather than use save I would recommend constructing a query with just the _id field from the original document and doing the update call manually. I would even go so far as you should enter a Jira ticket to get PyMongo to do this for you.
HTH,
Rob.

Related Links

mongoengine slow serialization of embedded documents with reference fields
Pymongo error : 'Cursor' object has no attribute 'find' , how to filter it?
Spark Streaming: foreachRDD insert into mongoDB using python?
how to find right version of bson from pip for pymongo/mongoengine
How to enable `projection` in pymongo query class `find()`?
Variables don't hold value for very long pymongo ipython
pymongo upgrade to 3.0.3 causes this statement to fail
UUID to NUUID in Python
how to test for empty mongodb query in pymong
read video file from mongodb with pymongo
Get OperationFailure error with PyMongo 2.7.1
Inserting string as regular string in mongodb
Mongos + Pymongo 2.5 ==>No suitable hosts found
pymongo collection add item using update
How to convert objectid to string
Conditional atomic mongo remove with different results

Categories

HOME
oracle
entity-framework
mc
yii
swi-prolog
foaf
slider
celery
jersey-2.0
portable-class-library
nsis
cryptography
minecraft
phpseclib
scrape
buffer
dma
temperature
user-defined-types
fosuserbundle
kamailio
multi-dimensional-scaling
onchange
monitor
objectmapper
publishing
memory-fences
wampsharp
knights-tour
overwrite
cloveretl
char-pointer
transform
xerces-c
nppexec
ftp-server
cordys-opentext
jslint
python-jira
flipkart
hypothesis-test
coverage.py
pyttsx
destroy
absolute
qvtkwidget
punctuation
freshdesk
nslayoutconstraint
asp.net-web-api-routing
gawk
cefpython
nunit-console
nvda
jenkins-workflow
declarative
eclemma
ajaxmin
gnucash
seadragon
onepage-checkout
pack
perceptron
directx-9
fanotify
canopy
roxygen2
explain
abstract-factory
subfolder
baseadapter
nhibernate-criteria
rubber
image-rotation
rpg
forceclose
reference-counting
xulrunner
webaii
wso2bam
map
mod-perl
strcmp
iirf
notifyjs
clrprofiler
clearinterval
uitextfielddelegate
groovy-console
mediacontroller
photo-management
hungarian-notation
pinax
icicles
raw-data
bindable-linq

Resources

Database Users
RDBMS discuss
Database Dev&Adm
javascript
java
csharp
php
android
javascript
java
csharp
php
python
android
jquery
ruby
ios
html
Mobile App
Mobile App
Mobile App