pymongo


Can't replace mongo document


I am attempting to save documents to a mongoDB cluster (sharded replica sets) and am having a strange issue. I am using pymongo 2.7.2 and TokuMX 1.5 mongodb 2.4.10.
When I attempt to save (overwrite) existing documents I am getting an exception that looks like the document I am saving is too large:
doc = db.collection.find_one()
db.collection.save(doc)
pymongo.errors.OperationFailure: BSONObj size: 18798961 (0x71D91E01) is invalid. Size must be between 0 and 16793600(16MB) First element: op: "u"
However this works fine:
doc = db.collection.find_one()
db.collection.remove({'_id': doc['_id']})
db.collection.save(doc)
The document in question is about 9mb, so it looks like when I attempt to replace the document it is somehow doubling the size of the document, exceeding the 16mb limit.
Any ideas as to what could cause this behavior?
Apparently this is a known issue with TokuMX. Oplog entries are twice the size of the document, so replacing a 9mb document will result in a 18mb oplog entry- which raises the exception.
The solution would be to limit document writes to less than 8mb so that oplog entries never exceed 16mb.
I think this is a side effect of how save is implemented in PyMongo.
Under the hood if the document has a _id then the save(doc) is turned into an update(doc, doc). That is where the doubling is coming into play since the query+update is 18MB.
When you removed the _id you changed the save(doc) into a insert(doc) of a new document with a new _id. I don't think that is what you wanted.
Rather than use save I would recommend constructing a query with just the _id field from the original document and doing the update call manually. I would even go so far as you should enter a Jira ticket to get PyMongo to do this for you.
HTH,
Rob.

Related Links

flask-admin and pymongo search
Running code when mongoengine loads a document
Can't replace mongo document
How do I call a function with pymongo using webpy?
python motor offers fetch_next exception on aggregation
python motor mongo cursor length or peek next
PyMongo check if update succeeded
insert and query an OrderedDict in MongoHQ
Return .str of ObjectID using pymongo
Selecting the lowest/highest with find_one in Pymongo
Safe use of Pymongo's find_one
Pymongo $in + $regex
How can I get pymongo to always return str and not unicode?
SON Manipulator in mongo queries
How to append a document to a 2nd level embedded model list document in Django and mongoengine
pymongo multiple update $in

Categories

HOME
qt
nam
yaml
kivy
hdfs
android-emulator
joomla3.2
mailing-list
scrape
slide
freertos
powermock
spring-jpa
cuba-platform
trace32
rest-assured
pygobject
azure-machine-learning
bosun
sharedpreferences
liferay-6.2
gollum-wiki
acl
websphere-portal
list-comprehension
django-import-export
sendmessage
microprocessors
advanced-filter
textmate2
redgate
twilio-php
resampling
coverage.py
webalizer
rm
datediff
asp.net-web-api-routing
dynamics-crm-4
svnserve
openseadragon
fiber
yii2-model
ajaxmin
uibinder
rails-engines
urbit
i2b2
wepay
miniconda
opencyc
chicagoboss
ambiguity
wso2ml
embeddedwebserver
darwin
clarion
tree-structure
mxe
jcheckbox
tabris
object-code
libgcc
category-theory
reference-counting
oxygene
meteorite
application-blocks
ctp
xsd2code
tnt4j
pacman
lynx
multiple-dispatch
xcode4.5
audiotoolbox
getopt-long
coordinate
quartz-core
background-repeat
mvcrecaptcha
litespeed
data-dump
mediacontroller
cewolf
servlet-container
android-2.1-eclair
mongrel
jdownloader
rational-unified-process
spring-modules
projectpier
rtti
datagridcolumn
jboss-mdb
objectinstantiation
regioninfo
html-generation
word-automation

Resources

Mobile Apps Dev
Database Users
javascript
java
csharp
php
android
MS Developer
developer works
python
ios
c
html
jquery
RDBMS discuss
Cloud Virtualization
Database Dev&Adm
javascript
java
csharp
php
python
android
jquery
ruby
ios
html
Mobile App
Mobile App
Mobile App