dashdb


Inconsistencies in bulk loading into DashDB via the REST API?


Using the POST multipart upload endpoint for file upload in the DashDB REST API (https://developer.ibm.com/clouddataservices/wp-content/themes/projectnext-clouddata/dashDB/), I've noticed random drops in records capturing.
I have tested this in single POST calls attached with a couple thousand csv files (netting close to 100k records) down to single POST calls with just a couple a handful of csv files (with less than 5000k records), and I can see that records committed would be different from one run to the next (on the same files).
Example output (with some formatting and such):
Run 1:
Read and attached part-00000 into POST request.
Read and attached part-00001 into POST request.
Read and attached part-00002 into POST request.
Read and attached part-00003 into POST request.
Read and attached part-00004 into POST request.
Read and attached part-00005 into POST request.
Read and attached part-00006 into POST request.
Read and attached part-00007 into POST request.
Read and attached part-00008 into POST request.
Read and attached part-00009 into POST request.
Read and attached part-00010 into POST request.
Read and attached part-00011 into POST request.
Read and attached part-00012 into POST request.
Read and attached part-00013 into POST request.
Read and attached part-00014 into POST request.
Finished DashDB POST upload request. Returned Log: aaaaaaaaaaaaa
Rows Committed: 3601
Rows Deleted: 1
Rows Skipped: 0
Success Percentage: 99.9722%
Total Run Time: 36.0263 seconds
Run 2:
Read and attached part-00000 into POST request.
Read and attached part-00001 into POST request.
Read and attached part-00002 into POST request.
Read and attached part-00003 into POST request.
Read and attached part-00004 into POST request.
Read and attached part-00005 into POST request.
Read and attached part-00006 into POST request.
Read and attached part-00007 into POST request.
Read and attached part-00008 into POST request.
Read and attached part-00009 into POST request.
Read and attached part-00010 into POST request.
Read and attached part-00011 into POST request.
Read and attached part-00012 into POST request.
Read and attached part-00013 into POST request.
Read and attached part-00014 into POST request.
Finished DashDB POST upload request. Returned Log: bbbbbbbbbbbbb
Rows Committed: 3509
Rows Deleted: 1
Rows Skipped: 0
Success Percentage: 99.9715%
Total Run Time: 36.6753 seconds
I don't know if this makes a difference, but the table I'm loading into is 128 columns deep.

Related Links

How to export IXF and LOB files from dashdb
Can not run sample R/dashDB script “Server Memory Usage”
Does dashDB supports spatial objects?
sql to list tables in a dashDB schema
Inconsistencies in bulk loading into DashDB via the REST API?
dashDB query node: Error: [IBM][CLI Driver][DB2/LINUXX8664] SQL0964C The transaction log for the database is full. SQLSTATE=57011
Why is loading dashDB analytics by trickle feed a bad idea?
how to get the table DDL in dashDB console?
Difference between dashDB for Analytics and dashDB for Transactions?
pip install ibmdbpy gives syntax error exec nested function
how to query disk used / available on dashDB
dashdb out of memory on Cloudant sync
how to perform 'SELECT TOP X FROM TABLE' type queries with DB2 / dashDB

Categories

HOME
php
libgdx
class
automapper
apple-push-notifications
openxml
jms
filesize
jersey-2.0
yql
facebook-android-sdk
phpmqtt
siesta
twitter-oauth
xades4j
sap-fiori
trace32
texas-instruments
reselect
user-defined-types
cloudsim
salt-cloud
adobe-illustrator
amazon-mws
data-type-conversion
activeadmin
workday
bsd
pymongo
android-service
iwebbrowser2
xcrun
om-next
cmis
transform
jslint
unification
klee
virtual-memory
morphline
oracle-bmcs
ipp-protocol
dredd
rm
palindrome
business-rules
http4s
oracle-fusion-apps
portal
rational-performance-test
pjax
openquery
pyaudio
coovachilli
cdo.message
strpos
teensy
grinder
cryptojs
auto-generate
vdm-sl
modern.ie
flow-js
dbmigrate
whitespace-language
mgwt
incron
baseadapter
adhoc-polymorphism
password-recovery
rubber
quantify
nidaqmx
distributed-r
stripe.net
miglayout
windows-taskbar
acitree
mp4parser
acceptance-testing
bcdedit
unicoins
dibs
jbox2d
ckeditor.net
getproperty
coordinate
mkannotation
django-apps
rtd
airprint
servlet-container
git-log
turbine
gaelyk
android-2.1-eclair
explicit
sscli
tabbarcontroller
fxruby
.net-client-profile
autobench
motif
regioninfo
webclient.uploaddata

Resources

Mobile Apps Dev
Database Users
javascript
java
csharp
php
android
MS Developer
developer works
python
ios
c
html
jquery
RDBMS discuss
Cloud Virtualization
Database Dev&Adm
javascript
java
csharp
php
python
android
jquery
ruby
ios
html
Mobile App
Mobile App
Mobile App