dashdb


Inconsistencies in bulk loading into DashDB via the REST API?


Using the POST multipart upload endpoint for file upload in the DashDB REST API (https://developer.ibm.com/clouddataservices/wp-content/themes/projectnext-clouddata/dashDB/), I've noticed random drops in records capturing.
I have tested this in single POST calls attached with a couple thousand csv files (netting close to 100k records) down to single POST calls with just a couple a handful of csv files (with less than 5000k records), and I can see that records committed would be different from one run to the next (on the same files).
Example output (with some formatting and such):
Run 1:
Read and attached part-00000 into POST request.
Read and attached part-00001 into POST request.
Read and attached part-00002 into POST request.
Read and attached part-00003 into POST request.
Read and attached part-00004 into POST request.
Read and attached part-00005 into POST request.
Read and attached part-00006 into POST request.
Read and attached part-00007 into POST request.
Read and attached part-00008 into POST request.
Read and attached part-00009 into POST request.
Read and attached part-00010 into POST request.
Read and attached part-00011 into POST request.
Read and attached part-00012 into POST request.
Read and attached part-00013 into POST request.
Read and attached part-00014 into POST request.
Finished DashDB POST upload request. Returned Log: aaaaaaaaaaaaa
Rows Committed: 3601
Rows Deleted: 1
Rows Skipped: 0
Success Percentage: 99.9722%
Total Run Time: 36.0263 seconds
Run 2:
Read and attached part-00000 into POST request.
Read and attached part-00001 into POST request.
Read and attached part-00002 into POST request.
Read and attached part-00003 into POST request.
Read and attached part-00004 into POST request.
Read and attached part-00005 into POST request.
Read and attached part-00006 into POST request.
Read and attached part-00007 into POST request.
Read and attached part-00008 into POST request.
Read and attached part-00009 into POST request.
Read and attached part-00010 into POST request.
Read and attached part-00011 into POST request.
Read and attached part-00012 into POST request.
Read and attached part-00013 into POST request.
Read and attached part-00014 into POST request.
Finished DashDB POST upload request. Returned Log: bbbbbbbbbbbbb
Rows Committed: 3509
Rows Deleted: 1
Rows Skipped: 0
Success Percentage: 99.9715%
Total Run Time: 36.6753 seconds
I don't know if this makes a difference, but the table I'm loading into is 128 columns deep.

Related Links

How to export IXF and LOB files from dashdb
Can not run sample R/dashDB script “Server Memory Usage”
Does dashDB supports spatial objects?
sql to list tables in a dashDB schema
Inconsistencies in bulk loading into DashDB via the REST API?
dashDB query node: Error: [IBM][CLI Driver][DB2/LINUXX8664] SQL0964C The transaction log for the database is full. SQLSTATE=57011
Why is loading dashDB analytics by trickle feed a bad idea?
how to get the table DDL in dashDB console?
Difference between dashDB for Analytics and dashDB for Transactions?
pip install ibmdbpy gives syntax error exec nested function
how to query disk used / available on dashDB
dashdb out of memory on Cloudant sync
how to perform 'SELECT TOP X FROM TABLE' type queries with DB2 / dashDB

Categories

HOME
powerbi
d3.js
writefile
enums
plaid
openlayers-3
odata
joomla3.2
tweepy
tup
roku
wine
yahoo
hystrix
driver
fireloop
alljoyn
parsley.js
erd
angular4
pinterest
bazaar
multiple-tables
restier
react-dnd
tracing
clipboard
onchange
code-climate
cube
region
unmarshalling
credit-card
dotcms
knights-tour
xmlunit
winrm
wdk
android-n
iscroll
microkernel
resampling
minikube
karabiner
connect-direct
mobile-angular-ui
adobe-reader
oracle-fusion-apps
scheduledexecutorservice
deviare
bigdecimal
nvda
coovachilli
openseadragon
facebook-ios-sdk
xcopy
vim-plugin
certificate-authority
lovefield
google-maps-api-2
arbre
maven-shade-plugin
relationships
maven-central
rpg
reference-counting
refit
jack
boinc
form-for
acceptance-testing
asp.net-mvc-3-areas
proc-open
xcode4.5
audiotoolbox
border-box
getopt-long
exponent
netstream
qtextbrowser
glassfish-esb
representation
office-automation
turbine
self-tracking-entities
fxruby
subtext
server-load
remote-working
html-generation
word-automation

Resources

Database Users
RDBMS discuss
Database Dev&Adm
javascript
java
csharp
php
android
javascript
java
csharp
php
python
android
jquery
ruby
ios
html
Mobile App
Mobile App
Mobile App