apache-toree


Apache Toree job progress


With Apache Toree one can execute arbitrary expressions on Spark. Say we want to execute some SQL query like:sqlContext.sql(..)
Is it possible to get a progress (like in Zeppelin) for such SQL query? Maybe Toree can provide some query metrics (like X tasks from N are done)?
The way Apache Zeppelin use is through sc.dagScheduler.
REST API should be a better choice if can not access the SparkContext directly.
package org.apache.zeppelin.spark
class SparkInterpreter {
#Override
public int getProgress(InterpreterContext context) {
String jobGroup = getJobGroup(context);
int completedTasks = 0;
int totalTasks = 0;
DAGScheduler scheduler = sc.dagScheduler();
if (scheduler == null) {
return 0;
}
HashSet<ActiveJob> jobs = scheduler.activeJobs();
if (jobs == null || jobs.size() == 0) {
return 0;
}
Iterator<ActiveJob> it = jobs.iterator();
while (it.hasNext()) {
ActiveJob job = it.next();
String g = (String) job.properties().get("spark.jobGroup.id");
if (jobGroup.equals(g)) {
int[] progressInfo = null;
try {
Object finalStage = job.getClass().getMethod("finalStage").invoke(job);
if (sparkVersion.getProgress1_0()) {
progressInfo = getProgressFromStage_1_0x(sparkListener, finalStage);
} else {
progressInfo = getProgressFromStage_1_1x(sparkListener, finalStage);
}
} catch (IllegalAccessException | IllegalArgumentException
| InvocationTargetException | NoSuchMethodException
| SecurityException e) {
logger.error("Can't get progress info", e);
return 0;
}
totalTasks += progressInfo[0];
completedTasks += progressInfo[1];
}
}
if (totalTasks == 0) {
return 0;
}
return completedTasks * 100 / totalTasks;
}
}

Related Links

Jupyter kernel Apache toree startup timeout
Apache Toree job progress

Categories

HOME
powerbi
list
clearcase
livecode
scroll
loopbackjs
hdfs
twitter-fabric
fogbugz
impala
sentry
tweepy
port
azure-logic-apps
gtk
log4j2
lda
yeoman-generator-angular
erd
tiff
responsivevoice
jodatime
xbox360
mahout-recommender
sonicwall
async-await
azure-machine-learning
activeadmin
chronicle-queue
ratio
web-push
prompt
fstar
adapter
phpstorm-2016.1
snap-framework
rspec-rails
overwrite
cups
testlink
cargo
strapi
unification
flowchart
minikube
httphandler
latexml
music21
cloudwatch
rm
beeline
visual-format-language
line-intersection
network-protocols
alertify
instafeedjs
nitrousio
yargs
researchkit
thread-exceptions
jenkins-workflow
fiber
qpixmap
frame-grab
sharepoint-apps
xcopy
textblock
django-redis
taco
okta-api
trust
deeplink
perceptron
largenumber
mixins
apple
swagger-maven-plugin
flow-js
xcode6.3.1
rpg
stripe.net
air-native-extension
stty
apportable
oscilloscope
system-requirements
lynx
system-information
taskkill
cakeyframeanimation
qtextbrowser
fork-join
directshow.net
grails-validation
nsconnection
qcar-sdk
pinax
getimagesize
bucket
iphone-sdk-3.1.3
chronic
webclient.uploaddata

Resources

Encrypt Message