Scientific Workflow Management Systems (SWfMS) [3] allow for scientists to specify workflows consisting of activities (i.e., program invocations) and their data dependencies. Scientific workflows can be executed using local computing resources and High Performance Computing (HPC) environments
such as computing clusters, grids and clouds. Although several SWfMS provide mechanisms for executing large-scale scientific workflows in distributed environments [4,8,9] most of them perform the workflow execution in an "offine" way, according to Ailamaki et al. [1]. Existing approaches provide results and provenance data [5] that can only be analyzed after processing the entire dataset within
the workflow. However, as the experiment complexity, the volume of data and the need for computing power are on the rise, scientists need mechanisms for monitoring, analyzing partial results, and taking action during workflow execution.
CFD analyses, for example, take several factors into consideration: geometry, viscosity, mesh partitioning, time step size, wall time and the frequency at which the results are stored; just to name a
few. According to the initial setup, the simulation may produce huge amounts of data. Based on the produced outcome, scientists may see they need to explore the simulation diFFerently. They may need to refine the mesh, change time step size or store more or less results during specific simulation time intervals. Nowadays, scientists simply run the simulation again from the beginning. However, if they
have a significant feedback of what is currently happening, they can take actions during the execution and profit from a better resource utilization. Besides, steering the execution of a workflow may help scientists to achieve the desired outcome faster.
In this work, we discuss algorithms and techniques that may give scientists the possibility to steer
their experiments taking advantage of querying provenance data at real-time. When scientists run their workflows, provenance records keep track of everything that has already happened, what is currently happening and what still needs to be executed in the workflow. Thus we present our ongoing approaches to handle what we believe are the three main issues related to steering in scientific
workflows: (i) monitoring of execution, (ii) data analysis at runtime, and (iii) dynamic interference in the execution.
For monitoring and notification, SciLightning [2] noties scientists about events that are important through mobile devices and social networks (e.g., Facebook, SMS, and Twitter) and opens a communication channel between the mobile device and the remote (e.g. cloud) execution. For data analysis,
Prov-Viz [6] allows for querying and traversing the provenance database, to stage out selected data and visualize them in a local machine or on tiled wall displays. We also show our ongoing approach to interfere in the execution using a provenance API for steering Chiron [7], our algebraic workflow engine.
[1] A. Ailamaki, Managing scientic data: lessons, challenges, and opportunities, Proceedings of the
2011 ACM SIGMOD International Conference on Management of Data, 1045-1046, 2011.
[2] F. Cota, F., V. Silva, D. Oliveira, K. Ocana, J. Dias, E. Ogasawara, M. Mattoso, Capturing and querying workflow runtime provenance with PROV: a practical approach, Proceedings of the International Workshop on Managing and Querying Provenance Data at Scale, 2013.
[3] E. Deelman, D. Gannon, M. Shields, I. Taylor, Workflows and e-Science: an overview of workflow
system features and capabilities, Future Generation Computer Systems, 25(5):528-540, 2009.
[4] E.Deelman, G. Mehta, G. Singh, M.-H. Su, K. Vahi, Pegasus: mapping large-scale workflows to distributed resources, Work
ows for e-Science, Springer, 376-394, 2007.
[5] J. Freire, D. Koop, E. Santos, E., Silva, C.T., Provenance for computational tasks: a survey, Computing in Science and Engineering, v.10(3):11-21, 2008.
[6] F. Horta, J. Dias, K. Oca~na, D. Oliveira, E. Ogasaw, M. Mattoso, Poster: using provenance to visualize data from large-scale experiments, Poster Proceedings of the International Conference on High Performance Computing, Networking, Storage and Analysis, 2012.
[7] E. Ogasawara, J. Dias, V. Silva, F. Chirigati, D. Oliveira, F. Porto, P. Valduriez, M. Mattoso, M. Chiron, A parallel engine for algebraic scientific workflows, Concurrency and Computation, 2013.
[8] D. Oliveira, K. Ocaña, F. Baião, M. Mattos, A provenance-based adaptive scheduling heuristic for parallel scientific workflows in clouds, Journal of Grid Computing, 10(3):521-552, 2012.
[9] M. Wilde, M., Hategan, J.M. Wozniak, B. Cliord, D.S. Katz, I. Foster, Swift: a language for distributed parallel scripting, Parallel Computing(37(9)):633-652, 2011.
×