Maestro Workflow Conductor (maestrowf)

11 May 2019

Reading time ~8 minutes

> Projects > Maestro Workflow Conductor (maestrowf)

Overview

What is Maestro?

Maestro is a Python library and command line tool for automating and monitoring software workflows that I authored at LLNL. The main concept centers around conducting “studies” using independent workflow steps that are chained together by user specified execution-order dependency. A study is defined in a YAML file that Maestro denotes as a “study specification”. A specification can be user written or programmatically generated. Maestro parses the a study specification, generating standard data structures for setting up, executing, and monitoring a study.

Why did you create Maestro?

I created Maestro because of my experiences running simulation workflows and the general trend of ignoring automation. In my experience, automation and infrastructure are often viewed as efforts that are lesser priority to accomplishing critical path milestones. In response, I wanted to design a lightweight tool that could be used to quickly stand up automated software workflows but that also accomplished a few other important properties:

encourage users to think about workflows as standalone steps with intent and purpose.
encourage users to better document their workflows.
make user workflows easier to share and communicate.
make workflows easier to parameterize and manage.

What features does Maestro have?

Maestro has a large set of features that help to make running software workflows easier:

Workspace set up and management.
- Initial workspace set up and basic study metadata generation.
- Automatic study workspace timestamping.
- Automatic step workspace generation before execution.

What does a study specification look like?

A basic single step “hello world” example looks as follows:

description:
    name: hello_world
    description: A simple 'Hello World' study.
env:
    variables:
        OUTPUT_PATH: ./sample_output/hello_world
study:
    - name: hello_world
      description: Say hello to the world!
      run:
          cmd: |
            echo "Hello, World!" > hello_world.txt

You can easily chain steps together by specifying dependencies. If you wanted to say “bye” after saying “hello”:

description:
    name: hello_bye
    description: A simple 'Hello World' study.
env:
    variables:
        OUTPUT_PATH: ./sample_output/hello_world
    labels:
        OUT_FORMAT: $(GREETING)_$(NAME).txt
study:
    - name: hello_world
      description: Say hello to someone!
      run:
          cmd: |
            echo "Hello, World!" > hello_world.txt
    - name: bye_world
      description: Say bye to someone!
      run:
          cmd: |
            echo "Bye, World!" > bye_world.txt
          depends: [hello_world]

Core Design Principles

When performing scientific software workflows, there are a few desirable properties that are essential to consider. Maestro focuses on the following:

Repeatability. Consistently repeating a workflow is the first step to reproducing it.
Extendibility. Workflows should be modular, allowing workflows to be extensible, testable, and robust.
Documentation. Good documentation provides an opportunity for the user to step back and explain to themselves what they’re doing.
Shareability. A workflow should be easy given to other users with minimal coupling to any one user’s environment.

Citations

Di Natale, F., Bhatia, H., Carpenter, T. S., Neale, C., Schumacher, S. K., Oppelstrup, T., … Ingólfsson, H. I. (2019). A Massively Parallel Infrastructure for Adaptive Multiscale Simulations: Modeling RAS Initiation Pathway for Cancer. In Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis (pp. 57:1–57:16). New York, NY, USA: ACM. https://doi.org/10.1145/3295500.3356197

Computational models can define the functional dynamics of complex systems in exceptional detail. However, many modeling studies face seemingly incommensurate requirements: to gain meaningful insights into some phenomena requires models with high resolution (microscopic) detail that must nevertheless evolve over large (macroscopic) length- and time-scales. Multiscale modeling has become increasingly important to bridge this gap. Executing complex multiscale models on current petascale computers with high levels of parallelism and heterogeneous architectures is challenging. Many distinct types of resources need to be simultaneously managed, such as GPUs and CPUs, memory size and latencies, communication bottlenecks, and filesystem bandwidth. In addition, robustness to failure of compute nodes, network, and filesystems is critical.

We introduce a first-of-its-kind, massively parallel Multiscale Machine-Learned Modeling Infrastructure (MuMMI), which couples a macro scale model spanning micrometer length- and millisecond time-scales with a micro scale model employing high-fidelity molecular dynamics (MD) simulations. MuMMI is a cohesive and transferable infrastructure designed for scalability and efficient execution on heterogeneous resources. A central workflow manager simultaneously allocates GPUs and CPUs while robustly handling failures in compute nodes, communication networks, and filesystems. A hierarchical scheduler controls GPU-accelerated MD simulations and in situ analysis.

We present the various MuMMI components, including the macro model, GPU-accelerated MD, in situ analysis of MD data, machine learning selection module, a highly scalable hierarchical scheduler, and detail the central workflow manager that ties these modules together. In addition, we present performance data from our runs on Sierra, in which we validated MuMMI by investigating an experimentally intractable biological system: the dynamic interaction between RAS proteins and a plasma membrane. We used up to 4000 nodes of the Sierra supercomputer, concurrently utilizing over 16,000 GPUs and 176,000 CPU cores, and running up to 36,000 different tasks. This multiscale simulation includes about 120,000 MD simulations aggregating over 200 milliseconds, which is orders of magnitude greater than comparable studies.

@inproceedings{DiNatale2019MPI32955003356197,
  author = {Di Natale, Francesco and Bhatia, Harsh and Carpenter, Timothy S. and Neale, Chris and Schumacher, Sara Kokkila and Oppelstrup, Tomas and Stanton, Liam and Zhang, Xiaohua and Sundram, Shiv and Scogland, Thomas R. W. and Dharuman, Gautham and Surh, Michael P. and Yang, Yue and Misale, Claudia and Schneidenbach, Lars and Costa, Carlos and Kim, Changhoan and D'Amora, Bruce and Gnanakaran, Sandrasegaram and Nissley, Dwight V. and Streitz, Fred and Lightstone, Felice C. and Bremer, Peer-Timo and Glosli, James N. and Ing\'{o}lfsson, Helgi I.},
  title = {A Massively Parallel Infrastructure for Adaptive Multiscale Simulations: Modeling RAS Initiation Pathway for Cancer},
  booktitle = {Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis},
  series = {SC '19},
  year = {2019},
  month = nov,
  isbn = {978-1-4503-6229-0},
  location = {Denver, Colorado},
  pages = {57:1--57:16},
  articleno = {57},
  numpages = {16},
  url = {http://doi.acm.org/10.1145/3295500.3356197},
  link = {https://dl.acm.org/doi/pdf/10.1145/3295500.3356197?download=true},
  doi = {10.1145/3295500.3356197},
  acmid = {3356197},
  publisher = {ACM},
  address = {New York, NY, USA},
  keywords = {adaptive simulations, cancer research, heterogenous architecture, machine learning, massively parallel, multiscale simulations},
  notes = {Won Best Paper}
}

Patki, T., Frye, Z., Bhatia, H., Di Natale, F., Glosli, J., Ingolfsson, H., & Rountree, B. (2019). Comparing GPU Power and Frequency Capping: A Case Study with the MuMMI Workflow. In 2019 IEEE/ACM Workflows in Support of Large-Scale Science (WORKS) (pp. 31–39). https://doi.org/10.1109/WORKS49585.2019.00009

Accomplishing the goal of exascale computing under a potential power limit requires HPC clusters to maximize both parallel efficiency and power efficiency. As modern HPC systems embark on a trend toward extreme heterogeneity leveraging multiple GPUs per node, power management becomes even more challenging, especially when catering to scientific workflows with co-scheduled components. The impact of managing GPU power on workflow performance and run-to-run reproducibility has not been adequately studied. In this paper, we present a first-of-its-kind research to study the impact of the two power management knobs that are available on NVIDIA Volta GPUs: frequency capping and power capping. We analyzed performance and power metrics of GPU’s on a top-10 supercomputer by tuning these knobs for more than 5,300 runs in a scientific workflow. Our data found that GPU power capping in a scientific workflow is an effective way of improving power efficiency while preserving performance, while GPU frequency capping is a demonstrably unpredictable way of reducing power consumption. Additionally, we identified that frequency capping results in higher variation and anomalous behavior on GPUs, which is counterintuitive to what has been observed in the research conducted on CPUs.

@inproceedings{8943552,
  author = {{Patki}, T. and {Frye}, Z. and {Bhatia}, H. and {Di Natale}, F. and {Glosli}, J. and {Ingolfsson}, H. and {Rountree}, B.},
  booktitle = {2019 IEEE/ACM Workflows in Support of Large-Scale Science (WORKS)},
  title = {Comparing GPU Power and Frequency Capping: A Case Study with the MuMMI Workflow},
  year = {2019},
  volume = {},
  number = {},
  pages = {31-39},
  keywords = {Workflows; Cancer MuMMI; GPU power capping; GPU frequency capping; Performance; Variation},
  doi = {10.1109/WORKS49585.2019.00009},
  issn = {null},
  month = nov
}

Peterson, J. L., Anirudh, R., Athey, K., Bay, B., Bremer, P.-T., Castillo, V., … Yeom, J.-S. (2019). Merlin: Enabling Machine Learning-Ready HPC Ensembles.

@misc{peterson2019merlin,
  title = {Merlin: Enabling Machine Learning-Ready HPC Ensembles},
  author = {Peterson, J. Luc and Anirudh, Rushil and Athey, Kevin and Bay, Benjamin and Bremer, Peer-Timo and Castillo, Vic and Natale, Francesco Di and Fox, David and Gaffney, Jim A. and Hysom, David and Jacobs, Sam Ade and Kailkhura, Bhavya and Koning, Joe and Kustowski, Bogdan and Langer, Steven and Robinson, Peter and Semler, Jessica and Spears, Brian and Thiagarajan, Jayaraman and Essen, Brian Van and Yeom, Jae-Seung},
  year = {2019},
  eprint = {1912.02892},
  archiveprefix = {arXiv},
  primaryclass = {cs.DC}
}

Ahn, D. H., Bass, N., Chu, A., Garlick, J., Grondona, M., Herbein, S., … others. (2018). Flux: Overcoming Scheduling Challenges for Exascale Workflows. In 2018 IEEE/ACM Workflows in Support of Large-Scale Science (WORKS) (pp. 10–19). IEEE.

@inproceedings{ahn2018flux,
  title = {Flux: Overcoming Scheduling Challenges for Exascale Workflows},
  author = {Ahn, Dong H and Bass, Ned and Chu, Albert and Garlick, Jim and Grondona, Mark and Herbein, Stephen and Koning, Joseph and Patki, Tapasya and Scogland, Thomas RW and Springmeyer, Becky and others},
  booktitle = {2018 IEEE/ACM Workflows in Support of Large-Scale Science (WORKS)},
  pages = {10--19},
  year = {2018},
  organization = {IEEE}
}

Carpenter, T. S., López, C. A., Neale, C., Montour, C., Ingólfsson, H. I., Di Natale, F., … Gnanakaran, S. (2018). Capturing Phase Behavior of Ternary Lipid Mixtures with a Refined Martini Coarse-Grained Force Field. Journal of Chemical Theory and Computation, 14(11), 6050–6062. https://doi.org/10.1021/acs.jctc.8b00496

@article{doi101021acsjctc8b00496,
  author = {Carpenter, Timothy S. and L\'{o}pez, Cesar A. and Neale, Chris and Montour, Cameron and Ing\'{o}lfsson, Helgi I. and Di Natale, Francesco and Lightstone, Felice C. and Gnanakaran, S.},
  title = {Capturing Phase Behavior of Ternary Lipid Mixtures with a Refined Martini Coarse-Grained Force Field},
  journal = {Journal of Chemical Theory and Computation},
  volume = {14},
  number = {11},
  pages = {6050-6062},
  year = {2018},
  doi = {10.1021/acs.jctc.8b00496},
  note = {PMID: 30253091},
  url = {https://doi.org/10.1021/acs.jctc.8b00496},
  eprint = {https://doi.org/10.1021/acs.jctc.8b00496}
}