Experience

> Experience

I currently work at Lawrence Livermore National Laboratory as a Computer Scientist and workflow expert. My work is focused in the topic areas of HPC simulation, simulation management, software engineering, and software workflow automation. My primary roles are:

Project Lead and Author of the Maestro Workflow Conductor: Maestro is a Python command line tool and library for specifying, automating, and monitoring software workflows. Users create YAML specifications called “study descriptions” that define a software workflows, which are then concretely expanded based on static variables and parameters. See our README about how to build a basic study or our sample studies.
- Maestro GitHub page
  Download Star Watch
- Maestro on PyPi
- Maestro on ReadtheDocs
- Maestro is part of the RADIUSS effort at LLNL
Workflow Expert and Researcher for the RAS protein pilot project (one of three pilots of JDACS4C). Our mission aims at scaling molecular dynamics to long enough timescales to better understand the RAS protein and its role in the development of cancer. Our team has successfully run large scale MD simulation campaigns on Sierra, the second fastest supercomputer on the TOP500 list (as of November 2018). As part of an interdisciplinary team of scientists, my responsibilities include managing the project’s code repositories, providing guidance on workflow/code design, and implementing new features.

FULL RESUME

Interests and Past Experience

My technical interests include (but are not limited to) Software Engineering, Software Design, Computer Architecture, simulation, simulation and software automation, Object-Oriented Design, Python, and HPC. I initially started with an interest in Computer Architecture, which led me to become a Performance Architect at Intel Corporation. My experiences at Intel maintaining simulators, running large numbers of simulations, and using the results to make architectural assessments allowed me to appreciate the software systems required to perform computational studies. Through these experiences I broadened my interests to simulation software workflows, co-designing and implementing Study Launcher, a workflow launcher that utilized an XML specification. In May 2016, I joined Lawrence Livermore National Laboratory as a Computer Scientist and workflow expert to continue to learn more about computational workflow and automation.

Skills and Proficiencies

Excellent (Go-to tools) ★ ★ ★

Python, Git, GitHub, Bash, LaTeX, SLURM, LSF, Unix

Proficient (Competent and comfortable) ★ ★ ☆

C++, C, C#, CSS, HTML, Linux

Basic (Essential foundation and basics) ★ ☆ ☆

Ruby, Java, SQL

Domain Knowledge (Fundamental Concepts)

Software Engineering, Python, Agile/Scrum Development Methods, Software System Design, Object Oriented Design, Algorithms, Simulation, Workflow automation and tools, HPC, Advanced Computer Architecture

Awards and Recognition

Lawarence Livermore National Laboratory Science & Technology Excellence in Publication Award (August 2020)
Best Paper at Supercomputing’19 for A Massively Parallel Infrastructure for Adaptive Multiscale Simulations: Modeling RAS Initiation Pathway for Cancer [ARTICLE]
Multiple ASQ Division Awards at LLNL
Two Intel Team Awards
Most Outstanding Graduate Award by USF’s CSE Department
Member of Tau Beta Pi engineering honor society (FL Gamma Chapter)

Publications

Ferreira da Silva, R., Casanova, H., Chard, K., Laney, D., Ahn, D., Jha, S., … Wozniak, J. (2021). Workflows Community Summit: Bringing the Scientific Workflows Community Together. Zenodo. https://doi.org/10.5281/zenodo.4606958

@misc{ferreira_da_silva_rafael_2021_4606958,
  author = {Ferreira da Silva, Rafael and Casanova, Henri and Chard, Kyle and Laney, Dan and Ahn, Dong and Jha, Shantenu and Goble, Carole and Ramakrishnan, Lavanya and Peterson, Luc and Enders, Bjoern and Thain, Douglas and Altintas, Ilkay and Babuji, Yadu and Badia, Rosa and Bonazzi, Vivien and Coleman, Taina and Crusoe, Michael and Deelman, Ewa and Di Natale, Frank and Di Tommaso, Paolo and Fahringer, Thomas and Filgueira, Rosa and Fursin, Grigori and Ganose, Alex and Gruning, Bjorn and Katz, Daniel S. and Kuchar, Olga and Kupresanin, Ana and Ludascher, Bertram and Maheshwari, Ketan and Mattoso, Marta and Mehta, Kshitij and Munson, Todd and Ozik, Jonathan and Peterka, Tom and Pottier, Loic and Randles, Tim and Soiland-Reyes, Stian and Tovar, Benjamin and Turilli, Matteo and Uram, Thomas and Vahi, Karan and Wilde, Michael and Wolf, Matthew and Wozniak, Justin},
  title = {{Workflows Community Summit: Bringing the 
                     Scientific Workflows Community Together}},
  month = mar,
  year = {2021},
  publisher = {Zenodo},
  doi = {10.5281/zenodo.4606958},
  url = {https://doi.org/10.5281/zenodo.4606958}
}

Peterson, J. L., Anirudh, R., Athey, K., Bay, B., Bremer, P.-T., Castillo, V., … Yeom, J.-S. (2019). Merlin: Enabling Machine Learning-Ready HPC Ensembles.

@misc{peterson2019merlin,
  title = {Merlin: Enabling Machine Learning-Ready HPC Ensembles},
  author = {Peterson, J. Luc and Anirudh, Rushil and Athey, Kevin and Bay, Benjamin and Bremer, Peer-Timo and Castillo, Vic and Natale, Francesco Di and Fox, David and Gaffney, Jim A. and Hysom, David and Jacobs, Sam Ade and Kailkhura, Bhavya and Koning, Joe and Kustowski, Bogdan and Langer, Steven and Robinson, Peter and Semler, Jessica and Spears, Brian and Thiagarajan, Jayaraman and Essen, Brian Van and Yeom, Jae-Seung},
  year = {2019},
  month = dec,
  eprint = {1912.02892},
  archiveprefix = {arXiv},
  primaryclass = {cs.DC},
  link = {https://arxiv.org/pdf/1912.02892.pdf}
}

Di Natale, F., Bhatia, H., Carpenter, T. S., Neale, C., Schumacher, S. K., Oppelstrup, T., … Ingólfsson, H. I. (2019). A Massively Parallel Infrastructure for Adaptive Multiscale Simulations: Modeling RAS Initiation Pathway for Cancer. In Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis (pp. 57:1–57:16). New York, NY, USA: ACM. https://doi.org/10.1145/3295500.3356197

Computational models can define the functional dynamics of complex systems in exceptional detail. However, many modeling studies face seemingly incommensurate requirements: to gain meaningful insights into some phenomena requires models with high resolution (microscopic) detail that must nevertheless evolve over large (macroscopic) length- and time-scales. Multiscale modeling has become increasingly important to bridge this gap. Executing complex multiscale models on current petascale computers with high levels of parallelism and heterogeneous architectures is challenging. Many distinct types of resources need to be simultaneously managed, such as GPUs and CPUs, memory size and latencies, communication bottlenecks, and filesystem bandwidth. In addition, robustness to failure of compute nodes, network, and filesystems is critical.

We introduce a first-of-its-kind, massively parallel Multiscale Machine-Learned Modeling Infrastructure (MuMMI), which couples a macro scale model spanning micrometer length- and millisecond time-scales with a micro scale model employing high-fidelity molecular dynamics (MD) simulations. MuMMI is a cohesive and transferable infrastructure designed for scalability and efficient execution on heterogeneous resources. A central workflow manager simultaneously allocates GPUs and CPUs while robustly handling failures in compute nodes, communication networks, and filesystems. A hierarchical scheduler controls GPU-accelerated MD simulations and in situ analysis.

We present the various MuMMI components, including the macro model, GPU-accelerated MD, in situ analysis of MD data, machine learning selection module, a highly scalable hierarchical scheduler, and detail the central workflow manager that ties these modules together. In addition, we present performance data from our runs on Sierra, in which we validated MuMMI by investigating an experimentally intractable biological system: the dynamic interaction between RAS proteins and a plasma membrane. We used up to 4000 nodes of the Sierra supercomputer, concurrently utilizing over 16,000 GPUs and 176,000 CPU cores, and running up to 36,000 different tasks. This multiscale simulation includes about 120,000 MD simulations aggregating over 200 milliseconds, which is orders of magnitude greater than comparable studies.

@inproceedings{DiNatale2019MPI32955003356197,
  author = {Di Natale, Francesco and Bhatia, Harsh and Carpenter, Timothy S. and Neale, Chris and Schumacher, Sara Kokkila and Oppelstrup, Tomas and Stanton, Liam and Zhang, Xiaohua and Sundram, Shiv and Scogland, Thomas R. W. and Dharuman, Gautham and Surh, Michael P. and Yang, Yue and Misale, Claudia and Schneidenbach, Lars and Costa, Carlos and Kim, Changhoan and D'Amora, Bruce and Gnanakaran, Sandrasegaram and Nissley, Dwight V. and Streitz, Fred and Lightstone, Felice C. and Bremer, Peer-Timo and Glosli, James N. and Ing\'{o}lfsson, Helgi I.},
  title = {A Massively Parallel Infrastructure for Adaptive Multiscale Simulations: Modeling RAS Initiation Pathway for Cancer},
  booktitle = {Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis},
  series = {SC '19},
  year = {2019},
  month = nov,
  isbn = {978-1-4503-6229-0},
  location = {Denver, Colorado},
  pages = {57:1--57:16},
  articleno = {57},
  numpages = {16},
  url = {http://doi.acm.org/10.1145/3295500.3356197},
  link = {https://dl.acm.org/doi/pdf/10.1145/3295500.3356197?download=true},
  doi = {10.1145/3295500.3356197},
  acmid = {3356197},
  publisher = {ACM},
  address = {New York, NY, USA},
  keywords = {adaptive simulations, cancer research, heterogenous architecture, machine learning, massively parallel, multiscale simulations},
  notes = {Won Best Paper}
}

Patki, T., Frye, Z., Bhatia, H., Di Natale, F., Glosli, J., Ingolfsson, H., & Rountree, B. (2019). Comparing GPU Power and Frequency Capping: A Case Study with the MuMMI Workflow. In 2019 IEEE/ACM Workflows in Support of Large-Scale Science (WORKS) (pp. 31–39). https://doi.org/10.1109/WORKS49585.2019.00009

Accomplishing the goal of exascale computing under a potential power limit requires HPC clusters to maximize both parallel efficiency and power efficiency. As modern HPC systems embark on a trend toward extreme heterogeneity leveraging multiple GPUs per node, power management becomes even more challenging, especially when catering to scientific workflows with co-scheduled components. The impact of managing GPU power on workflow performance and run-to-run reproducibility has not been adequately studied. In this paper, we present a first-of-its-kind research to study the impact of the two power management knobs that are available on NVIDIA Volta GPUs: frequency capping and power capping. We analyzed performance and power metrics of GPU’s on a top-10 supercomputer by tuning these knobs for more than 5,300 runs in a scientific workflow. Our data found that GPU power capping in a scientific workflow is an effective way of improving power efficiency while preserving performance, while GPU frequency capping is a demonstrably unpredictable way of reducing power consumption. Additionally, we identified that frequency capping results in higher variation and anomalous behavior on GPUs, which is counterintuitive to what has been observed in the research conducted on CPUs.

@inproceedings{8943552,
  author = {{Patki}, T. and {Frye}, Z. and {Bhatia}, H. and {Di Natale}, F. and {Glosli}, J. and {Ingolfsson}, H. and {Rountree}, B.},
  booktitle = {2019 IEEE/ACM Workflows in Support of Large-Scale Science (WORKS)},
  title = {Comparing GPU Power and Frequency Capping: A Case Study with the MuMMI Workflow},
  year = {2019},
  volume = {},
  number = {},
  pages = {31-39},
  keywords = {Workflows; Cancer MuMMI; GPU power capping; GPU frequency capping; Performance; Variation},
  doi = {10.1109/WORKS49585.2019.00009},
  issn = {null},
  month = nov
}

Pauli, E. T., Aschwanden, P. D., Laney, D. E., Dahlgren, T., Semler, J. A., Di Natale, F., … Administration, U. S. D. O. E. N. N. S. (2018). Simulation INsight and Analysis. https://doi.org/10.11578/dc.20190715.10

@misc{osti_1542560,
title = {Simulation INsight and Analysis},
author = {Pauli, Esteban T and Aschwanden, Pascal D and Laney, Daiel E and Dahlgren, Tamara and Semler, Jessica A and Di Natale, Francesco and Greco, Nathan S and Eklund, Joseph L and Haluska, Rebecca M and Administration, USDOE National Nuclear Security},
abstractnote = {Sina is a tool set for modern scientific data management that provides flexible, light-weight support of non-bulk data capture for retention in and queries against SQL and noSQL data stores. HPC simulations traditionally maintain their data in files. Extracting data of interest for subsequent analysis then requires the time-consuming process of traversing directories and scraping data from files in a variety of formats. Sina facilitates capturing relevant data during execution or post-processing of simulation runs for retention in and queries from a modern data store. The tools are sufficiently general to allow for the inclusion of new fields as scientists learn more about their data. Libraries, currently in C++ and Python, and a command line interface (CLI) are provided. Sina's flexibility starts with a general schema, in JSON, for the collection of non-bulk simulation data. JSON provides a flexible, human-readable representation of the data that of interest. Sina currently has a C++ library for simulations to write data to and read from a schema-compliant file for subsequent ingestion into one of the supported data stores. However, applications are free to write their data directly into a schema-compliant file. Python packages provide data ingestion, management, query, and export capabilities. A command line interface (CLI) provides simplified access to these features. A common application programming interface (API) is used to maintain and query data in any of the supported data stores, which are currently limited to SQL and Apache Cassandra (a column store). Tutorials, demonstrations, and examples illustrate aspects of the process using scripts and Jupyter notebooks.},
url = {https://www.osti.gov//servlets/purl/1542560},
doi = {10.11578/dc.20190715.10},
year = {2018},
month = nov
}

Carpenter, T. S., López, C. A., Neale, C., Montour, C., Ingólfsson, H. I., Di Natale, F., … Gnanakaran, S. (2018). Capturing Phase Behavior of Ternary Lipid Mixtures with a Refined Martini Coarse-Grained Force Field. Journal of Chemical Theory and Computation, 14(11), 6050–6062. https://doi.org/10.1021/acs.jctc.8b00496

@article{doi101021acsjctc8b00496,
  author = {Carpenter, Timothy S. and L\'{o}pez, Cesar A. and Neale, Chris and Montour, Cameron and Ing\'{o}lfsson, Helgi I. and Di Natale, Francesco and Lightstone, Felice C. and Gnanakaran, S.},
  title = {Capturing Phase Behavior of Ternary Lipid Mixtures with a Refined Martini Coarse-Grained Force Field},
  journal = {Journal of Chemical Theory and Computation},
  volume = {14},
  number = {11},
  pages = {6050-6062},
  year = {2018},
  doi = {10.1021/acs.jctc.8b00496},
  note = {PMID: 30253091},
  url = {https://doi.org/10.1021/acs.jctc.8b00496},
  eprint = {https://doi.org/10.1021/acs.jctc.8b00496}
}

Di Natale, F. (2017). Maestro Workflow Conductor. Retrieved from https://github.com/LLNL/maestrowf

MaestroWF is a Python tool and software package for loading YAML study specifications that represents a simulation campaign. The package is capable of parameterizing a study, pulling dependencies automatically, formatting output directories, and managing the flow and execution of the campaign. MaestroWF also provides a set of abstracted objects that can also be used to develop user specific scripts for launching simulation campaigns.

@misc{osti_1372046,
  title = {Maestro Workflow Conductor},
  author = {Di Natale, Francesco},
  year = {2017},
  month = jun,
  url = {https://github.com/LLNL/maestrowf}
}

For the most up-to-date publications, please see my Google Scholar profile

Invited Talks

Di Natale, F., Bhatia, H., Carpenter, T. S., Neale, C., Schumacher, S. K., Oppelstrup, T., … Ingólfsson, H. I. (2019). MuMMI: Massively Parallel Multiscale Simulation for Modeling RAS Protein and ML Workflow Challenges. In Supercomputing ’19.

@conference{mummi_sc19,
  title = {MuMMI: Massively Parallel Multiscale Simulation for Modeling RAS Protein and ML Workflow Challenges},
  author = {Di Natale, Francesco and Bhatia, Harsh and Carpenter, Timothy S. and Neale, Chris and Schumacher, Sara Kokkila and Oppelstrup, Tomas and Stanton, Liam and Zhang, Xiaohua and Sundram, Shiv and Scogland, Thomas R. W. and Dharuman, Gautham and Surh, Michael P. and Yang, Yue and Misale, Claudia and Schneidenbach, Lars and Costa, Carlos and Kim, Changhoan and D'Amora, Bruce and Gnanakaran, Sandrasegaram and Nissley, Dwight V. and Streitz, Fred and Lightstone, Felice C. and Bremer, Peer-Timo and Glosli, James N. and Ing\'{o}lfsson, Helgi I.},
  month = nov,
  day = {20},
  year = {2019},
  booktitle = {Supercomputing '19},
  location = {Denver, CO, USA},
  link = {https://sc19.supercomputing.org/presentation/?id=pap384&sess=sess165},
  notes = {Won Best Paper}
}

Di Natale, F., Bhatia, H., Ingólfsson, H. I., Streitz, F., & Nissley, D. V. (2019). MuMMI: Massively Parallel Multiscale Simulation for Modeling RAS Protein and ML Workflow Challenges. Data Science Workshop.

This session will cover the challenges for many applications of scientific machine learning related to both large scale workflows and incorporating experimental data with scientific simulations. Each of the talks identified will cover different aspects of these challenges and how they relate to multiple scientific areas.

@workshop{dsi_mummi,
  title = {MuMMI: Massively Parallel Multiscale Simulation for Modeling RAS Protein and ML Workflow Challenges},
  author = {Di Natale, Francesco and Bhatia, Harsh and Ing{\'o}lfsson, Helgi I. and Streitz, Fred and Nissley, Dwight V.},
  year = {2019},
  month = jul,
  day = {19},
  booktitle = {Data Science Workshop},
  location = {Livermore, CA, USA},
  link = {https://data-science.llnl.gov/latest/workshop-2019}
}

Posters

Chowdhury, F., Di Natale, F., Moody, A., Gonsiorowski, E., Mohror, K., & Yu, W. (2019). Understanding I/O Behavior in Scientific Workflows on High Performance Computing Systems. Supercomputing’19.

@unpublished{chowdhuryunderstanding,
  title = {Understanding I/O Behavior in Scientific Workflows on High Performance Computing Systems},
  author = {Chowdhury, Fahim and Di Natale, Francesco and Moody, Adam and Gonsiorowski, Elsa and Mohror, Kathryn and Yu, Weikuan},
  booktitle = {Supercomputing'19},
  year = {2019},
  month = nov,
  location = {Denver, Colorado, USA}
}

Community Involvement

Review Panelist for the Department of Energy Early-Career Research Program (ECRP) 2021