Mostofa Patwary
Director Of Large Foundational Language Model, Applied Deep Learning Research
South San Francisco, California, United States
Summary
👤
Senior🎓
Top SchoolMostofa Patwary is a technology leader and applied deep learning researcher with 8+ years driving foundational LLM pretraining and high-performance ML systems at NVIDIA. He led Megatron-LM and the Nemotron series (including Nemotron-4 15B trained on 8 trillion tokens) and contributed to landmark projects such as Megatron-Turing NLG 530B, StarCoder 2 and The Stack v2. His expertise spans scalable model-parallel training, dataset curation (Nemo Data Curator), and HPC-grade algorithm engineering that has run on 100K+ cores and terabyte-scale data. An active open-source contributor, he’s implemented optimizer and evaluation fixes in NVIDIA/Megatron-LM and blends systems-level C/C++/MPI optimization with cutting-edge NLP research. He combines rigorous academic training (PhD) and supercomputing experience with practical tooling and dataset innovations that accelerate large-model development.
8 years of coding experience
13 years of employment as a software developer
Doctor of Philosophy (PhD) Computer Science, Doctor of Philosophy (PhD) Computer Science at University of Bergen (UiB)Master's degree Computer Science and Engineering, Master's degree Computer Science and Engineering at Bangladesh University of Engineering and Technology
Stackoverflow
11reputation
244reached
1answer
1question
Github Skills (9)
transformers10
pytorch10
machine-learning10
large-language-models10
optimization9
nlp8
natural-language-processing8
cross-entropy6
softmax6
Programming languages (4)
Github contributions (5)
NVIDIA/Megatron-LM
Oct 2020 - Dec 2021
Ongoing research training transformer models at scale
Role in this project:
ML Engineer
ML EngineerContributions:85 commits, 1 comment in 1 year 2 months
Contributions summary:Mostofa primarily contributed to the integration of Adam optimizer parameters, including betas and epsilon values, into the training process. They fixed an evaluation issue in the wiki dataset. Additionally, the user made modifications to the biencoder model related to the Inverse Cloze Task (ICT), including adding features for score scaling and reporting top-k accuracies, indicating their involvement in the development and evaluation of the model. The user's work also includes refactoring the code by clearing out the commented-out parts.
pytorchnlplanguage-modeltransformer-modelsbert
mpatwary/DeepBench
Nov 2017 - Nov 2017
Contributions:2 pushes in 1 day
benchmarkingdeep-learningmachine-learningbenchmarkhardware
Find and Hire Top DevelopersWe’ve analyzed the programming source code of over 60 million software developers on GitHub and scored them by 50,000 skills. Sign-up on Prog,AI to search for software developers.