Kevin Stephano
Engineering Manager Of PyTorch Compilers at NVIDIA
San Francisco, California, United States
Summary
🤩
Rockstar🎓
Top SchoolKevin Stephano is an engineering leader and compiler expert with six years focused on PyTorch compilers and a long track record at NVIDIA driving high-performance deep learning infrastructure. As Engineering Manager of PyTorch Compilers and former lead of the nvFuser team, he blends hands-on C++/CUDA implementation skills with team leadership to deliver fusion, kernel, and optimizer improvements that materially accelerate training workloads. His open-source contributions include significant work in pytorch/pytorch and NVIDIA/apex—rewriting multihead attention for bandwidth efficiency and adding NVFuser caching and primitives—work that underpins MLPerf-scale training. Kevin’s background in GPU microarchitecture and performance modeling dating back to IBM and AMD gives him a rare systems-level view that informs pragmatic compiler and kernel design. He’s based in San Francisco and known for removing unnecessary data movement (novel weight layouts and plumbing) to cut CPU/GPU overhead in production transformer training.
6 years of coding experience
21 years of employment as a software developer
Other, Data Science Course, Other, Data Science Course at General AssemblyMS, Electrical and Computer Engineering, MS, Electrical and Computer Engineering at University of Illinois Urbana-Champaign
English
Github Skills (14)
cuda10
pytorch10
machine-learning10
c-language10
deep-learning10
cprogramming-language10
python10
gpu9
neural-network9
cublas9
tensor8
autograd7
numpy6
tensorflow3
Programming languages (2)
Github contributions (5)
NVIDIA/apex
Feb 2020 - Jun 2022
A PyTorch Extension: Tools for easy mixed precision and distributed training in Pytorch
Role in this project:
ML Engineer
ML EngineerContributions:6 reviews, 6 commits, 6 PRs in 2 years 4 months
Contributions summary:Kevin significantly contributed to the `nvidia/apex` repository, focusing on enhancing and optimizing multihead attention mechanisms for PyTorch. They implemented a C++ multihead attention implementation within the contrib module, and created several python versions of attention models, which indicates a significant amount of work in the domain of deep learning. Furthermore, the user improved the performance of existing kernels by updating to the current CUDA Stream and worked on integrating the Fused Lamb optimizer. The modifications include both forward and backward passes, suggesting a focus on both model functionality and training efficiency within the context of deep learning frameworks.
pytorchraymixed-precisiondeep-learningtemporal-data
pytorch/pytorch
Sep 2020 - Nov 2022
Tensors and Dynamic neural networks in Python with strong GPU acceleration
Role in this project:ML Engineer
ML EngineerContributions:87 reviews, 44 commits, 19 PRs in 2 years 2 months
Contributions summary:Kevin primarily contributes to the NVFuser Python frontend within the PyTorch repository, a key component for accelerating deep learning computations. Their work focuses on enhancing the NVFuser framework, including implementing caching mechanisms for fusion reuse and improving batch normalization functionality. These changes involve modifications to the C++ and Python bindings for NVFuser, specifically adding support for new primitives like `rand_like` and improving code organization and printing of function definitions. The impact of their work is aimed at improving performance and usability.
pythongpu-accelerationdeep-learninggpunumpy
Find and Hire Top DevelopersWe’ve analyzed the programming source code of over 60 million software developers on GitHub and scored them by 50,000 skills. Sign-up on Prog,AI to search for software developers.