A fast MoE impl for PyTorch
Role in this project:
ML Engineer Contributions:10 reviews, 15 commits, 7 PRs in 2 months
Contributions summary:Tiago primarily focused on optimizing the `fastmoe` library for PyTorch, a library for implementing Mixture-of-Experts (MoE) models. They made significant contributions by modifying and improving the CUDA kernels, specifically for the MOELinear layer, to incorporate bias calculations directly. The user also introduced new CUDA kernels for column reduction, enhancing performance. Furthermore, they added a test suite for different data types, ensuring cross-compatibility within the kernels.
pytorchmoemixture-of-expertsimplpytorch-lightning
Contributions:8 commits, 6 pushes, 1 branch in 2 years 4 months