Lu Wang
Senior Software Engineer at Databricks
San Francisco, California, United States
Summary
🤩
Rockstar🎓
Top SchoolLu Wang is a Senior Software Engineer in San Francisco with 8 years of experience building and leading ML infrastructure and tooling at Databricks, where they tech-led the ML training team and worked on Databricks Runtime for ML, AutoML, and sparkdl. Trained as a computational mathematician (PhD), Lu brings a strong research-to-production pedigree from high-performance scientific computing and GPU-accelerated solvers at national labs to large-scale ML platforms. They are an active open-source contributor to heavyweight projects like Apache Spark and Hyperopt, improving ML APIs, Spark integration, and test reliability across distributed systems. Comfortable across backend, ML, and tooling layers, Lu combines deep numerical expertise with practical engineering—evident in contributions that extended Spark ML features and hardened cross-version compatibility for Spark, Scala, TensorFlow/Keras pipelines.
8 years of coding experience
5 years of employment as a software developer
Doctor of Philosophy (PhD) Computational Mathematics, Doctor of Philosophy (PhD) Computational Mathematics at Penn State UniversityBachelor's degree Computational Mathematics, Bachelor's degree Computational Mathematics at Peking University
English, Chinese
Github Skills (33)
apache-spark10
benchmark10
spark10
pytest10
python10
clustering-algorithm10
r10
testing10
big-data10
machine-learning10
benchmarking10
ml10
scala10
mle10
keras10
Programming languages (6)
Github contributions (5)
databricks/spark-deep-learning
Apr 2018 - Nov 2018
Deep Learning Pipelines for Apache Spark
Role in this project:
ML Engineer
ML EngineerContributions:1 release, 6 reviews, 10 commits in 6 months
Contributions summary:Lu focused on updating and maintaining the `spark-deep-learning` repository, specifically addressing compatibility with Keras and TensorFlow versions. They modified code to ensure compatibility, including updating dependency versions such as tensorframes and spark. Furthermore, they optimized existing code and fixed issues in the testing framework. These updates included changes to various files, demonstrating their involvement in maintaining the core functionality and test infrastructure of the deep learning pipelines.
data-sciencedeep-learningmachine-learningapachespark
databricks/tensorframes
Aug 2018 - Sep 2019
[DEPRECATED] Tensorflow wrapper for DataFrames on Apache Spark
Role in this project:ML Engineer
ML EngineerContributions:1 release, 10 commits, 9 PRs in 1 year 1 month
Contributions summary:Lu's contributions primarily involved updating and maintaining the project's dependencies, particularly those related to TensorFlow and Spark. They updated the TensorFlow version, made modifications to the project's protocol buffer definitions, and addressed a known bug in the TensorFlow Java API, demonstrating a focus on keeping the project compatible with the latest releases of the core machine learning libraries. The user also updated the Spark version and managed the project's versioning, which includes both library versions and project release version, and the addition of the necessary fixes and patches.
dataframesapachemachine-learningsparkapache-spark
Find and Hire Top DevelopersWe’ve analyzed the programming source code of over 60 million software developers on GitHub and scored them by 50,000 skills. Sign-up on Prog,AI to search for software developers.