Dawei Hu
Lead Data Engineer
Singapore, Singapore
Summary
👤
Senior🎓
Top SchoolDawei Hu is a Lead Data Engineer based in Singapore with 11 years of experience building high-performance data platforms and web services. He has designed and implemented end-to-end data infrastructure for a bank, leading adoption of an Iceberg-based Lakehouse, DataOps low-code web services, and robust governance/RBAC and schema-evolution workflows. Previously he led large-scale Hadoop and Spark platforms for government projects at EY, owning cluster reliability, performance tuning, ETL and streaming pipelines. Technically hands-on across Rust, Scala, Spark, Kubernetes and distributed databases, he has contributed core RDD operations to a Rust reimplementation of Spark, showing deep interest in performant data processing primitives. Dawei combines systems-level engineering with product sensibility—able to translate regulatory and business requirements into auditable, production-grade data platforms. His original academic background in plant sciences hints at a pragmatic, research-minded approach to complex engineering problems.
11 years of coding experience
10 years of employment as a software developer
Bachelor's degree, Plant Sciences, Bachelor's degree, Plant Sciences at Huazhong Agricultural University
Chinese, English
Github Skills (6)
testing10
data-structures10
rust10
parallel-computing10
data-structure10
algorithms9
Programming languages (11)
Github contributions (5)
rajasekarv/vega
Apr 2020 - May 2020
A new arguably faster implementation of Apache Spark from scratch in Rust
Role in this project:
Back-end Developer
Back-end DeveloperContributions:41 commits, 8 PRs, 29 comments in 22 days
Contributions summary:Dawei primarily contributed to the development of core functionality for a Spark-like system in Rust. Their work focused on implementing and testing essential features such as `range`, `filter`, `max`, `min`, `key_by`, and `random_split` functions for RDDs. The user also introduced and tested the `top()` and `take_ordered()` methods, expanding the functionality of the data processing library. These contributions demonstrate a focus on building the fundamental data processing operations within the system.
rustapachefastersparkscala
ugoa/datafusion
Jul 2024 - Mar 2025
Apache DataFusion SQL Query Engine
Contributions:5 PRs, 72 pushes, 9 branches in 8 months
Find and Hire Top DevelopersWe’ve analyzed the programming source code of over 60 million software developers on GitHub and scored them by 50,000 skills. Sign-up on Prog,AI to search for software developers.