Shiyan Xu
Founding Team Member at Onehouse
San Francisco, California, United States
Summary
🤩
Rockstar🎓
Top SchoolShiyan Xu is an engineering leader with 11 years of experience building cloud-native data platforms, machine learning infrastructure, and mobile/backend systems, now based in San Francisco. He led core Apache Hudi engineering at Onehouse and at Zendesk, driving exabyte-scale lakehouse capabilities like CDC, multi-modal indexing, and low-latency streaming pipelines. An Apache Hudi PMC member and committer, he contributed backend features such as efficient bulk-insert and partition-path updates to the widely used Hudi project and authored a definitive guide on the technology. At Applied Intuition he continues to apply his systems expertise to production-critical platforms while mentoring contributors into sustained open-source committers. His background blends hands-on backend Java work, Rust-native implementations (Hudi-rs), and practical ML serving experiments, reflecting a rare combination of production-scale engineering and open-source community building.
11 years of coding experience
7 years of employment as a software developer
University of California, Los AngelesBachelor of Engineering (B.Eng.) Electrical and Electronics Engineering, Bachelor of Engineering (B.Eng.) Electrical and Electronics Engineering at Nanyang Technological University Singapore
English, Chinese
Stackoverflow
1,172reputation
552kreached
16answers
6questions
Github Skills (17)
apache-hudi10
incremental-processing10
data-engineering10
big-data10
java10
javas10
data-lake10
hdfs9
parquet9
swift6
virtualenv6
python6
django6
emr6
apache-spark6
Programming languages (11)
Github contributions (5)
apache/hudi
Nov 2019 - Jan 2023
Upserts, Deletes And Incremental Processing on Big Data.
Role in this project:
Data Engineer
Data EngineerContributions:1 release, 1636 reviews, 273 commits in 3 years 3 months
Contributions summary:Shiyan contributed to the Apache Hudi project, which focuses on building data lakes with incremental processing capabilities. They focused on implementing the "bulkinsert" feature within the HDFSParquetImporter for efficiently importing data, as well as adding support to update partition paths. Their contributions involved code modifications within core Java files, indicating backend development. Also worked on improvements to the testing aspects of the projects, for example, improving the Exporter test.
stream-processingbigdatahudiincrementalapacheflink
apache/hudi-rs
May 2024 - Apr 2025
The native Rust implementation for Apache Hudi, with Python API bindings.
Contributions:3 releases, 100 reviews, 295 PRs in 11 months
apachehudipythonrust
Find and Hire Top DevelopersWe’ve analyzed the programming source code of over 60 million software developers on GitHub and scored them by 50,000 skills. Sign-up on Prog,AI to search for software developers.