Cheng Lian
Staff Software Engineer at The Apache Software Foundation
San Francisco, California, United States
Summary
👤
Senior🎓
Top SchoolCheng Lian is a Staff Software Engineer at Databricks with 16 years of experience building high-performance distributed systems and query engines, and a long track record of shaping Apache Spark as a committer and PMC member. He led engineering for Databricks’ Unity Catalog, bringing production-grade data governance to lakehouses, and previously helped build Spark SQL from scratch and optimize Databricks Runtime. A substantive open-source contributor, Cheng has fixed deep bugs and added features in cornerstone projects like Apache Parquet, spark-avro, and Spark’s SQL engine, demonstrating strong expertise in data formats, predicate pushdown, and executor-side serialization. Based in San Francisco, he combines hands-on backend engineering with program-level leadership, and his background includes building debugging tools for distributed Spark jobs—an oft-overlooked capability that reflects his focus on observability and developer productivity.
16 years of coding experience
17 years of employment as a software developer
Bachelor Computer Science, Bachelor Computer Science at Zhejiang University
English, Chinese
Stackoverflow
663reputation
60kreached
7answers
1question
window-functions
top-5%
Github Skills (21)
avro10
apache-spark10
back-end-development10
data-engineering10
hadoop10
java10
scala10
javas10
orc10
parquet10
partitioning9
window-functions9
data-serialization8
serialization8
build-automation7
Programming languages (24)
Github contributions (5)
databricks/spark-avro
Aug 2015 - Feb 2017
Avro Data Source for Apache Spark
Role in this project:
Back-end Developer & Data Engineer
Back-end Developer & Data EngineerContributions:11 commits, 12 PRs, 7 pushes in 1 year 5 months
Contributions summary:Cheng primarily contributed to the development of the `spark-avro` data source, focusing on enhancements to Avro data handling within Apache Spark. They added partitioning support, addressed UUID settings for write jobs, and ported the library to Spark 2.0 using the file source API. Key contributions also include respecting the configuration setting for ignoring files without extensions and fixing a serialization issue in the configuration object used on the executors.
avroavro-datadata-sourceapachebig-data
apache/spark
Nov 2014 - Jul 2018
Apache Spark - A unified analytics engine for large-scale data processing
Role in this project:Back-end Developer
Back-end DeveloperContributions:8 reviews, 11 commits, 366 PRs in 3 years 8 months
Contributions summary:Cheng primarily focused on enhancing the functionality and performance of Apache Spark's SQL engine, specifically within the context of data sources. Their contributions included simplifying code using compact case class pattern matching, improving ORC predicate push-down, and implementing methods for efficient data reading, writing, and handling, particularly for handling tables with a data source. Furthermore, the user was involved in making improvements to schema management and in providing a more robust test environment.
analyticspythondata-processingsqlapache
Find and Hire Top DevelopersWe’ve analyzed the programming source code of over 60 million software developers on GitHub and scored them by 50,000 skills. Sign-up on Prog,AI to search for software developers.