Dan Harris
Principal Software Engineer at The Apache Software Foundation
Charlottesville, Virginia, United States
Summary
🤩
Rockstar🎓
Top SchoolDan Harris is a Principal Software Engineer with a decade of experience building high-performance data systems, currently driving backend and database-focused work at Coralogix from Charlottesville, VA. He is an Apache Software Foundation committer with notable contributions to the Rust implementation of Apache Arrow and the DataFusion query engine, where he’s improved Parquet handling, schema evolution, and IO pruning. Equally comfortable in Rust and Scala, he has expanded codec and migration support in zio-schema and implemented nuanced decoder behaviors for Parquet that reduce IO and improve performance. Dan combines deep systems-level engineering with practical product impact, moving low-level encoding/decoding improvements into production-grade tooling. A math graduate from the University of Virginia, he brings a rigorous, algorithmic mindset to real-world data engineering problems.
10 years of coding experience
16 years of employment as a software developer
Bachelor’s Degree Mathematics, Bachelor’s Degree Mathematics at University of Virginia
Github Skills (26)
algorithm10
algorithms10
zio10
decode10
arrow-keys10
data-structure10
decoding10
schema-migrations10
scala10
fusion10
apache-arrow10
sql10
encoding10
codec10
parquet10
Programming languages (6)
Github contributions (5)
zio/zio-schema
Mar 2021 - Mar 2022
Compositional, type-safe schema definitions, which enable auto-derivation of codecs and migrations.
Role in this project:
Back-end Developer
Back-end DeveloperContributions:2 releases, 126 reviews, 26 commits in 1 year 1 month
Contributions summary:Dan primarily contributed to implementing and refining the `ProtobufCodec`, a core component for encoding and decoding data in the `zio-schema` project. Their work involved adding support for various schema types, including failure schemas, and case classes with arity up to 22, indicating a focus on expanding the codec's capabilities. The commits also include the addition of features for implementing schema migrations, which enhances the project's data evolution capabilities. The user also worked on adding support for the `Schema.Enum` encoding for ADTs, which demonstrates their work on implementing encoding for core data types.
derivationziotype-safeschema-validationdefinitions
apache/datafusion
Dec 2021 - Nov 2022
Apache DataFusion SQL Query Engine
Role in this project:Back-end Developer
Back-end DeveloperContributions:165 reviews, 24 commits, 37 PRs in 11 months
Contributions summary:Dan focused on enhancing the Apache DataFusion SQL Query Engine by implementing features and addressing issues related to query execution and data processing. Their contributions included improvements to ParquetExec, such as handling schema evolution during merging, and implementing a SchemaAdapter trait for adapting file-level record batches. The user also worked on the serialization of pruning predicates in ParquetExec and UDAFs in the physical plan. Furthermore, they contributed to implementing features to support GROUPING SETS/CUBE/ROLLUP in the engine.
querypythonquery-enginedataframerust
Find and Hire Top DevelopersWe’ve analyzed the programming source code of over 60 million software developers on GitHub and scored them by 50,000 skills. Sign-up on Prog,AI to search for software developers.