Tathagata Das
Staff Software Engineer at Databricks
United States
Summary
🤩
Rockstar🎓
Top SchoolTathagata Das is a Staff Software Engineer at Databricks with 15 years of experience building and hardening large-scale streaming and distributed data systems. He is a long-time Apache Spark contributor and Spark PMC member who has driven core streaming features, checkpointing fixes, and API enhancements across high-profile projects including Apache Spark, Mesos integrations, and Delta Lake. His open-source work shows deep backend focus—improving streaming test suites, resilient Twitter/Kafka receivers, RDD checkpointing, and HDFS-compatible log stores—helping make real-time data pipelines more reliable. Trained at IIT Kharagpur and UC Berkeley, he combines rigorous academic foundations with pragmatic engineering that surfaces in both production systems at Databricks and community-facing OSS improvements. An understated strength is his knack for refactoring complex streaming code to simplify APIs while preserving fault tolerance at scale.
14 years of coding experience
Master of Science (M.S.), Master of Science (M.S.) at University of California, BerkeleyBachelor of Technology (B.Tech.) + Master of Technology (M.Tech.), Bachelor of Technology (B.Tech.) + Master of Technology (M.Tech.) at Indian Institute of Technology, Kharagpur
Stackoverflow
1,808reputation
118kreached
22answers
1question
hadoop
top-5%
csv
top-5%
apache-spark
top-5%
Github Skills (36)
apache-spark10
licensing10
checkpoint10
twitter10
apidoc10
testing10
key-management10
hdfs10
data-serialization10
spark-streaming10
hadoop10
java10
checkpointing10
performance-testing10
scala10
Programming languages (8)
Github contributions (5)
delta-io/delta
Apr 2019 - Jul 2022
An open-source storage framework that enables building a Lakehouse architecture with compute engines including Spark, PrestoDB, Flink, Trino, and Hive and APIs
Role in this project:
Back-end & DevOps Engineer
Back-end & DevOps EngineerContributions:7 releases, 792 reviews, 95 commits in 3 years 3 months
Contributions summary:Tathagata's primary focus was on improving the Delta Lake license management and the overall stability of the project. They fixed license headers, ensuring proper license management across various project files and replaced instances of "Databricks Delta" with "Delta Lake". Furthermore, the user implemented a HDFSLogStoreImpl for OSS, which involved ensuring atomic writes and consistent file listing. They also updated the README file.
analyticsprestodbflinkbig-dataspark
mesos/spark
Jun 2011 - Jul 2013
Lightning-fast cluster computing in Java, Scala and Python.
Role in this project:Back-end Developer
Back-end DeveloperContributions:225 commits in 2 years 1 month
Contributions summary:Tathagata implemented core functionality for saving RDDs in Spark Streaming, introducing file saving, and improved data serialization techniques. They also refactored the reduceByKeyAndWindow operation, consolidating multiple functions into a smaller set with optional parameters. Furthermore, they addressed bugs related to checkpointing by modifying various aspects of the RDD and DStream classes, which included changes to handle parent-child RDD dependencies, splits, and ensuring re-processing after master failures.
pythoncluster-computinglightningsparkscala
Find and Hire Top DevelopersWe’ve analyzed the programming source code of over 60 million software developers on GitHub and scored them by 50,000 skills. Sign-up on Prog,AI to search for software developers.