Dibyendu Bhattacharya
Sr. Software Development Engineer At Amazon
New York, New York, United States
Summary
👤
Senior🎓
Top SchoolDibyendu Bhattacharya is a senior software engineer with 11+ years building large-scale streaming and data platforms, currently a Senior SDE at Amazon in New York. He holds a Master’s in Software Systems and a B.Tech in Computer Science and has led architecture and delivery for features in Amazon MSK like Consumer Lag and Deep Health Check. His background spans principal and staff engineering roles at Salesforce, AWS, and companies acquired by Akamai, with deep expertise in Kafka, Spark, real-time ETL, and scalable analytics. He has a history of open-source impact—authoring a widely used low-level Kafka consumer for Spark—and has presented at major Big Data conferences. Known for turning research-grade ideas into production services, he combines hands-on implementation with system-level design to improve observability and resilience in streaming systems.
11 years of coding experience
19 years of employment as a software developer
Master's Degree, Software Systems, Master's Degree, Software Systems at BITS PilaniB.Sc, Physics, B.Sc, Physics at Presidency College CalcuttaB.Tech, Computer Science & Engineering, B.Tech, Computer Science & Engineering at University of Calcutta
Github Skills (42)
apache-spark10
spark10
controller10
python10
r10
kafka-connector10
big-data10
kafka10
pid10
hadoop10
spark-streaming10
apache10
jdbc10
analytics10
java10
Programming languages (2)
Github contributions (5)
dibbhatt/kafka-spark-consumer
Sep 2014 - Feb 2022
High Performance Kafka Connector for Spark Streaming.Supports Multi Topic Fetch, Kafka Security. Reliable offset management in Zookeeper. No Data-loss. No dependency on HDFS and WAL. In-built PID rate controller. Support Message Handler . Offset Lag checker.
Contributions:13 commits, 6 PRs, 93 pushes in 7 years 6 months
controllerzookeepercheckerkafka-connectorloss
dibbhatt/tachyon
May 2015 - Jun 2015
Contributions:4 pushes in 1 month
memorydistributed-storagereliablestoragedistributed
Find and Hire Top DevelopersWe’ve analyzed the programming source code of over 60 million software developers on GitHub and scored them by 50,000 skills. Sign-up on Prog,AI to search for software developers.