Adam Miller
Senior Crawl Engineer at Internet Archive
Spokane-Coeur d'Alene Area United States
Summary
🤩
RockstarAdam Miller is a Senior Crawl Engineer with 14 years of experience building and improving large-scale web crawling systems at the Internet Archive. Based in the Spokane–Coeur d'Alene area, he specializes in backend engineering for browser-driven and archival crawlers, contributing meaningful fixes and feature work to prominent open-source projects like brozzler and Heritrix3. His work has restructured core crawler behavior, added cookie and hop-path handling, and fixed nuanced REST/API and XML issues that improved job management and data presentation. He combines deep systems knowledge with practical maintenance sensibilities, routinely touching browser orchestration, frontier logic, and data model concerns. Notably, his contributions have directly enhanced the Internet Archive’s ability to capture complex, real-world web interactions. Colleagues rely on him for durable, production-focused solutions that keep large-scale archiving robust and reproducible.
14 years of coding experience
Github Skills (25)
web-crawler10
python10
webscraper10
webscraping10
java10
browser-automation10
crawling10
javas10
heritrix10
crawler10
rethinkdb-go9
freemarker9
restful-api9
api-design9
api-rest9
Programming languages (6)
Github contributions (5)
internetarchive/heritrix3
Jul 2012 - Jul 2021
Heritrix is the Internet Archive's open-source, extensible, web-scale, archival-quality web crawler project.
Role in this project:
Back-end Developer
Back-end DeveloperContributions:7 reviews, 111 commits, 76 PRs in 9 years 2 months
Contributions summary:Adam contributed to bug fixes and improvements within the Heritrix3 web crawler project, specifically focusing on the REST API for job management and bean browsing. The contributions addressed issues related to incorrect URLs in XML representations, conversion to Freemarker templates for the engine resource, and improvements to the display of CrawlJob and Engine information. The user's work involved modifying Java code and refactoring the codebase, which improved the overall functionality of the web crawler's user interface and data presentation.
heritrixweb-crawlerjavawebcrawlingwarc
internetarchive/brozzler
Oct 2015 - Apr 2022
brozzler - distributed browser-based web crawler
Role in this project:Back-end Developer
Back-end DeveloperContributions:46 reviews, 21 commits, 33 PRs in 6 years 7 months
Contributions summary:Adam primarily contributed to the backend logic of the brozzler web crawler, focusing on core functionality. They made significant changes to the `browser.py`, `worker.py`, `model.py`, and `frontier.py` files. These changes included restructuring browser behavior, adding support for cookie management, implementing hop path information, and fixing port search logic, enhancing the crawler's overall performance and feature set.
web-crawler
Find and Hire Top DevelopersWe’ve analyzed the programming source code of over 60 million software developers on GitHub and scored them by 50,000 skills. Sign-up on Prog,AI to search for software developers.