Ilya Kreymer
Lead Software Engineer at Webrecorder Software
San Francisco, California, United States
Summary
🤩
Rockstar🎓
Top SchoolIlya Kreymer is a lead software engineer and open-source founder with 14+ years building high-fidelity web archiving systems and infrastructure from the Wayback Machine to Webrecorder and Conifer. Based in San Francisco, he combines backend, full-stack and DevOps expertise to deliver robust crawlers, replay tooling and archival data pipelines used by institutions and individuals worldwide. He founded and leads projects like webrecorder/browsertrix-crawler and pywb, contributing pragmatic fixes for POST handling, multipart support, resilient ARC processing, and scalable deployment patterns including Docker and S3 uploads. Comfortable shipping across code, infrastructure and product, he’s known for improving fault tolerance in large crawlers and simplifying complex archival workflows. An amateur accordionist who once toured playing music, he brings a maker’s curiosity and a user-focused ethos to preserving the web as open, reusable history.
14 years of coding experience
12 years of employment as a software developer
Brown University
Github Skills (48)
puppeteer10
javascript10
web-crawler10
archiving10
rewrite10
webscraper10
python10
http10
archives10
webscraping10
request-handler10
java10
crawling10
webar10
data-storage10
Programming languages (19)
Github contributions (5)
webrecorder/pywb
Dec 2013 - Nov 2022
Core Python Web Archiving Toolkit for replay and recording of web archives
Role in this project:
Back-end Developer
Back-end DeveloperContributions:30 releases, 22 reviews, 2111 commits in 9 years
Contributions summary:Ilya contributed to the core functionality of the pywb toolkit by improving POST request support for URL rewriting, which included enhancements to the webagg input request, including json based post requests and extracting the full payload. They implemented features such as handling multipart/form-data encoding for POST requests and supported different content types. Additionally, they worked on adding support for different index sources (e.g. XMLQueryIndexSource), and adding the ability to configure a proxy's date settings using --proxy-default-timestamp.
pythonreplaypython-webpywbweb-archiving
webrecorder/browsertrix-crawler
Oct 2020 - Jan 2023
Run a high-fidelity browser-based web archiving crawler in a single Docker container
Role in this project:Full-stack Developer & DevOps Engineer
Full-stack Developer & DevOps EngineerContributions:88 releases, 212 reviews, 141 commits in 2 years 3 months
Contributions summary:Ilya primarily refactored the crawler and driver components, wrapping crawling functionality in a `Crawler` class and adding extensibility. They implemented features such as support for headless/non-headless operation, custom drivers, and collection names for `pywb`. Furthermore, the user made several infrastructure and build process improvements by switching to a more modern base image, added support for VNC-based profile creation, and implemented code to upload to S3. The user also introduced new command-line flags and improved the internal project logging.
high-fidelitybrowserweb-crawlerdockerdocker-container
Find and Hire Top DevelopersWe’ve analyzed the programming source code of over 60 million software developers on GitHub and scored them by 50,000 skills. Sign-up on Prog,AI to search for software developers.