Python Web scraping
- Merit Data
- 4 - 8 years
- Chennai
- 18 hours ago
- Email to a friend
- Report this job
Job Description
Mid-Senior Python Web Scraping Engineer
49 years
Job Summary
We are looking for a highly skilled Python Web Scraping Engineer to design, build, and maintain scalable, resilient web data extraction systems. The role requires deep hands-on experience with Scrapy-based architectures, advanced crawling strategies, and handling real-world challenges such as dynamic websites, anti-bot mechanisms, and frequent site changes.
The engineer will work closely with data engineering, ML, and product teams to deliver high-quality, structured data pipelines at scale.
Key Responsibilities
- Design, develop, and maintain large-scale web scraping systems using Python
- Build robust Scrapy spiders with custom middlewares, pipelines, and extensions
- Implement scraping using Scrapy, Requests/HTTPX, Selenium, and Playwright
- Handle anti-bot, CAPTCHA, geo-blocking, proxy rotation, and dynamic content
- Optimize performance, concurrency, retries, and crawl scheduling
- Ensure data quality through validation, logging, and monitoring
- Collaborate with Data Engineering and ML teams for downstream integration
Required Technical Skills
- Strong proficiency in Python
- Deep hands-on experience with Scrapy framework
- Experience with Requests / HTTPX
- Practical experience with Selenium and/or Playwright
- Strong understanding of HTML, DOM, XPath, CSS selectors
- Experience with large-scale scraping workloads
Nice to Have
- Apache Airflow
- Apache Spark
- Message Queues (Redis, SQS, Kafka)
- Cloud platforms (AWS/GCP/Azure)
- Docker, Kubernetes
- ML/LLM-assisted extraction
Soft Skills
- Strong problem-solving and debugging skills
- Ability to work independently
- Clear communication and ownership mindset
Pls share resumes to 8667517***
Job Classification
Industry: IT Services & ConsultingFunctional Area / Department: Engineering - Software & QARole Category: Software DevelopmentRole: Back End DeveloperEmployement Type: Full timeContact Details:
Company: Merit DataLocation(s): Chennai
Keyskills: Scrapy Docker Data Scraping Webscraping Kubernetes Redis Selenium
Similar positions
Merit Data
We build DATA + CODE for some of the worlds leading B2B brands. Through pioneering marketing data, product data,customised software and marketing automation, Merit helps brands of all sizes confidently embrace the challenges ofthe future.
Job Listings
We are in open beta version of website, please let us know if any issues, at: info(at)indigojobs.in