Tools #

Python #

• Beautiful soup (Python)
• Mechanize (Python)
• Twill (Python)
• http://github.com/petewarden/pyparallelcurl - A simple Python class for running multiple URL fetches in parallel
• pattern - "It bundles tools for data retrieval (Google + Twitter + Wikipedia API, web spider, HTML DOM parser), text analysis (rule-based shallow parser, WordNet interface, syntactical + semantical n-gram search algorithm, tf-idf + cosine similarity + LSA metrics) and data visualization (graph networks)."
• http://scrapy.org/
• lazynlp: Library to scrape and clean web pages to create massive datasets

Ruby #

• Nokogiri
• Anemone

Web #

• http://import.io

Browser plugins #

• imacros
• Selenium (software)

Etc. #

• http://sikuli.org

Tutorials #

• 국회 사이트, 국회의원 목록 크롤링 - how to use the developer tool to figure out asynchronous requests and crawl a webpage without using tools like Selenium.

• Automate the boring stuff with Python: Web Scraping

Articles #

• The Perils of Web Crawling
• http://www.bytemining.com/2011/02/web-mining-pitfalls/
• http://www.propublica.org/nerds/item/doc-dollars-guides-collecting-the-data
• http://vancouverdata.blogspot.com/2011/02/how-to-web-scraping-xpath-html-google.html
• Web scraping 101 with python
• How to Crawl the Web Politely with Scrapy

Incoming Links #

Related Articles (Article 0) #

• Python/Modules
• Regular expression
• Selenium (software)

Suggested Pages #

• 0.686 Regex
• 0.023 RPython
• 0.022 Python/Test
• 0.021 Python/Beautiful soup
• 0.019 PyPy
• 0.019 Scientific computing
• 0.019 Python/Packaging
• 0.018 CPython
• 0.015 Progress bar
• 0.014 Python/Debugging
• More suggestions...