crawler

BigPhish - An all-in-one solution to identify and monitor phishing kit deployment

Spremioloon.Com data scraper

Dark Web Search Engine : a dark net crawler and database viewer.

Web crawler for checking the validity of your documents.

Findpapers is an application that helps researchers who are looking for references for their work

Novel Feast is the website where you can read Chinese, Japanese, and Korean Web Novels, Web Comics Light Novels Online Free https://novelfeast.com/novels

Finds webpages containing recipes

Python module to implement API-like functionality for the FurAffinity.net website

HTTP request module customized for crawlers.

Discover apps by different mehtods. Mass download app packages and metadata.

Bilingual TED-TALK, Crawler, Crawling

A data scraping and manipulation library with command line interface with multi-database support for MSSQL, MySQL, PSQL, MariaDB and SQLite. View jsdoc and coverage reports at, https://rdfedor.gitlab.io/datahorde

A Python crawler to gather information about augmented reality applications. (For a research project.)

A Python app for scanning large data sets of URLs for a given signature and storing the results to an ElasticSearch index. Useful applications for CERTs and security researchers, maybe others.

Python tools for ethical hacking

Ethereum Internal Transaction Crawler lets you extract internal transactions from etherscan.io and transform them into CSV format.

A simple email extractor for obfuscated emails.

a GitLab.com network crawler

codes of An Analysis of GitLab's Users and Projects Networks DOI: 10.1109/IST50524.2020.9345844

moved to https://github.com/hadisfr/gitlab_crawler

A crawler that looks like a browser for distant servers.