Hakrawler - A fast CLI web crawler for hackers

(1,693 views)

Fast golang web crawler for gathering URLs and JavaScript file locations. This is basically a simple implementation of the awesome Gocolly library. Example usages Single URL: echo https://google.com | hakrawler Multiple URLs: cat urls.txt | hakrawler Timeout for each line of stdin after 5 seconds: cat urls.txt | hakrawler -timeout 5 Send all requests through a proxy: cat urls.txt | hakrawler -proxy https://localhost:8080 Include subdomains: echo https://google.com | hakrawler -subs Note: a common issue is that the tool returns no URLs. This usually happens when a domain is specified (https://example.com), but it redirects to a subdomain (https://www.example.com). The subdomain is not included in the scope, so the no URLs are printed. In order to overcome this, either specify the final URL in the redirect chain or use the -subs option to include subdomains. Example toolchain Get all subdomains of google, find the ones that respond to http(s), and crawl them....

August 10, 2022

Author

Hakin9 TEAM
Hakin9 is a monthly magazine dedicated to hacking and cybersecurity. In every edition, we try to focus on different approaches to show various techniques - defensive and offensive. This knowledge will help you understand how most popular attacks are performed and how to protect your data from them. Our tutorials, case studies and online courses will prepare you for the upcoming, potential threats in the cyber security world. We collaborate with many individuals and universities and public institutions, but also with companies such as Xento Systems, CATO Networks, EY, CIPHER Intelligence LAB, redBorder, TSG, and others.
Subscribe
Notify of
guest
1 Comment
Newest
Oldest Most Voted
Inline Feedbacks
View all comments
Lizzy Agnes
5 days ago

A great hacker is really worthy of good recommendation , Henry
really help to get all the evidence i needed against my husband and
and i was able to confront him with this details from this great hacker
to get an amazing service done with the help ,he is good with what he does and the charges are affordable, I think all I owe him is publicity for a great work done via, Henryclarkethicalhacker at gmail com, and you can text, call him on whatsapp him on +12014305865, or +17736092741, 

© HAKIN9 MEDIA SP. Z O.O. SP. K. 2023