Web scraper/parser and spider/crawler
#1
hello,

which Web scraper/parser you recommend to extract links from search engines?

which spider/crawler you recommend to extract all links from domains/sites?

Thank you
Reply
#2
(06-16-2018, 03:27 PM)ipwn Wrote: hello,

which Web scraper/parser you recommend to extract links from search engines?

which spider/crawler you recommend to extract all links from domains/sites?

Thank you

Are you asking for frameworks or libraries to use or you’re not planning on programming?
I’ve never used already made crawlers so I can’t help you there but in case you’re down to some programming I can help you out  Wink
Reply
#3
(06-16-2018, 06:29 PM)enmafia2 Wrote:
(06-16-2018, 03:27 PM)ipwn Wrote: hello,

which Web scraper/parser you recommend to extract links from search engines?

which spider/crawler you recommend to extract all links from domains/sites?

Thank you

Are you asking for frameworks or libraries to use or you’re not planning on programming?
I’ve never used already made crawlers so I can’t help you there but in case you’re down to some programming I can help you out  Wink

im asking for ready tools/scripts to run...

i only know about httrack and lynx.. does anyone know if these 2 accept input of list of domains/sites?

Many thanks!
Reply
#4
(06-16-2018, 08:15 PM)ipwn Wrote:
(06-16-2018, 06:29 PM)enmafia2 Wrote:
(06-16-2018, 03:27 PM)ipwn Wrote: hello,

which Web scraper/parser you recommend to extract links from search engines?

which spider/crawler you recommend to extract all links from domains/sites?

Thank you

Are you asking for frameworks or libraries to use or you’re not planning on programming?
I’ve never used already made crawlers so I can’t help you there but in case you’re down to some programming I can help you out  Wink

im asking for ready tools/scripts to run...

i only know about httrack and lynx.. does anyone know if these 2 accept input of list of domains/sites?

Many thanks!

What exactly are you trying to crawl? https://www.screamingfrog.co.uk/seo-spider/ this is a decent crawler depending on your needs...
Reply
#5
If you want to scrape from Google, I'd suggest using the googler tool. https://github.com/jarun/googler

If you want to spider a website, use Burp suites spider. https://portswigger.net/burp/help/spider_using

Really it depends what you want to do. But you can do a lot of damage with those two tools if you use them properly. Good luck.
Reply


Possibly Related Threads…
Thread Author Replies Views Last Post
  [Tutorial] Request header MySQL injection using netcat and burp suite Insider 0 619 06-16-2020, 02:53 AM
Last Post: Insider
  would this be a good way to start web hacking? QMark 19 7,451 04-04-2020, 06:28 AM
Last Post: QMark
  Basics of website and server hacking Insider 0 1,695 03-26-2020, 09:34 PM
Last Post: Insider
  Re-posted and Updated [Complete MySQL Injection] Insider 5 12,742 04-28-2019, 09:46 PM
Last Post: thunder