GreySec Scraper Tool Release
#1
Hello internet users. This is a public service announcement. The greysec scraper previously mentioned in this subforum is functional and it has gathered some data on around 2,000 of the 8,000ish greysec users. You can get the tool and see some of the data collected in its' github repo: https://github.com/ghostwalkr/GreySec-Statistics. Keep in mind though, this thing is in its' infancy. It will have bugs and I could use some help with it. I'll put a todo at the bottom of this thread. Send those pull requests. It's pretty simple to use. Just run it on any system with python3 installed like so (edit: also need to put your cookie into the code):
Code:
python3 scraper.py

 

It'll collect the username, UID, post count, and thread count of every user on GreySec. As I write this the scraper is collecting the data for all 8,000ish users. It'll probaby take between 1-3 hours to run. Feel free to play around with the code or data, it's open source software after all.

 
## EDIT ######
Tool updated on June 6, 2020
If you have any questions let me know below.
 

Todo

- Make it threaded so it runs faster.

- Better logging
Reply
#2
Hey there, as I mentioned before I was able to write up a bot that scrapes forum posts, were you interested in me adding my code to your project? I can submit a pull request once I've got it setup to work with what you've got :-)
Reply
#3
(06-01-2020, 03:53 PM)EpochRoot Wrote: Hey there, as I mentioned before I was able to write up a bot that scrapes forum posts, were you interested in me adding my code to your project? I can submit a pull request once I've got it setup to work with what you've got :-)
 
Absolutely! My scraper has a lot of work to be put into it. I'll probably have to rewrite it to work with threading a little better. But if you want to put some of your code in so that posts could be scraped that would be great.
Reply
#4
As I said in Discord, very cool project, I always loved scrapers.
I would really want to help with this development, but being honest I am busy with IRL lately.

Good luck sir I will be lurking as always.
Reply
#5
Cool project! I'm going to have a look and see if I can use this for thread statistics or maybe even archive threads!
Good work.
Reply


Possibly Related Threads…
Thread Author Replies Views Last Post
  GreySec Social thread. Vector 163 114,561 06-11-2020, 11:25 PM
Last Post: Insider
  How did you find GreySec? Insider 8 1,073 06-11-2020, 11:11 PM
Last Post: Insider
  GreySec Forum Statistics (Full Stats) Dismal_0x8 7 933 06-02-2020, 11:15 AM
Last Post: Insider
  Project: GreySec Data Scraper Program Dismal_0x8 14 2,325 05-17-2020, 02:36 AM
Last Post: Dismal_0x8