Digital Dumpster Diving - Searching for Value in Scraped Paste Site Data

In this talk, I will introduce the concepts of web scraping, how they apply to paste sites, and specific tools that I have used to gather data, categorize it, and search it. I will also walk through the setup and usage of PasteHunter, a publicly available Python tool that makes scraping and storing paste site data easy. Lastly, I'll walk through the setup of a basic ELK cluster using Docker, and demo how it can be used to index and search through a variety of datasets expanding beyond just paste data into publicly available datasets. I will finish with some overall thoughts on how many "big data" oriented tools can be used in the context of offensive and defense cybersecurity.