RTTM - Real Time Threat Monitoring Tool V2.0

Monitoring possible threats of your company on Internet is an impossible task to be achieved manually. Hence many threats of the company goes unnoticed until it becomes viral in public. Thus causing monetary/reputation damage. This is where RTTM comes into action. RTTM (Real Time Threat Monitoring Tool) is a tool developed to scrap all pasties,github,reddit..etc in real time to identify occurrence of search terms configured. Upon match an email will be triggered. Thus allowing company to react in case of leakage of code, any hacks tweeted..etc.. and harden themselves against an attack before it goes viral.

Over the past 2 years the tool has evolved from simple search. Artificial intelligence has been implemented to perform better search based on context. If regex is needed even that is supported. Thus behaviour is close to human and reduces false positives.

The best part of tool is that alert will be sent to email in less that 60 seconds from the time threat has made it to interent. Thus allowing response in real time to happen..

The same tool in malicious user hands can be used offensively to get update on any latest hacks, code leakage etc..

List of sites which will be monitored are:

  • Non-Pastie Sites
    • Twitter
    • Reddit
    • Github
  • Pastie Sites
    • Pastebin.com
    • Codepad.org
    • Dumpz.org
    • Snipplr.com
    • Paste.org.ru
    • Gist.github.com
    • Pastebin.ca
    • Kpaste.net
    • Slexy.org
    • Ideone.com
    • Pastebin.fr

Architecture:

How it works?

Once the tool is started , engine gets kicked off and it runs forever. The main input for this engine is the configuration file. Based on the configuration file data, engine goes ahead and probes twitter/github/reddit for matches configured in configuration file. Upon a match is found, the link of twitter/github/reddit pushed to sqlite DB and an email alert is triggered.

In case of pastie sites the logic is different. The reason being they do not support search nor streaming api's. Hence any new pastie made by any user, the link is fetched and pushed to kafka. From kafka any new link added is picked up and searched for matches configured in configuration file. Upon a match is found, the link of pastie site is pushed to sqlite DB and an email alert is triggered.

Over the past 2 years the tool has evolved from simple search. Artificial intelligence has been implemented to perform better search based on context. If regex is needed even that is supported. Thus behaviour is close to human and reduces false positives.

Configuration

Before using this tool is is neccessary to understand the properties file present in scrapper_config directory.

consumer.properties

Holds all the necessary config data needed for consumer of Kafka (Refer apache Kafka guide for more information). The values present here are default options and does nto require any changes

producer.properties

Holds all the necessary config data needed for Producer (Refer apache Kafka guide for more information).The values present here are default options and does nto require any changes

email.properties

Holds all the configuration data to send email.

scanner-configuration.properties

This is the core configuration file. Update all the config for enabling search on twitter/github(To get tokens and key refer respective sites).

For pastie sites and reddit there is no need for any changes in config.

Note:However in all cases make sure to change “searchterms” to values of our choice to search. If there are multiple search terms then add them seperate by comma like the example data provided in config file.

Understanding more about scanner-configuration.properties file.

For any pastie site configuration is as below:

Note: leave the pastie sites configuration as is and just change the search terms as requried by the organization. This will do good.

  • scrapper.(pastie name).profile=(Pastie profile name)
  • scrapper.(pastie name).homeurl=(URL from where pastie ids a extracted)
  • scrapper.(pastie name).regex=(Regex to fetch pastie ids)
  • scrapper.(pastie name).downloadurl= (URL to get information about each apstie)
  • scrapper.(pastie name).searchterms=(Mention terms to be searched seperated by comma)
  • scrapper.(pastie name).timetosleep=(Time for which pastie thread will sleep before fetching pastie ids again)

For github search configuration is as below:

  • scrapper.github.profile=Github
  • scrapper.github.baseurl=https://api.github.com/search/code?q={searchTerm}&sort=indexed&order=asc
  • scrapper.github.access_token=(Get your own github access token)
  • scrapper.github.searchterms=(Mention terms to be searched seperated by comma)
  • scrapper.github.timetosleep=(Time for which github thred should sleep before searching again)

For reditt search configuration is as below: * scrapper.reddit.profile=Reddit

  • scrapper.reddit.baseurl=https://www.reddit.com/search.json?q={searchterm}
  • scrapper.reddit.searchterms=(Mention terms to be searched seperated by comma)
  • scrapper.reddit.timetosleep=(Time for which github thred should sleep before searching again)

For Twitter search configuration is as below: * scrapper.twitter.apikey=test

  • scrapper.twitter.profile=Twitter
  • scrapper.twitter.searchterms=(Mention terms to be searched seperated by comma)
  • scrapper.twitter.consumerKey=(Get your own consumer key)
  • scrapper.twitter.consumerSecret=(Get your own consumerSecret)
  • scrapper.twitter.accessToken=(Get your own accessToken)
  • scrapper.twitter.accessTokenSecret=(Get your own accessTokenSecret)

API Key Setup

Twitter API Key:

  • Go to https://dev.twitter.com/apps/new and log in, if necessary
  • Enter your Application Name, Description and your website address. You can leave the callback URL empty.
  • Accept the TOS, and solve the CAPTCHA.
  • Submit the form by clicking the Create your Twitter Application
  • Copy the consumer key (API key) and consumer secret from the screen into your application

Pastebin.com

In case of pastebin.com it is not neccessary to generate apiKey but you need to whitelist your IP. For thsi you have to pay and get your IP whitelisted.

Refer https://pastebin.com/doc_scraping_api for detailed information.

SetupGuide

Install via Docker

  • Install docker in your system
  • Download Dockerfile from https://github.com/NaveenRudra/RTTM
  • Change directory to RTTM
  • execute docker build .
  • Run docker exec -it <container id> /bin/bash
  • Now once in docker navigate to /opt/RTTM/script
  • Run intialize.sh script. This will boot mysql server and starts kafka.
  • Run db_setup.sh this will created needed table.
  • Now from /opt/RTTM run command java -jar scraptool/target/scraptool-1.0-SNAPSHOT-standalone.jar -t test -c /home/n0r00ij/RTS/scrapper_config/

Detailed Tool Documentation:

https://real-time-threat-monitoring.readthedocs.io/en/latest/

Developers:

Authors:

  • Naveen Rudrappa

Contributors:

  • Sunny Sharma
  • Murali Segu

Website: https://github.com/NaveenRudra/RTTM

December 16, 2019

Leave a Reply

avatar

This site uses Akismet to reduce spam. Learn how your comment data is processed.

  Subscribe  
Notify of
© HAKIN9 MEDIA SP. Z O.O. SP. K. 2013

Privacy Preference Center

Necessary

Cookies that are necessary for the site to function properly. This includes, storing the user's cookie consent state for the current domain, managing users carts to using the content network, Cloudflare, to identify trusted web traffic. See full Cookies declaration

gdpr, PYPF, woocommerce_cart_hash, woocommerce_items_in_cart, _wp_wocommerce_session, __cfduid [x2]

Marketing


tr, fr
ads/ga-audiences