HackDig : Dig high-quality web security articles for hacker

«No Previous
No Next

Scripting Web Categorization, (Fri, Jan 29th)

2016-01-29 13:50

When you are dealing with a huge amount of data, it can be very useful to enhance them by adding more valuable content. Example:

  • Geolocalization for IP addresses
  • Get an IP address DShield score
  • Lookup domain names in list of malicious domains
  • ...

When you are processingmany URLs during a security incidentinvestigation or while extracting IOCs from a malware sample or logs, it can also be very interesting to categorize them. The process of categorization helps to tag an URL with a label like the classic Adult Content, Government, Forums, etc.Many commercial solutions offer this feature. It can be very powerful to configure your firewall to deny access to non-business categories. But, integrated in closed solutions, its not easy to re-use them to benefit of this information in your own scripts.For years, Bluecoat has a product called K9 that helps to protectkidssurfing the web. Its free, you just can get a license key and install the tool or... use the online API!I had to categorize a bunch of"> $ ./webcat.py isc.sans.orgisc.sans.org,Education

Multiple URLs can be passed on the same command line or the script can be fed via STDIN if you use -"> $ ./webcat.py isc.sans.org blog.rootshell.beisc.sans.edu,Educationblog.rootshell.be,Technology/Internet$ cat suspicious-urls.tmp | ./webcat.py -getmooresuccess.com,Business/Economyweddingme.net,Business/Economyriverbird.usa.cc,Malicious Outbound Data/Botnets1ntershipping.co,Malicious Outbound Data/Botnetssecureemail.bz,Malicious Sources/Malnetsvsreviewsa.com,Malicious Sources/Malnetsfelceconserve.com,Malicious Outbound Data/Botnetsflashsync.cf,Uncategorizedcy-m0ld.com,Malicious Outbound Data/Botnetsberettitdint.ru,Malicious Outbound Data/Botnetsvehanmace.ru,Malicious Outbound Data/Botnetsredderbest.gq,Uncategorizedgooglemails.ga,Uncategorizedmsportf1.com,Sports/Recreationwww.vai-t.com,Malicious Sources/Malnetsduotthenaning.ru,Malicious Sources/Malnetsduotthenaning.ru,Malicious Sources/Malnetslittrecdintoft.ru,Malicious Sources/Malnetsvsreviewsa.com,Malicious Sources/Malnetsdoncglobal.com,Malicious Outbound Data/Botnets

The API returns an hexadecimal code corresponding to the web category. That"> $ ./webcat.py -husage: webcat.py [-h] [-f CACHEFILE] [-F] [URL [URL ...]]Categorize URL using BlueCoat K9positional arguments: URL the URL(s) to check. Format: fqdn[:port]optional arguments: -h, --help show this help message and exit -f CACHEFILE, --file CACHEFILE Categories local cache file (default: /var/tmp/categories.txt) -F, --force force a fetch of categories

Before using the script, you have to register to get your K9 license, add it to the script (line 30).

Note: Im not aware of any rate-limit in place while querying the API. During my investigations,I was never blocked.

Xavier Mertens
ISC Handler - Freelance Security Consultant
PGP Key

(c) SANS Internet Storm Center. https://isc.sans.edu Creative Commons Attribution-Noncommercial 3.0 United States License.


Source: ssr;pma&96602=diyrots?lmth.yraid/ude.snas.csi

Read:3405 | Comments:0 | Tags:No Tag

“Scripting Web Categorization, (Fri, Jan 29th)”0 Comments

Submit A Comment

Name:

Email:

Blog :

Verification Code:

Announce

Share high-quality web security related articles with you:)

Tools

Tag Cloud