FILE_DATA_PORTS error in Snort, SOLVED

I'm basically putting this post up for Google to index it and maybe it'll help some people solve the problem in the future.

PortVar Lookup failed on '$FILE_DATA_PORTS'

If you came to this blog post by searching for the above error, or if you have the above error in Snort, you should read this post on the VRT blog that we wrote.  It'll help solve your problem:

http://vrt-blog.snort.org/2011/11/say-hello-to-file-identify-category.html

There. Check that out.

Let's just assume this pcap is bad...mkay?

Alerts (2.9.1.1, 4924362.pcap)
1:18347:3 BLACKLIST USER-AGENT known malicious user-agent string AutoIt Alerts: 4
1:19734:1 BLACKLIST DNS request for known malware domain 770304123.cn Alerts: 2
1:16816:5 BOTNET-CNC known command and control channel traffic Alerts: 1
1:18762:1 BLACKLIST URI request for known malicious URI /blog.updata?v= - Win32-Agent-GRW Alerts: 1
1:17834:3 BLACKLIST DNS request for known malware domain 343.boolans.com Alerts: 1
120:3:1 (http_inspect) NO CONTENT-LENGTH OR TRANSFER-ENCODING IN HTTP RESPONSE Alerts: 3
1:16815:4 BOTNET-CNC known command and control channel traffic Alerts: 1

---

Please leave comments below.

What have I been up to?

Well, as promised, I haven't written a post in awhile.  I've been really really busy, so I'll give you a crash course on what I've been doing that's kept me, and thoughts about things that have come to market in the past month or so.

1)  My Mother passed away.  As anyone who has had this happen knows, it's a pretty hard time, emotionally, as well as just, all the stuff you have to do.  Writing your own mother's obituary isn't necessarily a good time.  Selecting what she's going to wear, the casket, ...  just a lot.  The people that have written me and talked to me face to face have been great and I thank you all very much.

2)  VRT.  April 1st I moved to the Vulnerability Research Team at Sourcefire.  I'm one of the many other analysts responsible for writing Snort (and now recently ClamAV!) rules to detect the known, and the unknown.  It's a difficult job, it's challenging, it's fun, and it's busy.  I currently have over 100 bugs in my cue.  Lots of bugs and research to do.  My current focus is Malware and some redesign efforts.  We're trying to make the Snort rules easier to manage and provide more intelligence to the end user as well as increase our coverage in a lot of areas.  Making our rules harder to bypass and more and more adaptable to today's client-based landscape.  Over the rest of 2011, the VRT ruleset is going to change, for the better, and significantly.  There's essentially going to be three steps to this, and I'll post about the changes soon over on the Snort Blog.

3)  Snort Community.  It's growing.  When I took over the job in October of last year, I thought the Snort community had reached critical mass.  Most open source projects that I've seen plateau after awhile.  When I was running the BASE project we got up to about 15k downloads a day, and that was our plateau.  But since I took over, I've started to keep a lot of metrics.  Metrics about email postings, forum postings, users, downloads, etc.  Lots of metrics.  They are all going up.  We're doing well.

4)  Snort.  It's changing and evolving.  We're rolling out 2.9.1 soon with some very significant changes (read about PAF!) in detection and the IP reputation preprocessor.  The changes we have planned for post 2.9.1 make Snort even faster (we are already hitting WAY over 20 G/sec in detection, and the next number we are aiming for is unheard of in our industry), and easier to deploy.  Changes in it's detection will make it more accurate and significantly increase the effectiveness of our rules and keywords.

5)  ClamAV.  Also growing.  Now built into Immunet 3.0 (the company we acquired in December of last year) providing not only cloud based detection (so awesome), and offline detection.  Immunet is growing very fast by the looks of our daily metrics which means ClamAV use is increasing as well.  OEM solutions that are building ClamAV are also growing, and now recently we are going to start accepting community virus detection as well.  This will grow our detection rate exponentially.

6)  OSX Lion.  It's out.  I'm using it (have been for about a month and a half).  It works great.  The only thing I don't like about it is the deletion of the scroll bar.  I don't mind it as much as my wife will (I haven't converted her yet).

7)  Defcon.  We'll (VRT) be there.  Look for us in pink.  For those of you that were able to get an invite to TheBarCon, we'll see you there.

I can't think of anything more right now, and am being summoned for dinner.  I'll write more when I have a chance. If you have any questions, leave a comment below.

---

Please leave comments below.

Time to move on to a new job.

For the past, almost 6 years, my employment at Sourcefire has been great.  I've worked in Professional Services, going around to over 150 customers, educating them on Sourcefire, the product, the GUI, detection, and all the awesome that is, but..... it's time for me to move on.

I accepted a position last October with Sourcefire, being in charge of OpenSource Community Management, in charge of coordinating the communities themselves and being the liaison between the communities and the company (Sourcefire) for all OpenSource products.  This has been great.

Well, I'm pleased to announce that, I am retaining this new role as I go, so I will still be the OpenSource Community Manager for Sourcefire.  

Where am I moving to?  A company you may have heard of:

Sourcefire.

That's right, I'm staying right here.  Effective today, I'm moving out of Sourcefire's Professional Services team and moving to the Vulnerability Research Team (VRT), the team at Sourcefire that is responsible for publishing detection for Snort, ClamAV, and Razorback.

My new role has me writing detection for Snort primarily, moving into writing detection for ClamAV and even vulnerability research down the road.

I'm pretty excited about this move, as you can probably tell, and look forward to working with my new team.  

P.S.  I know I write this on April 1, but it's not an April Fools joke ;)

Let me tell you about my past two weeks

The past couple weeks I've had the opportunity to do some really amazing work, something that most people, if they could do, would understand a lot more of what goes on behind the veiled curtain.

The last two weeks I worked for Sourcefire's Vulnerability Research Team (VRT).

First I'd like to say that I've never worked with a more professional organization.  Period.  I came in to do some technical work with them, which consisted of analyzing hundreds of pcaps, tons of analysis, and as a result writing rules for those threats.  We did, kind of a tech exchange type of thing.

Now, we weren't shooting in the dark.  (even though there is no overhead lighting in the VRT offices, and you have to watch for getting hit in the head with a Nerf dart)  The VRT doesn't take the random vulnerability or exploit found on exploit-db.com or milw0rm or whatever, and just bang out a rule for it.  They do labor intensive work.

For instance, I had to write a rule for a vulnerability in a piece of software that had to do with email.  In order to test of this vulnerability, could I have taken a piece of a malicious attachment, or looked for a malicious attachment and written a "signature" to check for the exploit here.  Sourcefire's standard is higher than that.  We try to not do that kind of thing.  We try and write a rule to look for the vulnerability itself.  For example: If the vulnerability is actually the fact that a certain field, if it's over 512 bytes, can be used to overflow a buffer in the software, looking for a series of "A"s isn't going to work.  Looking to see if the field is bigger than 512 bytes is the correct way to do it.

But I digress….

The easiest way to emulate this problem is to send an email with an attachment on it, and capture the pcap, then pick it apart from there.  The problem with that is, most  email (well at least Sourcefire's) is encrypted.  So, I got with one of the other VRT guys and we came up with a solution.

Write an email delivery system.

So he did.  It's in ruby, and it allows you to send an email, just like any other email client would, unencrypted, and much faster and more reliable than a regular email client would, if we were trying to trick the client into doing something.

We took the ruby script that he wrote, made it attach a file in base64, and captured the pcap.  Now, you may ask me a question, "Heck, why didn't you just make a new email with Outlook Express and make an attachment and send it?"  Because Outlook Express uses a different attachment system, it's crackheaded, and it's non-standard.  Don't believe me?  Send an email with Outlook and then send an email with Outlook Express and compare the two pcaps.

So I captured the pcap -- that's all well and good, except that I noticed that the checksums in the pcap was wrong.  Sometimes when you capture traffic on an interface, on certain OSes, it will capture the traffic before the checksum is computed, so it will write to disk incorrectly.  So that has to be corrected before you can write a rule to look for the vulnerability.

So, I used tcprewrite to correct the checksums on the packet, and off I went from there.

Now, you come up with the realization that this happens, sometimes 10-20x a day for the VRT, and you come to realize that the rules that are written by these guys are very professional and come with a higher degree of accuracy and purpose.

I'd like to thank the VRT to allowing me to come in and learn and share with them.  I hope I helped them out as much as they helped me.

Final thought -- Take your time when writing your rules.  The time spent writing them makes for a much more reliable rule than just banging out a rule…. and I have seen a lot of "just banging out a quick rule" lately.  A quick rule usually isn't a rule.  It's a signature.  There is a difference.

Oh, and whomever wrote the Microsoft Word and Excel standard is a crazy crack smoker.

Long live Razorback.

Project Razorback has been unleashed on the World

For several months, the Vulnerability Research Team (VRT) here at Sourcefire has been heads down in coming up with a new framework for detection called Razorback, and now, it's been unveiled to the world this this morning.

Being announced at Defcon this weekend by the VRT, so if you are in Defcon this week, reading my posts, First: Have a beer for me, as I am not there this year due to the impending birth of my child, and Second: Attend this talk.  If no other talks are attended during your drunken hacking binge in Vegas, go to this talk.

OH AND BUY THE VRT BEER IF YOU MEET THEM.  Mkay?

What is Razorback?

In Marketing speak: "Razorback is an Open-Source Framework for an intelligence driven security solution."  Okay, okay, what does that mean?

Razorback is a system that detects and decodes, well, just about anything you need it to.  Following that, it has the ability to then block and alert on that activity.  So, for example:

• Obfuscated Javascript?  Decoded, Blocked?


• Bad PDFs? Decoded, Blocked?


• Bad Word Documents? Powerpoint Documents? Decoded, Blocked?

This framework is aimed primarily at these Client based attacks, and, dare I use it?  Advanced Persistent Threat (APT).  It was born out of necessity and a discussion with the VRT during a panel they participated in last year about detection.  The community asked for something to be able to perform a function like this, and well, here it is.  Better.  There is nothing to combat these threats, so Sourcefire created one.

So, say for example, a PDF comes in via email.  The PDF is sent to Razorback by the SMTP engine, Razorback runs it through the detection, -- which I'm not even going to begin to explain here, because it's extremely awesome and complicated, and you should go to the talk to fully understand --, and if the detection decides the PDF is bad, it will record that fact in it's database so that all further attempts with a PDF like that one will be blocked from there on out.   Now, that's just one example.

Since Razorback is an Open-Source project and framework, anyone can write a detection "nugget" for it.  These nuggets, written in C, can detect pretty much anything and provide actionable intelligence on it afterwards, and of course, since it's Open-Source, many different "feeds" can be provided to Razorback.

SMTP, ClamAV, Snort, Web proxies, Web filtering devices, et all.  They can all be written to feed data to Razorback which then can have the ability to take further action after it's analyzation.

This is a different approach to detection than what's been tried before.  While IPS is great, it can't really grab a PDF off the wire, reassemble it, decode it, and block it in real-time.  With Razorback, Snort can grab the PDF off the wire, pass it to Razorback where it will be analyzed, and so on.

After the talk if the VRT puts their slides and more info up on their website, I'll make sure that I post further information about it.  But for now, here it is:

Razorback.

Here's another article about Razorback over at DarkReading.

Snort 2.8.6 segfaults

Putting this post up for the people who will Google the error.

If you get an error that looks like something like this:
"segfault at 0 ip b7955947 sp bfa35d70 error 4 in libsf_engine.so.0.0.0[b7953000+8000]"

When you start Snort after you have upgraded to 2.8.6 from 2.8.5.3 (or whatever)

This means you are running 2.8.5.3 SO rules with the 2.8.6 engine. You need the 2.8.6 rules to run with the 2.8.6 engine.

You can get the rules here: http://www.snort.org/snort-rules

Make sure you read this post too: http://blog.joelesler.net/2010/04/new-vrt-rulepack-changes.html

VRT: Using Snort fast patterns wisely for fast rules

If you read my blog, chances are, it's because you have something to do with, or have heard of, Snort.  Check out the below link, because VRT shows you how the pattern matcher works, and how to make it bend it's will for awesome.

VRT: Using Snort fast patterns wisely for fast rules.

PulledPork v0.4.1 released!

New Features/changes:

- Flowbit tracking! - This means that all flowbits are not enabled when a specific base ruleset is specified (security etc...) but rather all flowbits are now tracked, allowing for only those that are required to be enabled.

- Adjusted pulledpork.conf to account for new snort rules tarball naming and packing scheme, post Snort 2.8.6 release.

- Added option to specify all rule modification files in the master pulledpork.conf file - feature request 19.

- Added capability to specify base ruleset (see README.RULESETS) in master pulledpork.conf file.

- Handle preprocessor and sensitive-information rulesets

Bug Fixes:

- 18 - non-rule lines containing the string sid:xxxx were being populated into the rule data structure, added an extra check to ensure that this does not occur

- Cleaned up href pointers, syntatical purposes only...

- Modified master config to allow for better readability on smaller console based systems

- Error output was not always returning full error

Be sure and go here to download the newest update!

http://code.google.com/p/pulledpork/

Be sure and read my other two posts in order to make sure you are fully up to date with everything going on.

New VRT Rulepack changes

There has been a lot of confusion between the rule update packs.  Some people would see the word "snortrules-snapshot-CURRENT_s.tar.gz" in the rulepack name, or the "snortrules-snapshot-2.8_s.tar.gz" name, and not know which ones to use, or which version of rulepack to use with which version of Snort, so hopefully with this change we've eliminated that confusion.  Now the Snort RulePacks are specific to "Version released".

What does that mean for you?

If you are using 2.8.5.3 and are updating to 2.8.6 (recommended)

You need to go into your oinkmaster / pulledpork / wget / any updater that you are using, and change the name of the rulepack you are grabbing to the version that is specific to your environment, so if you are changing to 2.8.6, you will not only need to update to 2.8.6, but you will also need to change your rulepack name to:

snortrules-snapshot-2860.tar.gz

If you are using 2.8.5.3, and are NOT planning to update to 2.8.6 at this time

You STILL need to go into your oinkmaster/pulledpork/wget/any updater that you are using and change the name of the rulepack you are pulling to the version that is specific to your environment.

In short, everyone that uses Snort will need to make this change.  For the next 30-days, the "snortrules-snapshot-CURRENT.tar.gz" and "snortrules-snapshot-2.8.tar.gz" links will symlink to the "snortrules-snapshot-2853.tar.gz".  So if you update to 2.8.6 you will need to change to the appropriate rulepack.

These symlinks will exist for the next 30-days.

If you are a Snort VRT rules subscriber (aka, you pay for it), the symlinks will be of use to you for 30-days, however, you are strongly encouraged to make the change now so that after the symlinks are removed, you won't get 404 errors.

If you are NOT a Snort VRT rules subscriber (aka, registered user, you don't pay for it, and you get the rulepack after the "30-day free window" is lifted) you need to make the change.  So for example, if snortrules-snapshot-CURRENT.tar.gz is in your rule download URL, you need to update it to snortrules-snapshot-2853.tar.gz (or snortrules-snapshot-2860.tar.gz if you update).  The Symlinks will NEVER apply to you, as the new packages won't be available to registered users for 30 days.

If you are running a version of Snort that is < 2.8.5.3.

You will need to modify oinkmaster / pulledpork / wget / whatever update system you are using to remove 2.8.5.3 version specific rule keywords or Snort will fail to load.  Please update to 2.8.5.3 at least, or move to 2.8.6.

Snort.conf

The Snort.conf file that is in each rulepack is ALSO version specific now.  (Yeah!)

The rulepacks will also be significantly smaller because of the fact that since the rulepacks are locked to the version of Snort they support, only the SO rules for the specific rulepack version are included.  For instance, the 2853 rulepack will only contain SO rules for 2.8.5.3.

Also be sure and read the VRT blog for further information: http://vrt-sourcefire.blogspot.com

Snort 2.8.6 is released!

[*] New Additions
* HTTP Inspect now splits requests into 5 components -
Method, URI, Header (non-cookie), Cookies, Body.
Content and PCRE rule options can now search one or more of these buffers.

HTTP server-specific configurations to normalize the HTTP header and/or cookies have been added.

Support gzip decompression across multiple packets.

* Added a Sensitive Data preprocessor, which performs detection of Personally Identifiable Information (PII).  A new rule option is available to define new PII.  See README.sensitive_data and the Snort Manual for configuration details.

* Added a new pattern matcher and related configurations.  The new pattern matcher is optimized to use less memory and perform at AC speed.

[*] Improvements
* Addressed problem to resolve output obfuscation affecting packets when Snort is inline.

* Preprocessors with memcap settings can now be configured in a "disabled" state.  This allows you to configure that memcap globally, but only enable the preprocessor in targeted configurations.

Go to http://www.snort.org to download the latest release!  I have two more posts that will be coming out later today with further updates, so make sure you read those as well. Also, make sure you read the VRT blog for further information: http://vrt-sourcefire.blogspot.com

VRT: The New Disclosure Debate and the Evil Mr. Moore

VRT: The New Disclosure Debate and the Evil Mr. Moore.

I am not trying to get into the business of reblogging Sourcefire VRT's blog entries, but I blog things that I think are interesting, or that I think my readers will find interesting and hopefully debate.  I think this is yet, ANOTHER insanely great article by Mr. Matt Olney.  Please click the link above and read it!

VRT: APT: Should your panties be in a bunch, and how do you un-bunch them?

VRT: APT: Should your panties be in a bunch, and how do you un-bunch them?.

I don't know how to say it anymore than this:

Matt Olney wrote a damn, a DAMN good post about APT on the VRT blog, and if you read my blog, and you don't go over to the VRT blog and read that post..  Heck I don't care if you don't read another post by the VRT that they have written in the past (although, you SHOULD!  They put a LOT of time into their posts!) you should read this one.

Matt, whom I play Xbox with nearly every night, talk to on a regular basis, and consider to be my friend..  I just wanted to let you know, seriously...

Damn fine job sir.

Sourcefire VRT Labs: MS to SID mappings

Sourcefire VRT Labs.

For those of you that are using Sourcefire VRT rules to protect your network with your Snort IDS/IPS installation, (as you should!).  There are mappings from MS vulnerability number to SID number, in the past, you either had to be a Sourcefire customer (we make this super easy in the Policy Editor GUI) or you had to be very patient and grep your way through the rules.

However, VRT put these mappings in a super easy to use interface at the link above.  Check it out.

Update:

Nigel corrected me, these mappings have always been on Snort.org, VRT just moved the hosting.  Duh.

Dojocon

Drove down to Dojocon at Capitol College in Maryland today.  Did the old, "Man the Sourcefire Booth" bit.  Except this time, it was for VRT, instead of at a big Orange Sourcefire booth full of literature about product, the questions this time were about Snort and VRT rules.  Quite a bit different from normal, great though.

Dojocon did quite well (It's still going on), 150-200 people there, I would guess, (I'm not good at people estimation), lots of good presentations and lots of good questions at the end of the talks. Food and drinks and snacks were provided (which is a nice change from other conferences I've been to).

I recommend going if you can next time they have it, great resources of information there, Marcus does a great job.


Please leave comments below.

Microsoft IIS 5/6 FTP 0Day

In my job, I see a lot of Snort rules being thrown around for this that, and the other thing. The thing I try to emphasize is not to make rules for rules sake. Don't write rules just because you can. Write rules because you have to.

So recently an exploit for Microsoft IIS's FTP daemon was released on Milw0rm. (Go find it yourself if you must.) Almost immediately I saw a ton of people trying to make rules for it. Turns out, rules didn't need to be made.

The Ftp_telnet preprocessor was written a long time ago to deal with these "buffer overflow" type of exploits. Plus, a lot of old rules were already in place to catch it.

Check out the VRT's blog post about it here. Use the rules and preprocessor alerts that they suggest.

So, lesson learned here? Before you try and write rules, get a pcap and run it through Snort with all the rules on already. You should have a separate instance of Snort that you use for running pcaps through that mimics your actual live set up. This instance of Snort should have every rule turned on and every preprocessor alert on. That way you can see, if you run a pcap through Snort, what alerts, and if you need to write a rule in the first place.

Sourcefire's Exploit Development Class

First off, if you were to go look at this class on Sourcefire’s website, it states “Exploit Development Class for Snort Rule Writers”. We need to fix this. In the words of Lurene, “This class has nothing to do with Defense. At all. Ever.” The class should be more appropriately named, “Fundamentals of Exploit Development”, or “Writing Exploits, we’re going to hurt you”

So, let’s describe this class in two words or less:

Freakin Awesome.

Beginning on day one with a lot of terminology, introduction and drinking from the firehose on Assembly and gdb, by the end of the first day, you are well versed in how to read assembly, pick it apart, and even being able to reverse simple programs at this point. Your Brain will hurt.

Day Two, more drinking from the firehose, more reversing, more assembly, more gdb, drawing stacks, and by the end of the second day, you are learning to control EIP, and doing it. Your Brain will hurt even more.

Day Three, you just sit all day and hack programs. From simple to intermediate, (you aren’t cracking Microsoft Office just yet ;), by the end of the day, you are using reverse shells and shellcode like nothing. Your Brain is now fried. Go drink beer. Seriously.

This was the best class I have ever taken in my life. Srsly.

You know those classes where you go and sit, and you could probably figure out 80% of it, and the other 20% of the class you pick up little tricks and tips on whatever you are learning? This is not one of those. If you know assembly, or have experience in reversing assembly, this is not the class for you (even though you will probably learn something). The class I took was taught by 4 of the Vulnerability Research Team members, people I am glad to call my friends.

So, my hat’s off to Lurene, Matt, Ryan, and Nigel, along with all the other members of the VRT that contributed, came out, and helped with the class. It was great, and I’d gladly take it again anytime.

The best part of the class, I thought, was during the class, on a separate projector, they fuzzed software and found some 0days. I don’t want to disclose which pieces of software were fuzzed, but let’s just say that they are pieces of software that people use everyday.

In one piece of software, over 200 crashes and bugs were caused. No word on how many were exploitable yet.

No, I will not tell you which piece of software it was either.

Why is your Blog named Finshake?

Someone wrote in and asked me why I named my blog “Finshake”. Well..

Finshake is an internal joke between me and the guys in VRT at Sourcefire. A while ago, I was an author on the “Snort IDS and IPS toolkit” book from Syngress. Well, with the rush to deadlines and things, there are several mistakes in the book. Okay, so there are alot of mistakes made in the book...

Well, one of the biggest mistakes in the book, actually happened in my chapter. (Chapter 6). I was talking about TCP Session initiation and TCP Session tear down and how Snort interprets those. In the final book, I wanted pictures of the TCP Handshake for session initiation, and the TCP exchange for session tear down.

In my copy of the manuscript I simply indicated where the pictures should go:

I didn’t actually draw the pictures. I knew Syngress had the pictures from the 2.1 book, and I just asked them to use those.

So in my final proofread of the pdf that I got from the publisher:

The place holder was there, but no picture. Oh well.

The picture was inserted later, and no one ever checked to see if the picture was right. Oh well.

So it’s become such a funny joke around the VRT, someone made the suggestion that I should rename my blog “Finshake”. (Since obviously, Session initiation does NOT take place with a “FIN” packet!?)