Friday, February 5

If you never knew it occurred, did it occur in the first place?

In my To-Do list, I have a section for Blog topics that I think of in $random_place and I want to jot down for brainstorming later. This topic has been on my to-do list for about a year.

I was standing on a stage giving a speech at a military base, in about 2004.  The people I was giving a speech to were about 200-250 different "network" and "Systems" administrators from all over this military base in tons of different units.  In this audience I had military, civilian, and contractor.  I was asked to give a speech to the system administrators because some of them didn't see the value in security in their systems.  It was an afterthought and people weren't terribly excited about having to follow $regulation that ensured proper lock down of various controls in the operating system and network.

I asked this question:  "If you never knew it occurred, did it occur in the first place?"  I paused for effect, waiting for an answer.  One didn't come.  Obviously they had no idea was I was talking about.

I proceeded to explain the importance of reviewing logs, system and network information, explaining to them the importance of what I had found that week upon a security audit I was doing of their Army post.

Hundreds of compromised machines, botnets, poor security controls, inadequate permissions, etc.  This was all from about 3 days of work.  I didn't even get into the trenches trying to find things, this was just surface level scanning and network monitoring.  Not even penetration testing, just scanning.

They didn't know.  They thought their network was perfect.  They thought it was clean.  They didn't need to review logs.  They thought wrong.

If you aren't going to review logs, if you aren't going to look at the system logs, the firewall logs, the IDS/IPS logs, then why collect them?  The problem is, we have things like SOX compliance now that mandates that we have some kind of logging system.  Which is fine, it's a great idea, but people are missing the point.  The point of the SOX compliance and log review is for people to REVIEW the logs.  Otherwise what is the point?  So you can go back and see when you were compromised?

Some people will agree with me here and say "Yes, I'd like to have historical information so I can go back and see when the intrusion occurred."

That's fine, I don't disagree, but stop for a second while reading this and meditate on this question "Why?"  What are you going to do about it?

If you are going to look at your logs and dismiss them, instead of looking at your logs and doing something about the mistakes that you find, then what's the point in looking at the logs.  Don't waste your time.

It's your JOB to be looking at these things, if you aren't going to DO your job, then quit.  We don't need you in our industry because it's people like YOU that are messing things up for the rest of us.

I'm going to do it...  I am going to use APT (Advanced Persistant Threat).  APT was found by looking at logs.  APT has been around for a long time.  Before I worked at Sourcefire, I worked for the Department of the Army in computer security, and we were dealing with APT (only it wasn't called that back then) then.  We didn't have an advanced term for the threat, we used terms like 'rootkit' and 'trojan'.  We were looking at hacks that we had never thought possible offloading information to countries that weren't ours.  Some of the techniques were so interesting and secret, they haven't been made public to this day, so I can't talk about them here.

But we found the compromises by looking through logs.  I've said this before, and I'll say it again, what's the point in having a security device that keeps logs if you aren't going to LOOK at it?


Nathan Christiansen said...

Isn't the point of reviewing logs regularly to catch APTs before they get entrenched? Or possibly even to remove the threat at the earliest possible time after intrusion, thus minimizing the effect of an intrusion?

I'm a software programmer, not a system admin guy. Even I think that those two things are self evident.

Joel said...

Yes, you are right. My point in the post was to say you have to review the logs. People don't do a lot of that and just think that there are devices that are 'turn key' and magical.

Nathan Christiansen said...

I'm sorry if my comment came out in an accusing tone. I was trying to reaffirm your point.

Anyone who does not review the logs needs a Security 101 class in my opinion. Reviewing logs for signs of intrusion seems self evident to me.

You can't have perfect security, nor is it desirable. No connectivity (read air gap) usually means you can't do business. So, you do what you can to secure your business to the point of acceptable risk. Then you look for signs of intrusions regularly and take care of them promptly.

I wonder why so many people do not get it.

Rob Lewis said...


Reviewing the logs is still about reacting faster and containment. In light of your past experience, would the difference between security attained on the NPRNET vs SPRNET be the use of high assurance systems etc. What is missing in terms of other defenses in the average network to fight APT?

Joel said...

NIPR and SIPR security are two different balls of wax. What is missing? NSM (Network Security Monitoring) the proper recording and correlation of logs. Not saying it's the end all, be all, but it will fight it.

Rob Lewis said...

Joel, I guess the question I am asking indirectly is that since corporate networks are closer to the NPRNET than SPRNET, in light of the nature of these attacks, does there need to be more features found in SPRNET like networks introduced into commercial networks to fend off this level of adversary? When you have Alan Paller saying that the maganitude of attack is one or a couple of levels above the defensive capabitlies of the status quo, what is he inferring exactly? He is not saying the adversary is getting in because of sloppy defenses, he is saying that current solutions are inadequate.

Based on your background, agree or disagree?

Joel said...

The traditional attacks have been server based. Someone attacking you. Recently this trend has switched to client based. You opening up your computer to the outside. This was the first of the client side attacks that was huge. Well done. To be honest, we don't know the full extent of what happened. I am not sure the companies involved do either.

So, do we know what happened? No, not fully. But it was well done, and no one caught it. That's what happened.

Joel said...

But do I think that keeping certain things (like your repository of internal code) on a separate network segment? Sure, but it makes accessibility hard. The security controls around these pieces of the network need to be very tightly controlled, obviously more than they are now.