Greg Reinacker’s Weblog

Justin writes:

Greg is asking about something that is near to my heart - Web Services Threat Detection. Or as I’ve heard marketing people call it (and this is long folks) - how the hell do we keep people from accessing the web services who are deliquent on payment and aren’t deliquent because they are disatisfied with the service?

That is one of the problems I tried to solve at my ex-gig. Although it was always unofficial because no one wanted to sanction that project.

Unfortunately the best answer I ever came up with is this - I couldn’t do it in 100% code. It requires people. I could monitor everything that comes in (who, what, when, and from where). I could check whether or not they were supposed to be sending the particular type of message at that particular moment in time. I could shut that message down or route it to what Greg calls a honey-pot (never heard that before). But when it comes to intelligently doing it - I could never figure out how to tie the above logic into the CRM system. Or if I would really want to. Should my system really try to be smart enough to detect AND take action other than notification?

So what did I end up with? Nothing tangible. Just a bunch of ideas. Maybe I’ll get a chance to implement them at the next gig…

But I do agree with Greg. A system that can do all that he lists is going to be a specialized system. You might be able to find something like what I built on the market. But the intelligence you will have to integrate into your CRM somehow.  [News from the Forest]

Well, actually the specific problem you mention there in marketing-ese :-) seems like the simplest subset of the problem.  If a customer is delinquent on their payment, but they’re not dissatisfied, then presumably your automated billing system has already sent them warnings; and at some point the web services infrastructure should just shut them off.  Disable their account.  Unless I’m misinterpreting the described scenario, this is the trivial case.  This would presumably be driven from the CRM system into the web services infrastructure, so there isn’t a difficult CRM tie-in required.

The more difficult scenarios, in my mind, are as follows:

1. You have a customer who has accidentally coded up a bug which calls your service in a very fast loop.  Instead of getting 1 request per second from this customer, you’re now getting 25 requests per second.  We need to, at some level, slow down or disable this customer’s access until they get it fixed.  Note that discerning this case from the case where the customer is having an unusually busy day is non-trivial, and will probably be business-specific.  As Justin says, this might require input from the CRM system, to look at typical usage patterns for this specific customer, and see if something appears to be amiss.

2. You have a customer who, potentially through no fault of their own, is repeatly making valid requests which trigger a bug in your system, possibly crashing it, or causing abnormally high resource usage.  If the customer keep retrying this request, for example, until he gets a valid response, this might consume huge amounts of resources.

3. You have a malicious customer, with a valid account, who decides that now is a good time to see if he can take down your system by swamping it with requests.  Now, this will probably be punishable legally, but punishment after the fact is not sufficient to keep our system running today.  We need to stop this attack as it is happening, to protect our system.

4. The malicious customer in (2) above becomes a bit more clever; instead of swamping you with requests, he repeatedly decrements your inventory, for example. 

5. A malicious attacker, who has somehow gained access to your system, doing any of the above attacks.

There are also lots of business-specific threats.  For example, if I’m selling cookies with my service, and you don’t have to pay until you take delivery, then an obvious attack is someone falsely reserving one box of cookies per minute for a week.  Difficult to detect real-time, but arguably necessary.

Oh and the honeypot thing - the idea here is if you detect an unauthorized intruder, either attempting to gain access or browsing around in your system after gaining access - you redirect him to a “honeypot”.  This would be a dedicated cluster of systems, which look just like your production systems, but are not actually production systems.  The attacker can do all the damage he wants, and you can even entice him with seemingly valuable data.  While he’s attacking these honeypot systems, you and/or the authorities can begin tracing his location and identity.

A while back in another life, we considered the idea of a real-time threat detection system for our web services. The idea was we could build (or buy) an infrastructure component that could analyze the incoming bit stream, detect anomalies, and react appropriately. The “anomalies” detected would be along the lines of:

1. Unauthorized customer repeatedly attempting to gain access
2. Repeated requests causing errors
3. Unusually heavy volume of requests coming from a specific customer (have to be careful here, can’t shut down amazon.com at christmas time)
4. Repeated malformed requests
5. Lots of other items…

Reactions would be along the lines of:

A. Cut off a customer’s access (realtime)
B. Re-route requests to a “honeypot” for analysis
C. Notifications (event log, email) for certain detected events
D. Other options (throttle requests, notify firewall to block IP, etc.)

This system would have to be blazingly fast; large added latency to the overall request/response could not be tolerated for many applications.  Building a system like this is highly complex and application-dependent; for example, the very existence of a credible threat might depend on the cost to service a request. If it costs more to turn away an evil (but authorized) request than it does to just process it, then you have to make a call.

It also ties in with things like guaranteed levels of service. If you have to guarantee some subset of your customers a certain response time, but you only have so many cycles to spare, then you’ve got to prioritize your traffic; but your threat detection system must, at the same time, analyze the traffic and help do this prioritization.

Everything I have mentioned here is outside the scope of a “typical” off-the-shelf intrusion detection system, I believe, as it must incorporate logic that knows about my services, the resources they use, and the cost of certain requests. The IDS is still required to detect “normal” attacks; but for service-dependent threats, you need a new system.

Any thoughts about this?

Announcing Groove Experiments shared space and work

Greg Reinacker and Sam Gentile would like to announce our creation of “Groove Experiments”. We have been working and Groovin to flesh out some ideas.This dynamic shared Groove space has three main purposes:

1.  Groove and Blogs. There are certain synergies between Groove spaces and weblogs. Let’s enumerate and expand a bit on the possibilities. In particular, one great synergy, especially in the Web Services and .NET communities, is the initial posting of some technical idea or topic on a Blog and then wanting to get into a more “direct” and detailed discussion or interaction about that. Groove is excellent for that immediate phase of fleshing out and discussing the idea(s). So the first main area and disussion is around providing the ability to have detailed disucssions about blog topics and explore the synergy further.
2. Groove and Web Services - We believe that there are some great ideas to be explored in this area. Lets get them on the table. Maybe have a groovespace aggregator that lets you see new entries for multiple groove spaces, so you can keep up with all the the activity for topics on your blog. There are also plenty of areas of discussion on Web Services in general and where they should go.
3. Groove Platform Applications - We believe that the value of Groove is not in the existing tools but in the underlying Platform and the decentralized peer-to-peer communications mechanism that can create some really interesting applications. Lets explore.

This is something Sam and I have been working on. I want to emphasize that neither of us is an employee of Groove Networks and that this shared space is not speaking for Groove or sanctioned by Groove. We believe that there are fine areas of interesting research to be done here. If you are interested in particpating, please email Greg or Sam to be sent an invitation. Ideally, we would like to keep it to under 12 particpants.