Customized Capistrano tasks per-host

This was something that took a while to piece together, and most of the links I found on the net pointed to vastly more difficult solutions.

The problem: I wanted to create iptables rules that would lock down the back-end private IP’s of my servers to allow access only from each other. Every time I add or remove a server, these rules need to be rewritten, so my Capistrano deployment seemed the logical place to do this (as opposed to my server installation scripts).

But…each server needed its own rules, which would be different from each other because the interfaces would be different. And in Capistrano, this isn’t really the default mode of operation. It’s easy to customize tasks by role, but doing it by host (while allowing for easy addition or removal of hosts) is harder.

You do have access to $CAPISTRANO:HOST$ within your tasks; however, this is only evaluated within the context of certain functions like “run”. Which may or may not help…in this case, it did not.

Here are the relevant snippets from what I did. First, I added a custom attribute ‘internal’ to each host, which is its internal IP address:

Then, inside my :update_rules task, I needed an array of all the internal IPs, so I find the servers for the task (using find_servers_for_task) and iterate over them to pull out the :internal attribute:

And finally, the part that does something different on each server…here, for each server, I add rules for all of the internal IPs; note the :hosts option on the run command, which specifies the task will run on only that host (sorry for the line wrapping).

I’m looping through all the servers, running a specific task on each of them. This isn’t perfect; it will run on only one host at a time, rather than running in parallel…but it gets the job done!

Sparrow

If you haven’t yet seen it, Sparrow (Mac app store) is an email client for the Mac that really focuses in on great Gmail support, and has a very different UI than Apple Mail. Ever since Sparrow was released, there has been a lot of chatter about it; nearly all of what I have read has been very positive. I’ve been using the app every day for about six weeks, and I wanted to post my thoughts about it.

If you’re a heavy email user, then you can’t take switching email clients lightly. You know all of the keyboard shortcuts of your client, and if something changes even subtly you will notice. Consistency and reliability are the most important traits. No one likes sitting around in their email app, so anything that helps you get in and get out quickly is what you’re looking for.

I’m not going to walk through how Sparrow works – there are videos on the web site you can watch, or you can read many reviews around the net going into detail on the app.

These are the things I found good about Sparrow, as a user who had been using Mail:

  • It’s very pretty. It looks more like Twitter for Mac than Mail, at least when you’re just looking at the message list.
  • It works really well with Gmail. There is a button for archiving, and shortcuts to label and archive. It displays conversations quite effectively.
  • Just added in 1.2, the Universal inbox feature was a great usability enhancement.
  • It has a great quick reply feature, where it opens a control for you to type a response without having to pop up a whole message window.
  • It’s “different”, and somehow more “fun”. I can’t quite put my finger on it, but there’s something satisfying about using it.

But not everything is roses. Here are the things that gave me fits:

  • It’s not so smart about image attachments. I took a screenshot of a window to email to a friend, and pasted it into a new message in Sparrow. It sent it as a 7MB TIFF file…but when I paste the same thing into Mail, it pastes in “actual size” as a 45KB PNG file. Needless to say, defaulting to a multi-megabyte TIFF is unexpected.
  • I have a colleague using Outlook on Windows, who sends me a regular email that has two attachments (a docx and a xlsx file). These do not make it into Sparrow intact, but rather show up as a “winmail.dat” attachment. Yikes – I thought this problem was behind us! I actually have to open these messages in the google web client to read them.
  • No spotlight integration. For me, this is a big one, because I use spotlight all the time…
  • I’m not sure I can put this down as a “con”, but some of the animations in Sparrow which are very sexy when you first get started, become less endearing as time goes on. Expanding out the message pane, for example, could be a little faster. I also notice choppy animations when having a new message window animate across multiple screens on my Mac Pro. This isn’t the end of the world, but it’s the little things that you notice day in and day out.

The first three issues on the list above are enough to have made me finally switch my accounts back to Mail. Proper encoding and decoding of attachments is pretty much table stakes in this game, and Spotlight is something I’ve grown to expect of an app like this as well. It’s a little bizarre to me that they’ve spent time adding things like Facebook integration (still not sure why I need that), as opposed to really solidifying the app, but clearly they have a vision in mind.

So I’m back to Mail. To help make Mail more usable with Gmail, I’ve added the Archive button plug-in, which adds a button and a keyboard shortcut to archive messages. So now, “delete” sends a message to the trash, and “archive” archives it to “all mail”, which is exactly how I would expect it to work. This alone has made Mail so much more usable with Gmail and Google Apps.

As for Sparrow, I’m not giving up on it, but I’ll wait for some of the problems and usability issues to get worked out. And I’m really looking forward to trying Mail in Lion, which looks like a major upgrade over prior versions.

Generic terms of service

I’ve written in the past about Facebook and Picasa and their (IMHO) ridiculous terms (at least at the time, I haven’t reviewed them lately). Now twitpic, a site that we all use used to upload pictures to show on twitter has decided that they are more than welcome to sell our pictures. They even tried to say “everyone does it” with the new, improved, “you own the copyright but we’ll still do whatever we want” terms. Luckily, we are welcome to switch services, and I suggest you think about it if you care about this sort of thing.

More generally, to solve this whole TOS problem, I hereby propose two sets of terms of service agreements, and a company can just pick one and put it on their web site. They’re even short enough to put on the front page, instead of behind the tiny link hidden below everything else. Short enough that *gasp* people might even read them.

Option 1: (e.g. twitpic, along with twitter and many others)

We provide a service. You upload your stuff. Don’t upload porn or illegal stuff. Once you upload it, we own it as much as you do, and we’ll do whatever we please with it. You promise to defend us from anyone who doesn’t like that.

Option 2: (e.g. yfrog, smugmug, and others)

We provide a service. You upload your stuff. Don’t upload porn or illegal stuff. We’ll put your stuff on our web site like you asked.

Add something to both of those like “we won’t tell anyone your email or contact info” and I think the terms would be pretty much complete.

Learning Ruby and Rails

A few weeks ago, I decided it was high time to get back to writing code. NewsGator’s code is based on Microsoft .net, and much of my career has been building products for Windows. Given that, I figured it was time to learn how the other half lives.

I started this adventure learning PHP (which reminded me a lot of pre-.net ASP, with some extra language constructs like classes sort of bolted on), and dabbling enough with MySQL that I could do what I wanted.

Then I decided it was time to learn Ruby and Rails. It actually took a fair amount of effort to figure out how to get started there, and I didn’t find any blog posts that really laid it out all in one place…so here is what I did.

First, I wanted to learn the language without the additional complexity of Rails on top of it. I downloaded the ruby_koans project, which essentially has a bunch of test cases, and code that you need to fix in order to continue. It was a unique way to learn, I thought, but I think I got a fair amount out of it.

After I was done with that, I thought I generally had the idea, but wanted to dive a little deeper into the language. So I read the Humble Little Ruby Book. I found the writing style a little distracting at times, but the book went beyond what I had learned with the ruby_koans, and after I was done I felt like I was ready to go. If you read this book, read the PDF version rather than the HTML version – the PDF one has much better code formatting (indents, etc.)

Ok, now that I was an expert in Ruby (ha), it was time to dive into Rails. Somehow I stumbled across the Ruby on Rails Tutorial by Michael Hartl. This was absolutely fantastic – I worked through it over a few days, and it provided a great foundation in Rails. I really can’t recommend this enough; he even covers rvm, git, rspec, heroku, and lots of other real-world topics. You can buy the book, buy a screencast version, or go through the free online version.

The beginning of that tutorial gave a little taste of using Git and Github; I realized I was going to need to learn a little more about git. To do this, I’ve been reading Pro Git by Scott Chacon, which seems like the go-to reference for git. You can read it online for free, or buy the book.

And then finally, as I’ve been working on a new project, I’ve been reading through the Rails Guides, a little at a time. They sort of pick up where the Ruby on Rails Tutorial leaves off, filling in details on specifics.

Hopefully this will be helpful for some folks…and I’m happy to finally have all these links in one place. If there are other great resources out there, please leave a comment and let me know!

Traffic threshold pepper extension for Mint stats

I’ve started using Mint for web stats on this site. I stumbled across a review from Shawn Blanc that he wrote about an older version a while back, and decided to try it. It’s a web stats package that you install on your server, and it’s really focused on what’s happening right now (as opposed to deep dives on old historical data). I’ve had it running for about a week, and I love it so far! It’s also extensible with add-ins called “peppers” which can be found in a directory on the site.

Not one to leave well enough alone, I also wrote a pepper. Sometimes this site gets bursts of traffic, which I don’t always find out about until later when I’m looking at the stats. So I wrote a “Traffic Threshold” pepper which will send an alert via email when a certain number of page views per minute has been hit.

PreferencesIt’s designed to be super fast. No extra script is used on the client side. The preferences are stored in the Mint database using the Pepper API, so Mint will load them (and it’s designed to load pepper preferences efficiently). The actual page view counting, though, isn’t done with the database or a file, but rather uses a unix system V shared memory segment. Web requests are served from multiple processes, and thus the page view counter needs to be saved somewhere where they can all access it; shared memory (with synchronization to protect against simultaneous updates) is one of the fastest ways to do this.

The shared memory is allocated on installation (when the pepper is testing to see if the required APIs are available), and will be cleaned up when the pepper is uninstalled. It won’t work on every system – for example, if you’re on a shared hosting plan, the required APIs may be disabled. But you can give it a shot, and you’ll see a message during configuration if the pepper can’t be installed.

It also assumes you have a default mailer set up on your system to send mail. It measures using clock minutes, rather than a 60-second sliding window. There are technical reasons for this, but most folks will never notice. And it will only work for sites hosted on a single server. If you’re a bigger site that’s hosted across a large web farm, you probably don’t need this pepper anyway!

I’ll submit this shortly to the Mint Peppermill, but in the meantime you can download it here.

Update: now available from the Mint site.

Update 2: source now available on Github; if you have ideas to make it better, feel free to contribute!

Don on the Amazon outage

An excellent article from the super-smart Don MacAskill, CEO of Smugmug, on how they survived the Amazon Web Services outage and a few tidbits about how their system is designed. One little tidbit:

Once your system seems stable enough, start surprising your Ops and Engineering teams by killing stuff in the middle of the day without warning them. They’ll love you.

You know you’re confident in your system when you throw a dart at a board full of “kill” buttons. :-)

Don was one of the early folks to make a big commitment with AWS – he’s been through a lot with them, and has learned a ton of useful things. Definitely worth a read!

Mac Pro fortune cookies

TUAW has an article up today wondering if Thunderbolt might mean the end of the line for the Mac Pro in its current form:

The arrival of the Thunderbolt interface, Meta Media says, will allow Apple to return to its beloved sealed-box model of computer production with no user-serviceable parts inside, just like the original Macintosh. No expansion cards, no hard disk upgrades, just Thunderbolt (aka Light Peak) interfaces to connect … well, to connect anything you like really.

The comments to that article are full of opinions, as expected. As someone who uses a Mac Pro every day, I’ve thought about this off and on for the last year or so. Thunderbolt definitely changes the game, providing two 10Gbps channels on one cable.

The big differentiators for the Mac Pro today over the other models in Apple’s lineup are CPU, RAM capacity, display capabilities, and internal storage. CPU and RAM are both significant, but the lower end machines are making this up quickly – witness the impressive performance of the new Sandy Bridge-based Macbook Pros. A next-gen iMac could be quite impressive on these fronts if Apple chose to push the envelope.

As for internal storage, the MP has four internal drive bays for SATA drives; I have all four filled in mine. However, a Thunderbolt port with two 10Gbps channels for external drives would certainly suffice, even compared to a potential future MP with 6Gbps SATA…and for the folks who really need the I/O performance – folks editing HD video, say – Thunderbolt RAID systems could be a step up over what they can do now with 2/4 Gbps fiber adapters.

Multiple displays make up another area where the MP shines – it’s the only Mac where you can have more than two displays (not counting network- or USB-connected displays of dubious performance). I use three (with two video adapters), and I know folks with more. While I could get by with two large 27-30 inch displays myself, there are others who would not be so understanding. As I understand it, Thunderbolt supports one Display Port display at the end of the chain. Seems to me a future MP replacement would ideally need to support at least 4 2560×1600 displays to be accepted by the real power users, so Thunderbolt isn’t helping too much here…unless they built in more ports.

In any case, it appears to me that Thunderbolt definitely enables Apple to make some real changes to the MP line if they want to, specifically with respect to storage and other peripherals that have traditionally used PCI Express. They could either redesign and slim down the existing MP by getting rid of most of the internal drive bays and the space used by the PCIe slots, or potentially even drop the MP completely in favor of a “super-iMac”.

Time will tell.

Choosing a colocation vendor – power and cooling (part 3)

In part 1, we had a quick overview of things to think about when looking at colocation vendors and data centers. In part 2, we looked into your network and your bandwidth usage. Today, we’ll talk about power.

I pretty much ignored this when I first moved into a top-tier facility. I assumed if I was leasing one rack of space, they’d connect up enough power for me to fill that rack up with servers, and I wouldn’t need to worry about it.

In reality, that was far from the truth. Power is the most important thing to think about in a data center.

Let me say that one more time, just in case you missed it. Power is the most important thing to think about in a data center. And you should spend some quality time thinking about it.

Power density

Here’s a scenario, which, uh, may or may not be based on a true story. You’re growing fast. You need servers, pronto. And you want to make sure you have enough room to scale to more servers. So you call IBM, and the sales guy comes in with small army and they start talking up their blade servers. And you just about fall over…not only do they look cool, but you can fit 14 blades in one 7U chassis! Six of those will fit in a rack, and you do some mental math, thinking about how many processor cores you can fit in those racks you just leased at your hosting facility. Growth problem solved. You’re almost ready to pull the trigger.

You just need to make sure you have enough power outlets in your rack to plug these babies in. You call your host’s engineer on the phone, bragging about how many processor cores you’re going to have, and has he heard about the new Binford SS-9000 blades with the 10-core processors? And you hear him tapping away on his keyboard in the background, and you’re just not getting the excited reaction from him you were hoping for.

Then he points out that those blade enclosures you are talking about have four 2300W power supplies each. Six enclosures. That’s 55 kW. And while you’re unlikely to actually use 55kW at any given moment, even say 25 kW would be a lot.

No problem, you think. I’ll just keep adding power circuits as I keep adding blade enclosures, as my business grows. What’s a few bucks for power when I’m getting 140 cores in 7U? This is gonna be great.

But over the course of the next half hour, as he explains the data center’s version of the birds and the bees to you, you start to see the light. See, the problem isn’t whether or not they can drag in enough power cables into your racks. They can.

The problem is cooling.

Every data center has some amount of power they can dissipate per square foot. It’s essentially the capability of the cooling systems they have. Some facilities are more efficient than others, but everyone has a number. And you can’t go over it. So you can have that rack that’s got 6 enclosures full of blades if you want it – but you might have to have, say, 100 sq. ft. of empty space around it to ensure it can be cooled. And yes, you’re going to pay for that space!

In real life, you will often end up leaving empty space in some of your racks for just this reason.

So now you know. Power density is one of the most important things you need to think about as you’re planning your data center future. Talk to your hosting company’s engineers about it; you’re not the first one to have questions about it, and it’s going to affect you. If not now, later.

Multiple power circuits

On its way to your rack, your power has to come through a bunch of distribution equipment. Someday there could be a failure in some piece of equipment, out of your control. So, if you are allowed to (and you should be), you should have power from multiple independent circuits coming into your equipment.

Mission-critical servers that will cause you pain if they go down, like database servers and such, should probably have multiple power supplies, and you should make sure each server is plugged into at least two independent circuits.

Other more failure-tolerant equipment (or as I like to call them, “disposable”) could be plugged into a single circuit each. So if you have 10 web servers, maybe plug 5 of them into each circuit.

But here’s the important part about this. Do some modeling as to what exactly will happen if you lose a power circuit. Depending on what’s connected, and how, it’s likely that your other circuit will need to absorb some of the load from the circuit that went down, and is now drawing more current. Make sure to model these loads (call your equipment manufacturers if you need more data from them), and understand what will happen in different power scenarios.

And while I’m thinking of hard-learned tips, here’s another one to think about while you’re doing your power modeling – don’t load your power circuits up anywhere near their max capacity. Something like 50% would be more reasonable.

At one time we were living on the edge in terms of power in one particular rack, because we didn’t do this modeling. For whatever reason we ticked over the max on one circuit, and a circuit breaker tripped, taking it down. This caused a small surge on the other circuit due to the power supplies connected to it having to work harder. (can you guess what’s coming?) And then, you guessed it, this additional load on the second circuit caused its circuit breaker to trip. Two circuits, two breakers, no power.

Do your modeling. It’s not optional.

Choosing a colocation vendor – network and bandwidth (part 2)

In part 1, I gave a quick overview of some things to think about when choosing the data center vendor you want to colocate with. Today, we’ll talk about one particular topic, the network and bandwidth.

At a high level, there are three parts of your network: the external connection to the internet, your internal network inside your own firewall or router, and the connection between those two networks.

Other than research, there isn’t a lot you can do about the external connection to the internet. But if you go with a top-tier hosting company, and you should, then they know what they’re doing, they’re really good at it, and they can keep that connection up. You should learn about it, and what kinds of connections they have, the available bandwidth, redundancy, and all that, but once you are comfortable you can worry less. Just make sure your growth projections don’t end up with you using an appreciable percentage of their total bandwidth. Also ask about things like their capabilities around things like denial-of-service attacks and such; they should be able to tell you if and how they can help if you run into network-related issues like that.

If you think you will eventually need to be hosted in multiple facilities, and your vendor has suitable ones, then ask about interconnections between those facilities. In many cases you will find they have private connections between their data centers, so ask about the available bandwidth and how that is priced.

On the opposite end of things is your own internal network that you design. I won’t touch on that, as only you know your requirements, and your network folks can do a great job there.

It’s the middle part that’s worth thinking about. Your firewall (or router) is connected to some piece of equipment that’s managed by your host. Sooner or later, that equipment needs to have a firmware update, or a new power supply, or something else. It will go down, if only for maintenance; everything does eventually. So think about the effect of that. If downtime is not acceptable, then make sure you can get (at least) two ethernet connections, connected to different equipment on their end. And on your end, you’ll need your own equipment which is capable of managing and failing over between those connections as necessary (two firewalls, for example, with a failover mechanism).

And on the topic of bandwidth, you’re a grown-up now in terms of hosting. You don’t buy 500GB of “bandwidth” per month any more; you pay for usage in megabits per second (Mbps). So if your system sustains 30Mbps most of the time, that’s what you will contract for. Your host will measure your sustained bandwidth, probably at the 90th or 95th percentile, and charge you overages if you go over your contracted amount. You may also be able to specify burst capability; so for example maybe you contract for 30Mbps sustained and 100Mbps burst, and they will cap your max bandwidth at your burst rate.

So that’s all pretty straightforward, and probably not a lot of surprises there for the startup CTO. Tomorrow, we’ll talk about power and cooling in part 3 – and that’s the part you definitely don’t want to miss.