Author Archives: gregr

Zero to PostgreSQL streaming replication in 10 mins

I’ve found a number of articles on the web for setting up streaming replication in PostgreSQL, but none of them seemed to actually put everything together that I needed, and none of them seemed to use the new pg_basebackup in PostgreSQL 9.1 and later. So with that in mind, here are a set of steps you can use to set up streaming replication, over the internet if you wish, using an encrypted SSL connection. We’re not going to set up log archiving – we’re going to rely solely on the streaming replication for now.

I’m assuming you have a master server set up on Ubuntu 10.04 or 12.04, running PostgreSQL 9.2.x, and you have a new slave server set up on the same OS and pg version. The IP of the master is 1.2.3.4, and the IP of the slave is 5.6.7.8.

First, create the replication user on the master:

Note that we are using REPLICATION permissions, rather than creating a superuser.

Next, configure the master for streaming replication. Edit postgresql.conf (on Ubuntu, this is at /etc/postgresql/9.2/main/postgresql.conf):

We’re configuring 8 WAL segments here; each is 16MB. If you expect your database to have more than 128MB of changes in the time it will take to make a copy of it across the network to your slave, or in the time you expect your slave to be down for maintenance or something, then consider increasing those values.

Then edit the access control on the master to allow the connection from the slave; in pg_hba.conf (/etc/postgresql/9.2/main/pg_hba.conf on Ubuntu):

In this case, 5.6.7.8 is the IP address of the slave that will be connecting for replication, and hostssl means this host can only connect via SSL.

You’ll need to restart the master after making the above changes.

Now on to the slave. In the slave’s postgresql.conf, add the following:

Then restart the slave. No changes are required in the slave’s pg_hba.conf specifically to support replication. You’ll still need to make whatever entries you need in order to connect to it from your application and run queries, if you wish.

That’s all the initial setup we need to do. After you’ve done the above configuration, running the following script on the slave will copy the database over and begin replication (1.2.3.4 is the IP of the master):

That script will stop the slave, delete the old slave cluster directory, run pg_basebackup connecting to the master to copy over the databases (you’ll see the progress as it goes), create a new recovery.conf file, and start the slave. If you look at the logs after this script completes, you should hopefully see messages about it having reached a consistent recovery state.

Be careful – that script is going to delete the old database cluster on your slave, so make sure to read through it and understand what it’s doing.

At this point, you can try writing data to the master database, and you should see it mirrored over to the slave. To check the replication status, you can run the following on the master:

Link

No one is “forced” into piracy

[...] if piracy is actually wrong, it doesn’t get less wrong simply because you can’t have the product exactly when and where you want it at a price you wish to pay.

This. Just because you can’t find the content you want in the form you’d like to have it, doesn’t mean it’s ok to steal it. [via Marco]

Link

Letterpress

A new word game from Loren Brichter (of Tweetie fame). If you love casual games, and you’re tired of Words with Friends, Letterpress is a lot of fun!

update: I’m ‘gregr’ on Game Center if you want to play!

MacBook Pro retina pricing

I usually like reading ReadWriteWeb, but I think this article this morning is just link bait (which obviously worked, because I’m linking to it):

Buying a MacBook Pro with Retina means shelling out at least $2,199 for a notebook with a 15.4-inch, 2880×1800 display. Top-end models approach $3,500!

By comparison, the cheapest 15-inch MacBook Pro starts at $1,799, with half the resolution and a different but roughly comparable set of features and specifications. (The Retina version is smaller and lighter but lacks a DVD drive, and uses expensive Flash storage instead of a slower conventional hard drive.)

Anyone who has used an SSD knows there’s nothing “roughly comparable” between the these two computers’ configurations. But we can help them with the math.

Let’s take the cheapest 15-inch at $1799. To get “roughly comparable”, we’ll need to upgrade the RAM to 8GB ($100), and upgrade to the same 256GB SSD ($500), for a total of $2399. You could add another $100 to upgrade to the 1680×1050 hi-res screen, but let’s assume you don’t want that.

To be fair, that machine will have a DVD drive and an ethernet port. So, let’s add those to the base retina MacBook Pro. Base model is $2199, add the USB SuperDrive ($79), and add the Thunderbolt ethernet adapter ($29), for a total of $2307.

So the retina MBP is actually cheaper – AND it comes with 1GB of video RAM, vs. 512MB in the classic MBP.

Is the retina MBP expensive for a laptop? Yes. But none of the 15-inch MacBook Pros are cheap. The new model with the retina display is actually quite aggressively priced, IMHO, as compared with the prior models.

On the MacBook Pro with Retina display

There has been much written about the new MacBook Pro with Retina display. I’ve had one for about a week; I’m not going to write a review, as I’m not sure how anyone could compete with this review…but rather I’ll just mention a few things I’ve noticed in using it for my work.

First, the retina display is quite striking when you use it with applications that have been updated with retina graphics. Most websites that have not updated their graphics don’t look good at all, as I said earlier:

The “blurry” effect is actually more noticeable on the MacBook Pro than it is on the iPad. So for the many, many folks I’ve heard saying “my site looks ok on the iPad, so I’m not going to worry about it” – my recommendation is take a look at it on the MacBook Pro, and make sure you’re comfortable with how it looks. Hint: it probably looks worse than you think.

There are two things that I’ve found a little painful at the moment when using the new MacBook Pro, as Joshua Johnson also noticed.

First, trying to edit 1x artwork on the retina screen is definitely a challenge. At the moment, you just can’t really tell how it’s going to look on a non-retina screen. Maybe a future update to Photoshop CS6 or Pixelmator or some other app will fix this; we’ll have to see.

And second, taking a screenshot on the retina MBP results in an image that’s twice the size you expect; for example, if you’re running at the “ideal” 1440×900 effective resolution, screen shots will be at 2880×1800. That’s great if you need a 2x screenshot to display on a retina display…but if you need a screenshot to display on a 1x display, you don’t have great options. You can downsize it in Photoshop or other editor, but you lose quality.

My solution to both of these problems has been to connect a regular non-retina screen for those tasks. This also has the advantage of letting me see quickly how things are looking on both retina and non-retina screens side by side, at the cost of being tethered to my desk…

All in all, the machine is beautiful, the screen is stunning when viewing high resolution content, and the machine is quite fast as compared to my other machines. I think the issues I mentioned above will probably (hopefully!) work themselves out as the software catches up with the display.

Glassboard 2.0 and Glassbot

The folks at Sepia Labs have released Glassboard 2.0 this morning. Lots of new features in the iPhone and Android apps…but my favorite part is the new web client.

When I first started using the web app a couple of weeks ago, it struck me as a game-changer for Glassboard. Where I would use the mobile apps for casual messages, with the web app I could keep it up on my desktop screen, and send messages and updates much more quickly. I found I was using it for real work.

Which got me to thinking, I’d like a way to automatically post content into a board. For example, maybe I’d like commit notices from Github to be automatically added to a board. Or maybe new support tickets. Or maybe just a way to have some fun with my friends – who doesn’t like seeing a coworker with a huge mustache?

So I created glassbot – a bot for Glassboard…and glassbot_recv, which allows you to post external content into a board.

If you want to try it, you can either:

  • Join the Glassbot Playground board with invitation code cfoli - once you’re in, post a message like “@glassbot help” (without the quotes) to see what he can do.
  • Invite the Glassbot to your own board! Just invite “glassyglassbot@gmail.com” to your board; he’ll accept within a few seconds, and then enter “@glassbot help” (again without the quotes) to see what he can do.

After you invite the glassbot, you can add a post-receive webhook from Github if you want to see notifications – use the URL

http://glassbot-recv.herokuapp.com/github/{BoardId}

as the endpoint URL, replacing {BoardId} with the ID of your board (you can see this in the Glassboard web app).

If you have ideas of things Glassbot could do, or you want to run your own, the code is available at glassbot (in Ruby) and glassbot-recv (Node.js) – add some new actions and let’s have some fun!

Because every board needs a bot.

UPDATE 06/12/2012 – the source code links have been removed at the request of the Glassboard team, while they are making changes to their API. The glassbot is still running, though, and you can still invite it to your boards. I’ll put the code back up when I get the ok.

Editing Canon 30p “PF30″ footage in FCPX

Based on what you usually see on this site, you might think came out of nowhere, but it took many hours to figure this out, and there is precious little information on the net with actual answers.

Canon’s current consumer and prosumer camcorders (HF M40, HF M41, HF M400, HF S30, HF G10, XA10, and others) encode in AVCHD, and have the ability to record in 30p. However, in the Canon specs, this is described as “30p progressive (records at 60i)”.

The common first-100-google-hits about this suggest doing one of two things:

1. Edit in a 60i (29.97i) interlaced timeline.

2. De-interlace the footage, and edit in a 29.97p timeline.

There are problems with both of these.

It’s not obvious, but it seems that the “PF30″ footage (as the Canon cameras call it in their configurations) is recorded as “Progressive Segmented Frame“. In layman’s terms (and believe me, I’m a layman when it comes to this), this means an entire frame is captured in the camera, and then it’s recorded into two separate fields in the 60i recording. The two fields, together, make up the frame.

Sounds a lot like interlaced footage, right? No. With true interlaced footage, half of the frame lines are recorded in each field…but each field is 1/60 second apart. So there are no two fields which, when combined, will yield a complete frame at a single point in time.

So this demonstrates the problem with both #1 and #2 above. If you edit in an interlaced timeline, you’re using footage that isn’t really interlaced. And if you de-interlace it, you’re close but you may not end up with what the camera recorded…and, it’s an unnecessary step, since the footage isn’t interlaced.

Ideally, the editor would detect this PF30 footage, and import it as 30p. However, most editors today do not. Here’s an article about this, re the Canon XA10. In Final Cut Pro X, importing this footage just using Import From Camera will show it as 29.97i.

So what to do? I’ve found two ways to get this footage interpreted correctly as 30p by Final Cut Pro X:

1. Use ClipWrap, a third-party tool (see article). Or…

2. Simply import into FCPX twice; do something like the following:

  • Use Import From Camera to import your footage into Event1.
  • Then, use Import Files, and import the files in Final Cut Events / Event1 / Original Media – and copy them into a new Event2.

When you import the second time, the files are just copied over – you can compare the timestamps if you want – so you’re not losing any quality. And after you do this, you’ll see the footage after the second import is shown as 29.97p…and will not need to render if dropped into a 29.97p timeline. Whew!

When I was trying to find the difference between the first and second import, I noticed the second had the Field Dominance Override set to “Progressive”. Aha, I thought…one could simply change this setting on the original first-import footage, and it will be interpreted as 29.97p. And at first glance, it appears to work. However, if you take this footage and put it in a 29.97p timeline, it will have to render, and this should be unnecessary. Just using a second import step eliminates this problem.

So anyway, hopefully this will save someone else a few hours of research!

Just another day at the office

Well, ok, maybe another day out of the office. :-)

The technical details…onboard footage was filmed in Bear Creek Canyon, just outside of Morrison, CO. It was shot with a GoPro HD; mounted via suction cup to the tank and to the side of the fairing. Other footage is from a Canon HF M40. The bike is my much-loved 2010 ZX-6R, which was badly in need of a bath!

Tradervue launches today!

Well, it’s been a while in the making, but today I’m very excited to announce the launch of Tradervue, a web application for active traders!

When I left my full-time position at NewsGator about a year and a half ago, I started actively trading equities intraday. Yep, one of those day traders. I was thinking “I’m an engineer, how hard can this be?” Ha! Turns out it was harder than I thought.

I spent some time searching for a trading methodology that worked for me, and one that specifically worked for my personality. I like instant gratification – I often use overnight shipping when I order things, I hate that my TV takes 10 seconds or so to warm up, and I like trading during the day where I’m not subject to the whims of the market overnight when I can’t do much about it.

I eventually settled into a rhythm, with the help of many smart traders I met online, where I was trading actively moving stocks that had some catalyst for moving that day (earnings announcement, fresh news, etc.), and I would watch the order flow and do my thing. I worked pretty hard at it – I was at the screens an hour before the market opened, would trade most of the day, and then a few hours of prep work at night for the next day.

I also kept a trading journal in Pages (a word processor), where I would write down why I was making certain trades, how I was feeling about it at the time (confident, anxious, etc.), and I’d paste in order execution data and charts from my trading platform at the end of the day. I’d review this journal at the end of the week, and try to learn from my successful and not-so-successful trades. All in all, this was one of the best tools I had for understanding my trading.

But I hated keeping it.

I didn’t mind writing in it – why I was taking a trade, what was making me nervous about it, etc. That part was easy, and pseudo-creative work. What I hated was having to paste in my execution data, and pasting charts into it from my trading platform. It ended up being about an hour of busy-work at the end of every trading day. Once I even caught myself not taking a quick trade because I didn’t want to add even more work to my after-close routine. Obviously not good; my very best tool for improving my trading was becoming so onerous it was discouraging me from trading.

On the advice of many experienced traders, I also made a list of trading goals for the year. For 2011, two of my non-P&L-related trading goals were a) continue keeping my trading journal, because I was learning a lot from doing it, and b) come up with a way to objectively analyze my data to understand strengths and weaknesses that might not be obvious. For the second item, my hope was to find a product that would just work for me; I looked around for a while, but never found anything that “clicked.”

So with these two things in the back of my mind, I set to work to build something, just for myself, to address them. Find a way to write in my journal, but have the busy work be automated. Find a way to load all of my trading data, and show me views of it I haven’t seen before. Show me what’s working. And show me what’s not.

As I was building this, somehow I got distracted and decided to refocus a bit, and build a web application that could do this for everyone. And so was born Tradervue.

As Tradervue was taking shape, in the back of my mind I was thinking about the trading community I had learned a lot from, and the traders that actively share their ideas online on Twitter, StockTwits, and their blogs. What I have rarely seen is traders sharing actual trades. I don’t mean the sensitive data like how many shares were traded, or how much money was made – that’s not important. Rather, things like where did you enter this trade? How did you get in when it popped through the price level you were watching, but then dropped 0.50 before moving higher? When did you start to sell? Questions like that. Execution is everything – and so perhaps show people how you executed.

As I thought more about this, I noted that Tradervue had all of the data necessary to share trades. The challenge was more a matter of figuring out specifically what information should be excluded and kept private, and then make it easy to share the more educational parts. Shouldn’t it just be a click or two to share a trade with the community, complete with charts and commentary? I thought so.

So I built the sharing into Tradervue. And combined with the trading journal capabilities (with generated charts) and the analysis it can do, allowing you to drill down as far as you want, I think it’s a pretty cool product.

There were beta users whose average session length was measured in hours, with no more than a few minutes idle during that period. It was quite amazing, and exciting; I’m even more excited to see where it goes from here.

So, happy birthday to Tradervue – today is its day!