Graphing Rails Performance With Scout

Posted in Scout | no comments

Derek

29

Jan

We’re using Scout, our monitoring and reporting application, to graph the performance of our Rails applications and servers.

I’ve uploaded a video that looks at how one of our applications, PlaceShout, impacts the server load and Mongrel memory usage. I also compare PlaceShout’s footprint to another server.

Watch the video: Graphing in Scout (1 min 47 sec)

Past Videos on Scout:

Installing the Scout Client (1 min 39 sec)

Installing the Rails Requests Plugin (1 min 55 sec)

Signup for our launch email list

We’ve started emailing invites to Scout. Signup on our homepage, and we’ll give you access to Scout before the public launch.

See PlaceShout, Other New Apps at SF Beta Tonight

Posted in PlaceShout | no comments

Derek

29

Jan

Andre and I will be at SF Beta tonight demoing PlaceShout, our short-form local reviews service.

We’re among 10 companies showing off their latest innovations (view the full list).

More information on tonight’s event is here.

Hope to see you there!

Making it easier to eat healthier

no comments

Derek

28

Jan

Better = more work. Tools that “make better easier” are rare.

I heard a radio advertisment for Safeway’s FoodFlex Program this morning. When you purchase groceries at Safeway and use your Safeway Card, Foodflex analyzes your purchases and suggests healthier alternatives. The program is accessed through the Safeway website.

Dieting often means counting calories. It makes dieting more work than not dieting. Foodflex can do the counting for you. Sure, Foodflex can’t track meal-by-meal, but you get a reasonable overview of your eating habits based on your grocery purchases. A couple of healthier suggestions a month is an easier, more sustanable path to eating better than religiously counting calories.

Web App Service Plan Subscriptions

Derek

22

Jan

We’re working on our subscription plans for Scout. While working on the plans, we took a look at what other subscription-based services are charging for access.

There are some definite similarities, but a couple of peculiar differences.

Below is a quick sampling of what we found:

* - All prices per/month
^ - Only applies to paid plans

All of the services but BlinkSale and Wufoo are dramatically more useful with more people. Why then does Highrise’s free plan only allows 2 users (initially it was a single user)? 37Signal’s older product on the list, Basecamp, has unlimited users on the free plan.

37Signal’s pricing model is widely copied (free/$12/$24/$49).

FamSpam and FogBuzz on Demand allow you to signup for a paying plan without supplying a credit card. I assume after 30 days you need to enter credit card information. It’s an interesting idea because sometimes you don’t know if you need an advanced feature until you try it. The tricky part about this strategy is handling accounts that don’t decide to pay. Do you suspend all functionality or just parts of the application that paid plans can’t access? How to you handle access to data they may no longer have access to on a free plan?

On the surface, CrazyEgg appears to be the most generous with their free plan. Many people may only want heat maps on a couple of pages anyway and how often the data is gathered may not be a major factor. There are a lot of people that could use their free plan perpetually.

Being too generous isn’t necessarily a good thing. It takes time and resources to support free accounts, and that can take away from enhancements and service offerings to paying customers. Blinksale may have the best balance. You get most of the features, and if you’re willing to pay for an invoicing solution, you’re probably sending more than 3 invoices a month.

Howto: PostgreSQL on Leopard gotchas

no comments

Andre

16

Jan

For a recent Rails project, I had to use PostgreSQL instead of the standard MySQL for the database. Setting up Postgres on Mac OS 10.5 has some quirks, which I will share with you here.

• There are three primary ways to get PostGRES onto Leopard: a pre-built binary, MacPorts, and source. My choice was to build with source. There are several good posts on building from source:
  • stiefels.net
  • developer.apple.com
  • brilliantcorners.org
  • forums.realsoftwrae.com

• I do not recommend creating a separate user to run postgres. Doing so clutters your startup screen with another user, and requires you to su to that user to start the database. Instead:

  1) install Postgres as root

  2) after install, from the command line: chown -R andre /usr/local/pgsql/data. The -r flag changes the ownership recursively. Obviously, substitute your own username for "andre" in the example.

• Since I don't use PostGres all the time, I don't want it to be started up automatically. Instead, I set up two aliases in by .bashrc:

  alias startpostgres='/usr/local/pgsql/bin/pg_ctl -D /usr/local/pgsql/data -l /usr/local/pgsql/data/logfile start' alias stoppostgres='/usr/local/pgsql/bin/pg_ctl -D /usr/local/pgsql/data -l /usr/local/pgsql/data/logfile stop'

• when you start postgres, you'll probably get the dreaded "shared memory error," which looks like this:

  FATAL: could not create shared memory segment: Cannot allocate memory DETAIL: Failed system call was shmget(key=1, size=1081344, 03600). HINT: This error usually means that PostgreSQL's request for a shared memory segment exceeded available memory or swap space. To reduce the request size (currently 1081344 bytes), reduce PostgreSQL's sharedbuffers parameter (currently 50) and/or its maxconnections parameter (currently 10).

• some resources will tell you to edit the /etc/rc file, which is incorrect for Leopard. Instead, create /etc/sysctl.conf (if it does not already exist -- mine didn't), and add the following lines:

  kern.sysv.shmmax=167772160 kern.sysv.shmmin=1 kern.sysv.shmmni=32 kern.sysv.shmseg=8 kern.sysv.shmall=65536

You'll need to su to root to create /etc/rc

Have fun with Postgres!

How popular are you? Find out with Scout

Posted in Scout | no comments

Derek

14

Jan

Tracking the results of your blatant self-promotion campaign can be a time-consuming effort. You might be using Google Analytics for web traffic and FeedBurner for blog subscribers. You’re probably checking link referrals. You’re querying the database for usage statistics (user signups, logins, etc.), etc.

Scout is an honest friend that gives it to you straight. Our friendly retriever will track, mash, and graph all of this data in real-time.

For example, below is a graph generated by Scout. It shows the FeedBurner circulation of this blog (in red) vs. unique visits from Google Analytics on our Highgroove homepage (in blue):

It doesn’t look like there’s a huge correlation there. What about unique visitors on PlaceShout (in red) vs. unique visitors on our Highgroove site (in blue)? Data via Google Analytics:

There’s a correlation there. Traffic to PlaceShout appears to drives traffic to Highgroove.

Currently, 3 Scout Plugins exist for grabbing external data:

• The number of page views and visits via Google Analytics
• FeedBurner subscribers and item hits
• Total entries that match a provided Yahoo! search term

Seeing this data is extremely useful for answering questions that take quite a bit of work to find out manually (and can’t be updated in real-time):

• How many of our unique visitors create a shoutout on PlaceShout?
• As the number of sites linking to us increases, how does this impact traffic on our site?
• How is traffic impacted when we publish our email newsletter?

The great thing about these reports is they don’t require any updates – Scout continually grabs new data and updates the graph.

Signup for our launch email list

We’re launching Scout this winter – click here to signup for our launch notification. We’ll email that list before the public launch.

Get Immediate Access to Scout

I’d like to create a plugin to report back the number of people linking to a url on del.icio.us, but haven’t had time yet. Want to create this plugin? Shoot me an email at (derek at highgroove dot com) and I’ll give you immediate access.

Past posts on Scout:

• Get notified of slow Rails web requests with Scout
• See Scout Run – 2 screencasts added

The Rails Way by Obie Fernandez

Posted in Community, Ruby on Rails | no comments

CBQ

11

Jan

Recently I’ve been reading Obie Fernandez’s book, The Rails Way. I made an appearance in one of the chapters, but the real credit goes to Obie for pulling together an exhaustive 850-page book.

It’s not for beginners, but if you’ve taken one of my Rails training classes, it’s a great reference book and next step.

Reviews of the Rails Way:

• has_many :through

• Ruby Inside (with some interesting comments)

Get notified of slow Rails web requests with Scout

Posted in Scout | no comments

Derek

07

Jan

When something bad happens you want to find out about it as quickly as possible.

You’re probably notified of uncaught exceptions. What about slow web requests, which can be just as annoying to a user?

Find about about slow web requests (and what might be causing them) in near real-time using Scout and the Ruby on Rails Request Monitoring Plugin.

Here’s how it goes down:

1. Install the Scout Client   watch a video (1 min 39 sec)

2. Install the Plugin   watch a video (1 min 55 sec)

...that’s it – you’re no longer a performance slacker.

3. Scout reports back data

Every 10 minutes, Scout collects information:

If you have a slow request, an alert is generated. You can view the offending requests and their request times:

View Web Requests on a Graph

You can easily graph this data as well with Scout’s built-in graph builder:

Compare to other Rails applications

...but that’s not all. You probably have multiple Rails applications. You can compare their performance on a single graph as well:

Look for trends against other data

...we’re not done yet though…how about comparing the average request time vs. the size of the mongrel threads through the Process Usage plugin?

Since it’s easy to plot different data stats on Scout, you can quickly rule out possible reasons for slow performance. That’s half the battle.

Signup for our launch email list

We’re launching Scout this winter. Signup on our homepage, and we’ll give you access to Scout before the public launch.

Geocoding Advanced Rails Recipes

Andre

04

Jan

I have a geocoding recipe coming out in Pragmatic’s Advanced Rails Recipes . The recipe is titled “Finding Things by Location,” and it offers how-tos and best practices on using my GeoKit plugin to, well, Find Things by Location. The most recent PDF of the beta book includes my recipe along with over 30 other recipes by community leaders like Chris Wanstrath, Dan Benjamin, and Geoffrey Grosenbach, to name just a few.

We Love Open Source

Posted in Open Source | no comments

James

03

Jan

I had an unusual Christmas Eve. While my wife was entertaining the family, I was in my office programming until nearly midnight. (Have I mentioned how much I adore my wife?!)

What had me so tied down at such an odd hour? One of the open source libraries Highgroove maintains: FasterCSV.

The Ruby core team gave me the nod to replace the standard CSV library with FasterCSV just before the Ruby 1.9.0 release. While a little more warning would have been nice, I was happy to do it.

I think it’s important to discuss why Highgroove, not just allows, but actually prefers that I spend some of my time maintaining our open source libraries, like FasterCSV.

The reason we created the library is nice and simple: we needed it. We import or export spreadsheet data at some point for most of our applications. We work with large data sets and we needed that to be fast. We also frequently use headers in the CSV files to map directly to database fields so a broader feature set to support such things was important to us.

Now we could have built it and just used it internally, but we decided to share. Did we do it because we are just really nice guys? Not really. (We are though, of course!) We did it because we wanted it to be great.

There are many subtle variations and edge cases for CSV. We debugged the library the best we could, but we’re human and we missed things. That meant our library would have trouble with some inputs. We (hopefully) would have noticed those eventually, but we’re just one set of users and by releasing it on the world we multiplied the user base by thousands. Needless to say early users found problems for us much faster than we could have alone. Better than that, they sometimes even fixed them for us! Even when they don’t, they usually send in an example that leads us straight to the problem which is almost as helpful.

That’s not all. Remember all of those features I told you we love? I wish I could brag and tell you I invented all of them, but some of my favorite features in FasterCSV were sent in by the users. I got the ball rolling with what I could see us needing, but the users ran with the ideas and made my library better than I knew it could be.

Highgroove isn’t losing hours on me applying a few patches here and there. They are gaining the help of many extra employees. We love that.

To top it all off, Matz gives us the ultimate Christmas present. By blessing our work, he saves us the trouble of even installing it for new projects. Very soon we will be able to count on our code being available in all modern Ruby installs. That’s just one less thing we need to worry about.

With open source everybody wins.