ActiveLabs blog

For some weeks now, one of my website has started to be heavy loaded and needs to move to a new home. Iy cost me 2 hard days last time to install the server, It’s boring to plan it again. Also I start to think about what could happen if the server crashes and if I need 2 days (less because I have written down the procedure) to get a new slice up. Here start my journey to find a simple solution to automate this deployment from scratch OS to a full running rails stack.

Common solution for big guys

• chef: used or promoted by EY and 37s’ it s the most popular, ruby syntax, quite Ubuntu oriented
• puppet: written in ruby, has it s own DSL, more open to other systems
• moonshine combine the best of the two worlds with a ruby DSL, but sadly it s opinionated and not easy to find recipes or to build recipes if you choose another road (example: nginx).

Currently using FreeBSD, I’d like to keep this config, that doesn’t seem to be an option in these solutions. Also they require a server to dispatch configurations as they are build for provisioning clusters and clouds solutions (chef supports a solo version).

to mention: puppetshadow, the DSL extracted from moonshine to use on puppet

Then I dug further to find some solution for the poor man’s server.

Capistrano

Since the beginning I feel like capistrano could do this job. I use it a lot to remote administrate and deploy application, then why not for setting up my server. And then i started to find some products doing it:

• carpet: still alive ?
• sprinkles with the passenger-stack
• deprec, old friend I’d like to see the evolution

Then I will try those 2 and give feed back later on.

Mixin

To mention also, initiatives built on sprinkles to start working with chef:

• chef bootstrap : to setup a chef client server
• sprinkle_chef same same but different

Take a look to solution as vagrant to setup a full VM stack with rails, quite useful for development and ensure every developer use the same environment. You can also build your own box.

If you know any other solution that can fit in this post (ruby, one server deployment, etc…), leave a comment.

I know it exists other information about this, but I want to summarize here the practise I use in several project now. It takes 4 steps:

• configure your database.yml
• create an abstract class for connection
• create new models within a module
• interact the 2 databases

1. Configure your database.yml

This is the same as you create production or development connection, just give it a specific name:

2. Create an abstract class for connection

Then create an abstract class you will inheritate by your new classes:

3. Create new models within a module

To make it easier to use and cleaner, create a subfolder OtherDatabase and then create in this folder your new classes (for each table you need) as following:

4. How to interact the 2 databases

Once you have created your new classes, you can simply access it using the module prefix:

Conclusion

That’s all, submodule is not really necessary but keeps your model folder cleaner. Any suggestion to improve this practice ?

Huh, seems simple to do :p, but I’ve been seeking for a while to find the answer and dig into rails code to find a simple method to retrieve the controller of a given path (could be useful to build a navigation). There it is:

ActionController::Routing::Routes.recognize_path(path)[:controller]

Ebb is new server to put in front of a rackable ruby application.

Have a look to the Ebb website to get some comparison with other app servers.

sudo gem install ebb

What I needed to do before on Mac OS X (leopard with MacPorts):

sudo port install glib2

This the third and last part for a summary about solutions to help better computing of your rails app

• 1st part.
• 2nd part.

Some home made benchmarks

NB: Following benchmarks have been made to a quick look study and not for a proof of concept.

The test machine (dedibox, ubuntu 6.10) is 1 core cpu @2Ghz – 1024 Mo memory. In each case the http server uses 10 processes on a real world application (not just ‘hello world!’). Sending 500 rq with 1 or 10 concurrency req:

Since last time, Rails world has a new nice app server: thin. Replacement of mongrel, based on rack interface, it’s also recently added a unix socket connector where usually we use IP connector. For better performance, avoid longest stuff as scripts (google analytics). Other scripts/images/css are keys in http performances also, usage of assets in recommended.

As we can see in equivalent configuration there is no major difference for http server. Thin seems to be definitly faster than mongrel when you need to server more than 1 request at a time. And of course on a 1 core cpu, having more than one app server doesnt make the content served really faster.

Memory usage

Mongrel (48 Mo) → Thin (40 Mo) less 15%

Apache (10 Mo) → Nginx (1.4 Mo) less 85%

Finally I decided to move to a nginx / thin on socket configuration because of the memory usage and the fair performances. I even moved my PHP stuff with an fcgi connector.

Other ideas of tests: Varnish, Nginx and memcached. Maybe next time :)

[update]

Please find a good benchmark about nginx vs apache on Joe’s blog

Overall conclusion

I think when we need to compare two framework, about which one is faster, we should keep in mind all these levels of optimisation to scale an application, and then the raw performance is a secondary issue.

If like me you have an error after installing scrubyt 0.3.4 (in console and launching mongrel):

can't activate RubyInline (= 3.6.3), already activated RubyInline-3.6.4]

I spent a lot of time to find the solution, and so easy solution … anyway could be useful for people googling about this. You just need to unsinstall the lastest installed version of RubyInline:

sudo gem uninstall RubyInline -v 3.6.4

Everything should be fine with this, even Scrubyt requiring version 3.6.3.

This the second part for a summary about solutions to help better computing of your rails app (1st part).

Application

To help your application to focus on replying fast to user, you can use a distributed queueing to give the job to other processes:

• Daemons
• Backgroundrb

And second thing, you can cache things you don’t need to compute for each request, this can incredibly increase the response of your server, but need to modify your code and it s recommend for very big sites:

• Memcache client, memcached server
• acts_as_cached

Back end

DB

For your database you can find some sql proxy, but I haven’t tried it yet. You have also the way to replicate your databse and serve data from several servers.

• sql relay
• use jruby with a jdbc pool connection and caching

Important things you have to check about your databse is if you have the right indexed fields. You should find some tools on mysql helping you to find where indexes missing. Check about cache too.

A last way is to serialize some of your data in a single object.

Mysql

Otherwise you can configure your mysql with some optimized configuration (mysql.ini):

• skip-name-resolve
• query_cache_size = 52428800;
• query_cache_type = 1

And in your ActiveRecord object you can use the :include parameter.

Filesystem

Optimizing the filesystem, in other sense than choose a good OS, can been done with not overloading the management of files by using some rules:

• max 10K files / subdir
• 16 top level / 256 sub / 10K per sub dir
• use a hasher to manage files

OS

As we have seen, file and memory management are keys of good performances. And these are managed by OS. The two best OSs mentionned are

• Sunsolaris
• FreeBSD (can be use for a all in one stack), avoid the version 5

And upcoming, last part of this article, some home made benchmarks for my own purpose.

One of the most interesting topic in the RailsConf was Scaling your Rails app. There are 2 ways of doing it:

• adapt your code
• adapt your servers

But the second one will not help to make your code faster. It will just help to give maximum resources to compute it right.

Summaries from the conference (sorry for the so few litterature it will be only a summarize):

Some theories

These theories are for big websites with millions of hits every month. This can involve thousand of servers, different location, … BTW some tips can be use to help smaller configurations.

The 7 layers to be considered, a la OSI.

• DNS
• frontend
• proxy
• application server
• application
• backend (Db , filesystem)
• OS

DNS

Nice website to check your DNS configuration: dnsstuff.com
Good tool to optimize your DNS: speed, dispatching, and so on…: powerdns

DNS can be quite fast to address the right server from a request, scripts allow to manage complex configurations. The tip is to use 1 DNS entry per 1 controller, then managing part of the routing.

Frontend and Proxy

Load balancing is about dispatching requests taking in account the charge of each cluster.

Load balancers:
Varnish
HAProxy

Both look promising and HAProxy seems to be more and more popular.

Front end with load balancer:
Nginx : fast, low memory usage, http server with load balancing, easy configuration, very good in front of mongrel cluster.
Apache 2.2 + mod_proxy_balancer, you can find how to setup it in previous post: apapche, mongrel cluster, rails on a load balancer

Application server

• mongrel server: enough for a personnal server
  gem install mongrel
  mongrel_rails start (more information)
• mongrel_cluster: useful to scale up your configuration, need at least a multi-processor or multi server environnment.
  gem install mongrel_cluster
  mongrel_rails cluster::configure …
  mongrel_rails cluster::start
• Evented Mongrel (swiftiply): it’s a mongrel based an event triggering
• jruby / glassfish: the technology java comes with some good environment to speed up part of your application like connection to database (jdbc, pool connection)

to be continued (application, backend, OS, and some home benchmark)

Nginx is a ‘small’ efficient http, load balancer server. More and more popular in Rails community, it’s a good front end for mongrel clusters. It’s seems to be a good replacement for apache, at least for the memory usage (on my test: 1 process use 1.3M against the more than 9M taken by 1 apache process). Benchmark on one single server are quite similar.

The process has been tested on debian 4.0r1 and should work on Ubuntu.

Installation

apt-get install nginx

or, if you want an up to date product:

1/ Prepare your system

apt-get install zlib1g-dev libgcrypt11-dev libpcre3-dev libssl-dev

2/ Download source

tar xzf nginx-0.6.12.tar.gz
cd nginx-0.6.12

3/ Compile it

./configure --pid-path=/usr/local/nginx/logs/nginx.pid \
	--sbin-path=/usr/local/sbin/nginx \
	--with-md5=/usr/lib \
	--with-sha1=/usr/lib \
	--with-http_ssl_module \
	--with-http_dav_module --prefix=/usr \
    --conf-path=/etc/nginx/nginx.conf \
    --pid-path=/var/run/nginx.pid \
    --lock-path=/var/run/nginx.lock \
    --error-log-path=/var/log/nginx/error.log \
    --http-log-path=/var/log/nginx/access.log 

make

sudo make install

Configuration

The following configuration is for 1 nginx server with load balancing 3 mongrel in cluster:

1/ Mongrel cluster configuration

gem install mongrel_cluster -y

cd /var/www/my-app-path/current/

configures 3 mongrel in cluster, in production mode, starting at port 5000 and listening on localhost

sudo mongrel_rails cluster::configure -e production \
    -p 5000 -N 3 -c /var/www/my-app-path/current/ -a 127.0.0.1 \
	--user www-data

mongrel_rails cluster::start

2/ Nginx

nano /etc/nginx.conf

Use the nginx.conf file from brainspl.at website, change the 2 or 4 (if you use ssl) references to the path of your application (/var/www/my-app-path/current/)

/etc/init.d/nginx start

All these information come from many website (thanks to all), I put them together for my own usage, hope it’s helpfull.

Following the conference, and some information about scaling Rails application, I decided to investigate about it and write a post on it.

First, I tried to install FreeBSD as recommended for small configuration… Well, I still don’t get it, I’m not fan of this installation and stop trying to understand, instead I installed a Debian 4.0R1, not so bad and closer to what I know :)

Secondly, I want to try Nginx. First, need to find the good one to install on Debian, not last dev, maybe the one on the testing repository. Finally I found package with the 0.5.31, latest stable version. Easy to install with dpkg, I only wanted to simply configure static server and point on php application I have. Ouch, the php-cgi stuff is pretty much more complicated than expected, I had a look to a Rails configuration and gave up :p.(update: configuration of one server is not so complicated, at least for rails, not more than load_balancer in apache, but it is for php and if you want to dispatch URL on the same domain, it doenst seem trivial)

Then I went back to something I can handle: apache. I have read some article about 2.2 you can use with load_balancer and several mongrel instances. Let’s go for it. I found a very complete article: Time For A Grown-Up Server: Rails, Mongrel, Apache, Capistrano and You used also by mongrel website about Apache Best Practice Deployment.

As I faced some issues, please be careful about the following:

1/ Mods to activate:

a2enmod proxy
a2enmod proxy_http
a2enmod rewrite
a2enmod deflate
a2enmod headers
a2enmod cache
a2enmod mem_cache
a2enmod file_cache 

2/ Forbidden access ?
If you have some issue with forbidden access please check your proxy conf file and replace

Order deny,allow
Deny from all

by

Order allow,deny
Allow from all

Enjoy!

I’ll try again Nginx and maybe also pen / pound and litespeed. A buzzy load balancer: HAProxy. A lot of work is coming :p. If you have some feedback about these configuration don’t hesitate to share it :)

And the winner of 68 votes is ….

acts_as_authenticated with 47,06% of the votes

followed by

• salt / sugar systems, 35,29%
• other…, 13,24%
• openid, 2,94%
• auth_generator, 1,47%

Congratulations to the winner, it is a very nice and neat plugin. BTW at the RailsConf, we had a demonstration of an HTTP_AUTHENTICATION as a new feature of Rails 2.0. More’s coming…

The conference has just finished, and the first thing to say “it’s a must go conference for all of you developing with Rails”. These 3 days were plenty of good information and teaching. You can read a lot of things on feeds or blogs, train yourself about every thing in Rails but nothing worth the real contact with the community. 500 750 participants but only 2% of french people “mais ou est la communauté francaise rails ?” and 95% of Mac users :D. Berlin is quite pretty city to visit, that’s was a good plus :).

I will write a serie of posts about some interesting conferences I attended to :

• Scaling rails app
• Security (website)
• Capistrano 2.0 (you should use it)
• RSpec and its stories (good ideas, the best is coming)
• Tavnav and other italian widgets (DRY)
• Rails 2.0 RC (not a revolution, but many nice evolutions)

and maybe another one about feedback of business creation around rails app.

Other subjects I have watched but will not write about:

• Good practises (no big stuff)
• Ferret (previous post)
• Presenter concept (maybe a good idea but not convinced by the presentation)
• REST (already covered by many posts)

I have been disappointed by one thing, there weren’t any RadiantCMS talk :(.

Link_to: presentations and descriptions

Congratulation to the team of Radiant CMS they have released a great piece of code with this version:

$ sudo gem install radiant

Announce

List of changes

Enjoy !

For my training on Radiant I did a small extension to manage config parameters, usually accessible with the console or sql.

installation:

svn co http://www.rubynaut.net/svn/radiant/extensions/config_editor/ config_editor

Done.

I liked:

• the api to add your admin controls
• an asbtract model controller to help in managing your model’s CRUD with only few lines.
• the good practises you can get from the Radiant code :)

Creating an extension with Radiant is so easy! I guess this system will become very popular, and it will give ideas to others :p. With this, Radiant become a very good base to create any kind of website.

PS: the extension has now a tag <r:config key="key"/> to access a value inside your page.

As I’m new in Radiant I will not explain every thing I don’t know about it, only the few first steps to create your pages. If you want more information please refer to the links at the bottom of the page.

please follow the guide

I use Typo for my blog for some month now and before for other blogs. For its purpose of blogging it is a good tool, and written with Rails :). As Mephisto I think both inherit of the Rails’ fashion. The past week I tried to implement inside Typo a small poll system I have developed in standalone application. I really got surprised by the complexity of Typo, and the lack of interface for extension, then I tried quickly its brother and it was maybe better, but so complex. We are far from the 20’ blog’s demo. These two blogs are made by great people knowing very well Rails then I guess the code is good and can handle every situation, but can’t we do simpler ?

Radiant 0.6.0

Radiant 0.6.0 is going out today. Don’t look at the version number (last stable was 0.5.2), this new version is a big change. I have tried it before but was not happy with the behavior system. Why add another concept to MVC? Why the basic content should be a page?

In this new version the concept of behavior disappears, I let you check on the website what is different. “Instead” there is a great extension system. You have an interface between radiant and extension to declare it (that is IMO a minimum to make a good system) and then you can access the Radiant API to easily extend it, by example, add a new tab in admin interface (try this in Typo ;) ).

I like:

• a clear and simple coding
• a neat admin interface
• a great extension system!

I don’t like:

• the concept of Page, website is not only pages but many kind of contents as files, folders (or node). Even if extensions bring Assets, the top class should be Content instead of Page.
• the single level of snippets and layouts (only to complain about something :p)

Link

• RadiantCMS website

To celebrate my new poll sidebar, I would like to ask everybody which authentication system do you use for your projects?

Please feel free to comment your answer :)

survey on the right side =>

First (and last?) sidebar I’ve made for Typo: a poll system. The code it’s at its early development but works. I started by a plugin, I found almost what I need then I wanted to integrate it to my Typo blog… It was more difficult to integrate this piece of code than to type it. And sadly I discovered that Typo is not really good to be extended, at least since you want more interaction between your sidebar with user than from a static page.

Poll sidebar

Create a plugin on rails is the best way to share a features or a code between applications and with others. I found a lot information about how to extend or mixin ActiveRecord, but no information about how to provide a piece of application. Maybe I do wrong but that’s how I did:

Goldberg

To continue the topic of builders, here we have a web site generator, including user authentication, security, and site navigation, provided by a gem:

$ sudo gem install goldberg_generator -y
$ rails _website_name_
$ cd _website_name_
$ script/generator goldberg _template_

where _template_ is one of the following template:

• default
• ewnf
• snooker
• spoiled_brat
  configure config/database.yml then

$ rake goldberg:install

One interesting thing is the possible use of FCKeditor.

Links

• Goldberg

If you don’t want to lost your time (about 2 hours for me), don’t understand why your change in your model are not taking in account, and then understand there with a thread that’s cannot be killed then block you model, create the ferret index at startup

In your environment.rb:

The last few days I found different solutions to use textile in Rails application. Let’s have a look to the different ways.

Requirement: RedCloth installation

First you will need to install RedCloth (generally already needed to install Ruby and Gem)

$ sudo gem install redcloth

Few month ago I started posts on admin tools for Rails and advanced ruby on rails to help in applications. This week end I found some frameworks doing that stuff: Application builders, that means things are on going.

hobo

• hobo
• hobo screencasts
• hobo quick start

The most “alive” and completed builder. It provides:

• assitant for model creation
• user management, roles
• an interface to manage models

If you can do with the default theme, this builder is a good start, the new website is nice and the you can find help and demos about the framework.

[EDIT: there is a long article on oreilly gmt]

ajuby

ajuby

It’s another try for an Application builder but I haven’t found any activity for the last month. Is it still alive ? [EDIT: see the comments]

Conclusion

I really think this kind of Application Builder are great for developers. Think about this, Rails is a great framework already, make development faster, but when you develop an entire Application from scratch it’s not really DRY, has you have to set up again and again basis of your business needs, as user management. With Application builder don’t need to repeat these steps and you have every things ready to work. It s going further than Rails then make development very Rapid.

links

• hobo
• ajuby

As linked in the last digest, the Craig Ambrose’s Redbox is a Rails plugin inspired by Thickbox. It provides a way to show box overlaying your page.

Example:

Not fully documented, I’ll give you the way I have used it. Before starting, install the plugin:

script/plugin install http://www.tinboxsoftware.com/plugins/show/redbox/

First you need to verify your CSS and if, same me, your application not cover all the visible page of your navigator you should add the following in your CSS:

html {
height: 100%;
width: 100%;
}

body {
height: 100%;
width: 100%;
}

This code allows the overlay to cover all the page. Verify also you have the default javascript loaded and add Redbox CSS and javascript:

<%= stylesheet_link_tag ‘redbox’ >

<= javascript_include_tag :defaults >

<= javascript_include_tag ‘redbox’ %> 

Next you need to create hidden div for each box you need. The name of the div must follow the Redbox convention: hidden_content_actionnamecontrollername. The default style have to be display: none;.

admin"
style=“display: none;”>

Once you have put this code in your layout, you can activate your overlay with the following code (in my case it was a call to an action rendering a piece of html in return):

<%= link_to_remote_redbox ‘link text’, :url => {:controller => ‘controllername’, :action => ‘actionname’} %>

That’s all, Redbox afford other ways to call box don’t hesitate to look at the demos.

When you install redbox, it copies css and javascript to the public folder. You can configure the CSS to modify the transparency of the overlay as you wish.

Hope this article would be helpful

• Redbox

Ok, I should have done my home work last week end :). Let’s have a look for what I have found these last few days :

• erubis: it’s a VM for ruby written in ruby. Rails is supported in the latest version. Have also for java-er
• ActiveScaffold: do you know about AjaxScaffold ? ActiveScaffold is the new version (v4) of this plugin. Don’t need to generate controller anymore, every thing is in methods.
• <a href=“http://www.craigambrose.com/projects/redbox”: if you are looking for some nice web2.0 features as the thickbox in your rails application look at this plugin. I will try to do an article soon about its use as it ’s not fully documented
• acts_as_authenticated plugin: great plugin for managing users and authentication. Has many other plugins use it for authorization by example. Excellent documentation on the wiki
• sitemeter: another great plugin, easy to install, let’s you have statistics on your rails application. Have a look.
• globalize: not yet tested, a plugin for i18nalize your application

Needed for a little project, I found a plugin: “acts_as_rated”.

“Adds rating capabilities to any ActiveRecord object.
It has the ability to work with objects that have or don’t special fields to keep a tally of the
ratings and number of votes for each object.
In addition it will by default use the User model as the rater object and keep the ratings per-user.
It can be configured to use another class, or not use a rater at all, just keeping a global rating”

Not yet released as gem please checkout the following trunk :
svn://rubyforge.org/var/svn/acts-as-rated/trunk/acts_as_rated

Installation:

script/plugin install svn://rubyforge.org/var/svn/acts-as-rated/trunk/acts_as_rated

Generate the Rating model:

The rater is User (by default) and rated object is Photo

script/generate migration add_rating

rake db:migrate

New objective for this blog: summarize the most interesting articles / products I found or rediscovered during my week of surf.

For this week :

• simplelog: Not new for most of you but I really discover this new rails blog system, if you dont know it yet, I warmly invite you to check about it.


• boxroom: I was looking for a tool to store my files online, I found one not so bad :-).


• pastie: Simple, nice, just have a look and use to this web2.0 paste tool.


• ruby search: An excellent tool for developers to find any detail about ruby and rails APIs.


• rb-appscript: You have a Mac, know ruby and don’t want to bother with Applescript, use this.


• railsdav: still in the same way for sharing files, a start of plugin to use WebDAV over rails


• beast: A rails forum, stay tuned, version 1.0 should go out soon.

If you are insterested in CMS and Ruby On Rails, the project Radiant is going to publish a new version: 0.6.

You can try it following the instructions :
http://lists.radiantcms.org/pipermail/radiant/2007-February/003395.html

Please have a look as the upgrade of version number is far to relate the numerous changes in the project.

I know it s a bit late, but the version 1.2 of Ruby On Rails has finally got out on 18th of january.

Have look here…

Ruby on Rails Backend

Have you ever though to have quicker site setting up with Ruby on Rails ? Each time I want to start a project I do design first same things: a backend.

• Manage your data
• Authentification system

And for all that you need of course a nice layout. Do you know something can do that out of the box ? let me know, untill I find this I will try to make something like that.

Zope on Rails

To go further, you can add some good stuff like :

• Workflow / machine state system
• A mini CMS with a flexible content with inline editor
• Indexing engine, tag system
• Navigation
• Cache
• eCommerce system with checkout and payment system
• Upload system / File system Handler
• Widgets

Most of these features exist. Then It s possible to have an extended framework with Ruby on Rails, just have to be done.

When choosing to develop with RoR instead of django, only one thing was missing : an admin interface. In django when you generate a project you have a small interface to manage your data. In Ruby on Rails you can use the magic scaffold but it’s not very convenient.

The first initiative I have discovered was ajax scaffold generator, an advanced scaffold with some ajax stuff and a real way to integrate an admin interface, but you still have to do it.

Then I found two other projects:

• auto-admin generating a django like interface
• streamlined a more completed generator

I let you play with these three different projects, for different using.

I have currently an ubuntu dapper running ruby, rails and gem.

As gem will need to compile some stuff you should do :

$ apt-get install build-essential

Then at first try I had an error :

Building native extensions. This could take a while…
extconf.rb:1:in `require’: no such file to load — mkmf (LoadError)
from extconf.rb:1
ERROR: While executing gem … (RuntimeError)
ERROR: Failed to build gem native extension.

I found my ruby installation was not complete, then you shall install following packages :

$ apt-get install ruby1.8-dev ruby1.8 ri1.8 rdoc1.8 irb1.8 /
libreadline-ruby1.8 libruby1.8

Installation of typo also needs sqlite3 (you will see, by default the installer creates a database and launchs mongrel)

$ apt-get install libsqlite3-0
$ apt-get install libsqlite3-dev
$ apt-get install sqlite3

Install the ruby sqlite debian packages.

$ apt-get install swig
$ apt-get install libsqlite3-ruby

Now it should work fine :)

$ sudo gem install typo —include-dependencies

$ typo install directory

At this step, the typo installer creates a directory with a full version of typo and embedded rails, creates a database and launch the blog behind mongrel.

To configure Apache, you can find in installer directory and example of the config file need to put in apache. For my configuration with Apache 2 I did the two commands :

$ ap2enmod proxy
$ ap2enmod proxy_http

and change the servername and proxy pass instruction :

ProxyPass / http://localhost:4434/
ProxyPassReverse / http://localhost:4434/

Where 4434 is the port configured by typo during installtion. Now just reload apache and you should see your website with a beautiful azure typo.