ActiveLabs blog

Installing redis (a crazy fast key/value store) for a website, I had a look to Resque, and give here the why I moved from the good DelayedJob. Here a short answer:

• Replacing a dirty cron / curl / controller setup to launch scheduled tasks
• Having a easy control on the scheduled tasks and failed jobs

Resque

Resque is a Redis-backed Ruby library for creating background jobs, placing those jobs on multiple queues, and processing them later.

For the application, the big differences are you cannot send ruby objects and you need to specify a queue.

Resque scheduler, a light-weight job scheduling system built on top of resque.

Quick installation

For full details: Resque on github

Redis:

curl -O http://redis.googlecode.com/files/redis-2.2.2.tar.gz
tar xzf redis-2.2.2.tar.gz
cd redis-2.2.2
make
sudo make install

Add to you Gemfile:

gem 'hiredis'
gem 'redis'
gem 'redis-namespace' 
gem 'yajl-ruby' 
gem 'vegas'
gem 'sinatra'
gem 'resque', '1.15'
gem 'resque-scheduler'

Add to your Rakefile:

require 'resque/tasks'
require 'resque_scheduler/tasks'

Configuration and authentication

Initializer for resque config/initializers/resque.rb:

Schedule configuration file config/resque_schedule.yml:

Let’s start

Resque workers:

COUNT=5 QUEUE=* rake resque:workers

Resque web interface:

bundle exec resque-web -e production -s thin -L config/initializers/resque.rb

And finally the resque scheduler worker:

rake resque:scheduler INITIALIZER_PATH=config/initializers/resque.rb 

To kill the resque-web:

bundle exec resque-web -K

Async

If you use DelayedJob and want a replacement for the delay/send_later method, here a simple async method:

Deployment

see the update.

Conclusion
With this setup I get a more robust system to manage jobs and scheduled tasks:

• The scheduler plugin, multiple queues
• Web interface, make easy to follow exception and what’s going on (DJ gets admin)
• Faster to insert thousand jobs without slowing the website db (DJ can manage a mongodb now)

Links:

• Resque on github
• Resque scheduler

update (02/04/2011): I ve just found a good post with full setup and capistrano deployment – setting up resque

UPDATE 30/03/2011: this article was started as a draft last year and unfortunatly published as is. As things changing quite fast in this part of the web, I update this article to keep only references to technologies. I m sure there a lot of more today.

For the website Pronostic Facile I m looking for technologies to provide real time information. I receive information of different kinds and want to dispatch it to users as soon as possible and make it available without the user reloading the page. This can be provided by a Pubsub architecture with a Comet layer.

Node.js

Node.js is the server technology build for this job. Scalable, asynchronous, it can handle a lot of requests as to receive and publish messages.

Faye

pubsub over http for node.js and rack, Faye

Nodule

a pubsub Key-value store, Nodule

Fanout.js

I needed a robust and scalable event-driven message fanout server for several Comet-based webapps where some process in the backend could send various messages of different types/channels to the clients (browsers) in the front-end. fanout.js implements such a server using standard TCP sockets so that it can be used in a wide variety of applications. Connection to web browsers is accomplished by using the TCPSocket abstraction in the Orbited.org project.

a simple and robust fanout pubsub messaging server for node.js, Fanout.js

Nodered

NodeRed is an extensible network service container for Node.js

Nordered

Express

Not really build for Pubsub, it s a good base to build a service on node.js. This framework try to bring to node.js all the power of other framework like rails and make development fast.

Express

Socket.io

A nice piece of software giving you server and client abstraction of sockets over technology and browser.

Socket.io client
Socket.io node.js server

I did look at XMPP and APE, but this solution is far simpler and is the new technology for real time application.

The move was longer than expected but it s now done, and my former typo blog has been moved to static Webby website. Here are the elements I used to migrate:

• posts: migrate_content_to_webby.rb
• comments to discuss: export_comments.rb
• using gist to embeded code and syntax color: gist.github.com
• tags system improved from http://aaronqian.com/
• ping from avdgaag

That’s about all you will need to start a blog. Later I will work on some tasks to publish best posts to twitter or rubycorner.

should be available soon on gihub…

I did it! After some days, I can move all the old comments to disqus and then link them to the webby posts. Here is the rake task:

export_comment_to_disqus.rb

Also you need to install the update disqus gem (API ver. 1.1) from my repo or link the lib, it fixes small things need to make the export works:

disqus lib

After 2 years of almost nonexistent posting (4 posts), I’ve moved this blog from Typo to Webby (as planned on Feb … 2008). The migration is on going, content was easy, other part of website, less. I will post later about the migration code.

Blogging from Textmate and Webby is cooool and my server says thanks.

TODO:

• migrate old comments to disqus done !
• code highlight (ultraviolet is too much to setup) (using gist)
• integrate tags (based on aaronqian.com)
• google search (could be improved)
• ping on posting (feedburner, rubycorner, etc…)

For some weeks now, one of my website has started to be heavy loaded and needs to move to a new home. Iy cost me 2 hard days last time to install the server, It’s boring to plan it again. Also I start to think about what could happen if the server crashes and if I need 2 days (less because I have written down the procedure) to get a new slice up. Here start my journey to find a simple solution to automate this deployment from scratch OS to a full running rails stack.

Common solution for big guys

• chef: used or promoted by EY and 37s’ it s the most popular, ruby syntax, quite Ubuntu oriented
• puppet: written in ruby, has it s own DSL, more open to other systems
• moonshine combine the best of the two worlds with a ruby DSL, but sadly it s opinionated and not easy to find recipes or to build recipes if you choose another road (example: nginx).

Currently using FreeBSD, I’d like to keep this config, that doesn’t seem to be an option in these solutions. Also they require a server to dispatch configurations as they are build for provisioning clusters and clouds solutions (chef supports a solo version).

to mention: puppetshadow, the DSL extracted from moonshine to use on puppet

Then I dug further to find some solution for the poor man’s server.

Capistrano

Since the beginning I feel like capistrano could do this job. I use it a lot to remote administrate and deploy application, then why not for setting up my server. And then i started to find some products doing it:

• carpet: still alive ?
• sprinkles with the passenger-stack
• deprec, old friend I’d like to see the evolution

Then I will try those 2 and give feed back later on.

Mixin

To mention also, initiatives built on sprinkles to start working with chef:

• chef bootstrap : to setup a chef client server
• sprinkle_chef same same but different

Take a look to solution as vagrant to setup a full VM stack with rails, quite useful for development and ensure every developer use the same environment. You can also build your own box.

If you know any other solution that can fit in this post (ruby, one server deployment, etc…), leave a comment.

I ve just spent 5 hours to understand what was going wrong with my ajaxy form. I use jquery on rails and ajaxForm plugin, I have some form already working good and tried to a new one.

But this time it doesnt work, I dont understand why my form send twice the request, first with ajax header and second time with html request. I even tried to write raw javascrit, livequery and so on. Nothing make it works. Finally I want to check a little thing I add on the form: ‘disable with text’ option. I disabled it and miracle it works now.

I ve lost 5 hour of my time for this little cosmetic tool that s mess up ajax binding.

Then my advice: DON’T USE “DISABLE WITH TEXT” OPTION WITH YOUR JQUERY AJAX BINDING.

I know it exists other information about this, but I want to summarize here the practise I use in several project now. It takes 4 steps:

• configure your database.yml
• create an abstract class for connection
• create new models within a module
• interact the 2 databases

1. Configure your database.yml

This is the same as you create production or development connection, just give it a specific name:

2. Create an abstract class for connection

Then create an abstract class you will inheritate by your new classes:

3. Create new models within a module

To make it easier to use and cleaner, create a subfolder OtherDatabase and then create in this folder your new classes (for each table you need) as following:

4. How to interact the 2 databases

Once you have created your new classes, you can simply access it using the module prefix:

Conclusion

That’s all, submodule is not really necessary but keeps your model folder cleaner. Any suggestion to improve this practice ?