Articles

Delaying File Processing & Uploading with DelayedJob and PaperClip

In delayedjob, paperclip, Rails, Ruby, s3 on August 11, 2009 by Matt Grande

I’m going to say it right now. I love DelayedJob. I’ve been working on a project that involves uploading assets, mostly images, to S3 and resizing the images to three sizes. With the addition of DelayedJob, this can happen much faster.

Before I go on, I should mention that I wouldn’t have been able to delay paperclip without this post, and I wouldn’t have been able to delay S3 uploads without this post. Thanks go out to both bloggers!

The original DelayedJob was done by tobi. I’ll be using the version created by collectiveidea, which has a few extra nice features.

So the first step is to install it. I had to install both the plugin and the gem. The plugin was unable to generate the database migration, and the gem was unable to run the rake task. Hopefully, it was either a mistake on my part, or they’ll fix it soon.

script/plugin install git://github.com/collectiveidea/delayed_job.git
sudo gem install collectiveidea-delayed_job

Collectiveidea’s delayed_job has a nice generation script which will make your delayed_jobs table for you.

script/generate delayed_job
rake db:migrate

You’ll also want to add a processing column to your asset table.

class AddProcessingToAsset < ActiveRecord::Migration
  def self.up
    add_column :assets, :processing, :boolean, :default => true
  end

  def self.down
    remove_column :assets, :processing
  end
end

Your next step is to move into the code. To prevent the S3 upload, we subclass our Asset model with TempAsset, and just save locally.

class TempAsset < Asset
  has_attached_file :media, :path => ":rails_root/tmp/uploads/:id/:basename.:extension"
end

In your controller, save your TempAsset instead of Asset, and call a method that will queue up the processing.

class AssetsController < ApplicationController
  def create
    @asset = TempAsset.new(params&#91;:asset&#93;)
    @asset.save
    @asset.queue_move_to_s3
    redirect_to @asset
  end
end
&#91;/sourcecode&#93;

From there, we want to make our <code>queue_move_to_s3</code> method.  I'm using <code>send_later</code> in this example, but <code>enqueue</code> works just as well.  While we're at it, we'll write the method that will perform the saving.


# In temp_asset.rb
def queue_move_to_s3
  self.send_later(:perform)
end

def perform
  asset = Asset.find(self.id) # This is the same db record as self
  asset.media = self.media.to_file
  asset.processing = false
  asset.save!  # This will re-upload & re-size your image as per your has_attached_file method in Asset

  path = self.media.path
  self.media.to_file.close
  File.delete(path)
end

One last thing you may want to do is have a “We’re Still Processing This File” image. Thanks to the processing column we added earlier, it’s a piece of cake.

# In asset.rb
def url(style=:thumb)
  if processing
    still_processing_image_path
  else
    self.media.url(style)
  end
end

There are unfortunately a few hoops to jump through to get this working, but in the end you have a much faster response time which leads to a much better user experience. Hope this helps!

Advertisements

5 Responses to “Delaying File Processing & Uploading with DelayedJob and PaperClip”

  1. Very useful post. Thank you.

    I am wondering how this might vary in my situation where I have a “Thing” model with a one to many relationship with an “Asset” model. The Asset model has the paperclip attachments. So each Thing might have several Assets (attachments) associated with it.

    I would be interested in potentially hiring you on a free-lance basis to modify this example for the scenario I describe above. How would I contact you to discuss?

  2. Hi Poul,

    In order to modify the above example to handle several assets, I believe you would just need to loop through the assets.

    In actuality, the above code does work with several assets. We’re using SWFUpload to upload multiple assets one at a time, and when they are uploaded they are sent to AssetsController’s create method.

    I am currently unavailable for freelance work in any sort of extensive capacity. However, I may be able to answer some specific question you have. If you’d like, feel free to email me at matt.grande@gmail.com

  3. Hi, I followed this guide and I get this error from the worker

    TempAsset#perform failed with Errno::EACCES: Permission denied – path here… – 5 failed attempts

    Have you any idea what is up?

  4. The blog with the post about delayed S3 uploads seems to have gone away, fortunately the wayback machine has a copy:

    http://web.archive.org/web/20100322203313/http://codewordstudios.com/posts/3-delayed-upload-delivery-to-s3-with-paperclip-delayed-job

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: