The Scariest Fun : notes from a web startup

We have been using Capistrano for awhile now and it has been working great. Recently, for reasons we are not sure of, migrations started failing. When we executed 'cap deploy:migrate', we would get an error “no such file to load -- rcov/rcovtask”. After a lot of investigation, I found that this meant that the system, for whatever reason, couldn’t find our gems anymore.

Since Capistrano hardly gives any information when failing, it took awhile to figure out what was going on. sshing in as our Capistrano user, and running printenv, showed that the environment variables for the Capistrano user were all set as expected. After some investigation, we learned that since Capistrano runs a non-interactive user, the environment isn’t set up the same. To figure out the environment that Capistrano was running as, I created a task and added it to config/deploy.rb

desc "Echo environment vars" 
namespace :env do
  task :echo do
    run "echo printing out cap info on remote server"
    run "echo $PATH"
    run "printenv"
  end
end

After saving your deploy.rb file you should be able to execute that command on your servers by running “cap env:echo”.

This printed out a different path than sshing in as our Cap user and also was missing the RUBYOPT=rubygems environment variable. Obviously, not having RUBYOPT set was causing the problem of not finding out gems.

So next, we just had to set up the correct environment for the non-interactive Cap user. This was a little tricky since non-interactive users don’t load all the environment settings like normal users. This is done for extra security. Unfortunately, it means it is a little more annoying to set up these variables, and not as well documented.

If you need to add environment variables or change the PATH for a non-interactive user, you need to edit two files. First ssh in as the user under which the non-interactive scripts will be run. Then, in that user's home directory, create a file ~/.ssh/environment (in our case /home/cap/.ssh/environment). Edit this file and add whatever environment variables you need (in our case RUBYOPT=rubygems), and save the file. Then edit /etc/ssh/sshd_config, and add the following line

PermitUserEnvironment yes

I would run 'man sshd_config' to make sure your version of sshd supports this option, as some older versions and distros (we are using Debian) do not. Then restart ssh by running “sudo /etc/init.d/ssh restart”. At this point, you should try running “cap env:echo” again, which should print the RUBYOPTS environment variable set correctly. This fixed our problems and migrations began running again without a problem.