Mascote

notes to self.

The Big Rewrite, Porting a Legacy Application to Rails

  • set your development database to a production database as readonly.
  • dump a schema of your current database using seed_dumper.
  • create a simple model for each table.
  • use safe_attributes to deal with columns with reserved names (ex: hash, type).
  • annotate your models with annotate_models gem.
  • run a desc db.table and paste the result inside the corresponding model.
  • ensure ActiveRecord detect your database columns datatypes correctly or in a compatible (ex: generating the same result).
  • start unit testing your model for the NOT NULL attributes and add validates_presence_of :column_name.
  • configure a code coverage report tool like simpleconv.
  • add self.table_name = 'old_table_name' to each model if not matches rails conventions.
  • add belongs_to according your memory and add tests checking assert #respond_to? :entity not assert #respond_to? :entity_id.
  • spin a rails console help with the tests.
  • add routes test, create route.
  • test your controllers at the boundaries, same input same output.
  • ensure the database are with the same behaviour (ex: NULL, 0).
  • deal with the error paths, return same errors (ex: request with empty or incorrect data, invalid api keys).
  • implement the old cron tasks as a batchable job, execute as a proccess using resque or sidekiq.

[draft]

Setting Up a Load Balancer Using Apache 2.4

UPDATE: Actualy my point about 2way ssl just on one location is wrong here. There is no sense in send the CA just to one location because of the basics of the HTTP protocol, first the connection is established, then the HTTP method is issued, so sending the CA is required at the SSL handshake.

The load balancing part still valid anyway. :)


On a high available architectures is common to use HAproxy to load balance request accross a set of application servers, sometimes we Sysadmins need to setup not so common features where we need to be aware about all others software in the stack, for example when you need to secure an expecific URI using 2way/mutual authentication where is used a client ssl certificate to authenticate access to a determined service.

This setup can be tricky because there is some pitfalls when using HAproxy and Apache:

  • you need to pass the ssl traffic through tcp to the backends
  • have to configure ssl directly on each application instance
  • use the PROXY protocol to have the information about the client ip address
  • deploys can break existing established connections

I can't setup 2way ssl directly on HAproxy because the verify option is global at the listen directive, neither on Nginx because they don't provide the possibility to have diferent verify levels on diferent locations.

To avoid these quirks and keeping my configuration as simple as possible I discovered the load balancing option on apache documentation allowing me to setup my virtual host with both 2way ssl and load balancing as shown below:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
  <VirtualHost _default_:443>
    ServerAdmin [email protected]

    DocumentRoot /var/www/foo

    LogLevel error

    ErrorLog ${APACHE_LOG_DIR}/error.log
    CustomLog ${APACHE_LOG_DIR}/access.log combined

    SSLEngine On
    SSLProxyEngine On

    SSLCertificateFile    /etc/apache2/ssl/cert.pem
    SSLCertificateKeyFile /etc/apache2/ssl/key.pem
    SSLCACertificateFile  /etc/apache2/ssl/ca-combined.pem

    ProxyPreserveHost On

    <Proxy balancer://default>
      BalancerMember http://app01:8080
      BalancerMember http://app02:8080
      BalancerMember http://app03:8080
      BalancerMember http://app04:8080
    </Proxy>

    # <Location /my_secure_location>
    #   SSLVerifyClient require
    #   SSLVerifyDepth  1
    #   SSLRequireSSL

    #   ProxyPass balancer://default/
    #   ProxyPassReverse balancer://default
    # </Location>

    <Location />
      ProxyPass balancer://default/
      ProxyPassReverse balancer://default
    </Location>

  </VirtualHost>

This configuration combined with Ansible loops provides a good automation strategy also keeping backend and load balancer configuration simple.

I'm using with the evented apache module to have the same benefits as the non-blocking Nginx reactor pattern, so is pretty fast.

I provided an Ansible role with the complete configuration on this repository.

Controlling HAProxy With Socat for Fun and Profit

One nice thing to do when hacking some deploy/provision tools is the hability to provide rolling restart of backend servers without affecting existing traffic.

Ensure you have the stats socket enabled on your haproxy.cfg.

1
# echo "disable server www-backend/web12" | socat /run/haproxy/admin.sock stdio

Your server web12 from backend server group www-backend will enter on the maintenance mode, you can see with:

1
2
3
4
5
# echo "show stat" | socat /run/haproxy/admin.sock stdio
...
www-http,FRONTEND,,,0,6,2000,18,17584,39934,0,0,0,,,,,OPEN,,,,,,,,,1,2,0,,,,0,0,0,4,,,,0,5,8,2,4,0,,0,4,19,,,0,0,0,0,,,,,,,,
www-backend,web12,0,0,0,1,,15,13936,39082,,0,,0,0,0,0,MAINT,1,1,0,,,,,,1,3,1,,15,,2,0,,4,,,,0,5,8,2,0,0,0,,,,0,0,,,,,865,,,0,0,0,17,
                                                      ^^^^^

Deploy and enable you backend again:

1
# echo "enable server www-backend/web12" | socat /run/haproxy/admin.sock stdio

You can mix as a before task with capistrano or a pre_tasks with ansible.

Crazy Fast Deployment of Composer Based Applications With Capistrano 3

If you already use capistrano you already know about the linked_dirs and probably use something like this to keep your PHP libraries across releases:

1
set :linked_dirs, fetch(:linked_dirs, []).push('vendor')

But sometimes you need to run composer to keep add another dependency and sometimes takes a lot of time to download everything or some composer server stop to respond or suffer a slowdown and your deploy breaks.

I developed a deployment strategy to not forget to run composer or install all dependencies everytime.

The ideia is have the composer.phar available before the deploy starts and faster deploy times. (< 30 seconds)

The magic start with the rake file task who is only invoked one time and only if necessary, so we can install composer just once:

1
2
3
file "/tmp/composer.phar" do |f|
  sh "wget -q https://getcomposer.org/composer.phar -O #{f.name}"
end

Here is another trick, if you do not put the composer.lock file on your repository, composer will try to install all the dependencies and create the lock file, so here whe create the file and let composer update their content later to speed next deployments:

1
2
3
file "/tmp/composer.lock" do |f|
  touch f.name
end

The remote_file set the first argument as a prerequisite and if the file no exist the file task with the name '/tmp/composer.phar' will be called. The '/tmp/composer.phar' is a rake task name and also a path. You can see more about remote_file here.

The file task will run and the file will be created on the shared_path, otherwise, nothing happens.

1
2
remote_file 'composer.phar' => '/tmp/composer.phar', roles: :app
remote_file 'composer.lock' => '/tmp/composer.lock', roles: :app

And we link these tasks above on the check flow who happens before the deploy to ensure the capistrano.phar and capistrano.lock will be available on the shared_path and linked to the release_path later:

1
2
3
4
5
namespace :deploy do
  namespace :check do
    task :linked_files => ["composer.phar", "composer.lock"]
  end
end

Finally we always run composer to ensure the dependencies will be on available to the application:

1
2
3
4
5
6
7
8
9
10
11
namespace :deploy do
  #...
  desc "updates composer"
  after :updated, :vendor do
    on roles(:app), in: :groups, limit: 3 do
      within release_path do
        execute :php, "composer.phar install --no-dev --optimize-autoloader"
      end
    end
  end
end

TL;DR

The whole thing:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
#...

set :linked_files, fetch(:linked_files, []).push('composer.phar', 'composer.lock')
set :linked_dirs, fetch(:linked_dirs, []).push('vendor')

file "/tmp/composer.phar" do |f|
  sh "wget -q https://getcomposer.org/composer.phar -O #{f.name}"
end

file "/tmp/composer.lock" do |f|
  touch f.name
end

remote_file 'composer.phar' => '/tmp/composer.phar', roles: :app
remote_file 'composer.lock' => '/tmp/composer.lock', roles: :app

namespace :deploy do
  namespace :check do
    task :linked_files => ["composer.phar", "composer.lock"]
  end

  desc "updates composer"
  after :updated, :vendor do
    on roles(:app), in: :groups, limit: 3 do
      within release_path do
        execute :php, "composer.phar install --no-dev --optimize-autoloader"
      end
    end
  end
end

Try to remove one or another dependency from the vendor directory and see composer installing only the missing piece on the next deploy!

You can find me on Twitter as @fgbreel.

See you next time!

Solving 'Undefined Method Authenticate' When Using MiniTest and Devise in Rails 4.2

If you are using methods like user_signed_in? inside your controllers and receiving errors like those when running MiniTest:

1
ActionView::Template::Error: undefined method `authenticate' for nil:NilClass

or

1
ArgumentError: uncaught throw :warden

just reopen ActionController::TestCase and include the Devise::TestHelpers putting this at the end of your test_helper.rb file:

1
2
3
class ActionController::TestCase
  include Devise::TestHelpers
end

No more errors!

tip: you can use the sign_in helper provided by Devise to use inside your controller tests.

HTTP and HTTPS Scheme Redirect With HAProxy.

If you are deploying HAProxy in front of some other web server like Varnish or Nginx and need to ensure certain URI will be accessed only over a secure connection can easily done with HAProxy.

Create a file with the URIs who need to be secured, one per line:

1
2
3
4
5
vim /etc/haproxy/secure-locations.map

/admin
/secure-area
/foo

and put on your frontend definition on haproxy.cfg:

1
2
3
frontend www-http
  ...
  redirect scheme https if { capture.req.uri,map_beg(/etc/haproxy/secure-locations.map) -m found }

and this into your secure frontend definition if you don't want another places to follow the current (https) scheme:

1
2
3
frontend www-https 
  ...
  redirect scheme http if !{ capture.req.uri,map_beg(/etc/haproxy/secure-locations.map) -m found }

Cheers!

Task to Show Deployed Commits, Revisions and Branch With Capistrano 3

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
desc "Show the difference between deployed revisions on server."
task :diff_report do
  on roles(:app) do
    current, previous, branch = fetch(:current_revision), fetch(:previous_revision), fetch(:branch)
    puts "\n" << "-"*63
    puts "===== [ \033[1;36m#{fetch(:application).capitalize} - #{fetch(:stage).capitalize}\033[0m ]"
    puts "=== Deployed Branch: \033[1;32m#{branch}\033[0m"
    puts "=== Deployed Revision: \033[1;32m#{current}\033[0m"
    puts "=== Previous Revision: \033[1;32m#{previous}\033[0m\n\n"

    # If deployed master branch, show the difference between the last 2 deployments
    # or show the difference between master and the deployed branch.
    base_rev, new_rev = branch != "master" ? ["master", branch] : [previous, current]

    # Show difference between master and deployed revisions.
    if (diff = `git log #{base_rev}..#{new_rev} --oneline`) != ""
      # Colorize refs
      diff.gsub!(/^([a-f0-9]+) /, "\033[1;32m\\1\033[0m - ")
      diff = "    " << diff.gsub("\n", "\n    ") << "\n"
      # Indent commit messages nicely, max 80 chars per line, line has to end with space.
      diff = diff.split("\n").map{|l|l.scan(/.{1,120}/).join("\n"<<" "*14).gsub(/([^ ]*)\n {14}/m,"\n"<<" "*14<<"\\1")}.join("\n")
      puts "=== Difference between current revision and deployed revision:\n\n"
      puts diff
    else
      puts "=== Deployed the last revision again.\n\n"
    end
  end
end

after :log_revision, :diff_report

Kudos for @ndbroadbent for the original post.

Configuring Automatic Package Upgrade on Debian/Ubuntu

UPDATE: To enable without pain just run: dpkg-reconfigure unattended-upgrades.

In order to enable automatic package upgrades you need to install unattended-upgrades package from repository:

1
2
3
aptitude install unattended-upgrades
#or
apt-get install unattended-upgrades

Then configure what will be upgraded periodicaly editing the file /etc/apt/apt.conf.d/50unattended-upgrades. The file is well commented, so in this case I configured to install security upgrades only and an email address to receive alert in case of problems:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
// Automatically upgrade packages from these (origin:archive) pairs
Unattended-Upgrade::Allowed-Origins {
    "${distro_id}:${distro_codename}-security"; 
//    "${distro_id}:${distro_codename}-updates";
//    "${distro_id}:${distro_codename}-proposed";
//    "${distro_id}:${distro_codename}-backports";
};

...

// Send email to this address for problems or packages upgrades
// If empty or unset then no email is sent, make sure that you
// have a working mail setup on your system. A package that provides
// 'mailx' must be installed.
Unattended-Upgrade::Mail "[email protected]";

Later configure the file /etc/apt/apt.conf.d/10periodic:

1
2
3
4
5
6
7
8
// Enable periodic
APT::Periodic::Enable "1";
// Download packages every n-days.
APT::Periodic::Download-Upgradeable-Packages "1";
// Refresh package list automaticaly every n-days, in this case, everyday.
APT::Periodic::Update-Package-Lists "1";
// Enable Unattended Upgrades to run every n-days, well, you know :)
APT::Periodic::Unattended-Upgrade "1";

That is.

How to Change Timezone on Ubuntu/Debian

To select another timezone for your Debian/Ubuntu based linux box run the following command and select your Country/Province.

1
dpkg-reconfigure tzdata

Just that!

Deploy With Capistrano Without SCM

To deploy with Capistrano withou using a SCM, you need to add these lines on your deploy.rb:

1
2
3
set :scm, :none             # If you set this line without the lines below, you will get an error with 'which' command when run cap deploy:check 
set :repository,  "."       # set the directory who will be compressed to deploy
set :deploy_via, :copy      # specify to use via compressed tar