At New Relic, we eat our own dog food. Politely put, that means we use New Relic RPM to tune itself. One of the biggest challenges we face is reproducing our production databases in a local environment. Our production data is vast and rich, while our local databases only have basic test data. We considered writing scripts that could export/import, however due to the size of our databases this wasn’t practical.
Our solution was to create a new type of RAILS_ENV called local_production. The “local” means it’s used on developers’ desktops and the “production” means active record is using production and not local databases. In a nutshell, what we’re doing is using a creating the facade of local databases that actually map to remote databases. This gives us the opportunity to reproduce bugs running the server locally in debug mode as well as collect more detailed page by page statistics using RPM Developer Mode.
When creating this type of environment, there are a few things you’ll need to do.
Create the environment script
<app>/config/environments/local_production.rb. We cloned our development.rb script but turned on class caching to reduce the database noise that Active Record class reloading tends to cause.
Also add a new section to your
config/newrelic.yml for the local_production environment. Copy the development section.
Setup port forwarding in your SSH config file in
~/.ssh/config. SSH port forwarding creates a local socket that is bound to a remote socket. SSH proxies traffic between the two, allowing you to open a connection to what looks like a local database but really is a remote database. Because we have multiple databases, we use a single ssh session to forward to multiple databases. Here’s what our config entry looks like:
LocalForward 3316 mysql50-account-master:3306
LocalForward 3317 mysql50-account-replica:3306
LocalForward 3318 mysql50-staging:3306
The next step is to create a new database user and give them read-only permission. We do this because it’s a really bad idea to use a user with write permission since it’s very easy to forget you’re running with the “local_production” RAILS_ENV. We started out using our normal production database user, but since switching to a read only user we sleep much better at night.
The last step is to configure <app>/config/database.yml with the database information. Based on the port forwarding we use, here is the corresponding database.yml entry:
<<: *defaults database: rpm_production username: newrelic_ro_db password: password
# our master database is the default
Verifying the Setup
Once you’ve got it all setup, you can test that it’s working by using the script console. Before you do this, make sure that you’ve got ssh opened to the host that is doing the port forwarding.
$ export RAILS_ENV=local_production
Loading local_production environment (Rails 2.1.2)
=> "NewRelic Administration"
Taking Advantage of Developer Mode
Now you’re ready to fire up a local rails server running as if it’s in production. Make sure your local schema version matches the remote and start
script/server -e local_production.
This gives you the opportunity to reproduce bugs in production running in debug mode or with more verbose logging, but we’ve also benefitted from running RPM Developer Mode. Open a browser on your site and then hit the pages of interest. Then open the page
/newrelic (not installed on your production site) and there you’ll find detailed transaction traces for each page including the actual SQL being executed and the query plans for each query.