So recently at work we’ve been looking to migrate a few services that are hosted on legacy infrastructure over to a much better set of servers at Amazon AWS. Along the way we hit a few issues, i wanted to share my experience with you so people never have to worry about these issues again…

Issue 1:
When using Amazon AWS Elastic Load balancers you get provided with a host name for the load balancer (not an IP – this is key). This is all good and well but DNS RFC 1034 states that the origin of a domain has to be a A record, but Amazon gave us a host name, so thats a CNAME? Our current DNS provider did not support host names as A records, so this isnt good.

Solution:
Use Route53 at Amazon. Amazon allows you to use their Route53 DNS service and create A records with an “AliasTarget”. These basically setup some crazy round robin DNS setup. There are a few steps to this:

1. Create the HostedZone using the Route53 API. Here is the XML required to create the hostedzone for my domain bitchasscode.com

<CreateHostedZoneRequest xmlns="https://route53.amazonaws.com/doc/2010-10-01/">
 <Name>bitchasscode.com.</Name>
 <CallerReference>dns_migration_amcgrath_20111018</CallerReference>
 <HostedZoneConfig>
 <Comment>Migrate an existing domain to Route 53</Comment>
 </HostedZoneConfig>
</CreateHostedZoneRequest>

2. Create the individual A records and Cnames for the domain using the Route53 API:

<?xml version="1.0" encoding="UTF-8"?>
<ChangeResourceRecordSetsRequest xmlns="https://route53.amazonaws.com/doc/2011-05-05/">
 <ChangeBatch>
 <Comment>Create record for bitchasscode.com.</Comment>
 <Changes>

 <Change>
 <Action>CREATE</Action>
 <ResourceRecordSet>
 <Name>www.bitchasscode.com.</Name>
 <Type>CNAME</Type>
 <TTL>300</TTL>
 <ResourceRecords>
 <ResourceRecord>
 <Value>bitchasscode.com<Value>
 </ResourceRecord>
 </ResourceRecords>
 </ResourceRecordSet>
 </Change>
 <Change>
 <Action>CREATE</Action>
 <ResourceRecordSet>
 <Name>bitchasscode.com.</Name>
 <Type>A</Type>
 <AliasTarget>
 <HostedZoneId>YOUR LOAD BALANCERS HOSTED ZONE ID</HostedZoneId>
 <DNSName>DNS NAME OF THE LOAD BALANCER</DNSName>
 </AliasTarget>
 </ResourceRecordSet>
 </Change>
</Changes>

</ChangeBatch>

</ChangeResourceRecordSetsRequest>

This is a key step, becsause this is what lets you have a “hostname” as a A record. Sexy stuff…

3. Test your setup, it worked fine for me, however honestly its a real pain in the ass to do all of this. This created a new issue for me, i didnt really trust what I just did…so i started thinking about creating a UI for Route53 so i could easily view my edits.

Issue 2:
There is no UI for route53 provided by Amazon.

Solution:
Use Interstate53 as your UI, you just need to provide an AWS API key & secret…then you’re done!

Honestly i dont know a lot about Interstate53, other than its very amazing. I tried a few other services which didnt do the job for me:

• nsRoute (Doesnt do Alias records to ELB’s properly)
• dns30 (Doesnt do Alias records to ELB’s properly)
• easydns (They wanted me to signup, pay or something – too hard, i gave up after i couldnt work out how to get an account.)

Interstate53 were super easy to get going with (just provide an api key and secret), there is no registration or anything and their UI is great. It works perfectly with Alias records going to ELB’s and then you’re cooking!

Recently I’ve run into so many poorly setup systems, and what makes me sad is that people paid for them to do this. If you wrote any code recently and you’re trying to get people to pay for it, do them (and yourself) a credit by at least doing the following:

• GET A LOAD BALANCER
  They are very simple to setup, services like Amazon AWS and Rackspace provide these for around $20 / month; they are not expensive. If you have your reputation on the line they are worth the spend! Even if you only have 1 machine behind the load balancer, if you dont use one from the start in order to introduce one later when things get serious you need to make a DNS change. This is a big issue because you probably want it to happen quickly.  Add the load balancer in from the start! You can always add more servers behind it and with sticky sessions as an option, you dont even need to make code changes to your app!
• DISABLE SSH ACCESS FROM THE ENTIRE WORLD
  At Amazon this is very simple, create a custom security group and whitelist only the IP’s you want to have access via SSH.
• DO NOT USE MYSQL WITH ROOT
  Always create custom usernames for applications. Its nice to tell the difference between your app, and someone using the root account.
• ALWAYS USE PASSWORDS WITH MYSQL
  I came across a case yesterday where people were using root with no password
• ONLY GIVE MYSQL ACCESS TO PEOPLE WHO NEED IT
  Again back to the security groups, dont let anyone get access to your machine on port 3306! Only white-list servers which need access to your database. Sadly the same case i ran into yesterday using root and no password also had their instance open to the entire world. Major no no!
• KNOW YOUR HOSTING PROVIDER
  Companies like Amazon provide some amazing tools that if used properly can result in some great results. If you’re at Amazon try use things like RDS (Rather than installing MYSQL on an EC2 instance), take advantage of different availability zones (dont put every machine behind your load balancer in the same zone), know how to secure your site using tools like “Security Groups”, and understand how to enable monitoring.
• Finally…MONITOR YOUR SERVICE
  At Amazon CloudWatch is your friend, use it! Its cheap ($3/month) and very useful…if you want to get an email when the number of unhealthy servers in a load balanced group increases, this service can tell you! Tie that baby in with companies like PagerDuty and hey presto, every time a server goes out of service you get a phone call.

If you use the tools you are provided with properly, your life is easy. If you dont someone like me will jump on your machine, drop your database, sit back and laugh because you’re not monitoring it and you dont have a clue it happened until the customer calls to yell at you.

Fix your shit, or i’ll take your customers money and keep it for myself!