Category Archives: Cloud

Adding Instance Storage after upgrading a Micro instance on Amazon EC2

Cloud-Storage I recently upgraded a Micro (t1.micro) instance to a Small (m1.small) version on Amazon EC2. This was performed with no real trouble at all (Stop, Change Instance Type, Start). However, I noticed upon boot that I did not have any of the promised Instance Storage available to me. On the m1.small, 160 gigabytes is mentioned. Where was it?

It turns out that for EBS-backed AMIs (including Amazon Linux, the OS in use for this example) , the instance storage is not automatically baked in. It is possible to ensure it is added when starting a new VM, but I was never given an option to do this due to going down the upgrade path. It turns out it is not possible to add it to an instance that is already in operation. So, what to do?

Firstly, the official Amazon documentation: Adding Instance Store Volumes to an AMI.

These are the steps I needed to take, which I accomplished via the AWS Console rather than via the command line.

  1. Stop the Instance. You don’t have to do this prior to the next step, but for consistency, it is best you do.
  2. Create a Snapshot.
  3. Create a new AMI from the Snapshot (This is where the steps in the official guide linked above come into play). For the m1.small, I added two Instance Store Volumes (as you get a ‘swap’ partition as well).
  4. Create a new instance using your new AMI as the source.
  5. If relevant, re-assign any Elastic IP Address to the new instance.
  6. Start the new instance.
  7. You’re done!

Upon boot, Amazon Linux auto-detects the new available partitions.

The main storage partition:

$ df -h
Filesystem            Size  Used Avail Use% Mounted on
/dev/xvda1            7.9G  2.3G  5.6G  30% /
tmpfs                 829M     0  829M   0% /dev/shm
/dev/xvdb             147G  188M  140G   1% /media/ephemeral0

… and the swap partition:

[  169.078672] Adding 917500k swap on /dev/xvda3.  Priority:-1 extents:1 across:917500k SS

Easy!

Note that by default, Amazon Linux formatted the storage partition as ext3. You may well want to convert it to ext4 (I just reformat it here, so don’t do it this way if you’ve put data there!)

$ sudo umount /dev/xvdb
$ sudo mke2fs -t ext4 /dev/xvdb
mke2fs 1.42.3 (14-May-2012)
Filesystem label=
OS type: Linux
Block size=4096 (log=2)
Fragment size=4096 (log=2)
Stride=0 blocks, Stripe width=0 blocks
9773056 inodes, 39088128 blocks
1954406 blocks (5.00%) reserved for the super user
First data block=0
Maximum filesystem blocks=4294967296
1193 block groups
32768 blocks per group, 32768 fragments per group
8192 inodes per group
Superblock backups stored on blocks: 
	32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632, 2654208, 
	4096000, 7962624, 11239424, 20480000, 23887872
 
Allocating group tables: done                            
Writing inode tables: done                            
Creating journal (32768 blocks): done
Writing superblocks and filesystem accounting information: done     
 
$ sudo mount /media/ephemeral0

One final and standard warning regarding Instance Storage on EC2: It will be erased upon your instance Terminating, which could include a failure condition. As a result, never use it for data you rely on (use EBS instead).

Migrating email to GMail

I recently moved extricate.org to sit on Amazon EC2. This meant I had to decide what to do with my email. Should I just leave it on Dreamhost and access it via IMAP?

In the end, I decided to enable Google Apps for my domain. Free for 10 users or less so that will do nicely. GMail has a fantastic interface nowadays and after using it for a while I was happy that I would use it instead of IMAP, although IMAP is on offer should it ever be needed.

Getting up and running was straightforward, with the main element being to point my DNS to the new servers. Google walk you through all this via their setup wizard, so well played to them there. One immediate issue was that my new email address already existed as a ‘personal’ Google account, and that conflict needed to be resolved. In the end I renamed that account, started a ‘blank’ new one, and migrated the data across.

I transferred most of my email from Dreamhost by using Thunderbird. Having enabled IMAP on GMail, the folders could be consolidated and dragged between accounts. This wasn’t particularly fast as it was reliant on my home broadband connection. When it came to larger folders, I needed a better way!

Take it to the cloud…

It made sense to perform the copying via my Amazon EC2 server, as it wouldn’t be constrained by my home network connection. It did mean finding some suitable Linux command-line software. I decided upon imapcopy. I downloaded it straight from the home site.

However, this software is quite old and does not support secure connections, as required by GMail’s IMAP servers! This is where stunnel came in, which may be installed from the Amazon Linux AMI repositories:

sudo yum install stunnel

Configuration guide: stunnel and imapcopy.

I then just needed to configure imapcopy (ImapCopy.cfg). Naturally, the DestServer had to be sent to ‘localhost:1143’ so that the routing occurred through the new tunnel. I also explicitly stated what I wanted to happen:

copyfolder INBOX.Sent
DstRootFolder “Migrated”

It was then just a case of running imapcopy and away it went! It worked flawlessly which meant that all my email was now accessible within GMail.

A successful migration!

Moving extricate.org to Amazon EC2

I have been continuing work with Amazon EC2 recently. I am a big cloud fan (evangelist?) and, as a result of that, it was time to eat my own dog food and move my domain over.

My host for a couple of years has been Dreamhost. They provide an excellent hosting service, including a very comprehensive control panel and secure shell access. Nothing wrong there at all: I just wanted to go down the Amazon route.

Another factor is that the Dreamhost servers are based in the US. This meant that the site always seemed slightly sluggish to me, so I was hopeful that moving hosting to Ireland (The ‘EU-West’ region in EC2 terms) would speed things up.

Instance and OS selection

It made sense to embrace the AWS Free Usage Tier, which meant I elected to go with a ‘Micro’ instance:

613 MB memory
Up to 2 EC2 Compute Units (for short periodic bursts)
EBS storage only
32-bit or 64-bit platform
I/O Performance: Low
API name: t1.micro

64-bit was a natural choice. There is no ephemeral storage with this instance size. That was fine by me, as I wanted to ensure that everything was secure in the event of the instance terminating for whatever reason. Hurrah for EBS!

I tried out two different Linux images: Amazon’s own flavour, and Ubuntu Cloud. There was not a great deal in it, and although I am more familiar with Ubuntu on a day-to-day basis, I went with Amazon Linux. It is optimised by them for their own platform and has a CentOS pedigree.

Getting up and running

The instance was very quickly provisioned. I followed the standard WordPress Installation Guide, which included getting Apache and MySQL up and running on the box. Both of those are in the Amazon repositories. WordPress itself I did directly from the source, copying themes, plugins and other content data from my old host via scp.

I needed to bring the old MySQL database across as well. A simple mysqldump got the required SQL which was trivial to import. I used phpMyAdmin to help out with user creation and permissions, as it is a lot friendlier than tapping things out on the command line.

Memory considerations

During testing, MySQL was terminated due to the instance running out of memory. By default, there is no swap space provided. Swap space is easy to provision if required, be that via a swap file or swap partition (allocating a new EBS volume).

However, it should be noted that using swap space will naturally increase I/O, along with charges!

I decided to keep things streamlined and not enable swap space. Instead, I toned down Apache’s memory usage, as by default it would spin up 10 servers to handle requests. I went with the following settings:

StartServers 2
MinSpareServers 1
MaxSpareServers 3
ServerLimit 8
MaxClients 8
MaxRequestsPerChild 1000

Everything has been stable since then.

CloudFront

WordPress a great cache plugin: WP Super Cache. I soon got this up and running, and took advantage of its CDN support. This allows it to rewrite wp-content URLs in order to be served up by the Content Delivery Network of your choice.

Here’s a good tutorial on this: ‘Setting Up Amazon CloudFront CDN in WordPress is Really Easy!’.

This handily takes further load off the micro instance, and therefore performance is improved.

Route 53 DNS

That just left DNS over at Dreamhost. Amazon offer this service as well, in the form of Route 53. I set up a Hosted Zone for extricate.org and it was very quick to set up the required entries (pretty much just copying over from Dreamhost). The zone was instantly provisioned and worked perfectly once I instructed my registrar to use the new servers. Very easy!

The results…

I’m happy to report that everything just works! Performance is snappier as well, although I am sure that the server now sitting more local to me is a big boost here. I also love having full ‘root’ access over the system now.

That did leave migrating my email, and that is a future article!