Using Amazon Elastic Block Storage (EBS) to secure data

Amazon EBSAmazon Elastic Block Store (EBS) provides raw block-level storage that can be attached to Amazon EC2 instances. These block devices can then be used like any raw block device.

There are two ways that data can be protected when using EBS:

  1. Use the Amazon Identity Management Service to control access Elastic Block Store Volumes. This can be complimented with policy options that enforce policies such as multi-factor authentication, SSL links etc in addition to controlling or locking down originating IP addresses.
  1. Encrypted EBS volumes can be created.These encrypt data at rest.Note that once set an encrypted volume cannot later be unencrypted (the same as an unencrypted volume cannot be later encrypted).

How to use Linux instances on EC2 with Active Directory

AWS Linux AD Linux instances running on Amazon EC2 can now be joined to Simple AD directories from the AWS Directory Service.

This enables users to log in to all of their EC2 instances with a single set of domain credentials (no key pair needed) and set access controls, allowing domain admins to control which users can access particular instances.

Simple AD is a managed directory powered by a Samba 4, Active Directory Compatible Server. It supports commonly used features such as user accounts, group memberships, domain-joining Amazon EC2 instances running Linux and Microsoft Windows, as well as Kerberos based single sign-on (SSO), and Group Policies.

This makes it very easy for companies to be able to manage Amazon EC2 instances in the Amazon Web Services cloud.

Amazon have more details on their blog.

Amazon EBS Provisioned IOPS volumes can now store up to 16 TB

Amazon EVS 16TBFrom  today, users of Amazon Web Services can create Amazon EBS Provisioned IOPS volumes that can store up to 16 TB, and process up to 20,000 input/output operations per second (IOPS).

Amazon Elastic Block Store (Amazon EBS) provides persistent block level storage volumes for use with Amazon EC2 (Elastic Compute) instances in the AWS Cloud.

Users can also create Amazon EBS General Purpose (SSD) volumes that can store up to 16 TB, and process up to 10,000 IOPS. These volumes are designed for five 9s of availability and up to 320 megabytes per second of throughput when attached to EBS optimized instances.

These performance improvements make it even easier to run applications requiring high performance or high amounts of storage, such as large transactional databases, big data analytics, and log processing systems. Users can now run large-scale, high performance workloads on a single volume, without needing to stripe together several smaller volumes.

Larger and faster volumes are available now in all commercial AWS regions and in AWS GovCloud (US). To learn more please check out the Amazon EBS details page.

Amazon – what is coming soon, and what is not !

We had a meeting with Amazon in the UK recently and covered off some off the pressing issues that we wanted to speak about and also learnt some other of what Amazon have lined up.

First, what is not going to happen anytime soon:

– From what we heard Amazon are not going to resolve the issue of billing in local currency with locally issued invoices any time soon. See our prior post on this topic. We did learn however that large organisations can request an invoice.

– Right now if you want to use your own AMI image to sell on a SaaS basis using Amazon infrastructure you have to a US organisation. Again Amazon don’t seem to have plans to change this in the immediate timeframe so that leaves out any organisation outside of the US who want to sell their product offering as SaaS on Amazon’s web services infrastructure unless they integrate their own commerce infrastructure and not use DevPay. This can be both a blessing (charge margin on Amazon’s infrastructure pieces like AMQS) but also a curse (can leave you exposed as you will be month behind in billing your clients). Even though Amazon are entrenched right now as the Public Cloud infrastructure of choice, it wouldn’t be the first time we have seen 100 pound gorilla displaced from it’s prime market position. If I were Amazon, I’d fix this and soon. Microsoft and RackSpace are looking more attractive all the time.

– Amazon’s ingestion services again require you to be a US organisation with a US return address. Are you detecting a common theme here….

And what we can expect to see soon:

– VPC (Virtual private cloud) access is in private beta now. This is a mechanism for securely connecting public and private clouds within the EC2 infrastructure.

– High memory instances analogous to High CPU instances are in the pipeline

– Shared EBS is in the pipeline

– Functionality for Multiple users associated with a single account is in the pipeline and will provide simple privileges too. This has long been a bone of contention for organisations using AWS so will be welcomed.

– Amazon is planning to have lot more EC2 workshops through local partners.

Other things of note that we learnt where:

– We learned that large physical instances currently have their own dedicated blade / box.

– As AWS has grown, large number of machines are available and organizations can request hundreds of machines easily. Even extreme cases are catered for i.e. even requests for 50000 machines.

– As a matter of policy new functionally will be rolled out simultaneously in EU and US unless there is a good reason.

All in all some exciting stuff, and there was other things in the pipeline they could not share, but the public cloud market is starting to get more players and I think Amazon need to get some of their infrastructure pieces in place sooner rather than later.

Amazon Elastic MapReduce now available in Europe

From the Amazon Web Services Blog:

 Earlier this year I wrote about Amazon Elastic MapReduce and the ways in which it can be used to process large data sets on a cluster of processors. Since the announcement, our customers have wholeheartedly embraced the service and have been doing some very impressive work with it (more on this in a moment).

Today I am pleased to announce Amazon Elastic MapReduce job flows can now be run in our European region. You can launch jobs in Europe by simply choosing the new region from the menu. The jobs will run on EC2 instances in Europe and usage will be billed at those rates.

 Because the input and output locations for Elastic MapReduce jobs are specified in terms of URLs to S3 buckets, you can process data from US-hosted buckets in Europe, storing the results in Europe or in the US. Since this is an internet data transfer, the usual EC2 and S3 bandwidth charges will apply.

Our customers are doing some interesting things with Elastic MapReduce.

 At the recent Hadoop Summit, online shopping site ExtraBux described their multi-stage processing pipeline. The pipeline is fed with data supplied by their merchant partners. This data is preprocessed on some EC2 instances and then stored on a collection of Elastic Block Store volumes.The first MapReduce step processes this data into a common format and stores it in HDFS form for further processing. Additional processing steps transform the data and product images into final form for presentation to online shoppers. You can learn more about this work in Jinesh Varia’s Hadoop Summit Presentation.

Online dating site eHarmony is also making good use of Elastic MapReduce, processing tens of gigabytes of data representing hundreds of millions of users, each with several hundred attributes to be matched. According to an article on, they are doing this work for $1,200 per month, a considerable savings from the $5,000 per month that they estimated it would cost them to do it internally.

We’ve added some articles to our Resource Center to help you to use Elastic MapReduce in your own applications. Here’s what we have so far:



You should also check out AWS Evangelist Jinesh Varia in this video from the Hadoop Summit:

— Jeff;

PS – If you have a lot of data that you would like to process on Elastic MapReduce, don’t forget to check out the new AWS Import/Export service. You can send your physical media to us and we’ll take care of loading it into Amazon S3 for you.

Is billing Amazon’s Achilles heel ?

istock_000000199356xsmallHaving worked on a number of projects with Amazon Web Services recently the one non-technical thing that has stood out is the billing model that Amazon adopts which basically forces the company to have a credit card available and then Amazon produce an email with the least amount of information possible on it to tell you that your credit card has been charged. If the user wants any kind of ‘Invoice’ they have to go back to their account and try and download usage amounts and associated bills. There is not one clean Invoice and a number of ‘features’ missing for this type of model…to name but a few:

What I am looking for is a way to put some control back into an Organisations hands, including:

– A way to grant access to more  granular access to users and therefore track who /which department in the company is using the service

– Central Management of billing, and an actual Invoice that can be submitted for recompense either to a another company or internally

– Ability to set budget limits, akin to what you can do to Google Adwords. 

– Alerting mechanisms to SMS when budgets near tolerance levels

– Ability to centrally track usage data so that chargeaback mechanisms can cleanly be built and used

There are numerous threads on the Amazon Web Service Community forum asking for hard copy invoices . Amazon does provide a page for tax help but its not that helpful 😉

Just some of the things floating around on the thread:

“Sounds silly, isnt’t it? But really, you can shake your head as long as you want, but tax authorities will not accept an invoice which does not state both partie’s VAT-ID number (here in italy, but its the same all over europe). 
If i go to dinner with my clients, the waiter will bring the bill in a carbon copy chemical paper. I HAVE to write my VAT-ID and full company name on it. 
Only THEN, he separates the first from the second sheet of paper, one stays in his records, one in my. 

If they check my books and find an invoice or bill which is not complaint to the formal requirements of having VAT-ID of both parties, they will not accept it and make you pay a fine. Its silly to discuss about the meaning of this, you would have to listen to a very long story about what cross-checks they do with these VAT-IDs. 

Any way, it’s not necessary that you send me a printed invoice, i can print it myself. But IT IS NECESSARY, that the invoice states clearly: 

name, address and VAT-ID of the seller 
name, address and VAT-ID of the purchaser 
description of goods and services 
invoice date, invoice number 

if any of these things are missing, the sheet of paper simply is not an invoice and trying to book it as an expense is a violation of law. 

Currently we are not able to detract AWS expenses of a few 100 US$/month due to these limitations.”

Reply to this post:

“In Czech it is even worse … we have to have hard copy with hand-writen _signature_ to be valid for tax authorities. Problems implications are then quite clear. Silly, but real in Czech. Another more detail, we can not add dinner with customer to our taxes. It has to be paid from the company net profit. “

Another example Reply:

“The same here in germany, we want to start using AWS for some projects but without a proper invoice our accounting will not give us a “go”. 

If this won’t change within this month we will either continue to work with dedicated server networks or might try the google appspot. 

Thats really a shame, because amazon does obviously know how to write correct invoices for 

I believe that this is probably tax related, with Amazon not wanting to amass taxes for Regional entities that would be liable for country specific tax, but its a great hole right now and I don’t have much doubt that it stops further adoption of the services themselves as organisational procedures are pretty inflexible when dealing with these issues.

Amazon EC2 News / Round Up

There is a good PDF whitepaper on using Oracle with Amazon Web Services which can be downloaded here.

A tutorial by Amazon on creating an Active Directory Domain on Amazon EC2 is a thorough article and well worth the read if you intend to implement this functionality on the cloud.

Simon Brunozzi from Amazon gives a good talk on “From zero to Cloud in 30 minutes” at the Next conference in Hamburg which can be viewed below.

Leventum talk about how they implemented the first ERP solution on the cloud using Compiere.

Jay Crossler Looks at how to visualize different cloud computing algorithms using serious Games technologies on the Amazon EC2 cloud below:

London Amazon Web Services Startup Event Videos

For those of you who missed the Amazon Web Services startup event in London, you can find the customer presentations on And view the videos from the links below:

Cedric Roll, Co-Founder, ORbyte Solutions

Felipe Padilla, Co-Founder, Skipso

Nigel Hamilton, CEO,

Simone Brunozzi, Getting Started with AWS

Tal Saraf, Accelerating Your Website with CloudFront