Is billing Amazon’s Achilles heel ?

istock_000000199356xsmallHaving worked on a number of projects with Amazon Web Services recently the one non-technical thing that has stood out is the billing model that Amazon adopts which basically forces the company to have a credit card available and then Amazon produce an email with the least amount of information possible on it to tell you that your credit card has been charged. If the user wants any kind of ‘Invoice’ they have to go back to their account and try and download usage amounts and associated bills. There is not one clean Invoice and a number of ‘features’ missing for this type of model…to name but a few:

What I am looking for is a way to put some control back into an Organisations hands, including:

– A way to grant access to more  granular access to users and therefore track who /which department in the company is using the service

– Central Management of billing, and an actual Invoice that can be submitted for recompense either to a another company or internally

– Ability to set budget limits, akin to what you can do to Google Adwords. 

– Alerting mechanisms to SMS when budgets near tolerance levels

– Ability to centrally track usage data so that chargeaback mechanisms can cleanly be built and used

There are numerous threads on the Amazon Web Service Community forum asking for hard copy invoices . Amazon does provide a page for tax help but its not that helpful 😉

Just some of the things floating around on the thread:

“Sounds silly, isnt’t it? But really, you can shake your head as long as you want, but tax authorities will not accept an invoice which does not state both partie’s VAT-ID number (here in italy, but its the same all over europe). 
If i go to dinner with my clients, the waiter will bring the bill in a carbon copy chemical paper. I HAVE to write my VAT-ID and full company name on it. 
Only THEN, he separates the first from the second sheet of paper, one stays in his records, one in my. 

If they check my books and find an invoice or bill which is not complaint to the formal requirements of having VAT-ID of both parties, they will not accept it and make you pay a fine. Its silly to discuss about the meaning of this, you would have to listen to a very long story about what cross-checks they do with these VAT-IDs. 

Any way, it’s not necessary that you send me a printed invoice, i can print it myself. But IT IS NECESSARY, that the invoice states clearly: 

name, address and VAT-ID of the seller 
name, address and VAT-ID of the purchaser 
description of goods and services 
invoice date, invoice number 

if any of these things are missing, the sheet of paper simply is not an invoice and trying to book it as an expense is a violation of law. 

Currently we are not able to detract AWS expenses of a few 100 US$/month due to these limitations.”

Reply to this post:

“In Czech it is even worse … we have to have hard copy with hand-writen _signature_ to be valid for tax authorities. Problems implications are then quite clear. Silly, but real in Czech. Another more detail, we can not add dinner with customer to our taxes. It has to be paid from the company net profit. “

Another example Reply:

“The same here in germany, we want to start using AWS for some projects but without a proper invoice our accounting will not give us a “go”. 

If this won’t change within this month we will either continue to work with dedicated server networks or might try the google appspot. 

Thats really a shame, because amazon does obviously know how to write correct invoices for 

I believe that this is probably tax related, with Amazon not wanting to amass taxes for Regional entities that would be liable for country specific tax, but its a great hole right now and I don’t have much doubt that it stops further adoption of the services themselves as organisational procedures are pretty inflexible when dealing with these issues.

Amazon EC2 News / Round Up

There is a good PDF whitepaper on using Oracle with Amazon Web Services which can be downloaded here.

A tutorial by Amazon on creating an Active Directory Domain on Amazon EC2 is a thorough article and well worth the read if you intend to implement this functionality on the cloud.

Simon Brunozzi from Amazon gives a good talk on “From zero to Cloud in 30 minutes” at the Next conference in Hamburg which can be viewed below.

Leventum talk about how they implemented the first ERP solution on the cloud using Compiere.

Jay Crossler Looks at how to visualize different cloud computing algorithms using serious Games technologies on the Amazon EC2 cloud below:

Practical Guide for Developing Enterprise Applications for the Cloud

This session was presented at Cloud Slam 09 by Nati Shalom CTO of GigaSpaces. It provides a practical guideline addressing the common challenges of developing and deploying an existing enterprise application on the cloud. Additionally, you will get the opportunity for hands-on experience running and deploying production ready applications in a matter of minutes on Amazon EC2.

London Amazon Web Services Startup Event Videos

For those of you who missed the Amazon Web Services startup event in London, you can find the customer presentations on And view the videos from the links below:

Cedric Roll, Co-Founder, ORbyte Solutions

Felipe Padilla, Co-Founder, Skipso

Nigel Hamilton, CEO,

Simone Brunozzi, Getting Started with AWS

Tal Saraf, Accelerating Your Website with CloudFront

Using Amazon EC2 for PCI DSS compliant applications

Compliance and regulatory concerns are often voiced when it comes to Cloud Computing, and often many of the interesting types of applications organisations would like to deploy to the cloud are  often those governed by some form of regulatory standard. Lets look in more details at one of these.

PCI DSS is a set of comprehensive requirements for enhancing payment account data security and was developed by the founding payment brands of the PCI Security Standards Council, including American Express, Discover Financial Services, JCB International, MasterCard Worldwide and Visa Inc. Inc. International, to help facilitate the broad adoption of consistent data security measures on a global basis.

The PCI DSS is a multifaceted security standard that includes requirements for security management, policies, procedures, network architecture, software design and other critical protective measures. This comprehensive standard is intended to help organizations proactively protect customer account data.

So, is it possible to create a PCI DSS compliant application that can be deployed to EC2 ?

In order for an application or system to become PCI DSS compliant requires an end to end system design (or a review if pre-existing) and implementation.  In the case of AWS customer’s attaining PCI compliance (certification), they would have to ensure they met all of the prescribed requirements through the use of encryption etc. very much like other customers have done with HIPAA applications.  The AWS design allows for customers with varying security and compliance requirements to build to those standards in a customized way.

There are different levels of PCI compliance and the secondary level is quite a straight forward configuration, but requires additional things such as 3rd party external scanning (annually).  You can find an example here of the PCI Scan report that is done on a quarterly basis for the Amazon platform.  This isn’t meant to be a replacement for the annual scan requirement. Customers undergoing PCI certification should have a dedicated scan that includes their complete solution, therefore certifying the entire capability, not just the Amazon infrastructure.

 The principles and accompanying requirements, around which the specific elements of the DSS are organized are:

 Build and Maintain a Secure Network

Requirement 1: Install and maintain a firewall configuration to protect cardholder data

Requirement 2: Do not use vendor-supplied defaults for system passwords and other security parameters Protect Cardholder Data

Requirement 3: Protect stored cardholder data

Requirement 4: Encrypt transmission of cardholder data across open, public networks Maintain a Vulnerability Management Program

Requirement 5: Use and regularly update anti-virus software

Requirement 6: Develop and maintain secure systems and applications Implement Strong Access Control Measures

Requirement 7: Restrict access to cardholder data by business need-to-know

Requirement 8: Assign a unique ID to each person with computer access

Requirement 9: Restrict physical access to cardholder data Regularly Monitor and Test Networks

Requirement 10: Track and monitor all access to network resources and cardholder data

Requirement 11: Regularly test security systems and processes Maintain an Information Security Policy

Requirement 12: Maintain a policy that addresses information security

Many of these requirements can’t be met strictly by a datacenter provider, but in Amazon’s case, they will be able to provide an SAS70 Type 2 Audit Statement in July that will provide much of the infrastructure information needed to meet PCI DSS certification.  The Control Objectives that the Amazon Audit will address are:

 Control Objective 1: Security Organization:  Management sets a clear information security policy. The policy is communicated throughout the organization to users

 Control Objective 2: Amazon Employee Lifecycle:  Controls provide reasonable assurance that procedures have been established so that Amazon employee accounts are added, modified and deleted in a timely manner and reviewed on a periodic basis to reduce the risk of unauthorized / inappropriate access

 Control Objective 3: Logical Security:  Controls provide reasonable assurance that unauthorized internal and external access to data is appropriately restricted

Control Objective 4: Access to Customer Data:  Controls provide reasonable assurance that access to customer data is managed by the customer and appropriately segregated from other customers

Control Objective 5: Secure Data Handling:  Controls provide reasonable assurance that data handling between customer point of initiation to Amazon storage location is secured and mapped accurately

 Control Objective 6: Physical Security:  Controls provide reasonable assurance that physical access to Amazon’s operations building and the data centers is restricted to authorized personnel

Control Objective 7: Environmental Safeguards:  Controls provide reasonable assurance that procedures exist to minimize the effect of a malfunction or physical disaster to the computer and data center facilities

Control Objective 8: Change Management:  Controls provide reasonable assurance that changes (including emergency / non-routine and configuration) to existing IT resources are logged, authorized, tested, approved and documented.

Control Objective 9: Data Integrity, Availability and Redundancy:  Controls provide reasonable assurance that data integrity is maintained through all phases including transmission, storage and processing and the Data Lifecycle is managed by customers

Control Objective 10: Incident Handling:  Controls provide reasonable assurance that system problems are properly recorded, analyzed, and resolved in a timely manner.

Many thanks to Carl from Amazon for his help with this information.

Update: Since this post was published Amazon updated their PCI DSS FAQ. You can find that here.

Overcoming the EC2 Windows AMI 10GB limit

Amazon limit the Windows AMI instance to 10GB in size which almost makes the image unusable if you try and add other software within the windows C Drive. Windows is notoriously heavy on disk space and whereas 10 GB may seem a lot believe us, it isn’t when it comes to windows and a combination of windows software.

So what can you do ? Well there are three potential options:

1. You can mount an EBS volume to a directory under C: MyDigitalLife has a great article on how to achieve this. This volume will become your E:

2. If more temporary space is needed for files or downloads etc than the 10 GB limit will give you, it is possible to make temporary folders outside of  the C: partition. 

– Right-click My Computer. 
– Click Properties 
– Click Advanced 
– Click Environment Variables 
– Change the tmp and temp to whatever you want.

3.  Use a combination of Junction link magic and webdrive. Firstly install whatever you need to the D: drive and use JLM to create junctions from C to D. Junctions are effectively a combination of symbolic links, and mount points. Install WebDrive to C: and then use it to copy the program files that are on D: to Amazon s3. As D: is transient this will mean if the instance goes down You can copy everything back from S3 to D:.

I’m sure at some point Amazon will get their act together on the instance size for Windows so you don’t have to navigate you way around this but right now at least this gives you some options.

McKinsey Cloud research kicks up a storm

A research paper on Cloud Computing by McKinsey & Company entitled ‘Clearing the Air on Cloud Computing’ has kicked up a right old storm with various luminaries either for or against it. The premise of the results of the article are that for large organisations, if they adopt the cloud model, then they would be making a mistake and most likely will lose money, as outsourcing from a more traditional data centre will likely double the cost (($150 per month per unit for data center vs $366 per month per unit for Amazon virtual cloud) . The New York times has an excellent summary of the study here.

Many of the complaints focus on McKinsey totally missing the “Private Cloud” and basing their assumptions on Public Clouds only. However there seems to be a general consensus that Amazon is too expensive and will need to adjust to survive. I’m not convinced about this. It is not the first study to suggest that Amazon are more expensive to use than a traditional data centre. Amazon seems to have been doing just fine up to now and they seem to be getting Enterprises to move across. Whether they replace a whole corporate data centre misses the point. I think this is unlikely, but for certain applications and service it makes perfect sense. Also, more competition unfolds then economics suggest that prices will naturally adjust if they need to.

You can download a PDF of the McKinsey presentation on this paper here.

Mosso come out fighting against S3 / Cloudfront with Cloudfiles and Limelight

Mosso are certainly not intent on letting Amazon have everything their own way, posting on their blog, Top 10 Reasons why Cloud Files + Limelight offers a better experience than S3 + CloudFront.

Competition is a great leveler and fuels innovation so I am glad to Mosso taking the lead here. The reasons that they give are reproduced below – I’d be interested in the thoughts of S3 / Cloudfront users as to whether these would make them consider moving across or what they think in general:

1.  World-class technical support is only one click away.

Live support, with real humans based right here in our offices, is available 24/7.  And they are really, really good!

2. World-class technical support is free.

Yes, free.  As in you don’t pay for it.  And it is really, really good support!

3. You can get started in as little as one minute.

Not a programmer?  Not a problem!  You do not have to know how to code to use Cloud Files + CDN!  Our simple web-based interface makes it a snap to share your content!

4. Limelight is a tier one CDN provider.

Really, Limelight is VERY cool, and one of the foremost CDN provider’s in the industry.  That’s why we chose to partner with them!

5. No API is required to share files.

Did we mention this already?  If so, it is worth mentioning again!

6. Language-specific APIs are available, if you need them.

Not everyone knows ReST and SOAP, so we’ve created and provide support for the following language APIs – PHP, Python, Java and .NET. We do this to allow you to work in the language you feel most comfortable with.

7. Pricing for data transfer does not vary depending on edge locations.

Data transfer starts at $0.22/GB, no matter what edge location is used to share your content. This should make it easier on you when you are trying to estimate your monthly bill.

8. There are no per requests fees for CDN.

Just another way we simplify your life, and your billing.

9. There is no limit to the number of CDN-enabled containers you can create.

As far as we can tell, you can only have up to 100 distributions in Amazon’s Cloud Front system. At Mosso, we try to keep these types of arbitrary limitations to a bare minimum, not just for Cloud Files, but for all of the services we offer.

10. The Cloud Files GUI is easy to use and navigate.

Our browser based GUI let’s you easily upload a file and share it on CDN without writing a single line of code.  Heck, you don’t even need to know a programmer to share content via Cloud Files!

What happens when the Cloud goes wrong ?

DeletetheCloud It is great enthusing about the benefits of Cloud Computing, but what are the consequences when it goes wrong ? Of course, there are different levels of  ‘going wrong’. We have often publicised outages from the likes of Amazon and Google, but given the publicised SLA’s of each some down time is expected. However things can get much more serious than this. In the last couple of days it has emerged that Carbonite had lost some of its customers data in 2007. According to TechCrunch Carbonite lost the data of 7,500+ customers who relied on the company to keep their files safe. This emerged because of a law suit that Carbonite filed on the providers of  their infrastructure. The Carbonite CEO provides more details:

“The failures of the Promise equipment occurred primarily during 2007. We stopped buying the Promise servers and switched suppliers. We allege that the Promise servers had defective firmware and were not reliable enough for Carbonite’s use. We are demanding that Promise compensate us for the cost of replacing their defective products. As for the 7,500 affected customers, their backups were restarted automatically and immediately on our new servers.”

In this case it seems the data loss had no effect and any data loss was mitigated by the companies internal backup procedures. However, in the case of JournalSpace, this unfortunately was not the case. JournalSpace was a blogging platform that had been around for about 6 years and, due to a disgruntled employee, all customers blogs were wiped out from their internal servers. Ouch !

This is not the first time such human error has lead to such problems. In August 2008 Cloud Platform FlexiScale, had an outage for over 2 days due to an engineer accidentally deleting a main storage volume.

Also the damage done it not just to the customer but also to the vendor’s reputation, especially if they are a smaller vendor trying to make a name for themselves. One such catastrophe can literally be the difference between success and failure in the market.

So what can we learn from this ? Well the first thing is that, just like Murphy’s law, if something can go wrong at some point it invariably will. With that in mind you should always take all steps to protect your applications and data. This could mean backing them up locally or keeping backups on different storage clouds and having a DR ready plan in place. If you don’t then you cannot just blame the cloud….

When Does Amazon EC2 Reserved Instance Pricing Save Money?

The simple answer is 4643.

Amazon recently announced new pricing option where you reserve an instance for one or three years and then have discount on the hourly rate. The table below describes  the cost per year if the instance is up for the whole year.

Instance Type Cost/Year On Demand instance $ Cost/Year for 1 year reserved instance $ Cost/Year for 3 Year reserved instance $
Extra Large
Medium High CPU
Extra Large High CPU

The above prices were calculated using Linux instance and United States prices

Cost/Year On Demand instance$ =  cost per hour * 365*24

Cost/Year for 1 year Reserved instance $ = 1 Year reserve price + (cost per hour *  365*24)

Cost/Year for 3 Year reserved instance $ = (3 Year reserve price /3)+ (cost per hour *  365*24)

Now this brings the interesting question when is it cost efficient reserve an instance ?

We can calculate the minimum usage hours over which the cheaper per hour price for a reserved instance starts to decrease the overall cost using the following formula

Instance Reserve Price/(On Demand Instance Hourly Price – Reserve Instance Hourly Price)

e.g for a small one year reserved instance

350/(0.10 – 0.03) = 4642.8 hours

For small 3 year reserved instance

500 / (0.10 – 0.03) = 7142.8 hours

The above calculated numbers are same for all instance types. So you should reserve a one year instance if you intend to use more than 4643 hours  or the instance is up 53% of the time in one year. With a 3 year reserved instance you can save money if over 3 year period you use 7143 or more hours or the instance is up 27% of the time over a  3 year time period.