Amazon EC2 News / Round Up

There is a good PDF whitepaper on using Oracle with Amazon Web Services which can be downloaded here.


A tutorial by Amazon on creating an Active Directory Domain on Amazon EC2 is a thorough article and well worth the read if you intend to implement this functionality on the cloud.


Simon Brunozzi from Amazon gives a good talk on “From zero to Cloud in 30 minutes” at the Next conference in Hamburg which can be viewed below.





Leventum talk about how they implemented the first ERP solution on the cloud using Compiere.


Jay Crossler Looks at how to visualize different cloud computing algorithms using serious Games technologies on the Amazon EC2 cloud below:


Practical Guide for Developing Enterprise Applications for the Cloud

This session was presented at Cloud Slam 09 by Nati Shalom CTO of GigaSpaces. It provides a practical guideline addressing the common challenges of developing and deploying an existing enterprise application on the cloud. Additionally, you will get the opportunity for hands-on experience running and deploying production ready applications in a matter of minutes on Amazon EC2.

Cloud Best Practice – what to be aware of !

Some of the key things to think about when putting your application on the cloud are discussed below. Cloud computing is relatively new, and best practice is still being established. However we can learn from earlier technologies and concepts such as utility compute, SaaS, outsourcing and even internal enterprise centre management, as well as from experience with vendors such as Amazon and FlexiScale.

Licensing: If you are using the cloud for spikes or overspill make sure that the products you want to use in the cloud can be used in this way. Certain products restrict their licenses to be used from a cloud perspective. This is especially true of commercial Grid, HPC or DataGrid vendors.

Data transfer costs:  When using a provider like Amazon with a detailed cost model, make sure that any data transfers are internal to the provider network rather than external. In the case of Amazon, internal traffic is free but you will be charged for any traffic over the external IP addresses.

Latency: If you have low latency requirements then the Cloud may not be the best environment to achieve this. If you are trying to run an ERP or some such system in the cloud then the latency may be good enough but if you are trying to run a binary or FX Exchange then of course the latency requirements are very different and more stringent. It is essential to make sure you understand the performance requirements of your application and have a clear understanding of what is deemed business critical.

One Grid vendor who has focused on attacking low latency in the cloud is GigaSpaces and so if you require cloud low latency then these are one of the companies you should evaluate. Also for processing distributed data loads there is the map reduce pattern and Hadoop. These type of architectures eliminating the boundaries created by scale-out database based approaches.

State: Check whether your cloud infrastructure providers have persistence.  When an application is brought down and then back up all local changes will be wiped and you start with a blank slate. This obviously has ramifications with instances that need to store user or application state.  To combat this on their platform Amazon rdelivered EC2 persistent storage in which data can remain linked to a specific computing instance. You should ensure you understand the state limitations of any Cloud Computing platform that you work with.

Data Regulations: If you are storing data in the cloud you may be breaching data laws depending where your data is stored i.e. which country or continent.  To combat this Amazon S3 now supports location constraints, which allow you to specify where in the world to store data for a bucket and provides a new API to retrieve the location constraint for an existing bucket. However if you are using another cloud provider you should check where your data is stored.

Dependencies:  Be aware of dependencies of service providers. If service ‘y’ is dependant on ‘x’ then if you subscribe to service ‘y’ and service ‘x’ goes down you lose your service. Always check any dependencies when you are using a cloud service.

Standardisation: A major issue with current cloud computing platforms is that there is no standardisation of the APIs and platform technologies that underpin the services provided. Although this represents a lack of maturity you need to consider how locked in you are when considering a Cloud platform or migrating between cloud computing platforms will be very difficult if not impossible. This may not be an issue if your supplier is IBM and always likely to be IBM, but it will be an issue if you are just dipping your toe in the water and discover that other platforms are better suited to your needs.

Security: Lack of security or apparent lack of security is one of the perceived major drawbacks of working with Cloud platform and Cloud technology. When moving sensitive data about or storing it in public cloud it should be encrypted. And it is important to consider a secure ID mechanism for authentication and authorisation for services. As with normal enterprise infrastructures only open the ports needed and consider installing a host based intrusion detection systems such as OSSEC. The advantage of working with an enterprise Cloud provider, such as IBM or Sun is that many of these security optimisations are already taken care of. See our prior blog entry for securing n-tier and distributed applications on the cloud.

Compliance:  Regulatory controls mean that certain applications may not be able to deployed in the Cloud. For example the US Patriot Act could have very serious consequences for non-US firms considering U.S. hosted cloud providers. Be aware that often cloud computing platforms are made up of components from a variety of vendors who may themselves provide computing in a variety of legal jurisdictions. Be very aware of the dependencies and ensure you factor this into any operational risk management assessment. See also our prior blog entry on this topic

Quality of service: You will need to ensure that the behaviour and effectiveness of the cloud application that you implement can be measured and tracked both to meet existing or new Service Level agreements. We have discussed previously some of the tools that come with this option built in (GigaSpaces) and other tools that provide functionality that enable you to use this with your Cloud Architecture (RightScale, Scalr etc). Achieving Quality of Service will encompass scaling, reliability, service fluidity, monitoring, management and system performance.

System hardening: Like all enterprise application infrastructures you need to harden the system so that it is secure, robust, and achieves the necessary functional requirements that you need. See our prior blog entry on system hardening for Amazon EC2.

Content adapted from “TheSavvyGuideTo HPC, Grid, DataGrid, Virtualisation and Cloud Computing” available on Amazon.



London Amazon Web Services Startup Event Videos

For those of you who missed the Amazon Web Services startup event in London, you can find the customer presentations on Slideshare.net. And view the videos from the links below:

Cedric Roll, Co-Founder, ORbyte Solutions http://www.vimeo.com/4409867

Felipe Padilla, Co-Founder, Skipso http://www.vimeo.com/4409569

Nigel Hamilton, CEO, Turbo10.com http://www.vimeo.com/4409682

Simone Brunozzi, Getting Started with AWS http://www.vimeo.com/4411474

Tal Saraf, Accelerating Your Website with CloudFront http://www.vimeo.com/4409756

Using Amazon EC2 for PCI DSS compliant applications

Compliance and regulatory concerns are often voiced when it comes to Cloud Computing, and often many of the interesting types of applications organisations would like to deploy to the cloud are  often those governed by some form of regulatory standard. Lets look in more details at one of these.

PCI DSS is a set of comprehensive requirements for enhancing payment account data security and was developed by the founding payment brands of the PCI Security Standards Council, including American Express, Discover Financial Services, JCB International, MasterCard Worldwide and Visa Inc. Inc. International, to help facilitate the broad adoption of consistent data security measures on a global basis.

The PCI DSS is a multifaceted security standard that includes requirements for security management, policies, procedures, network architecture, software design and other critical protective measures. This comprehensive standard is intended to help organizations proactively protect customer account data.

So, is it possible to create a PCI DSS compliant application that can be deployed to EC2 ?

In order for an application or system to become PCI DSS compliant requires an end to end system design (or a review if pre-existing) and implementation.  In the case of AWS customer’s attaining PCI compliance (certification), they would have to ensure they met all of the prescribed requirements through the use of encryption etc. very much like other customers have done with HIPAA applications.  The AWS design allows for customers with varying security and compliance requirements to build to those standards in a customized way.

There are different levels of PCI compliance and the secondary level is quite a straight forward configuration, but requires additional things such as 3rd party external scanning (annually).  You can find an example here of the PCI Scan report that is done on a quarterly basis for the Amazon platform.  This isn’t meant to be a replacement for the annual scan requirement. Customers undergoing PCI certification should have a dedicated scan that includes their complete solution, therefore certifying the entire capability, not just the Amazon infrastructure.

 The principles and accompanying requirements, around which the specific elements of the DSS are organized are:

 Build and Maintain a Secure Network

Requirement 1: Install and maintain a firewall configuration to protect cardholder data

Requirement 2: Do not use vendor-supplied defaults for system passwords and other security parameters Protect Cardholder Data

Requirement 3: Protect stored cardholder data

Requirement 4: Encrypt transmission of cardholder data across open, public networks Maintain a Vulnerability Management Program

Requirement 5: Use and regularly update anti-virus software

Requirement 6: Develop and maintain secure systems and applications Implement Strong Access Control Measures

Requirement 7: Restrict access to cardholder data by business need-to-know

Requirement 8: Assign a unique ID to each person with computer access

Requirement 9: Restrict physical access to cardholder data Regularly Monitor and Test Networks

Requirement 10: Track and monitor all access to network resources and cardholder data

Requirement 11: Regularly test security systems and processes Maintain an Information Security Policy

Requirement 12: Maintain a policy that addresses information security

Many of these requirements can’t be met strictly by a datacenter provider, but in Amazon’s case, they will be able to provide an SAS70 Type 2 Audit Statement in July that will provide much of the infrastructure information needed to meet PCI DSS certification.  The Control Objectives that the Amazon Audit will address are:

 Control Objective 1: Security Organization:  Management sets a clear information security policy. The policy is communicated throughout the organization to users

 Control Objective 2: Amazon Employee Lifecycle:  Controls provide reasonable assurance that procedures have been established so that Amazon employee accounts are added, modified and deleted in a timely manner and reviewed on a periodic basis to reduce the risk of unauthorized / inappropriate access

 Control Objective 3: Logical Security:  Controls provide reasonable assurance that unauthorized internal and external access to data is appropriately restricted

Control Objective 4: Access to Customer Data:  Controls provide reasonable assurance that access to customer data is managed by the customer and appropriately segregated from other customers

Control Objective 5: Secure Data Handling:  Controls provide reasonable assurance that data handling between customer point of initiation to Amazon storage location is secured and mapped accurately

 Control Objective 6: Physical Security:  Controls provide reasonable assurance that physical access to Amazon’s operations building and the data centers is restricted to authorized personnel

Control Objective 7: Environmental Safeguards:  Controls provide reasonable assurance that procedures exist to minimize the effect of a malfunction or physical disaster to the computer and data center facilities

Control Objective 8: Change Management:  Controls provide reasonable assurance that changes (including emergency / non-routine and configuration) to existing IT resources are logged, authorized, tested, approved and documented.

Control Objective 9: Data Integrity, Availability and Redundancy:  Controls provide reasonable assurance that data integrity is maintained through all phases including transmission, storage and processing and the Data Lifecycle is managed by customers

Control Objective 10: Incident Handling:  Controls provide reasonable assurance that system problems are properly recorded, analyzed, and resolved in a timely manner.

Many thanks to Carl from Amazon for his help with this information.

Update: Since this post was published Amazon updated their PCI DSS FAQ. You can find that here.

Overcoming the EC2 Windows AMI 10GB limit

Amazon limit the Windows AMI instance to 10GB in size which almost makes the image unusable if you try and add other software within the windows C Drive. Windows is notoriously heavy on disk space and whereas 10 GB may seem a lot believe us, it isn’t when it comes to windows and a combination of windows software.

So what can you do ? Well there are three potential options:

1. You can mount an EBS volume to a directory under C: MyDigitalLife has a great article on how to achieve this. This volume will become your E:

2. If more temporary space is needed for files or downloads etc than the 10 GB limit will give you, it is possible to make temporary folders outside of  the C: partition. 

– Right-click My Computer. 
– Click Properties 
– Click Advanced 
– Click Environment Variables 
– Change the tmp and temp to whatever you want.

3.  Use a combination of Junction link magic and webdrive. Firstly install whatever you need to the D: drive and use JLM to create junctions from C to D. Junctions are effectively a combination of symbolic links, and mount points. Install WebDrive to C: and then use it to copy the program files that are on D: to Amazon s3. As D: is transient this will mean if the instance goes down You can copy everything back from S3 to D:.

I’m sure at some point Amazon will get their act together on the instance size for Windows so you don’t have to navigate you way around this but right now at least this gives you some options.

System hardening guidelines for Amazon EC2

One of the biggest questions we get from Clients is “Is Amazon EC2 secure” . That is like saying is my Vanilla network secure. Like anything you can take some steps to make the environment as secure as you can, such as:

– First read the Amazon Security Whitepaper and the Amazon discussion of Security processes

– Ensure the system key is encrypted at start-up

– Ensure you plan for load balancing in case an instance goes down. Ensure you understand all the security implications of this and harden any other instances.

– Test or emulate the performance of applications deployed to the cloud in all geographies where you plan to deploy them. The latency could vary greatly for each.

– Never ever allow password base authentication for shell access.

– Encrypt all network traffic always.

– Always encrypt everything stored on S3

– Encrypt file systems for Block devices

– Open only the minimum required ports

– Include no authentication information in any AMI images

– Think about how your system can be hardened and what is out there such as SELinux, PAX,  ExecShield etc

– Don’t allows any decryption keys into the cloud – understand the perils of keys and security

– Install host based intrusion detection system such as OSSEC

– Regularly backup Amazon instances and store them securely. 

– Use Security Groups. With EC2 security groups, you can completely isolate every tier, even internally to the EC2 cloud.

– Design in a way you can issue security patches to AMI instances

The nightmare scenario that you cannot cater for is is that Xen has unforeseen security issues which would allow inter-VM communication and which in essence would enable instance spying. Amazons doomsday scenario…..

McKinsey Cloud research kicks up a storm

A research paper on Cloud Computing by McKinsey & Company entitled ‘Clearing the Air on Cloud Computing’ has kicked up a right old storm with various luminaries either for or against it. The premise of the results of the article are that for large organisations, if they adopt the cloud model, then they would be making a mistake and most likely will lose money, as outsourcing from a more traditional data centre will likely double the cost (($150 per month per unit for data center vs $366 per month per unit for Amazon virtual cloud) . The New York times has an excellent summary of the study here.


Many of the complaints focus on McKinsey totally missing the “Private Cloud” and basing their assumptions on Public Clouds only. However there seems to be a general consensus that Amazon is too expensive and will need to adjust to survive. I’m not convinced about this. It is not the first study to suggest that Amazon are more expensive to use than a traditional data centre. Amazon seems to have been doing just fine up to now and they seem to be getting Enterprises to move across. Whether they replace a whole corporate data centre misses the point. I think this is unlikely, but for certain applications and service it makes perfect sense. Also, more competition unfolds then economics suggest that prices will naturally adjust if they need to.


You can download a PDF of the McKinsey presentation on this paper here.

Mosso come out fighting against S3 / Cloudfront with Cloudfiles and Limelight

Mosso are certainly not intent on letting Amazon have everything their own way, posting on their blog, Top 10 Reasons why Cloud Files + Limelight offers a better experience than S3 + CloudFront.

Competition is a great leveler and fuels innovation so I am glad to Mosso taking the lead here. The reasons that they give are reproduced below – I’d be interested in the thoughts of S3 / Cloudfront users as to whether these would make them consider moving across or what they think in general:


1.  World-class technical support is only one click away.

Live support, with real humans based right here in our offices, is available 24/7.  And they are really, really good!

2. World-class technical support is free.

Yes, free.  As in you don’t pay for it.  And it is really, really good support!

3. You can get started in as little as one minute.

Not a programmer?  Not a problem!  You do not have to know how to code to use Cloud Files + CDN!  Our simple web-based interface makes it a snap to share your content!

4. Limelight is a tier one CDN provider.

Really, Limelight is VERY cool, and one of the foremost CDN provider’s in the industry.  That’s why we chose to partner with them!

5. No API is required to share files.

Did we mention this already?  If so, it is worth mentioning again!

6. Language-specific APIs are available, if you need them.

Not everyone knows ReST and SOAP, so we’ve created and provide support for the following language APIs – PHP, Python, Java and .NET. We do this to allow you to work in the language you feel most comfortable with.

7. Pricing for data transfer does not vary depending on edge locations.

Data transfer starts at $0.22/GB, no matter what edge location is used to share your content. This should make it easier on you when you are trying to estimate your monthly bill.

8. There are no per requests fees for CDN.

Just another way we simplify your life, and your billing.

9. There is no limit to the number of CDN-enabled containers you can create.

As far as we can tell, you can only have up to 100 distributions in Amazon’s Cloud Front system. At Mosso, we try to keep these types of arbitrary limitations to a bare minimum, not just for Cloud Files, but for all of the services we offer.

10. The Cloud Files GUI is easy to use and navigate.

Our browser based GUI let’s you easily upload a file and share it on CDN without writing a single line of code.  Heck, you don’t even need to know a programmer to share content via Cloud Files!

When Does Amazon EC2 Reserved Instance Pricing Save Money?

The simple answer is 4643.

Amazon recently announced new pricing option where you reserve an instance for one or three years and then have discount on the hourly rate. The table below describes  the cost per year if the instance is up for the whole year.

Instance Type Cost/Year On Demand instance $ Cost/Year for 1 year reserved instance $ Cost/Year for 3 Year reserved instance $
Small
876
587.8
429.4667
Large
3504
2351.2
1717.867
Extra Large
7008
4702.4
3435.733
Medium High CPU
1752
1175.6
858.9333
Extra Large High CPU
7008
4702.4
3435.733

The above prices were calculated using Linux instance and United States prices

Cost/Year On Demand instance$ =  cost per hour * 365*24

Cost/Year for 1 year Reserved instance $ = 1 Year reserve price + (cost per hour *  365*24)

Cost/Year for 3 Year reserved instance $ = (3 Year reserve price /3)+ (cost per hour *  365*24)

Now this brings the interesting question when is it cost efficient reserve an instance ?

We can calculate the minimum usage hours over which the cheaper per hour price for a reserved instance starts to decrease the overall cost using the following formula

Instance Reserve Price/(On Demand Instance Hourly Price – Reserve Instance Hourly Price)

e.g for a small one year reserved instance

350/(0.10 – 0.03) = 4642.8 hours

For small 3 year reserved instance

500 / (0.10 – 0.03) = 7142.8 hours

The above calculated numbers are same for all instance types. So you should reserve a one year instance if you intend to use more than 4643 hours  or the instance is up 53% of the time in one year. With a 3 year reserved instance you can save money if over 3 year period you use 7143 or more hours or the instance is up 27% of the time over a  3 year time period.