They note the issues seem only to be hitting the U.S. Standard centers — other S3 centers including Northern California, Europe and Asia are functioning correctly.

Amazon have introduced a new storage option for Amazon S3 called Reduced Redundancy Storage (RRS) that enables customers to reduce their costs by storing non-critical, reproducible data at lower levels of redundancy than the standard storage of Amazon S3.

It provides a cost-effective solution for distributing or sharing content that is durably stored elsewhere, or for storing thumbnails, transcoded media, or other processed data that can be easily reproduced. The RRS option stores objects on multiple devices across multiple facilities, providing 400 times the durability of a typical disk drive, but does not replicate objects as many times as standard Amazon S3 storage does, and thus is even more cost effective.

Both storage options are designed to be highly available, and both are backed by Amazon S3’s Service Level Agreement.

Once customer data is stored using either Amazon S3’s standard or reduced redundancy storage options, Amazon S3 maintains durability by quickly detecting failed, corrupted, or unresponsive devices and restoring redundancy by re-replicating the data. Amazon S3 standard storage is designed to provide 99.999999999% durability and to sustain the concurrent loss of data in two facilities, while RRS is designed to provide 99.99% durability and to sustain the loss of data in a single facility.

Pricing for Amazon S3 Reduced Redundancy Storage starts at only $0.10 per gigabyte per month and decreases as you store more data.

From a programming viewpoint to enable your storage to take advantage of RRS  you need to set the storage class of an object you upload to RRS. To enable this you set x-amz-storage-class to REDUCED_REDUNDANCY in a PUT request.

First, what is not going to happen anytime soon:

- From what we heard Amazon are not going to resolve the issue of billing in local currency with locally issued invoices any time soon. See our prior post on this topic. We did learn however that large organisations can request an invoice.

- Right now if you want to use your own AMI image to sell on a SaaS basis using Amazon infrastructure you have to a US organisation. Again Amazon don’t seem to have plans to change this in the immediate timeframe so that leaves out any organisation outside of the US who want to sell their product offering as SaaS on Amazon’s web services infrastructure unless they integrate their own commerce infrastructure and not use DevPay. This can be both a blessing (charge margin on Amazon’s infrastructure pieces like AMQS) but also a curse (can leave you exposed as you will be month behind in billing your clients). Even though Amazon are entrenched right now as the Public Cloud infrastructure of choice, it wouldn’t be the first time we have seen 100 pound gorilla displaced from it’s prime market position. If I were Amazon, I’d fix this and soon. Microsoft and RackSpace are looking more attractive all the time.

- Amazon’s ingestion services again require you to be a US organisation with a US return address. Are you detecting a common theme here….

And what we can expect to see soon:

- VPC (Virtual private cloud) access is in private beta now. This is a mechanism for securely connecting public and private clouds within the EC2 infrastructure.

- High memory instances analogous to High CPU instances are in the pipeline

- Shared EBS is in the pipeline

- Functionality for Multiple users associated with a single account is in the pipeline and will provide simple privileges too. This has long been a bone of contention for organisations using AWS so will be welcomed.

- Amazon is planning to have lot more EC2 workshops through local partners.

Other things of note that we learnt where:

- We learned that large physical instances currently have their own dedicated blade / box.

- As AWS has grown, large number of machines are available and organizations can request hundreds of machines easily. Even extreme cases are catered for i.e. even requests for 50000 machines.

- As a matter of policy new functionally will be rolled out simultaneously in EU and US unless there is a good reason.

All in all some exciting stuff, and there was other things in the pipeline they could not share, but the public cloud market is starting to get more players and I think Amazon need to get some of their infrastructure pieces in place sooner rather than later.

 Earlier this year I wrote about Amazon Elastic MapReduce and the ways in which it can be used to process large data sets on a cluster of processors. Since the announcement, our customers have wholeheartedly embraced the service and have been doing some very impressive work with it (more on this in a moment).

Today I am pleased to announce Amazon Elastic MapReduce job flows can now be run in our European region. You can launch jobs in Europe by simply choosing the new region from the menu. The jobs will run on EC2 instances in Europe and usage will be billed at those rates.

 Because the input and output locations for Elastic MapReduce jobs are specified in terms of URLs to S3 buckets, you can process data from US-hosted buckets in Europe, storing the results in Europe or in the US. Since this is an internet data transfer, the usual EC2 and S3 bandwidth charges will apply.

Our customers are doing some interesting things with Elastic MapReduce.

 At the recent Hadoop Summit, online shopping site ExtraBux described their multi-stage processing pipeline. The pipeline is fed with data supplied by their merchant partners. This data is preprocessed on some EC2 instances and then stored on a collection of Elastic Block Store volumes.The first MapReduce step processes this data into a common format and stores it in HDFS form for further processing. Additional processing steps transform the data and product images into final form for presentation to online shoppers. You can learn more about this work in Jinesh Varia’s Hadoop Summit Presentation.

Online dating site eHarmony is also making good use of Elastic MapReduce, processing tens of gigabytes of data representing hundreds of millions of users, each with several hundred attributes to be matched. According to an article on SearchCloudComputing.com, they are doing this work for $1,200 per month, a considerable savings from the $5,000 per month that they estimated it would cost them to do it internally.

We’ve added some articles to our Resource Center to help you to use Elastic MapReduce in your own applications. Here’s what we have so far:

• Developer Peter Skomoroch wrote about Finding Similar Items with Amazon Elastic MapReduce, Python, and Hadoop Streaming. This comprehensive article shows how to run a multi-stage processing pipeline to compute pairwise similarity in a large database of items using the Pearson correlation coefficient.  
•  We also have an article on Processing and Loading Data from Amazon S3 to the Vertica Analytic Database. This article shows how to use Elastic MapReduce to load the some Freebase data using the MapReduce model. The Map phase converts tab-delimited values into quoted strings and does some other data cleanup. The Reduce phase then performs batch insert operations, which the Vertica JDBC driver transforms into bulk loads. The code resides in a customer JAR and the article shows how to arrange for it to be loaded at run-time.  
•  If you want to learn more about Elastic MapReduce, you should definitely check out our Introduction to Elastic MapReduce. This PDF walkthrough will introduce you to the service with an example which shows you how to load data into Amazon SimpleDB.  

You should also check out AWS Evangelist Jinesh Varia in this video from the Hadoop Summit:

– Jeff;

PS – If you have a lot of data that you would like to process on Elastic MapReduce, don’t forget to check out the new AWS Import/Export service. You can send your physical media to us and we’ll take care of loading it into Amazon S3 for you.

What I am looking for is a way to put some control back into an Organisations hands, including:

- A way to grant access to more  granular access to users and therefore track who /which department in the company is using the service

- Central Management of billing, and an actual Invoice that can be submitted for recompense either to a another company or internally

- Ability to set budget limits, akin to what you can do to Google Adwords. 

- Alerting mechanisms to SMS when budgets near tolerance levels

- Ability to centrally track usage data so that chargeaback mechanisms can cleanly be built and used

There are numerous threads on the Amazon Web Service Community forum asking for hard copy invoices . Amazon does provide a page for tax help but its not that helpful

Just some of the things floating around on the thread:

“Sounds silly, isnt’t it? But really, you can shake your head as long as you want, but tax authorities will not accept an invoice which does not state both partie’s VAT-ID number (here in italy, but its the same all over europe). 
If i go to dinner with my clients, the waiter will bring the bill in a carbon copy chemical paper. I HAVE to write my VAT-ID and full company name on it. 
Only THEN, he separates the first from the second sheet of paper, one stays in his records, one in my. 

If they check my books and find an invoice or bill which is not complaint to the formal requirements of having VAT-ID of both parties, they will not accept it and make you pay a fine. Its silly to discuss about the meaning of this, you would have to listen to a very long story about what cross-checks they do with these VAT-IDs. 

Any way, it’s not necessary that you send me a printed invoice, i can print it myself. But IT IS NECESSARY, that the invoice states clearly: 

name, address and VAT-ID of the seller 
name, address and VAT-ID of the purchaser 
description of goods and services 
invoice date, invoice number 

if any of these things are missing, the sheet of paper simply is not an invoice and trying to book it as an expense is a violation of law. 

Currently we are not able to detract AWS expenses of a few 100 US$/month due to these limitations.”

Reply to this post:

“In Czech it is even worse … we have to have hard copy with hand-writen _signature_ to be valid for tax authorities. Problems implications are then quite clear. Silly, but real in Czech. Another more detail, we can not add dinner with customer to our taxes. It has to be paid from the company net profit. “

Another example Reply:

“The same here in germany, we want to start using AWS for some projects but without a proper invoice our accounting will not give us a “go”. 

If this won’t change within this month we will either continue to work with dedicated server networks or might try the google appspot. 

Thats really a shame, because amazon does obviously know how to write correct invoices for amazon.com/.de. ”

I believe that this is probably tax related, with Amazon not wanting to amass taxes for Regional entities that would be liable for country specific tax, but its a great hole right now and I don’t have much doubt that it stops further adoption of the services themselves as organisational procedures are pretty inflexible when dealing with these issues.

Cedric Roll, Co-Founder, ORbyte Solutions http://www.vimeo.com/4409867

Felipe Padilla, Co-Founder, Skipso http://www.vimeo.com/4409569

Nigel Hamilton, CEO, Turbo10.com http://www.vimeo.com/4409682

Simone Brunozzi, Getting Started with AWS http://www.vimeo.com/4411474

Tal Saraf, Accelerating Your Website with CloudFront http://www.vimeo.com/4409756

PCI DSS is a set of comprehensive requirements for enhancing payment account data security and was developed by the founding payment brands of the PCI Security Standards Council, including American Express, Discover Financial Services, JCB International, MasterCard Worldwide and Visa Inc. Inc. International, to help facilitate the broad adoption of consistent data security measures on a global basis.

The PCI DSS is a multifaceted security standard that includes requirements for security management, policies, procedures, network architecture, software design and other critical protective measures. This comprehensive standard is intended to help organizations proactively protect customer account data.

So, is it possible to create a PCI DSS compliant application that can be deployed to EC2 ?

In order for an application or system to become PCI DSS compliant requires an end to end system design (or a review if pre-existing) and implementation.  In the case of AWS customer’s attaining PCI compliance (certification), they would have to ensure they met all of the prescribed requirements through the use of encryption etc. very much like other customers have done with HIPAA applications.  The AWS design allows for customers with varying security and compliance requirements to build to those standards in a customized way. 

There are different levels of PCI compliance and the secondary level is quite a straight forward configuration, but requires additional things such as 3^rd party external scanning (annually).  You can find an example here of the PCI Scan report that is done on a quarterly basis for the Amazon platform.  This isn’t meant to be a replacement for the annual scan requirement. Customers undergoing PCI certification should have a dedicated scan that includes their complete solution, therefore certifying the entire capability, not just the Amazon infrastructure.

 The principles and accompanying requirements, around which the specific elements of the DSS are organized are:

 Build and Maintain a Secure Network

Requirement 1: Install and maintain a firewall configuration to protect cardholder data

Requirement 2: Do not use vendor-supplied defaults for system passwords and other security parameters Protect Cardholder Data

Requirement 3: Protect stored cardholder data

Requirement 4: Encrypt transmission of cardholder data across open, public networks Maintain a Vulnerability Management Program

Requirement 5: Use and regularly update anti-virus software

Requirement 6: Develop and maintain secure systems and applications Implement Strong Access Control Measures

Requirement 7: Restrict access to cardholder data by business need-to-know

Requirement 8: Assign a unique ID to each person with computer access

Requirement 9: Restrict physical access to cardholder data Regularly Monitor and Test Networks

Requirement 10: Track and monitor all access to network resources and cardholder data

Requirement 11: Regularly test security systems and processes Maintain an Information Security Policy

Requirement 12: Maintain a policy that addresses information security

Many of these requirements can’t be met strictly by a datacenter provider, but in Amazon’s case, they will be able to provide an SAS70 Type 2 Audit Statement in July that will provide much of the infrastructure information needed to meet PCI DSS certification.  The Control Objectives that the Amazon Audit will address are:

 Control Objective 1: Security Organization:  Management sets a clear information security policy. The policy is communicated throughout the organization to users

 Control Objective 2: Amazon Employee Lifecycle:  Controls provide reasonable assurance that procedures have been established so that Amazon employee accounts are added, modified and deleted in a timely manner and reviewed on a periodic basis to reduce the risk of unauthorized / inappropriate access

 Control Objective 3: Logical Security:  Controls provide reasonable assurance that unauthorized internal and external access to data is appropriately restricted

Control Objective 4: Access to Customer Data:  Controls provide reasonable assurance that access to customer data is managed by the customer and appropriately segregated from other customers

Control Objective 5: Secure Data Handling:  Controls provide reasonable assurance that data handling between customer point of initiation to Amazon storage location is secured and mapped accurately

 Control Objective 6: Physical Security:  Controls provide reasonable assurance that physical access to Amazon’s operations building and the data centers is restricted to authorized personnel

Control Objective 7: Environmental Safeguards:  Controls provide reasonable assurance that procedures exist to minimize the effect of a malfunction or physical disaster to the computer and data center facilities

Control Objective 8: Change Management:  Controls provide reasonable assurance that changes (including emergency / non-routine and configuration) to existing IT resources are logged, authorized, tested, approved and documented.

Control Objective 9: Data Integrity, Availability and Redundancy:  Controls provide reasonable assurance that data integrity is maintained through all phases including transmission, storage and processing and the Data Lifecycle is managed by customers

Control Objective 10: Incident Handling:  Controls provide reasonable assurance that system problems are properly recorded, analyzed, and resolved in a timely manner.

Many thanks to Carl from Amazon for his help with this information.

So what can you do ? Well there are three potential options:

1. You can mount an EBS volume to a directory under C: MyDigitalLife has a great article on how to achieve this. This volume will become your E:

2. If more temporary space is needed for files or downloads etc than the 10 GB limit will give you, it is possible to make temporary folders outside of  the C: partition. 

- Right-click My Computer. 
- Click Properties 
- Click Advanced 
- Click Environment Variables 
- Change the tmp and temp to whatever you want.

3.  Use a combination of Junction link magic and webdrive. Firstly install whatever you need to the D: drive and use JLM to create junctions from C to D. Junctions are effectively a combination of symbolic links, and mount points. Install WebDrive to C: and then use it to copy the program files that are on D: to Amazon s3. As D: is transient this will mean if the instance goes down You can copy everything back from S3 to D:.

I’m sure at some point Amazon will get their act together on the instance size for Windows so you don’t have to navigate you way around this but right now at least this gives you some options.