Ed Snowdon’s email service shuts down – advises not to trust physical data to US companies – what are options ?

It has been a while since we did a post and a lot has happened in that time including the explosion from Edward Snowdon and the PRISM snooping revelations. These have continued to gather momentum culminating  in the email service that Snowdon used, Lavabit, closing. The owner, Ladar Levision, said that he had to walk away to prevent becoming complicit in crimes against the American public. All very cryptic and chilling. He also had this advised that he “would  strongly recommend against anyone trusting their private data to a company with physical ties to the United States.” So what to do if you have data stored on remote servers ?

Well firstly you may not care. The data you are storing may no way be sensitive and that is the key ie. you need a strategy for how you deal with sensitive data and sharing of sensitive data so what can you do ?

1. You could consider encrypting the data that is stored on cloud servers. There are various ways to do this. There are client side tools such as BoxCryptor that do a good job of this, and there are also more enterprise type platform solutions such as CipherCloud and Storage Made Easy that enable private key encryption of data stored remotely . Both can be deployed on-premise behind the corporate firewall.

2. You could consider a different policy entirely for sharing sensitive data. On a personal basis you could use OwnCloud or even setup a RaspBerry Pi as your own personal DropBox or again you could use StorageMadeEasy to create your own business cloud for keeping sensitive data behind the firewall and encrypting remote data stored outside the firewall.

The bottom line is think about your data security, have a policy, think about how you protect sensitive data.


DropBox is just a frontend to Amazon S3 with a killer sync feature

Musing about iCloud, the forthcoming SkyDrive integration into Windows 8, and Google Drive  got me thinking about DropBox, the company whose business model is built on charging when everyone else is starting to give large amounts of storage away for free. DropBox killer feature is their sync replication. It just works, and consumers have shown they love the simplicity of it. However Apple have replicated the simplicity of the sync, albeit only for iOS users, and Microsoft are now close to the same with Live Mesh.

DropBox store the files you give them on Amazon S3. This surprises many people who had assumed that they are stored on DropBox Servers. This means that the entire DropBox business model is beholden to Amazon Web Services. Amazing when you think about it, and highly illustrative that what DropBox really brings to the table is great software with a killer feature, but what is going to happen when every one else has that killer feature, with 10x to 20x more storage for free?

recent article had DropBox valued at 4 billion dollars . This is a valuation on a company doing revenues between 100-200 million dollars per year in which investors have poured in 257 million dollars in funding. Perhaps it’s me, but I just don’t see it. Yes, they have a gazillion subscribers but so what? In a commodised industry that struggles to convert more than 2% of the user base, why should that get me excited? But there is DropBox Teams for businesses right? Ever used it? Then try it and you won’t need me to draw a conclusion.

So what for DropBox if there is no mega IPO coming along? They turned down Mr Jobs (a mistake), so who else would be interested? What about Amazon? After all DropBox really is the ultimate sync client for Amazon S3. With Amazon now looking twards  private cloud it would same a match made in heaven. As with all good things, time will tell……

Dealing with MySQL issues in the Cloud: Automating restart on error

MySQL is the mainstay of most Cloud Applications (including this WordPress Blog !), however if MySQL has an issue, either through number of connections maxing out, or MySQL being locked and not available it can result in site outages. We’ve seen clients who have ended up with their SQL DB down from a couple of hours to a couple of days before they suddenly realised there was an issue.

To that end we wrote a small script that can be used to automate the restarting of MySQL in such scenarios.

The script is called mysqlrestart.sh and is listed below. You need root access to be able to use it. If you use it and  ever reboot the server you will need to login as root and run nohup ./mysqlrestart.sh  & to restart it.

Set the script to run every 30 seconds using Cron. It will then check for a number of connections and if it cannot get a connection or the number of connections is greater than the number defined (defined as 90 in the example below), it will restart mysql.



echo `date`  sqlrestart started >> run.out

while true; do

        sqlconnections=`mysql –skip-column-names -s   -e  “SHOW STATUS LIKE ‘Threads_connected'” -u root | awk ‘{print $2}’`

        #exclude myself from the number of thread connections             

        sqlconnections=$((sqlconnections – 1))

        echo `date` sqlrestart connections  $sqlconnections  >> run.out

        if [ $sqlconnections -gt $SQLCONNECTION_THRESHOLD ] || [ $sqlconnections -lt 0 ]


                echo `date` restarting mysql server $sqlconnections  >> restart.out

                service mysql restart >> restart.out 2>&1

                echo `date` restart  complete   >> restart.out


        sleep 30


Amazon Cloud is now FISMA certified: Joins Google and Microsoft

The Amazon Cloud has now classed as being FISMA certified. FISMA is an acronym for Federal Information Security Management Act. FISMA sets security requirements for federal IT systems. and is a required certification for US federal government projects.

This is the third set of certifications Amazon has recently announced coming on top of VPC ISO 27001 certification and SAS 70 Type II certification.

The accreditation covers EC2 (Amazon Elastic Compute Cloud), S3 (Simple Storage Service), VPC (Virtual Private Cloud), and includes Amazon’s underlying infrastructure.

AWS’ accreditation covers FISMA’s low and moderate levels. This level of accreditation requires a set of security configurations and controls that includes documenting the management, operational and technical processes used in securing physical and virtual infrastructure, and a requirement for third-party audits.

Other vendors who recently announced FISMA certification recently where Google with Google Apps for Government and Microsoft with the Microsoft’s Business Productivity Online Suite among cloud services (although there was a spat between Microsoft and Google regarding these claims).

Expect to see further certifications as these are a pre-requisite of expansion into lucrative government and private sector contracts as vendors feels more comfortable choosing Cloud resources as commoditisation marches on.

The Cloud and the power of “one”

One of the interesting things that about the last 12-18 months is how the Cloud has put the power into the hands of consumers. What I mean by this is, imagine the following scenario in the world pre-cloud:

“A user buys some software over the Internet or Shrink wrapped. They receive it and install it. It either does not work for them or they cannot figure out how to use it so they basically write off the cash and don’t use the App. End of story.”

Now lets looks at what happens on Cloud:

“The user either buys an Application from an App Store be it desktop or mobile, or a holiday from a holiday store, and then decided either the application is rubbish, does not work for them (or they have not RTFM) or has a bad experience on holiday. The user then use Social networks and/or the review forum on the App Store to comment on the bas experience”

In the latter case this “review” and negative experience puts off other people of trying the App / the holiday / the hotel etc. In some cases it can mean the difference between continually selling product or selling nothing as users look at the last bad review and then move somewhere else to continue their search to buy. One person can have the power to seriously undermine your whole product marketing and application strategy.

Even worse, many of the review forums (Apple’s App Store and Google Marketplace come to mind…) don’t even let you post a counter-review to explain that either the user has got it wrong, or misunderstood, or to genuinely offer to correct a bug. Worse still, some vendors can use a strategy of targeting competitive products to “put people off” purchase. In some cases this has led to the vendors involved seeking out legal action.

So what can you do to protect your product and your reputation ?

Well the first step has already been taken, you are at least thinking about it and conscious of it which is half the battle. What you should do is have your marketing or support team have a strategy that includes:

– Monitoring App Stores and review forums where your product features
– Monitoring social media for keywords about your product or company
– Set up Google Alerts keywords to inform you of keywords about your product and company
– Ensure you check your Twitter messages and also posts on LinkedIn and Facebook pages.  These are easily missed.

When you see responses, always make sure to try and follow up with the user and engage and resolve their issue, even if this means refunding them, Even if a refund seems like the last thing you want to do, offer the refund, it is not worth your reputation. As part of the process try and see if they will change their review, even if it is only to neutral.

The Cloud brings power to the masses in more ways than one and a single user can have a dramatic network effect on your business if you are not careful !

Clustrix reveals some big names using its Webscale SQL DB

An interesting article in the New York Tines online by Derrick Harris of GigaOM outlined the release of some customers using Clustrix webscale SQL database, namely, Box.net, AOL, PhotoBox, and iOffer.

Clustrix launched in early May with the claim that it’s DB technology, which features MySQL like transactionality and reliability, could scale to billions of entries. All four customers announced by Clustrix in the press release announced that this was the case.

Clustrix provide this as an appliance as an option for online services that do not want to have to invest time and effort in figuring out how to a) shard their data for better performance or b) move to something relatively new such as MongDB, or Cassandra.


Further details (which are worth reading) can be found on the original post on GigaOM.

The rise of the Cloud Data Aggregators

As storing data in the cloud becomes increasingly more normal users will increasingly find themselves in the position of needing to access different types of data regularly.  To this end we are starting to see a new breed of applications and services which themselves provide a service that interacts with data stored on the cloud. The challenge is  that services that sell their products or service based on data access are in the position of having to choose which data services to support.

This is further exacerbated in the cloud storage space as their is no ubiquitous API (see our prior post on Amazon S3 becoming a de facto standard interface).

To this end we are starting to see services an applications that themselves are offering interesting aggregations of access to data clouds. We look at a few of these below:

GoodReader, Office2 HDQuickOfficeDocuments to Go, iSMEStorage, iWork:

The iPad,  iPhone, Android have some interesting applications which function on top of existing data clouds. All the aforementioned application work in this way, either letting you view the files (in the case of GoodReader) or letting you view and edit the files (in the case of Office2, QuickOffice, Documents to Go, iWork, and iSMEStorage). The premise is that if you have data stored in an existing cloud then you can load and view or edit it in this tools and store it locally.

Tools such as iWork (which encompasses iPages, iNumbers, and iKeynote) only work with MobileMe or the WebDav standard, although the iSMEStorage App gets around this by enabling you to use iWork as an editor for files accessed through it’s cloud gateway , that can be stored on any number of clouds, using WebDav, even if the underlying cloud does not support WebDav.

In fact some companies are making data access a feature in pricing, for example,  charging extra for increased connectivity.

Gladinet.com and StorageMadeEasy.com :

Both Gladinet and SMES are unique amongst the current Cloud vendors in that they enable aggregated access to multiple file clouds. They essentially enable you to access cloud files from multiple different providers from a single file system.

Gladinet is inherently a windows only solution with many different offerings whereas Storage Made Easy also has windows software but also has cloud drives for Linux, Mac and also mobile clients for iOS, Android and BlackBerry. Gladinet is  a client side service whereas SME is a server-based service using it’s Cloud Gateway Appliance ,which is also available as a virtual appliance for VMWAre, XEN etc.

Both offering support a dizzying array of Cloud, such as, Amazon S3, Windows Azure Blob Storage, Google Storage, Google Docs, RackSpace Cloud Files etc, plus many more.

Such solutions don’t just aggregate cloud services but bring the cloud into the desktop and onto the Mobile / Tablet, making the use of cloud data much more transparent.

As data become more outsourced (to the cloud) for all types of different applications and services I expect we will see more such innovative solutions, and applications that give access to aggregated cloud data, and extend the services and tools that are provided by the native data provider.

Gawker news sites cloud security breach

If you did not notice the Gawker set of news sites recently has it’s online security compromised. You may not have heard of Gawker but you will probably know of the set of news sites they encompass which includes Gizmodo, Lifehacker, Kotaku, io9 or Jezebel. Over 1.3 million passwords where stolen and uploaded as a 500MB torrent file. Also posted where Gawker’s source code and internal employee conversations. The disclosure of this authentication information led to a viral effect with increased spam attacks, for example, on Twitter being attributed to the breach. Many users use the same web password everywhere so such a breach could leave them exposed on every site where they use the same username and password.

Apparently the passwords where encrypted in the torrent but as Gawker used an outdated encryption scheme they are relatively straightforward to crack. If you have ever registered on any of these sites then and tend to use the same username and password then you should change your username and password anywhere else you have used it on the web. Some sites are already pro-actively forcing you to do this. I receive an email from LinkedIN today that made me go through the lost password security mechanism to reset my account.

So what does this mean for Cloud ? Can one site damage the concept of storing and accessing information on the Cloud ? I think for sure, yes. It will make companies who were reticent about going to Cloud because of security concerns even more reticent, and such a breach has an effect on other sites, and I am sure we have not seen the full fallout of this yet. As for Gawker’s brand, well I think it is hugely damaging, although the web can be a fickle place, it remains to be seen how badly affected the Gawker brand will be. I can imagine potential advertisers do not want to be associated with it.

What can you do to protect yourself ? Well first, for sure change any username/password combos that are the same as the one you registered on this site, and in future consider having a separate username/password combination for each site you register. I create email addresses specifically for a registration for such sites on the web and I file them in KeepPass to be able to remember them. Ulitmately, remember, as a user don’t rely that such sites will protect your data, and as a vendor, revisit your security mechanisms to ensure the next Gawker is not you !

Amazon S3, EC2 and VPC ISO 27001 certified

As well as being SAS 70 Type II-certified Amazon is now ISO 27001 certified. ISO/IEC 27001 formally outlines a management system that brings information security under management control, and mandates requirements that have to be met. Organisations that have adopted ISO/IEC 27001 may be formally audited to maintain compliance with the standard.

As stated on WikiPedia:

SO/IEC 27001 requires that management:

Systematically examine the organization’s information security risks, taking account of the threats, vulnerabilities and impacts;

Design and implement a coherent and comprehensive suite of information security controls and/or other forms of risk treatment (such as risk avoidance or risk transfer) to address those risks that are deemed unacceptable; and

Adopt an overarching management process to ensure that the information security controls continue to meet the organization’s information security needs on an ongoing basis.

“Amazon Web Services is continuing its commitment to provide further assurance of AWS security controls and practices through third-party audits and certifications such as SAS 70 Type II and ISO 27001,” said Stephen Schmidt, Chief Information Security Officer for Amazon Web Services.

“Via ISO 27001 and other certifications, we continue to provide our customers with confidence that our security controls and practices follow internationally-recognized security standards.”

You can learn more about Amazon and it’s compliance and security provisions here.

Cloud Failure – Files cannot be downloaded from Box.net

Again the ugly issue of what do you do when the cloud goes wrong rears it’s head. Right now if you login to box.net and try and download a file you cannot download a file. Instead you get a screen like the below. I’m sure Box are aware of this, but it again shows you the total reliance you have on an outsourced infrastructure on the cloud, and their problems become your problems.

Picture 12