Solving Time Sync Issue on Azure

We just came off an Azure project and we thought it would be useful to push out our notes on keeping a hosted server time in sync.

1. We were configuring a Linux hosted server on Azure and thehe NTP protocol uses UDP on port 123. But you don’t have to allow that on ‘iptables’ in Linux – NTP just gets passed through.

2.  On Azure you don’t have to define the port in the VM configuration, like you do for 22/80/443.

3. Old) posts say Azure doesn’t support UDP, but it seems to now.

4. In theory, Azure provides a service “time.windows.com” but it was 80ms behind the standard servers at [0123].centos.pool.ntp.org.

After configuring the clock on the hosted Linux appliance starts drifting very quickly.
The problem seems to be that there are problems with the time sync in Hyper-V on Windows Server 2008, which is what Azure is built on.

The solution is to look at the changes required to grub.conf and ntp.conf as described at:

http://www.hardanswers.net/correct-clock-drift-in-centos-hyper-v

Hardening RedHat (CentOS) Linux for use on Cloud

If you next to deploy Linux on Cloud you should consider hardening the Linux instance prior to any deployment. Below are guidelines we have pulled together with regards to hardening a RedHat or CentOS instance.

Hardening Redhat linux guidelines

enable selinux

Ensure that /etc/selinux/config includes the following lines:
SELINUX=enforcing
SELINUXTYPE=targeted

Run the following on commandline to allow httpd to create outbound network connections
setsebool httpd_can_network_connect=1

check using
sestatus
To enable/disable
echo 1 >/selinux/enforce

disable the services

chkconfig anacron off
chkconfig autofs off
chkconfig avahi-daemon off
chkconfig gpm off
chkconfig haldaemon off
chkconfig mcstrans off
chkconfig mdmonitor off
chkconfig messagebus off
chkconfig readahead_early
chkconfig readahead_early off
chkconfig readahead_later off
chkconfig xfs off

Disable SUID and SGID Binaries

chmod -s /bin/ping6
chmod -s /usr/bin/chfn
chmod -s /usr/bin/chsh
chmod -s /usr/bin/chage
chmod -s /usr/bin/wall
chmod -s /usr/bin/rcp
chmod -s /usr/bin/rlogin
chmod -s /usr/bin/rsh
chmod -s /usr/bin/write

Set Kernel parameters

At boot, the system reads and applies a set of kernel parameters from /etc/sysctl.conf. Add the following lines to that file to prevent certain kinds of attacks:

net.ipv4.conf.all.rp_filter=1
net.ipv4.conf.all.accept_source_route=0
net.ipv4.icmp_echo_ignore_broadcasts=1
net.ipv4.icmp_ignore_bogus_error_messages=1
kernel.exec-shield=1
kernel.randomize_va_space=1

Disable IPv6

Unless your policy or network configuration requires it, disable IPv6. To do so, prevent the kernel module from loading by adding the following line to /etc/modprobe.conf:
install ipv6 /bin/true
Next, add or change the following lines in /etc/sysconfig/network:
NETWORKING_IPV6=no
IPV6INIT=no

Nessus PCI Scan

Upgrade openssh to latest version

upgrade bash to latest version

http://www.thecpaneladmin.com/upgrading-openssh-on-centos-5/

Set HTTP headers off

In /etc/httpd/conf/httpd.conf set the following values
ServerTokens Prod
ServerSignature Off
TraceEnable off

In /etc/php.ini set
expose_php = Off

Change MySQL to listens on only localhost

Edit /etc/my.cnf and add following to mysqld section
bind-address = 127.0.0.1

Make sure only port 80 443 21 are open

vi /etc/sysconfig/iptables
and add
ACCEPT tcp anywhere anywhere state NEW tcp dpt:https
ACCEPT tcp — anywhere anywhere state NEW tcp dpt:ftp

Comprehensive overview of PaaS Platforms

Looking to implement a PaaS. Wondering what product to start with or how they compare ? Well, there may not be an App for that but there is a collaborative spreadsheet.

To view the spreadsheet directly on Google Docs click here (it seems Google only supports 50 concurrent connections of a spreadsheet so if you have an issue try again later)

Using the Power of Cloud Computing with SaaS Services

We recently started documenting the services we use on a day-to-day basis and it struck us just how much we use SaaS and Cloud Computing services. To that end we thought it would be fun / beneficial to share some of these and how and why we use them:

File Server: StorageMadeEasy

We use SkyDrive (25 GB free) in conjunction with Amazon S3 and an in-house PogoPlug installation to store files. We use the Storage Made Easy Cloud File Server to provide a unified view of all of ur files (we are trialling PogoPlug support with them). This also enables us to assign user / file permissions and governance on the consolidated information stores which is very useful. We can also create collaboration groups across the consolidated information stores for sharing with clients and where the files are stored is abstracted (to the client). We also use the service  to managea nd share files on our iPad (tablet) and Windows Phone 7 clients.

CRM: Zoho CRM

Having used several different CRM systems over time, personally we prefer Zoho. For up to 3 users it is free and adding users and services is reasonably priced. It’s also very easy to change the default templates and to use (using the HTML5 mobile App). SalesForce is synonymous with CRM systems and SaaS, but Zoho is good (cheaper) alternative.

Project Management: Basecamp

Basecamp can be a little expensive but if you manage projects and want to collaborate it is hard  to beat as it’s simplicity and easy to use web interface stand the test of time.

Source Code / Bug Tracker: BitBucket

There are many source code and bug tracking systems, many of them free but we like BitBucket. It’s free for up to 3 users and is a solid source code and bug tracker. For source code editing we use Textastic which enables us to hook into specific source files stored on SkyDrive via WebDav using the SMEStorage CloudDav feature which is enabled when the iPad app is purchased.

Analytics: Google

Google Analytics a essential for tracking website statistics. There are alternatives (such as the fantastic Piwik) but it is hard to beat for ease of use. It is not flawless, the lack of ability to track IP addresses (and therefore do a reverse DNS lookup) is frustrating for example. We use AnalyticsPro on the iPad for mobile access/tracking.

Email: Gmail

Google Apps GMail is a great email system. We’ve used it for years and only have good things to say about it. We backup our Gmail to SkyDrive using the SMEStorage Cloud File Server so that it can be indexed and searchable along with our other files, and of course also for resiliency. For offline access on our iPad’s we use a customised version of Remail that we enhanced for the iPad.

Inbound Lead Tracking: Leadlander

LeadLander enables us to track companies visiting our website, how often they visit, and the new people feature enables us to directly contact leads. It’s a great service and great value at a couple of thousand dollars per year.

Server Monitoring: Server Density & Pingdom & WatchMouse & PagerDuty

Server Density is great for monitoring thresholds on Apache, Server Processes, MySQL connections etc. We plug this into PagerDuty so we can be alerted by phone if major thresholds are breached. Server Density also provide an iOS App for push alerts. Pingdom is used an added check for server outages, and WatchMouse is used to check quality of service for times taken to load pages on a site.

MySQL Admin: PHPMyAdmin

We tend to use the command line but if we need to access MySQL graphically we use PHPMyAdmin.

Call Services: e-Receptionist and Skype and TollFreeforwarding

e-Receptionist is great for virtual teams if you want to route calls to Sales / Support etc and you are not physically in the same location. Skype of course needs no explanation, except that we back up our Skype conversations to SkyDrive using the SMEStorage Cloud File Server so they can be indexed and searchable as we do a lot of communication over Skype. TollFreeForwarding is used to give an international number and then route into the e-receptionist infrastructure.

Online Marketing: Google Adwords and BuySellAds.com and LinkedIn Ads

Google Adwords is synonymous with online marketing and can be a great sales tool if used correctly, and BuySellAds is useful to advertise services to targeted sites using banner ads. LinkedIn Ads is a great way to reach a very targeted audience and in our experience is a great way to compliment any online marketing campaign.

Invoicing Services: BlinkSale

Blinksale is a great, simple SaaS invoicing service that is low cost and very easy to use and administer. There are many others but for simplicity you can’t beat BlinkSale.

Blog: WordPress

There are many others, such as Google Sites, and Tumblr, but there are so many ways to use WordPress and so many plug-in’s that there really is nothing else to compete with it.

Social: Twitter and FriendFeed and Identi.ca and Google Plus and Hootsuite

We use a variety of social sites to push out news. There are of course so many now that it would be impossible to list them all but we covered the major ones. If you use multiple Twitter or social accounts then Hootsuite is a must.

……. and there you have it. The types of SaaS cloud services a distributed virtual company can use to run their business.

The Cloud and the power of “one”

One of the interesting things that about the last 12-18 months is how the Cloud has put the power into the hands of consumers. What I mean by this is, imagine the following scenario in the world pre-cloud:

“A user buys some software over the Internet or Shrink wrapped. They receive it and install it. It either does not work for them or they cannot figure out how to use it so they basically write off the cash and don’t use the App. End of story.”

Now lets looks at what happens on Cloud:

“The user either buys an Application from an App Store be it desktop or mobile, or a holiday from a holiday store, and then decided either the application is rubbish, does not work for them (or they have not RTFM) or has a bad experience on holiday. The user then use Social networks and/or the review forum on the App Store to comment on the bas experience”

In the latter case this “review” and negative experience puts off other people of trying the App / the holiday / the hotel etc. In some cases it can mean the difference between continually selling product or selling nothing as users look at the last bad review and then move somewhere else to continue their search to buy. One person can have the power to seriously undermine your whole product marketing and application strategy.

Even worse, many of the review forums (Apple’s App Store and Google Marketplace come to mind…) don’t even let you post a counter-review to explain that either the user has got it wrong, or misunderstood, or to genuinely offer to correct a bug. Worse still, some vendors can use a strategy of targeting competitive products to “put people off” purchase. In some cases this has led to the vendors involved seeking out legal action.

So what can you do to protect your product and your reputation ?

Well the first step has already been taken, you are at least thinking about it and conscious of it which is half the battle. What you should do is have your marketing or support team have a strategy that includes:

– Monitoring App Stores and review forums where your product features
– Monitoring social media for keywords about your product or company
– Set up Google Alerts keywords to inform you of keywords about your product and company
– Ensure you check your Twitter messages and also posts on LinkedIn and Facebook pages.  These are easily missed.

When you see responses, always make sure to try and follow up with the user and engage and resolve their issue, even if this means refunding them, Even if a refund seems like the last thing you want to do, offer the refund, it is not worth your reputation. As part of the process try and see if they will change their review, even if it is only to neutral.

The Cloud brings power to the masses in more ways than one and a single user can have a dramatic network effect on your business if you are not careful !

GigaSpaces take e-Gaming to the Cloud with Yazino

New York & London, December 7, 2010GigaSpaces Technologies, a leading provider of a new generation of application platforms for Java and .Net environments, has provided Yazino, a massively multiplayer online casino, the application infrastructure used to build the first cloud-based, social casino platform. Yazino is using GigaSpaces’ eXtreme Application Platform (XAP) to scale on demand while reducing costs and speeding time to market.

As a social gaming online casino, we knew from the beginning that scalability was critical to our success,” says Hussein Chahine, Yazino’s Founder and CEO. “With XAP, we have the flexibility to meet constantly changing business volumes, with linear, real-time dynamic scalability using a cloud-data center hybrid model.

Yazino fuses social interaction and multiplayer functionality, building a bridge between traditional online gambling and social gaming sites. Yazino has already registered more than 500,000 players, with more than 10,000 new players joining daily.

XAP provides a unique enterprise-grade, end-to-end application scalability platform, which can handle extremely large volumes, be scaled out (or in) in real-time, and is governed by preset business SLAs, and which is ultra-fast due to the whole platform running in memory. A strategic solution enhancing IT efficiency and agility, it guarantees performance under peak demand while improving hardware utilization by up to 500%. It allows developers to build, deploy, and operate their infrastructures in any environment without a single code change.

Yazino benefits from our long experience with cloud development,” says Adi Paz, GigaSpaces Executive Vice President of Marketing and Business Development. “With XAP as the underlying infrastructure, our clients like Yazino can focus on the business logic and speed time to market without concern for where the application will actually run.

By using XAP, Yazino now has a massively multiplayer/multi-game online casino platform optimized for extremely high throughput, with hundreds of thousands of concurrent, interactive players, while giving each player excellent response time.

GigaSpaces technology helped us build a hybrid infrastructure, where we can leverage the best of the cloud’s economies of scale while ensuring our data center can manage all of the regulatory-related processing,” continued Hussein. “This provides Yazino a valuable competitive edge as we use more costly hosting only for what is required by regulation, while other services sit entirely on the cloud.

About GigaSpaces

GigaSpaces Technologies is a leading provider of a new generation of virtualized application platforms. Our flagship product, eXtreme Application Platform (XAP), delivers end-to-end scalability across the entire stack, from the data all the way to the application. XAP is the only product that provides a complete in-memory solution on a single platform, enabling high-speed processing of extreme transactional loads, while scaling to meet any requirement – dynamically and linearly. XAP was designed from the ground up to support any cloud environment – private, public, or hybrid – and offers a pain-free, evolutionary path from today’s data center to the technologies of tomorrow.

More than 350 organizations worldwide are leveraging XAP to enhance IT efficiency and performance. Among our customers are Fortune Global 500 companies, including top financial services enterprises, telecom carriers, online gaming providers, and e-commerce companies, such as Dow Jones, NYSE, Société Générale, Virgin Mobile, and Sears.

About Yazino
Yazino, the world’s first social casino (www.yazino.com) was conceived by three friends who wanted to reinvent the social gaming and online casino worlds by connecting the two together. Yazino has built a bridge between traditional online gambling and social gaming sites, creating a whole new hybrid category. The entire brand and in-game experience is entertaining and social to its core. Yazino offers a uniquely fun and competitive environment to connect the world around casino games, such as Blackjack, Roulette, Texas Hold’em and Slots.  Constantly refreshed multiplayer content, tournaments and the engaging challenge of levels and achievements allow Yazino to define the next generation of online gambling.

Yazino, a wholly owned subsidiary of Yazino Group AG (Switzerland), was founded by Hussein Chahine, Bijan Khezri and Gojko Adzic in 2008.

The Cloud – A disruptive game changer – just ask Nokia !

iStock_000000579002XxxSmallIt’s often said that the Cloud will be a game changer and disruptive and that statement is put out there for the future but I believe we have already seen a huge example of this in the mobile telecommunications domain.  We have seen in the last twelve months the beginning of a fundamental change in a users relationship with services because of the ability of real time delivery over mobile and fixed broadband. Apple single handedly changed the perception of, not only what dollar value a user would pay, but that they would actually pay at all. In the first 60 days Apple had 100 million downloads from their App Store.  Just think about this…60 days, 100 million downloads ! Phenomenal. Even more phenomenal is they ripped up the script of the established model and established their own.

We have quickly seen other providers such as RIM, and Google quickly adopt the same model, with Nokia lagging behind and then news filtering out that they would launch an App Store at the Mobile World Conference in Barcelona and when they did,  well lets just say that it was not exactly a success.  Microsoft, late to the party as always, are also jumped on the bandwagon with the launch of their “My Phone” service . Samsung have also now launched their own Mobile Applications MarketPlace. This shift has hugely changed the whole model of the Telco market. Nokia, the 100 pound gorilla, is losing market share hand over fist as it struggles to get to grips with this new model. Motorola has lost $3.6 billion as they too struggle to get to grips with this new consumer model.

In 1 year Apple has become the eighth largest mobile phone vendor in the world (source: strategy analytics). This whilst only competing in the smartphone market and, at the time of the report, not selling into markets such as China. Overall during the past March quarter mobile phone sales fell 13% worldwide, the fastest rate of annual decline since record began, but in contrast sales rose 10% in the US, largely because of Apple. The top 5 handset vendors saw their market share fall from 83.5% to 78%, a decline that is predicted to continue as Android comes of age and Apple continues it’s dominance with low-end entry points into the consumer market.

The whole notion of how to sell to an individual has changed, it has become from the edge and back rather than the reverse i.e. it has proved that users are wiling to not only pay for real-time services and just-in-time applications, but will actually choose their handset provider based on the perceived value and breadth of those services. How many times have you read of a competitive phone review, “In some ways it is a better handset than the iPhone but it just cannot match the App Store for breadth of Appllications”. Interestingly not everyone agrees. MobileCrunch recently ran an article, “Not every Company needs an App Store“.  I believe they miss the point. The rules have changed and the humble phone has changed to become a platform to deliver services aided by on-demand cloud applications and services.  

I agree that ideally we would be able to write against one platform for services delivered. Unfortunately the mobile phone OS market is very segmented with lots of players such as Symbian (Nokia), Microsoft, Google, iPhone etc. Having said that their are some initiatives to try and provide some abstraction to allow code / services written for one platform to run on others, such as PhoneGap, which supports iPhone, Android and Blackberry. Ultimately the Genie is out of the bottle and new mobile companies can see the carrot that is new revenue and business models that Apple has made reality. Ultimately the will have no choice, with an Open Source OS model in Android squeezing them from one side, and Apple on the other, the landscape is being changed and the 100 pound gorilla is starting to look like an endangered species. Figures compiled by Gartner show that Apples Market share more than doubled in 2008, whilst Nokia’s Market share of the global smartphone Market fell from 47% in 2007 to 31% in 2008, and based on projections in the Gartner analysis, this would make Apple the leading global smartphone provider by 2011.

Supporting SLA’s on the Cloud

What does it take to make a Cloud Computing infrastructure enterprise ready ? Well, as always, this probably depends on the use case, but support for real-time scaling and SLA support must figure highly.

Software that purports to scale the applications on the cloud is not new, have a look at our prior blog post on this topic, and you will see some of the usual suspects such as RightScale, and Scalr. A new offering in this space is by Tibco with its Tibco Silver offering. Tibco Silver is trying to solve the problem of not whether cloud services can scale but whether the applications themselves can scale with them. This problem is addressed by Silver through ‘self aware elasticity’. Hmmm….sounds good but what exactly does that mean ? It means the system can automatically provision new cloud capacity (be that storage or compute) dependent upon fluctuations in application usage.

According to Tibco, unlike services in a service-oriented architecture cloud services are not aware of the SLA’s to which they are required to adhere and Tibco Silver is aimed at providing this missing functionality. Tibco claim that “Self-aware elasticity” is something no other vendor has developed. I would dispute this. GigaSpaces XAP with it’s ability to deploy to the cloud as well as on-premise using the same technology has very fine grained application level SLA control that when breached allows the application to react accordingly, whether this be to increase the number of threads, provision new instances or to distribute workloads in a different way. GigaSpaces Service Grid technology enables support for this real-times elasticity.  The GigaSpaces Service Grid originated from Sun’s RIO Project. (interestingly it seems GigaSpaces are doing some work on enabling their cloud tools to deploy to and manage VMWARE images on private clouds as they do with AMI’s on Amazon’s public cloud) 

Without a doubt the ability to react in real-time to application level SLA’s rather than just breaches of an SLA at an infrastructure level is something that will find a welcome home in both private and public clouds.

A brief overview of the Windows Azure Cloud Platform

Azure

The Windows Azure cloud platform includes Windows Azure, a Web-based Microsoft SQL Azure, and connectivity /  interoperability services with .NET Services. As with other Cloud platforms, Azure is a consumption-based pricing model.  A summary of the pricing is below and more details can be found here:

– Compute @ $0.12 / hour
– Storage @ $0.15 / GB stored
– Storage Transactions @ $0.01 / 10K

SQL Azure:

– Web Edition – Up to 1GB relational database @ $9.99
– Business Edition – Up to 10GB relational database @ $99.99

.NET Services:

Messages @ $0.15/100K message operations, including Service Bus messages and Access Control tokens

Bandwidth across all three services will be charged at $0.10 in / $0.15 out / GB.

The Windows Azure blog details a service-level agreement that covers service uptime, connectivity, and data availability. The Azure SDK can be downloaded here.

It remains to be seen how Azure pans out but I am sure it will be a serious cloud player and it is nice to see a little pressure put on Amazon not only in terms of pricing but also for the intellectual ‘one-man upmanship’ this is sure to bring which can only be good for all companies working with cloud.

Amazon Elastic MapReduce now available in Europe

From the Amazon Web Services Blog:

 Earlier this year I wrote about Amazon Elastic MapReduce and the ways in which it can be used to process large data sets on a cluster of processors. Since the announcement, our customers have wholeheartedly embraced the service and have been doing some very impressive work with it (more on this in a moment).

Today I am pleased to announce Amazon Elastic MapReduce job flows can now be run in our European region. You can launch jobs in Europe by simply choosing the new region from the menu. The jobs will run on EC2 instances in Europe and usage will be billed at those rates.

 Because the input and output locations for Elastic MapReduce jobs are specified in terms of URLs to S3 buckets, you can process data from US-hosted buckets in Europe, storing the results in Europe or in the US. Since this is an internet data transfer, the usual EC2 and S3 bandwidth charges will apply.

Our customers are doing some interesting things with Elastic MapReduce.

 At the recent Hadoop Summit, online shopping site ExtraBux described their multi-stage processing pipeline. The pipeline is fed with data supplied by their merchant partners. This data is preprocessed on some EC2 instances and then stored on a collection of Elastic Block Store volumes.The first MapReduce step processes this data into a common format and stores it in HDFS form for further processing. Additional processing steps transform the data and product images into final form for presentation to online shoppers. You can learn more about this work in Jinesh Varia’s Hadoop Summit Presentation.

Online dating site eHarmony is also making good use of Elastic MapReduce, processing tens of gigabytes of data representing hundreds of millions of users, each with several hundred attributes to be matched. According to an article on SearchCloudComputing.com, they are doing this work for $1,200 per month, a considerable savings from the $5,000 per month that they estimated it would cost them to do it internally.

We’ve added some articles to our Resource Center to help you to use Elastic MapReduce in your own applications. Here’s what we have so far:

 

 

You should also check out AWS Evangelist Jinesh Varia in this video from the Hadoop Summit:

— Jeff;

PS – If you have a lot of data that you would like to process on Elastic MapReduce, don’t forget to check out the new AWS Import/Export service. You can send your physical media to us and we’ll take care of loading it into Amazon S3 for you.