Open Source is just getting started, you have to adopt or die

.

Some might think open source has just been getting lucky.   A few suddenly successful products like Hadoop, RedHat, OpenStack and everybody is going crazy.   Come on people it’s just a lucky break for open source.   Soon enough everyone will realize that this stuff doesn’t scale, can’t be supported, is a JOKE!   The love affair will wear off.

NOT!!!!

This is a very sophisticated topic because we are talking about what many might think is the very lifeblood of capitalism, of America.   After all, a lot of Americans make a living producing intellectual property and the cloud and open source, digital copying it all seems to spell disaster for IP.   Also, many companies today obviously live off the idea they have an advantage because they have intellectual property that is protected from nefarious evil people that intend to copy and steal it.   Open source seems a total affront to this viewpoint by saying this stuff is free and that’s okay.   Why would anyone do that?   Can it possibly last?

Sometimes people are thinking about things completely the wrong way.     I am NOT going to argue that everything should be open, that there is no place for secrets or protected technology or art or anything.   I am not going to debate at this point the ultimate moral question of open source or what about sharing music, movies, lectures or anything that can be digitized.   I am simply going to make what I believe are indisputable points about how this whole thing seems to be working and why it is going to gain speed not slow down or stop.   Later maybe for a philosophy blog we can post ramblings about the society that will result or whatever concerns people have with open source.

Today open source is a movement that is 10-20 years old depending on when you look at the real taking off point.  It’s relatively recent.   The fact is that even though there are many thousands of open source projects probably a few hundred are successful by any measure but the success of these projects is inescapable and devastating to the traditional highly IP protecting corporations way of business.

The standard model of how big companies get undermined by disruptive forces is well established

Everybody should know this, but it is a fact we have seen repeated time and again.   Small upstarts can undermine the large players by going in at a low cost point for the lower “value” business that is unprofitable to the larger players.   The large companies look good for abandoning low margin products, i.e. their profits actually climb as a result of jettisoning low profit parts of their business and abandoning it to the small players while the small players benefit from success at the bottom.

In time the smaller players work their way up the food chain and at each point the big company looks smart for abandoning the lower profitability market.  At every step the smaller companies look smart to keep growing and gaining.  At some point the larger players are put into competition for the largest customers and then their stock plummets as they don’t have anywhere to go.   Open source has done this already in many markets and we are seeing just this quarter in my opinion the real start of the decline of the proprietary vendors models.     We are seeing Oracle and other companies really starting to scramble.  TIBCO recently went private and Software AG has had declines.   The Enterprise software model is under stress.

In January 2013, Gartner said that WSO2 had the complete application middleware stack equal to the other 7 large proprietary vendors.   It also said WSO2 was visionary in all 3 middleware categories whereas none of the proprietary vendors were classified as visionary in all 3 categories.  How can a company 100-1000 times smaller than these guys compete?  This is happening in the OS market with Redhat and the proprietary vendors.   It’s happening now with databases with the no-sql databases such as Hadoop, Cassandra, MongoDB, … becoming the biggest thing in DB history fast.   We are seeing it in the ALM space where Jenkins, Jira, Redmine, other open source ALM players are becoming industry standards and proprietary vendors are hanging on.

What happened?

During the period 2000-2009 Google, Netflix, Facebook, Twitter and Yahoo worked together on numerous open source projects cooperatively.  I talk about this in another blog.  These companies and others promulgated a more rapid evolution of software that I call the virtuous circle

Surveys recently done show that open source is considered the BEST quality software not the cheapest.   In many cases it is the only software you can use.

The Enterprise model is not always aligned with the customer needs anymore

I worked in Enterprise license model companies before.   I am quite familiar with this model.   There are 3 things I think everyone should realize that are also undermining the traditional model:

1) An Enterprise License model company (closed source) is interested  in moving you to the next licensed version.   They therefore hold up key features till the “next” version.   If you need features in this fast paced world their need to force you to the next version is at odds with your interest in agility.

2) An Enterprise License model company will be interested in selling you licensed versions of other things, some of which are available in open source.  They will not want to work with that open source software even if it is standard or the best because they will eventually want to force you to buy their enterprise licensed version.

3) They are not motivated to sell you SaaS versions of their products even if they have them.   Frequently they have SaaS products today but the salesman will be interested in selling you the licensed version or product rather than a SaaS version.

What are characteristics of open source projects?  

They have to be re-usable

The holy grail of re-usability that we as an industry have strived for over decades with various efforts has finally landed someplace where we can say it is working.  Open source projects have to be re-usable for them to succeed therefore by definition all successful open source projects are re-usable because people re-use them.  Tautology.   Done.   The ultimate in re-usability will be APIs and the story on this is just evolving.   That’s another blog.

It is remarkable in reliability, scalability, security, mission criticality, performance.   

Companies such as Facebook, Google, Twitter, Ebay, Netflix, … and I could go on and on have built a significant part of their technology on open source components.   These companies process billions of transactions per day, sometimes approaching a trillion.  They have billions of users.   They collect data on a scale unimaginable just a few years ago.   They operate with performance any enterprise would be happy to deliver to customers.  They have reliability that rivals any enterprises needs.  They deal with security threats as good as any company relying on proprietary software.   The fact is that the argument that proprietary vendors are the only ones who have the most mission critical, most high performance, most reliable software is simply laid bare by the obvious facts of the software we all use everyday in the cloud from these companies.

It is simply indefensible argument that IP protected software is more industrial, scalable or robust than open source software in general.

Open Source is evolving faster than closed source 

The companies I listed above and others are evolving at a breathtaking rate, both in terms of scaling by adding millions of customers a week or by adding new features and improving service daily.  These are some of the most profitable companies in the world.   Some of these companies update their software DAILY, put it into production and have users using it.   This is simply unheard of in the traditional enterprise.   The software is improved at the open source level daily and many of these companies and others contribute their improvements back to the projects to feed the engine of innovation.   New projects are born every day by the hundreds if not thousands and some of these by major companies putting back to the well what they have used or created.

What does all this mean?

Open Source cannot be inferior to proprietary software in any generic way.

This is a general statement but is borne out indisputably by the small amount of evidence I gave earlier.   Obviously open source cannot be less secure, less performant, reliable, scalable if the companies using it are able to succeed, grow and generally blow away a lot of traditional companies using proprietary software.   I would argue that these companies couldn’t use proprietary software companies solutions because they simply wouldn’t scale, have security, reliability, etc as the open source variants of these technologies have.   So, the argument open source is inferior to proprietary solutions is impossible to support in many of the cases.

The pace of innovation in open source is staggering.

As I said, the number of new projects created daily is estimated at hundreds if not thousands.   The projects that exist are expanding at a fast clip….   The successful projects are spawning new variants and innovations and improvements at a dizzying pace.     Some may complain that it is chaos.   Some may argue: “Open source will fail due to the sheer volume of success.”    How can somethings success lead to its failure?  Well, that has happened but if that is going to happen it’s not happening anytime soon.   It is clear as rain that the projects doing open source and the companies behind them are gaining enormously from the success and this doesn’t seem likely to change anytime soon.    It is simply not clear what will be the undoing of open source or why.  What we know is what is happening and the best predictor of the future is always to follow the trend we are in.     Many of the projects in open source are rapidly crossing the chasm and will revolutionize the landscape of enterprise software of the future.   It’s an inescapable conclusion.

Open source is being driven by basic demand for Cloud, Mobile and Social, now IoT.

I will write a blog about the pace of change because this is an important topic to understand for enterprises.   The reasons for the pace of change to accelerate are:

1) Mobile, Social are the key drivers of new technology disruption.   The mobile revolution is synergistic with social and the internet.   The growing affluence of key markets like China and India are also responsible for giving these 2 disruptive forces more momentum but even if they didn’t exist the change from the improvements spearheaded by mobile and social are unstoppable even if those countries didn’t exist.

2) The Cloud has enabled a lower cost of doing business especially for smaller companies.

3) These 2 factors have combined to allow innovation to flourish by providing a market for the new technologies, a way to innovate cheaply, to start new companies based on this lower cost of business.

4) The new technology has not just been limited to supplying Mobile and Social companies but has also infected itself by spawning new technology to make new technology cheaper to build, i.e. improvements in cloud services, improvements in languages, devOps has forced companies to adopt open source products to do development.   This has changed the landscape of development and the development process significantly putting most companies light years behind the processes used in the cloud.    The new development processes allow developers and marketers to create new technologies in days or weeks that took months or years before and at significantly lower capital expense and risk.

5) This has allowed companies using those technologies to rapidly innovate allowing Facebook, for instance, to put out daily updates to its service, Netflix, Google and many other companies to adopt a daily, weekly or monthly upgrade schedule.    This rapid innovation of companies in these spaces has forced competitors and other new entrants to use the new technology and open source to stay up.

6) Companies that put out mobile applications were generally small and built simple applications using this new technology.   The innovation of the APP STORE by Apple demonstrated another major paradigm shift.   Previously applications were heavy and upgrading them on all the desktops was a major hassle that companies and individuals tried to avoid unless forced   The APP STORE made it easy for people to acquire apps, to maintain them, manage them.   Therefore, users started acquiring apps by the dozens and hundred.

7) The Mobile world grew to the size of the social and cloud worlds rapidly and leveraged all the technology in this space to put out new versions, supported by the App Store concept this was feasible and more scalable than anybody anticipated.   The success of this model is now percolating through the entire ecosystem driving adoption of entirely new ways of thinking about hardware, software, how people use applications in enterprises.   The Mobile world also revolutionized SOA.   SOA was the dominant way people tried to build software.   With Mobile the important thing was the API that the application talked to on the backend to do the work of the app and store the data.   The API became important because Apps became important.

8) APIs becoming important drives adoption of new technologies that promote re-use and easier building of apps for mobile devices but also makes automation between companies, their ecosystems and their customers easier and more efficient.

9) All these technologies feed one another.  Better social drives the value of mobile, drives the value of apps, drives the value of APIs, drives the value of faster development, cheaper development, cloud use climbs, cheaper deployment, more innovation keeps driving new adoption and more value added.  More adoption and more value added drives the market further by increasing the money available.    All of these trends depend on a rapidly evolving common software base now comprised of the open source ecosystem NOT the proprietary ecosystem.   This further drives those companies from the proprietary vendors.   Since these companies are evolving fast and the new disruptive forces of mobile, cloud, social are impossible for almost any company to avoid they are forced to accept the new models as well.   The closer you are to the mobile, cloud, social disruptive forces the more benefit and more you need to adopt open source ecosystem.   Some companies may think they are immune but it is hard to say how these disruptive forces will affect each industry.    You risk a lot by saying you don’t need to leverage the mobile, social and/or cloud forces in your business.

The pace of innovation forces others to adopt open source.

I cannot think of an example in history of technology when a fast pace of improvement or change didn’t result in more adoption and lower costs for consumers and better services.   The companies using open source have a huge advantage of riding a cost curve down at a rapid pace.   When the cloud emerged about the same time the process of doing software development for most new companies required an investment of about 20-40 million dollars in VC money.   Over the last 10 years new companies adopting open source and the cloud have been able to start at a fraction of those costs.    The cloud and open source software has enabled these companies to build technology cheaply, deploy it cheaply, grow the costs as they grow revenue thereby reducing greatly the need for capital and increasing their profitability earlier.    These companies continue to take advantage of open source and the cloud to continue to lower costs rapidly, to innovate faster, to grow faster as I described earlier.

Traditional companies have simply been left behind.   Most companies have not adopted in any serious way any of this new technology.   They are just starting to look at it but the evidence is clear that the costs of using the cloud, using open source and more important of continuing to innovate is critical for anyone wanting to compete in the new market for consumers and enterprises.  You simply cannot be a player unless you are using this stuff, learning how to operate with it, helping to make this work.   Regardless of your moral opposition or feelings about IP or whatever the facts on the ground are that your proprietary vendors are not the only game in town and you are taking a risk with them because it is not clear that they can deliver software that will allow you to grow your organizations and to reduce your costs to compete in the future like the open source players can.

Right now the typical company is starting to use SaaS services which are built on the cloud and using open source in many cases.   These companies may think that they can go along and use proprietary vendor solutions for everything else but it is unlikely that the revolution driving adoption of open source is going to stop.   I believe it’s very clear that would be a losing bet.   I’m NOT saying that everything open source is golden.  I am NOT saying that it’s always cheaper to use the cloud or that it’s always best to go open source.     I am saying simply that you absolutely have to look at open source seriously and use it in some ways in your plans to reduce the risk that you are going down a path that is going to kill you in terms of cost or lack of agility in the future.   You need to make open source part of your enterprise plans at the minimum simply because if your proprietary vendors cannot adapt or reduce costs you may be driven out of business at some future point.

I have seen very big companies starting to make pivots.   Accenture has been choosing best of open source and best of proprietary when picking vendors recently.  IBM is contributing and using open source.  Companies such as HP, Citrix have joined open source coalitions.   It may be an attempt somehow to embrace and crush but that seems futile.   This is too big for any vendor no matter how large or powerful to derail.

In my estimation you are taking a greater risk NOT using open source than using it.

I don’t know if this is compelling enough logic to you or I am speaking to the choir.  I have thought about it a lot and I believe that this pace of innovation is unstoppable and growing.   I believe that more and more projects in open source will become the leading technologies in their space.

What about support?   Am I simply replacing proprietary vendor “IP Licenses” with open source run time support costs?

I have worked at proprietary software companies my whole life.   Proprietary vendors charge annual support costs.  These costs are quite high ON TOP of the costs for the license to the products.   Open source projects that have a supporting partner or partners charge for their support.   However, these costs are on the whole dramatically lower than the proprietary vendors costs.   On average these costs are 20-50% of what proprietary vendors charge.   In some cases with some open source you don’t need to have support.   For non-mission critical functions with commodity open source projects such as tomcat for instance there may not be a good reason to pay for support.   Tomcat has been working for a long time and it’s not changing fast.   You’re probably not going to need any new features or have a problem with Tomcat that hasn’t been discovered by someone else already.     On the other hand there are other Application Servers that provide considerable advantages that you should pay for support if you use it in a mission critical capacity.

It’s always been safer to buy from “IBM” or some big proprietary vendor.  Nobody ever lost their job buying from “BBB.”

I am saying this old adage is now being turned on its head.   The most demanding companies in the world are using open source.   The argument could be made that choosing Oracle version xx may in fact cost you your job when your product succeeds in the market and you need to support a vast database of usage data that just doesn’t make any sense to store in a relational database.   This usage data is critical to understand your customers and to keep enhancing your services to compete.  If you are using a database that costs 10 times the alternate to license and support before you factor in data costs which also can be more you may find you have painted your company into a disastrous situation.   You may find by using a proprietary companies IaaS cloud sounded good when the pitch came in but sometime down the line the costs turn out to be much more than competing vendors and you are facing a massive rewrite to be able to run on a different IaaS vendor stack.   More importantly and more likely you will just find that using a proprietary vendor results in you not being able to follow the latest mobile technology easily, the latest identity technology or to use some crazy new service in the cloud that others are using and that leaves you falling behind every day to what your competitors are doing.   In other words the most likely way that using proprietary is going to hurt you is in the incredible pace of change that you will be left out of.

What are the technologies and trends that you can’t afford to skip?

It’s clear certain open source categories get a lot of press.  These are ones that seem to get the most press but this is just a smattering of the technology out there.  I admit these are my favorites.   I apologize in advance for the obvious misses that I have in this list.   Let me know your favorites and I will keep a list of these resources on my blog for you to use in the future.

Open Source Enterprise Software

Operating Systems:  Redhat gets most of the press, but there are numerous competitors Suse, Ubuntu, Centos as well as Android, …

Cloud Operating Systems, Virtual Systems:  Openstack, Eucalyptus, Docker, Kerbenetes, Mesos, Xen, …

Operations Configuration management:  Puppet, Chef , …

Middleware:  WSO2 Carbon, Fuse, Camel, Mule, JBoss, Talend, …

Cloud Middleware (next layer on top of Cloud Operating systems):  WSO2 Stratos, Stockato, Cloud Foundry, jClouds, Rightscale, node.js, …

API Management: WSO2 API Management, apiGrove

PaaS:  WSO2 App Factory, Heroku, Cloudbees, EngineYard, …. many others… I will write the second part of my PaaS blog soon and delineate these in more depth

iPaaS:  WSO2, Mule 

Databases:  mysql, cassandra, Hadoop, mongodb, couchdb and many more

Development languages, tools:  Java, Scala, Spring, Ruby, php, …

ALM Tools: Jenkins, Maven, Redmine, Jira, also an incredibly long list

Mobile Tools:  openMobster, openMEAP, Titanium, PhoneGap, also a long list

 Identity Management: WSO2, Atricore, openIam

content management:  Drupal, Joomla, and WordPress

Again, this is nowhere near comprehensive list but it shows the breadth of Open source solutions and the quality of many of these products rivals or is superior to that of any proprietary vendors offerings.   More important is the rate of evolution in these products dwarfs their proprietary vendor competitors meaning that in a few years the pressure on proprietary vendors selling enterprise software will increase dramatically.

What are the open source projects that are working?

I would not have joined WSO2 if I didn’t believe the software was ready for prime-time.   WSO2 is used by Ebay to process >5 billion transactions a day during peak periods.   They run hundreds of servers using WSO2 flawlessly and faster than other companies.  They went through a lot looking for the best software to use open source or not.    WSO2 has greater than 150 companies using its software in serious mission critical capacities.  WSO2 is 100% open source.

Many Companies are scared to go Open Source

I talk to many entrepreneurs who are concerned about making the software they are building open source.   Of course they are concerned that if they make their software open source they are giving away the shop.  They wonder if the model of income can work?

I don’t think this will work for every company.    It doesn’t make sense for all software of every company.   I haven’t really thought about what if everyone was open source that much.  What I do know is that many people overestimate the risk of going open source, overestimate the value of their proprietary code and underestimate the advantage of being open source.

Whatever technology you develop you should be aware how easy it is for other companies to reinvent whatever proprietary advantage you think you’ve developed.  It is especially easy in this day of high productivity to do this without breaking any patents or looking at your source code.     So, if you think you are making it harder for your competitors by keeping your source code from them you are wrong.

Most likely the main thing you are benefiting from by keeping your source code secret is the FUD competitors may create by being able to see exactly what you do and don’t do.

It is extremely unlikely that some other group of developers or companies will take your source code and decide to use it to build their own company on and compete with you against your own installed base.  I’ve never really seen it.    There is an enormous cost to keeping a significant code base up to date and you would be in direct competition with the company that wrote it and has the majority of installed base.  It’s not really a workable business plan unless the company is really failing in supporting its customers in a big way.

At WSO2 we’ve had customers suggest they were willing to support the software themselves in an effort to get us to reduce our support fees.  We say fine.  Go ahead.  Hire 50 engineers and try to support the products with patches, keep them up to date with new standards and other open source projects.   How long before your code deviates significantly from ours and you are out alone building your own code and spending your time being an integration company rather than a online store or insurance company or whatever your main business is.  How long before your boss recognizes you are an idiot for spending so much time and effort to save so little money?  We have 90% + renewal rates because it makes no sense.

If your company is making software that is mission critical then companies will pay support fees.  Most companies are required to do this and most companies the person who used open source and didn’t buy support would be fired the first time an outage occurred and the company found out they didn’t buy the support.  The fact is if some company doesn’t pay the support fees there is a good chance they couldn’t pay the support fees because they simply can’t.   So, you are not losing any revenue if they use the open source software.

There are clearly some open source projects where support fees aren’t as critical.  Open source Word processor for instance may not generate significant support fees.  It may make more sense to have multiple versions including a licensed version.  It may make sense to consider a pay per use cloud model instead of source code.

Conclusion

Open Source is NOT having a lucky streak.   This is happening.  I am the kind of person who embraces change.   Maybe I’m seeing open source through rose colored glasses but I don’t think so.   When I was at TIBCO I argued that we give the “TIB” away as free or virtually free for the use in the general internet.   I always felt that the value of what we delivered was in what we did with it and how fast we could evolve it, our reputation for delivering to our customers solutions they needed and that worked not by protecting ourselves with patents.   I believe many proprietary vendors will recognize that they may have to go open source.   Many are considering or are making portions of their stacks open source.    We are seeing Citrix, HP, IBM and many other companies embracing open source.   I think any organization that doesn’t embrace open source and use it where possible is setting itself up for a costly transition and difficult time in the market.