Software Development today, Cool APIs, Open Source, Scala, Ruby, Chef, Puppet, IntelliJ, Storm, Kafka

What is software development like today?

A complaint I heard from a CIO recently was that when talking to the high flying technologists at his firm he said:  “they talked in strange names, Kafka this, docker that.  Names of open source projects, APIs and languages that make him wonder if these people can program or just piece together technology and names? It’s very easy to toss around a lot of names and convince somebody you know what you are talking about, but the real question: If you assume that you can do things like this; Is this the way we should be building software these days? Talk to developers today and a lot are doing something different than what we did just a few years ago.   Today developers may use languages such as Ruby or Scala, PHP or Clojure or combinations of these things in a single project, they may use google APIs and back end as a service APIs, Salesforce APIs and APIs for doing pattern discovery, blob storage.   They may use Storm or Kafka, App Factory, CEP, Stratos, docker, node.js or cassandra or an open source Identity Management.  They may develop on IntelliJ or CodeEnvy in the cloud.   They may build their technology with dependency injection, aspect oriented techniques, test using Chaos Monkey, appium and numerous other web services that in general allow them to have very high productivity.    It is very likely they will complain bitterly if you don’t have Git, Maven or Jenkins, Chef or Puppet and numerous other open source tools for them to use that they are familiar with.   It’s a new world that has evolved very rapidly.   This cacophony of names is scary for some. I love this diagram that Kinvey came up with some time ago that illustrates some of the complexity of just the APIs for backend as a service in the new paradigm: baas-ecosystem

What are advantages of these technologies?

The first thing I want to do is lay the groundwork that many of these tools are very productive and are certainly compelling on the face of them.  It would be impossible to give a complete list of the various tools, so the purpose of this blog is simply to give you a quick summary of some of the things and variety of tools you can find.   This is the new way to do software development and I think the world has to adapt. For 40 years we have been striving to get the holy grail of “reuse.”   Why should people in every enterprise build the same things over and over and over again?  It’s a huge waste of time and resources.  We have finally found a way to get the reuse we dreamed of.  Now we are facing the problems of that reality but let’s not go backwards.  Let’s figure out how to make this work.

clojure kafka programming ruby scalaLogo

1) New Programming Languages and Frameworks

Ruby on Rails interpreted functional language that makes building UI  and database access easy more intuitive, like SQL is built into the language.   Companies such as Twitter used Ruby for years to build most of their technology.  Heroku gained prominence because it allowed people to build and deploy Ruby on Rails apps in the cloud.

Groovy on Grails – Is considered the successor to Ruby and is faster, more tightly integrated with Java, supports modern powerful regexp operations and polymorphic iterations.  

Python and Django : interpreted procedural language with dynamic typing.  Programs are shorter, maybe 1/3 the size of Java.  Django is a MVC type web framework.   Python is used massively by web developers in small companies and consultants who want an efficient general purpose language that can be used to build stuff and test it very fast.

Scala and Lift – compiled functional concurrent language.  Can be many faster than java because of faster than thread concurrency.   With Lift added Scala has very powerful web development features that are a leap ahead of Ruby on rails. (my opinion)  Concise, the language is compiled into Java bytecodes so it can perform extremely fast and can also call any java function and interact with java seamlessly.  Functional aspect means safer language with high scalability and easy concurrency that allows you to build communication code that scales and performs well in all situations.

Jaggery is an open source language based on javascript that gives you dense easy to understand scripting framework for server side development.

Clojure – a powerful dynamic language with lisp like syntax for those who remember.  It has concurrency features not unlike Scala and is used in a number of open source projects. One advantage of all these languages and other older options like PHP is that they all share a Java JVM as the basic underlying execution environment meaning that in many cases interoperability is much easier than in past multi-language environments.  It means many of the support and deployment issues are the same as with Java in general.   This powerful feature of Java means that these languages can be used in an enterprise and may not be as difficult as earlier languages were to switch to or from or to use in one group or another without becoming a large burden.


2) Open Source Projects

Open source projects can provide an enormous step function in agility not only because you can take a project and incorporate it into your software but because of the ability to collaborate and continue to develop the technology, influence the roadmap of the software you can have other people test, develop new functionality for you that you can use.   Companies such as Yahoo, Google, Twitter started working on big data 10 years ago and the fruits of that have been enormous.  IBM claims 20 billion dollars in revenues in 2015 from BigData.   Here is a barely a start of a start of some of the open source projects that may not be tip of the tongue.   Trying to list the categories of open source software would be tedious, let alone all the projects. I am not going to list Operating Systems, IaaS platforms, Bigdata projects, Eclipse, other development tools only projects, all the middleware open source projects, mobile projects.  There are thousands.  So, here is a smattering. Storm – framework for building hadoop like distributed queries across a large number of nodes node.js – distributed development platform CourseFork – A way to create “pull” requests for courseware Docker – create containers for executable content to be distributed in a PaaS  API Management – API management software to start you on your way to an API future FFMPEG – an old favorite for doing almost anything with video Apache Zookeeper – help manage large bigdata clusters  Apache Stratos –  PaaS to do DevOps Apache Cordova/PhoneGap  – tools to help you build mobile applications Collabinate – open social API to help you build social applications

3) APIs in the cloud

This is one of the fastest growing of all the areas I’ve discussed above.  Companies are offering services at a heady rate for many reasons.   You can find an API to do just about anything you want.  There are APIs for all the basic functions: Databases, File Systems, Logging data, Big Data, Analytics, Identity Management

There are APIs for things in the real world:

Geolocation, Names of places, Address conversions, time conversion, language conversion, making phone calls, receiving phone calls, videos, storing videos, streaming videos, discovery of services, information on almost any  product, tagging photographs, storing photos and videos, prices of homes, estimates of home prices, mapping anything, getting directions, finding cabs, finding parking, finding the closest things, find out my physical activity, find out where my car is and what the battery charge is, honk the horn on my car

There are APIs for people related things:

tasks, calendars, social network relationships, financial information about people, credit information, interests of people, people you know, your position and job history, pictures of you, the popularity or influence of people, the projected impact of a person, what’s popular, what’s trending, what stories about any topic

There are APIs for esoteric functions:

find patterns in arbitrary data, recognize voices of people, recognize people in pictures, setup triggers for complex sequences of events in the real world using any other service in the cloud, for instance, if I post something to facebook with your name in it then copy it down and send it to me, if my house price changes value let me know. Calculating almost anything mathematical, performing statistical analysis, visualizing statistical analysis, various visualization tools, augmented reality, gaming, doing polls, earthquake detection (probably prediction too although I didn’t find anything), finding people affected by major disasters

APIs for financial information in general

stock prices, bond prices, other asset prices, historical, similar for mutual funds or other assets, transfer money, loans on your paycheck, risk calculations for credit, securities, people, sending money to people or banks

APIs for helping to build mobile applications

APIs around ad pricing, ad placement

4) New Development Tools and Methodologies

New development methodologies like DevOps / PaaS means deploying all this technology you are reusing easier, faster, cheaper. New testing approaches and architectures that build resilience instead of reliability emphasize accepting failure and building around the assumption that things will fail sometimes.  Have circuit breakers so that when something fails you cut and run fast, don’t wait around interminably because that can very quickly make any complex interdependent system come to a grinding halt.    Have agents in your test environment or even in production that wreak havoc on your services and applications to see how they will react to heavy load or difficult situations you can’t anticipate.  These things are sometimes called Chaos Monkeys and they were invented/promoted by Netflix which has had a huge impact on the way people think about building scalable systems. The use of bigdata heavily throughout a system so that you can accommodate learning into your application.   I call it “looking smart.”  If you record lots of stuff that is happening in your APIs, applications then as you discover things that is happening you can systematically learn how to respond intelligently to patterns.  This makes your system easy to improve and adapt and to look like it is learning or that you are improving things rapidly.   Bigdata is essential in todays new technology mix.

API Centric Orientation

Software development methodology is changing because of the need for agility and reducing cost. Using open source means NO LENGTHY AND EXPENSIVE software license negotiation.  Leverage smart people doing something you need who are on the cutting edge of software and are upgrading their software daily.  Leverage other people and companies who are bug testing and coming up with innovative ideas for the software for free for you. Building or using APIs means thinking about the user of a service as opposed to thinking about the function of the service.   What APIs did was change the paradigm for developing interfaces from one that was focused primarily on the technology behind the service to one focused on the user of the service.  This sea change is reflected in API Stores that give a social face to an API that encourages consumers of a service to comment, complain, offer advice, fixes, improvements which encourages more usage and further improvement.   The mantra in the API world is simplicity.  Make the API simple to use and people will use it.   Instead of “This is my service, use it or don’t.  I don’t case.”  We’ve gone to a “supply side” approach where because there is a plethora of APIs to gain adoption you need to make your API easier to use than anyone else.  Whether you are building an API for yourself, plan to market it outside your company or are utilizing an API from an external source this results in increased agility, faster time to market and a stream of faster improvements that reflect quickly on your end products features as well.   Today, many applications are simply composed of calls to APIs and is called API Centric programming.   To facilitate API Centric programming you need to have a social API store.


What does it mean that developers are using these tools?  We have been promoting reuse for years.   We finally have achieved a new level of reuse.  Do we think it shouldn’t be embraced?   I mentioned above that Twitter used Ruby on Rails for a lot of development in the early days.  They found that Ruby didn’t scale well.  It is interpreted language and had other problems that ultimately forced Twitter to move away from Ruby.  However, would Twitter have succeeded in the first place if they had chosen a less agile language to begin with?  I don’t know the answer to that question but I know that Twitter today is successful and they made the transition largely intact.

Does all this new era of programming mean the old era of the programmer finding neat algorithms to do one thing or another has died?  I don’t think so.  The current paradigms of open source and API services are very agile and promote rapid evolution.   One advantage of these new paradigms of reuse is that improvements can be leveraged by everyone faster.  There is still room for lots of creativity in programming but the effort can now be leveraged by everyone.

There is still the thorny question of what to use and what to avoid, i.e. what to tie one’s horse to.   Should you pick one bigdata database over another?  Which BaaS (Back end as a service) provider will win?   There was always risk in software development that you would pick a bad post to latch onto.   However, the social environment of open source projects and API services enables you to talk to other users directly and determine what the reusable pieces are good at, what the size of community using the projects and what is the future of those projects.   I believe this is generally a good trend.

Other Articles to read

The top 10 new open source projects of the year
API Directory
Want a composable business? You need Agile and DevOps
BigData All the time NSA for you
API centric – The evolution of reuse and API Management
Open Source is just getting started, you have to adopt or die

4 thoughts on “Software Development today, Cool APIs, Open Source, Scala, Ruby, Chef, Puppet, IntelliJ, Storm, Kafka

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s