This talk will focus on lessons learned by the Apache Bigtop community while trying to build a consistent, reliable and easy to use software distribution out of loosely coupled ASF projects. We will discuss patterns and anti-patterns of development, integration and system testing of complex software stacks.
While the core mission of Apache Bigtop is to build a Bigdata management distribution based on the Apache Hadoop, its uniqueness lies in the fact that it is the only ASF project that aims at integrating other ASF projects into a consistent, reliable and easy to use software distribution. Apache projects are typically very disciplined in dealing with internal consistency and correctness, but when it comes to the integration efforts (either with the OS or between the loosely coupled projects) those were largely shouldered by the ISVs outside of the ASF (Linux distribution, commercial vendors, etc.). There is, however, no reason why this aspect of software engineering shouldn't be tackled within the foundation itself. Ever since its inception in 2011, Apache Bigtop has made tremendous strides in making a 100% Apache Bigdata management distribution a reality and have also learned a number of valuable lessons in helping various loosely coupled Apache projects move towards a common set of goals.