Theo Beack's Virtual Whiteboard

SOA and the "Contextualization" of Reuse

Oh, what a tangled web we weave.

Earlier today I read a post by Joe McKendrick who was blogging about two blog posts he read (respectively by David Chappell and Harry Pierson) about SOA & Reuse. Reuse is a topic that is receiving a lot of attention from architects, developers, IT Executives, software vendors, etc. I have done quite a bit of SOA related architectural and development work during the last 3 years and in this time I worked with a large number of companies spanning the globe. I also published an article on this topic earlier this year.

After reading the 3 blog posts mentioned above, I decided to jump in and add my voice to the debate.

Harry mentioned that he thinks, "David does a great job blowing the SOA Reuse argument out of the water". I disagree; I don't think David blew the reuse argument away, but rather I think he did a very good job of highlighting some of the barriers that are preventing organizations from gaining desired levels of reuse from their SOA implementations. They key is to challenge people to think about reuse and what it really means within the context of SOA and how one can achieve it. Many assume that by going towards an SOA, reuse will automatically fall into place. This is obviously not the case.

In my opinion the prime reasons for this are organizational issues. Other factors that also play a role include:

• interoperability problems,
• poor internal communication,
  
• and lack of organizational standards.

David quite rightly mentioned that some organizations are able to achieve significant levels of reuse. I have worked with a number of companies who were able to achieve it. What most of them had in common was the ability to:

• foster a culture of reuse,
• provide incentives for developers and teams to create reusable services,
• implement mechanisms to measure how each services is being used, by whom and how often (this also resulted in their ability to measure the quality of each service),
• implement a governance structure, tools, processes, collaboration workspaces and blogs to facilitate improved communication and thus share knowledge and best practices that helps improve the reusability of the services being created, and
• provide management support to drive the cultural change.

What is important to understand is that each of these companies understood the cultural impact that SOA would have on their organizations and was prepared to drive this change throughout the IT organization so that it became a way of thinking. Again I agree with David, not all organizations will have the same discipline and dedication to these principles and thus they will not achieve their stated goals. To summarize this line of thinking I would say the 3 most important aspects that will impact the levels of reuse within your organization are:

• governance,
• cooperation, and
• incentive.

Harry also posed in interesting question:

"So here's the question: given that general object reuse has seen some success, what's so different about business objects that causes reuse to fail utterly? Since we're really interested in service reuse, knowing why some object reuse succeeds and other reuse fails will help us understand which services are likely to be reusable and which wont."

It is easy to fall into the trap of comparing objects and services. One should keep in mind that Object-orientation is a programming paradigm, whereas Service-oriented Architecture is an architectural approach. The scope of SOA extends beyond just the programming paradigm and thus it is difficult to equate objects with services. Objects generally are only reusable within the programming language of choice, whereas services can be called by consumers regardless of platform, programming paradigm or technology in question. The scope of potential reuse with SOA is significantly larger than in the OO paradigm. Through the use of standards such as SOAP or architectural styles such as REST one is able to connect applications that previously were unable to integrate with each other.

Almost all Fortune 500 companies have significant investments in legacy applications. Many still run their core business on applications that are written in COBOL, PL1, Natural, CICS, etc. and these applications have historically been very inaccessible to other platforms and applications, especially if they are hosted on platforms such as z/OS, OS/400, VSE and many others. The concept of objects simply does not exist on these platforms since most of these business applications were (and still are) written in 3GL or 4GL languages. This explains why companies view SOA as the approach with the most potential of achieving reuse.  Not one single organization I have dealt with is either a pure J2EE or .NET shop. This may occur within the SMB market, but it is a very rare occurrence within the Enterprise market (if it happens at all).

Another aspect that caught my attention from the blog post was the following statement:

"A business object on the other hand has significant contextual requirements, which makes reuse difficult or impossible."

I agree this might be the case when dealing with objects, but it doesn't have to be the case when dealing with services. Although the notion of "context coupling" may occur in SOA, it should not be the norm. Services should be designed in such a way that they are reusable, since the consumer doesn't have to know or even care about the service provider's context. The reason why "context coupling" may occur when dealing with objects, is that the business process may be encapsulated within the object. Many companies have fallen into this trap, not only with OO, but also with the older style legacy applications and it is the encapsulation of the business process into the application code that gave rise to multiple and redundant business applications. So the "context" should be elevated to the process layer and NOT encapsulated within the service.

SOA enables companies to remove context through the services that they expose, which gives rise to the reuse of services (such as Purchase_Order_Service) within the business process layer.  It is now possible to automate your processes through BPM and many organizations are using this as a method to reduce redundancies and remove duplicate systems through standardizing on a common set of applications. This in turns drive reuse and huge cost savings.

Last year I worked with a major US Bank who exposed one extremely vital calculation routine as a service, which they are now able to reuse throughout the entire bank. This occurred as a result of regulatory requirements they had to comply with. It is possible that the drive for "standardization" may be influenced by external forces that have nothing to do with technology and many companies are finding that SOA and reuse is the way to comply with or accommodate these external forces, as in the example I just used.

It is the combination of loose coupling, standardization, interoperability and the wider adoption of SOA related best practices that is making reuse a real possibility. However it is still up to the individual programmers, project teams and IT organizations as a whole that will be responsible for the ultimate success of their SOA implementations. I think we have a real shot at making this work; the question is do we have the discipline to do it right?

Jonathan Schwartz and open letter, the SEC and blogging

I just read a blog post by Jonathan Schwartz about using corporate blogs, RSS and the Internet to disclose information that is regulated via “Reg FD”. Included in his blog post is an open letter he wrote to the SEC chairman, Christopher Cox.

I think this could potentially be a significant step, should the SEC take Jonathan’s letter seriously and act upon it. This is one of the early signs of the potential impact of “Web 2.0” on the business community. There are  just so many  potential way in which business can benefit from using simple (yet powerful) mechanisms such as blogs and  syndication to communicate more effectively with their customers, investors and employees.  <tongue_in_cheek> The notion of monthly new letters and limited conference calls is very old school<tongue_in_cheek>

Companies can be using VOIP tools (such as Skype) to host much larger and cheaper conference calls (if they have to) and people can attend as they wish without the need to register before hand. These calls easily be recorded and blogged and/or syndicated as an audiocast (podcast). The notion however is that information can be published and made available publicly to a very large audience in a very efficient manner.

Having said all of that I also recognize it is not a technology issue, but rather a behavioral issue. Company executives, analysts and the press are very used to the current way of doing things and the status quo works well. Therefore their will be a reluctance to try something new or the change the existing way of doing things. Especially if the parties involved are more or less technophobes. I suspect it will be the more adventurous companies and those with a pioneer spirit that will attempt something and thus blaze the way and set the standard for how things will be done in the future.

I’ll be watching this thread with a lot of interest.

The "Web .02" ala Net Neutrality Act

I just saw the following video clips on YouTube, which I thought was quite funny. Both the clips are snippets from Jon Stewart's "The Daily Show". In the first clip he "reports" on comments made by Sen. Ted Stevens about the Internet and the need for the Net Neutrality Act.

The second clip is a follow-up of the first "report" and this clip features John Hodgman who helps to clarify the inner working of the Internet.

I thought both were really funny. Enjoy!

Future Impact of Web 2.0

I have following the Web 2.0 trends with a lot of interest. As I have mentioned before I think it will have a lasting impact on our lives. Even though I have been traveling all over the world the last week, I have tried to keep up to date with the latest developments in the Web 2.0 space. For the last two days I was totally immersed in catching up on what has transpired and it is interesting to watch the trends of how much focus is placed on the technology aspect of Web 2.0.

Most blogs, web sites and articles seem to zoom in on the coolest of cool companies, how much funding they have received, what their market valuations are and the features that have been added. I don't see to many people blogging about the impact Web 2.0 will have on our lives. Many bloggers express very vocal opinions regarding the hype that is Web 2.0 and how it is nothing more than the latest marketing buzzword.

I have to agree that there is a lot of hype and a tremendous amount of unrealistic expectations, but at the same time I think we should careful not to miss the obvious. I think that the most fundamental trend and implication of Web 2.0 is how it is in the process of impacting our lives and also how it will continue to impact our lives in the years to come.

We are undergoing a fundamental shift in how people use the Internet and as a result we are undergoing a major change in how we communicate, share experiences and knowledge, connect with one another, how we influence each other and also how we continue to use the web as a basic (yet sophisticated) communication platform. In another 5 or 6 or 10 years we will look back at the 2006 and recognized how far we have come. We won't recognize the technologies at play, we won't even think about them. We will however recognize the impact it has had not only on us as individuals, but also on society as a whole.

I did a presentation last week in Bonn at a SOA Conference hosted by Deutsche Post. In my presentation I used the following image to try and visualize the social impact of Web 2.0:

This image should not be viewed as an accurate representation of the relative community sizes. However, what is it trying to express is how our on-line behavior patterns are changing. We are becoming a lot more active, plus we are using the Internet for more and more of our daily activities.

This is the aspect of Web 2.0 that doesn't get enough attention. the focus now is on the cool technologies and business models (which is important  and I do not discount the role that they play in order to facility the growth and maturity of Web 2.0. I just think we should pay more attention to the broader social impact and implications. Quite rightly so it will have a tremendous impact on many business models. I can see how many companies will be severely challenged by much smaller challengers who are able to marshal the underlying forces that is making Web 2.0 such an important (and disruptive force). In this article Curt Finch starts to address the transformation. In the closing section he focuses on new business models and although I agree with his assessment, I think that even that perspective is too narrow. It still too focused on the content, on delivery; it still too focused on the technology savvy and doesn't take into account the impact it will have on average person. The web really became  an important business strategy once non-geeks started to feel comfortable using it, once we started using it to keep in contact with family and friends via email. It really started to make a positive impact on companies bottom lines when we started doing our Christmas shopping via the Internet. Now we use it to search for products, compare features, get products reviews and shop for best prices. We are now able to share our experiences and give feedback on what we think of the products we purchased, etc.

Now we are entering the next phase in which the Web is evolving as a platform, thus the term Web 2.0. As we experiment and use cool sites such as flickr, YouTube, Yelp, Wallop, etc. we should also keep in mind the how these sites are changing our behavior patterns . . . . or maybe they are just reinforcing our natural behavior patterns. Humans are very social creatures after all and we connected we share ideas, information, knowledge, experience and our vacation photos. :-) Web 2.0 enables us to do the same things but on a much larger scale, to potentially a much larger audience. The point being that we are doing so more consistently in an electronic fashion.  The momentum is starting to build, at first it happens very slowly (as it has been for the the last 2-3 years) and now the pace of adoption and change is accelerating. It will be exciting to look back in a couple years time to recognize how this trend has impacted our lives.

Mac Virtualization Feedback

I have swapped my Dell laptop for a MacBook Pro in the last couple of weeks. I'm really enjoying the MacBook Pro. It is great to finally just be able to shut the lid close and know that it will go into sleep mode and when I open the lid it will wake up and be ready. No more waiting for the laptop to wake up and start the OS.

A couple of weeks ago I blogged about Virtualization on the Mac. I have installed both Parallels Desktop and Desktop Manager on my Mac and these tools work great. Yes, I have to run Windows for some work related software and it is great to have the ability to boot another OS within the Mac OSX. If you have similar requirement you should consider this excellent piece of software.

Social Change, Data Aggregation and the role of SOA & Web 2.0

I’m in Germany this week and am writing this from the airport in Munich. And before you ask, no I’m not here for the world cup, although I saw the tail end of the Brazil/Ghana match last night. I wish I could say that I had the foresight to arrange business travel to Germany right around the world cup, but I cannot. The business schedule has been pretty hectic, so I had to resort to catching glimpses of matches between meetings and dinners. Anyway, this is not really what I wanted to blog about today.

Earlier this morning I watched a fascinating and quite entertaining presentation by Hans Rosling on “TEDTalks”.

I quite enjoyed the methods he used to visualize and present the statistical data. It made it quite intuitive to understand the data and impact of social change over time. I have always found this kind of representation of data difficult and tedious to do. So I really enjoyed the techniques used to animate the data.

What I found most intriguing about his talk, apart from the chimpanzee comments - ;-), is how he came to his conclusions. He had to research and process a large volume of raw data in order to build the neat models he presented.  At the end of his presentation he talks about the need to have free access to a broad base of data and how difficult it is to gain access to the data stored in so many different data sources all over the world.

This is a problem we (in IT) are very familiar with. Business users, end-user and consumers of computer systems want one thing beyond anything else . . . . access to data. This is the reason why we have created data warehouses and data marts. This is why we setup tedious ETL routines to extract huge amounts of data out of operational data sources into these “reporting databases”. With EAI we started to link systems together in real-time, in part to give users more access to data and to enable to “flow” of data between systems.

You may wonder how this is relevant in the context of the talk given by Hans Rosling. I think it is highly relevant, because the basic problem he and other researchers are faced with, is how to gain access to the “right” data. This is also the exact same problem the users of IT systems struggle with. 

There are a number of important questions locked up in this one statement. First of all how do you define and identify “the right data”. Secondly, how does one extract and aggregate data that comes from different data sources and present it into a usable format.

Until now it was mainly the developers, database administrators, architects and system analysts that decided how to design, implement and expose the computer solutions (and resulting data) to their users and consumers. However, what the consumers of these systems (and data) really want is access in their terms. They want to dictate what the data should look like, where it should come from and how it should be delivered and analyzed. They want control so that they can do their tasks more efficiently.

In my opinion this is where SOA is laying the foundation for the next phase in the evolution of the systems/applications that we use and build. Through the application of the principles of SOA we are starting to break down some of the barriers that restricted the free flow of information. We are now in the process of creating a new informational fabric with the intent of making not only data, but also real information available, on demand, to those that want it. Granted we are only in the early stages, but I already see the usage patterns emerging.

For one, when people such as Hans Rosling were to aggregate, massage, analyze and interpret data (as he did for his talk) they could potentially make it available to others. Rather than simply presenting it in a passive format, the information could be presented as a service (or set of services). So if I were interested in analyzing the social impact of the adoption of broadband in South Africa and compare it to other regions of the world, I could simply add the service provided by Hans and then merge it with my own set of data. I would not have to crawl through several different data sources and scrub them for analysis. Someone like Hans may have done that already, thus enabling me to focus on the information and trends I’m really interested in.

This is not only applicable to researchers and academia; this is extremely relevant to the business world. There is tremendous potential for people to share information with colleagues, who can actively collaborate to identify and analyze patterns and processes with the aim of improving business models, identifying areas of potential optimization, etc.

This is where the combination of SOA and Web 2.0 (or more specifically the content of “smashups”) really come into play. When I look at the way publicly available and published APIs are being used, it is clear that people are extremely creative and in ways the creators of the APIs never imagined. So much so that the original data (and application thereof) is being used in ways never anticipated. Google maps is a famous example, but I think Amazon is another wonderful example. Now Skype has published their API and so did Yahoo (for their IM tool). Who knows how people will be integrating the basic VOIP and IM capabilities into new kinds of applications? The point I’m trying to stress is that once this kind of mentatility takes hold within the enterprise, we should see a similar explosion of different usage patterns. It is the collaboration aspect that will enable people to solve complex problems more efficiently and to share their work with peers. For that to happen, we first need to reach a critical “services” mass. At this point most organizations are building the basic framework for creating, aggregating, managing, securing and publishing services. Once that framework is in place, the focus will shift to creating a network (or cloud) of services. This cloud of services should enable people to do their magic.

It is amazing to observe the explosion of mashed apps within the last 6 months. The sheer number of apps and solutions that are being created on a weekly basis is phenomenal. Not all of them are good or even relevant. Not all of them will survive for any length of time, but what is really important is the fact that people are able to create them in the first place and that people are able to experiment with ideas, content and information. The ideas that survive will strengthen the next generation of solutions and lead to a new generation of innovation. This is a really exciting time and I cannot wait to see how the web and technology will evolve over the next few years. I hope that we at BEA will play an important role in the innovation process and as a member of the Office of the CTO I can see the potential for how we incorporate some of these ideas into out technology and product roadmap. I hope to share some of the ideas with you in the months to come.

[Written in Munich and posted from Frankfurt, Germany via a very slow Internet connection]