Changing role of IT in the BI world

BI is Dead, Long live BI

Timo Elliot (blog|twitter) recently published a blog BI is Dead which draws from Gartner’s Magic Quadrant report and a detailed report.

The main take away from the post (which includes references from the Gartner research ,so not all are Timo’s points) include:

  •  The self service analytics tools have matured enough that the involvement of IT in BI and analytics projects is deemed optional. IT don’t need to model the data upfront (which needed gathering all or partial analytics requirements to start with), analysts can prototype and test it themselves.
  •  The balance of power for BI and analytics platform buying decisions has gradually shifted from IT to business.

  • BI and analytics tools which do require intervention from IT are not considered BI anymore; they are Enterprise Reporting Based Platforms. Admittedly, they take most share of the BI market. In other words Gartner has updated the definition of BI.

  • The bold headline – “BI is Dead” – is to gain attention on the changing landscape of BI tools.

  •  Organizations who do not embrace the new definition of BI, run the danger of turning into BI-nosaurs.

  •  Having single view of data through an EDW is pointless/extremely hard. This and this from Curt Monash also support this line of thinking.

  •  Most organizations believe that IT has role to play in BI although majority want the responsibility of authoring the content to end users. This has always been the holy grail of BI.

So it is really dead?

Well-not necessarily. It’s dead in the sense that PCs are dead as we know them from 1980 or before. PCs have clearly evolved: they don’t look like how they used to look and many aspects of PC have been democratized. Trivial things which needed programmer in those days can now be done by users themselves. Heck, you can even upgrade the hardware if you are into that sort of thing.

I think that’s exactly what is happening in the BI and analytics world. The self service analytics tools have evolved to the point that many of the trivial data munging tasks can be done by the analysts themselves and what they can do with these new tools does not look at all like how they have been doing it. Does that mean the BI is dead? No- the fundamental analytics is still the same. It has evolved just like the PCs and the role of IT has changed.

Role of IT in the new BI world

The changing role of IT in BI and analytics can be best described from picture below.

Changing role of IT in the BI world

IT is now viewed as (and rightly so) as data facilitators. They make data available to the analysts in palatable form. The responsibility of modelling it,authoring presentation components and analysis lies with analysts.

Does that mean the job of IT got easier?

Most-Definitely-Not, on the contrary it has got even more complicated. With the influx of new tools, the demand for data has dramatically increased. The analysts want to analyse all sorts of data, from all sorts of unlikely sources and with all sorts of analytical methods. Their expectations from IT is to facilitate this process which is where ITs job has got a lot harder. We have to deal with ever increasing data sizes, from ever expanding sources, and support ever changing analytical tools.

What else is keeping BI alive?

If we assume the following definition of BI,then here are more reasons why BI is definitely not dead in its current incarnation.

an umbrella term to describe “concepts and methods to improve business decision making by using fact-based support systems”

1. Not everybody is data scientist

a. Combining data from multiple sources, b. creating data model and c.authoring reports from it, need special skillset. Many a times, analysts just want an Excel connecting to OLAP cube to do their analysis. Traditional BI has a place in this space.

2. Robust production ready solutions

The analysis done in the analysts R code or ipython notebook is not usually production ready. BI will be very much needed when it needs to be productionised.

3. If ETL is part of BI then its here to stay

How else would you provide quality data to analysts otherwise? And if the definition of BI includes methods to improve process of decision making, then we need BI.

4. Operational reporting is a fact of life

As much as we harp about ad-hoc analysis, data science and self-service BI, plain old operational reporting is a big part of any organization. For that we need classic BI.


Building data model for SSAS Tabular

I have started preparing for Microsoft Certifications on SQL Server 2012 and I am spending a fair amount of time with SSAS Tabular since it’s the newest addition to MS BI stack and the one which I am not very accoustomed to. As I work with it more and more, I am starting to think about how would you go around designing the database on which the tabular model will be based. In the multi-dimensional world, this was an easy answer. You would almost always design the DW in Star/Snow-flake schema or atleast expose the DW as Star/Snow-flake schema through SQL Server Views or Data Source View (DSV). In SSAS Tabular, it appears to me, that this not really necessary.

Do I really need to segregate data in facts and dimension?
I can create measure out of any column in any table in SSAS Tabular. It doesn’t have to be from a fact table so why separate out facts and dimensions? Assume that I have a ‘House’ dimension with ‘NumberOfRooms’ as attribute. In SSAS Multidimensional, if the users wanted to slice and dice ‘NumberOfRooms’ by other attributes of house dimension or any other dimension, you would have to some how surface this as a fact table. SSAS Tabular will happily allow you to create measure out of this column although it’s in a dimension table. In theory, you can point the tabular model to the normalized OLTP database and still be able to give users the ability to slice and dice the data just like SSAS Multidimensional.

Normalized Or Denormalized, that is the question..
There would be obvious issues with building the tabular model on a normalized schema.One, since the database is normalized, the sheer number of tables will be very large. Two, maintaining relationships between the tables would be painful.Three, the model itself can be confusing for end users. On the other hand, building a fully denormalized schema when you don’t really need it adds complexity to the project, especially in the ETL. I don’t think I need to reiterate that the complex part of DW project is ETL and around 60-70% effort are spent on this phase. So it appears like we need some middle ground.

Enter Entity Based design..
I have always perceived the DW as a collection of entities like product, sales, employee etc.  which have relationship with each other. The star schema is similar but the dimensions don’t have relationship with each other directly. They are related to each other through the fact table. Plus, the dimensions are denormalized so as much as possible information is crammed into them. I guess what I am trying to say is we don’t need a fully denormalized structure and at the same time we don’t want the datawarehouse in 3NF. We would need to find a middle ground so that it is just about corretly denormalized and normalized at the same time. In my opinion, the way to get to this is to think of DW in terms of entities and how they interact with each other rather than classifying the data into dimension and facts.

Of course, these are all my prilimanary thoughts and are very much open to issues. I guess we would have to wait for wide spread adoption of the tabular model and learn from people who did it.

Why business intelligence?

I watched a very interesting talk on TED yesterday by Pankaj Ghemawat called “Actually the world isn’t flat”. I would highly recommend it especially if you are working in the Business Intelligence field. 

The argument in the talk is simple. We percieve our world as hyperconnected and globalized. We use devices ‘Designed in California Assembled in China’. We wear cloths made in India or Bangladesh. Our fruits and vegetables come from Africa. We bank with a Hong Kong based bank and we work in a country we were not born in. It does appear that everything is connected and that there is mass exodus from not-so-good places to better of places AKA immigration. However the actual numbers tell a different story. To quote an example from the talk, majority of french people thought the around 24% of the population are immigrants while the actual number is 8%. The external forces makes us exagarate the situation. Mr Ghemawat has collected a vast data sets which contradicts many of the popular belifs against what actual data suggests. 

This can be correlated in busineses as well. Managers tend to make intutional decisions based on the experiance, however, the stats paint a different picture. The aim of business intelligence to fill this void. So that managers can make informed decisions based on facts rather than the gut feeling. Data always speaks the truth. I would also recommend book by Davenport called ‘Competing on Analytics’. The argument is same. Take a decision based on data, not on intuition. And this where business intelligence will come handy. 

Microsoft Surface 199$ gamble

So if the news on the web is true then the new tablet kid on the block AKA Microsoft Surface is going cheap. Engadget and CNet are reporting that it will be around $199. Surprised! I know I am. When it was launched my gut estimate was around $499 for lowest Windows RT model going up $1500 for professional level high end model. $199 look really puny and I look like a fool. So what might the reason behind the magical $199 mark (magical because Nexus,Kindle & Nook are at same price range). Most will agree that Microsoft will not make much money at $199 price tag. Google hardly makes anything from 7″ Nexus 7 and Surface is going to have 10.1″ LCD. So profits on hardware seem to be out of question. Which means there must be a long term motive and here’s what I think it is. 

Microsoft is new in the already crowded market. That includes both hardware and software. So to take a slice out of it, the viable option for them would be to sell their wares at cheap. The target would be the large not-so-niche market. This would increase the user base. If the user base increase, other manufacturers can follow the suite and launch their own versions of Windows tablets. That means more options for end users and more Windows tablets in the market. Microsoft can licence the Windows copy on each of them and earn revenue. Just like what it does currently. IMHO, that’s the ultimate goal. Besides you always have the app store revenues from selling apps, music, movies, games etc. If down the line Surface starts making profits, that’s an added bonus. Launching Surface at cheap is just to kick-off things.

Let’s see how this thing shapes up. 😀

Apple,Google or Microsoft. Pick up your partner.

Reading about all the smartphone news these days, I cannot help to come to conclusion that ultimately it’s coming down to three major players Apple, Microsoft and Google. I mean it’s not that hard to see, is it? And I am not talking about the hardware. No sir. I am talking about the more lucarative software side of things : the OS, third party applications, media, cloud etc. Why Microsoft? You might be thinking. Read along to find out why I think so.

Let’s begin with the big daddy of all, the Apple. It is indeed in a very safe position right now. What with 68% tablet market share ,cool profit margins on its hardwarehealthy iPhone sale and awesome app system. It’s going to be here for long time.Period. Get over with it. But Apple’s ecosystem is tightly integrated with its hardware. So unless Apple allows third party access to its ecosystem,there will always be room for other players. I mean not everybody is going to buy Apple products suddenly. Also, Apple products are expensive which gives chance to other players to get in the not-so-niche markets.  RIM is somewhat in the same situation. Software tightly coupled with hardware. Except that people are not buying into Blackberry promise.

I guess Google understood that long before I did (I suppose that’s why they are Google). Giving the hardware manufacturers a free OS that can compete with Apple’s iOS, painting Google over it and generating revenue from it not only by way of Ads but also from Apps & Media, is ingenious. That seemed to have paid off well. Andriod is doing well. It is second only to iOS  Infact it is doing better than iOS. Couple that with low cost and variety of Andriod phones and tablets and I think Google should be pretty happy with itself. I am no expert but I think Nokia missed a big opertunity with Symbian there. Can you imagine if Symbian OS would have been modified quickly and efficiently enough for touch based devices and rolled out to handset manufactureres. Most of them already used Symbian the switch could have been smoother I suppose. Nokia could have been where Google is today with Android. Anyway.

Coming to the point of why Microsoft then? First and foremost I personally admire what they have done with the Metro UI changes. Second, all the services that an mobile ecosystem needs are already there from Microsoft. They just need to take these and integrate it into a platform in a seamless way. Take email for example. We always had hotmail and now the new There is skydriveXbox Live for media & games and office online for documents. Third, number of PC user’s are much more than of any other users. Yes, I know, the recent Mac sales are much more than PC’s but average Joe still has a PC and is conversant with using it. So if all his devices start talking to each other over the air, seamlessly and he doesn’t have to pay a fortune for it, wouldn’t that be cool?Fourth, Microsoft has a large developer community. If the user base increases, the number of apps will surely increase. Fifth, enterprises all over the world use Windows and Microsoft solutions. It’s inevitable that they will start offering these on mobile platforms one way or another.

As you would have guessed I am really hopefull that Microsoft platform picks up. Surface looked awesome and turned out what I predicted it to be. Apple started it all, still leads it but comes at a premium price. Google took the idea and opened it to the mass market. Now I think its Microsoft’s turn to make use of it’s strong user base and create a very unique offering. I, for one, am with them.