Any business intelligence have four layers, first one is source data to the staging for loading updated data. Then data warehouse and analytics and displaying data on the front-end are other layers.
Source Data
Data Analytics platform Challenges
Data source could be SAP HANA, oracle Microsoft SQL server or CSV. Most of the time data format is different for each and every source. Underlying technology is also sometimes different. That impact BI development and execution? To bring the data from one source systems to DW/BI we need to have the ETL Layer in a staging area for integration and data cleaning.
Most of the time data warehouse schema is quite different than staging area. Because staging layer is interface for source to connect enterprise data sources. So schema is different between staging and DW. That need another data transfer from the staging area to data.
Data pipeline for data warehouse
So this is the second layer for doing staging data to Data warehouse.
The finally the fourth player is the front end where we make the report, dashboard, interaction with report. Decision support system help report, several workbooks and formula needed with local data format. Sometimes they need on the desktop MS excel, web browser or mobile devices to view KPIs. So this fourth layer is the final one where they your data turn into action and enables decision-making.
These challenge given above are sometimes even more complex with business processes high then the solution becomes quite complex. Each and every layer has their different solution behavior and different technology stack. This makes sometimes solution impossible to implement and hard to maintain. Since there is no any single common framework from (ETL) extract transformers and loading to data warehouse and from the DW to processing service or from the processing service to report on the front of you this situation becomes quite complex and difficult to solve.
Data warehouse compute service
Generally the business intelligence solutions are distracted. At each and every activity we have several tools. For Analytical processing we are using Microsoft SSAS. To make the report Microsoft Excel power BI, micro strategy or web based custom development happens
Decision making and dynamic dashboard and reporting
DW designed as per data model for the reporting model. The computational capability need to be very strong because this report can be in any shape and the query could be quite complex. Task you execute on Relational database system will be heavy. So we need analytical computer service which is good with the Microsoft analysis services, Oracle BI or SAP BI. When data volume is really high and you need to bring big data. So this third layer solve as analysis layer where the compute is some happens.