Save content
Have you found this content useful? Use the button above to save it to your profile.
AIA

There is a simpler way to create business intelligence. By Olly Bond

by
22nd Jun 2006
Save content
Have you found this content useful? Use the button above to save it to your profile.

The Japanese have a word for technology overkill: "chindogu". The term originally referred to consumer electronics products that were so complex and over-specialised they complicated life rather than improving it.

But chindogu can also apply to the business intelligence systems from which managers and workers get the data they need to do their jobs effectively. Software solutions intended to let end users get their own data can be maddeningly complex.

Now more than ever, businesses need greater access to more information more quickly to make decisions. But are they getting it? Despite collectively spending billions on data management and business intelligence tools, the answer in many cases is no. Instead, many BI solutions cause more work, both for end users and the IT department.

This introductory extract from the Datawatch whitepaper, 'Report Mining: An Easier Way to Access Corporate Information' provides a basic introduction to the different breeds of business intelligence and sets the scene for an alternative solution: report mining.

Hardcopy reports
Far and away the most frequently used corporate information remains the printed report. ERP and other enterprise information systems typically offer a
large library of existing ('canned') reports, which can be run anytime. Often these reports contain too much information. And, of course, all the information is frozen on the printed page. There's just no way to get at that data, it seems, without digging through mountains of printouts and wasting more time manually rekeying data from reports into spreadsheets or other applications.

Hardcopy reports
Pros
  • An existing informational asset: no new programming work required
  • Very easy to re-produce anytime

Cons

  • Enormous waste of paper
  • No way to interactively work with the data
  • Incurs many costs beyond paper, including storage costs, man-hours lost re-keying data from reports into spreadsheets, etc.

Data query tools applied to operational databases
SQL-based (BI) applications seek to provide direct access to data within production databases, so managers can then request the exact information they need anytime, on an ad hoc basis. Using BI tools to retrieve information directly from databases was supposed to end the need to run and print most 'canned' reports, but organisations are printing more reports than ever before.

One problem is the data typically accessed by BI solutions is sitting in operational systems, which are designed to quickly and efficiently process
and store data for individual business transactions, not to quickly and efficiently enable analysis of data and making decisions.

In many cases, direct end user access to data often yields unsatisfactory results. Operational systems suffer under the heavy load of query processing, frustrated managers spend too much time simply trying to build the correct query for the information they need, and the data query burden often falls on the IT department.

SQL query tools
Pros
  • Delivers live data to users
  • Enables interactive analysis

Cons

  • Frequent or complex data queries slow down operational systems,
    often dramatically
  • Tools often prove too difficult for end users to work with
  • IT department often incurs long queue of user requests for help
  • Potential security issues associated with granting direct access to core databases.

Data warehouses & data marts
To reap the promised benefits of high-end BI solutions, companies might implement a data warehouse, or a data mart; a smaller, tactical data warehouse. Both are intended to provide access to data for analysis and decision making without impacting operational systems.

To start building the data warehouse, 'raw' data is extracted from operational databases at pre-determined intervals, and is then combined into just a few monolithic tables. This data cleansing and migration process is referred to as Extraction, Transformation and Loading (ETL).

Organisations can yield a compelling return on investment, and even reap tremendous business-transforming insights from a data warehouse/mart. However, data warehousing projects can also fail completely, often due to a faulty ETL process: data from different systems is mapped together improperly; some needed data is not included in the data warehouse at all; or just plain wrong data is included into the data warehouse due to miscommunication between end users and IT personnel. The result: a 'data outhouse,' a label of failure applied to a data warehouse with dirty,
incorrect and worthless data.

SQL query tools
Pros
  • Delivers live data to users
  • Enables interactive analysis
  • Avoids adversely affecting operational system performance
  • Can yield tremendous insight

Cons

  • High risk of failure if data ETL process not performed properly
  • Data warehouse/mart creation very expensive, even if done right

Faced with these limitations, Datawatch pioneered the concept of "report mining", which can eliminate many of the risks within the ETL process to give data warehouse and data mart projects a far greater, and faster, opportunity for success. In essence, report mining captures the data contained in reports and allows it to be analysed, transformed or extended.

Report mining works by taking an instance of a report and applying a predefined model to it. This produces a spreadsheet-style presentation of the tdata as well as a variety of analyses and charts if required. The caputred data can be copied and pasted into other applications - Excel, for example, - or it can be exported in formats ranging from Access and SCV to HTML and PDF.

You can find out more about how the technique works in detail by downloading the 15-page Datawatch whitepaper, 'Report Mining: An Easier Way to Access Corporate Information'.

About the author
Olly Bond is product manager for Datawatch Europe, the business intelligence speciast responsible for the Monarch report mining application.

Tags:

Replies (6)

Please login or register to join the discussion.

avatar
By User deleted
26th Jul 2006 12:24

Why does anyone purchase ......
a system that does not allow paper based reports to be exported in some form (i.e. Excel, csv etc)

The thrust of the article seems to be about stripping data out of existing reports; however this would surely not be necessary if exports existed

From memory there was a product Monarch?? in existance over 10 years ago that provided just this functionality - i.e. create a template & fire the report through the system; resulting in an export file of some type

Thanks (0)
John Stokdyk, AccountingWEB head of insight
By John Stokdyk
26th Jul 2006 12:45

About the author...
Thanks for your comment, JC. I should probably clarify matters by pointing out that Olly Bond is product manager for Datawatch, which developed the Monarch system. This piece is based on a company whitepaper posted on AccountingWEB. I found the explanations of the report types quite helpful.

The paper has a lot more detail about how Monarch actually works, if you want to find out more about it.

My apologies for omitting to add a clear identification at the end of the piece. This will be rectified very shortly.

John Stokdyk
Technology editor
AccountingWEB.co.uk

Thanks (0)
avatar
By User deleted
27th Jul 2006 12:11

Database knowledge is the key element
To my mind the extraction of B.I. data from reports is prone to the same problems and limitations as those that apply to SQL Queries and Data Warehouses. In that a fundimental understanding of data structures, their relationships and limitations is essential before B.I. data can be extracted and used with any confidence.

Unless an end user has such knowledge, or can easily communicate with someone who has that knowledge, the type of vehicle used to extract the B.I. data becomes subordinate to this key element. So the placement of IT staff into end user departments is the critical step that many organisations miss for whatever reason.

Also, to be useful B.I. data must be timely. Getting the required data after a deadline has passed is of little or no benefit. That is why I believe that the ability to construct SQL Queries directly in the operational ERP system is vitual.
As it greatly reduces the I.T. delay in constructing Data Warehouses/external databases.
Once a SQL Query has been constructed it can be incorporated into a B.I. Score Card for running as many times as required (system performance permitting). Allowing managers near instant access to B.I. data. In this way fluctuations/deviations can be quickly spotted and appropriate management actions taken.

Such an implementation may not be suitable for very large international organisations (due mainly to the volume of data), but can work remarkably well for the vast majority of small/medium sized organisations.

Regards Robert Brook

Director of Rowanberry Consultancy Ltd.
Seller of Acceptum Business Software.

Thanks (0)
avatar
By User deleted
26th Jul 2006 15:16

Appologies missed the reference to Monarch ....
but Alastair is quite right about his mainframe comments - good product in its day

The only possible exception is where auditors need to extract data in special circumstances

He is also perfectly correct about users (not)knowing what they want - otherwise it is cystal ball gazing

Thanks (0)
avatar
By listerramjet
26th Jul 2006 13:25

its in the title!
there is a cost of getting information out of a system or systems, and often part of that cost is getting the base system to capture the required data.

data in a production system is geared to the needs of that system/business process - it would be strange if it was not. and the design of that rarely (never) takes account of information needs from a BI system.

this type of requirement can never be met from a commodity application - many have tried (and failed). It has to capture data from disparate systems, and often it needs interpretation before it gets near a user, and of course identifying generic requirements is difficult/impossible.

That is all implicit in this piece, simply because it talks about technical solutions rather than business solutions; but the main problem (that this masks) is that end users actually don't know what they want, and even if they do then those wants change over time - and they are driven by the dynamics of the business and its market place(s). By the time you have built what you want today, it is out of date!

This becomes more obvious when you try and implement this stuff - it is like a new years resolution, lots of enthusiasm over christmas, but by january 5th it is gathering cobwebs!

In fact the highlight of this item is in the title, and it is the word simpler. Work out in simple terms what you want, and perhaps you might both get it and use it, but accept that you are likely to have to rap with a techie. And if you don't believe me then read the latest installment of the CEOs diary.

Thanks (0)
avatar
By listerramjet
26th Jul 2006 13:29

ps
I use Monarch - it's a great way of getting data out of mainframe reports, but it has had its day. Our mainframe is on its way out, and it is easier to specify reports and data extracts and get them to users from the modern network system that is supplanting the mainframe. Monarch is becoming redundant.

Thanks (0)