Tuesday, May 11, 2010

Enterprise Data Quality and SSIS

I'm reading: Enterprise Data Quality and SSISTweet this !
A general perception of Business Intelligence is Data Warehousing and using Analytical solutions. But this is a 10,000 Ft picture. If an end-to-end BI solution is considered, ETL, Operational Reporting, Relational and Analytical Data Marts or Data Warehouse development, Dashboard Development and deployment over a collaborative platform like Sharepoint becomes visible from a 1,000 Ft. I find a major part of the MS BI Community busy in taking steroids of T-SQL Performance Tuning or engrossed in wrestle mania of re-blogging content that has been published tonnes of times and is available even on MSDN, many community websites are even busy in saving the legacy of SQL Server 7.0 / SQL Server 2000 and entertaining blind community challenges on the same. If you are working with a solution provider where you get all different kinds of projects from a vast client base, you would find that the world is big enough and there is more spectrum in BI and more sensible exercises to carry out than the typical ones or the obsolete ones. When you take a 100 Ft level closer look at any of the development aspect like ETL for example, you would find that non-microsoft players would also come into picture that would integrate with MS BI technology stack, aspects like Data Quality, Data Cleansing, Systems Integration would start popping up. This would feel like I am describing a wonderland, but the fact is that you realize this only when you work in that role.

As usual, enough of theory and coming down to the subject of this post, I just have two points to mention. Firstly, Data Quality is a very important aspect of any Data Migration or Data Cleansing or Systems Integration Programme. Even when compliance or audits becomes the main objective, more than 80% of the project depends upon data quality. Perception of compliance is limited to auditing like C2, SOX, HIPPA and others. But auditing is one of the aspect of these compliance and there's more to it which also includes data quality compliance. Secondly, in SSIS, Data Profiling task is one of the best improvements after 2005 version. It can be used to make intelligent decisions based on the quality of the data, when the same is processed.

If you really intend to learn more on data quality, I suggest to read this whitepaper on Enterprise Data Compliance. This whitepaper explains the legal complexities and implications of data quality, business process that affects data quality, and technical architecture and components needed to establish enterprise data quality. After you are able to grasp this architecture, you would definitely find that SSIS can be used exhaustively as the primary tool to craft a data quality solution. Though just the Data Profiling task would fall short and rest of the requirements would have to be tailored manually using different tasks and transforms, but I am very sure that SSIS still has a large scope to evolve and the present potential is reasonable enough to consider it's use for data quality exercises.

Monday, May 10, 2010

Free SQL Server Ebook : Defensive database programming with SQL Server

I'm reading: Free SQL Server Ebook : Defensive database programming with SQL ServerTweet this !
It's been raining free content and resources for SQL Server these days. Every next day when I think I would blog about a technical topic and I get hit by a new free resource on SQL Server, good enough to make me hold my technical post and blog about this free content. Today it's the favourite food for developers, a free ebook on SQL Server.

Most people look out for interview questions and answers ebooks to cross technical interviews. In my views, if you really want to cross a technical interview, read such books and check yourself whether you are up-to-date with your concepts and viewpoints by matching it with the viewpoint and guidelines presented in such books. I have not read this entire book completely, as the moment I came to know about this book, I took a 10-15 min glance of this book and made up my mind to blog about it. It's a nice ebook and the subject is relevant itself to reveal what the book is about. For dummies, this book mostly discusses how NOT to do database programming from a long and/or short term vision. Book contains many code examples which is like a feast for SQL Developers. Click here to download this book.


Reference: SQL Aloha

Saturday, May 08, 2010

Free SQL Azure Videos and Guide

I'm reading: Free SQL Azure Videos and GuideTweet this !
Free downloads of developer resources is one of the most sought after thing in any technology. Not to mention anyone is specific, but I have even seen many sites posting lots of free content for download like Free SQL Server 2008 Book / Ebook and Trial download of SQL Server 2008 and other such contents, which pumps up their readership exponentially. But when you see the stats of pure technical content that is posted free on such sites, it would be shocking to know that it would have comparatively far lesser hits. And nope, I am not discussing about my blog, neither I am trying to discuss how to increase your sql server blog traffic nor I am going to reveal traffic stats of any community websites, but I am trying to give a viewpoint on how much crowd is after the quest of knowledge and how much is after just gathering free download like books, tools and other such stuff.

I agree that I could have just posted link to the free sql azure resources without the lecture in the above paragraph, but I thought I can make you forcibly read this lecture as you too came for free stuff on my blog :) So below is the prize for visiting my blog.

Thursday, May 06, 2010

Free Visio Viewer

I'm reading: Free Visio ViewerTweet this !
IT cannot work in Isolation, and after a certain level in one's career, one cannot just stick with routine code business. Whether from a modeling perspective or from a data processing perspective, Visio finds it's tiny space in SQL Server ecosystem in the form of Visio as a tool for modeling diagrams or Visio services for strategy maps in Sharepoint 2010.

Even for our day to day purpose, usage of Visio file formats like .vsd among the developer community are as frequent as .gif or .jpg in the office users community. Microsoft Visio is not a free tool, but viewing should not cost anything. Visio diagrams are much more packed with data and information compared to other file formats and for the same a more intelligent viewer is required. Fortunately Microsoft has released a free viewer, and I feel this is one of the must have additional tools to keep in your kit. Visio 2010 viewer is a free download available from Microsoft and can be downloaded from here. This viewer can view files of .vsd, .vss, .vst, .vdx, .vsx, or .vtx extension.

Monday, May 03, 2010

Lack of support for spatial data movement in SSMS and SSIS

I'm reading: Lack of support for spatial data movement in SSMS and SSISTweet this !
Spatial data has definitely got some level of support in SSMS, but that is more limited in terms of visualization of data in the results tab. When I consider to move the same data out of SQL Server i.e when I want to import and/or export this data using the Import Export Wizard, it breaks up as it is not upgraded with a corresponding data type in SSIS.

Even if you modify the config file that Import Export Wizard uses to map it to any data-type and facilitate the data movement, it does not help. Using varchar(max) or varbinary(max) also helps for some destination and breaks for others. To the best of my knowledge, I am not able to figure out a generic datatype that can take the helm of pulling or pushing any spatial datatype on it's back for a smooth import - export.

The same issue is faced in SSIS. As of date, there is no straight mapping for spatial data types. Though spatial datatypes got introduced in SQL Server 2008, SSIS has not been upgraded even in the R2 version to accommodate smooth movement of spatial data. DT_IMAGE is the best option as of now to move the same in SSIS, or the best approach is to convert spatial data in WKT format before it leaves SQL Server and read the same in SSIS. Converting in WKT makes it easier to analyze and manipulate this data in SSIS and then use a Script task or the same WKT to push data back into the spatial columns.

Acceptance of spatial datatypes and it's use would grow gradually, but R2 would have been good release to incorporate the support for spatial data movement in and out of SQL Server as well as in SSIS too.
Related Posts with Thumbnails