Databases are closed

Believe it or not, most likely databases in your organization are closed. By closed I mean that access to data is hindered to the people that should be able to access them freely - data analysts, DBAs, software developers, support consultants. How is that? Well, let me explain first, that it has nothing to do with access. Database systems have well established access control solutions. This limitation has a different nature - data is hidden in plain sight. How is this possible? The answer is: because many databases are complex and messy. This creates real barrier for everyone that needs to find and access data.

1 likes

Dataedo Cartoon

Why is it so?

There are many reasons why databases are complex and messy. More on that in another post. But, in my opinion, the reason why organizations don't deal with this sufficiently, is that they don't collect and preserve knowledge about their databases.

Many applications are third party

Most ERP, CRM, HRM, billing and other packaged applications are delivered by third party vendors. Those vendors often have documentation of the databases and applications they develop. The same goes for a company or a team that is responsible for implementation of the application in your organization. But after implementation is over they are often gone, and the documentation with them.

It is also important to understand that "standard" documentation is fit for a standard blank database. Your implementation is unique - it has customizations and configuration and you use just a subset of the overall solution which is often very big.

You don't ask

If you are responsible for implementation of new applications or development of new functionalities, do you request a documentation of an underlying database? Most likely not, because it doesn't meet any immediate need (that need often comes much later) and it seems a technical detail you don't need to worry about. You need to worry about functionalities. It is also not a part of the scope of the project and acceptance criteria. And the supplier or development team has no incentive to provide it. On the contrary, it has a good incentive to make you dependent on them.

Documentation is overlooked anyway

Any kind of technical documentation is quite often considered second-class deliverable. Even if requested, it can be quite often omitted without anyone noticing. If checked for, no-one tests it for quality or completeness. A simple document with a few pages will do. Even if it's properly tested, it's then rarely kept up to date with all the changes. And those changes always happen. This means that the value of documentation (that is not up to date) after a while might not be much.

Why open databases?

But why should we open databases, you'll probably ask. There are many reasons, the most important being - you will most likely need to make use of your data beyond what your existing applications enable you to.

Cheaper and faster data analysis, reporting and data warehousing

The first time you'll probably face the problem of lack of understanding your databases, is when you want to analyze your data or create simple reports. You'll learn that for people that were not part of the original team that implemented the particular application, it might be hard to find the right data.

Another case where a severe problem might be, is with implementation of any of the data analytics solutions - Data Warehouse, Business Intelligence, Performance Management Dashboards or Big Data.

You might not always be aware, but it will manifest itself in:

• cost of development,
• long time of delivery,
• errors in numbers, often critical and
• dependency on people.

Cheaper application development and integration

No application is a silo. Applications in your organization need to be integrated, more and more each year. Organizations need to be more and more agile and adopt changes in their applications and processes more quickly. To be able to implement new applications and integrate with existing infrastructure or change or develop new functionalities in existing applications you need to understand database schema.

Integration often occurs directly in databases - applications read and write data directly to/from tables or views.

You don't lose data: migration

A day will come when your existing cutting-edge application will face the end of its life cycle. It will be obsolete and will probably be about to be replaced with the next best thing in the area. You will most likely be happy to close it away in a dark warehouse where it will belong. Your new application would be designed for all the processes, functions, data and reporting requirements. But you won't be able to get rid of the old application just like that - you need your old data, right? You would be able delete all the code of the old application, but you'll need to migrate most of its data. And this is where it gets tricky - you (and the new supplier) will need to understand old database schema well. But the supplier of the old application won't be there any more or won't be interested to help you.

And this is where it might be quite hard - not only will it take more time and money to learn and migrate data, but what is even more important, you might migrate something incorrectly or even loose data. Imagine that couple months after migration you realized you have lost the values of discounts you provided to your customers - sales figures were ok, but you didn't specify discounts as a part of acceptance criteria. Terrifying, right? A good documentation of your database can lower the risk of such incidents.

How to open databases

There is one simple solution to complexity and an untidiness of databases - documenting them with Data Dictionaries and sharing this documentation within the organization. It's not that hard, but requires a small effort and organizational changes to do so.

We are suggesting the following steps in the process:

1. Choose a tool,
2. Create a team for each database,
3. Share throughout the organization,

4. Make documentation an integral part of working with your databases - update when:

   a) developing databases and

   b) as users learn more about schema and data.

More on the process in the future. Make sure to subscribe to our blog.

Start today

Check out our tool that will help your entire organization start building and maintaining Data Dictionaries for your databases.

Read more

1. Why It's Hard to Find Data And Why You Need a Map: Data Dictionary
2. Is Lack of Database Documentation a Global Developer Problem?

or

Start building Data Dictionaries today