Current Technology Challenges

Governments and corporations have to choose between a high rate of data ingest and fast data access. Currently there is not a cost effective approach that addresses both these issues. In many cases, current systems simply cannot support current requirements for databases, real-time needs or scaling.

Compounding the challenge, databases continue to grow, and applications have an ever growing need for better data access. As databases grow in size, it is critical to have access to data without the inherent delay of accessing large volumes of information. Traditional approaches of indexing very large databases are inadequate and have failed to meet even the current demand levels. As databases grow in size and complexity, the time required to retrieve information increases non-linearly. Simply put, as the size of the DB increases the slower it becomes. Adding more processing capacity has little value as data size mushrooms. Databases are growing faster than the capacity of traditional hosting technologies.

Efficient, high performance indexing of very large databases is a good way to achieve performance improvement. However with very large indexes these systems face challenges from managing indexes in memory or the I/O limits.

Governments and corporations with very large databases experience high latency from:

• High Query Rates: The volume of requests from databases exceeds the databases’ ability to return the information in a timely manner.
• High Ingest: Due to the large volume of data added to the system, the timeliness and accuracy of information is not readily available when immediately required.

Many advanced database applications are designed to access, store and retrieve vast amounts of data. However, even the most robust database solution has an inherent limitation of performance when:

• The amount of query requests made to the database exceeds the database’s ability to provide real-time access to information.
• The database has to search each index tree for the required information.
• The high rates of data ingest limit systems’ ability to provide real time access to information.
• Scaling the database infrastructure to meet the demand is not a cost-effective solution.

Beyond performance limitations, there are significant costs associated with adding additional complexity, personnel resources, and maintenance costs for existing solutions.

Adding to the problem is the number of users accessing (concurrency) and processing information (analytic processing) across multiple databases. Database administrators have a tremendous challenge of:

1. Maximizing system performance by balancing the index maintenance against system performance.
2. Reducing additional costs in IT infrastructure and index maintenance without disrupting the current environment.

Additional challenges in addressing database latency from a very large database have included:

• Price constraints – Adding infrastructure requires additional costs for index maintenance and IT resources, and leads to costly system and database downtime.
• Dividing database data into smaller segments - Resulting in the added time and resources required to maintain data accuracy and integrity. This solution prevents the database system from scaling as the need for increased data grows and becomes very costly to manage.
• Adding infrastructure – Resulting in excessive demand on IT resources that may disrupt the current environment, increasing the amount of disk I/O and requiring additional index maintenance.
• Limiting the use of data - Resulting in the lack of dissemination of information where the data is most required or resulting in limiting of ad hoc capabilities of the system.

Very Large Databases - The Problem.

Many of the performance and scalability challenges mentioned are partly due to limited budgets. But if IT budgets were unlimited, and there was unlimited servers, there would still be performance limitations due to I/O and issues of iterative tuning as applications and users evolve.

Because of the combination of price and performance advantages, adding Quasar to an existing database environment can immediately solve performance and scalability issues, all within the existing system budget. For the price of a typical server upgrade, Quasar's performance can be added to expand ingest/query rates, reduce I/O bottlenecks, provide expanded analytic performance, lift user concurrency limits, meet service levels for production reporting, enable adding new users and applications without fear, and enable obtaining reliable, consistent query performance.