news

한어Русский языкEnglishFrançaisIndonesianSanskrit日本語DeutschPortuguêsΕλληνικάespañolItalianoSuomalainenLatina

it home reported on september 3 that google ai researchers have recently launched the cardbench benchmark, which is mainly for learning cardinality estimation to meet the needs of system evaluation framework.

the cardbench benchmark is a comprehensive evaluation framework that includes thousands of queries across 20 different real-world databases, significantly exceeding any previous benchmark.

project background

cardinality estimation (ce) is the key to optimizing relational database query performance. it involves predicting the number of intermediate results that a database query will return, which directly affects the query optimizer's choice of execution plan.

accurate embeddedness estimates are critical for choosing an efficient join order, deciding whether to use indexes, and choosing the best join method.

these decisions can have a significant impact on query execution time and overall database performance. inaccurate estimates can lead to poor execution plans that can significantly degrade performance, sometimes by orders of magnitude.

cardinality estimation techniques widely used in modern database systems rely on heuristic methods and simplified models, such as assuming data uniformity and column independence.

although these methods are computationally efficient, they often require accurate predictions of cardinality, which is particularly evident in complex queries involving multiple tables and filters.

recent data-driven approaches attempt to model the distribution of data within and across tables without executing queries, which reduces some of the overhead but still requires retraining when the data changes.

despite these advances, the lack of comprehensive benchmarks makes it difficult to compare different models and evaluate their generalizability across different datasets.

CardBench

cardbench enables a more comprehensive evaluation of learned cardinality models under various conditions. the benchmark supports three key settings:

instance-based models, i.e. trained on a single dataset;

zero-shot models, which are pre-trained on multiple datasets and then tested on an unseen dataset;

fine-tune the model, that is, pre-train it and then fine-tune it using a small amount of data from the target dataset.

the benchmark provides two sets of training data: one for single table queries with multiple filter predicates and the other for binary join queries involving two tables.

the benchmark consists of 9125 single-table queries and 8454 binary join queries on one of the smaller datasets, ensuring a robust and challenging environment for model evaluation.

for example, the fine-tuned graph neural network (gnn) model achieves a median q-error of 1.32 and a 95th percentile of 120 on binary join queries, significantly outperforming the zero-shot model. the results show that fine-tuning a pre-trained model can significantly improve its performance even with 500 queries. this makes them feasible for real-world applications with limited training data.

in summary, cardbench represents a significant advance in learned cardinality estimation. researchers can spur further innovation in this critical area by providing a comprehensive and diverse benchmark to systematically evaluate and compare different ce models.this baseline is able to support fine-tuning models that require less data and training time.it provides a practical solution for real-world applications where training new models is too expensive.