Add vector support to MariaDB #3257
Draft
+2,951
−2,262
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
as it can never be null (only "" or "disabled")
Bounded_queue<> pretended to be a typesafe C++ wrapper on top of pure C queues.h. But it wasn't, it was tightly bounded to filesort and only useful there. * implement Queue<> - a typesafe C++ wrapper on top of QUEUE * move Bounded_queue to filesort.cc, remove pointless "generalizations" change it to use Queue. * remove bounded_queue.h * change subselect_rowid_merge_engine to use Queue, not QUEUE
the information about index algorithm was stored in two places inconsistently split between both. BTREE index could have key->algorithm == HA_KEY_ALG_BTREE, if the user explicitly specified USING BTREE or HA_KEY_ALG_UNDEF, if not. RTREE index had key->algorithm == HA_KEY_ALG_RTREE and always had key->flags & HA_SPATIAL FULLTEXT index had key->algorithm == HA_KEY_ALG_FULLTEXT and always had key->flags & HA_FULLTEXT HASH index had key->algorithm == HA_KEY_ALG_HASH or HA_KEY_ALG_UNDEF long unique index always had key->algorithm == HA_KEY_ALG_LONG_HASH In this commit: All indexes except BTREE and HASH always have key->algorithm set, HA_SPATIAL and HA_FULLTEXT flags are not used anymore (except for storage to keep frms backward compatible).
…uild with libxml2 2.12 libxml2 2.12.0 made `xmlGetLastError()` return `const` pointer: https://gitlab.gnome.org/GNOME/libxml2/-/commit/61034116d0a3c8b295c6137956adc3ae55720711 Clang 16 does not like this: error: assigning to 'xmlErrorPtr' (aka '_xmlError *') from 'const xmlError *' (aka 'const _xmlError *') discards qualifiers error: cannot initialize a variable of type 'xmlErrorPtr' (aka '_xmlError *') with an rvalue of type 'const xmlError *' (aka 'const _xmlError *') Let’s update the variables to `const`. For older versions, it will be automatically converted. But then `xmlResetError(xmlError*)` will not like the `const` pointer: error: no matching function for call to 'xmlResetError' note: candidate function not viable: 1st argument ('const xmlError *' (aka 'const _xmlError *')) would lose const qualifier Let’s replace it with `xmlResetLastError()`. ALso remove `LIBXMLDOC::Xerr` protected member property. It was introduced in 65b0e54 along with the `xmlResetError` calls. It does not appear to be used for anything.
needed to get partitioning and information about secondary objects
…EM VERSIONING" This partially reverts 43623f0 Engines have to set ::position() after ::write_row(), otherwise the server won't be able to refer to the row just inserted. This is important for high-level indexes. heap part isn't reverted, so heap doesn't support high-level indexes. to fix this, it'll need info->lastpos in addition to info->current_ptr
MDEV-33407 Parser support for vector indexes The syntax is create table t1 (... vector index (v) ...); limitation: * v is a binary string and NOT NULL * only one vector index per table * temporary tables are not supported MDEV-33404 Engine-independent indexes: subtable method added support for so-called "high level indexes", they are not visible to the storage engine, implemented on the sql level. For every such an index in a table, say, t1, the server implicitly creates a second table named, like, t1#i#05 (where "05" is the index number in t1). This table has a fixed structure, no frm, not accessible directly, doesn't go into the table cache, needs no MDLs. MDEV-33406 basic optimizer support for k-NN searches for a query like SELECT ... ORDER BY func() optimizer will use item_func->part_of_sortkey() to decide what keys can be used to resolve ORDER BY.
Introduce scripts and Dockerfile for executing the `ann-benchmarks` tool, aimed at vector search performance testing. Support running the ANN benchmarking both in GitLab CI and manually. Developer Interface: Both of the scripts provide flexibility for altering default behavior via environment variables. Refer to the detailed description in the scripts' documentation section. - `run-local.sh`: This script facilitates the execution of the ANN (Approximate Nearest Neighbors) benchmarking test either against local builds or a specified folder where the MariaDB server is installed. - `run-docker.sh`: This script automates the execution of the ANN benchmarking test within a Docker container. It builds the required Docker image if it doesn't exist or if forced, then build the source code and runs the benchmark in the specified workspace. GitLab CI Build: - A new job `ann-benchmark` is included in the test stage. This job runs ann-benchmark against the MariaDB server built in Ubuntu 22.04. Initially, we are using the `random-xs-20-euclidean` dataset with 20 dimensions and 10000 records.
|
|
cvicentiu
force-pushed
the
bb-11.4-vec-preview
branch
from
201fff7
to
5958acb
Compare
This commit includes the work done in collaboration with Hugo Wen from
Amazon:
MDEV-33408 Alter HNSW graph storage and fix memory leak
This commit changes the way HNSW graph information is stored in the
second table. Instead of storing connections as separate records, it now
stores neighbors for each node, leading to significant performance
improvements and storage savings.
Comparing with the previous approach, the insert speed is 5 times faster,
search speed improves by 23%, and storage usage is reduced by 73%, based
on ann-benchmark tests with random-xs-20-euclidean and
random-s-100-euclidean datasets.
Additionally, in previous code, vector objects were not released after
use, resulting in excessive memory consumption (over 20GB for building
the index with 90,000 records), preventing tests with large datasets.
Now ensure that vectors are released appropriately during the insert and
search functions. Note there are still some vectors that need to be
cleaned up after search query completion. Needs to be addressed in a
future commit.
All new code of the whole pull request, including one or several files
that are either new files or modified ones, are contributed under the
BSD-new license. I am contributing on behalf of my employer Amazon Web
Services, Inc.
As well as the commit:
Introduce session variables to manage HNSW index parameters
Three variables:
hnsw_max_connection_per_layer
hnsw_ef_constructor
hnsw_ef_search
ann-benchmark tool is also updated to support these variables in commit
HugoWenTD/ann-benchmarks@e09784e for branch
https://github.com/HugoWenTD/ann-benchmarks/tree/mariadb-configurable
All new code of the whole pull request, including one or several files
that are either new files or modified ones, are contributed under the
BSD-new license. I am contributing on behalf of my employer Amazon Web
Services, Inc.
Co-authored-by: Hugo Wen <wenhug@amazon.com>
cvicentiu
force-pushed
the
bb-11.4-vec-preview
branch
from
5958acb
to
ba7e8d8
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This is a WIP commit for MariaDB Vector. The PR will be rebased on top of 11.6, as soon as the branch contains all the most recent fixes.