PROJECT TITLE :

A High-Performance Index for Real-Time Matrix Retrieval

ABSTRACT:

The representation of data has been significantly altered as a result of the "embedding" method, which is a fundamental part of machine learning. Word embedding, image embedding, and audio embedding are some examples of embedding that can be found online. Using embedding techniques, matrices can be used to represent a wide variety of objects that exist in the real world. For illustration purposes, a document might be depicted as a matrix, with each row of the matrix standing in for a different word. On the other hand, we have seen that many applications constantly generate new data that is represented by matrices and require real-time query answering on the data. This is something that we have witnessed. In order to retrieve information quickly and easily from these continuously generated matrices, proper management is required. We propose a high-performance index for real-time matrix retrieval in this piece of research that we have written. In addition to providing a quick response to queries, the index also allows for real-time insertions by making use of a log-structured merge tree (LSM-tree). Because the index was designed to work with matrices, it requires a lot more time to search through it and uses a lot more memory than the traditional index does when it comes to retrieving information. In order to address the issues, we have proposed an index that contains precise and fuzzy inverted lists. Additionally, we have designed a number of innovative techniques that will improve the index's memory consumption as well as its search efficiency. To ensure that the index is of high quality, the techniques that have been suggested include vector signature, vector residual sorting, hashing-based lookup, and dictionary initialization. Comprehensive experimental results indicate that our proposed index is superior to the current state-of-the-art method in terms of both the amount of time it saves and the amount of memory it requires to operate.