Hybrid Storage Models for High-throughput Vector Retrieval
Keywords:
vector retrieval, hybrid storage, nearest neighbor search, vector databases, HNSW, DiskANN, data quantization, high-performance computing, AI data managementAbstract
This study examines the characteristics of employing hybrid models for high-performance vector search. The objective of this paper is to substantiate and systematize existing hybrid data storage schemes based on a memory hierarchy (DRAM ? SSD ? HDD) in order to enhance the efficiency of vector retrieval procedures. As a methodological foundation, a broad review of key publications devoted to graph index structures, inverted files and their hybrid combinations was carried out, supplemented by a comparative analysis of their performance according to primary metrics. On the basis of the obtained data, a conceptual model of a multilevel storage architecture is described, demonstrating pathways to achieve an optimal balance between query processing speed (QPS) and search completeness (recall) through adaptive quantization and the rational construction of index structures. The scientific novelty lies in the description of a unified architectural scheme integrating various memory types and indexing approaches to ensure highly efficient and scalable vector search in dynamically updated environments. The results presented in this work will be of interest to data engineers, AI system architects and researchers in the field of big data management.
References
[1]. Grand View Research. (2024). Artificial intelligence market size. https://www.grandviewresearch.com/industry-analysis/artificial-intelligence-ai-market#:~:text=The%20global%20artificial%20intelligence%20market,35.9%25%20from%202025%20to%202030 (Retrieved June 10, 2025).
[2]. Forbes Technology Council. (2023, May 23). Lessons learned from selfless leadership. Forbes. https://www.forbes.com/councils/forbestechcouncil/2023/05/23/lessons-learned-from-selfless-leadership/ (Retrieved April 11, 2025).
[3]. Acer S. et al. (2021). EXAGRAPH: Graph and combinatorial methods for enabling exascale applications. In The International Journal of High Performance Computing Applications, 35(6): 553-5717 https://doi.org/10.1177/10943420211029299 .
[4]. Diwan, H., et al. (2024). Navigable graphs for high-dimensional nearest neighbor search: Constructions and limits. Advances in Neural Information Processing Systems, 37, 59513–59531.
[5]. Shrivastava, A., Song, Z., & Xu, Z. (2023). A theoretical analysis of nearest neighbor search on approximate near neighbor graph. arXiv Preprint arXiv:2303.06210. https://doi.org/10.48550/arXiv.2303.06210.
[6]. Khan, S., et al. (2024). BANG: Billion-scale approximate nearest neighbor search using a single GPU. arXiv Preprint arXiv:2401.11324. https://doi.org/10.48550/arXiv.2401.11324.
[7]. Gollapudi, S., et al. (2023). Filtered-DiskANN: Graph algorithms for approximate nearest neighbor search with filters. In Proceedings of the ACM Web Conference 2023 (pp. 3406–3416). https://doi.org/10.1145/3543507.3583552.
[8]. Chen, Q., et al. (2024). MS MARCO Web Search: A large-scale information-rich web dataset with millions of real click labels. In Companion Proceedings of the ACM Web Conference 2024 (pp. 292–301). https://doi.org/10.1145/3589335.3648327.
[9]. Chakraborty T., Bera D. A sketch-based approach towards scalable and efficient attributed network embedding : dis. – IIIT-Delhi, 2021 (pp. 19–44).
[10]. Muhamed A. et al. (2023). Web-scale semantic product search with large language models. In Pacific-Asia Conference on Knowledge Discovery and Data Mining. – Cham : Springer Nature Switzerland, 73-85.
[11]. Majumdar A. et al. (2021)7 Zson: Zero-shot object-goal navigation using multimodal goal embeddings In Advances in Neural Information Processing Systems, 35, 32340-32352.
[12]. Wang, M., et al. (2021). A comprehensive survey and experimental comparison of graph-based approximate nearest neighbor search. arXiv Preprint arXiv:2101.12631. https://doi.org/10.48550/arXiv.2101.12631.
[13]. Li Z., Li H., Meng L. (2023). Model compression for deep neural networks: A survey In Computers, 12 (3). https://doi.org/10.3390/computers12030060 .
[14]. Activeloop. (2024). Activeloop named 2024 Gartner Cool Vendor in data management. https://www.activeloop.ai/resources/gartner-cool-vendor/ (Retrieved May 12, 2025).
[15]. Gartner. (2024). Cool vendors in data management: GenAI disrupts traditional technologies. https://www.gartner.com/en/documents/5476395 (Retrieved April 30, 2025).
Downloads
Published
Issue
Section
License
Copyright (c) 2025 American Scientific Research Journal for Engineering, Technology, and Sciences
This work is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License.
Authors who submit papers with this journal agree to the following terms.
