Skip to content

Commit 4a5187d

Browse files
fix(vector): Fix similarity-based HNSW search for cosine and dot product metrics (#9559)
### Summary This PR fixes a critical bug in HNSW vector search where **cosine similarity and dot product metrics returned incorrect results**. The search algorithm was treating all metrics as distance metrics (lower is better), causing similarity metrics (higher is better) to return the *worst* matches instead of the best. ### Problem The HNSW implementation had two issues with similarity-based metrics: 1. **Search phase**: The candidate heap in persistent_hnsw.go::searchPersistentLayer always used a min-heap, which pops the lowest value first. For similarity metrics where higher values are better, this caused the algorithm to explore the worst candidates first and terminate prematurely. 2. **Edge pruning phase**: The helper.go::addNeighbors function used a fixed comparison (`>`) when pruning edges, which is correct for distance metrics but inverted for similarity metrics. This resulted in keeping the worst edges instead of the best. ### Root Cause The original code assumed all metrics behave like distance metrics: ```go // Always used min-heap (pops lowest first) candidateHeap := *buildPersistentHeapByInit(elements) // Edge pruning always used > comparison compare: func(i, j uint64) bool { return ph.distance_betw(..., i, ...) > ph.distance_betw(..., j, ...) } ``` For **Euclidean distance**, lower values = better matches → min-heap is correct. For **Cosine/DotProduct similarity**, higher values = better matches → need max-heap. ### Solution #### 1. Added candidateHeap interface with metric-aware heap selection ```go type candidateHeap[T c.Float] interface { Len() int Pop() minPersistentHeapElement[T] Push(minPersistentHeapElement[T]) PopLast() minPersistentHeapElement[T] } func buildCandidateHeap[T c.Float](array []minPersistentHeapElement[T], isSimilarityMetric bool) candidateHeap[T] { if isSimilarityMetric { return &maxHeapWrapper[T]{...} // Pops highest first } return &minHeapWrapper[T]{...} // Pops lowest first } ``` #### 2. Added isSimilarityMetric flag to SimilarityType ```go type SimilarityType[T c.Float] struct { // ... existing fields isSimilarityMetric bool // true for cosine, dotproduct; false for euclidean } ``` #### 3. Fixed edge pruning comparison in addNeighbors ```go compare: func(i, j uint64) bool { distI := ph.distance_betw(ctx, tc, uuid, i, &inVec, &outVec) distJ := ph.distance_betw(ctx, tc, uuid, j, &inVec, &outVec) return !ph.simType.isBetterScore(distI, distJ) } ``` ### Files Changed | File | Changes | |------|---------| | tok/hnsw/heap.go | Added candidateHeap interface, minHeapWrapper, maxHeapWrapper, and buildCandidateHeap factory | | tok/hnsw/helper.go | Added isSimilarityMetric field to SimilarityType; fixed edge pruning comparison | | tok/hnsw/persistent_hnsw.go | Updated searchPersistentLayer to use metric-aware candidate heap | | tok/hnsw/persistent_hnsw_test.go | Added comprehensive unit tests for heap behavior and search correctness | ### Testing Added new tests covering: - TestCandidateHeapMinHeap: Verifies min-heap pops in ascending order - TestCandidateHeapMaxHeap: Verifies max-heap pops in descending order - TestCandidateHeapPushPop: Tests Push/Pop operations for both heap types - TestCandidateHeapPopLast: Tests PopLast for both types - TestSimilarityTypeIsSimilarityMetric: Verifies flag is set correctly for each metric - TestSearchReturnsCorrectOrderForAllMetrics: End-to-end test for Euclidean, Cosine, and DotProduct - TestEdgePruningKeepsBestEdges: Verifies edge pruning keeps best edges for each metric ### Performance Note This fix builds on PR #9514 which corrected the early termination condition. Together, these changes ensure HNSW search explores the correct number of candidates and returns properly ordered results. Users experiencing slower insert/search times compared to v25.1.0 can tune performance by lowering efConstruction and efSearch parameters when creating your vector indexes. Lower values trade recall for speed. The default values (efConstruction=128, efSearch=64) prioritize recall. ### GenAI Notice Parts of this implementation and all of the testing was generated using Claude Opus 4.5 (thinking). ### Checklist - [x] The PR title follows the [Conventional Commits](https://www.conventionalcommits.org/en/v1.0.0/#summary) syntax, leading with `fix:`, `feat:`, `chore:`, `ci:`, etc. - [x] Code compiles correctly and linting (via trunk) passes locally - [x] Tests added for new functionality, or regression tests for bug fixes added as applicable Fixes #9558 ### Benchmarks Our BEIR SciFact Information Retrieval Benchmarks now show recall rates close to or exceeding acceptable and excellent performance for all metrics. ``` ============================================================================================================================================ NDCG@k Comparison ============================================================================================================================================ NDCG@1: -------------------------------------------------------------------------------------------------------------------------------------------- BEIR Excellent (State-of-the-art baseline (768-dim)) ██████████████████░░░░░░░░░░░░ 0.6200 BEIR Acceptable (Acceptable baseline (384-dim)) ███████████████░░░░░░░░░░░░░░░ 0.5200 Dgraph v25.1.0 (euclidean) ███████████████░░░░░░░░░░░░░░░ 0.5000 Dgraph v25.1.0 (cosine) ████████░░░░░░░░░░░░░░░░░░░░░░ 0.2767 Dgraph v25.1.0 (dotproduct) ████████░░░░░░░░░░░░░░░░░░░░░░ 0.2867 Dgraph staged-fix (euclidean) ███████████████░░░░░░░░░░░░░░░ 0.5233 Dgraph staged-fix (cosine) ███████████████░░░░░░░░░░░░░░░ 0.5300 Dgraph staged-fix (dotproduct) ███████████████░░░░░░░░░░░░░░░ 0.5167 NDCG@3: -------------------------------------------------------------------------------------------------------------------------------------------- BEIR Excellent (State-of-the-art baseline (768-dim)) ████████████████████░░░░░░░░░░ 0.6700 BEIR Acceptable (Acceptable baseline (384-dim)) ██████████████████░░░░░░░░░░░░ 0.6000 Dgraph v25.1.0 (euclidean) ████████████████░░░░░░░░░░░░░░ 0.5588 Dgraph v25.1.0 (cosine) █████████░░░░░░░░░░░░░░░░░░░░░ 0.3043 Dgraph v25.1.0 (dotproduct) █████████░░░░░░░░░░░░░░░░░░░░░ 0.3164 Dgraph staged-fix (euclidean) █████████████████░░░░░░░░░░░░░ 0.5918 Dgraph staged-fix (cosine) █████████████████░░░░░░░░░░░░░ 0.5957 Dgraph staged-fix (dotproduct) █████████████████░░░░░░░░░░░░░ 0.5830 NDCG@5: -------------------------------------------------------------------------------------------------------------------------------------------- BEIR Excellent (State-of-the-art baseline (768-dim)) ████████████████████░░░░░░░░░░ 0.6900 BEIR Acceptable (Acceptable baseline (384-dim)) ██████████████████░░░░░░░░░░░░ 0.6300 Dgraph v25.1.0 (euclidean) █████████████████░░░░░░░░░░░░░ 0.5858 Dgraph v25.1.0 (cosine) █████████░░░░░░░░░░░░░░░░░░░░░ 0.3197 Dgraph v25.1.0 (dotproduct) █████████░░░░░░░░░░░░░░░░░░░░░ 0.3290 Dgraph staged-fix (euclidean) ██████████████████░░░░░░░░░░░░ 0.6168 Dgraph staged-fix (cosine) ██████████████████░░░░░░░░░░░░ 0.6240 Dgraph staged-fix (dotproduct) ██████████████████░░░░░░░░░░░░ 0.6081 NDCG@10: -------------------------------------------------------------------------------------------------------------------------------------------- BEIR Excellent (State-of-the-art baseline (768-dim)) █████████████████████░░░░░░░░░ 0.7000 BEIR Acceptable (Acceptable baseline (384-dim)) ███████████████████░░░░░░░░░░░ 0.6500 Dgraph v25.1.0 (euclidean) ██████████████████░░░░░░░░░░░░ 0.6118 Dgraph v25.1.0 (cosine) █████████░░░░░░░░░░░░░░░░░░░░░ 0.3305 Dgraph v25.1.0 (dotproduct) ██████████░░░░░░░░░░░░░░░░░░░░ 0.3423 Dgraph staged-fix (euclidean) ███████████████████░░░░░░░░░░░ 0.6461 Dgraph staged-fix (cosine) ███████████████████░░░░░░░░░░░ 0.6505 Dgraph staged-fix (dotproduct) ███████████████████░░░░░░░░░░░ 0.6369 NDCG@100: -------------------------------------------------------------------------------------------------------------------------------------------- BEIR Excellent (State-of-the-art baseline (768-dim)) █████████████████████░░░░░░░░░ 0.7200 BEIR Acceptable (Acceptable baseline (384-dim)) ████████████████████░░░░░░░░░░ 0.6800 Dgraph v25.1.0 (euclidean) ███████████████████░░░░░░░░░░░ 0.6418 Dgraph v25.1.0 (cosine) ██████████░░░░░░░░░░░░░░░░░░░░ 0.3445 Dgraph v25.1.0 (dotproduct) ██████████░░░░░░░░░░░░░░░░░░░░ 0.3555 Dgraph staged-fix (euclidean) ████████████████████░░░░░░░░░░ 0.6794 Dgraph staged-fix (cosine) ████████████████████░░░░░░░░░░ 0.6849 Dgraph staged-fix (dotproduct) ████████████████████░░░░░░░░░░ 0.6707 ============================================================================================================================================ MAP@k Comparison ============================================================================================================================================ MAP@1: -------------------------------------------------------------------------------------------------------------------------------------------- BEIR Excellent (State-of-the-art baseline (768-dim)) ██████████████████░░░░░░░░░░░░ 0.6000 BEIR Acceptable (Acceptable baseline (384-dim)) ███████████████░░░░░░░░░░░░░░░ 0.5000 Dgraph v25.1.0 (euclidean) ██████████████░░░░░░░░░░░░░░░░ 0.4812 Dgraph v25.1.0 (cosine) ███████░░░░░░░░░░░░░░░░░░░░░░░ 0.2586 Dgraph v25.1.0 (dotproduct) ████████░░░░░░░░░░░░░░░░░░░░░░ 0.2747 Dgraph staged-fix (euclidean) ███████████████░░░░░░░░░░░░░░░ 0.5046 Dgraph staged-fix (cosine) ███████████████░░░░░░░░░░░░░░░ 0.5112 Dgraph staged-fix (dotproduct) ██████████████░░░░░░░░░░░░░░░░ 0.4979 MAP@3: -------------------------------------------------------------------------------------------------------------------------------------------- BEIR Excellent (State-of-the-art baseline (768-dim)) ███████████████████░░░░░░░░░░░ 0.6400 BEIR Acceptable (Acceptable baseline (384-dim)) █████████████████░░░░░░░░░░░░░ 0.5700 Dgraph v25.1.0 (euclidean) ████████████████░░░░░░░░░░░░░░ 0.5357 Dgraph v25.1.0 (cosine) ████████░░░░░░░░░░░░░░░░░░░░░░ 0.2883 Dgraph v25.1.0 (dotproduct) █████████░░░░░░░░░░░░░░░░░░░░░ 0.3022 Dgraph staged-fix (euclidean) ████████████████░░░░░░░░░░░░░░ 0.5663 Dgraph staged-fix (cosine) █████████████████░░░░░░░░░░░░░ 0.5707 Dgraph staged-fix (dotproduct) ████████████████░░░░░░░░░░░░░░ 0.5579 MAP@5: -------------------------------------------------------------------------------------------------------------------------------------------- BEIR Excellent (State-of-the-art baseline (768-dim)) ███████████████████░░░░░░░░░░░ 0.6600 BEIR Acceptable (Acceptable baseline (384-dim)) █████████████████░░░░░░░░░░░░░ 0.5900 Dgraph v25.1.0 (euclidean) ████████████████░░░░░░░░░░░░░░ 0.5544 Dgraph v25.1.0 (cosine) ████████░░░░░░░░░░░░░░░░░░░░░░ 0.2993 Dgraph v25.1.0 (dotproduct) █████████░░░░░░░░░░░░░░░░░░░░░ 0.3113 Dgraph staged-fix (euclidean) █████████████████░░░░░░░░░░░░░ 0.5838 Dgraph staged-fix (cosine) █████████████████░░░░░░░░░░░░░ 0.5902 Dgraph staged-fix (dotproduct) █████████████████░░░░░░░░░░░░░ 0.5755 MAP@10: -------------------------------------------------------------------------------------------------------------------------------------------- BEIR Excellent (State-of-the-art baseline (768-dim)) ████████████████████░░░░░░░░░░ 0.6700 BEIR Acceptable (Acceptable baseline (384-dim)) ██████████████████░░░░░░░░░░░░ 0.6000 Dgraph v25.1.0 (euclidean) █████████████████░░░░░░░░░░░░░ 0.5676 Dgraph v25.1.0 (cosine) █████████░░░░░░░░░░░░░░░░░░░░░ 0.3045 Dgraph v25.1.0 (dotproduct) █████████░░░░░░░░░░░░░░░░░░░░░ 0.3175 Dgraph staged-fix (euclidean) █████████████████░░░░░░░░░░░░░ 0.5987 Dgraph staged-fix (cosine) ██████████████████░░░░░░░░░░░░ 0.6035 Dgraph staged-fix (dotproduct) █████████████████░░░░░░░░░░░░░ 0.5900 MAP@100: -------------------------------------------------------------------------------------------------------------------------------------------- BEIR Excellent (State-of-the-art baseline (768-dim)) ████████████████████░░░░░░░░░░ 0.6800 BEIR Acceptable (Acceptable baseline (384-dim)) ██████████████████░░░░░░░░░░░░ 0.6100 Dgraph v25.1.0 (euclidean) █████████████████░░░░░░░░░░░░░ 0.5746 Dgraph v25.1.0 (cosine) █████████░░░░░░░░░░░░░░░░░░░░░ 0.3074 Dgraph v25.1.0 (dotproduct) █████████░░░░░░░░░░░░░░░░░░░░░ 0.3203 Dgraph staged-fix (euclidean) ██████████████████░░░░░░░░░░░░ 0.6060 Dgraph staged-fix (cosine) ██████████████████░░░░░░░░░░░░ 0.6113 Dgraph staged-fix (dotproduct) █████████████████░░░░░░░░░░░░░ 0.5977 ============================================================================================================================================ RECALL@k Comparison ============================================================================================================================================ Recall@1: -------------------------------------------------------------------------------------------------------------------------------------------- BEIR Excellent (State-of-the-art baseline (768-dim)) ██████████████████░░░░░░░░░░░░ 0.6000 BEIR Acceptable (Acceptable baseline (384-dim)) ███████████████░░░░░░░░░░░░░░░ 0.5000 Dgraph v25.1.0 (euclidean) ██████████████░░░░░░░░░░░░░░░░ 0.4812 Dgraph v25.1.0 (cosine) ███████░░░░░░░░░░░░░░░░░░░░░░░ 0.2586 Dgraph v25.1.0 (dotproduct) ████████░░░░░░░░░░░░░░░░░░░░░░ 0.2747 Dgraph staged-fix (euclidean) ███████████████░░░░░░░░░░░░░░░ 0.5046 Dgraph staged-fix (cosine) ███████████████░░░░░░░░░░░░░░░ 0.5112 Dgraph staged-fix (dotproduct) ██████████████░░░░░░░░░░░░░░░░ 0.4979 Recall@3: -------------------------------------------------------------------------------------------------------------------------------------------- BEIR Excellent (State-of-the-art baseline (768-dim)) █████████████████████░░░░░░░░░ 0.7300 BEIR Acceptable (Acceptable baseline (384-dim)) ███████████████████░░░░░░░░░░░ 0.6500 Dgraph v25.1.0 (euclidean) █████████████████░░░░░░░░░░░░░ 0.5984 Dgraph v25.1.0 (cosine) █████████░░░░░░░░░░░░░░░░░░░░░ 0.3248 Dgraph v25.1.0 (dotproduct) ██████████░░░░░░░░░░░░░░░░░░░░ 0.3377 Dgraph staged-fix (euclidean) ███████████████████░░░░░░░░░░░ 0.6384 Dgraph staged-fix (cosine) ███████████████████░░░░░░░░░░░ 0.6401 Dgraph staged-fix (dotproduct) ██████████████████░░░░░░░░░░░░ 0.6284 Recall@5: -------------------------------------------------------------------------------------------------------------------------------------------- BEIR Excellent (State-of-the-art baseline (768-dim)) ███████████████████████░░░░░░░ 0.7900 BEIR Acceptable (Acceptable baseline (384-dim)) █████████████████████░░░░░░░░░ 0.7200 Dgraph v25.1.0 (euclidean) ███████████████████░░░░░░░░░░░ 0.6638 Dgraph v25.1.0 (cosine) ██████████░░░░░░░░░░░░░░░░░░░░ 0.3632 Dgraph v25.1.0 (dotproduct) ███████████░░░░░░░░░░░░░░░░░░░ 0.3697 Dgraph staged-fix (euclidean) ████████████████████░░░░░░░░░░ 0.6988 Dgraph staged-fix (cosine) █████████████████████░░░░░░░░░ 0.7088 Dgraph staged-fix (dotproduct) ████████████████████░░░░░░░░░░ 0.6888 Recall@10: -------------------------------------------------------------------------------------------------------------------------------------------- BEIR Excellent (State-of-the-art baseline (768-dim)) █████████████████████████░░░░░ 0.8400 BEIR Acceptable (Acceptable baseline (384-dim)) ███████████████████████░░░░░░░ 0.7800 Dgraph v25.1.0 (euclidean) ██████████████████████░░░░░░░░ 0.7368 Dgraph v25.1.0 (cosine) ███████████░░░░░░░░░░░░░░░░░░░ 0.3950 Dgraph v25.1.0 (dotproduct) ████████████░░░░░░░░░░░░░░░░░░ 0.4074 Dgraph staged-fix (euclidean) ███████████████████████░░░░░░░ 0.7808 Dgraph staged-fix (cosine) ███████████████████████░░░░░░░ 0.7834 Dgraph staged-fix (dotproduct) ███████████████████████░░░░░░░ 0.7701 Recall@100: -------------------------------------------------------------------------------------------------------------------------------------------- BEIR Excellent (State-of-the-art baseline (768-dim)) ████████████████████████████░░ 0.9500 BEIR Acceptable (Acceptable baseline (384-dim)) ███████████████████████████░░░ 0.9000 Dgraph v25.1.0 (euclidean) ██████████████████████████░░░░ 0.8717 Dgraph v25.1.0 (cosine) █████████████░░░░░░░░░░░░░░░░░ 0.4589 Dgraph v25.1.0 (dotproduct) █████████████░░░░░░░░░░░░░░░░░ 0.4658 Dgraph staged-fix (euclidean) ████████████████████████████░░ 0.9350 Dgraph staged-fix (cosine) ████████████████████████████░░ 0.9417 Dgraph staged-fix (dotproduct) ███████████████████████████░░░ 0.9250 ============================================================================================================================================ PRECISION@k Comparison ============================================================================================================================================ Precision@1: -------------------------------------------------------------------------------------------------------------------------------------------- BEIR Excellent (State-of-the-art baseline (768-dim)) ██████████████████░░░░░░░░░░░░ 0.6200 BEIR Acceptable (Acceptable baseline (384-dim)) ███████████████░░░░░░░░░░░░░░░ 0.5200 Dgraph v25.1.0 (euclidean) ███████████████░░░░░░░░░░░░░░░ 0.5000 Dgraph v25.1.0 (cosine) ████████░░░░░░░░░░░░░░░░░░░░░░ 0.2767 Dgraph v25.1.0 (dotproduct) ████████░░░░░░░░░░░░░░░░░░░░░░ 0.2867 Dgraph staged-fix (euclidean) ███████████████░░░░░░░░░░░░░░░ 0.5233 Dgraph staged-fix (cosine) ███████████████░░░░░░░░░░░░░░░ 0.5300 Dgraph staged-fix (dotproduct) ███████████████░░░░░░░░░░░░░░░ 0.5167 Precision@3: -------------------------------------------------------------------------------------------------------------------------------------------- BEIR Excellent (State-of-the-art baseline (768-dim)) ██████████████████████████████ 0.2700 BEIR Acceptable (Acceptable baseline (384-dim)) █████████████████████████░░░░░ 0.2300 Dgraph v25.1.0 (euclidean) ████████████████████████░░░░░░ 0.2178 Dgraph v25.1.0 (cosine) █████████████░░░░░░░░░░░░░░░░░ 0.1211 Dgraph v25.1.0 (dotproduct) █████████████░░░░░░░░░░░░░░░░░ 0.1211 Dgraph staged-fix (euclidean) █████████████████████████░░░░░ 0.2311 Dgraph staged-fix (cosine) █████████████████████████░░░░░ 0.2322 Dgraph staged-fix (dotproduct) █████████████████████████░░░░░ 0.2278 Precision@5: -------------------------------------------------------------------------------------------------------------------------------------------- BEIR Excellent (State-of-the-art baseline (768-dim)) ██████████████████████████████ 0.1800 BEIR Acceptable (Acceptable baseline (384-dim)) ██████████████████████████░░░░ 0.1600 Dgraph v25.1.0 (euclidean) ████████████████████████░░░░░░ 0.1480 Dgraph v25.1.0 (cosine) █████████████░░░░░░░░░░░░░░░░░ 0.0827 Dgraph v25.1.0 (dotproduct) █████████████░░░░░░░░░░░░░░░░░ 0.0807 Dgraph staged-fix (euclidean) █████████████████████████░░░░░ 0.1553 Dgraph staged-fix (cosine) ██████████████████████████░░░░ 0.1573 Dgraph staged-fix (dotproduct) █████████████████████████░░░░░ 0.1533 Precision@10: -------------------------------------------------------------------------------------------------------------------------------------------- BEIR Excellent (State-of-the-art baseline (768-dim)) ██████████████████████████████ 0.1000 BEIR Acceptable (Acceptable baseline (384-dim)) ██████████████████████████░░░░ 0.0900 Dgraph v25.1.0 (euclidean) █████████████████████████░░░░░ 0.0837 Dgraph v25.1.0 (cosine) █████████████░░░░░░░░░░░░░░░░░ 0.0453 Dgraph v25.1.0 (dotproduct) █████████████░░░░░░░░░░░░░░░░░ 0.0447 Dgraph staged-fix (euclidean) ██████████████████████████░░░░ 0.0887 Dgraph staged-fix (cosine) ██████████████████████████░░░░ 0.0887 Dgraph staged-fix (dotproduct) ██████████████████████████░░░░ 0.0873 Precision@100: -------------------------------------------------------------------------------------------------------------------------------------------- BEIR Excellent (State-of-the-art baseline (768-dim)) ████████████████████████████░░ 0.0100 BEIR Acceptable (Acceptable baseline (384-dim)) ████████████████████████████░░ 0.0100 Dgraph v25.1.0 (euclidean) ███████████████████████████░░░ 0.0100 Dgraph v25.1.0 (cosine) ██████████████░░░░░░░░░░░░░░░░ 0.0053 Dgraph v25.1.0 (dotproduct) ██████████████░░░░░░░░░░░░░░░░ 0.0051 Dgraph staged-fix (euclidean) █████████████████████████████░ 0.0106 Dgraph staged-fix (cosine) ██████████████████████████████ 0.0107 Dgraph staged-fix (dotproduct) █████████████████████████████░ 0.0105 ``` --------- Co-authored-by: Joe Lamming <191030909+joelamming@users.noreply.github.com>
1 parent c5de8a7 commit 4a5187d

5 files changed

Lines changed: 593 additions & 106 deletions

File tree

tok/hnsw/heap.go

Lines changed: 108 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -13,7 +13,7 @@ import (
1313

1414
const notAUid uint64 = 0
1515

16-
type minPersistentHeapElement[T c.Float] struct {
16+
type persistentHeapElement[T c.Float] struct {
1717
value T
1818
index uint64
1919
// An element that is "filteredOut" is one that should be removed
@@ -23,15 +23,15 @@ type minPersistentHeapElement[T c.Float] struct {
2323
}
2424

2525
func initPersistentHeapElement[T c.Float](
26-
val T, i uint64, filteredOut bool) *minPersistentHeapElement[T] {
27-
return &minPersistentHeapElement[T]{
26+
val T, i uint64, filteredOut bool) *persistentHeapElement[T] {
27+
return &persistentHeapElement[T]{
2828
value: val,
2929
index: i,
3030
filteredOut: filteredOut,
3131
}
3232
}
3333

34-
type minPersistentTupleHeap[T c.Float] []minPersistentHeapElement[T]
34+
type minPersistentTupleHeap[T c.Float] []persistentHeapElement[T]
3535

3636
func (h minPersistentTupleHeap[T]) Len() int {
3737
return len(h)
@@ -46,7 +46,7 @@ func (h minPersistentTupleHeap[T]) Swap(i, j int) {
4646
}
4747

4848
func (h *minPersistentTupleHeap[T]) Push(x interface{}) {
49-
*h = append(*h, x.(minPersistentHeapElement[T]))
49+
*h = append(*h, x.(persistentHeapElement[T]))
5050
}
5151

5252
func (h *minPersistentTupleHeap[T]) PopLast() {
@@ -61,12 +61,113 @@ func (h *minPersistentTupleHeap[T]) Pop() interface{} {
6161
return x
6262
}
6363

64-
// buildPersistentHeapByInit will create a tuple heap using the array of minPersistentHeapElements
64+
// buildMinPersistentHeapByInit will create a min-heap for distance metrics
6565
// in time O(n), where n = length of array
66-
func buildPersistentHeapByInit[T c.Float](array []minPersistentHeapElement[T]) *minPersistentTupleHeap[T] {
66+
func buildMinPersistentHeapByInit[T c.Float](array []persistentHeapElement[T]) *minPersistentTupleHeap[T] {
6767
// initialize the MinTupleHeap that has implement the heap.Interface
6868
minPersistentTupleHeap := &minPersistentTupleHeap[T]{}
6969
*minPersistentTupleHeap = array
7070
heap.Init(minPersistentTupleHeap)
7171
return minPersistentTupleHeap
7272
}
73+
74+
// maxPersistentTupleHeap is a max-heap for similarity metrics (cosine, dot-product)
75+
// where higher values indicate better matches.
76+
type maxPersistentTupleHeap[T c.Float] []persistentHeapElement[T]
77+
78+
func (h maxPersistentTupleHeap[T]) Len() int {
79+
return len(h)
80+
}
81+
82+
func (h maxPersistentTupleHeap[T]) Less(i, j int) bool {
83+
return h[i].value > h[j].value // reversed for max-heap
84+
}
85+
86+
func (h maxPersistentTupleHeap[T]) Swap(i, j int) {
87+
h[i], h[j] = h[j], h[i]
88+
}
89+
90+
func (h *maxPersistentTupleHeap[T]) Push(x interface{}) {
91+
*h = append(*h, x.(persistentHeapElement[T]))
92+
}
93+
94+
func (h *maxPersistentTupleHeap[T]) PopLast() {
95+
heap.Remove(h, h.Len()-1)
96+
}
97+
98+
func (h *maxPersistentTupleHeap[T]) Pop() interface{} {
99+
old := *h
100+
n := len(old)
101+
x := old[n-1]
102+
*h = old[:n-1]
103+
return x
104+
}
105+
106+
// buildMaxPersistentHeapByInit will create a max-heap for similarity metrics
107+
// in time O(n), where n = length of array
108+
func buildMaxPersistentHeapByInit[T c.Float](array []persistentHeapElement[T]) *maxPersistentTupleHeap[T] {
109+
maxHeap := &maxPersistentTupleHeap[T]{}
110+
*maxHeap = array
111+
heap.Init(maxHeap)
112+
return maxHeap
113+
}
114+
115+
// candidateHeap is an interface for the candidate heap used in HNSW search.
116+
// It abstracts over min-heap (for distance metrics) and max-heap (for similarity metrics).
117+
type candidateHeap[T c.Float] interface {
118+
Len() int
119+
Push(x persistentHeapElement[T])
120+
Pop() persistentHeapElement[T]
121+
PopLast()
122+
}
123+
124+
// minHeapWrapper wraps minPersistentTupleHeap to implement candidateHeap interface
125+
type minHeapWrapper[T c.Float] struct {
126+
h *minPersistentTupleHeap[T]
127+
}
128+
129+
func (w *minHeapWrapper[T]) Len() int {
130+
return w.h.Len()
131+
}
132+
133+
func (w *minHeapWrapper[T]) Push(x persistentHeapElement[T]) {
134+
heap.Push(w.h, x)
135+
}
136+
137+
func (w *minHeapWrapper[T]) Pop() persistentHeapElement[T] {
138+
return heap.Pop(w.h).(persistentHeapElement[T])
139+
}
140+
141+
func (w *minHeapWrapper[T]) PopLast() {
142+
w.h.PopLast()
143+
}
144+
145+
// maxHeapWrapper wraps maxPersistentTupleHeap to implement candidateHeap interface
146+
type maxHeapWrapper[T c.Float] struct {
147+
h *maxPersistentTupleHeap[T]
148+
}
149+
150+
func (w *maxHeapWrapper[T]) Len() int {
151+
return w.h.Len()
152+
}
153+
154+
func (w *maxHeapWrapper[T]) Push(x persistentHeapElement[T]) {
155+
heap.Push(w.h, x)
156+
}
157+
158+
func (w *maxHeapWrapper[T]) Pop() persistentHeapElement[T] {
159+
return heap.Pop(w.h).(persistentHeapElement[T])
160+
}
161+
162+
func (w *maxHeapWrapper[T]) PopLast() {
163+
w.h.PopLast()
164+
}
165+
166+
// buildCandidateHeap creates the appropriate heap based on whether we're using
167+
// a distance metric (lower is better) or similarity metric (higher is better).
168+
func buildCandidateHeap[T c.Float](array []persistentHeapElement[T], isSimilarityMetric bool) candidateHeap[T] {
169+
if isSimilarityMetric {
170+
return &maxHeapWrapper[T]{h: buildMaxPersistentHeapByInit(array)}
171+
}
172+
return &minHeapWrapper[T]{h: buildMinPersistentHeapByInit(array)}
173+
}

0 commit comments

Comments
 (0)