Skip to content

Commit 396ad18

Browse files
murphyjacob4murphyjacob4
authored
feat: Add support for clustered Memorystore products. (#94)
* Add support for clustered Memorystore products. Signed-off-by: Jacob Murphy <jkmurphy@google.com> * Run black formatter. Signed-off-by: Jacob Murphy <jkmurphy@google.com> * Fix documentation per PR review comments. Signed-off-by: Jacob Murphy <jkmurphy@google.com> * Run isort to fix import order. Signed-off-by: Jacob Murphy <jkmurphy@google.com> * Remove duplicated get_env_var helper. Signed-off-by: Jacob Murphy <jkmurphy@google.com> * Remove trailing newline Signed-off-by: Jacob Murphy <jkmurphy@google.com> --------- Signed-off-by: Jacob Murphy <jkmurphy@google.com>
1 parent f0364ed commit 396ad18

15 files changed

Lines changed: 451 additions & 138 deletions

‎DEVELOPER.md‎

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -53,6 +53,7 @@ substitutions:
5353
_VERSION: "3.8"
5454
_WORKER_POOL: ""
5555
_REDIS_URL: ""
56+
_REDIS_CLUSTER_URL: ""
5657
```
5758

5859
Use `gcloud builds triggers import --source=trigger.yaml` to create triggers via the command line
@@ -68,7 +69,7 @@ Use `gcloud builds triggers import --source=trigger.yaml` to create triggers via
6869
* Run integration test:
6970

7071
```bash
71-
gcloud builds submit --config integration.cloudbuild.yaml --region us-central1 --substitutions=_WORKER_POOL="path-to-worker-pool",_REDIS_URL="redis://..."
72+
gcloud builds submit --config integration.cloudbuild.yaml --region us-central1 --substitutions=_WORKER_POOL="path-to-worker-pool",_REDIS_URL="redis://...",_REDIS_CLUSTER_URL="redis://..."
7273
```
7374

7475
#### Trigger

‎README.rst‎

Lines changed: 15 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -84,6 +84,21 @@ Use a vector store to store embedded data and perform vector search.
8484
embeddings=embeddings_service
8585
)
8686
87+
You can also use a clustered client:
88+
89+
.. code-block:: python
90+
91+
from langchain_google_memorystore_redis import RedisVectorStore
92+
93+
redis_client = redis.cluster.RedisCluster.from_url("redis://127.0.0.1:6379")
94+
95+
embeddings_service = VertexAIEmbeddings(model_name="textembedding-gecko@003")
96+
vectorstore = RedisVectorStore(
97+
client=redis_client,
98+
index_name="my_vector_index",
99+
embeddings=embeddings_service
100+
)
101+
87102
See the full `Vector Store`_ tutorial.
88103

89104
.. _`Vector Store`: https://github.com/googleapis/langchain-google-memorystore-redis-python/blob/main/docs/vector_store.ipynb

‎docs/chat_message_history.ipynb‎

Lines changed: 50 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -10,7 +10,11 @@
1010
"\n",
1111
"> [Google Memorystore for Redis](https://cloud.google.com/memorystore/docs/redis/memorystore-for-redis-overview) is a fully-managed service that is powered by the Redis in-memory data store to build application caches that provide sub-millisecond data access. Extend your database application to build AI-powered experiences leveraging Memorystore for Redis's Langchain integrations.\n",
1212
"\n",
13-
"This notebook goes over how to use [Memorystore for Redis](https://cloud.google.com/memorystore/docs/redis/memorystore-for-redis-overview) to store chat message history with the `MemorystoreChatMessageHistory` class.\n",
13+
"> [Google Memorystore for Redis Cluster](https://cloud.google.com/memorystore/docs/cluster/memorystore-for-redis-cluster-overview) is a fully-managed service that is powered by the Redis in-memory data store. Applications running on Google Cloud can achieve extreme performance by leveraging the highly scalable, available, secure Redis service without the burden of managing complex Redis deployments.\n",
14+
"\n",
15+
"> [Google Memorystore for Valkey](https://cloud.google.com/memorystore/docs/valkey/product-overview) is a fully-managed Valkey Cluster service for Google Cloud, and is 100% compatible with the Google Memorystore for Redis Cluster LangChain integration.\n",
16+
"\n",
17+
"This notebook goes over how to use [Memorystore for Redis](https://cloud.google.com/memorystore/docs/redis/memorystore-for-redis-overview), [Memorystore for Redis Cluster](https://cloud.google.com/memorystore/docs/cluster/memorystore-for-redis-cluster-overview) or [Memorystore for Valkey](https://cloud.google.com/memorystore/docs/valkey/product-overview) to store chat message history with the `MemorystoreChatMessageHistory` class.\n",
1418
"\n",
1519
"Learn more about the package on [GitHub](https://github.com/googleapis/langchain-google-memorystore-redis-python/).\n",
1620
"\n",
@@ -26,8 +30,13 @@
2630
"To run this notebook, you will need to do the following:\n",
2731
"\n",
2832
"* [Create a Google Cloud Project](https://developers.google.com/workspace/guides/create-project)\n",
29-
"* [Enable the Memorystore for Redis API](https://console.cloud.google.com/flows/enableapi?apiid=redis.googleapis.com)\n",
30-
"* [Create a Memorystore for Redis instance](https://cloud.google.com/memorystore/docs/redis/create-instance-console). Ensure that the version is greater than or equal to 5.0.\n",
33+
"* Enable relevant APIs for your use case\n",
34+
" * [Enable the Memorystore for Redis API](https://console.cloud.google.com/flows/enableapi?apiid=redis.googleapis.com)\n",
35+
" * [Enable the Memorystore for Valkey API](https://console.cloud.google.com/flows/enableapi?apiid=memorystore.googleapis.com)\n",
36+
"* Create Memorystore resources for your use case. You may do any of the following:\n",
37+
" * [Create a Memorystore for Redis instance](https://cloud.google.com/memorystore/docs/redis/create-instance-console). Ensure that the version is greater than or equal to 7.2.\n",
38+
" * [Create a Memorystore for Redis cluster](https://cloud.google.com/memorystore/docs/cluster/instance-provisioning-vpc)\n",
39+
" * [Create a Memorystore for Valkey instance](https://cloud.google.com/memorystore/docs/valkey/instance-provisioning-vpc)\n",
3140
"\n",
3241
"After confirmed access to database in the runtime environment of this notebook, filling the following values and run the cell before running example scripts."
3342
]
@@ -156,6 +165,13 @@
156165
"1. `session_id` - Each chat message history object must have a unique session ID. If the session ID already has messages stored in Redis, they will can be retrieved."
157166
]
158167
},
168+
{
169+
"cell_type": "markdown",
170+
"metadata": {},
171+
"source": [
172+
"#### Method 1: Standalone Redis Client"
173+
]
174+
},
159175
{
160176
"cell_type": "code",
161177
"execution_count": null,
@@ -173,6 +189,37 @@
173189
"message_history = MemorystoreChatMessageHistory(redis_client, session_id=\"session1\")"
174190
]
175191
},
192+
{
193+
"cell_type": "code",
194+
"execution_count": null,
195+
"metadata": {},
196+
"outputs": [],
197+
"source": [
198+
"message_history.messages"
199+
]
200+
},
201+
{
202+
"cell_type": "markdown",
203+
"metadata": {},
204+
"source": [
205+
"#### Method 2: Clustered Redis Client"
206+
]
207+
},
208+
{
209+
"cell_type": "code",
210+
"execution_count": null,
211+
"metadata": {},
212+
"outputs": [],
213+
"source": [
214+
"import redis\n",
215+
"from langchain_google_memorystore_redis import MemorystoreChatMessageHistory\n",
216+
"\n",
217+
"# Connect to a Memorystore for Redis instance\n",
218+
"redis_client = redis.cluster.RedisCluster.from_url(\"redis://127.0.0.1:6379\")\n",
219+
"\n",
220+
"message_history = MemorystoreChatMessageHistory(redis_client, session_id=\"session1\")"
221+
]
222+
},
176223
{
177224
"cell_type": "code",
178225
"execution_count": null,

‎docs/document_loader.ipynb‎

Lines changed: 61 additions & 11 deletions
Original file line numberDiff line numberDiff line change
@@ -8,9 +8,15 @@
88
"source": [
99
"# Google Memorystore for Redis\n",
1010
"\n",
11-
"> [Google Memorystore for Redis](https://cloud.google.com/memorystore/docs/redis/memorystore-for-redis-overview) is a fully-managed service that is powered by the Redis in-memory data store to build application caches that provide sub-millisecond data access. Extend your database application to build AI-powered experiences leveraging Memorystore for Redis's Langchain integrations.\n",
11+
"> [Google Memorystore for Redis](https://cloud.google.com/memorystore/docs/redis/memorystore-for-redis-overview) is a fully-managed service that is powered by the Redis in-memory data store to build application caches that provide sub-millisecond data access.\n",
1212
"\n",
13-
"This notebook goes over how to use [Memorystore for Redis](https://cloud.google.com/memorystore/docs/redis/memorystore-for-redis-overview) to [save, load and delete langchain documents](https://python.langchain.com/docs/modules/data_connection/document_loaders/) with `MemorystoreDocumentLoader` and `MemorystoreDocumentSaver`.\n",
13+
"> [Google Memorystore for Redis Cluster](https://cloud.google.com/memorystore/docs/cluster/memorystore-for-redis-cluster-overview) is a fully-managed service that is powered by the Redis in-memory data store. Applications running on Google Cloud can achieve extreme performance by leveraging the highly scalable, available, secure Redis service without the burden of managing complex Redis deployments.\n",
14+
"\n",
15+
"> [Google Memorystore for Valkey](https://cloud.google.com/memorystore/docs/valkey/product-overview) is a fully-managed Valkey Cluster service for Google Cloud, and is 100% compatible with the Google Memorystore for Redis Cluster LangChain integration.\n",
16+
"\n",
17+
"Extend your database application to build AI-powered experiences leveraging Memorystore's Langchain integrations.\n",
18+
"\n",
19+
"This notebook goes over how to use [Memorystore for Redis](https://cloud.google.com/memorystore/docs/redis/memorystore-for-redis-overview), [Memorystore for Redis Cluster](https://cloud.google.com/memorystore/docs/cluster/memorystore-for-redis-cluster-overview) or [Memorystore for Valkey](https://cloud.google.com/memorystore/docs/valkey/product-overview) to [save, load and delete langchain documents](https://python.langchain.com/docs/modules/data_connection/document_loaders/) with `MemorystoreDocumentLoader` and `MemorystoreDocumentSaver`.\n",
1420
"\n",
1521
"Learn more about the package on [GitHub](https://github.com/googleapis/langchain-google-memorystore-redis-python/).\n",
1622
"\n",
@@ -26,8 +32,13 @@
2632
"To run this notebook, you will need to do the following:\n",
2733
"\n",
2834
"* [Create a Google Cloud Project](https://developers.google.com/workspace/guides/create-project)\n",
29-
"* [Enable the Memorystore for Redis API](https://console.cloud.google.com/flows/enableapi?apiid=redis.googleapis.com)\n",
30-
"* [Create a Memorystore for Redis instance](https://cloud.google.com/memorystore/docs/redis/create-instance-console). Ensure that the version is greater than or equal to 5.0.\n",
35+
"* Enable relevant APIs for your use case\n",
36+
" * [Enable the Memorystore for Redis API](https://console.cloud.google.com/flows/enableapi?apiid=redis.googleapis.com)\n",
37+
" * [Enable the Memorystore for Valkey API](https://console.cloud.google.com/flows/enableapi?apiid=memorystore.googleapis.com)\n",
38+
"* Create Memorystore resources for your use case. You may do any of the following:\n",
39+
" * [Create a Memorystore for Redis instance](https://cloud.google.com/memorystore/docs/redis/create-instance-console). Ensure that the version is greater than or equal to 7.2.\n",
40+
" * [Create a Memorystore for Redis cluster](https://cloud.google.com/memorystore/docs/cluster/instance-provisioning-vpc)\n",
41+
" * [Create a Memorystore for Valkey instance](https://cloud.google.com/memorystore/docs/valkey/instance-provisioning-vpc)\n",
3142
"\n",
3243
"After confirmed access to database in the runtime environment of this notebook, filling the following values and run the cell before running example scripts."
3344
]
@@ -141,6 +152,49 @@
141152
"## Basic Usage"
142153
]
143154
},
155+
{
156+
"cell_type": "markdown",
157+
"metadata": {},
158+
"source": [
159+
"### Setup a Redis connection"
160+
]
161+
},
162+
{
163+
"cell_type": "markdown",
164+
"metadata": {},
165+
"source": [
166+
"#### Method 1: Standalone Redis Client"
167+
]
168+
},
169+
{
170+
"cell_type": "code",
171+
"execution_count": null,
172+
"metadata": {},
173+
"outputs": [],
174+
"source": [
175+
"import redis\n",
176+
"\n",
177+
"redis_client = redis.from_url(ENDPOINT)"
178+
]
179+
},
180+
{
181+
"cell_type": "markdown",
182+
"metadata": {},
183+
"source": [
184+
"#### Method 2: Clustered Redis Client"
185+
]
186+
},
187+
{
188+
"cell_type": "code",
189+
"execution_count": null,
190+
"metadata": {},
191+
"outputs": [],
192+
"source": [
193+
"import redis\n",
194+
"\n",
195+
"redis_client = redis.cluster.RedisCluster.from_url(ENDPOINT)"
196+
]
197+
},
144198
{
145199
"cell_type": "markdown",
146200
"metadata": {},
@@ -149,7 +203,7 @@
149203
"\n",
150204
"Save langchain documents with `MemorystoreDocumentSaver.add_documents(<documents>)`. To initialize `MemorystoreDocumentSaver` class you need to provide 2 things:\n",
151205
"\n",
152-
"1. `client` - A `redis.Redis` client object.\n",
206+
"1. `client` - A `redis.Redis` or `redis.cluster.RedisCluster` client object.\n",
153207
"1. `key_prefix` - A prefix for the keys to store Documents in Redis.\n",
154208
"\n",
155209
"The Documents will be stored into randomly generated keys with the specified prefix of `key_prefix`. Alternatively, you can designate the suffixes of the keys by specifying `ids` in the `add_documents` method."
@@ -161,7 +215,6 @@
161215
"metadata": {},
162216
"outputs": [],
163217
"source": [
164-
"import redis\n",
165218
"from langchain_core.documents import Document\n",
166219
"from langchain_google_memorystore_redis import MemorystoreDocumentSaver\n",
167220
"\n",
@@ -181,7 +234,6 @@
181234
"]\n",
182235
"doc_ids = [f\"{i}\" for i in range(len(test_docs))]\n",
183236
"\n",
184-
"redis_client = redis.from_url(ENDPOINT)\n",
185237
"saver = MemorystoreDocumentSaver(\n",
186238
" client=redis_client,\n",
187239
" key_prefix=KEY_PREFIX,\n",
@@ -198,11 +250,11 @@
198250
"source": [
199251
"### Load documents\n",
200252
"\n",
201-
"Initialize a loader that loads all documents stored in the Memorystore for Redis instance with a specific prefix.\n",
253+
"Initialize a loader that loads all documents stored in the Memorystore instance with a specific prefix.\n",
202254
"\n",
203255
"Load langchain documents with `MemorystoreDocumentLoader.load()` or `MemorystoreDocumentLoader.lazy_load()`. `lazy_load` returns a generator that only queries database during the iteration. To initialize `MemorystoreDocumentLoader` class you need to provide:\n",
204256
"\n",
205-
"1. `client` - A `redis.Redis` client object.\n",
257+
"1. `client` - A `redis.Redis` ro `redis.cluster.RedisCluster` client object.\n",
206258
"1. `key_prefix` - A prefix for the keys to store Documents in Redis."
207259
]
208260
},
@@ -214,10 +266,8 @@
214266
},
215267
"outputs": [],
216268
"source": [
217-
"import redis\n",
218269
"from langchain_google_memorystore_redis import MemorystoreDocumentLoader\n",
219270
"\n",
220-
"redis_client = redis.from_url(ENDPOINT)\n",
221271
"loader = MemorystoreDocumentLoader(\n",
222272
" client=redis_client,\n",
223273
" key_prefix=KEY_PREFIX,\n",

‎docs/vector_store.ipynb‎

Lines changed: 54 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -7,9 +7,13 @@
77
"source": [
88
"# Google Memorystore for Redis\n",
99
"\n",
10-
"> [Google Memorystore for Redis](https://cloud.google.com/memorystore/docs/redis/memorystore-for-redis-overview) is a fully-managed service that is powered by the Redis in-memory data store to build application caches that provide sub-millisecond data access. Extend your database application to build AI-powered experiences leveraging Memorystore for Redis's Langchain integrations.\n",
10+
"> [Google Memorystore for Redis](https://cloud.google.com/memorystore/docs/redis/memorystore-for-redis-overview) is a fully-managed service that is powered by the Redis in-memory data store to build application caches that provide sub-millisecond data access.\n",
1111
"\n",
12-
"This notebook goes over how to use [Memorystore for Redis](https://cloud.google.com/memorystore/docs/redis/memorystore-for-redis-overview) to store vector embeddings with the `MemorystoreVectorStore` class.\n",
12+
"> [Google Memorystore for Redis Cluster](https://cloud.google.com/memorystore/docs/cluster/memorystore-for-redis-cluster-overview) is a fully-managed service that is powered by the Redis in-memory data store. Applications running on Google Cloud can achieve extreme performance by leveraging the highly scalable, available, secure Redis service without the burden of managing complex Redis deployments.\n",
13+
"\n",
14+
"> [Google Memorystore for Valkey](https://cloud.google.com/memorystore/docs/valkey/product-overview) is a fully-managed Valkey Cluster service for Google Cloud, and is 100% compatible with the Google Memorystore for Redis Cluster LangChain integration.\n",
15+
"\n",
16+
"This notebook goes over how to use [Memorystore for Redis](https://cloud.google.com/memorystore/docs/redis/memorystore-for-redis-overview), [Memorystore for Redis Cluster](https://cloud.google.com/memorystore/docs/cluster/memorystore-for-redis-cluster-overview) or [Memorystore for Valkey](https://cloud.google.com/memorystore/docs/valkey/product-overview) to store vector embeddings with the `MemorystoreVectorStore` class.\n",
1317
"\n",
1418
"Learn more about the package on [GitHub](https://github.com/googleapis/langchain-google-memorystore-redis-python/).\n",
1519
"\n",
@@ -33,8 +37,15 @@
3337
"To run this notebook, you will need to do the following:\n",
3438
"\n",
3539
"* [Create a Google Cloud Project](https://developers.google.com/workspace/guides/create-project)\n",
36-
"* [Enable the Memorystore for Redis API](https://console.cloud.google.com/flows/enableapi?apiid=redis.googleapis.com)\n",
37-
"* [Create a Memorystore for Redis instance](https://cloud.google.com/memorystore/docs/redis/create-instance-console). Ensure that the version is greater than or equal to 7.2."
40+
"* Enable relevant APIs for your use case\n",
41+
" * [Enable the Memorystore for Redis API](https://console.cloud.google.com/flows/enableapi?apiid=redis.googleapis.com)\n",
42+
" * [Enable the Memorystore for Valkey API](https://console.cloud.google.com/flows/enableapi?apiid=memorystore.googleapis.com)\n",
43+
"* Create Memorystore resources for your use case. You may do any of the following:\n",
44+
" * [Create a Memorystore for Redis instance](https://cloud.google.com/memorystore/docs/redis/create-instance-console). Ensure that the version is greater than or equal to 7.2.\n",
45+
" * [Create a Memorystore for Redis cluster](https://cloud.google.com/memorystore/docs/cluster/instance-provisioning-vpc)\n",
46+
" * [Create a Memorystore for Valkey instance](https://cloud.google.com/memorystore/docs/valkey/instance-provisioning-vpc)\n",
47+
"\n",
48+
"After confirmed access to database in the runtime environment of this notebook, filling the following values and run the cell before running example scripts."
3849
]
3950
},
4051
{
@@ -143,6 +154,13 @@
143154
"### Initialize a Vector Index"
144155
]
145156
},
157+
{
158+
"cell_type": "markdown",
159+
"metadata": {},
160+
"source": [
161+
"#### Method 1: Standalone Redis Client"
162+
]
163+
},
146164
{
147165
"cell_type": "code",
148166
"execution_count": 13,
@@ -168,6 +186,38 @@
168186
"RedisVectorStore.init_index(client=redis_client, index_config=index_config)"
169187
]
170188
},
189+
{
190+
"cell_type": "markdown",
191+
"metadata": {},
192+
"source": [
193+
"#### Method 2: Clustered Redis Client"
194+
]
195+
},
196+
{
197+
"cell_type": "code",
198+
"execution_count": null,
199+
"metadata": {},
200+
"outputs": [],
201+
"source": [
202+
"import redis\n",
203+
"from langchain_google_memorystore_redis import (\n",
204+
" DistanceStrategy,\n",
205+
" HNSWConfig,\n",
206+
" RedisVectorStore,\n",
207+
")\n",
208+
"\n",
209+
"# Connect to a Memorystore for Redis Cluster instance\n",
210+
"redis_client = redis.cluster.RedisCluster.from_url(\"redis://127.0.0.1:6379\")\n",
211+
"\n",
212+
"# Configure HNSW index with descriptive parameters\n",
213+
"index_config = HNSWConfig(\n",
214+
" name=\"my_vector_index\", distance_strategy=DistanceStrategy.COSINE, vector_size=128\n",
215+
")\n",
216+
"\n",
217+
"# Initialize/create the vector store index\n",
218+
"RedisVectorStore.init_index(client=redis_client, index_config=index_config)"
219+
]
220+
},
171221
{
172222
"attachments": {},
173223
"cell_type": "markdown",
@@ -225,7 +275,6 @@
225275
"from langchain_community.embeddings.fake import FakeEmbeddings\n",
226276
"\n",
227277
"embeddings = FakeEmbeddings(size=128)\n",
228-
"redis_client = redis.from_url(\"redis://127.0.0.1:6379\")\n",
229278
"rvs = RedisVectorStore.from_documents(\n",
230279
" docs, embedding=embeddings, client=redis_client, index_name=\"my_vector_index\"\n",
231280
")"

‎integration.cloudbuild.yaml‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -36,6 +36,7 @@ steps:
3636
]
3737
env:
3838
- "REDIS_URL=$_REDIS_URL"
39+
- "REDIS_CLUSTER_URL=$_REDIS_CLUSTER_URL"
3940

4041
substitutions:
4142
_VERSION: "3.8"

0 commit comments

Comments
 (0)