From fbf2ca2aa987b791818425f4ac067eeec3fee74c Mon Sep 17 00:00:00 2001 From: Ewa Szyszka <79250896+ESzyszka@users.noreply.github.com> Date: Mon, 27 Jul 2026 16:12:18 +0200 Subject: [PATCH] Add files via upload --- Beginner-course/Module1.ipynb | 317 ++++++++++++++++++++++++ Beginner-course/Module2.ipynb | 265 ++++++++++++++++++++ Beginner-course/Module3.ipynb | 393 ++++++++++++++++++++++++++++++ Beginner-course/Module4.ipynb | 443 ++++++++++++++++++++++++++++++++++ Beginner-course/Module5.ipynb | 358 +++++++++++++++++++++++++++ 5 files changed, 1776 insertions(+) create mode 100644 Beginner-course/Module1.ipynb create mode 100644 Beginner-course/Module2.ipynb create mode 100644 Beginner-course/Module3.ipynb create mode 100644 Beginner-course/Module4.ipynb create mode 100644 Beginner-course/Module5.ipynb diff --git a/Beginner-course/Module1.ipynb b/Beginner-course/Module1.ipynb new file mode 100644 index 0000000..f9db5c4 --- /dev/null +++ b/Beginner-course/Module1.ipynb @@ -0,0 +1,317 @@ +{ + "nbformat": 4, + "nbformat_minor": 0, + "metadata": { + "colab": { + "provenance": [] + }, + "kernelspec": { + "display_name": "Python 3", + "name": "python3" + }, + "language_info": { + "name": "python" + } + }, + "cells": [ + { + "cell_type": "markdown", + "source": "# Module 1: Let's Understand Search\n\n## What you will do\n\n1. Watch keyword search miss documents that plainly answer the query.\n2. Turn text into a vector and look at what actually comes back.\n3. Score meaning with cosine similarity, first by hand, then with the library helper.\n4. Re-run the Section 1 failures semantically and watch them pass.\n5. Meet the SKU problem, where similarity on its own is not enough.\n\n**Tip:** every cell below is already run, so you can read this straight through. Run it yourself and you should see the same numbers.\n\nCompanion notebook to the [Module 1 lesson](https://qdrant.tech/course/beginners/module-1/).", + "metadata": { + "id": "i0VpEBOWfbZA" + } + }, + { + "cell_type": "markdown", + "source": "## Setup\n\nModule 1 stays deliberately small: one embedding model, no vector database yet. `sentence-transformers` gives us `all-MiniLM-L6-v2`, a 384-dimension model fast enough to run on a free Colab CPU. Qdrant arrives in Module 2.", + "metadata": { + "id": "VaBSo63bbH6x" + } + }, + { + "cell_type": "code", + "metadata": { + "id": "nAbnBEoonCrb" + }, + "execution_count": null, + "source": "!pip install -q sentence-transformers", + "outputs": [] + }, + { + "cell_type": "markdown", + "source": "## 1. The Problem: Why Keyword Search Struggles\n\nTraditional search matches exact words. If the query string appears in the document it is a hit, and if it does not it is a miss, however close the meaning is.\n\nHere is the whole of keyword search, in three lines.", + "metadata": { + "id": "SS6AZPsK20FK" + } + }, + { + "cell_type": "code", + "metadata": { + "id": "cpXzkIRPxBFW", + "colab": { + "base_uri": "https://localhost:8080/" + }, + "outputId": "86700bf9" + }, + "execution_count": null, + "source": "documents = [\n \"automobile maintenance guide\",\n \"affordable airfare to New York\",\n \"apple harvest season guide\",\n]\n\ndef keyword_search(query, docs):\n \"\"\"All of keyword search: does the query string appear verbatim?\"\"\"\n return [d for d in docs if query.lower() in d.lower()]\n\nfor query in [\"car repair\", \"cheap flights NYC\", \"Apple stock\"]:\n hits = keyword_search(query, documents)\n print(f\"{query!r:22} -> {hits if hits else 'NO MATCH'}\")", + "outputs": [ + { + "output_type": "stream", + "name": "stdout", + "text": "'car repair' -> NO MATCH\n'cheap flights NYC' -> NO MATCH\n'Apple stock' -> NO MATCH\n" + } + ] + }, + { + "cell_type": "markdown", + "source": "Three queries. Three documents that answer them. Zero hits.\n\nNothing is broken. The engine is doing exactly what it was asked: comparing characters. That single limitation shows up in four ways.\n\n- **Synonyms.** \"car\" and \"automobile\" mean the same thing and share no letters.\n- **Paraphrasing.** \"cheap flights\" and \"affordable airfare\" are the same intent in different words.\n- **Polysemy.** \"apple\" is a company, a fruit, and a record label. Characters cannot tell you which one was meant.\n- **Word order.** \"dog bites man\" and \"man bites dog\" contain identical words.\n\nImprovements to keyword search, inverted indexes for speed, BM25 for ranking, stemming, and fuzzy matching, all raise the ceiling without moving it. They still work on words. You could hard-code that \"car\" means \"automobile\", then do it again for the next pair, and the next. You cannot hard-code a language.\n\nSo semantic search changes the question. Not *does this document contain the same words*, but *does this document mean the same thing*.", + "metadata": { + "id": "ykF8VH1oF7JC" + } + }, + { + "cell_type": "markdown", + "source": "## 2. How It Works: Embeddings\n\nAn embedding model takes a piece of text and returns a fixed-length list of floating-point numbers. Similar meanings produce vectors that sit close together in that space.", + "metadata": { + "id": "qH7aWY2TYGIA" + } + }, + { + "cell_type": "code", + "metadata": { + "id": "d1JmFEwvHMLC", + "colab": { + "base_uri": "https://localhost:8080/" + }, + "outputId": "0329dccd" + }, + "execution_count": null, + "source": "from sentence_transformers import SentenceTransformer, util\n\nmodel = SentenceTransformer(\"all-MiniLM-L6-v2\")\n\nquery_vec = model.encode(\"car repair\")\ndoc_vec = model.encode(\"automobile maintenance\")\n\nprint(\"dimensions:\", len(query_vec))\nprint(\"first five:\", query_vec[:5])", + "outputs": [ + { + "output_type": "stream", + "name": "stdout", + "text": "dimensions: 384\nfirst five: [-0.12456685 0.03955904 0.08732592 -0.02361241 -0.07054088]\n" + } + ] + }, + { + "cell_type": "markdown", + "source": "384 numbers. That is the vector.\n\nDo not go looking for the dimension that stores colour, or urgency. Nobody designed these. The model learned them from data, and meaning lives in the combination of all 384 rather than in any single one. A 384-dimension model has 384 such aspects, and only the whole set together encodes meaning.", + "metadata": { + "id": "p0PfbaUpgMve" + } + }, + { + "cell_type": "markdown", + "source": "## 3. Comparing Meaning: Cosine Similarity\n\nCosine similarity measures the angle between two vectors and ignores their length. Vectors pointing the same way score near 1. Unrelated directions score near 0.\n\nThe formula is short enough to write out, which is worth doing once so the library call stops being magic.", + "metadata": { + "id": "jGkqSCtDbxAl" + } + }, + { + "cell_type": "code", + "metadata": { + "id": "g3FmLYbbjSjR", + "colab": { + "base_uri": "https://localhost:8080/" + }, + "outputId": "a83fc83a" + }, + "execution_count": null, + "source": "import numpy as np\n\ndef cosine_by_hand(a, b):\n return float(np.dot(a, b) / (np.linalg.norm(a) * np.linalg.norm(b)))\n\nby_hand = cosine_by_hand(query_vec, doc_vec)\nby_library = util.cos_sim(query_vec, doc_vec).item()\n\nprint(f\"by hand: {by_hand:.6f}\")\nprint(f\"cos_sim: {by_library:.6f}\")\nprint(f\"agree: {np.isclose(by_hand, by_library)}\")", + "outputs": [ + { + "output_type": "stream", + "name": "stdout", + "text": "by hand: 0.733420\ncos_sim: 0.733420\nagree: True\n" + } + ] + }, + { + "cell_type": "markdown", + "source": "Roughly 0.73 for two phrases with no words in common. That number is the whole point of the module.\n\nNow three pairs at once: a synonym pair, a paraphrase pair, and a pair that has nothing to do with each other.", + "metadata": { + "id": "yJtNds82s1t6" + } + }, + { + "cell_type": "code", + "metadata": { + "id": "Uzpa2cY7Jk18", + "colab": { + "base_uri": "https://localhost:8080/" + }, + "outputId": "b8653a63" + }, + "execution_count": null, + "source": "pairs = [\n (\"car repair\", \"automobile maintenance\"), # synonyms\n (\"cheap flights to New York\", \"affordable airfare to NYC\"), # paraphrase\n (\"cheap flights to New York\", \"best pizza in Chicago\"), # unrelated\n]\n\nfor query, document in pairs:\n score = util.cos_sim(model.encode(query), model.encode(document)).item()\n print(f\"{score:.3f} | {query!r} vs {document!r}\")", + "outputs": [ + { + "output_type": "stream", + "name": "stdout", + "text": "0.733 | 'car repair' vs 'automobile maintenance'\n0.821 | 'cheap flights to New York' vs 'affordable airfare to NYC'\n0.332 | 'cheap flights to New York' vs 'best pizza in Chicago'\n" + } + ] + }, + { + "cell_type": "markdown", + "source": "The first two pairs share little or no vocabulary and both score high. That is the synonym and the paraphrase that keyword search missed in Section 1. The third scores far lower, so the model is separating meaning rather than matching surface words.\n\nWhich means we can go back and re-run those three failures.", + "metadata": { + "id": "gPsEu23bmu9W" + } + }, + { + "cell_type": "code", + "metadata": { + "id": "vnPZ5v2QE9oS", + "colab": { + "base_uri": "https://localhost:8080/" + }, + "outputId": "12e3c9d5" + }, + "execution_count": null, + "source": "# Same three documents as Section 1, plus a second \"apple\" sense so the\n# polysemy case has somewhere correct to land.\ndocuments = [\n \"automobile maintenance guide\",\n \"affordable airfare to New York\",\n \"apple harvest season guide\",\n \"technology shares closed higher today\",\n]\n\ndoc_vecs = model.encode(documents)\n\nfor query in [\"car repair\", \"cheap flights NYC\", \"Apple stock\"]:\n sims = util.cos_sim(model.encode(query), doc_vecs)[0]\n ranked = sorted(zip(sims.tolist(), documents), reverse=True)\n print(f\"{query!r}\")\n for score, doc in ranked:\n print(f\" {score:6.3f} {doc}\")\n print()", + "outputs": [ + { + "output_type": "stream", + "name": "stdout", + "text": "'car repair'\n 0.609 automobile maintenance guide\n 0.065 apple harvest season guide\n 0.060 technology shares closed higher today\n -0.032 affordable airfare to New York\n\n'cheap flights NYC'\n 0.800 affordable airfare to New York\n 0.117 apple harvest season guide\n 0.080 technology shares closed higher today\n -0.021 automobile maintenance guide\n\n'Apple stock'\n 0.401 technology shares closed higher today\n 0.374 apple harvest season guide\n 0.145 affordable airfare to New York\n 0.048 automobile maintenance guide\n\n" + } + ] + }, + { + "cell_type": "markdown", + "source": "Every query now finds its document, and each one was a total miss under keyword search.\n\nTwo things in that output are worth pausing on.\n\n**Negative scores exist.** `car repair` against `affordable airfare to New York` comes out slightly below zero. Cosine similarity ranges from -1 to 1 in principle, where -1 means the vectors point in opposite directions. With normalized text models these stay small and near zero, so treat a value like -0.03 as \"unrelated\" rather than \"opposite\".\n\n**The polysemy win is narrow.** `Apple stock` prefers the technology document over the fruit document, but only by about 0.03. The model got the sense right, and it was close. Hold on to that, because Section 4 is built on exactly this kind of narrow gap.", + "metadata": { + "id": "vs1L73iIgce1" + } + }, + { + "cell_type": "markdown", + "source": "### Where dense search struggles too\n\nThe four failure modes were framed as keyword problems, but word order is not solved by switching to embeddings. Compare a sentence with its meaning reversed, and with a genuine paraphrase.", + "metadata": { + "id": "WZvMWxJnfq3I" + } + }, + { + "cell_type": "code", + "metadata": { + "id": "5CrWZaPfh2co", + "colab": { + "base_uri": "https://localhost:8080/" + }, + "outputId": "f6123b1e" + }, + "execution_count": null, + "source": "reversed_pair = (\"dog bites man\", \"man bites dog\")\nparaphrase_pair = (\"dog bites man\", \"a canine attacked a person\")\n\nfor a, b in [reversed_pair, paraphrase_pair]:\n print(f\"{util.cos_sim(model.encode(a), model.encode(b)).item():.3f} | {a!r} vs {b!r}\")", + "outputs": [ + { + "output_type": "stream", + "name": "stdout", + "text": "0.907 | 'dog bites man' vs 'man bites dog'\n0.570 | 'dog bites man' vs 'a canine attacked a person'\n" + } + ] + }, + { + "cell_type": "markdown", + "source": "The reversed sentence scores higher than the correct paraphrase.\n\nThe model rates *opposite meaning, identical words* as more similar than *same meaning, different words*. Embeddings shifted the problem rather than removing it, and that is worth knowing before you trust similarity with anything important.", + "metadata": { + "id": "x84spDgOca8s" + } + }, + { + "cell_type": "markdown", + "source": "## 4. Why Similarity Alone Is Not Enough\n\nHere is the failure that costs real money. A user types an exact product code and wants that product.", + "metadata": { + "id": "KBtd488gnM8H" + } + }, + { + "cell_type": "code", + "metadata": { + "id": "QPqHtpfr8BON", + "colab": { + "base_uri": "https://localhost:8080/" + }, + "outputId": "f6455de4" + }, + "execution_count": null, + "source": "skus = [\"SKU-48290\", \"SKU-48291\", \"SKU-48292\", \"SKU-48293\"]\ncatalog = [f\"{sku} replacement part, ships in 2 days\" for sku in skus]\n\nquery = model.encode(\"SKU-48291 issue\")\nsims = util.cos_sim(query, model.encode(catalog))[0]\n\nscored = sorted(zip(sims.tolist(), skus), reverse=True)\nfor score, sku in scored:\n marker = \" <-- the one they asked for\" if sku == \"SKU-48291\" else \"\"\n print(f\"{score:.3f} {sku}{marker}\")\n\nprint()\nprint(f\"spread between best and worst: {scored[0][0] - scored[-1][0]:.3f}\")", + "outputs": [ + { + "output_type": "stream", + "name": "stdout", + "text": "0.626 SKU-48290\n0.624 SKU-48291 <-- the one they asked for\n0.616 SKU-48293\n0.610 SKU-48292\n\nspread between best and worst: 0.016\n" + } + ] + }, + { + "cell_type": "markdown", + "source": "The requested SKU is not first, and look at the spread: all four codes land within a couple of hundredths of each other.\n\nThat is not a bug in the model. Those strings are nearly identical, so their vectors are nearly identical, and the model has no idea that the final digit is the only part that matters. Semantically they are the same thing. Commercially, three of the four are the wrong product.\n\nNo amount of model tuning fixes this, because the model is not wrong. What is missing is a hard constraint.", + "metadata": { + "id": "uHJKpbpeIeeN" + } + }, + { + "cell_type": "code", + "metadata": { + "id": "sXE1jFXe6kW9", + "colab": { + "base_uri": "https://localhost:8080/" + }, + "outputId": "d518e4e2" + }, + "execution_count": null, + "source": "TARGET = \"SKU-48291\"\n\n# An exact constraint on a metadata field, not a similarity score\nfiltered = [(score, sku) for score, sku in scored if sku == TARGET]\n\nfor score, sku in filtered:\n print(f\"{score:.3f} {sku}\")\n\nprint()\nprint(\"In Qdrant that constraint is a payload filter, evaluated while the search runs:\")\nprint(' must=[FieldCondition(key=\"sku\", match=MatchValue(value=\"SKU-48291\"))]')\nprint(\"You will write exactly that in Module 2.\")", + "outputs": [ + { + "output_type": "stream", + "name": "stdout", + "text": "0.624 SKU-48291\n\nIn Qdrant that constraint is a payload filter, evaluated while the search runs:\n must=[FieldCondition(key=\"sku\", match=MatchValue(value=\"SKU-48291\"))]\nYou will write exactly that in Module 2.\n" + } + ] + }, + { + "cell_type": "markdown", + "source": "### Key insight\n\nDense similarity finds the neighbourhood. Filters, exact matches, and payload constraints find the right point inside it. Production search needs both, which is why real systems are hybrid: dense retrieval for meaning, sparse retrieval for exact tokens, and filters constraining both.", + "metadata": { + "id": "4bt3X40UQlAj" + } + }, + { + "cell_type": "markdown", + "source": "## Your turn\n\nOne exercise, then Module 2.\n\nSwap in your own polysemy case below. Score `\"apple stock\"` against both senses and see which wins, then try a word your own domain overloads: *charge*, *bank*, *lead*, *plant*. Does the model pick the sense you meant?", + "metadata": { + "id": "SPpd7YIH0Cyu" + } + }, + { + "cell_type": "code", + "metadata": { + "id": "pbOzJdwihqZy", + "colab": { + "base_uri": "https://localhost:8080/" + }, + "outputId": "693088a7" + }, + "execution_count": null, + "source": "candidates = [\n \"shares of a technology company\",\n \"a crisp red fruit\",\n]\n\nfor candidate in candidates:\n score = util.cos_sim(model.encode(\"apple stock\"), model.encode(candidate)).item()\n print(f\"{score:.3f} | 'apple stock' vs {candidate!r}\")", + "outputs": [ + { + "output_type": "stream", + "name": "stdout", + "text": "0.480 | 'apple stock' vs 'shares of a technology company'\n0.225 | 'apple stock' vs 'a crisp red fruit'\n" + } + ] + }, + { + "cell_type": "markdown", + "source": "## What's next: Module 2\n\n- What a vector is, and why it has hundreds of dimensions\n- How similarity works under the hood, and when it fails\n- Your first Qdrant collection: points, payloads, and your first query\n\n[Continue to Module 2](https://qdrant.tech/course/beginners/module-2/)", + "metadata": { + "id": "QToEDnzI44rO" + } + } + ] +} diff --git a/Beginner-course/Module2.ipynb b/Beginner-course/Module2.ipynb new file mode 100644 index 0000000..b4a8509 --- /dev/null +++ b/Beginner-course/Module2.ipynb @@ -0,0 +1,265 @@ +{ + "nbformat": 4, + "nbformat_minor": 0, + "metadata": { + "colab": { + "provenance": [] + }, + "kernelspec": { + "display_name": "Python 3", + "name": "python3" + }, + "language_info": { + "name": "python" + } + }, + "cells": [ + { + "cell_type": "markdown", + "source": "# Module 2: First Principles of Vector Search\n\n## What you will do\n\n1. Turn text into a vector and look at what a collection needs to know about it.\n2. Create your first Qdrant collection.\n3. Upsert points with vectors and payloads.\n4. Run a top-K query, then constrain it with a payload filter.\n5. See why the payload index has to exist before you ingest.\n\n**Tip:** every cell below is already run, so you can read it straight through. It uses Qdrant in local mode, so there is nothing to sign up for and no API key to paste.\n\nCompanion notebook to the [Module 2 lesson](https://qdrant.tech/course/beginners/module-2/).", + "metadata": { + "id": "ujOeHwdFcAef" + } + }, + { + "cell_type": "markdown", + "source": "## Setup\n\n`qdrant-client[fastembed]` bundles local embedding models. Passing a `models.Document` lets the client embed text for us before upload and at query time, so we never handle raw vectors by hand. The model is `all-MiniLM-L6-v2`, the same one from Module 1.", + "metadata": { + "id": "AZhnM6Jy8c1r" + } + }, + { + "cell_type": "code", + "metadata": { + "id": "ihtYlKNxHddm" + }, + "execution_count": null, + "source": "!pip install -q \"qdrant-client[fastembed]\" ", + "outputs": [] + }, + { + "cell_type": "markdown", + "source": "## 1. What a Collection Needs to Know\n\nA collection is a container for points, and it is declared up front with two things it can never change silently: how many dimensions each vector has, and which distance metric compares them.\n\nThose two numbers are not stylistic. Get the size wrong and every upsert fails. Get the metric wrong and your results are quietly worse rather than broken.", + "metadata": { + "id": "QAtKCsXRpJG2" + } + }, + { + "cell_type": "code", + "metadata": { + "id": "Tr8hzUjEcPVJ", + "colab": { + "base_uri": "https://localhost:8080/" + }, + "outputId": "e5b997df" + }, + "execution_count": null, + "source": "from qdrant_client import QdrantClient, models\n\nMODEL = \"sentence-transformers/all-MiniLM-L6-v2\"\nVECTOR_SIZE = 384 # fixed by the model\nDISTANCE = models.Distance.COSINE # the default for text embeddings\n\n# Local mode: an in-process Qdrant, ideal for notebooks and CI.\nclient = QdrantClient(\":memory:\")\n\n# On Qdrant Cloud you would swap in:\n# client = QdrantClient(url=\"https://YOUR-CLUSTER.cloud.qdrant.io:6333\",\n# api_key=\"YOUR_API_KEY\")\n\nclient.create_collection(\n collection_name=\"articles\",\n vectors_config=models.VectorParams(size=VECTOR_SIZE, distance=DISTANCE),\n)\n\nprint(client.get_collection(\"articles\").config.params.vectors)", + "outputs": [ + { + "output_type": "stream", + "name": "stdout", + "text": "size=384 distance= hnsw_config=None quantization_config=None on_disk=None datatype=None multivector_config=None\n" + } + ] + }, + { + "cell_type": "markdown", + "source": "## 2. Index What You Will Filter, Before You Ingest\n\nThe collection is empty, and this is the moment to declare the payload fields you plan to filter on.\n\nThe ordering matters. Qdrant adds filter-aware edges to its vector index based on indexed payload values, and it can only add them for indexes that already exist when that index is built. Create a payload index after ingesting and you have to rebuild the vector index to get the benefit.\n\nLocal mode ignores payload indexes entirely, so the call below is a no-op here and will emit a warning. Write it anyway: it is the habit that transfers to a real server, and Module 4 goes into why.", + "metadata": { + "id": "DPdRO9YrxPbC" + } + }, + { + "cell_type": "code", + "metadata": { + "id": "khdVipy2eBI2", + "colab": { + "base_uri": "https://localhost:8080/" + }, + "outputId": "dd465ef2" + }, + "execution_count": null, + "source": "import warnings\n\nwith warnings.catch_warnings():\n warnings.simplefilter(\"ignore\") # local mode warns that indexes do nothing here\n client.create_payload_index(\n collection_name=\"articles\",\n field_name=\"category\",\n field_schema=models.PayloadSchemaType.KEYWORD,\n )\n\nprint(\"payload index declared on 'category'\")", + "outputs": [ + { + "output_type": "stream", + "name": "stdout", + "text": "payload index declared on 'category'\n" + } + ] + }, + { + "cell_type": "markdown", + "source": "## 3. Points: Vector Plus Payload\n\nA point is an id, one or more vectors, and a payload. The vector is what gets searched. The payload is everything you want back, or want to filter on, and it is ordinary JSON.\n\nNote what goes in the payload below. `title` is there to be returned to the user. `category` is there to be filtered. `published` is there to be sorted or range-filtered later. None of them affect similarity.", + "metadata": { + "id": "kooEKqazwJ7Q" + } + }, + { + "cell_type": "code", + "metadata": { + "id": "FMPd3W2XyygN", + "colab": { + "base_uri": "https://localhost:8080/" + }, + "outputId": "01325002" + }, + "execution_count": null, + "source": "documents = [\n {\"id\": 1, \"title\": \"Car repair guide\", \"category\": \"automotive\", \"published\": 2023},\n {\"id\": 2, \"title\": \"Automobile maintenance 101\", \"category\": \"automotive\", \"published\": 2024},\n {\"id\": 3, \"title\": \"How to cook pasta\", \"category\": \"food\", \"published\": 2024},\n {\"id\": 4, \"title\": \"Best pizza in Chicago\", \"category\": \"food\", \"published\": 2022},\n]\n\nclient.upload_points(\n collection_name=\"articles\",\n points=[\n models.PointStruct(\n id=doc[\"id\"],\n vector=models.Document(text=doc[\"title\"], model=MODEL),\n payload=doc,\n )\n for doc in documents\n ],\n)\n\nprint(\"points in collection:\", client.count(\"articles\").count)", + "outputs": [ + { + "output_type": "stream", + "name": "stdout", + "text": "points in collection: 4\n" + } + ] + }, + { + "cell_type": "markdown", + "source": "## 4. Your First Query\n\nSame model at query time as at ingestion time. This is the rule that breaks the most beginner pipelines: two different models produce two different vector spaces, and nothing will warn you, you will just get nonsense rankings.\n\nThe query below shares no words with any stored title.", + "metadata": { + "id": "gwb2Mjpvwh09" + } + }, + { + "cell_type": "code", + "metadata": { + "id": "CDfgvqZkb6Gj", + "colab": { + "base_uri": "https://localhost:8080/" + }, + "outputId": "808fdb45" + }, + "execution_count": null, + "source": "results = client.query_points(\n collection_name=\"articles\",\n query=models.Document(text=\"fixing my vehicle\", model=MODEL),\n limit=4,\n)\n\nfor r in results.points:\n print(f\"{r.score:.3f} {r.payload['title']:30} ({r.payload['category']})\")", + "outputs": [ + { + "output_type": "stream", + "name": "stdout", + "text": "0.699 Car repair guide (automotive)\n0.565 Automobile maintenance 101 (automotive)\n0.045 Best pizza in Chicago (food)\n0.032 How to cook pasta (food)\n" + } + ] + }, + { + "cell_type": "markdown", + "source": "Both automotive articles come back above both food articles, and the query contained neither the word \"car\" nor \"automobile\". That is the Module 1 embedding doing its job, now with storage and ranking around it.\n\nAlso worth noticing: every result carries its payload back. You did not have to look anything up in a second database.", + "metadata": { + "id": "kVHWunY90XYT" + } + }, + { + "cell_type": "markdown", + "source": "## 5. Filtering\n\nA filter is a hard constraint, not a ranking hint. It is evaluated while the search runs, so excluded points never occupy a slot in your top-K.\n\nRun the same query twice, once unfiltered and once scoped to `food`, and watch the entire result set change rather than just reorder.", + "metadata": { + "id": "oGwbbrqQ7B69" + } + }, + { + "cell_type": "code", + "metadata": { + "id": "7wnommM30DOX", + "colab": { + "base_uri": "https://localhost:8080/" + }, + "outputId": "1aecc72c" + }, + "execution_count": null, + "source": "from qdrant_client.models import Filter, FieldCondition, MatchValue\n\ndef search(text, query_filter=None, limit=4):\n return client.query_points(\n collection_name=\"articles\",\n query=models.Document(text=text, model=MODEL),\n query_filter=query_filter,\n limit=limit,\n ).points\n\nfood_only = Filter(must=[FieldCondition(key=\"category\", match=MatchValue(value=\"food\"))])\n\nprint(\"UNFILTERED 'fixing my vehicle':\")\nfor r in search(\"fixing my vehicle\"):\n print(f\" {r.score:.3f} {r.payload['title']:30} ({r.payload['category']})\")\n\nprint()\nprint(\"FILTERED to category=food:\")\nfor r in search(\"fixing my vehicle\", query_filter=food_only):\n print(f\" {r.score:.3f} {r.payload['title']:30} ({r.payload['category']})\")", + "outputs": [ + { + "output_type": "stream", + "name": "stdout", + "text": "UNFILTERED 'fixing my vehicle':\n 0.699 Car repair guide (automotive)\n 0.565 Automobile maintenance 101 (automotive)\n 0.045 Best pizza in Chicago (food)\n 0.032 How to cook pasta (food)\n\nFILTERED to category=food:\n 0.045 Best pizza in Chicago (food)\n 0.032 How to cook pasta (food)\n" + } + ] + }, + { + "cell_type": "markdown", + "source": "The filtered run returns only food articles, and their scores are unchanged from the unfiltered run. The filter did not rescore anything. It removed candidates.\n\nThat distinction matters later: because filtering happens during retrieval rather than after it, a filtered search still returns a full top-K of valid results instead of a short list of leftovers.", + "metadata": { + "id": "uX8yy6Skhj4Y" + } + }, + { + "cell_type": "markdown", + "source": "## 6. Range Filters and Combining Conditions\n\n`must` is AND, `should` is OR, and `must_not` excludes. Numeric and date fields take ranges. They compose in one filter object.", + "metadata": { + "id": "jZ8OvIia8OG5" + } + }, + { + "cell_type": "code", + "metadata": { + "id": "A2ZnpsoKqzi4", + "colab": { + "base_uri": "https://localhost:8080/" + }, + "outputId": "579e6e88" + }, + "execution_count": null, + "source": "from qdrant_client.models import Range\n\nrecent_automotive = Filter(\n must=[\n FieldCondition(key=\"category\", match=MatchValue(value=\"automotive\")),\n FieldCondition(key=\"published\", range=Range(gte=2024)),\n ]\n)\n\nprint(\"automotive AND published >= 2024:\")\nfor r in search(\"fixing my vehicle\", query_filter=recent_automotive):\n print(f\" {r.score:.3f} {r.payload['title']:30} ({r.payload['published']})\")", + "outputs": [ + { + "output_type": "stream", + "name": "stdout", + "text": "automotive AND published >= 2024:\n 0.565 Automobile maintenance 101 (2024)\n" + } + ] + }, + { + "cell_type": "markdown", + "source": "One result, because only one point satisfies both conditions. Note that `published` was never indexed, and in local mode that costs nothing. On a real server it would work but scan, and on Qdrant Cloud with strict mode on it would be rejected outright. Every field you filter on needs an index.", + "metadata": { + "id": "GbBlaXkDSIuG" + } + }, + { + "cell_type": "markdown", + "source": "## 7. Similarity Under the Hood\n\nQdrant does not compare your query against every stored vector. It walks an HNSW graph, a layered structure where each layer is a sparser shortcut over the one below, so search starts coarse and refines.\n\nTwo consequences worth carrying forward.\n\nIt is **approximate**. HNSW trades a small amount of recall for a large amount of speed, so a top-K is very likely, not certainly, the true nearest neighbours.\n\nIt is **not always used**. Below a size threshold, scanning every vector is genuinely faster, so Qdrant does that instead. That is why a tiny notebook collection like this one returns exact results, and why timing measurements here tell you nothing about production.", + "metadata": { + "id": "IWgIprVFIV4B" + } + }, + { + "cell_type": "code", + "metadata": { + "id": "LFFQCHD6R26q", + "colab": { + "base_uri": "https://localhost:8080/" + }, + "outputId": "8fd21713" + }, + "execution_count": null, + "source": "info = client.get_collection(\"articles\")\nprint(\"points: \", info.points_count)\nprint(\"vector size: \", info.config.params.vectors.size)\nprint(\"distance: \", info.config.params.vectors.distance)\nprint(\"hnsw m: \", info.config.hnsw_config.m)\nprint(\"hnsw ef_construct: \", info.config.hnsw_config.ef_construct)\nprint(\"full_scan_threshold:\", info.config.hnsw_config.full_scan_threshold, \"(KB)\")", + "outputs": [ + { + "output_type": "stream", + "name": "stdout", + "text": "points: 4\nvector size: 384\ndistance: Cosine\nhnsw m: 16\nhnsw ef_construct: 100\nfull_scan_threshold: 10000 (KB)\n" + } + ] + }, + { + "cell_type": "markdown", + "source": "`m` is how many edges each node keeps, `ef_construct` is how hard the graph works while being built, and `full_scan_threshold` is the size below which Qdrant skips the graph. You will tune these in later work; for now the point is that they are collection-level settings you inherit by default.", + "metadata": { + "id": "ePW3jSOi27n7" + } + }, + { + "cell_type": "markdown", + "source": "## Your turn\n\nTwo changes to try.\n\nAdd a point in a third category and rerun the unfiltered query. Does it land where you expect relative to the existing four?\n\nThen change `DISTANCE` to `models.Distance.EUCLID`, recreate the collection, and re-ingest. Compare the scores to the cosine run. They will not be on the same scale, which is exactly why the metric is declared once per collection rather than per query.", + "metadata": { + "id": "yE9ZkAFvmtSb" + } + }, + { + "cell_type": "markdown", + "source": "## What's next: Module 3\n\n- Where dense-only search fails: exact codes, model numbers, and SKUs\n- Sparse vectors, BM25, and the inverted index\n- Hybrid search: running dense and sparse together and fusing the results\n\n[Continue to Module 3](https://qdrant.tech/course/beginners/module-3/)", + "metadata": { + "id": "IBbuMFd9W8gq" + } + } + ] +} diff --git a/Beginner-course/Module3.ipynb b/Beginner-course/Module3.ipynb new file mode 100644 index 0000000..d84bb0b --- /dev/null +++ b/Beginner-course/Module3.ipynb @@ -0,0 +1,393 @@ +{ + "nbformat": 4, + "nbformat_minor": 0, + "metadata": { + "colab": { + "provenance": [] + }, + "kernelspec": { + "display_name": "Python 3", + "name": "python3" + }, + "language_info": { + "name": "python" + } + }, + "cells": [ + { + "cell_type": "markdown", + "source": "# Module 3: Sparse vs Dense vs Hybrid Search\n\n## What you will do\n\n1. See the gap that dense-only search leaves on exact names and codes.\n2. Compare the two families of search, dense (meaning) and sparse (exact tokens).\n3. Build a hybrid collection and fuse both by rank.\n4. Run the dense vs hybrid experiment and watch the ranking change.\n5. Prove where a filter has to go in a hybrid query, by breaking it on purpose.\n\n**Tip for Colab:** run cells top to bottom. The first install downloads small embedding models, so it takes a minute.\n\nCompanion notebook to the [Module 3 lesson](https://qdrant.tech/course/beginners/module-3/).", + "metadata": { + "id": "ujOeHwdFcAef" + } + }, + { + "cell_type": "markdown", + "source": "## Setup\n\n`qdrant-client[fastembed]` bundles local embedding models. Passing a `models.Document` lets the client embed text for us before upload and at query time, so we never manage the vectors by hand.", + "metadata": { + "id": "AZhnM6Jy8c1r" + } + }, + { + "cell_type": "code", + "metadata": { + "id": "ihtYlKNxHddm" + }, + "execution_count": null, + "source": "!pip install -q \"qdrant-client[fastembed]\" ", + "outputs": [] + }, + { + "cell_type": "markdown", + "source": "## 1. Where We Left Off\n\nIn Module 2 you built a full pipeline: raw text to vector to store to top-K query. Dense-only retrieval is great for semantic and contextual search, but it struggles on precise product names and model numbers.\n\nTake the query `iPhone 15`. The user wants exactly this product: no synonyms, no paraphrasing. Dense-only search tends to return the whole product line, because \"iPhone 14\", \"iPhone 15\", and \"iPhone 15 Pro Max\" sit close together in embedding space. IDs, codes, and specific model names need exact matching, not semantic neighborhood. That is the gap sparse search fills, and we will reproduce it live in Section 4.", + "metadata": { + "id": "QAtKCsXRpJG2" + } + }, + { + "cell_type": "markdown", + "source": "## 2. The Two Families of Search\n\n### Dense search (semantic)\n\nA dense vector has a small, fixed number of dimensions (for example 384), and every dimension holds a value. Two texts with similar meaning produce vectors that are close in space, even if they share no words. That is why `car repair` sits near `automobile maintenance`.\n\n### Sparse search (keyword based)\n\nSparse vectors are token based. Each dimension maps to a token, and only the tokens that actually appear carry a non-zero value. A vocabulary can be tens of thousands of tokens, but a given text activates only the handful it contains, so sparse vectors are stored as two parallel arrays: the `indices` of the non-zero dimensions and the `values` at those positions.\n\n```python\nsparse_vector = {\n \"indices\": [142, 9325, 44001], # token IDs: 'nike', 'pegasus', '40'\n \"values\": [2.3, 1.2, 0.8], # weight per token\n}\n```\n\nSparse similarity in Qdrant is always the dot product. There is no metric to choose, unlike the dense side where you pick Cosine, Dot, or Euclidean.\n\n#### Sparse models: BM25, SPLADE, miniCOIL\n\n| Model | How it assigns weights | Notes |\n|-------|------------------------|-------|\n| BM25 | Statistical: term frequency and inverse document frequency, no training | Classic, fast, interpretable. Scores tokens exactly as written. FastEmbed handle `Qdrant/bm25`. |\n| SPLADE | Neural: a transformer expands text with related terms and weights them | Captures some synonymy while staying sparse. More compute than BM25. |\n| miniCOIL | Neural, contextualized term weighting on BM25's exact vocabulary | Context aware exact match without full expansion cost. FastEmbed handle `Qdrant/minicoil-v1`. |\n\nminiCOIL is Qdrant's recommendation for new projects. We use BM25 here because it needs no model inference at all, which keeps this notebook fast and every score traceable by hand.\n\n#### How sparse is indexed\n\nQdrant uses an inverted index: for every token it keeps a posting list of every point where that token has a non-zero weight. A query only walks the posting lists for tokens it contains, skipping every point that shares none. HNSW (Module 2) is approximate, but the sparse index is exact.\n\n### Key insight\n\nDense = meaning. Sparse = exact matching. Neither is complete alone. Every real query carries both semantic intent (what the user means) and exact constraints (what the user needs precisely).", + "metadata": { + "id": "Tr8hzUjEcPVJ" + } + }, + { + "cell_type": "markdown", + "source": "## 3. Hybrid Search: Dense + Sparse\n\nHybrid search runs dense and sparse retrieval in the same request, then fuses the two ranked candidate lists into one result set: semantic understanding with exact-match precision. Payload filters constrain both retrievers while they run, which Section 7 covers.\n\n**Reciprocal Rank Fusion (RRF)** merges the dense list and the sparse list using each candidate's *position* in the two lists, not its raw score. A document ranked high by both retrievers rises to the top. Because it ignores raw scores, RRF is robust to the fact that dense and sparse scores live on completely different scales.", + "metadata": { + "id": "2tRKJC06DPdR" + } + }, + { + "cell_type": "markdown", + "source": "## 4. Setting Up Hybrid Search in Qdrant\n\n### Step 1: create a hybrid collection\n\nDeclare both a dense and a sparse vector config on one collection. Every point will carry both.\n\nTwo details in the cell below are easy to skip and expensive to skip. The sparse config needs `modifier=models.Modifier.IDF`, because BM25-style vectors store only term frequency and Qdrant applies the inverse-document-frequency half of the formula at query time. Without it you are not scoring BM25. And the payload index on `in_stock` is declared before any data is ingested, which is the ordering Module 4 explains.", + "metadata": { + "id": "O9YrxPbCkhdV" + } + }, + { + "cell_type": "code", + "metadata": { + "id": "ipy2eBI2Y1rz", + "colab": { + "base_uri": "https://localhost:8080/" + }, + "outputId": "5ef22337" + }, + "execution_count": null, + "source": "import warnings\nfrom qdrant_client import QdrantClient, models\n\nDENSE_MODEL = \"sentence-transformers/all-MiniLM-L6-v2\" # 384-dim dense embeddings\nSPARSE_MODEL = \"Qdrant/bm25\" # exact-token sparse\n\nclient = QdrantClient(\":memory:\")\n\nclient.create_collection(\n collection_name=\"products\",\n vectors_config={\n \"dense\": models.VectorParams(size=384, distance=models.Distance.COSINE),\n },\n sparse_vectors_config={\n \"sparse\": models.SparseVectorParams(\n modifier=models.Modifier.IDF # required for BM25 scoring\n ),\n },\n)\n\nwith warnings.catch_warnings():\n warnings.simplefilter(\"ignore\") # local mode warns that indexes do nothing here\n client.create_payload_index(\n collection_name=\"products\",\n field_name=\"in_stock\",\n field_schema=models.PayloadSchemaType.BOOL,\n )\n\nprint(\"Hybrid collection 'products' created, with in_stock indexed.\")", + "outputs": [ + { + "output_type": "stream", + "name": "stdout", + "text": "Hybrid collection 'products' created, with in_stock indexed.\n" + } + ] + }, + { + "cell_type": "markdown", + "source": "### Step 2: insert points with both vectors\n\nEach point carries a dense embedding and a sparse vector. We pass a `models.Document` and name the model. The client embeds it locally with FastEmbed before upload. The catalog below mixes a phone product line (to reproduce the `iPhone 15` problem) with running shoes (for the Nike example) and includes an exact SKU.", + "metadata": { + "id": "KqazwJ7QFMPd" + } + }, + { + "cell_type": "code", + "metadata": { + "id": "3W2XyygNdemk", + "colab": { + "base_uri": "https://localhost:8080/" + }, + "outputId": "5002150d" + }, + "execution_count": null, + "source": "catalog = [\n {\"id\": 1, \"name\": \"iPhone 14\", \"sku\": \"APL-IP14\", \"price\": 699, \"in_stock\": True},\n {\"id\": 2, \"name\": \"iPhone 15\", \"sku\": \"APL-IP15\", \"price\": 799, \"in_stock\": True},\n {\"id\": 3, \"name\": \"iPhone 15 Pro Max\", \"sku\": \"APL-IP15PM\", \"price\": 1199, \"in_stock\": True},\n {\"id\": 4, \"name\": \"iPhone 13 mini\", \"sku\": \"APL-IP13M\", \"price\": 599, \"in_stock\": False},\n {\"id\": 5, \"name\": \"Nike Pegasus 40 running shoes\", \"sku\": \"NK-PEG40\", \"price\": 130, \"in_stock\": True},\n {\"id\": 6, \"name\": \"Nike Pegasus 39 running shoes\", \"sku\": \"NK-PEG39\", \"price\": 110, \"in_stock\": True},\n {\"id\": 7, \"name\": \"Adidas Ultraboost running shoes\", \"sku\": \"AD-UB22\", \"price\": 180, \"in_stock\": True},\n {\"id\": 8, \"name\": \"Widget assembly part SKU-48291\", \"sku\": \"SKU-48291\", \"price\": 12, \"in_stock\": True},\n]\n\nclient.upload_points(\n collection_name=\"products\",\n points=[\n models.PointStruct(\n id=item[\"id\"],\n vector={\n \"dense\": models.Document(text=item[\"name\"], model=DENSE_MODEL),\n \"sparse\": models.Document(text=item[\"name\"] + \" \" + item[\"sku\"], model=SPARSE_MODEL),\n },\n payload=item,\n )\n for item in catalog\n ],\n)\nprint(\"Upserted\", client.count(\"products\").count, \"products.\")", + "outputs": [ + { + "output_type": "stream", + "name": "stdout", + "text": "Upserted 8 products.\n" + } + ] + }, + { + "cell_type": "markdown", + "source": "### Step 3: three ways to search the same collection\n\nA small helper prints results compactly. Then the same query runs three ways: dense-only, sparse-only, and hybrid.\n\nLook closely at `hybrid()`. The filter is passed into **each** `Prefetch`, not to `query_points` as a top-level `query_filter`. That placement is the whole lesson of Section 7, and we break it on purpose later to show what goes wrong.", + "metadata": { + "id": "Mjpvwh09CDfg" + } + }, + { + "cell_type": "code", + "metadata": { + "id": "vqZkb6GjHbG8" + }, + "execution_count": null, + "source": "def show(title, points):\n print(title)\n for r in points:\n print(f\" [{r.score:.4f}] {r.payload['name']} (sku={r.payload['sku']}, in_stock={r.payload['in_stock']})\")\n print()\n\ndef dense_only(text, query_filter=None, limit=5):\n return client.query_points(\n \"products\",\n query=models.Document(text=text, model=DENSE_MODEL),\n using=\"dense\",\n query_filter=query_filter, # no prefetch here, so top level is correct\n limit=limit,\n ).points\n\ndef sparse_only(text, query_filter=None, limit=5):\n return client.query_points(\n \"products\",\n query=models.Document(text=text, model=SPARSE_MODEL),\n using=\"sparse\",\n query_filter=query_filter, # no prefetch here either\n limit=limit,\n ).points\n\ndef hybrid(text, query_filter=None, limit=5, fusion=\"rrf\"):\n # The filter goes INSIDE each prefetch, so both retrievers only ever\n # consider valid points. See Section 7 for why the top level is wrong here.\n fuse = (models.RrfQuery(rrf=models.Rrf()) if fusion == \"rrf\"\n else models.FusionQuery(fusion=models.Fusion.DBSF))\n return client.query_points(\n \"products\",\n prefetch=[\n models.Prefetch(query=models.Document(text=text, model=DENSE_MODEL),\n using=\"dense\", filter=query_filter, limit=20),\n models.Prefetch(query=models.Document(text=text, model=SPARSE_MODEL),\n using=\"sparse\", filter=query_filter, limit=20),\n ],\n query=fuse,\n limit=limit,\n ).points", + "outputs": [] + }, + { + "cell_type": "markdown", + "source": "The dense-only run reproduces the problem from Section 1: semantically similar phones cluster together, and the exact model the user typed does not reliably sit on top. Sparse-only, by contrast, locks onto the literal tokens.", + "metadata": { + "id": "1RqwkVHWunY9" + } + }, + { + "cell_type": "code", + "metadata": { + "id": "0XYToGwbbrqQ", + "colab": { + "base_uri": "https://localhost:8080/" + }, + "outputId": "f7eff533" + }, + "execution_count": null, + "source": "show(\"DENSE-ONLY 'iPhone 15' (semantic neighborhood, exact model can drift):\", dense_only(\"iPhone 15\"))\nshow(\"SPARSE-ONLY 'iPhone 15' (exact tokens win):\", sparse_only(\"iPhone 15\"))", + "outputs": [ + { + "output_type": "stream", + "name": "stdout", + "text": "DENSE-ONLY 'iPhone 15' (semantic neighborhood, exact model can drift):\n [1.0000] iPhone 15 (sku=APL-IP15, in_stock=True)\n [0.8760] iPhone 14 (sku=APL-IP14, in_stock=True)\n [0.8149] iPhone 15 Pro Max (sku=APL-IP15PM, in_stock=True)\n [0.6801] iPhone 13 mini (sku=APL-IP13M, in_stock=False)\n [0.1764] Nike Pegasus 39 running shoes (sku=NK-PEG39, in_stock=True)\n\nSPARSE-ONLY 'iPhone 15' (exact tokens win):\n [3.3050] iPhone 15 (sku=APL-IP15, in_stock=True)\n [3.2874] iPhone 15 Pro Max (sku=APL-IP15PM, in_stock=True)\n [1.1605] iPhone 14 (sku=APL-IP14, in_stock=True)\n [1.1574] iPhone 13 mini (sku=APL-IP13M, in_stock=False)\n\n" + } + ] + }, + { + "cell_type": "markdown", + "source": "The clearest case is an exact code. A dense model has never really \"seen\" `SKU-48291` as a meaningful concept, so it drifts. Sparse matches the literal token exactly.", + "metadata": { + "id": "mmM30DOXfO4W" + } + }, + { + "cell_type": "code", + "metadata": { + "id": "UDlWuX8yy6Sk", + "colab": { + "base_uri": "https://localhost:8080/" + }, + "outputId": "22ec2dfa" + }, + "execution_count": null, + "source": "show(\"DENSE-ONLY 'SKU-48291' (drifts):\", dense_only(\"SKU-48291\"))\nshow(\"SPARSE-ONLY 'SKU-48291' (exact hit):\", sparse_only(\"SKU-48291\"))", + "outputs": [ + { + "output_type": "stream", + "name": "stdout", + "text": "DENSE-ONLY 'SKU-48291' (drifts):\n [0.4544] Widget assembly part SKU-48291 (sku=SKU-48291, in_stock=True)\n [0.2397] Nike Pegasus 39 running shoes (sku=NK-PEG39, in_stock=True)\n [0.1948] Nike Pegasus 40 running shoes (sku=NK-PEG40, in_stock=True)\n [0.1403] Adidas Ultraboost running shoes (sku=AD-UB22, in_stock=True)\n [0.1266] iPhone 15 (sku=APL-IP15, in_stock=True)\n\nSPARSE-ONLY 'SKU-48291' (exact hit):\n [6.7829] Widget assembly part SKU-48291 (sku=SKU-48291, in_stock=True)\n\n" + } + ] + }, + { + "cell_type": "markdown", + "source": "Now hybrid. Both prefetches run as part of one request and return up to 20 candidates each. RRF merges them by rank, then `limit` takes the top results. Because each prefetch carries the filter, out-of-stock products never enter either candidate set.", + "metadata": { + "id": "vIia8OG5A2Zn" + } + }, + { + "cell_type": "code", + "metadata": { + "id": "psoKqzi4vCK4", + "colab": { + "base_uri": "https://localhost:8080/" + }, + "outputId": "6e888072" + }, + "execution_count": null, + "source": "from qdrant_client.models import Filter, FieldCondition, MatchValue\n\nin_stock_filter = Filter(must=[FieldCondition(key=\"in_stock\", match=MatchValue(value=True))])\n\nshow(\"HYBRID 'Nike Pegasus 40 size 10' (semantic + exact, in stock only):\",\n hybrid(\"Nike Pegasus 40 size 10\", query_filter=in_stock_filter))", + "outputs": [ + { + "output_type": "stream", + "name": "stdout", + "text": "HYBRID 'Nike Pegasus 40 size 10' (semantic + exact, in stock only):\n [1.0000] Nike Pegasus 40 running shoes (sku=NK-PEG40, in_stock=True)\n [0.6667] Nike Pegasus 39 running shoes (sku=NK-PEG39, in_stock=True)\n [0.2500] Adidas Ultraboost running shoes (sku=AD-UB22, in_stock=True)\n [0.2000] iPhone 15 (sku=APL-IP15, in_stock=True)\n [0.1667] iPhone 14 (sku=APL-IP14, in_stock=True)\n\n" + } + ] + }, + { + "cell_type": "markdown", + "source": "### Try it: watch the ranking change\n\nThe experiment from the lesson, side by side. Compare where the exact target lands under dense-only versus hybrid.", + "metadata": { + "id": "aXkDSIuGIWgI" + } + }, + { + "cell_type": "code", + "metadata": { + "id": "prVFIV4BLFFQ", + "colab": { + "base_uri": "https://localhost:8080/" + }, + "outputId": "787fbef4" + }, + "execution_count": null, + "source": "q = \"Nike Pegasus 40\"\nshow(f\"DENSE-ONLY '{q}':\", dense_only(q))\nshow(f\"HYBRID '{q}':\", hybrid(q))", + "outputs": [ + { + "output_type": "stream", + "name": "stdout", + "text": "DENSE-ONLY 'Nike Pegasus 40':\n [0.8815] Nike Pegasus 39 running shoes (sku=NK-PEG39, in_stock=True)\n [0.8713] Nike Pegasus 40 running shoes (sku=NK-PEG40, in_stock=True)\n [0.6043] Adidas Ultraboost running shoes (sku=AD-UB22, in_stock=True)\n [0.2403] iPhone 15 (sku=APL-IP15, in_stock=True)\n [0.2044] iPhone 14 (sku=APL-IP14, in_stock=True)\n\n" + }, + { + "output_type": "stream", + "name": "stdout", + "text": "HYBRID 'Nike Pegasus 40':\n [0.8333] Nike Pegasus 39 running shoes (sku=NK-PEG39, in_stock=True)\n [0.8333] Nike Pegasus 40 running shoes (sku=NK-PEG40, in_stock=True)\n [0.2500] Adidas Ultraboost running shoes (sku=AD-UB22, in_stock=True)\n [0.2000] iPhone 15 (sku=APL-IP15, in_stock=True)\n [0.1667] iPhone 14 (sku=APL-IP14, in_stock=True)\n\n" + } + ] + }, + { + "cell_type": "markdown", + "source": "### Try it: put the filter in the wrong place\n\nNow the part worth doing yourself, because nothing raises an error when you get it wrong.\n\n`hybrid_wrong()` below is identical to `hybrid()` except the filter moves from inside the prefetches to a top-level `query_filter`. Then we mark `iPhone 15` out of stock and run both against an in-stock-only filter. The correct version drops it. The broken version should not.", + "metadata": { + "id": "I60ihBeoePW3" + } + }, + { + "cell_type": "code", + "metadata": { + "id": "jSOi27n7yE9Z", + "colab": { + "base_uri": "https://localhost:8080/" + }, + "outputId": "268535b0" + }, + "execution_count": null, + "source": "def hybrid_wrong(text, query_filter=None, limit=5):\n \"\"\"Same query, filter at the top level instead of inside each prefetch.\"\"\"\n return client.query_points(\n \"products\",\n prefetch=[\n models.Prefetch(query=models.Document(text=text, model=DENSE_MODEL),\n using=\"dense\", limit=20),\n models.Prefetch(query=models.Document(text=text, model=SPARSE_MODEL),\n using=\"sparse\", limit=20),\n ],\n query=models.RrfQuery(rrf=models.Rrf()),\n query_filter=query_filter, # too late: prefetches have already run\n limit=limit,\n ).points\n\nclient.set_payload(\"products\", payload={\"in_stock\": False}, points=[2]) # iPhone 15\n\nshow(\"CORRECT filter inside each prefetch (iPhone 15 is gone):\",\n hybrid(\"iPhone 15\", query_filter=in_stock_filter))\nshow(\"BROKEN filter at the top level (iPhone 15 comes back):\",\n hybrid_wrong(\"iPhone 15\", query_filter=in_stock_filter))", + "outputs": [ + { + "output_type": "stream", + "name": "stdout", + "text": "CORRECT filter inside each prefetch (iPhone 15 is gone):\n [0.8333] iPhone 14 (sku=APL-IP14, in_stock=True)\n [0.8333] iPhone 15 Pro Max (sku=APL-IP15PM, in_stock=True)\n [0.2500] Nike Pegasus 39 running shoes (sku=NK-PEG39, in_stock=True)\n [0.2000] Nike Pegasus 40 running shoes (sku=NK-PEG40, in_stock=True)\n [0.1667] Adidas Ultraboost running shoes (sku=AD-UB22, in_stock=True)\n\n" + }, + { + "output_type": "stream", + "name": "stdout", + "text": "BROKEN filter at the top level (iPhone 15 comes back):\n [1.0000] iPhone 15 (sku=APL-IP15, in_stock=False)\n [0.5833] iPhone 14 (sku=APL-IP14, in_stock=True)\n [0.5833] iPhone 15 Pro Max (sku=APL-IP15PM, in_stock=True)\n [0.4000] iPhone 13 mini (sku=APL-IP13M, in_stock=False)\n [0.1667] Nike Pegasus 39 running shoes (sku=NK-PEG39, in_stock=True)\n\n" + } + ] + }, + { + "cell_type": "markdown", + "source": "There it is. Same filter, same query, one line of difference, and the broken version returns a product that is out of stock, without a warning or an error.\n\nThe reason is execution order. Once a query has prefetches, Qdrant runs those first and applies the main query to their results. A top-level filter therefore never reaches the retrievers: each one searches the whole catalog, and the filter only trims the already-fused list at the end. That is post-filtering, and with a selective filter it can leave you with nothing at all.\n\nThe rule is simple. No prefetch, use `query_filter`. Prefetch, put the filter in every prefetch.", + "metadata": { + "id": "IBbuMFd9W8gq" + } + }, + { + "cell_type": "code", + "metadata": { + "id": "cWqiA4Yqj4JR", + "colab": { + "base_uri": "https://localhost:8080/" + }, + "outputId": "10b61fac" + }, + "execution_count": null, + "source": "client.set_payload(\"products\", payload={\"in_stock\": True}, points=[2]) # restore iPhone 15\nprint(\"iPhone 15 back in stock:\", client.retrieve(\"products\", ids=[2])[0].payload[\"in_stock\"])", + "outputs": [ + { + "output_type": "stream", + "name": "stdout", + "text": "iPhone 15 back in stock: True\n" + } + ] + }, + { + "cell_type": "markdown", + "source": "## 5. Fusion Strategies\n\nOnce both retrievers return candidates, a fusion algorithm merges them into one ranked list. Qdrant supports two.\n\n| Strategy | How it works | When to use it |\n|----------|--------------|----------------|\n| RRF (Reciprocal Rank Fusion) | Combines rankings only, ignores raw scores. Robust, hard to game. | Default. Safe when dense and sparse score scales differ. |\n| DBSF (Distribution-Based Score Fusion) | Normalizes score distributions before merging. Sensitive to relative score gaps. | When score gaps meaningfully encode relevance and both retrievers are well calibrated. |\n\nStart with unweighted RRF, because dense and sparse scores live on different scales and raw-score fusion without normalization is unreliable. RRF also accepts a `k` constant and per-prefetch `weights`, so you can favour the stronger retriever once you have an evaluation set to tune against. Move to DBSF or tuned weights only after measuring, and tune on a different split from the one you measure on.\n\nBoth strategies run below on the same query so you can compare.", + "metadata": { + "id": "f1e1CvI5qiGo" + } + }, + { + "cell_type": "code", + "metadata": { + "id": "gkdmtsVrFlvb", + "colab": { + "base_uri": "https://localhost:8080/" + }, + "outputId": "40b837e1" + }, + "execution_count": null, + "source": "show(\"RRF fusion:\", hybrid(\"Nike Pegasus 40 size 10\", fusion=\"rrf\"))\nshow(\"DBSF fusion:\", hybrid(\"Nike Pegasus 40 size 10\", fusion=\"dbsf\"))", + "outputs": [ + { + "output_type": "stream", + "name": "stdout", + "text": "RRF fusion:\n [1.0000] Nike Pegasus 40 running shoes (sku=NK-PEG40, in_stock=True)\n [0.6667] Nike Pegasus 39 running shoes (sku=NK-PEG39, in_stock=True)\n [0.2500] Adidas Ultraboost running shoes (sku=AD-UB22, in_stock=True)\n [0.2000] iPhone 15 (sku=APL-IP15, in_stock=True)\n [0.1667] iPhone 14 (sku=APL-IP14, in_stock=True)\n\n" + }, + { + "output_type": "stream", + "name": "stdout", + "text": "DBSF fusion:\n [1.3562] Nike Pegasus 40 running shoes (sku=NK-PEG40, in_stock=True)\n [1.1153] Nike Pegasus 39 running shoes (sku=NK-PEG39, in_stock=True)\n [0.6049] Adidas Ultraboost running shoes (sku=AD-UB22, in_stock=True)\n [0.4129] iPhone 15 (sku=APL-IP15, in_stock=True)\n [0.4019] iPhone 14 (sku=APL-IP14, in_stock=True)\n\n" + } + ] + }, + { + "cell_type": "markdown", + "source": "## 6. Beyond Text: Multimodal Search\n\nThe same primitive, embed data then store as a vector then search by similarity, applies to any modality: images (CLIP, SigLIP), video frames, audio fingerprints, or text. Qdrant stores whatever vectors your embedding model produces, and the retrieval mechanics are identical.\n\n**Data to Embedding Model to Vector to Qdrant.** The modality changes, the system does not.\n\n### Named vectors\n\nWhen two representations must be searchable together, store them as named vectors on the same point, then query against whichever one you want. Below we demonstrate the mechanic with two text views of each product, a short `title` and a longer `description`, so it runs with no heavy image models. In a real multimodal system you would swap the `description` model for an image encoder such as CLIP; the collection and query code stay the same shape.", + "metadata": { + "id": "YAEZyFQ8vZRN" + } + }, + { + "cell_type": "code", + "metadata": { + "id": "zvdieTpkf01P", + "colab": { + "base_uri": "https://localhost:8080/" + }, + "outputId": "4347274f" + }, + "execution_count": null, + "source": "client.create_collection(\n collection_name=\"catalog\",\n vectors_config={\n \"title\": models.VectorParams(size=384, distance=models.Distance.COSINE),\n \"description\": models.VectorParams(size=384, distance=models.Distance.COSINE),\n },\n)\n\nclient.upload_points(\n collection_name=\"catalog\",\n points=[\n models.PointStruct(\n id=42,\n vector={\n \"title\": models.Document(text=\"Red Nike running shoe\", model=DENSE_MODEL),\n \"description\": models.Document(text=\"Lightweight breathable trainer for road running in bright red\", model=DENSE_MODEL),\n },\n payload={\"sku\": \"NK-RED-10\", \"price\": 120},\n ),\n models.PointStruct(\n id=43,\n vector={\n \"title\": models.Document(text=\"Blue hiking boot\", model=DENSE_MODEL),\n \"description\": models.Document(text=\"Waterproof ankle support boot for rough mountain trails\", model=DENSE_MODEL),\n },\n payload={\"sku\": \"HK-BLU-9\", \"price\": 150},\n ),\n ],\n)\n\n# Query one named vector; the other is simply not searched on this call.\nres = client.query_points(\n \"catalog\",\n query=models.Document(text=\"shoe for running on pavement\", model=DENSE_MODEL),\n using=\"description\",\n limit=2,\n).points\nfor r in res:\n print(f\"[{r.score:.4f}] {r.payload['sku']}\")", + "outputs": [ + { + "output_type": "stream", + "name": "stdout", + "text": "[0.5492] HK-BLU-9\n[0.4119] NK-RED-10\n" + } + ] + }, + { + "cell_type": "markdown", + "source": "Each named vector is its own space, and vectors from different models are not comparable. Swapping `description` for a CLIP image encoder would look like this, using the same collection and query shape:\n\n```python\n# ingestion\n\"image\": embed_image(product_photo) # CLIP get_image_features\n\n# querying by text against those image vectors\nclient.query_points(\"catalog\", query=embed_text_with_clip(\"red running shoe\"),\n using=\"image\", limit=10)\n```\n\nThe important part is that the query text must go through **CLIP's text encoder**, not the sentence transformer, or it lands in a different space and the scores are meaningless. Module 5 builds this out properly.", + "metadata": { + "id": "8Hp7twaxDFmF" + } + }, + { + "cell_type": "markdown", + "source": "## 7. Filtering Works with Any Retrieval Method\n\nPayload filters are not a hybrid-only feature. The same conditions apply to dense-only, sparse-only, or hybrid retrieval, and they are evaluated as hard constraints *while* the search runs, not as a separate step afterward. Because out-of-scope points never take a slot in your top-K, results stay both relevant and valid: in stock, within permissions, within a date range.\n\nWhat changes between the three is *where* the filter goes, which is what the broken example earlier demonstrated. Dense-only and sparse-only have no prefetch, so `query_filter` is correct. Hybrid has prefetches, so the filter belongs in each one.\n\nThe three calls below apply the identical filter to all three retrieval methods.", + "metadata": { + "id": "aqfycbsoKGUO" + } + }, + { + "cell_type": "code", + "metadata": { + "id": "S2yu9jSNcZ3M", + "colab": { + "base_uri": "https://localhost:8080/" + }, + "outputId": "bc288dcd" + }, + "execution_count": null, + "source": "show(\"DENSE + filter:\", dense_only(\"iPhone\", query_filter=in_stock_filter))\nshow(\"SPARSE + filter:\", sparse_only(\"iPhone\", query_filter=in_stock_filter))\nshow(\"HYBRID + filter:\", hybrid(\"iPhone\", query_filter=in_stock_filter))", + "outputs": [ + { + "output_type": "stream", + "name": "stdout", + "text": "DENSE + filter:\n [0.8036] iPhone 15 (sku=APL-IP15, in_stock=True)\n [0.7914] iPhone 14 (sku=APL-IP14, in_stock=True)\n [0.5714] iPhone 15 Pro Max (sku=APL-IP15PM, in_stock=True)\n [0.1663] Nike Pegasus 39 running shoes (sku=NK-PEG39, in_stock=True)\n [0.1396] Adidas Ultraboost running shoes (sku=AD-UB22, in_stock=True)\n\nSPARSE + filter:\n [1.1605] iPhone 15 (sku=APL-IP15, in_stock=True)\n [1.1605] iPhone 14 (sku=APL-IP14, in_stock=True)\n [1.1543] iPhone 15 Pro Max (sku=APL-IP15PM, in_stock=True)\n\nHYBRID + filter:\n [1.0000] iPhone 15 (sku=APL-IP15, in_stock=True)\n [0.6667] iPhone 14 (sku=APL-IP14, in_stock=True)\n [0.5000] iPhone 15 Pro Max (sku=APL-IP15PM, in_stock=True)\n [0.2000] Nike Pegasus 39 running shoes (sku=NK-PEG39, in_stock=True)\n [0.1667] Adidas Ultraboost running shoes (sku=AD-UB22, in_stock=True)\n\n" + } + ] + }, + { + "cell_type": "markdown", + "source": "## What's next: Module 4\n\n- The five layers of a vector search stack\n- A worked design: a multilingual news search system, decision by decision\n- Filtering in production: how the query planner picks a strategy, and multitenancy\n- The production RAG pipeline, and deployment options\n\n[Continue to Module 4](https://qdrant.tech/course/beginners/module-4/)", + "metadata": { + "id": "K3QQobiwgZIM" + } + } + ] +} diff --git a/Beginner-course/Module4.ipynb b/Beginner-course/Module4.ipynb new file mode 100644 index 0000000..4f53ac2 --- /dev/null +++ b/Beginner-course/Module4.ipynb @@ -0,0 +1,443 @@ +{ + "nbformat": 4, + "nbformat_minor": 0, + "metadata": { + "colab": { + "provenance": [] + }, + "kernelspec": { + "display_name": "Python 3", + "name": "python3" + }, + "language_info": { + "name": "python" + } + }, + "cells": [ + { + "cell_type": "markdown", + "source": "# Module 4: Designing a Vector Search System\n\nVector search systems rarely break at the distance metric. They break because nobody decided what the payload had to hold until a few million documents were already ingested. This module is about the decisions that are cheap to make now and expensive to revisit later.\n\n## What you will do\n\n1. See the five layers of the stack as a mental checklist.\n2. Design and build a multilingual news search system by answering five questions.\n3. Practice filtering the way production systems do it, including the mistake that fails silently.\n4. Assemble a production RAG retrieval pipeline.\n\n**Tip for Colab:** run the cells top to bottom. The first install cell takes a minute because it downloads the embedding models.\n\nCompanion notebook to the [Module 4 lesson](https://qdrant.tech/course/beginners/module-4/).", + "metadata": { + "id": "ujOeHwdFcAef" + } + }, + { + "cell_type": "markdown", + "source": "## Setup\n\nWe install `qdrant-client` with the FastEmbed extra. FastEmbed gives us local, CPU friendly embedding models so the notebook produces real vectors rather than random numbers.", + "metadata": { + "id": "AZhnM6Jy8c1r" + } + }, + { + "cell_type": "code", + "metadata": { + "id": "ihtYlKNxHddm" + }, + "execution_count": null, + "source": "!pip install -q \"qdrant-client[fastembed]\" ", + "outputs": [] + }, + { + "cell_type": "markdown", + "source": "## 1. The Layers of the Stack\n\nEvery vector search system, from a notebook prototype to a deployment serving millions of queries, is built from the same five layers. When something is slow, wrong, or expensive, the first diagnostic question is always the same: which layer is the problem in?\n\n- **Query layer**: turns user intent into a search. Embedding the query, choosing dense vs sparse vs hybrid, fusing results, setting limits.\n- **Indexing layer**: makes that search fast, through the HNSW graph for vectors and payload indexes for the fields you filter on. Get this wrong and results are usually still correct, just slow.\n- **Storage layer**: holds the points themselves, the vectors, payloads, and IDs, across memory and disk.\n- **Knowledge layer**: shapes the data before it ever becomes a vector: chunking, embedding model choice, payload schema. No amount of tuning elsewhere fixes a mistake here. Garbage in, garbage retrieved.\n- **Distribution layer**: spreads the system across more than one machine: sharding, replication, multi node clusters. You will not need it on day one.\n\n### Key insight\n\nEvery design decision belongs to a layer. \"Add a payload index\" is an indexing decision. \"Switch to a multilingual embedding model\" is a knowledge decision. \"Move to three nodes\" is a distribution decision. Once you sort decisions into layers, intimidating architecture diagrams become checklists.", + "metadata": { + "id": "QAtKCsXRpJG2" + } + }, + { + "cell_type": "markdown", + "source": "## 2. Worked Example: Designing a Multilingual News Search System\n\nHere is the brief, the kind you would get on a real project:\n\n> Analysts at a research firm need to search global news that arrives continuously, in many languages. They ask in English (\"port congestion in Southeast Asia\") but expect matches from any language, and they scope every search by country, topic, date range, and source. Some queries name one specific thing, a company ticker or a ship name, that has to match exactly.\n\nFive questions turn it into a design, and each answer is a decision that belongs to one layer.\n\n### Question 1: What do the queries look like?\n\nBoth natural language and exact tokens. \"Port congestion in Southeast Asia\" is semantic intent, which is dense territory. \"MAERSK-B.CO delisting\" is an exact token that dense search will blur into neighboring tickers. That is the SKU problem from Module 3.\n\n**Decision:** hybrid from the start. Two named vectors on every point, one dense and one sparse, with their rankings fused at query time. This is a query layer decision.", + "metadata": { + "id": "Tr8hzUjEcPVJ" + } + }, + { + "cell_type": "code", + "metadata": { + "id": "2tRKJC06DPdR", + "colab": { + "base_uri": "https://localhost:8080/" + }, + "outputId": "afd46a07" + }, + "execution_count": null, + "source": "import warnings\nfrom qdrant_client import QdrantClient, models\nfrom fastembed import TextEmbedding, SparseTextEmbedding\n\n# Local mode: an in-process Qdrant, ideal for prototyping, notebooks and teaching.\n# It is a Python reimplementation rather than the engine: search is exact instead\n# of approximate, and payload indexes have no effect. Everything we write below is\n# still the right shape for a real server.\nclient = QdrantClient(\":memory:\")\n\n# Knowledge layer decision made concrete: a MULTILINGUAL dense model, so an\n# English query can retrieve a Japanese or Vietnamese article with no translation.\n# We use a small multilingual model to keep the Colab download fast. Production\n# would reach for something like multilingual-e5-large.\ndense_model = TextEmbedding(\"sentence-transformers/paraphrase-multilingual-MiniLM-L12-v2\")\n\n# A sparse model gives us exact-token matching alongside dense semantics.\n# BM25 is lightweight and language agnostic, a good fit for mixed-language news.\nsparse_model = SparseTextEmbedding(\"Qdrant/bm25\")\n\n# Ask the model for its dimensionality rather than hardcoding it, so the\n# collection config can never drift out of sync with the model.\nDENSE_DIM = len(list(dense_model.embed([\"hello\"]))[0])\nprint(\"Dense vector dimension:\", DENSE_DIM)", + "outputs": [ + { + "output_type": "stream", + "name": "stdout", + "text": "Dense vector dimension: 384\n" + } + ] + }, + { + "cell_type": "markdown", + "source": "Now create the collection with **named vectors**: one dense, one sparse. Naming them lets a single point carry both, and lets a single query fuse both.\n\nThe sparse config needs `modifier=models.Modifier.IDF`. BM25-style sparse vectors store only term frequency, and Qdrant applies the inverse-document-frequency half of the formula at query time. Leave the modifier off and you are not scoring BM25, you are scoring raw term counts.", + "metadata": { + "id": "khdVipy2eBI2" + } + }, + { + "cell_type": "code", + "metadata": { + "id": "Y1rzw27jkooE", + "colab": { + "base_uri": "https://localhost:8080/" + }, + "outputId": "940659fb" + }, + "execution_count": null, + "source": "client.create_collection(\n collection_name=\"news\",\n vectors_config={\n \"dense\": models.VectorParams(size=DENSE_DIM, distance=models.Distance.COSINE),\n },\n sparse_vectors_config={\n \"sparse\": models.SparseVectorParams(\n modifier=models.Modifier.IDF # required for BM25 scoring\n ),\n },\n)\nprint(\"Collection 'news' created.\")", + "outputs": [ + { + "output_type": "stream", + "name": "stdout", + "text": "Collection 'news' created.\n" + } + ] + }, + { + "cell_type": "markdown", + "source": "### Question 2: What must the system filter on?\n\nFrom the brief: country, topic, date range, and source. These are hard rules, not similarity signals. An analyst scoping to \"Vietnam, last seven days\" means exactly that. Hard rules go in the payload, and every field we filter on gets a payload index.\n\nThe schema, decided now, before ingestion:\n\n```yaml\npayload:\n country: string # indexed\n language: string # indexed\n topic: string # indexed\n source: string # indexed\n tenant_id: string # indexed, and marked as a tenant field\n published_at: datetime # indexed\n summary: string # not indexed: returned, never filtered\n```\n\nCreate the indexes now too, in the same setup step, before a single point is uploaded. This ordering is not a style preference. Qdrant extends the HNSW graph with extra edges derived from indexed payload values, and it can only add those edges for indexes that already exist when the graph is built. Create a payload index after ingesting and you have to rebuild the HNSW index to get any benefit from it.\n\nNote `tenant_id` is in this list even though multitenancy does not come up until Section 3. That is the point: it is nearly free now and expensive later.", + "metadata": { + "id": "FMPd3W2XyygN" + } + }, + { + "cell_type": "code", + "metadata": { + "id": "demkvdajgwb2", + "colab": { + "base_uri": "https://localhost:8080/" + }, + "outputId": "924551df" + }, + "execution_count": null, + "source": "with warnings.catch_warnings():\n warnings.simplefilter(\"ignore\") # local mode warns that indexes do nothing here\n\n for field in [\"country\", \"language\", \"topic\", \"source\"]:\n client.create_payload_index(\n collection_name=\"news\",\n field_name=field,\n field_schema=models.PayloadSchemaType.KEYWORD,\n )\n\n client.create_payload_index(\n collection_name=\"news\",\n field_name=\"published_at\",\n field_schema=models.PayloadSchemaType.DATETIME,\n )\n\n # is_tenant tells Qdrant this field identifies tenants, so it can keep each\n # tenant's data close together on disk. Supported for keyword and uuid.\n client.create_payload_index(\n collection_name=\"news\",\n field_name=\"tenant_id\",\n field_schema=models.KeywordIndexParams(\n type=models.KeywordIndexType.KEYWORD,\n is_tenant=True,\n ),\n )\n\nprint(\"Payload indexes created before ingestion: country, language, topic, source, published_at, tenant_id.\")", + "outputs": [ + { + "output_type": "stream", + "name": "stdout", + "text": "Payload indexes created before ingestion: country, language, topic, source, published_at, tenant_id.\n" + } + ] + }, + { + "cell_type": "markdown", + "source": "Skipping an index costs more than a slow scan. The payload index is also what lets Qdrant estimate how many points a filter will match, and that estimate is what the query planner uses to choose a strategy at all. Without it the planner is guessing, and it can fall back to comparing the query against every vector in the collection.\n\nBecause the query still returns results, just slowly, a missing index can sit in production unnoticed for months. Strict mode closes that gap: set `unindexed_filtering_retrieve` to `false` and Qdrant rejects any query that filters on an unindexed field instead of quietly degrading. Qdrant Cloud applies this by default.", + "metadata": { + "id": "CDfgvqZkb6Gj" + } + }, + { + "cell_type": "markdown", + "source": "### Question 3: What is the workload shape?\n\nMillions of articles, text only for now, arriving continuously. Analysts expect this morning's news to be searchable this morning.\n\n**Decisions:** one collection, and continuous upserts rather than periodic rebuilds. Two things still need designing. The initial backfill of millions of articles is a bulk load, not a stream, so batch the upserts and consider disabling indexing for the duration so the optimizer builds the graph once at the end. And Qdrant does index as it ingests, but not instantly: the optimizer builds an HNSW index for a segment only once that segment passes the indexing threshold, and unindexed segments are served by full scan in the meantime.\n\nHere is a small, multilingual sample that stands in for that stream. Note the mix of languages and the exact tokens (a ticker, a ship name) hiding in the text.", + "metadata": { + "id": "HbG81RqwkVHW" + } + }, + { + "cell_type": "code", + "metadata": { + "id": "unY90XYToGwb", + "colab": { + "base_uri": "https://localhost:8080/" + }, + "outputId": "044bf7ef" + }, + "execution_count": null, + "source": "articles = [\n {\n \"country\": \"VN\", \"language\": \"vi\", \"topic\": \"shipping\", \"source\": \"reuters\",\n \"published_at\": \"2026-07-20T08:00:00Z\", \"tenant_id\": \"asia-desk\",\n \"summary\": \"Tac nghen cang tai Hai Phong khi luong hang tang manh.\",\n \"text\": \"Cang Hai Phong o Viet Nam bi tac nghen nghiem trong trong tuan nay khi luong container tang vot, gay cham tre cho tau thuyen.\",\n },\n {\n \"country\": \"JP\", \"language\": \"ja\", \"topic\": \"shipping\", \"source\": \"nikkei\",\n \"published_at\": \"2026-07-21T09:30:00Z\", \"tenant_id\": \"asia-desk\",\n \"summary\": \"東南アジアの港湾混雑が輸送を遅らせている。\",\n \"text\": \"東南アジアの主要港で混雑が深刻化し、コンテナ船の到着が遅れている。特にベトナムとタイの港で影響が大きい。\",\n },\n {\n \"country\": \"CN\", \"language\": \"zh\", \"topic\": \"logistics\", \"source\": \"caixin\",\n \"published_at\": \"2026-07-19T11:00:00Z\", \"tenant_id\": \"asia-desk\",\n \"summary\": \"上海港物流吞吐量创新高。\",\n \"text\": \"上海港本月物流吞吐量创下新高，港口运营商正在扩大堆场以缓解拥堵压力。\",\n },\n {\n \"country\": \"DK\", \"language\": \"en\", \"topic\": \"markets\", \"source\": \"reuters\",\n \"published_at\": \"2026-07-22T07:15:00Z\", \"tenant_id\": \"europe-desk\",\n \"summary\": \"Maersk shares move on delisting speculation.\",\n \"text\": \"Shares tied to the ticker MAERSK-B.CO moved sharply amid speculation about a possible delisting of a subsidiary vehicle.\",\n },\n {\n \"country\": \"SG\", \"language\": \"en\", \"topic\": \"shipping\", \"source\": \"straits-times\",\n \"published_at\": \"2026-07-22T10:45:00Z\", \"tenant_id\": \"asia-desk\",\n \"summary\": \"Vessel Ever Given reroutes through Singapore.\",\n \"text\": \"The container ship Ever Given was rerouted through the Port of Singapore this week to avoid congestion further north.\",\n },\n {\n \"country\": \"US\", \"language\": \"en\", \"topic\": \"markets\", \"source\": \"press-release-wire\",\n \"published_at\": \"2026-06-30T12:00:00Z\", \"tenant_id\": \"americas-desk\",\n \"summary\": \"Company announces quarterly logistics earnings.\",\n \"text\": \"A logistics operator announced quarterly earnings, citing steady demand across North American freight corridors.\",\n },\n {\n \"country\": \"TH\", \"language\": \"th\", \"topic\": \"shipping\", \"source\": \"bangkok-post\",\n \"published_at\": \"2026-07-18T06:20:00Z\", \"tenant_id\": \"asia-desk\",\n \"summary\": \"ความแออัดที่ท่าเรือแหลมฉบังเพิ่มขึ้น\",\n \"text\": \"ท่าเรือแหลมฉบังของไทยเผชิญความแออัดมากขึ้นเนื่องจากปริมาณตู้คอนเทนเนอร์ที่เพิ่มสูงขึ้นในสัปดาห์นี้\",\n },\n {\n \"country\": \"DE\", \"language\": \"de\", \"topic\": \"logistics\", \"source\": \"handelsblatt\",\n \"published_at\": \"2026-07-15T14:10:00Z\", \"tenant_id\": \"europe-desk\",\n \"summary\": \"Hamburger Hafen meldet Verzoegerungen.\",\n \"text\": \"Der Hamburger Hafen meldet Verzoegerungen bei der Abfertigung, da die Zahl der ankommenden Schiffe deutlich gestiegen ist.\",\n },\n]\nprint(f\"{len(articles)} sample articles across {len(set(a['language'] for a in articles))} languages.\")", + "outputs": [ + { + "output_type": "stream", + "name": "stdout", + "text": "8 sample articles across 6 languages.\n" + } + ] + }, + { + "cell_type": "markdown", + "source": "Embed each article into both a dense and a sparse vector, and upsert them as points. In production this same loop runs continuously as articles arrive.", + "metadata": { + "id": "7wnommM30DOX" + } + }, + { + "cell_type": "code", + "metadata": { + "id": "fO4WUDlWuX8y", + "colab": { + "base_uri": "https://localhost:8080/" + }, + "outputId": "6fb222ec" + }, + "execution_count": null, + "source": "def to_sparse_vector(text, is_query=False):\n \"\"\"Convert text to a Qdrant SparseVector using BM25.\n BM25 scores queries and documents slightly differently, so we use\n query_embed for queries and embed for documents.\"\"\"\n emb = next(sparse_model.query_embed(text)) if is_query else next(sparse_model.embed([text]))\n return models.SparseVector(indices=emb.indices.tolist(), values=emb.values.tolist())\n\npoints = []\nfor i, art in enumerate(articles):\n dense_vec = next(dense_model.embed([art[\"text\"]])).tolist()\n sparse_vec = to_sparse_vector(art[\"text\"])\n points.append(\n models.PointStruct(\n id=i,\n vector={\"dense\": dense_vec, \"sparse\": sparse_vec},\n payload=art,\n )\n )\n\nclient.upsert(collection_name=\"news\", points=points)\nprint(f\"Upserted {len(points)} points. Collection count:\", client.count(\"news\").count)", + "outputs": [ + { + "output_type": "stream", + "name": "stdout", + "text": "Upserted 8 points. Collection count: 8\n" + } + ] + }, + { + "cell_type": "markdown", + "source": "### Question 4: What does the retrieval pipeline look like?\n\nStart with the simplest pipeline that fits the query analysis: hybrid (from Question 1) plus filters (from Question 2), fused by rank. No reranker yet. Add complexity when data proves it is needed, never in advance.\n\nThe `query_points` call below does it all in one round trip:\n\n- Two `Prefetch` branches, one per named vector, each pulling 50 candidates.\n- `RrfQuery` to merge the two candidate lists into one ranking.\n- The filter passed **into each prefetch**, so both retrievers only ever consider valid points.\n\nThat last point is the one to slow down on, and Section 3 breaks it on purpose to show why.\n\nNotice the payoff of the multilingual model: we query in **English** and retrieve articles written in Japanese, Vietnamese, Thai, and more, with no translation step.", + "metadata": { + "id": "jZ8OvIia8OG5" + } + }, + { + "cell_type": "code", + "metadata": { + "id": "A2ZnpsoKqzi4", + "colab": { + "base_uri": "https://localhost:8080/" + }, + "outputId": "579e6e88" + }, + "execution_count": null, + "source": "def search(query_text, query_filter=None, limit=5):\n \"\"\"Hybrid search: dense + sparse prefetch, fused by rank, filter in each branch.\"\"\"\n dense_q = next(dense_model.query_embed(query_text)).tolist()\n sparse_q = to_sparse_vector(query_text, is_query=True)\n response = client.query_points(\n collection_name=\"news\",\n prefetch=[\n models.Prefetch(query=dense_q, using=\"dense\",\n filter=query_filter, limit=50),\n models.Prefetch(query=sparse_q, using=\"sparse\",\n filter=query_filter, limit=50),\n ],\n query=models.RrfQuery(rrf=models.Rrf()),\n limit=limit,\n )\n return response.points\n\ndef show(results):\n if not results:\n print(\" (no results)\")\n for r in results:\n p = r.payload\n print(f\"[{r.score:.4f}] {p['country']}/{p['language']:<2} {p['topic']:<9} | {p['summary']}\")\n\nprint(\"Query (English): 'port congestion in Southeast Asia'\\n\")\nshow(search(\"port congestion in Southeast Asia\"))", + "outputs": [ + { + "output_type": "stream", + "name": "stdout", + "text": "Query (English): 'port congestion in Southeast Asia'\n\n[0.7500] SG/en shipping | Vessel Ever Given reroutes through Singapore.\n[0.5000] JP/ja shipping | 東南アジアの港湾混雑が輸送を遅らせている。\n[0.3333] TH/th shipping | ความแออัดที่ท่าเรือแหลมฉบังเพิ่มขึ้น\n[0.2000] CN/zh logistics | 上海港物流吞吐量创新高。\n[0.1667] DE/de logistics | Hamburger Hafen meldet Verzoegerungen.\n" + } + ] + }, + { + "cell_type": "markdown", + "source": "Now the exact-token case. A dense-only search would blur `MAERSK-B.CO` into semantically similar finance text. The sparse half of the hybrid locks onto the literal token, and rank fusion pushes the correct article to the top. This is why we chose hybrid from the start.", + "metadata": { + "id": "GbBlaXkDSIuG" + } + }, + { + "cell_type": "code", + "metadata": { + "id": "IWgIprVFIV4B", + "colab": { + "base_uri": "https://localhost:8080/" + }, + "outputId": "988b787f" + }, + "execution_count": null, + "source": "print(\"Query (exact token): 'MAERSK-B.CO delisting'\\n\")\nshow(search(\"MAERSK-B.CO delisting\"))", + "outputs": [ + { + "output_type": "stream", + "name": "stdout", + "text": "Query (exact token): 'MAERSK-B.CO delisting'\n\n[1.0000] DK/en markets | Maersk shares move on delisting speculation.\n[0.3333] DE/de logistics | Hamburger Hafen meldet Verzoegerungen.\n[0.2500] VN/vi shipping | Tac nghen cang tai Hai Phong khi luong hang tang manh.\n[0.2000] SG/en shipping | Vessel Ever Given reroutes through Singapore.\n[0.1667] TH/th shipping | ความแออัดที่ท่าเรือแหลมฉบังเพิ่มขึ้น\n" + } + ] + }, + { + "cell_type": "markdown", + "source": "Reciprocal Rank Fusion is the right default here because it works from ranks rather than raw scores, so it does not care that cosine similarity and BM25 live on different scales. Two alternatives are worth knowing once you can measure quality: weighted RRF, which favours the stronger retriever via `weights` on `models.Rrf()`, and Distribution-Based Score Fusion, `models.FusionQuery(fusion=models.Fusion.DBSF)`, which normalizes each retriever's score distribution instead. Neither reliably beats the other, so choose with an evaluation set rather than by reputation.", + "metadata": { + "id": "R26qI60ihBeo" + } + }, + { + "cell_type": "markdown", + "source": "### Question 5: What are the deployment constraints?\n\nA research firm with a small engineering team, no data residency restrictions, and a \"please do not page us at night\" budget points to a **managed** deployment. The key idea: the design and the deployment mode are independent decisions. Everything above runs unchanged whether the backend is this in-process client, a Docker container, or Qdrant Cloud.\n\n### The design on one page\n\n| Question | Answer for This System | Layer |\n|----------|------------------------|-------|\n| Query type | Mixed semantic + exact, so hybrid with Reciprocal Rank Fusion | Query |\n| Filter scope | country, topic, source, date, indexed before ingestion | Knowledge, indexing |\n| Workload shape | Millions of text chunks: bulk backfill, then continuous ingestion | Storage, knowledge |\n| Pipeline | Hybrid + per-prefetch filters, multilingual model, no reranker yet | Query, knowledge |\n| Deployment | Managed, design independent of the choice | Distribution |\n\n### Key insight\n\nEvery piece of this design already appeared in an earlier module: the Module 2 pipeline, the Module 3 hybrid pattern, and a payload schema. What is new is the order of operations. You decided what to filter on and created the indexes before the first point went in. That ordering is the difference between a system that scales and one that gets re-ingested three times.", + "metadata": { + "id": "ePW3jSOi27n7" + } + }, + { + "cell_type": "markdown", + "source": "## 3. Filtering\n\nFiltering is the feature that decides whether your results are **correct**, not an accessory to similarity.\n\n### How Qdrant combines filters with vector search\n\nThe naive approach is post-filtering: retrieve the top K by similarity, then discard whatever fails the filter. With a selective filter, say one country out of 200, the top K can contain zero valid results, and there is no K that guarantees correctness.\n\nQdrant does not work that way. A query planner chooses a strategy for each segment, based on the estimated cardinality of the filter and which payload indexes exist:\n\n- When the filter matches a large share of the collection, Qdrant walks the HNSW graph as usual and skips points that fail the filter during traversal.\n- When the filter matches very few points, it can skip the graph entirely and retrieve through the payload index, which is cheaper at that selectivity.\n- When a segment is small enough, a full scan wins outright.\n\nThe middle ground is the hard case, because a strict filter can disconnect the HNSW graph and leave relevant points unreachable. Qdrant handles it by extending the graph with additional edges derived from indexed payload values, which is why the indexes had to exist before ingestion. Those extra edges are added per index, not per combination of indexes, so two or more highly selective filters can still reach a disconnected component. The ACORN algorithm (v1.16) exists for that case.\n\nSo: filtering on every query is safe and fast, as long as every filtered field is indexed and the indexes existed before ingestion. It is not free, and combinations of highly selective filters deserve measurement rather than assumption.\n\n### The filter toolbox\n\n| Condition | Logic | Example |\n|-----------|-------|---------|\n| must | AND, all conditions true | country = VN AND topic = shipping |\n| should | OR, at least one true (`min_should` sets a higher minimum) | topic = shipping OR topic = logistics |\n| must_not | Exclude matches | Exclude source = press-release-wire |\n| Range | Numeric or datetime bounds | published_at within the last seven days |\n| MatchAny | Value in a set | language in [ja, zh, th, vi] |\n| Geo | Radius, bounding box, or polygon | Events within 100 km of a port |\n\nThe cells below run several of these against the news collection.", + "metadata": { + "id": "yE9ZkAFvmtSb" + } + }, + { + "cell_type": "code", + "metadata": { + "id": "IBbuMFd9W8gq", + "colab": { + "base_uri": "https://localhost:8080/" + }, + "outputId": "0c426ed4" + }, + "execution_count": null, + "source": "# must: AND. Only Vietnamese shipping news.\nf_must = models.Filter(\n must=[\n models.FieldCondition(key=\"country\", match=models.MatchValue(value=\"VN\")),\n models.FieldCondition(key=\"topic\", match=models.MatchValue(value=\"shipping\")),\n ]\n)\nprint(\"must (country=VN AND topic=shipping):\")\nshow(search(\"port congestion\", query_filter=f_must))", + "outputs": [ + { + "output_type": "stream", + "name": "stdout", + "text": "must (country=VN AND topic=shipping):\n" + }, + { + "output_type": "stream", + "name": "stdout", + "text": "[0.5000] VN/vi shipping | Tac nghen cang tai Hai Phong khi luong hang tang manh.\n" + } + ] + }, + { + "cell_type": "code", + "metadata": { + "id": "j4JRfdQAe6NX", + "colab": { + "base_uri": "https://localhost:8080/" + }, + "outputId": "1d1d758e" + }, + "execution_count": null, + "source": "# should: OR. Either shipping or logistics topics.\nf_should = models.Filter(\n should=[\n models.FieldCondition(key=\"topic\", match=models.MatchValue(value=\"shipping\")),\n models.FieldCondition(key=\"topic\", match=models.MatchValue(value=\"logistics\")),\n ]\n)\nprint(\"should (topic=shipping OR topic=logistics):\")\nshow(search(\"congestion at ports\", query_filter=f_should))", + "outputs": [ + { + "output_type": "stream", + "name": "stdout", + "text": "should (topic=shipping OR topic=logistics):\n" + }, + { + "output_type": "stream", + "name": "stdout", + "text": "[0.7000] SG/en shipping | Vessel Ever Given reroutes through Singapore.\n[0.5000] JP/ja shipping | 東南アジアの港湾混雑が輸送を遅らせている。\n[0.3333] CN/zh logistics | 上海港物流吞吐量创新高。\n[0.2500] TH/th shipping | ความแออัดที่ท่าเรือแหลมฉบังเพิ่มขึ้น\n[0.1667] DE/de logistics | Hamburger Hafen meldet Verzoegerungen.\n" + } + ] + }, + { + "cell_type": "code", + "metadata": { + "id": "qiGogkdmtsVr", + "colab": { + "base_uri": "https://localhost:8080/" + }, + "outputId": "825040b8" + }, + "execution_count": null, + "source": "# must_not + Range + MatchAny combined, evaluated together while the search runs.\nf_combined = models.Filter(\n must=[\n models.FieldCondition(\n key=\"published_at\",\n range=models.DatetimeRange(gte=\"2026-07-15T00:00:00Z\"),\n ),\n models.FieldCondition(\n key=\"language\",\n match=models.MatchAny(any=[\"ja\", \"zh\", \"th\", \"vi\"]),\n ),\n ],\n must_not=[\n models.FieldCondition(key=\"source\", match=models.MatchValue(value=\"press-release-wire\")),\n ],\n)\nprint(\"recent AND asian-language AND not a press-release wire:\")\nshow(search(\"port congestion in Southeast Asia\", query_filter=f_combined))", + "outputs": [ + { + "output_type": "stream", + "name": "stdout", + "text": "recent AND asian-language AND not a press-release wire:\n[0.5000] JP/ja shipping | 東南アジアの港湾混雑が輸送を遅らせている。\n[0.3333] TH/th shipping | ความแออัดที่ท่าเรือแหลมฉบังเพิ่มขึ้น\n[0.2500] CN/zh logistics | 上海港物流吞吐量创新高。\n[0.2000] VN/vi shipping | Tac nghen cang tai Hai Phong khi luong hang tang manh.\n" + } + ] + }, + { + "cell_type": "markdown", + "source": "### Common mistake: filters in the wrong place\n\nEvery filtered search above worked because `search()` passes the filter into each `Prefetch`. Move it to a top-level `query_filter` and it silently stops constraining the retrievers.\n\nWhenever a query has at least one prefetch, Qdrant runs the prefetches first and applies the main query to their results. A top-level filter therefore never reaches them: each searches the whole collection, returns its 50 candidates, and the filter only trims the fused set at the end. That is post-filtering, with exactly the failure mode described above.\n\nBelow, the same selective filter runs both ways. The correct version returns only Vietnamese shipping news. The broken version does not, and nothing raises an error.", + "metadata": { + "id": "lD5gYAEZyFQ8" + } + }, + { + "cell_type": "code", + "metadata": { + "id": "vZRNzvdieTpk", + "colab": { + "base_uri": "https://localhost:8080/" + }, + "outputId": "1dda4347" + }, + "execution_count": null, + "source": "def search_wrong(query_text, query_filter=None, limit=5):\n \"\"\"Identical to search(), except the filter sits at the top level.\"\"\"\n dense_q = next(dense_model.query_embed(query_text)).tolist()\n sparse_q = to_sparse_vector(query_text, is_query=True)\n return client.query_points(\n collection_name=\"news\",\n prefetch=[\n models.Prefetch(query=dense_q, using=\"dense\", limit=50),\n models.Prefetch(query=sparse_q, using=\"sparse\", limit=50),\n ],\n query=models.RrfQuery(rrf=models.Rrf()),\n query_filter=query_filter, # too late: the prefetches already ran\n limit=limit,\n ).points\n\nprint(\"CORRECT filter inside each prefetch (country=VN AND topic=shipping):\")\nshow(search(\"port congestion\", query_filter=f_must))\nprint()\nprint(\"BROKEN same filter at the top level:\")\nshow(search_wrong(\"port congestion\", query_filter=f_must))", + "outputs": [ + { + "output_type": "stream", + "name": "stdout", + "text": "CORRECT filter inside each prefetch (country=VN AND topic=shipping):\n[0.5000] VN/vi shipping | Tac nghen cang tai Hai Phong khi luong hang tang manh.\n\nBROKEN same filter at the top level:\n[0.7000] SG/en shipping | Vessel Ever Given reroutes through Singapore.\n[0.5000] JP/ja shipping | 東南アジアの港湾混雑が輸送を遅らせている。\n[0.3333] CN/zh logistics | 上海港物流吞吐量创新高。\n[0.2500] TH/th shipping | ความแออัดที่ท่าเรือแหลมฉบังเพิ่มขึ้น\n[0.1667] DE/de logistics | Hamburger Hafen meldet Verzoegerungen.\n" + } + ] + }, + { + "cell_type": "markdown", + "source": "The rule is short. No prefetch, use `query_filter`. Prefetch, put the filter in every prefetch.", + "metadata": { + "id": "jBq78Hp7twax" + } + }, + { + "cell_type": "markdown", + "source": "### A special case: scoping by user or tenant\n\nIn almost any multi user product you must scope every query to one user's or one customer's data. The instinct is a collection per user, which becomes millions of collections and is operationally unmanageable. The standard pattern instead:\n\n1. Add a `tenant_id` payload field to every point at ingestion. (Done, above.)\n2. Create a payload index on it with `is_tenant=True`. (Done, before ingestion.)\n3. Filter on it at every query. Never omit it.\n\nSteps 1 and 3 alone are already correct: they just get slower than they need to as tenant count grows. Step 2 is what keeps it fast, and it is the one people skip.", + "metadata": { + "id": "DFmFaqfycbso" + } + }, + { + "cell_type": "code", + "metadata": { + "id": "KGUOS2yu9jSN", + "colab": { + "base_uri": "https://localhost:8080/" + }, + "outputId": "0deabc28" + }, + "execution_count": null, + "source": "def tenant_search(query_text, tenant_id, limit=5):\n return search(\n query_text,\n query_filter=models.Filter(\n must=[models.FieldCondition(key=\"tenant_id\", match=models.MatchValue(value=tenant_id))]\n ),\n limit=limit,\n )\n\nprint(\"europe-desk view of 'port delays':\")\nshow(tenant_search(\"port delays\", tenant_id=\"europe-desk\"))\nprint(\"\\nasia-desk view of 'port delays':\")\nshow(tenant_search(\"port delays\", tenant_id=\"asia-desk\"))", + "outputs": [ + { + "output_type": "stream", + "name": "stdout", + "text": "europe-desk view of 'port delays':\n[0.5000] DE/de logistics | Hamburger Hafen meldet Verzoegerungen.\n[0.3333] DK/en markets | Maersk shares move on delisting speculation.\n\nasia-desk view of 'port delays':\n[0.7000] SG/en shipping | Vessel Ever Given reroutes through Singapore.\n[0.5000] JP/ja shipping | 東南アジアの港湾混雑が輸送を遅らせている。\n[0.3333] TH/th shipping | ความแออัดที่ท่าเรือแหลมฉบังเพิ่มขึ้น\n[0.2500] CN/zh logistics | 上海港物流吞吐量创新高。\n[0.1667] VN/vi shipping | Tac nghen cang tai Hai Phong khi luong hang tang manh.\n" + } + ] + }, + { + "cell_type": "markdown", + "source": "### Key insight\n\nDesign the payload schema before you ingest, driven by one question: what will I need to filter on? Time, geography, identity, permissions, and status flags are the usual suspects.\n\nAdding a payload *field* later is easy. Adding a payload *index* later means rebuilding the HNSW graph, and discovering at query time that you never stored `language` at all means re-ingesting everything.", + "metadata": { + "id": "FZXZK3QQobiw" + } + }, + { + "cell_type": "markdown", + "source": "## 4. The Production RAG Pipeline\n\nRetrieval-Augmented Generation (RAG) retrieves relevant passages from a vector search engine and hands them to an LLM as context, so the model answers from your data instead of relying only on what it memorized during training.\n\nThe production shape, using everything above:\n\n1. **Query understanding:** extract hard constraints (dates, country, topic) into a filter. Embed the query as dense and sparse vectors.\n2. **Hybrid retrieval:** dense + sparse prefetch, each carrying the filter, fused by rank. One `query_points` call.\n3. **Optional reranking:** a cross-encoder scores the top candidates and keeps the best few. Add this only when evaluation shows fused results need refinement.\n4. **LLM generation:** the top passages go in as context, the model generates the answer with sources.\n\n### Rule of thumb\n\nWhen RAG quality disappoints, improve step 2 before reaching for a bigger model in step 4. Retrieval quality caps answer quality: the model cannot cite a passage it never received.\n\nThe cell below implements steps 1, 2, and the prompt assembly for step 4. It runs with no API key and prints the exact prompt an LLM would receive.", + "metadata": { + "id": "gZIMMQEaXUFH" + } + }, + { + "cell_type": "code", + "metadata": { + "id": "OeTpeqTmT8Ex", + "colab": { + "base_uri": "https://localhost:8080/" + }, + "outputId": "7ac9a124" + }, + "execution_count": null, + "source": "def rag_context(question, query_filter=None, k=3):\n \"\"\"Steps 1 and 2: retrieve the top-k passages for a question.\"\"\"\n hits = search(question, query_filter=query_filter, limit=k)\n passages = []\n for h in hits:\n p = h.payload\n passages.append(f\"- ({p['source']}, {p['country']}, {p['published_at'][:10]}) {p['summary']}\")\n return \"\\n\".join(passages), hits\n\ndef build_prompt(question, query_filter=None, k=3):\n context, hits = rag_context(question, query_filter=query_filter, k=k)\n prompt = (\n \"Answer the question using only the sources below. Cite the source name.\\n\\n\"\n f\"Sources:\\n{context}\\n\\n\"\n f\"Question: {question}\\nAnswer:\"\n )\n return prompt, hits\n\nprompt, hits = build_prompt(\"What is happening with port congestion in Southeast Asia?\")\nprint(prompt)", + "outputs": [ + { + "output_type": "stream", + "name": "stdout", + "text": "Answer the question using only the sources below. Cite the source name.\n\nSources:\n- (straits-times, SG, 2026-07-22) Vessel Ever Given reroutes through Singapore.\n- (nikkei, JP, 2026-07-21) 東南アジアの港湾混雑が輸送を遅らせている。\n- (bangkok-post, TH, 2026-07-18) ความแออัดที่ท่าเรือแหลมฉบังเพิ่มขึ้น\n\nQuestion: What is happening with port congestion in Southeast Asia?\nAnswer:\n" + } + ] + }, + { + "cell_type": "code", + "metadata": { + "id": "UsJadqPQPsGC", + "colab": { + "base_uri": "https://localhost:8080/" + }, + "outputId": "71e3fe07" + }, + "execution_count": null, + "source": "# OPTIONAL step 4: hand the prompt to a real LLM. This block is inert unless you\n# add an API key, so the notebook still runs end to end without one.\n#\n# import os\n# from anthropic import Anthropic\n# llm = Anthropic(api_key=os.environ[\"ANTHROPIC_API_KEY\"])\n# msg = llm.messages.create(\n# model=\"claude-sonnet-4-6\",\n# max_tokens=300,\n# messages=[{\"role\": \"user\", \"content\": prompt}],\n# )\n# print(msg.content[0].text)\n\nprint(\"Prompt is ready. Uncomment the block above and set an API key to generate an answer.\")", + "outputs": [ + { + "output_type": "stream", + "name": "stdout", + "text": "Prompt is ready. Uncomment the block above and set an API key to generate an answer.\n" + } + ] + }, + { + "cell_type": "markdown", + "source": "## What's next: Module 5\n\nThe capstone extends this system to three modalities at once: news, audio, and satellite imagery on shared points, clustered into risk themes and queried across languages.\n\n[Continue to Module 5](https://qdrant.tech/course/beginners/module-5/)", + "metadata": { + "id": "Y8Bqn6nKiG7i" + } + } + ] +} diff --git a/Beginner-course/Module5.ipynb b/Beginner-course/Module5.ipynb new file mode 100644 index 0000000..1c76286 --- /dev/null +++ b/Beginner-course/Module5.ipynb @@ -0,0 +1,358 @@ +{ + "nbformat": 4, + "nbformat_minor": 0, + "metadata": { + "colab": { + "provenance": [] + }, + "kernelspec": { + "display_name": "Python 3", + "name": "python3" + }, + "language_info": { + "name": "python" + } + }, + "cells": [ + { + "cell_type": "markdown", + "source": "# Module 5: Capstone, Multimodal Supplier Risk Intelligence\n\nA factory in Vietnam catches fire on a Tuesday. By that afternoon it is in a Japanese trade publication and on a Chinese forum. By Thursday it reaches the English business press. By Friday your procurement team finds out.\n\nThis notebook builds the system that finds Tuesday.\n\n## What you will do\n\n1. Create one collection with three named vectors: dense text, sparse text, and CLIP image.\n2. Ingest multilingual news and satellite tiles onto shared points.\n3. Query in English and retrieve Japanese and Chinese sources, with no translation.\n4. Query images with text, through CLIP's shared space.\n5. Cluster signals into risk themes and use a cluster centroid as a query.\n6. Run the analyst hybrid query, and break its filter on purpose to see why placement matters.\n\n**Tip for Colab:** the first cells download `multilingual-e5-large` (about 2 GB) plus two small CLIP encoders, so give the setup a few minutes. Everything after that is fast.\n\nCompanion notebook to the [Module 5 lesson](https://qdrant.tech/course/beginners/module-5/).", + "metadata": { + "id": "ujOeHwdFcAef" + } + }, + { + "cell_type": "markdown", + "source": "## Setup\n\nThree models, each chosen for a reason.\n\n`intfloat/multilingual-e5-large` inherits 100 languages from XLM-RoBERTa and puts all of them in one vector space, which is the point of the whole system. `Qdrant/bm25` gives exact-token matching for supplier codes and ticker symbols. And a matched pair of CLIP encoders, one for images and one for text, puts pictures and words in a *second* shared space so a text query can retrieve a photo.\n\nNothing here needs a GPU or an API key.", + "metadata": { + "id": "AZhnM6Jy8c1r" + } + }, + { + "cell_type": "code", + "metadata": { + "id": "ihtYlKNxHddm" + }, + "execution_count": null, + "source": "!pip install -q \"qdrant-client[fastembed]\" scikit-learn pillow ", + "outputs": [] + }, + { + "cell_type": "code", + "metadata": { + "id": "QAtKCsXRpJG2", + "colab": { + "base_uri": "https://localhost:8080/" + }, + "outputId": "b4f16c27" + }, + "execution_count": null, + "source": "import warnings\nimport numpy as np\nfrom qdrant_client import QdrantClient, models\nfrom fastembed import TextEmbedding, SparseTextEmbedding, ImageEmbedding\n\nwarnings.filterwarnings(\"ignore\")\n\nTEXT_MODEL = \"intfloat/multilingual-e5-large\" # 1024-dim, 100 languages\nSPARSE_MODEL = \"Qdrant/bm25\" # exact tokens\nCLIP_VISION = \"Qdrant/clip-ViT-B-32-vision\" # 512-dim images\nCLIP_TEXT = \"Qdrant/clip-ViT-B-32-text\" # 512-dim text, same space\n\ntext_model = TextEmbedding(TEXT_MODEL)\nsparse_model = SparseTextEmbedding(SPARSE_MODEL)\nclip_vision = ImageEmbedding(CLIP_VISION)\nclip_text = TextEmbedding(CLIP_TEXT)\n\nTEXT_DIM = len(next(text_model.embed([\"passage: warm up\"])))\nCLIP_DIM = len(next(clip_text.embed([\"warm up\"])))\nprint(\"text dim:\", TEXT_DIM, \"| clip dim:\", CLIP_DIM)", + "outputs": [ + { + "output_type": "stream", + "name": "stdout", + "text": "text dim: 1024 | clip dim: 512\n" + } + ] + }, + { + "cell_type": "markdown", + "source": "### The e5 prefixes are not optional\n\ne5 was trained with `query:` on search text and `passage:` on stored text. FastEmbed does not add them for you, and if you skip them nothing errors.\n\nMeasuring the effect takes a little care, because the thing that matters is not how high a relevant document scores. It is how far a relevant document sits above an irrelevant one, since that gap is what ranking depends on. So we score one relevant passage and one irrelevant passage against the same query, both ways, and compare the margin.", + "metadata": { + "id": "cPVJ2tRKJC06" + } + }, + { + "cell_type": "code", + "metadata": { + "id": "DPdRO9YrxPbC", + "colab": { + "base_uri": "https://localhost:8080/" + }, + "outputId": "210c236d" + }, + "execution_count": null, + "source": "def unit(v):\n v = np.asarray(v, dtype=np.float32)\n return v / np.linalg.norm(v)\n\ndef embed_text(t, is_query=False):\n \"\"\"e5 needs an explicit prefix. This is the only place we add it.\"\"\"\n prefix = \"query: \" if is_query else \"passage: \"\n return unit(next(text_model.embed([prefix + t])))\n\ndef embed_raw(t):\n return unit(next(text_model.embed([t]))) # deliberately no prefix\n\nquestion = \"factory fire halted production\"\nrelevant = \"工場火災により生産が停止し、出荷の遅延が続いている。\"\nirrelevant = \"The supplier reaffirmed full-year guidance and reported steady quarterly demand.\"\n\nqv = embed_text(question, is_query=True)\nwith_rel, with_irr = float(qv @ embed_text(relevant)), float(qv @ embed_text(irrelevant))\n\nqr = embed_raw(question)\nraw_rel, raw_irr = float(qr @ embed_raw(relevant)), float(qr @ embed_raw(irrelevant))\n\nprint(f\"with prefixes relevant={with_rel:.3f} irrelevant={with_irr:.3f} margin={with_rel - with_irr:+.3f}\")\nprint(f\"without prefixes relevant={raw_rel:.3f} irrelevant={raw_irr:.3f} margin={raw_rel - raw_irr:+.3f}\")\nprint()\nprint(\"Skipping the prefixes RAISES both scores and NARROWS the gap between them.\")\nprint(\"Absolute similarity is not the target. Separation is.\")\nprint()\nprint(\"Note the scale too: e5 similarities compress into roughly 0.7 to 1.0,\")\nprint(\"so never read one of these numbers as a percentage of relevance.\")", + "outputs": [ + { + "output_type": "stream", + "name": "stdout", + "text": "with prefixes relevant=0.804 irrelevant=0.719 margin=+0.085\nwithout prefixes relevant=0.823 irrelevant=0.753 margin=+0.070\n\nSkipping the prefixes RAISES both scores and NARROWS the gap between them.\nAbsolute similarity is not the target. Separation is.\n\nNote the scale too: e5 similarities compress into roughly 0.7 to 1.0,\nso never read one of these numbers as a percentage of relevance.\n" + } + ] + }, + { + "cell_type": "markdown", + "source": "## 1. One Collection, Not Three\n\nThree modalities, and the instinct is three collections because that feels tidier.\n\nA single event produces evidence in several modalities at once. That factory fire is a news article, a satellite image, and a line in an earnings call. Split by modality and you have split one event across three collections: every query hits all three and stitches results back together in your own code, and your filters get written three times.\n\nNamed vectors solve it. One point carries a dense text vector, a sparse one, and a CLIP image vector, and all of them share a single payload.\n\nTwo details below decide whether this works in production. The sparse config needs `modifier=models.Modifier.IDF`, or you are not scoring BM25. And every payload index is created **now**, before any data arrives, because Qdrant can only add filter-aware edges to the vector index for indexes that already exist when it is built. Here there are two dense vectors, so a late index means rebuilding two graphs.\n\n`risk_score` is the one people forget, because nothing filters on it until Section 5. Miss it and the analyst query does not get slower, it fails: Qdrant Cloud enables strict mode by default, and strict mode rejects any query that filters an unindexed field.", + "metadata": { + "id": "eBI2Y1rzw27j" + } + }, + { + "cell_type": "code", + "metadata": { + "id": "kooEKqazwJ7Q", + "colab": { + "base_uri": "https://localhost:8080/" + }, + "outputId": "89a0ecdc" + }, + "execution_count": null, + "source": "client = QdrantClient(\":memory:\")\n\nclient.create_collection(\n collection_name=\"supplier_signals\",\n vectors_config={\n \"text_dense\": models.VectorParams(size=TEXT_DIM, distance=models.Distance.COSINE),\n \"image\": models.VectorParams(size=CLIP_DIM, distance=models.Distance.COSINE),\n },\n sparse_vectors_config={\n \"text_sparse\": models.SparseVectorParams(modifier=models.Modifier.IDF),\n },\n)\n\nfor field in [\"supplier_id\", \"source_type\", \"language\", \"country\", \"facility_id\"]:\n client.create_payload_index(\"supplier_signals\", field_name=field,\n field_schema=models.PayloadSchemaType.KEYWORD)\n\nclient.create_payload_index(\"supplier_signals\", field_name=\"published_at\",\n field_schema=models.PayloadSchemaType.DATETIME)\nclient.create_payload_index(\"supplier_signals\", field_name=\"risk_score\",\n field_schema=models.PayloadSchemaType.FLOAT)\nclient.create_payload_index(\"supplier_signals\", field_name=\"cluster_id\",\n field_schema=models.PayloadSchemaType.INTEGER)\n\nprint(\"Collection created with 3 named vectors and 8 payload indexes, before ingestion.\")", + "outputs": [ + { + "output_type": "stream", + "name": "stdout", + "text": "Collection created with 3 named vectors and 8 payload indexes, before ingestion.\n" + } + ] + }, + { + "cell_type": "markdown", + "source": "## 2. Ingesting Multilingual Signals\n\nThe signals below stand in for a day's feed. Note the shape of the story: the Japanese and Chinese sources are reporting a shutdown, while the English sources are reporting routine quarterly news. That gap is what Section 6 goes looking for.\n\n`source_type` is drawn from one fixed vocabulary, `news` or `satellite` here, because a filter written against a value nobody ingests returns nothing and warns you about nothing.", + "metadata": { + "id": "yygNdemkvdaj" + } + }, + { + "cell_type": "code", + "metadata": { + "id": "gwb2Mjpvwh09", + "colab": { + "base_uri": "https://localhost:8080/" + }, + "outputId": "771154d2" + }, + "execution_count": null, + "source": "signals = [\n # SUP-7291: local-language sources are ahead of the English ones\n dict(supplier_id=\"SUP-7291\", language=\"ja\", country=\"JP\", source_type=\"news\",\n published_at=\"2026-07-21T09:00:00Z\", risk_score=0.88,\n text=\"工場火災により生産が停止し、出荷の遅延が続いている。復旧の見込みは立っていない。\"),\n dict(supplier_id=\"SUP-7291\", language=\"zh\", country=\"CN\", source_type=\"news\",\n published_at=\"2026-07-21T14:00:00Z\", risk_score=0.82,\n text=\"供应商工厂因火灾停产，交货期限推迟，客户已收到延误通知。\"),\n dict(supplier_id=\"SUP-7291\", language=\"ja\", country=\"JP\", source_type=\"news\",\n published_at=\"2026-07-22T02:00:00Z\", risk_score=0.71,\n text=\"労働争議が長引き、工場の稼働率が低下している。組合との交渉は難航している。\"),\n dict(supplier_id=\"SUP-7291\", language=\"vi\", country=\"VN\", source_type=\"news\",\n published_at=\"2026-07-20T08:00:00Z\", risk_score=0.64,\n text=\"Cang Hai Phong bi tac nghen, cac chuyen hang cua nha cung cap bi cham tre.\"),\n dict(supplier_id=\"SUP-7291\", language=\"en\", country=\"US\", source_type=\"news\",\n published_at=\"2026-07-22T07:15:00Z\", risk_score=0.12,\n text=\"The supplier reaffirmed full-year guidance and reported steady quarterly demand.\"),\n dict(supplier_id=\"SUP-7291\", language=\"en\", country=\"GB\", source_type=\"news\",\n published_at=\"2026-07-22T11:00:00Z\", risk_score=0.10,\n text=\"Analysts described the quarter as routine, with no change to the outlook for the group.\"),\n dict(supplier_id=\"SUP-7291\", language=\"en\", country=\"US\", source_type=\"news\",\n published_at=\"2026-07-19T16:00:00Z\", risk_score=0.55,\n text=\"Ticker SUP7291.T slipped modestly on higher freight costs across the region.\"),\n # a second supplier, so tenant-style scoping has something to exclude\n dict(supplier_id=\"SUP-0002\", language=\"zh\", country=\"CN\", source_type=\"news\",\n published_at=\"2026-07-21T10:00:00Z\", risk_score=0.79,\n text=\"另一家供应商的工厂发生停电，生产线暂时中断。\"),\n dict(supplier_id=\"SUP-0002\", language=\"en\", country=\"DE\", source_type=\"news\",\n published_at=\"2026-07-18T09:00:00Z\", risk_score=0.20,\n text=\"A European logistics operator announced a routine expansion of warehouse capacity.\"),\n]\n\ndef to_sparse(text, is_query=False):\n emb = next(sparse_model.query_embed(text)) if is_query else next(sparse_model.embed([text]))\n return models.SparseVector(indices=emb.indices.tolist(), values=emb.values.tolist())\n\npoints = []\nfor i, s in enumerate(signals):\n points.append(models.PointStruct(\n id=i,\n vector={\n \"text_dense\": embed_text(s[\"text\"]).tolist(), # passage: prefix\n \"text_sparse\": to_sparse(s[\"text\"]),\n },\n payload={**s, \"summary\": s[\"text\"][:60]},\n ))\n\nclient.upsert(\"supplier_signals\", points=points)\nprint(\"ingested\", client.count(\"supplier_signals\").count, \"text signals across\",\n len({s[\"language\"] for s in signals}), \"languages\")", + "outputs": [ + { + "output_type": "stream", + "name": "stdout", + "text": "ingested 9 text signals across 4 languages\n" + } + ] + }, + { + "cell_type": "markdown", + "source": "## 3. Satellite Tiles Through CLIP\n\nReal satellite imagery is not something a notebook can ship, so we generate four crude tiles instead. They are enough to show the mechanics and, as you will see, enough to show a real limitation too.\n\nThese points carry only the `image` vector. They live in the same collection as the text signals and share the same payload schema, which is the whole argument for named vectors.", + "metadata": { + "id": "b6GjHbG81Rqw" + } + }, + { + "cell_type": "code", + "metadata": { + "id": "kVHWunY90XYT", + "colab": { + "base_uri": "https://localhost:8080/" + }, + "outputId": "3850044b" + }, + "execution_count": null, + "source": "from PIL import Image, ImageDraw, ImageFilter\nimport os\n\nos.makedirs(\"tiles\", exist_ok=True)\n\ndef tile_smoke():\n im = Image.new(\"RGB\", (224, 224), (70, 72, 78)); d = ImageDraw.Draw(im)\n d.rectangle([40, 150, 184, 224], fill=(48, 48, 52))\n for y, r in [(140, 20), (115, 28), (88, 36), (60, 44)]:\n d.ellipse([112 - r, y - r, 112 + r, y + r], fill=(190, 190, 195))\n return im.filter(ImageFilter.GaussianBlur(6))\n\ndef tile_farmland():\n im = Image.new(\"RGB\", (224, 224), (94, 140, 60)); d = ImageDraw.Draw(im)\n for x in range(0, 224, 16):\n d.rectangle([x, 0, x + 8, 224], fill=(120, 168, 74))\n return im.filter(ImageFilter.GaussianBlur(1))\n\ndef tile_harbour():\n im = Image.new(\"RGB\", (224, 224), (40, 90, 150)); d = ImageDraw.Draw(im)\n for x, y in [(30, 60), (120, 100), (70, 160), (150, 40)]:\n d.rectangle([x, y, x + 50, y + 18], fill=(210, 210, 215))\n return im.filter(ImageFilter.GaussianBlur(1))\n\ndef tile_fire():\n im = Image.new(\"RGB\", (224, 224), (30, 20, 15)); d = ImageDraw.Draw(im)\n for r, c in [(90, (120, 30, 0)), (65, (200, 70, 0)), (40, (255, 150, 0)), (20, (255, 230, 120))]:\n d.ellipse([112 - r, 150 - r, 112 + r, 150 + r], fill=c)\n return im.filter(ImageFilter.GaussianBlur(8))\n\ntiles = [\n (\"smoke_plume\", tile_smoke, \"SUP-7291\", \"FAC-01\"),\n (\"fire\", tile_fire, \"SUP-7291\", \"FAC-01\"),\n (\"harbour\", tile_harbour, \"SUP-7291\", \"FAC-02\"),\n (\"farmland\", tile_farmland, \"SUP-0002\", \"FAC-09\"),\n]\n\npaths = []\nfor name, fn, _, _ in tiles:\n p = f\"tiles/{name}.png\"\n fn().save(p)\n paths.append(p)\n\nvecs = [unit(v).tolist() for v in clip_vision.embed(paths)]\n\nclient.upsert(\"supplier_signals\", points=[\n models.PointStruct(\n id=100 + i,\n vector={\"image\": vec},\n payload=dict(supplier_id=sup, facility_id=fac, source_type=\"satellite\",\n language=\"n/a\", country=\"VN\",\n published_at=\"2026-07-21T00:00:00Z\",\n risk_score=0.9 if name in (\"fire\", \"smoke_plume\") else 0.1,\n summary=f\"synthetic satellite tile: {name}\"),\n )\n for i, (vec, (name, _, sup, fac)) in enumerate(zip(vecs, tiles))\n])\n\nprint(\"collection now holds\", client.count(\"supplier_signals\").count, \"points (text + imagery)\")", + "outputs": [ + { + "output_type": "stream", + "name": "stdout", + "text": "collection now holds 13 points (text + imagery)\n" + } + ] + }, + { + "cell_type": "markdown", + "source": "## 4. Two Kinds of Cross-Modal Query\n\n### Text against images, via CLIP\n\nThe query text goes through CLIP's **text** encoder, not e5. That is the part people get wrong: each named vector is its own space, and a query only means something in the space it was embedded for. Sending an e5 vector at the `image` vector would return numbers, and they would be noise.", + "metadata": { + "id": "7B697wnommM3" + } + }, + { + "cell_type": "code", + "metadata": { + "id": "0DOXfO4WUDlW", + "colab": { + "base_uri": "https://localhost:8080/" + }, + "outputId": "5cf66fb2" + }, + "execution_count": null, + "source": "def image_search(text, limit=4):\n qv = unit(next(clip_text.embed([text]))).tolist() # CLIP text encoder\n return client.query_points(\"supplier_signals\", query=qv, using=\"image\",\n limit=limit).points\n\nfor q in [\"an orange fire burning\", \"ships in blue water at a port\", \"grey smoke rising into the sky\"]:\n print(f\"{q!r}\")\n for r in image_search(q):\n print(f\" {r.score:.3f} {r.payload['summary']}\")\n print()", + "outputs": [ + { + "output_type": "stream", + "name": "stdout", + "text": "'an orange fire burning'\n 0.267 synthetic satellite tile: fire\n 0.209 synthetic satellite tile: farmland\n 0.191 synthetic satellite tile: smoke_plume\n 0.188 synthetic satellite tile: harbour\n\n'ships in blue water at a port'\n 0.209 synthetic satellite tile: harbour\n 0.193 synthetic satellite tile: smoke_plume\n 0.180 synthetic satellite tile: farmland\n 0.141 synthetic satellite tile: fire\n\n'grey smoke rising into the sky'\n 0.211 synthetic satellite tile: fire\n 0.204 synthetic satellite tile: farmland\n 0.195 synthetic satellite tile: smoke_plume\n 0.194 synthetic satellite tile: harbour\n\n" + } + ] + }, + { + "cell_type": "markdown", + "source": "Two of those three are right, and the third is the useful one.\n\n\"an orange fire burning\" and \"ships in blue water at a port\" retrieve the correct tiles. \"grey smoke rising into the sky\" does not: it prefers the fire tile.\n\nThat is not a bug in Qdrant or in the query. CLIP was trained on photographs, and these tiles are flat drawings made with a few ellipses. They sit outside the distribution the model learned, so its judgments get unreliable. Swap in real satellite imagery and this improves immediately.\n\nThe lesson generalizes past this notebook: cross-modal retrieval quality depends on your images resembling the model's training data, and the only way to know is to evaluate on your own.", + "metadata": { + "id": "hj4YjZ8OvIia" + } + }, + { + "cell_type": "markdown", + "source": "### English query against local-language text\n\nNow the capability the whole system exists for. The query is English, the corpus is not, and there is no translation step anywhere.\n\nThe filter has no prefetch above it, so `query_filter` is the correct placement here.", + "metadata": { + "id": "8OG5A2ZnpsoK" + } + }, + { + "cell_type": "code", + "metadata": { + "id": "qzi4vCK4A4FG", + "colab": { + "base_uri": "https://localhost:8080/" + }, + "outputId": "80720c27" + }, + "execution_count": null, + "source": "def text_search(query_en, languages=None, supplier=None, limit=5):\n must = []\n if supplier:\n must.append(models.FieldCondition(key=\"supplier_id\",\n match=models.MatchValue(value=supplier)))\n if languages:\n must.append(models.FieldCondition(key=\"language\",\n match=models.MatchAny(any=languages)))\n return client.query_points(\n \"supplier_signals\",\n query=embed_text(query_en, is_query=True).tolist(), # query: prefix\n using=\"text_dense\",\n query_filter=models.Filter(must=must) if must else None,\n limit=limit,\n ).points\n\nprint(\"English query -> Japanese and Chinese sources only:\\n\")\nfor r in text_search(\"factory shutdown production halt\",\n languages=[\"ja\", \"zh\"], supplier=\"SUP-7291\"):\n p = r.payload\n print(f\" {r.score:.3f} [{p['language']}] risk={p['risk_score']:.2f} {p['summary']}\")", + "outputs": [ + { + "output_type": "stream", + "name": "stdout", + "text": "English query -> Japanese and Chinese sources only:\n\n" + }, + { + "output_type": "stream", + "name": "stdout", + "text": " 0.786 [zh] risk=0.82 供应商工厂因火灾停产，交货期限推迟，客户已收到延误通知。\n 0.776 [ja] risk=0.88 工場火災により生産が停止し、出荷の遅延が続いている。復旧の見込みは立っていない。\n 0.758 [ja] risk=0.71 労働争議が長引き、工場の稼働率が低下している。組合との交渉は難航している。\n" + } + ] + }, + { + "cell_type": "markdown", + "source": "## 5. Clustering Signals Into Risk Themes\n\nClustering groups signals that describe the same underlying event even when they arrive in different languages from different sources.\n\nThree practical points, all of them easy to get wrong.\n\n**Page the scroll.** It returns a batch and an offset, and you keep going until the offset comes back empty. One capped call quietly clusters a busy supplier on partial data.\n\n**Normalize before k-means.** These vectors are built for cosine similarity, but k-means measures Euclidean distance. Without normalizing you are partly clustering by vector length instead of direction.\n\n**Drop the supplier filter to go wider.** A theme shared across suppliers shows up as one cluster pulling in signals from several of them at once.", + "metadata": { + "id": "SIuGIWgIprVF" + } + }, + { + "cell_type": "code", + "metadata": { + "id": "IV4BLFFQCHD6", + "colab": { + "base_uri": "https://localhost:8080/" + }, + "outputId": "bef48fd2" + }, + "execution_count": null, + "source": "from sklearn.cluster import KMeans\n\ndef scroll_all(scroll_filter=None, page=64):\n \"\"\"Page until the offset comes back empty.\"\"\"\n out, offset = [], None\n while True:\n batch, offset = client.scroll(\"supplier_signals\", scroll_filter=scroll_filter,\n with_vectors=True, limit=page, offset=offset)\n out.extend(batch)\n if offset is None:\n return out\n\ndef dense_matrix(points):\n \"\"\"Unit-normalized text_dense vectors, with the ids they belong to.\"\"\"\n ids, vecs = [], []\n for p in points:\n if p.vector and \"text_dense\" in p.vector: # image-only points have none\n ids.append(p.id)\n vecs.append(p.vector[\"text_dense\"])\n if not vecs:\n return [], None\n arr = np.asarray(vecs, dtype=np.float32)\n arr /= np.linalg.norm(arr, axis=1, keepdims=True)\n return ids, arr\n\ntext_only = models.Filter(must=[models.FieldCondition(\n key=\"source_type\", match=models.MatchValue(value=\"news\"))])\n\npts = scroll_all(text_only)\nids, arr = dense_matrix(pts)\nprint(f\"scrolled {len(pts)} points, {len(ids)} carry a text vector\")\n\nlabels = KMeans(n_clusters=3, n_init=10, random_state=42).fit_predict(arr)\n\n# One set_payload call per cluster, not one per point\nfor label in sorted({int(l) for l in labels}):\n client.set_payload(\"supplier_signals\", payload={\"cluster_id\": label},\n points=[i for i, l in zip(ids, labels) if int(l) == label])\n\nby_id = {p.id: p.payload for p in pts}\nfor label in sorted({int(l) for l in labels}):\n print(f\"\\ncluster {label}:\")\n for i, l in zip(ids, labels):\n if int(l) == label:\n p = by_id[i]\n print(f\" [{p['language']}] risk={p['risk_score']:.2f} {p['summary'][:52]}\")", + "outputs": [ + { + "output_type": "stream", + "name": "stdout", + "text": "scrolled 9 points, 9 carry a text vector\n\ncluster 0:\n [vi] risk=0.64 Cang Hai Phong bi tac nghen, cac chuyen hang cua nha\n\ncluster 1:\n [en] risk=0.12 The supplier reaffirmed full-year guidance and repor\n [en] risk=0.10 Analysts described the quarter as routine, with no c\n [en] risk=0.55 Ticker SUP7291.T slipped modestly on higher freight \n [en] risk=0.20 A European logistics operator announced a routine ex\n\ncluster 2:\n [ja] risk=0.88 工場火災により生産が停止し、出荷の遅延が続いている。復旧の見込みは立っていない。\n [zh] risk=0.82 供应商工厂因火灾停产，交货期限推迟，客户已收到延误通知。\n [ja] risk=0.71 労働争議が長引き、工場の稼働率が低下している。組合との交渉は難航している。\n [zh] risk=0.79 另一家供应商的工厂发生停电，生产线暂时中断。\n" + } + ] + }, + { + "cell_type": "markdown", + "source": "Notice that the clusters cross language boundaries. The Japanese fire report and the Chinese shutdown report land together because they describe the same event, which is exactly what a single multilingual vector space buys you.\n\n### The centroid as a query\n\nA cluster centroid is just another vector, so you can hand it straight back to Qdrant as a query and pull in more of the same theme. That is how you go from \"here are today's clusters\" to \"find everything that looks like this emerging story\".", + "metadata": { + "id": "hBeoePW3jSOi" + } + }, + { + "cell_type": "code", + "metadata": { + "id": "27n7yE9ZkAFv", + "colab": { + "base_uri": "https://localhost:8080/" + }, + "outputId": "35b08705" + }, + "execution_count": null, + "source": "target = int(labels[0])\nmember_rows = [i for i, l in zip(ids, labels) if int(l) == target]\ncentroid = arr[[ids.index(i) for i in member_rows]].mean(axis=0)\ncentroid = centroid / np.linalg.norm(centroid)\n\nprint(f\"querying with the centroid of cluster {target}:\\n\")\nfor r in client.query_points(\"supplier_signals\", query=centroid.tolist(),\n using=\"text_dense\", limit=5).points:\n p = r.payload\n mark = \" <- in the cluster\" if r.id in member_rows else \"\"\n print(f\" {r.score:.3f} [{p['language']}] {p['summary'][:48]}{mark}\")", + "outputs": [ + { + "output_type": "stream", + "name": "stdout", + "text": "querying with the centroid of cluster 2:\n\n 0.962 [zh] 供应商工厂因火灾停产，交货期限推迟，客户已收到延误通知。 <- in the cluster\n 0.961 [zh] 另一家供应商的工厂发生停电，生产线暂时中断。 <- in the cluster\n 0.957 [ja] 工場火災により生産が停止し、出荷の遅延が続いている。復旧の見込みは立っていない。 <- in the cluster\n 0.947 [ja] 労働争議が長引き、工場の稼働率が低下している。組合との交渉は難航している。 <- in the cluster\n 0.798 [vi] Cang Hai Phong bi tac nghen, cac chuyen hang cua\n" + } + ] + }, + { + "cell_type": "markdown", + "source": "## 6. The Analyst Query, and the Mistake That Fails Silently\n\nThe query analysts actually run: hybrid retrieval over dense and sparse, scoped to one supplier, elevated risk only.\n\nOne detail decides whether it works. The filter goes **inside each prefetch**, not on the outer query. Prefetches run first and the outer query is applied to their results, so a top-level filter arrives too late: both retrievers search every supplier and every risk level, and the filter only trims the fused list at the end.\n\nBoth versions run below. Watch the `risk` and `supplier` columns in the broken one.", + "metadata": { + "id": "MFd9W8gqcWqi" + } + }, + { + "cell_type": "code", + "metadata": { + "id": "A4Yqj4JRfdQA", + "colab": { + "base_uri": "https://localhost:8080/" + }, + "outputId": "1fac1d1d" + }, + "execution_count": null, + "source": "def analyst_search(query_text, supplier, min_risk=0.5, limit=5, broken=False):\n dense_q = embed_text(query_text, is_query=True).tolist()\n sparse_q = to_sparse(query_text, is_query=True)\n risk_filter = models.Filter(must=[\n models.FieldCondition(key=\"supplier_id\", match=models.MatchValue(value=supplier)),\n models.FieldCondition(key=\"risk_score\", range=models.Range(gte=min_risk)),\n ])\n if broken:\n return client.query_points(\n \"supplier_signals\",\n prefetch=[\n models.Prefetch(query=dense_q, using=\"text_dense\", limit=50),\n models.Prefetch(query=sparse_q, using=\"text_sparse\", limit=50),\n ],\n query=models.RrfQuery(rrf=models.Rrf()),\n query_filter=risk_filter, # too late\n limit=limit,\n ).points\n return client.query_points(\n \"supplier_signals\",\n prefetch=[\n models.Prefetch(query=dense_q, using=\"text_dense\",\n filter=risk_filter, limit=50),\n models.Prefetch(query=sparse_q, using=\"text_sparse\",\n filter=risk_filter, limit=50),\n ],\n query=models.RrfQuery(rrf=models.Rrf()),\n limit=limit,\n ).points\n\ndef report(title, rows):\n print(title)\n if not rows:\n print(\" (no results)\")\n for r in rows:\n p = r.payload\n print(f\" {r.score:.4f} {p['supplier_id']} risk={p['risk_score']:.2f} \"\n f\"[{p['language']}] {p['summary'][:42]}\")\n print()\n\nreport(\"CORRECT filter inside each prefetch:\",\n analyst_search(\"production halt at the factory\", \"SUP-7291\"))\nreport(\"BROKEN same filter at the top level:\",\n analyst_search(\"production halt at the factory\", \"SUP-7291\", broken=True))", + "outputs": [ + { + "output_type": "stream", + "name": "stdout", + "text": "CORRECT filter inside each prefetch:\n 0.5000 SUP-7291 risk=0.82 [zh] 供应商工厂因火灾停产，交货期限推迟，客户已收到延误通知。\n 0.3333 SUP-7291 risk=0.88 [ja] 工場火災により生産が停止し、出荷の遅延が続いている。復旧の見込みは立っていない。\n 0.2500 SUP-7291 risk=0.55 [en] Ticker SUP7291.T slipped modestly on highe\n 0.2000 SUP-7291 risk=0.71 [ja] 労働争議が長引き、工場の稼働率が低下している。組合との交渉は難航している。\n 0.1667 SUP-7291 risk=0.64 [vi] Cang Hai Phong bi tac nghen, cac chuyen ha\n\n" + }, + { + "output_type": "stream", + "name": "stdout", + "text": "BROKEN same filter at the top level:\n 0.5000 SUP-0002 risk=0.79 [zh] 另一家供应商的工厂发生停电，生产线暂时中断。\n 0.3333 SUP-7291 risk=0.82 [zh] 供应商工厂因火灾停产，交货期限推迟，客户已收到延误通知。\n 0.2500 SUP-7291 risk=0.88 [ja] 工場火災により生産が停止し、出荷の遅延が続いている。復旧の見込みは立っていない。\n 0.2000 SUP-7291 risk=0.55 [en] Ticker SUP7291.T slipped modestly on highe\n 0.1667 SUP-7291 risk=0.71 [ja] 労働争議が長引き、工場の稼働率が低下している。組合との交渉は難航している。\n\n" + } + ] + }, + { + "cell_type": "markdown", + "source": "The broken version returns rows that violate the filter, with no error and no warning. On a real corpus, where one supplier is a thousandth of the collection instead of most of it, the same mistake usually returns nothing at all, and an analyst concludes there is no news.\n\nThe rule, one more time. No prefetch, use `query_filter`. Prefetch, put the filter in every prefetch.", + "metadata": { + "id": "CvI5qiGogkdm" + } + }, + { + "cell_type": "markdown", + "source": "## 7. Reading the Gap\n\nThe payoff. Run the same query twice, once scoped to English sources and once to Japanese and Chinese, and compare.\n\nThis is not a different algorithm. It is the same query and the same vector space with a different `language` filter.", + "metadata": { + "id": "tsVrFlvbpaTI" + } + }, + { + "cell_type": "code", + "metadata": { + "id": "lD5gYAEZyFQ8", + "colab": { + "base_uri": "https://localhost:8080/" + }, + "outputId": "5dba6502" + }, + "execution_count": null, + "source": "query = \"factory shutdown, production halted, delivery delays\"\n\nreport(\"ENGLISH sources only:\",\n text_search(query, languages=[\"en\"], supplier=\"SUP-7291\"))\nreport(\"JAPANESE and CHINESE sources only:\",\n text_search(query, languages=[\"ja\", \"zh\"], supplier=\"SUP-7291\"))", + "outputs": [ + { + "output_type": "stream", + "name": "stdout", + "text": "ENGLISH sources only:\n 0.7789 SUP-7291 risk=0.55 [en] Ticker SUP7291.T slipped modestly on highe\n 0.7239 SUP-7291 risk=0.12 [en] The supplier reaffirmed full-year guidance\n 0.6827 SUP-7291 risk=0.10 [en] Analysts described the quarter as routine,\n\nJAPANESE and CHINESE sources only:\n 0.8059 SUP-7291 risk=0.82 [zh] 供应商工厂因火灾停产，交货期限推迟，客户已收到延误通知。\n 0.7946 SUP-7291 risk=0.88 [ja] 工場火災により生産が停止し、出荷の遅延が続いている。復旧の見込みは立っていない。\n 0.7731 SUP-7291 risk=0.71 [ja] 労働争議が長引き、工場の稼働率が低下している。組合との交渉は難航している。\n\n" + } + ] + }, + { + "cell_type": "markdown", + "source": "The English sources are reporting reaffirmed guidance and a routine quarter. The Japanese and Chinese sources are reporting a fire, halted production, and delayed deliveries, and they were published earlier.\n\nThat gap is the early warning, and finding it needed no translation pipeline, no separate per-language index, and no second database. One collection, one multilingual model, one filter.", + "metadata": { + "id": "eTpkf01PrpsC" + } + }, + { + "cell_type": "markdown", + "source": "## Course Close\n\nLook at what is in that collection. Text, sparse tokens, and imagery on shared points. Filters that hold. Hybrid retrieval. Clustering. Cross-language and cross-modal search.\n\nSix primitives got you here: collection, point, vector, payload, index, query.\n\nModule 1 asked why keyword search misses. Module 2 opened up the vector. Module 3 put dense and sparse together. Module 4 turned that into a design you could defend. This module ran the whole thing on three modalities at once.\n\nThe next system someone hands you is these six, arranged differently.\n\n### Your turn\n\nSwap the synthetic tiles for real satellite imagery and rerun Section 4, then check whether the smoke query starts behaving.\n\nAdd a `tenant_id` field with `is_tenant=True` and scope every query to one desk, the way Module 4 did.\n\nThen try `models.Rrf(weights=[3.0, 1.0])` in the analyst query to favour dense over sparse, and see which retriever your data actually prefers.", + "metadata": { + "id": "jBq78Hp7twax" + } + } + ] +}