|
5 | 5 | "metadata": { |
6 | 6 | "id": "kTas9ZQ7lXP7" |
7 | 7 | }, |
8 | | - "source": "# Tutorial: Creating a Hybrid Retrieval Pipeline\n\n- **Level**: Intermediate\n- **Time to complete**: 15 minutes\n- **Components Used**: [`DocumentSplitter`](https://docs.haystack.deepset.ai/docs/documentsplitter), [`SentenceTransformersDocumentEmbedder`](https://docs.haystack.deepset.ai/docs/sentencetransformersdocumentembedder), [`InMemoryDocumentStore`](https://docs.haystack.deepset.ai/docs/inmemorydocumentstore), [`InMemoryBM25Retriever`](https://docs.haystack.deepset.ai/docs/inmemorybm25retriever), [`InMemoryEmbeddingRetriever`](https://docs.haystack.deepset.ai/docs/inmemoryembeddingretriever), and [`SentenceTransformersSimilarityRanker`](https://docs.haystack.deepset.ai/docs/sentencetransformerssimilarityranker)\n- **Prerequisites**: None\n- **Goal**: After completing this tutorial, you will have learned about creating a hybrid retrieval and when it's useful." |
| 8 | + "source": [ |
| 9 | + "# Tutorial: Creating a Hybrid Retrieval Pipeline\n", |
| 10 | + "\n", |
| 11 | + "- **Level**: Intermediate\n", |
| 12 | + "- **Time to complete**: 15 minutes\n", |
| 13 | + "- **Components Used**: [`DocumentSplitter`](https://docs.haystack.deepset.ai/docs/documentsplitter), [`SentenceTransformersDocumentEmbedder`](https://docs.haystack.deepset.ai/docs/sentencetransformersdocumentembedder), [`InMemoryDocumentStore`](https://docs.haystack.deepset.ai/docs/inmemorydocumentstore), [`InMemoryBM25Retriever`](https://docs.haystack.deepset.ai/docs/inmemorybm25retriever), [`InMemoryEmbeddingRetriever`](https://docs.haystack.deepset.ai/docs/inmemoryembeddingretriever), and [`SentenceTransformersSimilarityRanker`](https://docs.haystack.deepset.ai/docs/sentencetransformerssimilarityranker)\n", |
| 14 | + "- **Prerequisites**: None\n", |
| 15 | + "- **Goal**: After completing this tutorial, you will have learned about creating a hybrid retrieval and when it's useful." |
| 16 | + ] |
9 | 17 | }, |
10 | 18 | { |
11 | 19 | "cell_type": "markdown", |
|
38 | 46 | "id": "L40ZxZW8lXQh" |
39 | 47 | }, |
40 | 48 | "outputs": [], |
41 | | - "source": "%%bash\n\npip install haystack-ai\npip install \"datasets>=2.6.1\"\npip install sentence-transformers-haystack" |
| 49 | + "source": [ |
| 50 | + "%%bash\n", |
| 51 | + "\n", |
| 52 | + "pip install haystack-ai\n", |
| 53 | + "pip install \"datasets>=2.6.1\"\n", |
| 54 | + "pip install sentence-transformers-haystack" |
| 55 | + ] |
42 | 56 | }, |
43 | 57 | { |
44 | 58 | "cell_type": "markdown", |
|
144 | 158 | "\n", |
145 | 159 | "document_splitter = DocumentSplitter(split_by=\"word\", split_length=512, split_overlap=32)\n", |
146 | 160 | "document_embedder = SentenceTransformersDocumentEmbedder(\n", |
147 | | - " model=\"BAAI/bge-small-en-v1.5\", device=ComponentDevice.from_str(\"cuda:0\")\n", |
| 161 | + " model=\"BAAI/bge-small-en-v1.5\", device=ComponentDevice.from_str(\"cuda:0\") # remove the device argument if you are not using a GPU\n", |
148 | 162 | ")\n", |
149 | 163 | "document_writer = DocumentWriter(document_store)\n", |
150 | 164 | "\n", |
|
194 | 208 | }, |
195 | 209 | { |
196 | 210 | "cell_type": "code", |
197 | | - "execution_count": 7, |
| 211 | + "execution_count": null, |
198 | 212 | "metadata": { |
199 | 213 | "id": "DVfQvnWYrMWr" |
200 | 214 | }, |
|
204 | 218 | "from haystack_integrations.components.embedders.sentence_transformers import SentenceTransformersTextEmbedder\n", |
205 | 219 | "\n", |
206 | 220 | "text_embedder = SentenceTransformersTextEmbedder(\n", |
207 | | - " model=\"BAAI/bge-small-en-v1.5\", device=ComponentDevice.from_str(\"cuda:0\")\n", |
| 221 | + " model=\"BAAI/bge-small-en-v1.5\", device=ComponentDevice.from_str(\"cuda:0\") # remove the device argument if you are not using a GPU\n", |
208 | 222 | ")\n", |
209 | 223 | "embedding_retriever = InMemoryEmbeddingRetriever(document_store)\n", |
210 | 224 | "bm25_retriever = InMemoryBM25Retriever(document_store)" |
|
215 | 229 | "metadata": { |
216 | 230 | "id": "r8_jHzmosbC_" |
217 | 231 | }, |
218 | | - "source": "### 2) Rank the Results\n\nUse the [SentenceTransformersSimilarityRanker](https://docs.haystack.deepset.ai/docs/sentencetransformerssimilarityranker) that scores the relevancy of all retrieved documents for the given search query by using a cross encoder model. In this example, you will use [BAAI/bge-reranker-base](https://huggingface.co/BAAI/bge-reranker-base) model to rank the retrieved documents but you can replace this model with other cross-encoder models on Hugging Face." |
| 232 | + "source": [ |
| 233 | + "### 2) Rank the Results\n", |
| 234 | + "\n", |
| 235 | + "Use the [SentenceTransformersSimilarityRanker](https://docs.haystack.deepset.ai/docs/sentencetransformerssimilarityranker) that scores the relevancy of all retrieved documents for the given search query by using a cross encoder model. In this example, you will use [BAAI/bge-reranker-base](https://huggingface.co/BAAI/bge-reranker-base) model to rank the retrieved documents but you can replace this model with other cross-encoder models on Hugging Face." |
| 236 | + ] |
219 | 237 | }, |
220 | 238 | { |
221 | 239 | "cell_type": "code", |
|
224 | 242 | "id": "cN0woIxHs4Ng" |
225 | 243 | }, |
226 | 244 | "outputs": [], |
227 | | - "source": "from haystack_integrations.components.rankers.sentence_transformers import SentenceTransformersSimilarityRanker\n\nranker = SentenceTransformersSimilarityRanker(model=\"BAAI/bge-reranker-base\")" |
| 245 | + "source": [ |
| 246 | + "from haystack_integrations.components.rankers.sentence_transformers import SentenceTransformersSimilarityRanker\n", |
| 247 | + "\n", |
| 248 | + "ranker = SentenceTransformersSimilarityRanker(model=\"BAAI/bge-reranker-base\")" |
| 249 | + ] |
228 | 250 | }, |
229 | 251 | { |
230 | 252 | "cell_type": "markdown", |
|
0 commit comments