Site Logo

Building a Document Search Engine with LangChain and OpenAI: A Comprehensive Guide

Document search engines powered by large language models (LLMs) and vector-based retrieval enable fast, accurate searches across large document collections, making them essential for research, enterprise knowledge management, and customer support. By combining LangChain and OpenAI, you can create a robust search engine that retrieves relevant documents and generates context-aware responses.

Introduction to Document Search Engines and LangChain

A document search engine retrieves relevant documents based on a user’s query, often using vector embeddings for semantic search and LLMs for response generation. Unlike traditional keyword-based search, semantic search understands query intent, making it ideal for complex document sets. LangChain simplifies this with tools for document loading, vector storage, and retrieval chains. OpenAI’s API, powering models like gpt-3.5-turbo, enhances generation, while LangChain manages retrieval and context.

This tutorial assumes basic Python knowledge, with references to LangChain’s getting started guide, OpenAI’s API documentation, and FAISS documentation.

Prerequisites for Building the Document Search Engine

Ensure you have:

pip install langchain openai faiss-cpu pypdf
  • Development Environment: Use a virtual environment, as detailed in LangChain’s environment setup guide.
  • Sample Documents: Prepare a collection of PDFs or text files (e.g., articles, manuals) in a directory.
  • Basic Python Knowledge: Familiarity with syntax and package installation, with resources in Python’s documentation.

Step 1: Setting Up the Development Environment

Configure your environment by importing libraries and setting the OpenAI API key.

import os
from glob import glob
from langchain_openai import ChatOpenAI
from langchain.embeddings.openai import OpenAIEmbeddings
from langchain.vectorstores import FAISS
from langchain.document_loaders import PyPDFLoader
from langchain.chains import RetrievalQA
from langchain.text_splitter import RecursiveCharacterTextSplitter

# Set your OpenAI API key
os.environ["OPENAI_API_KEY"] = "your-openai-api-key"

Replace "your-openai-api-key" with your actual key. Environment variables enhance security, as explained in LangChain’s security and API keys guide. The imported modules are core to the search engine, detailed in LangChain’s core components overview.

Step 2: Loading and Processing Documents

Load multiple documents (e.g., PDFs) from a directory and split them into chunks for efficient retrieval.

# Load all PDFs from a directory
pdf_files = glob("path/to/documents/*.pdf")
documents = []
for pdf_file in pdf_files:
    loader = PyPDFLoader(
        file_path=pdf_file,
        extract_images=False
    )
    documents.extend(loader.load())

# Split documents into chunks
text_splitter = RecursiveCharacterTextSplitter(
    chunk_size=1000,
    chunk_overlap=200,
    length_function=len,
    add_start_index=True
)
docs = text_splitter.split_documents(documents)

Key Parameters for PyPDFLoader

  • file_path: Path to a PDF file. Iterated over multiple files.
  • extract_images: If True, extracts images as text (requires dependencies). False for text-only.

Key Parameters for RecursiveCharacterTextSplitter

  • chunk_size: Maximum characters per chunk (e.g., 1000). Ensures manageable retrieval.
  • chunk_overlap: Overlapping characters (e.g., 200). Preserves context.
  • length_function: Measures text length (default: len).
  • add_start_index: If True, includes chunk start index in metadata.

For other document types, see LangChain’s document loaders.

Step 3: Creating Embeddings and Vector Store

Convert document chunks into embeddings and store them in a FAISS vector database for semantic search.

embeddings = OpenAIEmbeddings(
    model="text-embedding-ada-002",
    chunk_size=1000,
    max_retries=3
)
vectorstore = FAISS.from_documents(
    documents=docs,
    embedding=embeddings,
    distance_strategy="COSINE",
    normalize_L2=True
)

Key Parameters for OpenAIEmbeddings

  • model: Embedding model (e.g., text-embedding-ada-002). Determines vector quality.
  • chunk_size: Texts processed per API call (e.g., 1000). Balances speed and limits.
  • max_retries: Retry attempts for API failures (e.g., 3). Enhances reliability.

Key Parameters for FAISS.from_documents

  • documents: Document chunks to embed.
  • embedding: Embedding model instance.
  • distance_strategy: Similarity metric (e.g., "COSINE", "L2", "IP"). "COSINE" suits semantic search.
  • normalize_L2: If True, normalizes vectors for consistent similarity scores.

For alternatives, explore Pinecone or Weaviate.

Step 4: Initializing the Language Model

Initialize the OpenAI LLM for response generation.

llm = ChatOpenAI(
    model_name="gpt-3.5-turbo",
    temperature=0.5,
    max_tokens=512,
    top_p=0.9,
    frequency_penalty=0.2,
    presence_penalty=0.1,
    n=1
)

Key Parameters for ChatOpenAI

  • model_name: OpenAI model (e.g., gpt-3.5-turbo, gpt-4). gpt-3.5-turbo is efficient; gpt-4 excels in reasoning. See OpenAI’s model documentation.
  • temperature (0.0–2.0): Controls randomness. At 0.5, balances coherence and creativity.
  • max_tokens: Maximum response length (e.g., 512). Adjust for detail vs. cost. See LangChain’s token limit handling.
  • top_p (0.0–1.0): Nucleus sampling. At 0.9, focuses on likely tokens.
  • frequency_penalty (–2.0–2.0): Discourages repetition. At 0.2, promotes variety.
  • presence_penalty (–2.0–2.0): Encourages new topics. At 0.1, mild novelty boost.
  • n: Number of responses to generate (e.g., 1). Higher values return multiple options.

See LangChain’s OpenAI integration guide.

Step 5: Building the RetrievalQA Chain

Create a RetrievalQA chain to combine retrieval and generation.

qa_chain = RetrievalQA.from_chain_type(
    llm=llm,
    chain_type="stuff",
    retriever=vectorstore.as_retriever(
        search_type="similarity",
        search_kwargs={"k": 5, "fetch_k": 10}
    ),
    return_source_documents=True,
    verbose=True,
    input_key="query",
    output_key="result"
)

Key Parameters for RetrievalQA.from_chain_type

  • llm: The initialized LLM.
  • chain_type: Document processing method (e.g., "stuff", "map_reduce", "refine"). "stuff" combines documents into one prompt.
  • retriever: Retrieval mechanism.
  • return_source_documents: If True, includes retrieved documents.
  • verbose: If True, logs execution.
  • input_key: Input variable name (e.g., "query").
  • output_key: Output variable name (e.g., "result").

Key Parameters for as_retriever

  • search_type: Retrieval method (e.g., "similarity", "mmr"). "similarity" prioritizes closest matches.
  • search_kwargs: Settings, e.g., k (top results, 5), fetch_k (initial candidates, 10).

See LangChain’s RetrievalQA chain guide.

Step 6: Querying the Search Engine

Test the search engine with queries.

query = "What are the key insights from the documents?"
response = qa_chain({"query": query})
print(response["result"])
print("Sources:", [doc.metadata for doc in response["source_documents"]])

Example Output:

The documents highlight insights on AI-driven automation, focusing on efficiency, scalability, and ethical considerations in enterprise applications.
Sources: [
    {'page': 4, 'source': 'doc1.pdf', 'start_index': 0},
    {'page': 15, 'source': 'doc2.pdf', 'start_index': 1200},
    {'page': 8, 'source': 'doc3.pdf', 'start_index': 500}
]

The chain retrieves relevant chunks and generates a response. For examples, see LangChain’s document QA chain or conversational flows.

Step 7: Customizing the Search Engine

Enhance with custom prompts, additional data, or tools.

7.1 Custom Prompt Engineering

Modify the prompt for precise responses.

from langchain.prompts import PromptTemplate

custom_prompt = PromptTemplate(
    input_variables=["context", "query"],
    template="You are a technical analyst. Provide a concise, evidence-based answer based on the context:\n\n{context}\n\nQuery: {query}\n\nAnswer: ",
    validate_template=True
)

qa_chain = RetrievalQA.from_chain_type(
    llm=llm,
    chain_type="stuff",
    retriever=vectorstore.as_retriever(search_kwargs={"k": 5}),
    return_source_documents=True,
    verbose=True,
    prompt=custom_prompt
)

PromptTemplate Parameters:

  • input_variables: Variables (e.g., ["context", "query"]).
  • template: Defines tone and structure.
  • validate_template: If True, validates variables.

See LangChain’s prompt templates guide.

7.2 Adding Diverse Data Sources

Incorporate web content using LangChain’s web loaders.

from langchain.document_loaders import WebBaseLoader

web_loader = WebBaseLoader(
    web_path="https://example.com/articles",
    verify_ssl=True
)
web_docs = web_loader.load()
all_docs = docs + text_splitter.split_documents(web_docs)
vectorstore = FAISS.from_documents(all_docs, embeddings)

WebBaseLoader Parameters:

  • web_path: URL to load.
  • verify_ssl: If True, enforces SSL verification.

7.3 Tool Integration with Agents

Add tools like SerpAPI for real-time data.

from langchain.agents import initialize_agent, Tool
from langchain.tools import SerpAPIWrapper

search = SerpAPIWrapper()
tools = [
    Tool(
        name="Search",
        func=search.run,
        description="Fetch current information."
    )
]

agent = initialize_agent(
    tools=tools,
    llm=llm,
    agent="zero-shot-react-description",
    verbose=True,
    max_iterations=3,
    early_stopping_method="force"
)

response = agent.run("Supplement document insights with recent trends.")
print(response)

initialize_agent Parameters:

  • tools: List of tools.
  • llm: The LLM.
  • agent: Agent type.
  • verbose: If True, logs decisions.
  • max_iterations: Limits steps.
  • early_stopping_method: Stops execution.

See LangChain’s agents guide.

Step 8: Deploying the Search Engine

Deploy as a Streamlit app for an interactive interface.

import streamlit as st

st.title("Document Search Engine")
st.write("Search across your document collection!")

if "messages" not in st.session_state:
    st.session_state.messages = []

for message in st.session_state.messages:
    with st.chat_message(message["role"]):
        st.markdown(message["content"])

if query := st.chat_input("Enter your search query:"):
    st.session_state.messages.append({"role": "user", "content": query})
    with st.chat_message("user"):
        st.markdown(query)
    with st.chat_message("assistant"):
        with st.spinner("Searching..."):
            response = qa_chain({"query": query})
            st.markdown(response["result"])
            st.write("**Sources:**")
            for doc in response["source_documents"]:
                st.write(f"- {doc.metadata['source']} (Page {doc.metadata['page']})")
            st.session_state.messages.append({"role": "assistant", "content": response["result"]})

Save as app.py, install Streamlit (pip install streamlit), and run:

streamlit run app.py

Visit http://localhost:8501. Deploy to Streamlit Community Cloud by pushing to GitHub and configuring secrets. See LangChain’s Streamlit tutorial or Streamlit’s deployment guide.

Step 9: Evaluating and Testing the Search Engine

Evaluate responses using LangChain’s evaluation metrics.

from langchain.evaluation import load_evaluator

evaluator = load_evaluator(
    "qa",
    criteria=["correctness", "relevance"]
)
result = evaluator.evaluate_strings(
    prediction="The documents focus on AI automation and ethics.",
    input="What are the key insights?",
    reference="The documents discuss AI-driven automation, efficiency, and ethical considerations."
)
print(result)

load_evaluator Parameters:

  • evaluator_type: Metric type (e.g., "qa").
  • criteria: Evaluation criteria.

Test with diverse queries. Debug with LangSmith per LangChain’s LangSmith intro.

Advanced Features and Next Steps

Enhance with:

See LangChain’s startup examples or GitHub repos.

Conclusion

Building a document search engine with LangChain and OpenAI enables semantic search and context-aware responses across large document sets. This guide covered setup, document processing, vector storage, LLM integration, deployment, evaluation, and parameters. Leverage LangChain’s chains, vector stores, and integrations to create powerful search engines.

Explore agents, tools, or evaluation metrics. Debug with LangSmith. Happy coding!