LangChain + YouTube transcripts: build a knowledge base

LangChain is the go-to framework for building LLM-powered apps. It handles chains, agents, vector stores, and document loaders so you can focus on your application logic.

But connecting LangChain to YouTube data? That has always been a weak spot.

LangChain's built-in YouTubeLoader wraps the open-source youtube-transcript-api Python library. It works in demo notebooks. It breaks in production. And it only handles single video transcripts, with no support for channels, playlists, or search.

In this tutorial, you will build a complete LangChain knowledge base powered by YouTube transcripts. We will replace the fragile built-in loader with a custom document loader that calls TranscriptAPI, giving you reliable transcript extraction plus search and channel capabilities.

By the end, you will have a system that ingests transcripts from any YouTube channel, stores them as embeddings, and answers natural-language questions with source citations.

Four steps. Real code. Let's go.

Why LangChain's built-in YouTube loader falls short

Before building our custom loader, it is worth understanding why the default one causes problems.

The problem with YouTubeLoader

LangChain ships with a YouTubeLoader class. Under the hood, it wraps youtube-transcript-api, which is a Python library that talks directly to YouTube's undocumented internal API.

That means three things:

1. It breaks when YouTube changes its frontend code (which happens regularly)

2. Your server's IP address makes the requests, and YouTube blocks cloud IPs aggressively

3. You get no channel listing, no playlist support, no search. Just single-video transcript extraction

I have seen teams build entire prototypes on YouTubeLoader, get excited, deploy to production, and watch everything crash within a week. The library's GitHub issues page is full of reports like "TranscriptsDisabled error on videos that clearly have captions."

That is frustrating. Especially when your pipeline was working fine yesterday.

Using TranscriptAPI as a custom document loader

The fix is to build a custom LangChain document loader that calls TranscriptAPI's REST endpoints instead.

You get:

• Reliable transcript extraction (49ms median response time, 15M+ transcripts processed per month)

• YouTube search, channel listing, channel search, and playlist extraction

• Structured error handling with clear error codes

• Works from any programming language, not just Python

And the best part: LangChain makes custom loaders easy to build. It is about 50 lines of code.

Step 1: build a custom TranscriptAPI document loader

The loader class

Here is the core loader. It extends LangChain's BaseLoader and returns Document objects:

import os
import requests
from typing import List, Optional
from langchain.docstore.document import Document
from langchain.document_loaders.base import BaseLoader


class TranscriptAPILoader(BaseLoader):
    """Load YouTube transcripts via TranscriptAPI."""

    def __init__(self, video_urls: List[str], api_key: Optional[str] = None):
        self.video_urls = video_urls
        self.api_key = api_key or os.environ["TRANSCRIPTAPI_KEY"]

    def load(self) -> List[Document]:
        docs = []
        for url in self.video_urls:
            try:
                response = requests.get(
                    "https://transcriptapi.com/api/v2/youtube/transcript",
                    params={
                        "video_url": url,
                        "format": "json",
                        "send_metadata": "true"
                    },
                    headers={
                        "Authorization": f"Bearer {self.api_key}"
                    }
                )
                response.raise_for_status()
                data = response.json()

                text = " ".join(
                    seg["text"] for seg in data["transcript"]
                )

                docs.append(Document(
                    page_content=text,
                    metadata={
                        "video_id": data["video_id"],
                        "title": data.get("title", ""),
                        "author": data.get("author_name", ""),
                        "source_url": url,
                        "language": data.get("language", "en")
                    }
                ))
            except requests.HTTPError as e:
                if e.response.status_code == 404:
                    print(f"No captions available: {url}")
                    continue
                raise

        return docs

Usage is just like any other LangChain loader:

loader = TranscriptAPILoader(
    video_urls=[
        "https://www.youtube.com/watch?v=VIDEO_ID_1",
        "https://www.youtube.com/watch?v=VIDEO_ID_2"
    ]
)

docs = loader.load()
print(f"Loaded {len(docs)} transcripts")

The loader handles errors cleanly. If a video has no captions (404), it logs a message and moves on. Your pipeline does not crash because one video out of 500 lacks transcripts.

Compare this to the built-in YouTubeLoader:

# Built-in LangChain loader (fragile)
from langchain.document_loaders import YoutubeLoader

loader = YoutubeLoader.from_youtube_url(
    "https://www.youtube.com/watch?v=VIDEO_ID"
)
docs = loader.load()  # might crash with IP block or parser error

Same number of lines. But our custom loader calls a managed API instead of scraping YouTube directly. That is the difference between "works in my notebook" and "works in production."

Adding channel and playlist support

Single videos are fine for testing. For a real knowledge base, you want entire channels.

class TranscriptAPIChannelLoader(BaseLoader):
    """Load all transcripts from a YouTube channel."""

    def __init__(self, channel: str, api_key: Optional[str] = None):
        self.channel = channel
        self.api_key = api_key or os.environ["TRANSCRIPTAPI_KEY"]

    def _get_channel_videos(self) -> List[str]:
        """Fetch all video IDs from the channel."""
        video_ids = []
        params = {"channel": self.channel}

        while True:
            response = requests.get(
                "https://transcriptapi.com/api/v2/youtube/channel/videos",
                params=params,
                headers={
                    "Authorization": f"Bearer {self.api_key}"
                }
            )
            response.raise_for_status()
            data = response.json()

            for video in data["videos"]:
                video_ids.append(video["videoId"])

            if not data.get("has_more"):
                break
            params = {"continuation": data["continuation_token"]}

        return video_ids

    def load(self) -> List[Document]:
        video_ids = self._get_channel_videos()
        print(f"Found {len(video_ids)} videos in {self.channel}")

        urls = [
            f"https://youtube.com/watch?v={vid}"
            for vid in video_ids
        ]

        loader = TranscriptAPILoader(
            video_urls=urls,
            api_key=self.api_key
        )
        return loader.load()

Now you can load an entire channel in two lines:

loader = TranscriptAPIChannelLoader(channel="@3blue1brown")
docs = loader.load()
print(f"Loaded {len(docs)} transcripts from the channel")

The channel videos endpoint returns about 100 videos per page and supports pagination through continuation tokens. For a channel with 500 videos, that is 5 API calls to list the videos plus 500 calls for the transcripts. Total cost: about 505 credits.

Want to load a playlist instead? The same pattern works with the /youtube/playlist/videos endpoint. Just swap the channel parameter for a playlist ID.

Have you tried loading a full channel with LangChain's built-in loader? It does not even support it. You would have to manually collect every video URL first, probably by scraping the channel page or using a separate YouTube API. That is two tools to do what our loader does in one.

The channel and playlist support is where TranscriptAPI really pulls ahead. Most YouTube transcript tools only handle single videos. But knowledge bases need hundreds or thousands of videos. You need an API that thinks about scale from the start.

Step 2: chunk and embed the transcripts

Raw transcripts from a full video can be 5,000 to 10,000 words. That is way too long for an embedding model to handle as a single chunk. You need to split them.

Choosing a text splitter

LangChain has several text splitters. For transcripts, RecursiveCharacterTextSplitter works well:

from langchain.text_splitter import RecursiveCharacterTextSplitter

splitter = RecursiveCharacterTextSplitter(
    chunk_size=1000,
    chunk_overlap=200,
    separators=[". ", "? ", "! ", "\n", " "]
)

# Split all documents
chunks = splitter.split_documents(docs)
print(f"Split {len(docs)} transcripts into {len(chunks)} chunks")

Why chunk_size=1000 and chunk_overlap=200? These are character counts, not word counts.

A 1,000-character chunk is roughly 150-200 words. That is long enough to carry a complete thought from a speaker but short enough for precise retrieval.

The 200-character overlap means consecutive chunks share about 30 words. This prevents losing context at boundaries. If a speaker makes a point that spans the chunk boundary, both chunks will contain enough of it to be retrievable.

The separator list matters too. We prioritize splitting on sentence boundaries (period, question mark, exclamation mark) before resorting to spaces. That keeps sentences intact.

You can also use TranscriptAPI's timestamp data for more natural chunking. Timestamps mark where each spoken segment begins, which often aligns with topic shifts. Check our RAG pipeline guide for a timestamp-based chunking approach.

Creating embeddings

Now turn those text chunks into vectors:

from langchain_openai import OpenAIEmbeddings

embeddings = OpenAIEmbeddings(model="text-embedding-3-small")

That is the setup. OpenAI's text-embedding-3-small model is the best balance of quality and cost at $0.02 per million tokens.

For self-hosted deployments where you do not want to send data to OpenAI:

from langchain.embeddings import HuggingFaceEmbeddings

embeddings = HuggingFaceEmbeddings(
    model_name="sentence-transformers/all-MiniLM-L6-v2"
)

This runs locally. No API calls. No cost per embedding. The trade-off is slightly lower quality than OpenAI's model, but it is free and private.

For large transcript collections (1,000+ videos), batch your embedding calls. LangChain handles this automatically when you use from_documents, but be aware that embedding 10,000 chunks takes a few minutes even with API-based models.

Which model should you pick? If you are building a commercial product, go with OpenAI. The quality is worth the small cost. If you are building something for personal use or you have privacy requirements, HuggingFace gives you full control over your data.

Either way, pick one and stick with it. You cannot mix embedding models in the same vector store. The dimensions are different, and the vector spaces are not compatible.

Step 3: store in a vector database

You have chunks. You have an embedding model. Now you need somewhere to store the vectors.

Local development with Chroma

Chroma is the easiest vector database to set up. Zero configuration. Runs in-process.

from langchain.vectorstores import Chroma

vectorstore = Chroma.from_documents(
    documents=chunks,
    embedding=embeddings,
    persist_directory="./youtube_kb",
    collection_name="transcripts"
)

# Verify it works
results = vectorstore.similarity_search(
    "What is gradient descent?",
    k=3
)

for doc in results:
    print(f"[{doc.metadata['title']}] {doc.page_content[:100]}...")

For local development and prototyping, Chroma is hard to beat. Your data persists to disk, so you do not need to re-embed on every restart.

Production setup with Pinecone or Weaviate

For production, use a managed vector database. Here is Pinecone:

from langchain.vectorstores import Pinecone
from pinecone import Pinecone as PineconeClient

pc = PineconeClient(api_key=os.environ["PINECONE_API_KEY"])

vectorstore = Pinecone.from_documents(
    documents=chunks,
    embedding=embeddings,
    index_name="youtube-knowledge-base"
)

Pinecone handles scaling, backups, and high-availability. You do not manage any infrastructure.

For Weaviate, the pattern is similar. LangChain supports both as first-class vector store backends.

Whichever you choose, make sure your index dimensions match your embedding model. OpenAI text-embedding-3-small uses 1,536 dimensions. HuggingFace all-MiniLM-L6-v2 uses 384.

Adding new videos later is simple. Embed the new chunks and upsert them into the existing vector store. No need to rebuild the entire index.

One production tip: keep a record of which video IDs you have already ingested. Before processing a new batch, check against this list to avoid duplicate chunks in your vector store. A simple SQLite database or even a JSON file works for tracking this.

Step 4: query the knowledge base

The fun part. Asking questions and getting answers.

Building a RetrievalQA chain

from langchain.chains import RetrievalQA
from langchain_openai import ChatOpenAI

llm = ChatOpenAI(model="gpt-4o", temperature=0)

qa_chain = RetrievalQA.from_chain_type(
    llm=llm,
    chain_type="stuff",
    retriever=vectorstore.as_retriever(search_kwargs={"k": 5}),
    return_source_documents=True
)

That is your complete question-answering system. The retriever finds the 5 most relevant transcript chunks. The LLM reads them and generates an answer.

Example queries and results

# Ask a question
result = qa_chain.invoke(
    "How does backpropagation work?"
)

print("Answer:", result["result"])
print("\nSources:")
for doc in result["source_documents"]:
    vid = doc.metadata.get("video_id", "unknown")
    title = doc.metadata.get("title", "untitled")
    print(f"  - {title} (youtube.com/watch?v={vid})")

You can also filter by channel or date. LangChain's retriever supports metadata filtering:

# Only search within a specific channel
retriever = vectorstore.as_retriever(
    search_kwargs={
        "k": 5,
        "filter": {"author": "3Blue1Brown"}
    }
)

When the knowledge base has no relevant content for a query, the LLM should say so. Add instructions to your prompt template like "If the provided context does not contain enough information to answer, say you don't have that information."

This prevents hallucination. Your knowledge base is grounded in actual video content, and the system should be honest when a question falls outside that scope.

Conversation memory

Want your knowledge base to support follow-up questions? Use LangChain's ConversationalRetrievalChain:

from langchain.chains import ConversationalRetrievalChain
from langchain.memory import ConversationBufferMemory

memory = ConversationBufferMemory(
    memory_key="chat_history",
    return_messages=True
)

conv_chain = ConversationalRetrievalChain.from_llm(
    llm=llm,
    retriever=vectorstore.as_retriever(search_kwargs={"k": 5}),
    memory=memory,
    return_source_documents=True
)

# First question
result = conv_chain.invoke({"question": "What is attention in transformers?"})
# Follow-up
result = conv_chain.invoke({"question": "How does multi-head attention differ?"})

The memory keeps track of the conversation, so follow-up questions understand the context. The user does not have to repeat themselves.

This is useful for building chatbot-style interfaces over your YouTube knowledge base. Users can ask a question, get an answer, and dig deeper naturally.

Going further: agent-based workflows

Once you have a working knowledge base, you can build on top of it.

Adding search and real-time ingestion

Create a LangChain agent that can search YouTube, ingest new transcripts, and query the knowledge base in a single conversation:

from langchain.agents import tool

@tool
def search_youtube(query: str) -> str:
    """Search YouTube for videos matching a query."""
    response = requests.get(
        "https://transcriptapi.com/api/v2/youtube/search",
        params={"q": query, "limit": 5},
        headers={
            "Authorization": f"Bearer {os.environ['TRANSCRIPTAPI_KEY']}"
        }
    )
    data = response.json()
    results = []
    for v in data["results"]:
        results.append(f"{v['title']} ({v['videoId']})")
    return "\n".join(results)

Now your agent can search for videos on a topic, ingest the transcripts into the knowledge base, and answer questions about them. The knowledge base grows as users ask questions. That is a powerful pattern.

Multi-channel knowledge bases

The real power shows up when you combine transcripts from multiple channels in the same knowledge base.

Say you want to build an ML knowledge assistant. You ingest:

• 3Blue1Brown for visual math explanations

• Two Minute Papers for research summaries

• Yannic Kilcher for paper deep-dives

• Andrej Karpathy for practical implementation advice

Each transcript gets tagged with its channel in the metadata. Users can ask broad questions ("How do transformers work?") and get answers that pull from multiple perspectives. Or they can filter to a specific channel ("What did Karpathy say about fine-tuning?").

The metadata filtering you set up earlier makes this trivial. No code changes needed, just more data in the same vector store.

Keeping it fresh with automatic ingestion

A static knowledge base goes stale. Channels publish new videos weekly or daily.

TranscriptAPI's /youtube/channel/latest endpoint is free. Use it to check for new uploads:

def get_latest_videos(channel):
    response = requests.get(
        "https://transcriptapi.com/api/v2/youtube/channel/latest",
        params={"channel": channel},
        headers={
            "Authorization": f"Bearer {os.environ['TRANSCRIPTAPI_KEY']}"
        }
    )
    return response.json()["videos"]

Run this on a cron job. Compare against your list of already-ingested video IDs. When new ones appear, run them through your loader, chunker, and embedding pipeline. The knowledge base grows automatically.

Since the latest endpoint costs zero credits, you can poll every hour without impacting your budget.

Frequently asked questions

Can I use this with LlamaIndex instead of LangChain?

Yes. The custom document loader pattern works the same way in LlamaIndex. Create a custom reader that calls TranscriptAPI and returns LlamaIndex Document objects. The rest of the pipeline (embedding, indexing, querying) uses LlamaIndex's native components.

How do I update the knowledge base when new videos are published?

Use TranscriptAPI's /youtube/channel/latest endpoint (free, costs zero credits) to check for new videos on a schedule. When new videos appear, extract their transcripts, embed them, and add them to the vector store. No need to rebuild the entire index.

What is the cost to index an entire YouTube channel?

At 1 credit per transcript, a 500-video channel costs 500 credits. On the annual plan, that is about $2.50 with top-up pricing at $1.50 per 1,000 credits. Embedding costs depend on your provider, roughly $0.01-0.05 per video with OpenAI's text-embedding-3-small.

Start building

LangChain plus TranscriptAPI gives you a production-ready YouTube knowledge base. The built-in YouTubeLoader cannot match it for reliability, features, or scale.

The custom document loader pattern takes 50 lines of code to set up. After that, you have access to transcripts, search, channel listing, and playlist extraction, all through LangChain's familiar interface.

Want the full RAG pipeline walkthrough with vector stores and prompt engineering? Read our guide on building a RAG pipeline with YouTube transcripts.

Start building your LangChain YouTube knowledge base with 100 free TranscriptAPI credits. No credit card needed.

What will you build with YouTube's knowledge base?