Working with LLMs and Chat Models in LangChain: OpenAI, Anthropic, and Local Models via Ollama

The most underrated feature of LangChain is model portability. You build a chain against the ChatOpenAI interface, and switching to Claude or a locally-hosted Llama model is a single-line change. This works because every LLM in LangChain implements the same Runnable protocol with identical .invoke(), .stream(), and .batch() methods. Understanding how each provider is wired up — including their differences in parameter names and token counting — is what makes that portability practical rather than aspirational.

The BaseChatModel Interface

All chat model classes in LangChain extend BaseChatModel. The key methods:

Method	Returns	Use case
.invoke(messages)	AIMessage	Single synchronous call
.stream(messages)	Generator of AIMessageChunk	Token-by-token streaming
.batch([msgs1, msgs2])	List of AIMessage	Parallel calls, rate-limited
.ainvoke(messages)	AIMessage (async)	Async frameworks

Every model class also exposes a .bind() method that attaches fixed parameters (tools, stop sequences, temperature) to the model without calling it.

OpenAI: ChatOpenAI

ChatOpenAI wraps the OpenAI chat completions API. Install: pip install langchain-openai.

from langchain_openai import ChatOpenAI
from langchain_core.messages import HumanMessage, SystemMessage

model = ChatOpenAI(
    model="gpt-4o-mini",
    temperature=0,
    max_tokens=256,
    # api_key="sk-..." # or set OPENAI_API_KEY env var
)

messages = [
    SystemMessage(content="Be concise. Answer in 2 sentences max."),
    HumanMessage(content="What is gradient descent?"),
]

response = model.invoke(messages)
print(type(response))
print(response.content)
print(f"\nToken usage: {response.response_metadata.get('token_usage', {})}")

Output:

<class 'langchain_core.messages.ai.AIMessage'>
Gradient descent is an iterative optimization algorithm that minimizes a function by repeatedly moving in the direction of the negative gradient. In machine learning, it updates model parameters to reduce the loss function by small steps proportional to the learning rate.

Token usage: {'completion_tokens': 47, 'prompt_tokens': 31, 'total_tokens': 78}

Note: LLM output content varies by run.

response_metadata contains the token usage, model name, and finish reason. This is where you track costs in production.

ChatOpenAI configuration options

from langchain_openai import ChatOpenAI

# With logprobs and streaming
model = ChatOpenAI(
    model="gpt-4o",
    temperature=0.7,
    max_tokens=512,
    logprobs=True,       # returns token log probabilities
    top_p=0.95,
    frequency_penalty=0.1,
    presence_penalty=0.1,
    timeout=30,          # request timeout in seconds
    max_retries=3,       # automatic retry on rate limits
)

print(f"Model: {model.model_name}")
print(f"Temperature: {model.temperature}")

Output:

Model: gpt-4o
Temperature: 0.7

Anthropic: ChatAnthropic

ChatAnthropic wraps the Anthropic Claude API. Install: pip install langchain-anthropic.

from langchain_anthropic import ChatAnthropic
from langchain_core.messages import HumanMessage, SystemMessage

model = ChatAnthropic(
    model="claude-3-5-haiku-20241022",
    temperature=0,
    max_tokens=256,
    # api_key="sk-ant-..." # or set ANTHROPIC_API_KEY env var
)

messages = [
    SystemMessage(content="Be concise. Answer in 2 sentences max."),
    HumanMessage(content="What is attention mechanism in transformers?"),
]

response = model.invoke(messages)
print(response.content)
print(f"\nUsage: {response.usage_metadata}")

Output:

The attention mechanism allows each token in a sequence to weigh the relevance of all other tokens when computing its representation, enabling the model to capture long-range dependencies. It computes query-key-value dot products to produce weighted sums of value vectors, with softmax normalization ensuring the weights sum to one.

Usage: {'input_tokens': 28, 'output_tokens': 59, 'total_tokens': 87}

Note: LLM output content varies by run.

The interface is identical to ChatOpenAI — same invoke(), same AIMessage return type, same usage metadata pattern. The difference is in the constructor parameters (model names, max_tokens vs no max, Claude-specific features like extended thinking).

Swapping models in a chain: zero chain code change

from langchain_openai import ChatOpenAI
from langchain_anthropic import ChatAnthropic
from langchain_core.prompts import ChatPromptTemplate
from langchain_core.output_parsers import StrOutputParser

prompt = ChatPromptTemplate.from_template("Explain {concept} in one sentence.")
parser = StrOutputParser()

# Swap models here — chain code is identical
openai_chain  = prompt | ChatOpenAI(model="gpt-4o-mini", temperature=0)  | parser
claude_chain  = prompt | ChatAnthropic(model="claude-3-5-haiku-20241022", temperature=0) | parser

input_data = {"concept": "backpropagation"}

openai_result = openai_chain.invoke(input_data)
claude_result = claude_chain.invoke(input_data)

print(f"OpenAI:  {openai_result}")
print(f"Claude:  {claude_result}")

Output:

OpenAI:  Backpropagation is the algorithm that computes gradients of the loss with respect to each parameter by applying the chain rule backwards through the network.
Claude:  Backpropagation is an algorithm that efficiently calculates how much each parameter in a neural network contributed to the error, enabling gradient-based optimization.

Note: LLM outputs vary by run.

Same chain template, same parser, same input — only the model line changed. This is the portability LangChain was designed for.

Local Models via Ollama

Ollama runs open-source models (Llama, Mistral, Phi, Gemma) locally with a REST API. No cloud, no API key, no cost per token. Install Ollama, pull a model, and LangChain talks to it via ChatOllama.

Install: pip install langchain-ollama and Ollama on your machine.

from langchain_ollama import ChatOllama
from langchain_core.messages import HumanMessage

# Requires Ollama running locally (ollama serve)
# Pull model first: ollama pull llama3.2
model = ChatOllama(
    model="llama3.2",
    temperature=0,
    base_url="http://localhost:11434",  # default Ollama endpoint
)

response = model.invoke([HumanMessage(content="What is 12 * 15?")])
print(response.content)

Output:

12 * 15 = 180

Note: Requires Ollama running locally with llama3.2 pulled. Output may vary.

ChatOllama implements the same BaseChatModel interface — your existing chains work with zero modification.

Running all three in a benchmark loop

from langchain_openai import ChatOpenAI
from langchain_anthropic import ChatAnthropic
from langchain_core.prompts import ChatPromptTemplate
from langchain_core.output_parsers import StrOutputParser
import time

prompt = ChatPromptTemplate.from_template(
    "Answer in exactly one sentence: {question}"
)
parser = StrOutputParser()

models = {
    "gpt-4o-mini": ChatOpenAI(model="gpt-4o-mini", temperature=0),
    "claude-haiku": ChatAnthropic(model="claude-3-5-haiku-20241022", temperature=0),
    # "llama3.2": ChatOllama(model="llama3.2", temperature=0),  # uncomment if Ollama installed
}

question = {"question": "Why do neural networks use non-linear activation functions?"}

for name, model in models.items():
    chain = prompt | model | parser
    start = time.time()
    result = chain.invoke(question)
    latency = time.time() - start
    print(f"\n[{name}] ({latency:.2f}s)")
    print(f"  {result}")

Output:

[gpt-4o-mini] (0.87s)
  Neural networks use non-linear activation functions because linear transformations can only represent linear functions regardless of depth, while non-linearity enables them to approximate arbitrary complex functions.

[claude-haiku] (1.12s)
  Non-linear activation functions allow neural networks to learn complex, non-linear patterns and representations that purely linear transformations cannot capture.

Note: Latencies and content vary by run and network conditions.

bind(): Attaching Parameters to a Model

.bind() creates a new Runnable with parameters pre-attached. Use it to configure stop sequences, tools, or format options without modifying the chain structure.

from langchain_openai import ChatOpenAI
from langchain_core.prompts import ChatPromptTemplate
from langchain_core.output_parsers import StrOutputParser

model = ChatOpenAI(model="gpt-4o-mini", temperature=0)

# Bind stop sequences — model stops when it generates "---"
model_with_stop = model.bind(stop=["---"])

chain = (
    ChatPromptTemplate.from_template("List 5 Python libraries for {task}. Separate with '---'.")
    | model_with_stop
    | StrOutputParser()
)

result = chain.invoke({"task": "data science"})
print(result)
print(f"Stopped at separator: {'---' not in result}")

Output:

NumPy
Stopped at separator: True

The model generated "NumPy" and then started generating "---", which triggered the stop sequence. Only the content before the stop sequence is returned.

Gotchas: Provider Differences

1. System message handling

Claude treats SystemMessage differently than OpenAI — it maps to Anthropic's system parameter rather than a message in the conversation. Both work via LangChain's abstraction, but raw API calls differ.

2. token counting

Different providers count tokens differently. OpenAI uses tiktoken; Anthropic uses their own tokenizer. For accurate cost tracking, use the provider's native token counter rather than estimating from character count.

3. Streaming chunks for OpenAI vs Anthropic

from langchain_openai import ChatOpenAI
from langchain_core.prompts import ChatPromptTemplate
from langchain_core.output_parsers import StrOutputParser

chain = (
    ChatPromptTemplate.from_template("Count from 1 to 5, one number per line.")
    | ChatOpenAI(model="gpt-4o-mini", temperature=0)
    | StrOutputParser()
)

chunks = []
for chunk in chain.stream({}):
    chunks.append(chunk)

print(f"Number of chunks received: {len(chunks)}")
print(f"First 5 chunks: {chunks[:5]}")
print(f"Assembled: {''.join(chunks)}")

Output:

Number of chunks received: 14
First 5 chunks: ['', '1', '\n', '2', '\n']
Assembled: 1
2
3
4
5

Note: Chunk count varies by model and network conditions.

OpenAI streams at approximately the token boundary. Anthropic streams at similar granularity. Ollama streaming cadence depends on the model and hardware.

4. Context window limits

Model	Context window
gpt-4o	128K tokens
claude-3-5-sonnet	200K tokens
llama3.2 (3B)	128K tokens
mistral-7b	32K tokens

Exceeding the context window raises a provider-specific error. LangChain does not automatically truncate — you need to manage context length explicitly in your chain.

Conclusion

Every LLM in LangChain speaks the same Runnable interface. Switching from OpenAI to Claude to a local Llama model is a constructor swap, not a refactor. The practical differences — model names, token counting, context limits, streaming granularity — are worth knowing but don't change the chain architecture. bind() attaches provider-specific parameters cleanly without polluting chain logic. Build your chains against the interface, not the provider, and model selection becomes a deployment decision rather than an architectural one.

The next post covers document loaders and text splitters — loading PDFs, web pages, and raw text, then splitting them into chunks that fit in context windows.