What you'll build
By the end of this tutorial you'll have a working local LLM setup in three pieces: a typed TypeScript client that talks to Ollama, a CLI that streams completions to your terminal, and a small Express HTTP API you can call from any app. The whole thing runs on your machine, needs no API key, and costs nothing per query after setup.
Every file you need is in this page. Paste each one into a fresh project, follow the setup steps below, and it runs. No repo to clone, no hidden files. This guide also explains every decision behind the code so you can change it with confidence.
Should you run a model locally? The honest verdict
Yes, for most development tooling and privacy-sensitive use cases. The tradeoffs are real but specific.
The case for it: After the model download, inference is free. You can run a prompt loop 10,000 times a day and pay nothing. Every prompt stays on your machine, so nothing sensitive crosses a network boundary you do not control. For developer scripts, internal tools, and experiments, local inference removes the operational overhead of API key management, rate limits, and billing alerts.
The honest limitation: A 3B or 7B parameter model running on a laptop is not going to match a frontier hosted model on hard reasoning tasks. You trade raw capability for cost control and privacy. For summarization, classification, code generation on well-defined tasks, and conversational tools around routine questions, the smaller models are good enough. The only way to know whether that is true for your case is to try.
This tutorial gives you a working setup so you can make that call yourself.
Why local inference is worth understanding in 2026
Local inference is not a niche technique anymore. Ollama ships with macOS, Linux, and Windows installers. Model sizes have dropped: a model with competitive quality at routine tasks fits in about 2 GB. The tooling around node-llama-cpp and Ollama's REST API means you do not need to understand GPU memory layout or CUDA to get something working.
The developer pattern this tutorial targets is common: you want an LLM as part of your Node.js app or script, you do not want a per-token bill, and you want to understand what is actually happening in the network calls. That combination is why this guide exists.
What you need
- Node.js 18.19 or newer. The client uses the built-in
fetch, which became stable in Node 18. If you are on an older version, update. - Ollama. Install from ollama.com/download. It runs as a local daemon on
http://localhost:11434and manages model downloads for you. - A model. Pull
llama3.2to follow along. The download is about 2 GB. Want smaller?qwen2.5:0.5bis fast and light. Want stronger reasoning?llama3.1:8bon a machine with 16+ GB RAM.
Pull the model, create a fresh project, and install the tooling:
ollama pull llama3.2
mkdir local-llm-node
cd local-llm-node
npm init -y
npm install express
npm install -D typescript tsx @types/node @types/expressExpress is the one runtime dependency; everything else is for building and running the .ts files. tsx runs TypeScript directly, so there is no separate build step while you follow along. Open package.json and add "type": "module" plus these scripts:
{
"type": "module",
"scripts": {
"cli": "tsx src/cli.ts",
"start": "tsx src/server.ts",
"build": "tsc"
}
}Then add a tsconfig.json at the project root. This one runs cleanly with tsx in development and compiles with tsc for a production build:
{
"compilerOptions": {
"target": "ES2022",
"module": "ESNext",
"moduleResolution": "bundler",
"strict": true,
"esModuleInterop": true,
"skipLibCheck": true,
"rootDir": "src",
"outDir": "dist",
"types": ["node"]
},
"include": ["src"]
}All the source files below go in a src/ directory you create in the project root.
How Ollama works
Ollama is a separate process that serves a plain HTTP API on port 11434. It handles model loading, quantization, and inference. Your Node.js code talks to it by sending JSON to its REST endpoints.
The three endpoints this tutorial uses:
POST /api/generatewithstream: falsefor a single complete responsePOST /api/generatewithstream: truefor a streaming response (newline-delimited JSON)POST /api/chatfor multi-turn conversations with a message history
This architecture has a useful property: your Node.js client is just an HTTP client. If you want to swap Ollama for a different inference server later (LM Studio, vLLM, a remote machine), you change one URL. The application code stays the same.
Build the Ollama client
Create src/client.ts. This is the whole file: the defaults, an error type, the option types, the streaming parser, and the OllamaClient class with generate, generateStream, and chat. Paste it in, then read the walk-through underneath.
const DEFAULT_BASE_URL = 'http://localhost:11434';
const DEFAULT_MODEL = 'llama3.2';
export class OllamaError extends Error {
status?: number;
constructor(message: string, opts: { status?: number; cause?: unknown } = {}) {
super(message, opts.cause ? { cause: opts.cause } : undefined);
this.name = 'OllamaError';
this.status = opts.status;
}
}
export interface OllamaClientOptions {
baseUrl?: string;
model?: string;
fetchImpl?: typeof fetch;
}
export interface GenerateOptions {
prompt: string;
model?: string;
options?: Record<string, unknown>;
}
export interface ChatMessage {
role: 'system' | 'user' | 'assistant';
content: string;
}
// fetch's web ReadableStream, or a Node async-iterable stream. The same parser
// handles both, so the client works with the built-in fetch or a mock.
type ByteStream =
| ReadableStream<Uint8Array>
| AsyncIterable<Uint8Array>
| null
| undefined;
export async function* parseNdjsonStream(
body: ByteStream,
pick: (obj: Record<string, unknown>) => unknown,
): AsyncGenerator<string> {
if (!body) return;
const decoder = new TextDecoder();
let buffer = '';
function* handleChunk(chunk: Uint8Array): Generator<string> {
buffer += decoder.decode(chunk, { stream: true });
let nl: number;
while ((nl = buffer.indexOf('\n')) !== -1) {
const line = buffer.slice(0, nl).trim();
buffer = buffer.slice(nl + 1);
if (!line) continue;
let obj: Record<string, unknown>;
try {
obj = JSON.parse(line);
} catch {
continue; // skip a malformed line, keep going
}
const value = pick(obj);
if (typeof value === 'string' && value) yield value;
}
}
if ('getReader' in body && typeof body.getReader === 'function') {
const reader = body.getReader();
for (;;) {
const { done, value } = await reader.read();
if (done) break;
if (value) yield* handleChunk(value);
}
} else {
for await (const chunk of body as AsyncIterable<Uint8Array>) {
yield* handleChunk(chunk);
}
}
}
export class OllamaClient {
readonly baseUrl: string;
readonly model: string;
private readonly fetchImpl: typeof fetch;
constructor(opts: OllamaClientOptions = {}) {
this.baseUrl = (opts.baseUrl || DEFAULT_BASE_URL).replace(/\/$/, '');
this.model = opts.model || DEFAULT_MODEL;
this.fetchImpl = opts.fetchImpl || globalThis.fetch;
if (typeof this.fetchImpl !== 'function') {
throw new OllamaError('No fetch implementation available. Use Node 18+ or pass fetchImpl.');
}
}
private async post(path: string, payload: unknown): Promise<Response> {
let res: Response;
try {
res = await this.fetchImpl(`${this.baseUrl}${path}`, {
method: 'POST',
headers: { 'Content-Type': 'application/json' },
body: JSON.stringify(payload),
});
} catch (err) {
throw new OllamaError(
`Could not reach Ollama at ${this.baseUrl}. Is it running? Try "ollama serve".`,
{ cause: err },
);
}
if (!res.ok) {
const body = await res.text().catch(() => '');
throw new OllamaError(
`Ollama returned ${res.status} ${res.statusText}: ${body.slice(0, 500)}`,
{ status: res.status },
);
}
return res;
}
async generate({ prompt, model, options }: GenerateOptions): Promise<string> {
if (!prompt) throw new OllamaError('generate() needs a prompt.');
const res = await this.post('/api/generate', {
model: model || this.model,
prompt,
stream: false,
...(options ? { options } : {}),
});
const data = (await res.json()) as { response?: string };
return data.response ?? '';
}
async *generateStream({ prompt, model, options }: GenerateOptions): AsyncGenerator<string> {
if (!prompt) throw new OllamaError('generateStream() needs a prompt.');
const res = await this.post('/api/generate', {
model: model || this.model,
prompt,
stream: true,
...(options ? { options } : {}),
});
yield* parseNdjsonStream(res.body, (obj) => obj.response);
}
async chat(messages: ChatMessage[], model?: string): Promise<string> {
const res = await this.post('/api/chat', {
model: model || this.model,
messages,
stream: false,
});
const data = (await res.json()) as { message?: { content?: string } };
return data.message?.content ?? '';
}
}Two decisions worth noting.
`fetchImpl` is injectable for testing. The constructor accepts a custom fetch function. This is how you can drive every code path with no real Ollama server running: pass a mock that returns canned responses, exercise generate and generateStream, and stay deterministic. It costs nothing and makes the whole network layer testable. If you take one pattern from this client, take this one.
Error messages say what to do. When Ollama is not running, fetch throws a connection error with a raw message like ECONNREFUSED. The client catches it and throws Could not reach Ollama at http://localhost:11434. Is it running? Try "ollama serve". That is the message a developer actually needs at 11pm.
Level up your Node.js
Want to go deeper on Node.js before continuing? The Odin Project's NodeJS path covers async Node, HTTP, and APIs end to end for free. Back End Development and APIs (freeCodeCamp) is a strong alternative if you prefer a more structured curriculum. We line both up on our Node.js and JavaScript courses page.
Stream tokens to your terminal
Streaming is what makes local LLMs useful for developer scripts. Instead of waiting for the full completion, you get tokens as they arrive. The experience looks like ChatGPT: text appears word by word.
The generateStream method above does the streaming. It leans on parseNdjsonStream, which handles the tricky part: Ollama streams responses as newline-delimited JSON, and network chunks do not respect line boundaries. A single JSON object can arrive split across two chunks. The parser appends each chunk to a buffer, scans for newlines, parses complete lines, and leaves the rest in the buffer for the next chunk. That handles every split-chunk case, including the one where a chunk boundary lands inside a JSON token.
Create src/cli.ts to drive the client from the terminal. It takes the prompt from the command line or from piped stdin, then streams the response:
import { OllamaClient } from './client';
async function readStdin(): Promise<string> {
if (process.stdin.isTTY) return '';
let data = '';
for await (const chunk of process.stdin) data += chunk;
return data.trim();
}
async function main(): Promise<void> {
const client = new OllamaClient({ model: process.env.OLLAMA_MODEL });
const prompt = process.argv.slice(2).join(' ') || (await readStdin());
if (!prompt) {
console.error('Usage: npm run cli -- "your prompt" (or pipe text on stdin)');
process.exit(1);
}
process.stdout.write(`> ${prompt}\n\n`);
for await (const chunk of client.generateStream({ prompt })) {
process.stdout.write(chunk);
}
process.stdout.write('\n');
}
main().catch((err: Error) => {
console.error(err.message);
process.exit(1);
});Make sure Ollama is running first (ollama serve), then run it. You should see the model's response appear token by token:
npm run cli -- "Explain closures in JavaScript in two sentences."
# Or pipe stdin:
echo "Write a haiku about Node.js" | npm run cliBuild the HTTP API
Once you have a working client, wrapping it in an HTTP API is the next natural step. You might want other services in your stack to call your local LLM, or to add pre/post-processing middleware.
Create src/server.ts using Express, the HTTP framework most Node developers already know. express.json() handles body parsing, so there is no hand-rolled request reader to maintain. Three endpoints:
GET /healthreturns{ ok: true }POST /generatereturns the full completion as JSONPOST /generate/streamstreams tokens astext/plain
The createServer function wires the client to the routes. It takes an optional client, so you can pass a fake in tests instead of hitting a real model. The file ends by starting the server:
import express, { type Express, type Request, type Response } from 'express';
import { OllamaClient, OllamaError } from './client';
export interface CreateServerOptions {
client?: OllamaClient;
}
export function createServer(opts: CreateServerOptions = {}): Express {
const client =
opts.client ||
new OllamaClient({
baseUrl: process.env.OLLAMA_URL,
model: process.env.OLLAMA_MODEL,
});
const app = express();
app.use(express.json({ limit: '1mb' }));
app.get('/health', (_req: Request, res: Response) => {
res.json({ ok: true });
});
app.post('/generate', async (req: Request, res: Response) => {
const { prompt, model } = req.body ?? {};
if (!prompt) {
res.status(400).json({ error: 'Missing "prompt".' });
return;
}
try {
const response = await client.generate({ prompt, model });
res.json({ response });
} catch (err) {
const status = err instanceof OllamaError ? 502 : 400;
res.status(status).json({ error: (err as Error).message });
}
});
app.post('/generate/stream', async (req: Request, res: Response) => {
const { prompt, model } = req.body ?? {};
if (!prompt) {
res.status(400).json({ error: 'Missing "prompt".' });
return;
}
res.setHeader('Content-Type', 'text/plain; charset=utf-8');
try {
for await (const chunk of client.generateStream({ prompt, model })) {
res.write(chunk);
}
res.end();
} catch (err) {
if (res.headersSent) {
res.end();
return;
}
const status = err instanceof OllamaError ? 502 : 400;
res.status(status).json({ error: (err as Error).message });
}
});
return app;
}
const port = Number(process.env.PORT) || 3000;
createServer().listen(port, () => {
console.log(`local-llm-node API listening on http://localhost:${port}`);
});Each res.write() sends a chunk to the client immediately; Express uses chunked transfer encoding under the hood, so tokens stream out as the model produces them. res.end() closes the response when the model finishes.
Start the server and test it with curl:
npm start
# -> local-llm-node API listening on http://localhost:3000
# Health check
curl http://localhost:3000/health
# Non-streaming
curl -X POST http://localhost:3000/generate \
-H "Content-Type: application/json" \
-d '{"prompt":"What is a closure in JavaScript?"}'
# Streaming (tokens arrive as they are generated)
curl -N -X POST http://localhost:3000/generate/stream \
-H "Content-Type: application/json" \
-d '{"prompt":"Write a Node.js hello world."}'Point the server at a remote Ollama instance with an env var:
OLLAMA_URL=http://192.168.1.50:11434 npm startGo deeper on backend Node.js
Ready to go further on Node.js and full-stack backend architecture? Full Stack Open (University of Helsinki) has a free, project-based Node.js and Express track. Back End Development and APIs (freeCodeCamp) is a good parallel if you want a certification path. Both sit on our backend developer roadmap.
Get the cheat sheet
You now have three files that fit together: src/client.ts, src/cli.ts, and src/server.ts. It is intentionally small: one client, one CLI, one Express server, all in TypeScript. You can read the whole thing in an afternoon and understand every line, then extend it into whatever you are building.
Want a quick-reference card for the Ollama endpoints, the streaming pattern, and the common model options (temperature, num_predict, stop)? Grab the free cheat sheet below, and keep our Node.js cheat sheet open while you build.
Get the free coding cheat sheet
The whole path from zero to job-ready on one page: what to learn, which free platform to use for each step, and how long it really takes. Enter your email and it unlocks below, ready to print or save as a PDF.
No spam. One short email when we ship a new roadmap or guide, and you can unsubscribe anytime. See our privacy policy.
Where to go next
Running a model locally is the foundation. Where you take it depends on what you are building.
Build LLM pipelines and agents: LangChain for LLM Application Development (DeepLearning.AI) covers chains, memory, and agents in about two hours. AI Agents in LangGraph follows up with stateful multi-agent workflows.
Add retrieval-augmented generation (RAG): Building and Evaluating Advanced RAG (DeepLearning.AI) covers chunking, embedding models, and evaluation. That is where most RAG builds fall down, and the course hits it directly.
Understand what is actually inside the model: Practical Deep Learning for Coders (fast.ai) and Andrej Karpathy's neural-networks series are the two best free paths into the internals. Neither needs a math degree.
Build production AI features: if you want to pair local inference in development with a hosted model in production, our How to become an AI engineer guide maps the free courses that get you there.
Frequently asked questions
Do I need a GPU to run this?
No. Ollama runs on CPU, and models at 3B parameters run in real time on modern laptop hardware. A 3B model on an M1 MacBook Pro generates tokens at a comfortable reading pace. Larger models (8B and up) work on CPU but are noticeably slower. GPU acceleration is used automatically if you have one.
Which model should I start with?
Start with llama3.2, the default in the client. It is a good balance of quality and speed at 3B parameters. If you need better reasoning and have 16+ GB RAM, try llama3.1:8b. For the smallest footprint, qwen2.5:0.5b handles classification and simple generation.
Why Express and TypeScript instead of a plain Node script?
Both match what you would use on the job. Express is the HTTP framework most Node developers already know, so express.json() and the app.post() routes read the way real APIs do, instead of a hand-rolled http.createServer. TypeScript catches whole classes of bugs before you run the code and documents the shapes (GenerateOptions, ChatMessage) as you type. You run the .ts files directly with tsx in development and compile with tsc for a production build.
How does this compare to the OpenAI or Claude API?
The Ollama API is similar in intent but different in request and response shape, so you cannot drop hosted-API client code in directly. Ollama does expose an optional OpenAI-compatible endpoint (/v1/chat/completions) if you need that. The client here uses the native Ollama format, which is slightly simpler and skips the compatibility layer.
Can I run multiple models at the same time?
Ollama loads one model into memory at a time by default. Switching models takes a few seconds while it unloads one and loads the other. To serve several models at once in production you would run multiple Ollama instances or a different setup. For development, single-model is fine.
Is this production-ready?
The code here is a tutorial implementation, not a production service. For production you would add authentication and rate limiting (both easy to drop in as Express middleware), a process manager (PM2 or systemd), structured logging, and monitoring. Treat it as a solid working prototype to harden from.
How do I pass conversation history for multi-turn chat?
The client includes a chat() method that takes a messages array of { role, content } objects, the same shape hosted chat APIs use. Managing the history (when to trim it, how to persist it across requests) is left to your application layer.
Does this work with models other than llama3.2?
Yes. Any model in the Ollama library works with this client. Set the OLLAMA_MODEL env var to the tag you pulled (for example OLLAMA_MODEL=mistral or OLLAMA_MODEL=qwen2.5:0.5b) and the client passes the name straight to the API. No other changes needed.