Skip to main content

Ollama

Ollama runs open models locally. The llms/ollama package talks to a local (or remote) Ollama server — no API key required.

Prerequisites​

Setup​

// Defaults to the local server at http://localhost:11434
llm, err := ollama.New(ollama.WithModel("llama2"))

// Point at a remote server
llm, err = ollama.New(
    ollama.WithServerURL("http://custom-server:11434"),
    ollama.WithModel("codellama"),
)

Ollama supports schema-constrained output through its native format field, wired up by the provider-neutral llms.WithStructuredOutput option.

For a full walkthrough, see the Ollama quickstart.

Example​

package main

import (
	"context"
	"fmt"
	"log"

	"github.com/vxcontrol/langchaingo/llms"
	"github.com/vxcontrol/langchaingo/llms/ollama"
	"github.com/vxcontrol/langchaingo/llms/streaming"
)

func main() {
	llm, err := ollama.New(ollama.WithModel("llama2"))
	if err != nil {
		log.Fatal(err)
	}
	ctx := context.Background()
	completion, err := llms.GenerateFromSinglePrompt(
		ctx,
		llm,
		"Human: Who was the first man to walk on the moon?\nAssistant:",
		llms.WithTemperature(0.8),
		llms.WithStreamingFunc(func(_ context.Context, chunk streaming.Chunk) error {
			fmt.Println(chunk.String())
			return nil
		}),
	)
	if err != nil {
		log.Fatal(err)
	}

	_ = completion
}