Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Feat: Add OLLAMA_LOAD_TIMEOUT env variable #4123

Closed
wants to merge 3 commits into from
Closed

Feat: Add OLLAMA_LOAD_TIMEOUT env variable #4123

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
11bcc40
Add env variable OLLAMA_LOAD_TIMEOUT in seconds
dcfidalgo May 2, 2024
0905a7b
Add entry in FAQ
dcfidalgo May 3, 2024
2befddf
log instead of fail
dcfidalgo May 6, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
4 changes: 4 additions & 0 deletions docs/faq.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -148,6 +148,10 @@ If a different directory needs to be used, set the environment variable `OLLAMA_

Refer to the section [above](#how-do-i-configure-ollama-server) for how to set environment variables on your platform.

### How do I increase the timeout when i get "timed out waiting for llama runner to start"?

To increase the timeout for loading models from disk, set the environment variable `OLLAMA_LOAD_TIMEOUT` (default 600s).

## Does Ollama send my prompts and answers back to ollama.com?

No. Ollama runs locally, and conversation data does not leave your machine.
Expand Down
11 changes: 10 additions & 1 deletion llm/server.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -425,7 +425,16 @@ func (s *llmServer) Ping(ctx context.Context) error {
func (s *llmServer) WaitUntilRunning(ctx context.Context) error {
start := time.Now()
// TODO we need to wire up a better way to detect hangs during model load and startup of the server
expiresAt := time.Now().Add(10 * time.Minute) // be generous with timeout, large models can take a while to load
timeout := 600
if timeoutEnv := os.Getenv("OLLAMA_LOAD_TIMEOUT"); timeoutEnv != "" {
timeoutInt, err := strconv.Atoi(timeoutEnv)
if err != nil || timeoutInt <= 0 {
slog.Info(fmt.Sprintf("invalid OLLAMA_LOAD_TIMEOUT=%s must be an integer greater than zero. Defaulting to %ds", timeoutEnv, timeout))
} else {
timeout = timeoutInt
}
}
expiresAt := time.Now().Add(time.Duration(timeout) * time.Second) // be generous with timeout, large models can take a while to load
ticker := time.NewTicker(50 * time.Millisecond)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should we print the message "loading the model" for each tick?
Without the message or a spinning, the compute seems being stuck for 10 mins.

defer ticker.Stop()

Expand Down