local llms with ollama + open-webui + litellm configured with some apis

the free Claude 3.5 sonnet is my daily driver now. When I run out of free messages, I switch over to open-webui and have access to various flagship models for a few cents. The local models usually suffice when I'm asking a dumb question

Here's my `shell.nix`:

```

{ pkgs ? import {} }:

(pkgs.buildFHSEnv {

name = "simple-fhs-env";

targetPkgs = pkgs: with pkgs; [

tmux

bash

python311

];

runScript = ''

#!/usr/bin/env bash

set -x

set -e

source .venv/bin/activate

tmux new-session -d -s textgen

tmux send-keys -t textgen "open-webui serve" C-m

tmux split-window -v -t textgen

tmux send-keys -t textgen "LITELLM_MASTER_KEY=hunter2 litellm --config litellm.yaml --port 8031" C-m

tmux attach -t textgen

'';

}).env

```

And here's `litellm.yaml`:

```

model_list:

- model_name: codestral

litellm_params:

model: mistral/codestral-latest

api_key: hunter2

- model_name: claude-3.5

litellm_params:

model: anthropic/claude-3-5-sonnet-20240620

api_key: sk-hunter2

- model_name: gemini-pro

litellm_params:

model: gemini/gemini-1.5-pro-latest

api_key: hunter2

safety_settings:

- category: HARM_CATEGORY_HARASSMENT

threshold: BLOCK_NONE

- category: HARM_CATEGORY_HATE_SPEECH

threshold: BLOCK_NONE

- category: HARM_CATEGORY_SEXUALLY_EXPLICIT

threshold: BLOCK_NONE

- category: HARM_CATEGORY_DANGEROUS_CONTENT

threshold: BLOCK_NONE

```