llama.cppだけでなくollamaもLLMアプリケーションのProxyにできる。
例えば以下のモデルがダウンロード済みとする。
(.venv) $ ollama list NAME ID SIZE MODIFIED phi3.5:latest 61819fb370a3 2.2 GB 2 weeks ago llama3.1:latest 62757c860e01 4.7 GB 7 weeks ago llava:13b 0d0eb4d7f485 8.0 GB 3 months ago solar:latest 059fdabbe6e6 6.1 GB 3 months ago mistral:latest 2ae6f6dd7a3d 4.1 GB 3 months ago phi3:medium 1e67dff39209 7.9 GB 3 months ago gemma:7b a72c7f4d0a15 5.0 GB 3 months ago
この場合、BASE_URLに"http://localhost:11434/v1"、modelに適当なモデルを設定すると、
(全てではないが)多くのOpenAIを使ったアプリケーションをローカルで実行できる。
$ cat test.py
import os
from openai import OpenAI
OPENAI_KEY = "no_key"
OPENAI_BASE_URL = "http://localhost:11434/v1"
client = OpenAI(api_key=OPENAI_KEY, base_url=OPENAI_BASE_URL)
#model="phi3.5:latest"
#model="llama3.1:latest"
#model="llava:13b"
#model="solar:latest"
#model="mistral:latest"
model="phi3:medium"
#model="gemma:7b"
response = client.chat.completions.create(
model=model,
messages=[
{
"role": "user",
"content": "What is the tallest mountain in Japan?"
}
],
temperature=0.2
)
print(response)
実行結果は以下の通り。
$ python test.py ChatCompletion(id='chatcmpl-123', choices=[Choice(finish_reason='stop', index=0, logprobs=None, message=ChatCompletionMessage(content=" Mount Fuji, also known as Fujisan or Fuji-san, is the highest mountain in Japan. It stands at 3,776 meters (12,389 feet) and is an iconic symbol of the country. Located on Honshu Island, it's a stratovolcano that last erupted in the early 18th century. Climbers often hike Mount Fuji during the summer months when the weather conditions are most favorable for reaching its summit.", refusal=None, role='assistant', function_call=None, tool_calls=None))], created=1727317603, model='phi3:medium', object='chat.completion', service_tier=None, system_fingerprint='fp_ollama', usage=CompletionUsage(completion_tokens=110, prompt_tokens=12, total_tokens=122, completion_tokens_details=None))
