3
$\begingroup$

I have a local service running on https://localhost:PORT/v1 that duplicates the Open AI API functionality available at https://api.openai.com/v1. It appears the only way to use something like LLMSynthesize is to have an Open AI key. Is it possible to replace the base URL for any LLM calls? I presume one way to do this may be to create a ServiceObject for authentication?

Improve this question
$\endgroup$
3
  • $\begingroup$ Hmm... yeah, I "complained" in a relatively recent Community post that it is not clear how to plug in other LLM services in WL's LLM functions framework. In the corresponding Python and Raku packages it is easy. So, are you interested in a Python solution which you can use in Mathematica? $\endgroup$
    – Anton Antonov
    Commented Sep 27, 2023 at 19:47
  • 1
    $\begingroup$ Yeah, in Python I'm running vLLM but there are a whole multitude of options for serving local LLMs. Another is llama.cpp. $\endgroup$
    – James Rohal
    Commented Sep 27, 2023 at 20:17
  • $\begingroup$ Thanks for the link! I was referring to an approach closer to Mathematica's -- "LLMFunctionObjects". $\endgroup$
    – Anton Antonov
    Commented Sep 27, 2023 at 20:23

1 Answer 1

Reset to default
2
$\begingroup$

Edit file C:\Program Files\Wolfram Research\Wolfram\14.1\SystemFiles\Components\OpenAI\Kernel\OpenAI.m

enter image description here

For example, LLaMA 3.1 via LM Studio:

enter image description here

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.