How to set up a custom OpenAI-compatible Server in BoltAI
BoltAI supports a custom OpenAI-compatible Server such as an OpenAI proxy server, LocalAI or LM Studio Local Inference Servers.
There are a few options to run a local OpenAI-compatible server.
1. LM Studio
The easiest way to do this is to use LM Studio. Follow this guide by Ingrid Stevens to start.
2. Ollama + LiteLLM
Ollama is another fantastic option. It's opensource and easy to use. Unfortunately, its server is not compatible with OpenAI so you will need to use LiteLLM for that.
LocalAI is another option if you're comfortable with docker and building it yourself. Follow their guide here:
How to use it in BoltAI
Go to Settings > Models, click the (+) button and choose "OpenAI-compatible Server"
Fill the form and click "Save Changes"
- Give it a friendly name.
- Enter the exact url for the chat completions endpoint. For LM Studio, the default is
- (Optional) Enter the model id. This will be sent with each chat request (the params
modelin OpenAI API spec)
- Enter the context length of this model. You need to refer to the original model to find this configuration. In LM Studio, find the "Context Length" configuration on the right pane.
- Enable streaming if the server supports it
Click "Save Changes".
IMPORTANT: if you don't intend to use OpenAI, make sure to set this as default (6)
How to use AI Command with a custom server
This feature is still in beta. Please reach out if you run into any issue.
To use AI Command with a custom server, make sure you set it as the default AI service (screenshot below)
If you are new here, BoltAI is a native macOS app that allows you to access ChatGPT inside any app. Download now.