KoboldCPP is an application that allows you to run AI models directly on your local device.
¶ Install KoboldCPP
- Refer to this document for instructions on installing KoboldCPP.
¶ Get Models
- KoboldCPP runs
*.ggufmodel files. These files can be found on platforms like HuggingFace. - Find the appropriate model on HuggingFace and download the GGUF file. For example, this is the GGUF file for the phi-4 model.
¶ Start KoboldCPP
- Once you have your model file, run KoboldCPP. In the Quick Launch menu, click GGUF Text Model to select the model you downloaded.
- Then, click Launch and wait a moment. The KoboldCPP UI will open in your browser. You can close this browser tab as we will be using astrsk.ai.
¶ Connect KoboldCPP in astrsk
- Within astrsk.ai, navigate to the provider settings. Select KoboldCPP as the source, verify the Base URL (default
http://localhost:5001), and then press Connect. - Check the model list to ensure your models are available.
