Quickstart
warning
🚧 Cortex.cpp is currently under development. Our documentation outlines the intended behavior of Cortex, which may not yet be fully implemented in the codebase.
Installation​
To install Cortex, download the installer for your operating system from the following options:
- Stable Version
Start Cortex.cpp Processes and API Server​
This command starts the Cortex.cpp API server at localhost:3928
.
- MacOs/Linux
- Windows
# Stablecortex start# Betacortex-beta start# Nightlycortex-nightly start
# Stablecortex.exe start# Betacortex-beta.exe start# Nightlycortex-nightly.exe start
Run a Model​
This command downloads the default gguf
model format from the Cortex Hub, starts the model, and chat with the model.
- MacOs/Linux
- Windows
# Stablecortex run mistral# Betacortex-beta run mistral# Nightlycortex-nightly run mistral
# Stablecortex.exe run mistral# Betacortex-beta.exe run mistral# Nightlycortex-nightly.exe run mistral
info
All model files are stored in the ~users/cortex/models
folder.
Using the Model​
API​
curl http://localhost:3928/v1/chat/completions \-H "Content-Type: application/json" \-d '{ "model": "", "messages": [ { "role": "user", "content": "Hello" }, ], "model": "mistral", "stream": true, "max_tokens": 1, "stop": [ null ], "frequency_penalty": 1, "presence_penalty": 1, "temperature": 1, "top_p": 1}'
Cortex.js​
const resp = await cortex.chat.completions.create({ model: "mistral", messages: [ { role: "system", content: "You are a chatbot." }, { role: "user", content: "What is the capital of the United States?" }, ], });
Cortex.py​
completion = client.chat.completions.create( model=mistral, messages=[ { "role": "user", "content": "Say this is a test", }, ],)
Stop a Model​
This command stops the running model.
- MacOs/Linux
- Windows
# Stablecortex models stop mistral# Betacortex-beta models stop mistral# Nightlycortex-nightly models stop mistral
# Stablecortex.exe models stop mistral# Betacortex-beta.exe models stop mistral# Nightlycortex-nightly.exe models stop mistral
Show the System State​
This command displays the running model and the hardware system status.
- MacOs/Linux
- Windows
# Stablecortex ps# Betacortex-beta ps# Nightlycortex-nightly ps
# Stablecortex.exe ps# Betacortex-beta.exe ps# Nightlycortex-nightly.exe ps
Run Different Model Variants​
- MacOs/Linux
- Windows
# Stable## Run HuggingFace model with HuggingFace Repocortex run TheBloke/Mistral-7B-Instruct-v0.2-GGUF# Run Mistral in ONNX formatcortex run mistral:onnx# Run Mistral in TensorRT-LLM formatcortex run mistral:tensorrt-llm# Beta## Run HuggingFace model with HuggingFace Repocortex-beta run TheBloke/Mistral-7B-Instruct-v0.2-GGUF# Run Mistral in ONNX formatcortex-beta run mistral:onnx# Run Mistral in TensorRT-LLM formatcortex-beta run mistral:tensorrt-llm# Nightly## Run HuggingFace model with HuggingFace Repocortex-nightly run TheBloke/Mistral-7B-Instruct-v0.2-GGUF# Run Mistral in ONNX formatcortex-nightly run mistral:onnx# Run Mistral in TensorRT-LLM formatcortex-nightly run mistral:tensorrt-llm
# Stable## Run HuggingFace model with HuggingFace Repocortex.exe run TheBloke/Mistral-7B-Instruct-v0.2-GGUF# Run Mistral in ONNX formatcortex.exe run mistral:onnx# Run Mistral in TensorRT-LLM formatcortex.exe run mistral:tensorrt-llm# Beta## Run HuggingFace model with HuggingFace Repocortex-beta.exe run TheBloke/Mistral-7B-Instruct-v0.2-GGUF# Run Mistral in ONNX formatcortex-beta.exe run mistral:onnx# Run Mistral in TensorRT-LLM formatcortex-beta.exe run mistral:tensorrt-llm# Nightly## Run HuggingFace model with HuggingFace Repocortex-nightly.exe run TheBloke/Mistral-7B-Instruct-v0.2-GGUF# Run Mistral in ONNX formatcortex-nightly.exe run mistral:onnx# Run Mistral in TensorRT-LLM formatcortex-nightly.exe run mistral:tensorrt-llm
What's Next?​
Now that Cortex.cpp is set up, here are the next steps to explore:
- Adjust the folder path and configuration using the
.cortexrc
file. - Explore the Cortex.cpp data folder to understand how it stores data.
- Learn about the structure of the
model.yaml
file in Cortex.cpp. - Integrate Cortex.cpp libraries seamlessly into your Python or JavaScript applications.