Skip to main content

Quickstart

warning

🚧 Cortex.cpp is currently under development. Our documentation outlines the intended behavior of Cortex, which may not yet be fully implemented in the codebase.

Installation​

To install Cortex, download the installer for your operating system from the following options:

Start Cortex.cpp Processes and API Server​

This command starts the Cortex.cpp API server at localhost:3928.


# Stable
cortex start
# Beta
cortex-beta start
# Nightly
cortex-nightly start

Run a Model​

This command downloads the default gguf model format from the Cortex Hub, starts the model, and chat with the model.


# Stable
cortex run mistral
# Beta
cortex-beta run mistral
# Nightly
cortex-nightly run mistral

info

All model files are stored in the ~users/cortex/models folder.

Using the Model​

API​


curl http://localhost:3928/v1/chat/completions \
-H "Content-Type: application/json" \
-d '{
"model": "",
"messages": [
{
"role": "user",
"content": "Hello"
},
],
"model": "mistral",
"stream": true,
"max_tokens": 1,
"stop": [
null
],
"frequency_penalty": 1,
"presence_penalty": 1,
"temperature": 1,
"top_p": 1
}'

Cortex.js​


const resp = await cortex.chat.completions.create({
model: "mistral",
messages: [
{ role: "system", content: "You are a chatbot." },
{ role: "user", content: "What is the capital of the United States?" },
],
});

Cortex.py​


completion = client.chat.completions.create(
model=mistral,
messages=[
{
"role": "user",
"content": "Say this is a test",
},
],
)

Stop a Model​

This command stops the running model.


# Stable
cortex models stop mistral
# Beta
cortex-beta models stop mistral
# Nightly
cortex-nightly models stop mistral

Show the System State​

This command displays the running model and the hardware system status.


# Stable
cortex ps
# Beta
cortex-beta ps
# Nightly
cortex-nightly ps

Run Different Model Variants​


# Stable
## Run HuggingFace model with HuggingFace Repo
cortex run TheBloke/Mistral-7B-Instruct-v0.2-GGUF
# Run Mistral in ONNX format
cortex run mistral:onnx
# Run Mistral in TensorRT-LLM format
cortex run mistral:tensorrt-llm
# Beta
## Run HuggingFace model with HuggingFace Repo
cortex-beta run TheBloke/Mistral-7B-Instruct-v0.2-GGUF
# Run Mistral in ONNX format
cortex-beta run mistral:onnx
# Run Mistral in TensorRT-LLM format
cortex-beta run mistral:tensorrt-llm
# Nightly
## Run HuggingFace model with HuggingFace Repo
cortex-nightly run TheBloke/Mistral-7B-Instruct-v0.2-GGUF
# Run Mistral in ONNX format
cortex-nightly run mistral:onnx
# Run Mistral in TensorRT-LLM format
cortex-nightly run mistral:tensorrt-llm

What's Next?​

Now that Cortex.cpp is set up, here are the next steps to explore:

  1. Adjust the folder path and configuration using the .cortexrc file.
  2. Explore the Cortex.cpp data folder to understand how it stores data.
  3. Learn about the structure of the model.yaml file in Cortex.cpp.
  4. Integrate Cortex.cpp libraries seamlessly into your Python or JavaScript applications.
info

Cortex.cpp is still in early development, so if you have any questions, please reach out to us: