Requirements

warning

🚧 Cortex.cpp is currently under development. Our documentation outlines the intended behavior of Cortex, which may not yet be fully implemented in the codebase.

To run LLMs on-device or on-premise, Cortex has the following requirements:

Hardware Requirements

CPU

macOs

• Mac Intel CPU.
• Mac Apple Silicon.

Intel

info

• Jan supports a processor that can handle AVX2. For the full list, please see here.
• We support older processors with AVX and AVX-512, though this is not recommended.

• Haswell processors (Q2 2013) and newer.
• Tiger Lake (Q3 2020) and newer for Celeron and Pentium processors.

AMD

info

• Jan supports a processor that can handle AVX2. For the full list, please see here.
• We support older processors with AVX and AVX-512, though this is not recommended.

• Excavator processors (Q2 2015) and newer.

RAM

• 8GB for running up to 3B models (int4).
• 16GB for running up to 7B models (int4).
• 32GB for running up to 13B models (int4).

info

We support DDR2 RAM as the minimum requirement but recommend using newer generations of RAM for improved performance.

GPU

• 6GB can load the 3B model (int4) with ngl at 120 ~ full speed on CPU/ GPU.
• 8GB can load the 7B model (int4) with ngl at 120 ~ full speed on CPU/ GPU.
• 12GB can load the 13B model (int4) with ngl at 120 ~ full speed on CPU/ GPU.

info

Having at least 6GB VRAM when using NVIDIA, AMD, or Intel Arc GPUs is recommended.

Disk

• Having at least 10GB is recommended.

info

The app is 1.02 MB, but models are usually 4GB+.

Operating System Requirements

Mac

• MacOSX 13.6 or higher.
• Dependencies:
  • Node.js version 18 or higher.
  • NPM version 9 or higher.
  • Homebrew version 3 or higher.
  • CPU Instructions Sets: AVX AVX2 AVX512.

Windows

• Windows 10 or higher.
• Dependencies:
  • Node.js version 18 or higher.
  • NPM version 9 or higher.
  • CPU Instructions Sets: AVX AVX2 AVX512.

Linux

• Debian-based (Supports .deb and AppImage )
• Ubuntu-based
  • Ubuntu Desktop LTS (official)/ Ubuntu Server LTS (only for server)
  • Edubuntu (Mainly desktop)
  • Kubuntu (Desktop only)
  • Lubuntu (Both desktop and server, though mainly desktop)
  • Ubuntu Budgie (Mainly desktop)
  • Ubuntu Cinnamon (Desktop only)
  • Ubuntu Kylin (Both desktop and server)
  • Ubuntu MATE (Desktop only)
• Pacman-based
  • Arch Linux based
    • Arch Linux (Mainly desktop)
    • SteamOS (Desktop only)
• RPM-based (Supports .rpm and AppImage )
• Fedora-based
  • RHEL-based (Server only)
• openSUSE (Both desktop and server)
• Dependencies:
  • Node.js version 18 or higher.
  • NPM version 9 or higher.
  • Homebrew version 3 or higher.
  • CPU Instructions Sets: AVX AVX2 AVX512.
  • MPI or OpenMPI.

info

• Please check whether your Linux distribution supports desktop, server, or both environments.
• For server versions, please refer to the server installation.

GPU Requirements

NVIDIA

• NVIDIA driver version 470.63.01 or higher.
• CUDA Toolkit version 12.3 or higher.

info

CUDA Toolkit dependencies will be installed when you install Cortex.

AMD

• AMD driver version 22.7.1 or higher.

MPI/OpenMPI

Linux

1. Update the package list:

sudo apt update

2. Install the OpenMPI binaries and development libraries:

sudo apt install openmpi-bin libopenmpi-dev