Requirements
To run LLMs on-device or on-premise, Cortex has the following requirements:
Hardware Requirements
- CPU
- RAM
- GPU
- Disk
- macOs
- Intel
- AMD
- Mac Intel CPU.
- Mac Apple Silicon.
info
- Jan supports a processor that can handle AVX2. For the full list, please see here.
- We support older processors with AVX and AVX-512, though this is not recommended.
- Haswell processors (Q2 2013) and newer.
- Tiger Lake (Q3 2020) and newer for Celeron and Pentium processors.
info
- Jan supports a processor that can handle AVX2. For the full list, please see here.
- We support older processors with AVX and AVX-512, though this is not recommended.
- Excavator processors (Q2 2015) and newer.
- 8GB for running up to 3B models (int4).
- 16GB for running up to 7B models (int4).
- 32GB for running up to 13B models (int4).
info
We support DDR2 RAM as the minimum requirement but recommend using newer generations of RAM for improved performance.
- 6GB can load the 3B model (int4) with
nglat 120 ~ full speed on CPU/ GPU. - 8GB can load the 7B model (int4) with
nglat 120 ~ full speed on CPU/ GPU. - 12GB can load the 13B model (int4) with
nglat 120 ~ full speed on CPU/ GPU.
info
Having at least 6GB VRAM when using NVIDIA, AMD, or Intel Arc GPUs is recommended.
- Having at least 10GB is recommended.
info
The app is 1.02 MB, but models are usually 4GB+.
Operating System Requirements
- Mac
- Windows
- Linux
- MacOSX 13.6 or higher.
- Dependencies:
- Node.js version 18 or higher.
- NPM version 9 or higher.
- Homebrew version 3 or higher.
- CPU Instructions Sets: AVX AVX2 AVX512.
- Windows 10 or higher.
- Dependencies:
- Node.js version 18 or higher.
- NPM version 9 or higher.
- CPU Instructions Sets: AVX AVX2 AVX512.
- Debian-based (Supports
.debandAppImage) - Ubuntu-based
- Ubuntu Desktop LTS (official)/ Ubuntu Server LTS (only for server)
- Edubuntu (Mainly desktop)
- Kubuntu (Desktop only)
- Lubuntu (Both desktop and server, though mainly desktop)
- Ubuntu Budgie (Mainly desktop)
- Ubuntu Cinnamon (Desktop only)
- Ubuntu Kylin (Both desktop and server)
- Ubuntu MATE (Desktop only)
- Pacman-based
- Arch Linux based
- Arch Linux (Mainly desktop)
- SteamOS (Desktop only)
- Arch Linux based
- RPM-based (Supports
.rpmandAppImage) - Fedora-based
- RHEL-based (Server only)
- openSUSE (Both desktop and server)
- Dependencies:
- Node.js version 18 or higher.
- NPM version 9 or higher.
- Homebrew version 3 or higher.
- CPU Instructions Sets: AVX AVX2 AVX512.
- MPI or OpenMPI.
info
- Please check whether your Linux distribution supports desktop, server, or both environments.
- For server versions, please refer to the server installation.
GPU Requirements
- NVIDIA
- AMD
- MPI/OpenMPI
- NVIDIA driver version 470.63.01 or higher.
- CUDA Toolkit version 12.3 or higher.
info
CUDA Toolkit dependencies will be installed when you install Cortex.
- AMD driver version 22.7.1 or higher.
- Linux
- Update the package list:
sudo apt update
- Install the OpenMPI binaries and development libraries:
sudo apt install openmpi-bin libopenmpi-dev