Reverse Proxy

A reverse proxy server retrieves resources from one or more servers on a client’s behalf. These resources are then returned to the client, appearing to originate from the source server itself. It abstracts your infrastructure setup from the end users, which is useful for implementing scaling, security middleware, and load balancing.

While forward and reverse proxies may seem functionally similar, their differences lie primarily in their use cases and perspectives. A forward proxy acts on behalf of clients seeking resources from various servers, often used for client privacy and access control. A reverse proxy acts on behalf of servers, making resources available to clients while hiding the backend server details.

That means a reverse proxy hides its presence from the clients and acts as an intermediary between them and the servers. When you communicate with a reverse proxy, it is as if you communicated directly with the target server.

That is one of the primary differences between forward proxy and a reverse proxy.

You can combine both forward proxy and reverse proxy to create a gateway.

Paddler

Paddler is a reverse proxy server and load balancer made specifically for llama.cpp. You can communicate with it like a regular llama.cpp instance. You can learn more on it’s dedicated page.

Gateway

Functionally, forward and reverse proxies are similar in that they both act as intermediaries handling requests. The key difference lies in the direction of the request. A forward proxy is used by clients to forward requests to other servers, often used to bypass network restrictions or for caching. It announces its presence to the end user. In contrast, servers use a reverse proxy to forward responses to clients, often for load balancing, security, and caching purposes, and it hides its presence from the end user.

When combined, forward and reverse proxies can create a gateway. A gateway serves as a front-end for underlying services and acts as an entry point for users to the application. It handles both incoming client requests and outgoing server responses.

Temperature

The temperature parameter in LLMs controls the randomness of the output. Lower temperatures make the output more deterministic (less creative), while higher temperatures increase variability. Even at low temperatures, some variability remains due to the probabilistic nature of the models.

llama.cpp

Llama.cpp is a production-ready, open-source runner for various Large Language Models.

It has an excellent built-in server with HTTP API.

In this handbook, we will use Continuous Batching, which in practice allows handling parallel requests.

Installing on AWS EC2 with CUDA

This tutorial was tested on g4dn.xlarge instance with Ubuntu 22.04 operating system. This tutorial was written explicitly to perform the installation on a Ubuntu 22.04 machine.

Installation Steps

1. Start an EC2 instance of any class with a GPU with CUDA support.

   If you want to compile llama.cpp on this instance, you will need at least 4GB for CUDA drivers and enough space for your LLM of choice. I recommend at least 30GB. Perform the following steps of this tutorial on the instance you started.

2. Install build dependencies:

   sudo apt update
   

   sudo apt install build-essential ccache
   

3. Install CUDA Toolkit (only the Base Installer). Download it and follow instructions from https://developer.nvidia.com/cuda-downloads

   At the time of writing this tutorial, the highest available supported version of the Ubuntu version was 22.04. But do not fear! :) We’ll get it to work with some minor workarounds (see the Potential Errors section)

4. Install NVIDIA Drivers:

   sudo apt install nvidia-driver-555
   

5. Compile llama.cpp:

   git clone https://github.com/ggerganov/llama.cpp.git
   

   cd llama.cpp
   

   GGML_CUDA=1 make -j
   

6. Benchmark llama.cpp (optional):

   Follow the official tutorial if you intend to run the benchmark. However, keep using GGML_CUDA=1 make to compile the llama.cpp (do not use LLAMA_CUBLAS=1): https://github.com/ggerganov/llama.cpp/discussions/4225

   Instead of performing a model quantization yourself, you can download quantized models from Hugging Face. For example, Mistral Instruct you can download from https://huggingface.co/TheBloke/Mistral-7B-Instruct-v0.2-GGUF/tree/main

Potential Errors

CUDA Architecture Must Be Explicitly Provided

ERROR: For CUDA versions < 11.7 a target CUDA architecture must be explicitly 
provided via environment variable CUDA_DOCKER_ARCH, e.g. by running 
"export CUDA_DOCKER_ARCH=compute_XX" on Unix-like systems, where XX is the 
minimum compute capability that the code needs to run on. A list with compute 
capabilities can be found here: https://developer.nvidia.com/cuda-gpus

You need to check the mentioned page (https://developer.nvidia.com/cuda-gpus) and pick the appropriate version for your instance’s GPU. g4dn instances use T4 GPU, which would be compute_75.

For example:

CUDA_DOCKER_ARCH=compute_75 GGML_CUDA=1 make -j

Failed to initialize CUDA

ggml_cuda_init: failed to initialize CUDA: unknown error

Sometimes can be solved with sudo modprobe nvidia_uvm.

You can also create a Systemd unit that loads the module on boot:

[Unit]
After=nvidia-persistenced.service

[Service]
Type=oneshot
ExecStart=/usr/sbin/modprobe nvidia_uvm

[Install]
WantedBy=multi-user.target

NVCC not found

/bin/sh: 1: nvcc: not found

You need to add CUDA path to your shell environmental variables.

For example, with Bash and CUDA 12:

export PATH="/usr/local/cuda-12/bin:$PATH"

export LD_LIBRARY_PATH="/usr/local/cuda-12/lib64:$LD_LIBRARY_PATH"

cannot find -lcuda

/usr/bin/ld: cannot find -lcuda: No such file or directory

That means your Nvidia drivers are not installed. Install NVIDIA Drivers first.

Cannot communicate with NVIDIA driver

NVIDIA-SMI has failed because it couldn't communicate with the NVIDIA driver. Make sure that the latest NVIDIA driver is installed and running.

If you installed the drivers, reboot the instance.

Failed to decode the batch

failed to decode the batch, n_batch = 0, ret = -1
main: llama_decode() failed

There are two potential causes of this issue.

Option 1: Install NVIDIA drivers

Make sure you have installed the CUDA Toolkit and NVIDIA drivers. If you do, restart your server and try again. Most likely, NVIDIA kernel modules are not loaded.

sudo reboot

Option 2: Use different benchmarking parameters

For example, with Mistral Instruct 7B what worked for me is:

./llama-batched-bench -m ../mistral-7b-instruct-v0.2.Q4_K_M.gguf 2048 2048 512 0 999 128,256,512 128,256 1,2,4,8,16,32

Installing with AWS Image Builder

This tutorial explains how to install llama.cpp with AWS EC2 Image Builder.

By putting llama.cpp in EC2 Image Builder pipeline, you can automatically build custom AMIs with llama.cpp pre-installed.

You can also use that AMI as a base and add your foundational model on top of it. Thanks to that, you can quickly scale up or down your llama.cpp groups.

We will repackage the base EC2 tutorial as a set of Image Builder Components and Workflow.

You can complete the tutorial steps either manually or by automating the setup with Terraform/OpenTofu. Terraform source files are linked to their respective tutorial steps.

Installation Steps

1. Create an IAM imagebuilder role (source file)

   Go to the IAM Dashboard, click “Roles” from the left-hand menu, and select “AWS service” as the trusted entity type. Next, select “EC2” as the use case:

   

   Next, assign the following policies:

   • arn:aws:iam::aws:policy/service-role/AmazonEC2ContainerServiceforEC2Role
   • arn:aws:iam::aws:policy/EC2InstanceProfileForImageBuilderECRContainerBuilds
   • arn:aws:iam::aws:policy/EC2InstanceProfileForImageBuilder
   • arn:aws:iam::aws:policy/AmazonSSMManagedInstanceCore

   Name your role (for example, “imagebuilder”) and finish creating it. You should end up with permissions and trust relationships looking like this:

   

2. Create components.

   We’ll need the following four components:

   • llama.cpp build dependencies. It needs to install build-essentials and ccache (source file)
   • CUDA toolkit (source file)
   • NVIDIA driver (source file)
   • llama.cpp itself (source file)

   To create the component via GUI, navigate to EC2 Image Builder service on AWS. From there, select “Components” from the menu. We’ll need to add four components that will act as the building blocks in our Image Builder pipeline. You can refer to the generic EC2 tutorial for more details for more information.

   Click “Create component”. Next, for each component:

   • Choose “Build” as the component type
   • Select “Linux” as the image OS
   • Select “Ubuntu 22.04” as the compatible OS version

   Provide the following as component names and contents in YAML format:

   Component name: apt_build_essential

    name: apt_build_essential
    description: "Component to install build essentials on Ubuntu"
    schemaVersion: '1.0'
    phases:
      - name: build
        steps:
          - name: InstallBuildEssential
            action: ExecuteBash
            inputs:
              commands:
                - sudo apt-get update
                - DEBIAN_FRONTEND=noninteractive sudo apt-get install -yq build-essential ccache
            onFailure: Abort
            timeoutSeconds: 180
   

   Component name: apt_nvidia_driver_555

    name: apt_nvidia_driver_555
    description: "Component to install NVIDIA driver 550 on Ubuntu"
    schemaVersion: '1.0'
    phases:
      - name: build
        steps:
          - name: apt_nvidia_driver_555
            action: ExecuteBash
            inputs:
              commands:
                - sudo apt-get update
                - DEBIAN_FRONTEND=noninteractive sudo apt-get install -yq nvidia-driver-550
            onFailure: Abort
            timeoutSeconds: 180
          - name: reboot
            action: Reboot
   

   Component name: cuda_toolkit_12

    name: cuda_toolkit_12
    description: "Component to install CUDA Toolkit 12 on Ubuntu"
    schemaVersion: '1.0'
    phases:
      - name: build
        steps:
          - name: apt_cuda_toolkit_12
            action: ExecuteBash
            inputs:
              commands:
                - DEBIAN_FRONTEND=noninteractive sudo apt-get -yq install nvidia-cuda-toolkit
            onFailure: Abort
            timeoutSeconds: 600
          - name: reboot
            action: Reboot
   

   Component name: llamacpp_gpu_compute_75

    name: llamacpp_gpu_compute_75
    description: "Component to install and compile llama.cpp with CUDA compute capability 75 on Ubuntu"
    schemaVersion: '1.0'
    phases:
      - name: build
        steps:
          - name: compile
            action: ExecuteBash
            inputs:
              commands:
                - cd /opt
                - git clone https://github.com/ggerganov/llama.cpp.git
                - cd llama.cpp
                - |
                  CUDA_DOCKER_ARCH=compute_75 \
                  LD_LIBRARY_PATH="/usr/local/cuda-12/lib64:$LD_LIBRARY_PATH" \
                  GGML_CUDA=1 \
                  PATH="/usr/local/cuda-12/bin:$PATH" \
                  make -j
            onFailure: Abort
            timeoutSeconds: 1200
   

   Once you’re finished, you’ll see all the created components you added on the list:

   

3. Add Infrastructure Configuration source file

   Next, we’ll create a new Infrastructure Configuration. Select it from the left-hand menu and click “Create”. You’ll need to use g4dn.xlarge instance type or any other instance type that supports CUDA. Name your configuration, select the IAM role you created in step 1, and select the instance, for example:

   

4. Add Distribution Configuration source file

   Select Distribution settings in the left-hand menu to create a Distribution Configuration. It specifies how the AMI should be distributed (on what type of base AMI it will be published). Select Amazon Machine Image, name the configuration, and save:

   

5. Add Image Pipeline source file

   Next, we’ll add the Image Pipeline. It will use the Components, Infrastructure Configuration, and Distribution Configuration we prepared previously. Select “Imagie Pipeline” from the left-hand menu and click “Create”. Name your image pipeline, and select the desired build schedule.

   As the second step, create a new recipe. Choose AMI, name the recipe:

   

   Next, select the previously created components:

   

6. The next step is to build the image. You should be able to run the pipeline:

   

7. Launch test EC2 Instance.

   When launching EC2 instance, the llama.cpp image we prepared should be available under My AMIs list:

   

Summary

Feel free to open an issue if you find a bug in the tutorial or have ideas on how to improve it.

Ollama

Ollama is a convenient and easy-to-use wrapper around llama.cpp.

It acts like a llama.cpp multiplexer. You can start a new conversation or request completion from a specific LLM without manually downloading weights and setting them up.

For example, when you request completion from a model that is not yet loaded, it checks if it is possible to fit that new model into RAM or VRAM. If so, it internally starts a new llama.cpp instance and uses it to load the LLM. If the requested model has not yet been downloaded, it will download it for you.

In general terms, it acts like a llama.cpp forward proxy and a supervisor.

For example, if you load both llama3 and phi-3 into the same Ollama instance, you will get something like this:

flowchart TD
    Ollama --> llama1[llama.cpp with llama3]
    Ollama --> llama2[llama.cpp with phi-3]
    llama1 --> VRAM
    llama2 --> VRAM

Viability for Production

Predictability

Although the Ollama approach is convenient for local development, it causes some deployment problems (compared to llama.cpp).

With llama.cpp, it is easily possible to divide the context of the loaded model into a specific number of slots, which makes it extremely easy to predict how many parallel requests the current server can handle.

It is not easy to predict with Ollama. It manages the slots internally and does not expose the equivalent with the llama.cpp /health endpoint to monitor the currently used resources. Even if it did, it is always possible to have a few different models loaded simultaneously that share server resources.

We might end up in a situation where Ollama keeps both llama3 (which is 70B parameter model) and phi-3 (which is 3.8B parameter model). A completion request towards llama3 will use many more resources than asking phi-3 for completion. 8 slots of llama3 require many more resources than 8 of phi-3.

How can that be balanced effectively? As a software architect, you would have to plan an infrastructure that does not allow developers to randomly load models into memory and force a specific number of slots, which defeats the purpose of Ollama.

Good Parts of Ollama

I greatly support Ollama because it makes it easy to start your journey with large language models. You can use Ollama in production deployments, but llama.cpp is a better choice because it is predictable.

Ollama is better suited than llama.cpp for end-user distributable applications. By that, I mean the applications that do not use an external server but are installed and run in their entirety on the user’s device. The same thing that makes it less predictable regarding resource usage makes it more resilient to end-user errors. In that context, resource usage predictability is less important than on the server side.

That is why this handbook is almost entirely based on vanilla llama.cpp as it is much better for server-side deployments (based on all the reasons above).

The situation might change with the future Ollama releases.

Paddler

Paddler is an open-source, stateful load balancer and reverse proxy designed for servers running llama.cpp. Unlike typical strategies like round-robin or least connections, Paddler uses each server’s available slots.

It uses agents to monitor the health of llama.cpp instances and dynamically adjust to adding or removing servers, making it easier to integrate with autoscaling tools.

Fine-tuning

Fine-tuning is taking a pre-trained model and further training it on a new task. This is typically useful when you want to repurpose a model trained on a large-scale dataset for a new task with less data available.

In practice, that means fine-tuning allows the model to adapt to the new data without forgetting what it has learned before.

A good example might be the sqlcoder model, which is a fine-tuned starcoder model (which is a general coding model) to be exceptionally good at producing SQL.

Retrieval Augmented Generation

Retrieval augmented generation does not modify the underlying model in any way. Instead, it is an approach to directly influence its responses.

In practice, and in a significant simplification, RAG is about injecting data into Large Language Model prompt.

For example, let’s say the user asks the LLM:

• What are the latest articles on our website?

To augment the response, you need to intercept the user’s question and tell LLM to respond in a way more or less like:

• You are a <inser persona here>. Tell the user that the latest articles on our site are <insert latest articles metadata here>

That is greatly simplified, but generally, that is how it works. Along the way, embeddings and vector databases are involved.

Predictability

The common issue with Large Language Models is the consistency and structure of outputs.

Software Engineering vs AI

The last few decades of IT developments have accustomed us to extreme predictability. Each time we call a specific API endpoint or use a specific button, the same thing happens consistently, under our complete control.

That is not the case with AI, which operates on probabilities. That stems from the approach to creating software. Neural network designers design just the network and the training process, but they do not design the actual reasoning. The reasoning is learned by the network during training, and it is not under the control of the designer.

That is totally different from the traditional software development process, where we design the reasoning and the process, and the software just executes it.

That is why you might feed Large Language Models with the same prompt multiple times and get different outputs each time. Temperature parameter may be used to limit the “creativeness” of the model, but even setting it to zero does not guarantee predictable outputs.

Structured Outputs

While LLMs not being completely predictable may cause some issues, but no technical solution is completely one-sided and we can turn that flexibility into our advantage.

LLMs are extremely good in understanding natural language. In practice we can finally communicate with computers in a similar way we communicate with other people. We can create systems that interpret such unstructured inputs and react to them in a structured and predictable way. This way we can use the good parts of LLMs to our advantage and mitigate most of the unredictability issues.

Use Cases

Some use cases include (but are not limited to):

• Searching through unstructured documents (e.g., reports in .pdf, .doc, .csv, or plain text)
• Converting emails into actionable structures (e.g., converting requests for quotes into API calls with parameters for internal systems)
• Question answering systems that interpret the context of user queries

Matching the Grammar

Some Large Language Model runners support formal grammars. It can be used to force the output to follow a certain structure (for example, speaking only in emojis or outputting just the valid moves from the Portable Game Notation).

It still does not guarantee that the output will be valid (in a semantic sense), but at least matching the formal grammar guarantees it will follow the correct structure.

One of the popular uses is to force a Large Language Model to match a specific JSON Schema.

llama.cpp

llama.cpp supports GBNF formal grammars, which is an extension of Backus-Naur form, but with the support of some regular expressions.

Application Layer

This chapter is not strictly related to LLMOps, but discussing the best practices for architecting and developing applications that use them would be a good idea.

Those applications have to deal with some issues that are not typically met in traditional web development, primarily long-running HTTP requests or MLOps - using custom models for inference.

Up until Large Language Models became mainstream and in demand by a variety of applications, the issue of dealing with long-running requests was much less prevalent. Typically, due to functional requirements, all the microservice requests normally would take 10ms or less, while waiting for a Large Language Models to complete the inference can take multiple seconds.

That calls for some adjustments in the application architecture, non-blocking Input/Output and asynchronous programming.

This is where asynchronous programming languages shine, like Python with its asyncio library or Rust with its tokio library, Go with its goroutines, etc.

Programming languages like PHP, which are synchronous by default, might struggle unless supplemented by extensions like Swoole (which essentially gives PHP Go-like coroutines) or libraries like AMPHP. Introducing support for asynchronous programming in PHP can be a challenge, but it is possible.

Long-Running

In web development, there are two primary application models:

1. Long-running processes (for example, a web application written in Go that keeps running and the same process responds to multiple incoming requests)
2. Worker-based, single-threaded, synchronous (for example, PHP with FPM, Ruby, and some Python setups - it is generally used by scripted languages)

It is not necessarily connected to a specific language (for example, PHP can also start a long-running process with a web server, but most PHP frameworks were not designed with that application model in mind and without extensions like Swoole, it won’t be preemptive).

Python can run synchronously with some frameworks and WSGI, but it can also be run as a long-running application with ASGI or projects like Granian.

The Problem with the Worker-Based Synchronous Model

We will use PHP-FPM as an example. On Debian, it comes preconfigured with the max_children parameter set to 5 by default, which means it can spawn at most 5 workers and handle at most 5 requests in parallel. This parameter can be tweaked, and under normal circumstances, it can be changed to a much higher value at the cost of RAM memory used.

Let’s assume we have 32 workers running. Normally, the time to respond to a request takes at most milliseconds, and this runtime model is not an issue, but working with LLMs or ML models turns that around. Typical requests to LLM can take multiple seconds to complete, sometimes 20-30 seconds, depending on the number of tokens to be generated. That means if we have 32 requests in parallel to our application (which is not a lot), all the synchronous PHP workers can be blocked while waiting for tokens to be generated.

We can end up in a situation where our server’s CPU is almost idling, but it can’t accept more requests due to an Input/Output bottleneck.

flowchart TD
    client[Client]

    client---|Request|response_handler

    subgraph worker[Worker]
    response_handler[Response Handler]---response["Response"]
    end

    subgraph llm[LLM]
    response_handler-->|Waits for|llmserver[Completion]
    llmserver-->response
    end

While the tokens are generated, the worker is at a standstill and cannot accept additional requests.

The Solution

The solution is to use languages and frameworks that support long-running processes (that do not rely on spawning workers) and with any form of asynchronicity.

The perfect example of such language is Go with its goroutines. Each goroutine uses just a few kilobytes of memory, and a server with a few gigabytes of RAM can potentially spawn millions of them. They run asynchronously and are preemptive, so there shouldn’t be a situation where just a few requests can exhaust the entire server capacity.

Asynchronous Programming

By asynchronous programming, we mean the ability to execute multiple tasks concurrently without blocking the main thread; that does not necessarily involve using threads and processes. A good example is the JavaScript execution model, which is, by default, single-threaded but asynchronous. It does not offer parallelism (without worker threads), but it can still issue concurrent network requests, database queries, etc.

Considering that most of the bottlenecks related to working with Large Language Models stem from Input/Output issues (primarily the LLM APIs response times and the time it takes to generate all the completion tokens) and not the CPU itself, asynchronous programming techniques are often the necessity when architecting the applications.

When it comes to network requests, large language models pose a different challenge than most web applications. While most of the REST APIs tend to have consistent response times below 100ms, when working with large language model web APIs, the response times might easily reach 20-30 seconds until all the requested tokens are generated and streamed.

Affected Runtimes

Scripting languages like PHP and Ruby are primarily affected because they are synchronous by default. That is especially cumbersome with PHP, which uses FPM pool of workers as a common hosting method. For example, Debian’s worker pool amounts to five workers by default. That means if each of them would be busy handling 30-second requests, the sixth request would have to wait for the first one to finish. That also means that you can easily run into a situation where your server’s CPU is idling, but it can’t accept more requests simultaneously.

Coroutines, Promises to the Rescue

To mitigate the issue, you can use any programming language supporting async, which primarily manifests in supporting Promises (Futures) or Coroutines. That includes JavaScript, Golang, Python (with asyncio), and PHP (with Swoole).

Preemptive vs Cooperative Scheduling

It is also really important to understand the preemptive aspect of async languages. Although preemptiveness is an aspect primarily of threading, it plays a role when scheduling promises and coroutines. For example, PHP natively implements Fibers, which grants it some degree of asynchronicity, although they are not preemptive. This means if you try something silly in your code, like, for example:

<?php

startCoroutine(function () {
    while (true) { echo 'hi'; }
});

PHP’s built-in scheduler will never give time to other asynchronous functions, meaning your script will get stuck on that specific coroutine. The same applies to Node.js and JavaScript Promises.

On the other hand, Go is preemptive by default, meaning that the runtime will automatically switch between coroutines, and you don’t have to worry about it. That is especially useful because you do not have to worry about infinite loops or blocking requests as long as you structure your code around coroutines.