# How a Sawmill Helped Speed Up AI **URL:** \ **Category:** Client Agentic AI\ **Tags:** ai, performance, gpu, open-source\ **Created:** [June 23, 2026, 12:07pm UTC](https://devcommunity.amd.com/t/how-a-sawmill-helped-speed-up-ai/632 "2026-06-23T12:07:15Z")\ **Posts on this page:** 9\ **Page:** 1
**Author:** ![Serhii\_Trykhlieb](https://avatars.discourse-cdn.com/v4/letter/s/ebca7d/32.png) [@Serhii\_Trykhlieb](https://devcommunity.amd.com/u/Serhii_Trykhlieb)\ **Post date:** [June 23, 2026, 12:07pm UTC](https://devcommunity.amd.com/t/how-a-sawmill-helped-speed-up-ai/632/1 "2026-06-23T12:07:15Z")
![TurboPrefill_GoldWood](https://global.discourse-cdn.com/amd/original/1X/9502de57cd57859d13d57a25c83aaaaeb6b4cd38.jpeg) For more than 20 years I have been finding and eliminating bottlenecks in factories, production lines, machine tools, and automation systems. In industry, I often saw that system performance is determined not only by the power of individual units, but also by how the workflow between them is organized. Recently, I was simply playing with an old mining rig built from 10 NVIDIA P104-100 GPUs. I ran GPT-OSS-120B on it using llama.cpp. While watching GPU utilization, I noticed a strange picture. Despite having ten GPUs, at any given moment only one GPU was doing most of the work, while the others were waiting for their turn. This looked surprisingly familiar. I had seen something similar on one of my sawmill production lines, where a log passes through several machines sequentially: while one machine is working, the rest of the line sits idle. Just like on the sawmill line, I decided to feed the “tree trunks” more densely. But first, a little theory. llama.cpp has a mode that splits a model across multiple GPUs by layers. This makes it possible to distribute a large model across the VRAM of several GPUs when a single GPU does not have enough memory. Each GPU processes its own part of the data and then passes the already “chewed” data to the next GPU. It is important that the data passes through the layers strictly in sequence — from the first layer to the last. Later data must not overtake earlier data. Continuing the sawmill analogy, the next tree trunk had to wait until the previous tree trunk passed through every machine in the line. For the “decode” stage this is indeed a necessary condition — generating the next token requires the previous one. For the “prefill” stage, however, this is not necessary. A tree trunk can be fed into a machine as soon as the previous tree trunk has cleared that machine, without waiting for it to pass through all the remaining machines in the line. The only requirement is to preserve the correct order. This is how TurboPrefill was born — a mechanism called “Intra-Prompt Pipeline Scheduling for Multi-GPU Prefill”. To avoid breaking existing mechanisms, I added two things to llama.cpp: 1. A “sorting table” that selects and accumulates suitable “tree trunks” (micro-batches (UB) belonging to a batch (B)). 2. A separate turbo pipeline into which “tree trunks” can be fed more densely. The difference from the standard pipeline is that each machine has its own table where a tree trunk can sit and wait until the machine becomes available. In the standard pipeline there is only one table shared by all GPUs, and it effectively “runs” from machine to machine as the tree trunk progresses through the line. “Tree trunks” that did not meet the required conditions were sent to the regular pipeline. In the first implementation: GitHub repository: sergey-automation/TurboPrefill I used strict selection criteria and only accepted “tree trunks containing text data only”. In the next implementation: GitHub repository: sergey-automation/TurboPrefill-VLM-Validation I relaxed the selection criteria and, in addition to text data, started accepting “visual” data as well — situations where a user uploads an image (or screenshot) and asks the AI to answer questions about it. As a result, the responsiveness of AI interactions improved significantly. The waiting time between asking a question and receiving the beginning of an answer was reduced by 1.6× for two-GPU pipelines and by up to 2.2× for eight-GPU pipelines. For example, if you upload a screenshot of an error and ask how to fix it, or upload a document and ask for a brief explanation of its contents, TurboPrefill allows the answer to begin several seconds sooner. The larger the document, image, or request, the greater the time savings compared to the standard pipeline. The speedup is achieved entirely through scheduling. The mathematics and logic of the model remain unchanged and do not affect answer quality. The mechanism itself is not tied to any particular GPU architecture. It has been tested on several GPU generations — from Pascal (GTX 1080, P104-100) to Ampere (RTX 3090) and Blackwell (RTX 5060 Ti). I proposed, introduced, and described the “Intra-Prompt Pipeline Scheduling for Multi-GPU Prefill” mechanism here: > > > \# \[RFC\]\[PoC\] Intra-Prompt Pipeline Scheduling for Multi-GPU Prefill > > > \## 1. S…tatus: Proof of Concept (PoC) > > This RFC describes an experimental scheduling mechanism implemented in the TurboPrefill project and intended for discussion of the architectural approach. > > \## 2. The Problem > > With \`split\_mode = layer\`, processing of a single request during the \`prefill\` phase is performed by passing each \`ubatch\` sequentially through the GPU pipeline. > > When processing a single \`ubatch\`, GPU stages are utilized sequentially: early GPUs become idle after completing their work, while later GPUs remain idle waiting for input data. > > This limits GPU utilization efficiency and reduces performance scaling as the number of devices increases. > > \## 3. Proposed Approach > > Within the scope of this RFC, the term \*\*Intra-Prompt Pipeline Scheduling\*\* refers to the request-internal scheduling mechanism for the \`prefill\` phase described below. > > The proposed approach does not modify the division of a request into \`ubatch\` units. > > During the \`prefill\` phase, \`ubatch\` instances are pre-classified. \`ubatch\` instances requiring the standard execution order continue to be processed by the existing scheduling mechanism. \`ubatch\` instances suitable for pipeline execution are routed to the Intra-Prompt Pipeline Scheduling mode implemented in TurboPrefill. > > In this mode, \`ubatch\` instances are accumulated until the final \`ubatch\` of the current batch is received, after which they are executed sequentially through the layer-split GPU pipeline. > > The next \`ubatch\` begins processing immediately after the previous \`ubatch\` completes execution on the corresponding GPU stage. It is therefore not necessary to wait for the previous \`ubatch\` to pass through all GPUs before starting execution of the next one. > > As a result, the number of GPUs simultaneously performing useful work increases. > > \## 4. Scope > > Intra-Prompt Pipeline Scheduling is intended to accelerate the \`prefill\` phase when processing large single requests on multi-GPU configurations. > > The approach is designed for \*\*Single-User Mode\*\*, where compute resources are dedicated to a single active request. > > \## 5. Limitations > > Intra-Prompt Pipeline Scheduling is a request-internal scheduling mechanism. It does not introduce parallelism between independent requests and does not replace existing multi-user batching mechanisms. > > The approach is intended to accelerate the \`prefill\` phase and does not affect \`decode\` performance. > > Intra-Prompt Pipeline Scheduling is applied only to \`ubatch\` instances that do not contain logits output requests. > > For this reason, the standard \`llama-bench\` tool is not suitable for evaluating this mode, since its prefill tests contain logits output requests and therefore continue to be processed by the standard scheduling path. > > \## 6. PoC Results > > Results for a context length of 16,373 tokens. Full results for all tested context lengths are provided below and are available in the repository. > > \`\`\`text > Platform GPU Baseline TurboPrefill Gain > \------------------------------------------------------------------ > 10x P104-100 10 77 181 +135% > 4x RTX 3090 4 1477 2758 +87% > 5x RTX 5060 Ti 5 1993 3886 +95% > 8x RTX 5060 Ti 8 1963 4380 +123% > \`\`\` > > \## 7. Computational Correctness > > Intra-Prompt Pipeline Scheduling does not modify the mathematical computation model. > > The execution order of individual layers and the order of data propagation through each layer remain unchanged. > > Only the execution scheduling order between GPU stages is modified. > > \## 8. Discussion Topics > > \* Possible integration into \`ggml-backend-sched\`. > \* Applicability to other split modes. > \* Applicability to systems with P2P/NVLink support. > > This RFC is published for discussion of the architectural approach and for collecting feedback on possible future directions of development. > \## 9. Implementation Status > > The current implementation is provided as an isolated overlay module and does not require modifications to the model architecture. > > The mode can be enabled through the \`TURBOPREFILL=1\` environment variable. > > Source code, installation scripts, benchmark data, and measurement results are available in the separate TurboPrefill repository. > > \## 10. Full Benchmark Results > > All measurements in this section were performed using GPT-OSS-120B Q4\_K\_M. > > \### Multi-GPU Scaling with and without Intra-Prompt Pipeline Scheduling > > !\[Multi-GPU Scaling with and without Intra-Prompt Pipeline Scheduling\](https://raw.githubusercontent.com/sergey-automation/TurboPrefill/main/doc/graphs/ipps\_prefill\_scaling\_5gpu\_8gpu\_rtx5060ti.png) > > \*5× and 8× NVIDIA RTX 5060 Ti 16GB, UB=768.\* > > \--- > > \### Prefill Throughput: Baseline vs Intra-Prompt Pipeline Scheduling (4× RTX 3090) > > !\[Prefill Throughput on 4× RTX 3090\](https://raw.githubusercontent.com/sergey-automation/TurboPrefill/main/doc/graphs/ipps\_prefill\_4gpu\_rtx3090.png) > > \*4× NVIDIA RTX 3090, UB=768.\* > > \--- > > \### Prefill Throughput on 10× NVIDIA P104-100 (Pascal) > > !\[Prefill Throughput on 10× NVIDIA P104-100 (Pascal)\](https://raw.githubusercontent.com/sergey-automation/TurboPrefill/main/doc/graphs/ipps\_prefill\_p104\_pascal\_ub32\_vs\_ub512.png) > > \*10× NVIDIA P104-100 (Pascal). Baseline UB=32, Baseline UB=512, Intra-Prompt Pipeline Scheduling UB=32.\* > > Author: Serhii Trykhlieb > > Proposed mechanism: > Intra-Prompt Pipeline Scheduling for Multi-GPU Prefill > > PoC implementation: > TurboPrefill > > Repository: > https://github.com/sergey-automation/TurboPrefill And created several implementations: sergey-automation/TurboPrefill > **[GitHub - sergey-automation/TurboPrefill-VLM-Validation: Validation of Intra-Prompt Pipeline Scheduling for...](https://github.com/sergey-automation/TurboPrefill-VLM-Validation)** > > Validation of Intra-Prompt Pipeline Scheduling for Multi-GPU Prefill using VLM workloads The project is evolving. I invite you to participate in testing. Best regards, Serhii Trykhlieb ---
**Author:** ![Hakob\_Arzumanyan](https://dub1.discourse-cdn.com/amd/user_avatar/devcommunity.amd.com/hakob_arzumanyan/32/214_2.png) [@Hakob\_Arzumanyan](https://devcommunity.amd.com/u/Hakob_Arzumanyan)\ **Post date:** [June 23, 2026, 5:37pm UTC](https://devcommunity.amd.com/t/how-a-sawmill-helped-speed-up-ai/632/2 "2026-06-23T17:37:27Z")
Hey, love this, neat work. Two things I’d keep an eye on: adding more trunks at once means more GPUs need to hold activations simultaneously, so with a tight VRAM split, a long prompt can hit memory limits even if the old setup didn’t, did you hit it already? Also, mining rigs often use x1 risers, so I would have expected the PCIe transfer between GPUs to be the main bottleneck, not GPU idle time. How are yours connected, was the bus not actually the bottleneck? ---
**Author:** ![Serhii\_Trykhlieb](https://avatars.discourse-cdn.com/v4/letter/s/ebca7d/32.png) [@Serhii\_Trykhlieb](https://devcommunity.amd.com/u/Serhii_Trykhlieb)\ **Post date:** [June 23, 2026, 7:53pm UTC](https://devcommunity.amd.com/t/how-a-sawmill-helped-speed-up-ai/632/3 "2026-06-23T19:53:54Z")
Thanks! The PCIe side was actually one of the biggest surprises. The original experiments were performed on a mining rig with 10× NVIDIA P104-100 GPUs connected through PCIe Gen1 x1 risers. I initially expected PCIe bandwidth to be the primary bottleneck. However, measurements showed that prefill throughput increased from approximately 77 tok/s to 181 tok/s after introducing TurboPrefill. In this configuration, the observed limitation appeared to be GPU idle time and scheduling inefficiencies rather than raw PCIe bandwidth. So while PCIe Gen1 x1 is certainly not an ideal interconnect, these results suggest that significant performance gains can still be achieved by improving scheduling even on extremely bandwidth-constrained systems. ---
**Author:** ![Serhii\_Trykhlieb](https://avatars.discourse-cdn.com/v4/letter/s/ebca7d/32.png) [@Serhii\_Trykhlieb](https://devcommunity.amd.com/u/Serhii_Trykhlieb)\ **Post date:** [June 23, 2026, 8:02pm UTC](https://devcommunity.amd.com/t/how-a-sawmill-helped-speed-up-ai/632/4 "2026-06-23T20:02:33Z")
> [@Hakob\_Arzumanyan](#): > > Hey, love this, neat work. Two things I’d keep an eye on: adding more trunks at once means more GPUs need to hold activations simultaneously, so with a tight VRAM split, a long prompt can hit memory limits even if the old setup didn’t, did you hit it already? Regarding memory usage, in my experiments TurboPrefill was usually tuned with a smaller UB value than the baseline configuration. As a result, the CUDA compute buffer was often not larger and in some cases was actually smaller than in the baseline runs that used a larger UB. ---
**Author:** ![Serhii\_Trykhlieb](https://avatars.discourse-cdn.com/v4/letter/s/ebca7d/32.png) [@Serhii\_Trykhlieb](https://devcommunity.amd.com/u/Serhii_Trykhlieb)\ **Post date:** [June 23, 2026, 8:08pm UTC](https://devcommunity.amd.com/t/how-a-sawmill-helped-speed-up-ai/632/5 "2026-06-23T20:08:11Z")
The logs are available on GitHub, so memory usage and GPU utilization can be compared directly between the baseline and TurboPrefill runs. The repositories also contain the scripts needed to reproduce the measurements independently. ---
**Author:** ![Serhii\_Trykhlieb](https://avatars.discourse-cdn.com/v4/letter/s/ebca7d/32.png) [@Serhii\_Trykhlieb](https://devcommunity.amd.com/u/Serhii_Trykhlieb)\ **Post date:** [July 1, 2026, 9:25pm UTC](https://devcommunity.amd.com/t/how-a-sawmill-helped-speed-up-ai/632/6 "2026-07-01T21:25:00Z")
![image](https://global.discourse-cdn.com/amd/original/1X/3b71a8dfcf6545e04a4951334a70b0c165b69d09.png) ---
**Author:** ![Serhii\_Trykhlieb](https://avatars.discourse-cdn.com/v4/letter/s/ebca7d/32.png) [@Serhii\_Trykhlieb](https://devcommunity.amd.com/u/Serhii_Trykhlieb)\ **Post date:** [July 1, 2026, 9:25pm UTC](https://devcommunity.amd.com/t/how-a-sawmill-helped-speed-up-ai/632/7 "2026-07-01T21:25:34Z")
![image](https://global.discourse-cdn.com/amd/original/1X/dfdbe2a2c1d615c381bee796d539f0806fe32ec6.png) ---
**Author:** ![Serhii\_Trykhlieb](https://avatars.discourse-cdn.com/v4/letter/s/ebca7d/32.png) [@Serhii\_Trykhlieb](https://devcommunity.amd.com/u/Serhii_Trykhlieb)\ **Post date:** [July 1, 2026, 9:30pm UTC](https://devcommunity.amd.com/t/how-a-sawmill-helped-speed-up-ai/632/8 "2026-07-01T21:30:04Z")
![image](https://global.discourse-cdn.com/amd/original/1X/f5c1d8b1c936c081e60441cb4383c3b6956e1e83.png) ---
**Author:** ![Serhii\_Trykhlieb](https://avatars.discourse-cdn.com/v4/letter/s/ebca7d/32.png) [@Serhii\_Trykhlieb](https://devcommunity.amd.com/u/Serhii_Trykhlieb)\ **Post date:** [July 1, 2026, 9:30pm UTC](https://devcommunity.amd.com/t/how-a-sawmill-helped-speed-up-ai/632/9 "2026-07-01T21:30:40Z")
> **[TurboPrefill-VLM-Validation/benchmarks/Llama-3-70B-Dense at main ·...](https://github.com/sergey-automation/TurboPrefill-VLM-Validation/tree/main/benchmarks/Llama-3-70B-Dense)** > > main/benchmarks/Llama-3-70B-Dense