Year of our Lord 2026. LLMs are everywhere.
Built Lex, a context-aware dictionary.
|
|
Components: lex-cgi, lex-llm, lex-mailer. User emails word and context. lex-cgi receives webhook, runs lex-llm, writes definition to file. lex-mailer emails past entries daily, timed for spaced repetition.
Tried local inference using llama.cpp + Llama 3.1 8B (GGUF) first:
llama_backend_init();
mparams = llama_model_default_params();
mparams.n_gpu_layers = 0; /* force all layers onto CPU */
model = llama_model_load_from_file(MODEL_PATH, mparams);
cparams = llama_context_default_params();
cparams.n_ctx = N_CTX; /* 1024 */
llama_init_from_model(model, cparams);
Generation loop synthesizes definition:
for (i = 0; i < MAX_TOKENS; i++) {
new_token_id = llama_sampler_sample(smpl, ctx, -1);
llama_sampler_accept(smpl, new_token_id);
if (llama_vocab_is_eog(vocab, new_token_id))
break;
char buf[128];
n = llama_token_to_piece(vocab, new_token_id, buf, sizeof(buf), 0, false);
if (n > 0) {
fwrite(buf, 1, (size_t)n, out);
fflush(out);
}
batch = llama_batch_get_one(&new_token_id, 1);
if (llama_decode(ctx, batch) != 0)
break;
}
Build what’s needed:
cmake -S $(LLAMA_SRC_DIR) -B $(LLAMA_BUILD) \
-DBUILD_SHARED_LIBS=OFF \
-DGGML_OPENMP=OFF \
-DGGML_VULKAN=OFF \
-DLLAMA_BUILD_SERVER=OFF \
-DLLAMA_BUILD_UI=OFF
# + misc size-reduction flags, omitted
cmake --build $(LLAMA_BUILD) --config Release -j$$(sysctl -n hw.ncpu)
Functional, but requires a $40/mo VPS (8 GB RAM). Benchmarked alternative Q4_K_M models on T490 (i7-10510U, OpenBSD 7.9, CPU only):
+--------------+--------+-------------+---------------------+----------+ | Model | Tokens | Tok/s (μ±σ) | Time s (Med/Range) | RAM (MB) | +--------------+--------+-------------+---------------------+----------+ | Qwen 2.5 3B | 114.0 | 5.22 ± 0.31 | 21.79 (19.39-25.20) | 3,235.22 | | Qwen 2.5 7B | 104.5 | 2.09 ± 0.07 | 49.27 (45.50-52.98) | 7,335.18 | | Phi 3.5 3.8B | 128.2 | 2.06 ± 0.10 | 71.87 (48.67-72.37) | 3,922.55 | | LLaMA 3.1 8B | 115.5 | 2.01 ± 0.08 | 57.10 (53.95-66.09) | 7,873.52 | | Mistral 7B | 106.6 | 1.67 ± 0.13 | 64.25 (57.85-66.86) | 7,535.61 | +--------------+--------+-------------+---------------------+----------+
Larger models caught nuances smaller ones missed, but definitions weren’t consistently better. Gemma 2 9B generated empty responses due to chat template bug; fixed, but 7.8 GB RAM usage didn’t justify re-running benchmarks. Smaller models handled shorter prompts better. Qwen 2.5 struggled with formality level, but produced tighter, faster definitions at 3.2 GB RAM.
Qwen 2.5 3B halves the VPS cost—still wasteful for the occasional prompt. Decoupling lex-llm from lex-cgi lets lex-llm run on a 4 GB Raspberry Pi 5; rest could run on a $5/mo 1 GB VPS.
Refactored lex-llm to communicate over Unix domain sockets. lex-cgi now spools incoming emails to disk. A shell daemon periodically fetches them via SSH, runs lex-llm, and syncs output back to VPS.
while true; do
# fork to avoid resource leakage on unexpected errors
(
raw_prompt=$(ssh -F "${SSH_CONFIG}" -n "${SSH_HOST}" \
"cat '${SPOOL_DIR}/${SPOOL_FILE}'")
response=$(print -r -- "${raw_prompt}" \
| nc -w 300 -U "${SOCK_PATH}" \
| awk '# formatting fix, omitted...'
)
# validate ${target_word}, write to tmp_file,
# atomically mv into place, rm spool file—omitted
print -r -- "${response}" \
| fold -s -w 72 \
| ssh -F "${SSH_CONFIG}" "${SSH_HOST}" "${remote_cmd}"
)
done
Performance: lex-llm was 8x slower (190 s/req) on the RPi5. Benchmarked version loaded the model every request. Loading it once at init reduced execution time to 55 seconds.
Forcing the model to RAM at init via LLAMA_LOAD_MODE_MLOCK would have been desirable, but RPi5 lacks RAM to grant the mlock. Between LLAMA_LOAD_MODE_NONE and LLAMA_LOAD_MODE_MMAP, former yielded better steady-state performance (1.3 GB less MAX RSS, 1-2 seconds faster).
RPi5’s slower Cortex-A76 processor was still 2x behind benchmarks. llama_decode() was evaluating system + user prompt every request. System prompt doesn’t change between requests; cached token offsets and decoded only the part that changed:
/* drop everything after common prefix. */
llama_memory_seq_rm(llama_get_memory(llm->ctx), 0, n_common, -1);
/* decode the diff */
if (n_common < n_new_tokens) {
batch = llama_batch_get_one(new_tokens + n_common,
n_new_tokens - n_common);
if (llama_decode(llm->ctx, batch) != 0)
return;
memcpy(llm->cached_tokens + n_common,
new_tokens + n_common,
(size_t)(n_new_tokens - n_common) * sizeof(llama_token));
llm->n_cached_tokens = n_new_tokens;
}
Prefix-caching reduced execution time to 35 seconds—acceptable for an email-driven async dictionary.
Note on measurements: 3.4 A supply likely caused CPU throttling on the RPi5 during measurements. Final version uses the recommended 5 A supply.
Fidelity: Didn’t notice qualitative differences between T490 and RPi5 output under normal operation. They exist, however. System prompt fragment duplicated the SYNONYMS instruction inside GENERAL TONE:
One terse sentence of describing general tone. One terse sentence of
standard conversational synonyms.
RPi5 echoed the second stray sentence verbatim; T490 did not:
GENERAL TONE:
The phrase suggests the impending end of life with somber and formal
language.
One terse sentence of standard conversational synonyms: Buried or
surrounded by darkness.
Possible artifact of non-associative floating-point arithmetic (x86 SIMD vs ARM NEON instruction ordering). Dropping duplicate sentence from prompt reconciled the outputs.
Storage: lex-llm binary is 12.9 MB. GGUF takes up 2 GB. Standard OpenBSD install recommends 8 GB. SD card I have is 4 GB—3.5 GB usable.
Flashed OpenBSD onto SD card; tossed man, game, and X file sets. Kept comp set to build lex-llm. Disabled boot-time library re-linking and purged /usr/share/relink to reclaim a further 450 MB.
SD card P/E cycles are limited. lex-llm avoids disk writes entirely. Mounted /tmp and /var/log on mfs to reduce flash wear by the OS.
OS + lex-llm + model now fits:
Filesystem Size Used Avail Capacity Mounted on
/dev/sd0a 3.5G 3.0G 325M 91% /
mfs:7428 15.4M 1.1M 13.6M 8% /var/log
mfs:93633 124M 4.0K 118M 1% /tmp
Installation went smoothly, except the RPi5 wouldn’t boot unless a serial adapter was plugged into the JST port. RX pin floated with no internal pull-up enabled. Resolved by adding an external 10 kΩ pull-up (RX to 3.3 V).
Security: PGP keys authenticate requests. lex-cgi (chrooted), lex-llm, lex-mailer employ pledge and unveil:
my $pid = fork();
if ($pid == 0) {
pledge(qw(stdio proc exec));
# sendmail
}
# unveil rules...
unveil($dict_dir, 'r');
unveil(); # unveil lock
# main process drops 'exec'
pledge(qw(stdio rpath wpath cpath proc));
Packet filter blocks network scans, access to other hosts:
set skip on lo
block all
pass in on $net_if inet proto tcp from $home_net to ($net_if) port ssh
pass out on $net_if inet proto tcp from ($net_if) to $vps port ssh
UDP traffic is blocked; reserved RPi5 IP address in the router, disabled DHCP.
End of line.
Source: 4c1da52