EdgeBench: uma plataforma aberta de benchmarking para modelos de linguagem em hardware de borda

Abstract

Currently, the proliferation of Large Language Models (LLMs) on edge devices poses unprecedented evaluation challenges, since isolated performance metrics are insufficient to characterize the behavior of quantized models on heterogeneous hardware. Accordingly, this work presents EdgeBench, an open-source benchmarking platform that unifies performance measurement, hardware telemetry, and quality assessment for LLMs executed across two complementary execution paths: Path A (local agent with Ollama/llama.cpp) and Path B (in-browser inference via WebGPU). In this regard, the architecture comprises five elements: (i) a Python agent (edgebench-agent) exposing an HTTP+SSE API with hardware instrumentation via RAPL, NVML, and INA219, together with a Go TUI featuring a setup wizard and auto-update; (ii) a browser inference layer via @mlc-ai/web-llm with WebGPU, requiring zero software installation; (iii) a quality evaluation layer computing perplexity, an MMLU subset, consistency, and format compliance locally, together with asynchronous LLM-as-Judge evaluation (Claude Sonnet); (iv) a Fastify backend with a BullMQ/Redis queue and PostgreSQL+TimescaleDB persistence; and (v) a Next.js 16 web interface with OAuth authentication, the Mutum design system, and a community leaderboard. Regarding validation, experiments on real hardware, namely a browser with WebGPU (70 tok/s on an NVIDIA GPU), a Jetson Nano (7.29 tok/s), and a Raspberry Pi 4 (3.81 tok/s) running TinyLlama-1.1B on CPU, demonstrate system feasibility and reveal behavioral differences between platforms, as the Jetson Nano outperforms the Raspberry Pi 4 despite its older-generation CPU, owing to the Pi 4's thermal throttling under load, a characterization inaccessible from datasheets. Moreover, the local quality suite (50 cases, 8 dimensions) and the LLM-as-Judge pipeline complement performance analysis with an assessment of the model's cognitive capability. Finally, the source code, experimental data, and documentation are publicly available in the project repository.