laya-rust: pure-Rust Laya inference on candle, CPU/Metal/CUDA

PaperinikSep 2014 stars

No Python, no torch, no ONNX export. Loads the ungated safetensors directly and runs any of the three checkpoints on CPU, Metal or CUDA.

Open on GitHub

Preview of laya-rust: pure-Rust Laya inference on candle, CPU/Metal/CUDA

Launch: Laya
Use case: Automation
Format: Repo
Source: github.com

Paperinik

Cloud software architect. Full stack developer minus frontend, Rust supporter. Formerly at Microsoft and Google.

See all

mizorewww

@mizorewww · Sep 19

Validated Core ML ports of all Laya checkpoints. One short multilingual decision in 4.98 ms P50 on M3 Max with ANE FP16, at 2.78x better energy per decision than MLX FP16.

FluidInference

@FluidInference · Sep 21

Swift package that reads a form in a running Mac app through the Accessibility API, asks an on-device model what belongs in each field, and types it in. ~1 ms per decision.

Alex

@Alex_tra_memory · Sep 21

FluidInference ported Laya to CoreML with 99.5% of ops running on the ANE and benchmarked it at 3.7 ms per decision on an M5 Pro.

lkarlslund

@lkarlslund · Sep 20

Ggml-based C++ runtime for all three checkpoints. Native tokenization, decision heads and JSON output; optimized FP16/BF16 CUDA on RTX with FP32 accumulation, plus Vulkan.

Open on GitHub

laya-rust: pure-Rust Laya inference on candle, CPU/Metal/CUDA

More built with Laya