Rendered at 05:29:46 GMT+0000 (Coordinated Universal Time) with Cloudflare Workers.
scottcodie 31 minutes ago [-]
I did took a native c++ approach when writing a relational transformers engine (RelativeDB). My journey was pytorch -> c++ -> Triton (lang). While C++ was more performant than Triton, I couldn't afford to optimize on every gpu. I just accepted the ~15% throughput loss for my cloud service, which honestly wasn't bad for the amount of flexibility I got out of it.
But the cpp port of vllm looks great, that'd be great if you'll maintain that. I hit the same limitations with vllm.
dennis16384 1 hours ago [-]
I had a similar success with Model2Vec static embedder and NER inference (both GGUF, compiled for WASM), ported to plain C from ONNX Runtime.
Wasm size from 30Mb to 300kb and 1.5x speedup. It's definitely worth it for performance or distribution size.
stephbook 2 hours ago [-]
Should have started with writing your own blog posts.
nnevatie 1 hours ago [-]
Came here to say the same. Really tiring to read these slop-infested posts, where everything has the “right shape”.
altmanaltman 1 hours ago [-]
I went through the post because of your comment but it really doesn't look like AI slop. Can you please share why you feel like its slop and not written by a human? I can also say "should have started writing your own comments" to you and its unfalsifiable. Blanket accusations with no proof is not a good move really.
nnevatie 8 minutes ago [-]
The post is full of signs. Here's only a couple of examples:
> The method, the measurements, and what it costs us.
> That is the general shape of these wins.
> Parity is the gate, speed is the follow-up
I could go on and on, but you probably get the point. If you don't find anything funny with the above, you might have not been enough-exposed to slop.
But the cpp port of vllm looks great, that'd be great if you'll maintain that. I hit the same limitations with vllm.
Wasm size from 30Mb to 300kb and 1.5x speedup. It's definitely worth it for performance or distribution size.
> The method, the measurements, and what it costs us.
> That is the general shape of these wins.
> Parity is the gate, speed is the follow-up
I could go on and on, but you probably get the point. If you don't find anything funny with the above, you might have not been enough-exposed to slop.