I’ve been using NixOS/nix containers to host some llama.cpp servers and Hermes Agent. Though I wrote a flake to get Project Maya working so I can serve GLM-5.3-Flash and in legit blown away by how well it works. It’s a fork of strata/llama.cpp that focuses on where the relevant experts are loaded and has a neat dashboard to track their use. https://github.com/mw00/project-maya

My server is assembled into an openair mining rig with a Tesla P40, RTX 3080, and a few Tesla V100s. All of which I bought used off eBay.

I purchased my first domain to self host a matrix server that I can access via my phone and I also have tailscale set up so I can ssh into the server via my laptop while traveling

How do you do, fellow human? 🤖🥸

I have it on my 3:2 laptop that I occasionally plug into my 21:9 monitor and it generally just works; though the trackpad navigation is what really makes it for me.

Oh this is news to me lmao. Brb, looks like I need to make some updates to my server 😂

Thanks for the heads up!

That’s awesome to hear, I’m glad it’s working well for you!

No problem! The company who provides the models also forked llama.cpp while they wait for their PR to be approved if you’re curious about it.

https://github.com/ifm-ai/llama.cpp/tree/model/K2Horizon

You might be interested in these:

https://ifm.ai/blog/k2/

Your sacrifice will not be forgotten.

One sec, I’ll go ask my neighbor whose car has a decal of Carlson’s pouty little face on it what he thinks about this.

::: Just kidding, I don’t speak with them. :::

I’m using a surface laptop studio with the linux-surface kernel (NixOS, btw :p).

It’s actually quite repairable compared to some other offerings and are generally available used on ebay. I won a $610 bid for mine with an i7, 32gb RAM and a 3050ti. I think all the models have the same 120hz 3:2 screen too. I’m incredibly pleased with it.

Helllooooo my future doom emacs PDA. Thanks for posting this!