The first question you might ask: why not use your Mac Studio? (M3 Ultra, 96GB RAM)
The answer is that the prefill (prompt-processing) speeds of MLX on a Mac isn’t all that fast while token generation is very fast because of unified memory bandwidth.
Mac Studio M3 Ultra memory bandwidth is >800GB/s and the DGX Spark is 273GB/s. But the PP side of the DGX Spark is upwards of 3.8-4x the speed of the M3 ultra.
During my initial testing, the LoRA training is far faster on the DGX Spark. No surprise!
When I was testing LoRA on my Mac Studio, the 31B model easily fits in memory though during training, my computer would hit over 50GB of swap and then kernel panic. That did suck as this is my primary machine.
The next question: Why do retraining?
I wanted to see if I could get a model to stop sounding like a customer support bot. The goal was directness. No fluff. Just information delivered in a way that actually sounds like me with a technical opinion. And maybe some snark.
Gemma 4 is considered a great model for writing. It is also capable of text-to-speech and imaging but I disabled that in my exported model.
I used the 31B model because the smaller versions just aren’t smart enough to do the nuance well. If you are burning the compute, you might as well do it on something that has the capacity to actually understand the logic you are trying to bake in.
For the technical side, I used Hugging Face PEFT and Transformers.
The gist of LoRA (Low-Rank Adaptation) is that it does not train the whole model. Instead it inserts small, low-rank matrices into the layers. You update these small weights while the main model stays frozen. It reduces VRAM requirements massively because you aren’t dealing with the full gradient of a massive parameter set.
If you want the details on the mechanics, this Medium article does a decent job: Efficient LLM Fine-Tuning with LoRA
The whole idea is decomposing a large weight matrix into two smaller ones. That’s the trick. The original weights of the model stay exactly as they are, while the LoRA adapter learns the specific patterns required for the new task—in this case, learning how to sound like me.
Data was my own shit I trawled: personal sent emails (only my words), announcements of old, and previous blog posts.
I did a lot of data curation to make sure the cadence was right. I wanted short, punchy statements followed by technical explanations. That is the pattern.
The 31B model has the capacity to do this but the base version is far too polite. That is what the adapter fixes.
It’s not perfect.
It still tries to be helpful sometimes. I have a low tolerance for that because that is just another word for wordy. The adapter also has a tendency to overfit if you run too many epochs. If you do that, you end up with a parrot. I don’t want a parrot.
But it is usable.
It is better than the base model. It stopped saying “As an AI language model” and just got to the point.
The training run took a while but the DGX Spark handled it fine.
Next step is figuring out exactly how small the dataset can be. I do not think I need a huge corpus. I just need a clean, concentrated dose of the right data.
The more I play with this, the more I think that the data quality matters way more than the quantity. I’d rather have 500 perfect examples than 50,000 mediocre ones.
That’s all.