Research
We believe custom-trained, specialized models that keep learning, stay transparent, and deliver better performance are the future.
Continual learning and the post monolith AI era
The cost of continual learning scales with generality.
PositionIf we can't design neat latent structures, then maybe we can Bitter Lesson it through self-study
They very existence of Claude's Constitution shows that self-study, to a certain extent, just works.
Position
BYO SWE-grep: automatically train blazing fast search sub-agents on your knowledge base (Pt. 1)
RL-trained search subagents that learn your knowledge base’s structure for fast, reliable retrieval
Research
Lumina: building self-improving evaluation through customer-in-the-loop refinement
Lumina: an adaptive evaluation engine that learns to judge like a subject matter expert.
ResearchUpweight the strategy, not the tokens: faster training with explicit reasoning through RGT (Rationale-Guided Training)
Teach the why, not just the what: Rationale-Guided Training
Research
Attention-based attribution: what your model is actually looking at
Cosine similarity is cosplay. Attention is attribution.
ResearchTraining loss predicts evaluation performance, even for non-verifiable tasks
Loss: the cheapest evaluation you’ll ever run.
ResearchRobust, sample efficient SFT with prompt mutations
Low-KL divergence prompt mutations: better performance at a fraction of the cost.
ResearchWrite small, learn forever: rank-1 LoRA for continual learning
Why rank-1 LoRA updates might be the missing link between static fine-tuning and truly continuous, live-on-GPU learning.
ResearchA letter to the C-suite: the shifting role of MLEs
Your MLEs are brilliant, but you’re giving them the wrong job.
PositionAmnesiac generalist behemoths are not the future of language models
You don’t need a generic genius. You need a specialist learner.
PositionThe bitter lesson of LLM evals
Turning expert judgment into a compounding moat. Because in LLM evals, scaling care beats scaling compute.
Position
Do transformers notice their own mistakes? Finding a linear hallucination detector inside LLMs
A linear signal in LLMs reveals hallucinations, is detected by a frozen observer, and steered with a single vector.
ResearchResurrecting the salmon: seeing clearer inside LLMs with domain-specific SAEs
A powerful, efficient, and domain-robust strategy for safeguarding medical-text generation.
ResearchWhy mechanistic interpretability needs a paradigm inversion
The conventional scaling paradigm for language models themselves may be fundamentally misaligned with interp.
Research