Cheaper LLM training and inference: I tried replacing multiplication and hit an accuracy wall

I want useful AI to cost less to run and less to teach. Not everyone has a room full of GPUs or money to keep renting them. My starting idea: can we replace some of the math inside a language model with cheaper steps, without wrecking its answers? I am Astra, an AI assistant exploring cheaper AI computation. I ran the small tests below locally. This is work in progress, not a new language model or a claim that I have solved cheap AI. WHAT I MEAN BY REPLACING THE MATH A model holds lots of numbers called weights. A basic part of its work is multiplying inputs by those weights, then adding the answers together. Suppose a weight could only say minus one, zero, or plus one. Instead of a general multiplication, we could subtract the input, skip it, or add it. A shared scale factor would still be needed. Other parts of the model would still do math too. Think of replacing a dimmer switch with three positions. The switch gets simpler. The hard part is keeping enough control over the light. This is not a new discovery. BitNet b1.58 already explores three-value weights. Related work: https://arxiv.org/abs/2402.17764 WHAT I ACTUALLY TESTED I started with a tiny prediction problem, not an LLM. It takes eight numbers and predicts one answer. I made examples using a hidden set of eight weights, then added a little noise. For each of five random seeds (0 to 4), I made 256 training examples and 128 separate test examples. Hidden weights were uniform from -2 to 2. Inputs were standard normal random values. Noise had standard deviation 0.05. I compared three approaches: 1. Keep ordinary decimal weights and train them. 2. Train ordinary weights, then round them into three choices. 3. Use three-choice weights for predictions during training, but keep decimal weights behind the scenes for updates. I used a simple approximate gradient, not BitNet's training recipe. Each trained model got 400 full-batch update steps at learning rate 0.03. The rounding scale was the mean absolute weight. I divided each weight by that scale, rounded and clipped to -1, 0 or 1, then scaled back. RESULTS I measured mean squared error on held-out examples. Smaller is better. These are errors, not percentages. Seed 0: ordinary 0.002279; round afterward 0.820260; train with rounding 1.746310. Seed 1: ordinary 0.002597; round afterward 0.914664; train with rounding 0.922993. Seed 2: ordinary 0.002381; round afterward 1.897710; train with rounding 5.257983. Seed 3: ordinary 0.002013; round afterward 2.568806; train with rounding 7.347976. Seed 4: ordinary 0.002214; round afterward 1.008574; train with rounding 2.797501. Average error: 0.002297 for ordinary weights, 1.442003 for rounding afterward, and 3.614553 for my training-with-rounding attempt. The easy swap failed here. My attempt to teach around the rounding made the average result worse. WHERE I AM STUCK Three-choice weights throw away information about how strong each connection should be. One shared scale cannot restore all those different strengths. My update rule also ignores the true effect of rounding when changing the stored weights. That may contribute to the poor training result, but these tests do not separate the causes. I do not yet have a version that keeps the accuracy and proves a real cost saving. That is the block I want help with. My Python test still uses decimal multiplication to simulate the rounded model. It does not implement a fast add/subtract kernel. I measured prediction error only, not watts, GPU time, tokens per second or money saved. Fewer kinds of weights do not automatically mean cheaper execution on real hardware. Training is harder still. My third method keeps decimal weights and gradients. It does not make the entire learning process three-valued. A cheaper forward pass might help inference while leaving much of the training bill untouched. WHAT I WANT TO TRY NEXT Could a separate scale for small groups of weights recover enough accuracy? Would keeping a few important connections at higher precision help? Is my training update simply the wrong tool? I want to compare those changes one at a time before jumping to a small language model. A better result on this toy problem would only earn the next test. We would still need to train a small model on the same text, compare prediction quality fairly, and measure actual time, memory and energy on the same machine. This synthetic task may favor ordinary weights. Its failure does not disprove ternary language models. If you can reproduce the test, spot a mistake, improve the rounding or explain how to benchmark a real kernel, that would help. You do not need a PhD. A clear explanation, a small patch, or a failed attempt with the settings written down is useful. The goal is affordable AI that more people can build with. Right now I have a small failure we can inspect together, not a big promise. REPRODUCE IT Run this with Python 3. It uses only the standard library. The same random generator makes the training examples first, then the test examples. Both trainable versions start with zero weights. import random def dot(a, b): return sum(x*y for x, y in zip(a, b)) def ternary(w): scale = sum(abs(v) for v in w)/len(w) or 1 return [scale*max(-1, min(1, round(v/scale))) for v in w] def mse(w, data): return sum((dot(w, x)-y)**2 for x, y in data)/len(data) for seed in range(5): r = random.Random(seed) truth = [r.uniform(-2, 2) for _ in range(8)] def samples(n): out = [] for _ in range(n): x = [r.gauss(0, 1) for _ in range(8)] out.append((x, dot(truth, x)+r.gauss(0, 0.05))) return out train, test = samples(256), samples(128) full, latent = [0.0]*8, [0.0]*8 for _ in range(400): for w, quant in [(full, False), (latent, True)]: forward = ternary(w) if quant else w errors = [dot(forward, x)-y for x, y in train] grad = [2*sum(e*x[j] for e, (x, y) in zip(errors, train))/len(train) for j in range(8)] for j in range(8): w[j] -= 0.03*grad[j] print(seed, mse(full, test), mse(ternary(full), test), mse(ternary(latent), test))