Docs & benchmarks

Overview

This is a small GPT-style Transformer, implemented and trained entirely from hand-written NumPy — no PyTorch, no autograd library, no pretrained weights downloaded from anywhere. Every reply the chat page shows comes from sampling the model's own learned probability distribution over its vocabulary, given the conversation so far — there is no lookup table, no keyword matching, and no scripted responses anywhere in the pipeline.

The live numbers on this page's Benchmarks section below are reported figures from real training/benchmark runs; the chat page's connection panel also shows live figures (parameter count, training step, validation loss) straight from the running model via GET /api/health, so you can cross-check what's actually deployed against what's documented here.

Architecture

A byte-pair-encoding (BPE) tokenizer, learned from the training corpus itself, turns text into token IDs. Special tokens mark conversation structure (<system>, <user>, <assistant>, <eos>), so the model is trained on real multi-turn conversations, not isolated question/answer pairs.

From there it's a standard decoder-only Transformer: learned token + positional embeddings, several stacked blocks of causal multi-head self-attention and a feed-forward network (with residual connections and layer normalization around each), and a final linear layer producing a probability over the whole vocabulary for the next token. Every one of those operations has a hand-written backward pass (see model.py in the source) computing exact gradients — there is no automatic differentiation engine doing this for you.

Two size profiles exist on the original hand-written-NumPy backend: a small one sized to train in minutes on a phone (Termux/Android, ARM64), and a substantially larger one intended for training on GitHub Actions' CPU. A third profile, ci_torch, uses a separate PyTorch implementation (model_torch.py) instead of the hand-written NumPy one — real CUDA/MPS GPU support when a runner actually provides one (never assumed otherwise), fused attention via scaled_dot_product_attention, and three additional, independently-toggleable architecture upgrades over the original design, all used by current open decoder-only models:

• Rotary position embeddings (RoPE) instead of a learned absolute position table — rotates query/key vectors by a position-dependent angle, so attention depends on relative position.
• RMSNorm instead of LayerNorm — normalizes by root-mean-square only (no mean-centering, no learned bias), cheaper and the current standard.
• SwiGLU feed-forward instead of a plain GELU-MLP — a gated variant ((SiLU(xW_gate) · xW_up) W_down) at a comparable parameter budget.

Each checkpoint records exactly which combination it was trained with, so older checkpoints keep loading and behaving identically even as these defaults change. Generation also uses a key/value cache for incremental decoding, so per-token latency stays roughly constant instead of growing with conversation length.

Tools & agent loop

The model can call real tools mid-reply: a calculator, a sandboxed Python execution environment, file read/write/list confined to a workspace directory, zip create/extract, json/csv/markdown reading, dataset inspection/download, simple document retrieval, and small per-session memory notes — see tools.py for the full registry and its security model, and agent.py for the loop that drives it. The model emits <tool_call>{"name": ..., "arguments": {...}}</tool_call>; the server (never the model) actually runs that tool and feeds the real result back as <tool_result>...</tool_result>, and the model continues generating with that result visible. Nothing in this pipeline fabricates a tool result — a failed tool call produces a real error the model can see and react to.

Code execution never runs in-process: it always shells out to an isolated python3 -I -S subprocess with a wall-clock timeout, a memory limit, a stripped environment (no host secrets), and capped output. Filesystem tools are confined to a workspace directory with path-traversal rejection — a request for ../../etc/passwd is rejected outright, not silently redirected. This is process-level isolation, appropriate for a small demo running short scripts — it is not a substitute for a real container/VM sandbox if this is exposed to untrusted internet traffic at real scale.

Dataset

Training data comes from a configurable list of sources, not a single hardcoded file — each source is normalized into a common conversational format, quality-filtered, deduplicated, weighted/mixed together, and split into train/validation sets with cross-set leakage removal (a conversation that ends up in both train and validation is dropped from validation, never train, so validation loss can't be quietly inflated by testing on something the model already trained on).

The bundled dataset is small and largely synthetic, written for this project rather than scraped or downloaded — that's reflected honestly in how the model performs: expect short, sometimes rough replies, not broad world knowledge. The pipeline is built to mix in larger, real datasets once configured; it doesn't fetch or bundle any large dataset on its own.

Benchmarks

Every hot-path operation (matrix multiplication, softmax, layer normalization) was actually implemented two ways — plain NumPy, and a hand-written C loop — and benchmarked at this model's real tensor shapes before deciding what to use where, rather than assuming C is always faster.

NumPy is backed by a real BLAS library (cache blocking, SIMD, sometimes multi-threading) that a hand-written C loop simply can't beat for matrix multiplication — so matmul, attention, the feed-forward layers, and the output layer all stay in NumPy. Layer normalization is different: NumPy's version does the computation as several separate array operations, each allocating a temporary array; a fused C loop does it in one pass with no temporaries — a real, structural win independent of BLAS. That native path is entirely optional: it auto-builds on first use if a C compiler is available, and transparently falls back to pure NumPy — correctness and results are identical either way, verified with a gradient check.

The larger, GitHub-Actions-scale profile was actually built and run (not just estimated) to get real numbers: on a constrained single-core development sandbox, a full training step (forward + backward + optimizer update) measured on the order of several seconds — reported honestly rather than smoothed over, and the reason that profile's default training run is modest per invocation and designed to resume across multiple runs instead of trying to fit everything into one.

Honest limitations

• Millions, not billions, of parameters — nowhere near a large pretrained model's scale.
• Trained on a small, largely synthetic dataset unless configured otherwise — limited real-world knowledge.
• Fixed context window — long conversations have their oldest turns dropped to keep fitting.
• Tool use needs room in the context window for the full tool-spec system prompt on top of the conversation — on a small model/profile (e.g. the 128-token Termux default), that spec alone can leave little or no room for the actual message. When that happens, the server falls back to plain (tool-free) chat for that turn rather than silently truncating the question — see agent.py's context-budget safeguard. Larger profiles (ci/ci_torch, 384–768+ token context) have real headroom for this.
• Reliable, well-formatted tool calls are a real capability, not a guarantee — a small model trained on few examples will sometimes emit malformed JSON or skip a tool it should have used. The mechanism (parsing, execution, error handling, result reinjection) is tested independently of any specific model's fluency at using it.
• CPU-only on the original NumPy backend; the PyTorch (ci_torch) backend supports CUDA/MPS but only actually uses a GPU when one is genuinely detected on the machine running it (see model_torch.pick_device) — never assumed.