# Ares Seed 6M Model

A larger Ares seed checkpoint has been trained from scratch to increase Ares' parameters beyond the 1M proof model.

Model repo:

```text
https://huggingface.co/jacmor64/ares-seed-6m
```

Previous 1M model repo:

```text
https://huggingface.co/jacmor64/ares-seed-1m
```

## What this model is

- About 6.23M parameters.
- Decoder-only Transformer using the Ares code in this repo.
- 128-token context window.
- 2048-token BPE tokenizer.
- Trained from scratch for 2,500 CPU steps.
- Uses the same seed data mixture: Simple Wikipedia, ML-process curriculum, and roleplay curriculum.

## Result

Best validation loss:

```text
3.918 at step 1749
```

For comparison, the 1M model reached about:

```text
4.422 at step 4999
```

So the increased-parameter 6M model achieved a better validation loss with fewer steps.

## Caveat

It is still a tiny seed model. It overfits if trained too long, and its generations remain rough/repetitive. This is a proof-of-scaling checkpoint. The serious next step is the 30M Colab run in:

```text
colab/Ares_Wikipedia_Roleplay_Brain_Training.ipynb
```