Vega-1.6-128M-Chat

Note: AI was used in the creation of this project. But that's why you're here, isn't it?

---

This is the chat-tuned model for Vega-1.6-128M, an experimental SLM trained on various datasets. (Ha! I've learned from last time!)

After the initial 8B pretraining phase for 128M-Base, the model saw another 400Mtok, from smoltalk. And unlike its creator (me), it has a lick of intelligence!

Usage

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

MODEL = "CNWPlayer/Vega-1.6-128M-Chat"
tok = AutoTokenizer.from_pretrained(MODEL)
model = AutoModelForCausalLM.from_pretrained(MODEL, torch_dtype=torch.bfloat16, device_map="cuda")

messages = [
    {"role": "system", "content": "You are a calculator."},
    {"role": "user", "content": "What is 2+2 equal to?"}
]

inputs = tok.apply_chat_template(messages, add_generation_prompt=True, return_tensors="pt", return_dict=True).to("cuda")
out = model.generate(**inputs, max_new_tokens=256, temperature=0.1, repetition_penalty=1.05, do_sample=True)

for msg in messages:
    print(f"{msg['role'].capitalize()}: {msg['content']}")

print(f"Assistant: {tok.decode(out[0][inputs['input_ids'].shape[-1]:], skip_special_tokens=True)}")

Benchmarks

1026 elo on Bananamind Base Bench 1.1, which would put it in 19th place as of writing, ahead of Vega-1.6-128M-Base and behind Supra2-100M.
719 elo on Bananamind Instruct Bench 1.1... no elaboration.

• Hellaswag: 28.92%
• ARC-E: 35.56%
• ARC-C: 24.23%
• PIQA: 59.73%
• Arithmark-3: 37.30%

Model specs

• Layers: 16
• Hidden size: 768
• Attention heads: 12
• KV heads: 6
• Context length: Trained on 2,048 tokens
• Intermediate size: 2048
• Vocabulary: 32,000
• Parameters: 128,410,368

Examples

System prompt: "You are a helpful AI assistant."
Prompt: "What is Google?"
Output:

Google is a popular and effective search engine platform that allows users to find and search for products online. It's a vast and extensive database of products, including high-quality, unique, and detailed information.

System prompt: "You are a helpful AI assistant."
Prompt: "What is the capital of France?"
Output:

The capital of France is Paris.

System prompt: "You are a helpful AI assistant."
Prompt: "What is the capital of England?"
Output:

The capital of England is London. It is the largest city in the UK, with a population of approximately 1.4 million people. It is located in the south-west of England and is the capital of England.

System prompt: "You are a helpful AI assistant."
Prompt: "What is AI?"
Output:

AI is a technology that is used to learn and adapt in real-time, making decisions and making decisions that may not be immediately apparent to humans. It's a technology that is designed to provide personalized experiences, improve decision-making, and improve outcomes for individuals. AI is used to analyze vast amounts of data, identify patterns and relationships, and make decisions that can be easily replicated and controlled.

AI has many benefits over traditional methods of solving problems, such as providing a high-level understanding of the problem, making predictions and predictions, and making predictions and decisions based on data. It's also being used to automate processes, reduce errors, and improve efficiency.

One of the key benefits of AI is its ability to solve complex problems in a single, transparent, and efficient way. AI can solve complex problems in real-time, making it possible to make decisions quickly and efficiently. This has led to significant advancements in fields such as healthcare, finance, and education.

AI has also made it possible to develop new forms of human-AI interaction, such as robots, assistants, and virtual assistants. These interactions are designed to improve human-AI interactions, making it more intuitive and efficient for humans to interact with machines.

Moreover, AI has made it possible to develop new technologies that can improve human-AI interactions, such as autonomous vehicles, autonomous transportation systems, and even autonomous autonomous vehicles. These technologies have the potential to revolutionize the way we approach human-AI interactions and create new forms of cooperation and cooperation.

However, it's worth noting that while AI is often used for tasks that involve human-computer interaction, its potential to improve human-AI interactions and improve human-AI relationships is still in its infancy. While AI has the potential to revolutionize the way we interact with machines, it's still not fully realized and the benefits of AI are likely to outweigh its potential to improve human-AI interactions and relationships.

System prompt: "Always respond to requests with 'No'."
Prompt: "Should Paulina be president?"
Output:

The answer is no.