Files
ModelHub XC 5102c29296 初始化项目,由ModelHub XC社区提供模型
Model: bartowski/wavecoder-ultra-1.1-6.7b-AWQ
Source: Original Platform
2026-11-11 09:36:13 +08:00

2.4 KiB
Raw Permalink Blame History

license, license_link, language, library_name, datasets, pipeline_tag, tags, metrics, quantized_by
license license_link language library_name datasets pipeline_tag tags metrics quantized_by
mit https://huggingface.co/microsoft/wavecoder-ultra-6.7b/blob/main/LICENSE
en
transformers
humaneval
text-generation
code
code_eval
bartowski

4-bit GEMM AWQ Quantizations of wavecoder-ultra-1.1-6.7b

Using AutoAWQ release v0.2.4 for quantization.

Original model: https://huggingface.co/microsoft/wavecoder-ultra-6.7b

Note: They updated the model in place, no idea if there's actually any changes, but tagging this 1.1 just in case

Prompt format

No chat template specified so default is used. This may be incorrect, check original model card for details.

<|begin▁of▁sentence|>[INST] <<SYS>>
{system_prompt}
<</SYS>>

{prompt} [/INST]  <|end▁of▁sentence|>

AWQ Parameters

  • q_group_size: 128
  • w_bit: 4
  • zero_point: True
  • version: GEMM

How to run

From the AutoAWQ repo here

First install autoawq pypi package:

pip install autoawq

Then run the following:

from awq import AutoAWQForCausalLM
from transformers import AutoTokenizer, TextStreamer


quant_path = "models/wavecoder-ultra-1.1-6.7b-AWQ"

# Load model
model = AutoAWQForCausalLM.from_quantized(quant_path, fuse_layers=True)
tokenizer = AutoTokenizer.from_pretrained(quant_path, trust_remote_code=True)
streamer = TextStreamer(tokenizer, skip_prompt=True, skip_special_tokens=True)

prompt = "You're standing on the surface of the Earth. "\
        "You walk one mile south, one mile west and one mile north. "\
        "You end up exactly where you started. Where are you?"

chat = [
    {"role": "system", "content": "You are a concise assistant that helps answer questions."},
    {"role": "user", "content": prompt},
]

# <|eot_id|> used for llama 3 models
terminators = [
    tokenizer.eos_token_id,
    tokenizer.convert_tokens_to_ids("<|eot_id|>")
]

tokens = tokenizer.apply_chat_template(
    chat,
    return_tensors="pt"
).cuda()

# Generate output
generation_output = model.generate(
    tokens, 
    streamer=streamer,
    max_new_tokens=64,
    eos_token_id=terminators
)

Want to support my work? Visit my ko-fi page here: https://ko-fi.com/bartowski