Georgi Gerganov f4ab2a4147 llama : fix BPE pre-tokenization (#6920) ...

* merged the changes from deepseeker models to main branch

* Moved regex patterns to unicode.cpp and updated unicode.h

* Moved header files

* Resolved issues

* added and refactored unicode_regex_split and related functions

* Updated/merged the deepseek coder pr

* Refactored code

* Adding unicode regex mappings

* Adding unicode regex function

* Added needed functionality, testing remains

* Fixed issues

* Fixed issue with gpt2 regex custom preprocessor

* unicode : fix? unicode_wstring_to_utf8

* lint : fix whitespaces

* tests : add tokenizer tests for numbers

* unicode : remove redundant headers

* tests : remove and rename tokenizer test scripts

* tests : add sample usage

* gguf-py : reader prints warnings on duplicate keys

* llama : towards llama3 tokenization support (wip)

* unicode : shot in the dark to fix tests on Windows

* unicode : first try custom implementations

* convert : add "tokenizer.ggml.pre" GGUF KV (wip)

* llama : use new pre-tokenizer type

* convert : fix pre-tokenizer type writing

* lint : fix

* make : add test-tokenizer-0-llama-v3

* wip

* models : add llama v3 vocab file

* llama : adapt punctuation regex + add llama 3 regex

* minor

* unicode : set bomb

* unicode : set bomb

* unicode : always use std::wregex

* unicode : support \p{N}, \p{L} and \p{P} natively

* unicode : try fix windows

* unicode : category support via std::regex

* unicode : clean-up

* unicode : simplify

* convert : add convert-hf-to-gguf-update.py

ggml-ci

* lint : update

* convert : add falcon

ggml-ci

* unicode : normalize signatures

* lint : fix

* lint : fix

* convert : remove unused functions

* convert : add comments

* convert : exercise contractions

ggml-ci

* lint : fix

* cmake : refactor test targets

* tests : refactor vocab tests

ggml-ci

* tests : add more vocabs and tests

ggml-ci

* unicode : cleanup

* scripts : ignore new update script in check-requirements.sh

* models : add phi-3, mpt, gpt-2, starcoder

* tests : disable obsolete

ggml-ci

* tests : use faster bpe test

ggml-ci

* llama : more prominent warning for old BPE models

* tests : disable test-tokenizer-1-bpe due to slowness

ggml-ci

---------

Co-authored-by: Jaggzh <jaggz.h@gmail.com>
Co-authored-by: Kazim Abrar Mahi <kazimabrarmahi135@gmail.com>

2024-04-29 16:58:41 +03:00

..

.gitignore

tests : gitignore ggml-common.h

2024-03-09 14:17:11 +02:00

CMakeLists.txt

llama : fix BPE pre-tokenization (#6920)

2024-04-29 16:58:41 +03:00

get-model.cpp

ci : add model tests + script wrapper (#4586)

2024-01-26 14:18:00 +02:00

get-model.h

ci : add model tests + script wrapper (#4586)

2024-01-26 14:18:00 +02:00

run-json-schema-to-grammar.mjs

json-schema-to-grammar improvements (+ added to server) (#5978)

2024-03-21 11:50:43 +00:00

test-autorelease.cpp

ggml : add numa options (#5377)

2024-02-16 11:31:07 +02:00

test-backend-ops.cpp

ggml : group all experts in a single ggml_mul_mat_id (#6505)

2024-04-18 15:18:48 +02:00

test-c.c

Nomic Vulkan backend (#4456)

2024-01-29 15:50:50 -05:00

test-chat-template.cpp

llama : add phi 3 chat template (#6857)

2024-04-24 11:52:37 +03:00

test-double-float.cpp

ggml : move FP16 <-> FP32 code to ggml-impl.h (#3861)

2023-10-30 19:19:15 +02:00

test-grad0.cpp

cuda : improve cuda pool efficiency using virtual memory (#4606)

2023-12-24 14:34:22 +01:00

test-grammar-integration.cpp

grammars: 1.5x faster inference w/ complex grammars (vector reserves / reuses) (#6609)

2024-04-11 19:47:34 +01:00

test-grammar-parser.cpp

ggml, common, examples, tests : fixed type arguments in printf (#5528)

2024-02-18 18:20:12 +02:00

test-json-schema-to-grammar.cpp

JSON schema conversion: ⚡️ faster repetitions, min/maxLength for strings, cap number length (#6555)

2024-04-12 19:43:38 +01:00

test-llama-grammar.cpp

ggml, common, examples, tests : fixed type arguments in printf (#5528)

2024-02-18 18:20:12 +02:00

test-model-load-cancel.cpp

ggml : add numa options (#5377)

2024-02-16 11:31:07 +02:00

test-opt.cpp

code : normalize enum names (#5697)

2024-02-25 12:09:09 +02:00

test-quantize-fns.cpp

tests : include IQ2_XXS and IQ2_XS in test-quantize-fns (#6303)

2024-03-25 19:33:15 +02:00

test-quantize-perf.cpp

ggml : add mmla kernels for quantized GEMM (#4966)

2024-02-11 15:22:33 +02:00

test-rope.cpp

llama : custom attention mask + parallel decoding + no context swaps (#3228)

2023-09-28 19:04:36 +03:00

test-sampling.cpp

sampling: fix top_k <= 0 (#5388)

2024-02-08 09:46:30 +01:00

test-tokenizer-0-bpe.py

llama : fix BPE pre-tokenization (#6920)

2024-04-29 16:58:41 +03:00

test-tokenizer-0-spm.py

llama : fix BPE pre-tokenization (#6920)

2024-04-29 16:58:41 +03:00

test-tokenizer-0.cpp

llama : fix BPE pre-tokenization (#6920)

2024-04-29 16:58:41 +03:00

test-tokenizer-1-bpe.cpp

llama : refactor unicode stuff (#5992)

2024-03-11 17:47:47 +02:00

test-tokenizer-1-spm.cpp

llama : fix BPE pre-tokenization (#6920)

2024-04-29 16:58:41 +03:00