Shipped: GPT2-BG-MEDIUM
A pioneering experimental GPT-2-Medium model for Bulgarian language, created in the summer of 2021; it is a ~ 355M parameter model, trained from scratch on a dataset from selected Bulgarian books and literature, author's own books and other publications and writings by himself, and other selected material. At the time of creation this was one of the earliest and largest 6-7 GPT2-like models in the world, developed for languages different than English. (The most of the experimenters at the time limited their models to GPT2-Small, ~ 3 times fewer parameters; for comparison, Bulgarian Academy of Science - BAS - trained a GPT2-Small model in 2023 - 2 years later - and the rich international AI institute "INSAIT", based in Sofia, funded with hundreds of millions of Euro, published their fine-tuned Mistral-7B - "BgGPT" - in February-March 2024, more than 2.5 years later. In contrast, GPT2-BG-MEDIUM-BG was developed and trained for free from scratch on the free-tier Google Colaboratory, using Tensorflow and a single Tesla T4 with custom sampling methods to allow for interrupted training session and the iterative addition of new material to the dataset. The project included an original experimental method for unlimited length multi-step generation for longer texts, applying hidden injections of tokens during the generation run, for user-directed topic change. The methods are explained in videos on Youtube, in a Github repo and shared on Huggingface - where you can find more details, the weigths, source code and instructions in videos. The training tutorial video is perhaps one of the most popular GPT2 videos with ~ 4.5K views, 71 likes, 37 direct subscribers as of Nov 2026. github.com/Twenkid/GPT2-Bulgarian-Training-Tips-and-Tools






Comments
0 comments