Objective Function: Cross Entropy with mean reduction (see API documentation).

Sequence length of 2048 tokens used (see BLOOM tokenizer, tokenizer description) Layer normalization applied to word embeddings layer ( StableEmbedding see code, paper)ĪLiBI positional encodings (see paper), with GeLU activation functions Modified from Megatron-LM GPT2 (see paper, BLOOM Megatron code): Please see the BLOOM training README for full details on replicating training. It is useful for people interested in model development. This section includes details about the model objective and architecture, and the compute infrastructure. (Further breakdown of organizations forthcoming.) Send Questions to: as: BigScience, BigScience Language Open-science Open-access Multilingual (BLOOM) Language Model. Release Date Estimate: Monday, 11.July.2022 License: RAIL License v1.0 ( link / article and FAQ) Model Type: Transformer-based Language ModelĬheckpoints format: transformers (Megatron-DeepSpeed format available here) (Further breakdown of participants forthcoming.) Click to expandĪll collaborators are either volunteers or have an agreement with their employer. It is useful for anyone who wants to reference the model.

This section provides information about the model type, version, license, funders, release date, developers, and contact information. BLOOM can also be instructed to perform text tasks it hasn't been explicitly trained for, by casting them as text generation tasks. As such, it is able to output coherent text in 46 languages and 13 programming languages that is hardly distinguishable from text written by humans. BigScience Large Open-science Open-access Multilingual Language ModelĬurrent Checkpoint: Training Iteration 95000īLOOM is an autoregressive Large Language Model (LLM), trained to continue text from a prompt on vast amounts of text data using industrial-scale computational resources.

Entropy zero 2 download