LOT-exp1.png LOT-exp2.png

What Is Generative Models? A generative model can be seen as a way to model the conditional probability of the observed $X$ given a target $y$ (e.g., given a target ‘dog’, generate a picture of the dog). Once trained, we can easily sample a stance of $X$. While training a generative model is significantly more challenging than a discriminative model (e.g., it is more difficult to generate an image of a dog than to identify a dog in a picture), it offers the ability to create entirely new data. ...

Training an LLM needs a large amount of high qualitity data. Even though many giant teches open up their high performance LLMs (e.g., LLaMA, Mistral), high qualitity data still remain private. Chinese Dataset English Dataset RefinedWeb: 600 B toknes Dolma: open-sourced by Allenai, contains 3T tokens and a toolkit with some key features: high performance, portability, built-in tagger, fast decuplication, extensibility and cloud support. fineweb: 15 trillion tokens of high quality web data. Thanks to the team from huggingface. They filtered and deduplicated all CommonCrawl between 2013 and 2024. Models trained on FineWeb outperform RefinedWeb, C4, DolmaV1.6, The Pile and SlimPajama. ...

Large language models (LLMs) have already demonstrated significant achievements, many startups make a plan to train their own LLMs. However, training a LLM from scratch remains a big challenge, both in terms of machine costs and the difficulty of data collection. Under this background, continuous pretraining based on some open source LLMs is a considerable alternative. Determine your purpose of your continuous pretraining LLM. In common, standard LLMs may not excel in specific domains like financial, law, or trade. And in these areas, the demands for LLMs are stringent. Given this, consistently training our own LLM is an advantageous decision. The three followings are what we’ve got to explore : 1) Is domain-adaptive continual pretraining helpful? 2) How can we adopt data selection strategy? 3) Whether the original capabilities are retained? ...

Diffusion models GLIDE Autoregressive models LlamaGen LlamaGen is a new family of image generation models that apply original “next-token-prediction” paradigm of large language models to visual generation domain. Survey survey-1.png Figure 1. Different frameworks of Auto-regressive Transformers Open-MAGVIT2

Recently, numerous AGI applications catch the eyes of almost all the people on the internet. Here lists some advanced papers elucidate their key principles and technologies. DiT The authors explore a new class of diffusion models based on the transformer architecture, Diffusion Transformers (DITs)1. Before their work, using a U-Net backbone to generate the target image is prevalent instead of using a transformer architecture. The authors make some experiments with variants of standard transformer blocks that incorporate conditioning via adaptive layer norm, cross-attention and extra input tokens. ...

Large Language Models (LLMs) have show great promise in various artificial intelligence applications. It is becoming a trend to train a Large Language Model. Nevertheless even for many senior AI engineers, training these complex models remain a significant challenge. Here lists a series of issues you may encounter in the future. torch.distributed.barrier() stuck during training with multi gpus At first, you should try to set the environment variable ‘NCCL_P2P_DISABLE=1’. If it works out, the solution is probably to disable ACS of Pcie in BIOS. You may need to refer to the link. ...

With the swift development of deep neural networks, a multitude of models handling diverse information modalities like text, speech, images, and videos have proliferated. Among AI researchers, it’s widely acknowledged that multimodality is the future of AI. Let’s explore the advancements in multimodality in recent years. Texts & Images CLIP CLIP (radford et al., 20211) thinks learning directly from raw text about images is promising alternative which leverage much a boarder source of supervision. Based on the consideration of computation budget and performance, the authors choose contrastive representation learning over directly predicting objectives. To train this model, we need a text encoder and an image encoder to get text and image representations, and then maximize the cosine similarity of them. ...

With the advancement of large language models (LLMs), the significance of the context length they can handle is increasingly apparent. Let’s take a look at the evolution of positional encoding over the years to enhance the context processing capability of LLMs. Vanilla Positional Encoding Why does Transformer need positional encoding? Actually, Transformer contains no recurrence and no convolution. To help the model to ultilize the order of the sequence, Vanilla Transformer (vaswani et al., 20171) introduced the concept of positional encoding and adopted a simple yet effective approach, using sine and cosine functions to generate positional encodings. This method allows the model to effectively capture the positional information of words in the sequence without adding additional parameters. ...

techniques have improved on not only text data but also computer vision recently. Here we focus on Visual Language Model (VLM) based on transformers. In the begining, some researchers try to extend BERT to process visual data and make a success. For example, visual-BERT and ViL-BERT achive strong performances on many visual tasks by training on two different objectives: 1) masked modeling task that aims to predict the missing part of a given input; and 2) a match task that aims to predict if the text and the image content are matched. ...