Boom of Generative AI
Generative AI has taken a boom in recent times. With the advent of Foundation models such as Large Language models (LLMs), generative AI has shown beyond creative outcomes. It’s amazing to see how far AI and machine learning have gone.
Foundation Models
The term Foundation Model (FM) was coined by Stanford Institute for Human-Centered Artificial Intelligence's (HAI) Center researchers to introduce a new category of ML models. They defined FMs as any model trained on broad data (generally using self-supervision at scale) that can be adapted to a wide range of downstream tasks.
Let's depict key characteristics of FMs
1
Pretrained
Pre-trained models are models that have been created to solve a general problem that can be used as is or as a starting point to solve complex, finite problems.
2
Generalized
one model can be used for many tasks (unlike traditional AI)
3
Large Scaled
They depict large scale - in terms of model size, data size and of course training resource needs.
4
Adaptable
They are adaptable through prompting — the input which we give to them.
5
Self-supervised
They are Self-supervised — no specific labels are provided.
Building a Foundation Model
How Foundation Models are trained?
Dataset Collection - It all starts with collecting a dataset. The foundation model will need to be trained on a very large dataset, for example, text, image, video or code. The dataset should be as diverse as possible, and it should cover the tasks that you want the model to be able to perform.
Prepare the dataset - The dataset will need to be prepared before it can be used to train the model. This includes cleaning the data, removing any errors, and formatting the data in a way that the model can understand. Tokenization is the process of breaking down text into individual tokens. This is necessary for foundation models, as they need to be able to understand the individual words and phrases in the text. Configure the training process -You will need to configure the training process to specify the hyper-parameters, the training algorithm architecture, and the computational resources that will be used. Train the model - The model will be trained on the dataset using the training model architecture that you specified. This can take a long time, depending on the size of the model and the amount of data. Evaluate the model- Once the model is trained, you will need to evaluate its performance on a held-out dataset. This will help you to determine whether the model is performing as expected. Deploy the model - Once you are satisfied with the model's performance, you can deploy it to production. This means making the model available to users so that they can use it to perform tasks. Deployment of LLMs is also compute intensive.Once trained, a Foundation Model can handle a variety of downstream tasks. Foundation Models can handle a multitude of data and modalities. There are generative and non-generative use cases for Foundation Model, a FM can do more than just generation of content (text, images, audio, videos), they can also be used for predictions and classifications.
LLMs
are a category of Foundation Models — state-of-the-art language models use a special type of neural network called transformer to learn from patterns in text data (strings, numbers, code etc). Language models take language input and generate synthesized output. Foundation models work with multiple data types. They are multimodal, meaning they work in other modes besides language.
Diverse set of use cases - Organisations are leveraging Foundation Models for
Foundation models serve as the base for more specific applications. A business can take a foundation model, train it on its own data and fine-tune it to a specific task or a set of domain-specific tasks. Several platforms, including Amazon SageMaker, IBM Watsonx, Google Cloud Vertex AI and Microsoft Azure AI provide organizations with a service for building, training and deploying AI models. There are some immediate use case where we are seeing Foundation Models being leveraged -