2021 was the yr of monster AI fashions

What does it imply for a mannequin to be massive? The dimensions of a mannequin—a educated neural community—is measured by the variety of parameters it has. These are the values within the community that get tweaked again and again throughout coaching and are then used to make the mannequin’s predictions. Roughly talking, the extra parameters a mannequin has, the extra data it may possibly take in from its coaching knowledge, and the extra correct its predictions about contemporary knowledge will likely be.

GPT-3 has 175 billion parameters—10 occasions greater than its predecessor, GPT-2. However GPT-3 is dwarfed by the category of 2021. Jurassic-1, a commercially obtainable massive language mannequin launched by US startup AI21 Labs in September, edged out GPT-3 with 178 billion parameters. Gopher, a brand new mannequin launched by DeepMind in December, has 280 billion parameters. Megatron-Turing NLG has 530 billion. Google’s Change-Transformer and GLaM fashions have one and 1.2 trillion parameters, respectively. 

The pattern is not only within the US. This yr the Chinese language tech large Huawei constructed a 200-billion-parameter language mannequin known as PanGu. Inspur, one other Chinese language agency, constructed Yuan 1.0, a 245-billion-parameter mannequin. Baidu and Peng Cheng Laboratory, a analysis institute in Shenzhen, introduced PCL-BAIDU Wenxin, a mannequin with 280 billion parameters that Baidu is already utilizing in quite a lot of purposes, together with web search, information feeds, and good audio system. And the Beijing Academy of AI introduced Wu Dao 2.0, which has 1.75 trillion parameters. 

In the meantime, South Korean web search agency Naver introduced a mannequin known as HyperCLOVA, with 204 billion parameters.

Each considered one of these is a notable feat of engineering. For a begin, coaching a mannequin with greater than 100 billion parameters is a fancy plumbing drawback: a whole bunch of particular person GPUs—the {hardware} of selection for coaching deep neural networks—have to be linked and synchronized, and the coaching knowledge break up have to be into chunks and distributed between them in the correct order on the proper time. 

Giant language fashions have grow to be status initiatives that showcase an organization’s technical prowess. But few of those new fashions transfer the analysis ahead past repeating the demonstration that scaling up will get good outcomes. 

There are a handful of improvements. As soon as educated, Google’s Change-Transformer and GLaM use a fraction of their parameters to make predictions, in order that they save computing energy. PCL-Baidu Wenxin combines a GPT-3-style mannequin with a information graph, a way utilized in old-school symbolic AI to retailer information. And alongside Gopher, DeepMind launched RETRO, a language mannequin with solely 7 billion parameters that competes with others 25 occasions its dimension by cross-referencing a database of paperwork when it generates textual content. This makes RETRO more cost effective to coach than its large rivals.