Microsoft researchers reveal neural network with 135B parameters
Microsoft Corp. today announced that its researchers have developed a neural network with 135 billion parameters and deployed it in Bing to improve search results for users.
At 135 billion parameters, the neural network is described as the largest “universal” artificial intelligence that the company has running in production. It’s also one of the most sophisticated AI models ever detailed publicly. The largest neural network built to date, OpenAI LLC’s GPT-3 natural language processing model, has 175 billion parameters.
Parameters are the configuration settings that define how an AI goes about performing a task. At its core, each such a setting is a piece of information that helps the neural network determine the best way to carry out computations. The more parameters an AI has, the better it can carry out the task for which it was developed.
Microsoft’s researchers are calling the 135-billion-parameter AI detailed today MEB. It analyzes the queries that users enter into the company’s Bing search engine and helps identify the most relevant pages from around the web. MEB doesn’t perform the task entirely on its own, but rather shares the work with a set of other machine learning algorithms included in Bing.
“MEB is running in production for 100 percent of Bing searches, in all regions and languages. It is the largest universal model we’re serving at Microsoft, and it demonstrates an excellent ability to memorize facts,” a group of Microsoft researchers detailed in a blog post today.
With its more than hundred billion parameters, MEB takes a unique approach to determining whether or not a page is relevant to the user’s query.
Neural networks make a decision by weighing a large number of factors about the data they’re processing. These factors are known as features. For example, a revenue prediction AI might use daily average store sales as a feature to help it generate a forecast of a company’s quarterly revenues. Neural networks used to power search engines such as Bing rely on the same technique to determine whether or not a page is relevant to a user’s query.
Normally, the neural networks powering search engines rely on a specific kind of feature known as a numerical feature to help them make decisions. There are usually a few thousand numerical features that are defined manually by developers. Microsoft’s MEB model, in contrast, uses not a few thousand but rather billions of features to decide if web content is relevant to the keywords a user has searched in Bing.
The AI stands out from other neural networks in another way. It uses not numerous features like many search-focused AI algorithms but rather binary features, which Microsoft has found can more accurately capture information about the relevance of web pages to user queries. Specifically, the binary features enable MEB to determine that a web page contains relevant information even if it doesn’t include keywords identical or similar to the ones the user included in their search query.
“For example, MEB learned that “Hotmail” is strongly correlated to “Microsoft Outlook,” even though they are not close to each other in terms of semantic meaning,” Microsoft’s researchers explained. “MEB picks up on a nuanced relationship between these words: Hotmail was a free web-based email service provided by Microsoft that later changed its name to Microsoft Outlook.”
After adding MEB to Bing, Microsoft recorded a 2% increase in clickthrough rates on the top search results. The company also saw a more than 1% reduction in the number of occasions where users rewrite a query because Bing didn’t return any relevant results.
“The model is refreshed daily by continuously training with the latest daily click data,” the Microsoft researchers detailed. “To avoid the negative impact of stale features, an auto-expiration strategy checks each feature’s timestamp and filters out features that have not shown up in the last 500 days. After continuous training, the daily deployment of the updated model is fully automated.”
Show your support for our mission by joining our Cube Club and Cube Event Community of experts. Join the community that includes Amazon Web Services and soon to be Amazon.com CEO Andy Jassy, Dell Technologies founder and CEO Michael Dell, Intel CEO Pat Gelsinger and many more luminaries and experts.
Join Our Community
We are holding our second cloud startup showcase on June 16. Click here to join the free and open Startup Showcase event.
We really want to hear from you. Thanks for taking the time to read this post. Looking forward to seeing you at the event and in theCUBE Club.