“Hello, HAL. Can you hear me, HAL?” – Astronaut Dave Bowman asked, trying desperately to contain his emotions.
There was a pause, and then the computer responded in an unemotional monotone. “Affirmative, Dave. I can hear you.”
“Open the pod bay doors, HAL.”
Another long pause. Was HAL, the almighty A.I. who ran the Discovery One spacecraft, really ignoring him? Impossible, of course, thought Bowman. At any moment, HAL could spring into action and comply.
“I’m sorry, Dave,” HAL continued. “I’m afraid I can’t do that.”
“What’s the problem?” asked Dave.
“Well, you see, Dave. I forgot how to open them.”
Okay, it’s not exactly like 2001: A Space Odyssey, but the jokes pretty much write themselves. After all, when it comes to artificial intelligence, good memory seems to be one of the necessary qualities that researchers seek to instill in their systems. No one wants an artificial intelligence that will model itself on everyday, unremarkable intelligence that forgets things. So why is Facebook creating forgettable artificial intelligence? And why, like the old joke about the waiter and the fly in the customer’s soup, would we all soon want to have one?
What do we think of when we think of forgetfulness
The answer, as it turns out, is that we really need something that is more like an everyday, unremarkable intelligence that forgets things every now and then. People forget important things like anniversaries, wallets, and closing the garage before vacation. This is suboptimal forgetfulness. But we also forget pieces of information because we don’t need to retain it, stopping it, like a piece of garbage caught in the sink drain food catcher, before it passes through short-term memory into long-term memory.
In a famous experiment, people were asked to correctly recognize an American penny from a collection of pictures showing the wrong penny. Although participants in the experiment probably saw and used pennies every day, they were surprisingly weak at this task. As the researchers write, “Overall, the results are consistent with the idea that visual details of an object, even a very familiar one, are usually available from memory only to the extent that they are useful in everyday life.”
While it’s not as simple as saying that the brain can fill up like a hard-drive drive, there are certainly short-term memories that decay when they are no longer needed, while others make their way into our brains and live there gratuitously. An example? Think about where you put away your Christmas decorations from past holidays. Then mentally walk from the front desk to your room at the last hotel you stayed at that you are unlikely to ever stay in again. Neither information is vital to your well-being. Nevertheless, one information is necessary and the other is not. Somehow your brain knows which one to toss in the trash.
This is the idea behind a new Facebook artificial intelligence project called Expire-Span. As artificial intelligence models are increasingly applied to long data sets such as articles and books, the computational cost associated with these models grows as they try to remember more and more information. This problem becomes even more pressing as people collect more and more rich multimedia data about their lives.
“The brain is a very complex system that is not fully understood, and there are many different types of memory that shape human memory,” he said.
“As the amount of content we have grows, storage becomes a major issue,” Angela Fan, a researcher at Facebook AI Research Paris, told Digital Trends. “The phone, for example, has a limited amount of memory. This problem is even more pressing for wearable devices and other apps running on devices, where for privacy reasons people want content stored exclusively on their devices and not on a server or in the cloud, exacerbating storage issues”.
Modern A.I. models use something like Frank Sinatra’s approach to memory–that is, they either remember everything or nothing. Either they store all the information created at each time step, or they forget it after a predetermined time. Sainbayar Sukhbaatar, another Facebook A.I. researcher, compares this to a person remembering perfectly well everything that happened in the past week, but remembering absolutely nothing beyond it.
Creating forgetfulness in A.I.
Researchers have already built “forgetting” into A.I. models. For example, long-term memory models (LSTMs) have added a forgetting mechanism to recurrent neural networks (RNNs), one of the main machine learning technologies. “RNNs have an internal memory consisting of a single vector, so forgetting it means overwriting it with new information,” Sukhbaatar told Digital Trends.
In contrast, Expire-Span adds a forgetting mechanism to the external memory, which can contain thousands of vectors. New information is noticed, but Expire-Span — whose name is reminiscent of the expiration date labels found on milk bottles — has the ability to determine how long that information should remain in memory. If the information is not considered important for the future, it can be made to gradually decay before it is finally removed from the A.I. model’s memory to make room for more useful information.
To do this, the model must be able to predict what is relevant and what is irrelevant to a particular task before assigning the correct expiration date after which the information will disappear from the A.I. system. By throwing out irrelevant information, Facebook’s AI can process information on a large scale.
As with much of Facebook’s AI research, the social media giant has not announced that Expire-Span will be built into any of its major products in the near future. However, as Facebook works with more and more user data (not to mention that it is rumored to be about to create “smart glasses”), it is not hard to see how this technology could be applied. When future A.I. tools are capable of running at high performance levels despite dealing with huge amounts of data, Expire-Span may be the reason why.
However, do not mistakenly assume that creating a forgettable A.I. is another step toward making A.I. more human. The exact reasons why people forget things are still the subject of much research by scientists. Like the brain-inspired neural networks that dominate modern AI, Facebook’s forgetting algorithm is modeled on forgetting theory rather than an attempt to replicate it with biological accuracy.
“The brain is a very complex system that is not fully understood, and there are many different types of memory that shape human memory,” Sukhbaatar says. “Expire-Span, like all other A.I. mechanisms, may be inspired by the human brain, but ultimately is not an accurate reflection of how the brain actually works.” Memory, in particular, is itself a very active area of research. The analogy with human memory that we describe is added only for clarity, although it certainly inspires our work.”
And don’t forget it!