It's based on a novel he wrote, and will be released later this year
By Ben Popper
Inventor Ray Kurzweil made his name as a pioneer in technology that
helped machines understand human language, both written and spoken.
These days he is probably best known as a prophet of The Singularity,
one of the leading voices predicting that artificial intelligence will
soon surpass its human creators — resulting in either our enslavement or
immortality, depending on how things shake out.
Back in 2012 he was
hired at Google as a director of engineering to work on natural language recognition,
and today we got another hint of what he is working on. In a video from
a recent Singularity conference Kurzweil says he and his team at Google
are building a chatbot, and that it will be released sometime later
this year.
Kurzweil was answering questions from the audience, via telepresence
robot naturally. He was asked when he thought people would be able to
have meaningful conversations with artificial intelligence, one that
might fool you into thinking you were conversing with a human being.
"That's very relevant to what I'm doing at Google," Kurzweil said. "My
team, among other things, is working on chatbots. We expect to release
some chatbots you can talk to later this year.
One of the bots will be named Danielle, and according to Kurzweil, it
will draw on dialog from a character named Danielle, who appears in a
novel he wrote — a book titled, what else, Danielle. Kurzweil is a best
selling author, but so far has only published non-fiction. He said that
anyone will be able to create their own unique chatbot by feeding it a
large sample of your writing, for example by letting it ingest your
blog. This would allow the bot to adopt your "style, personality, and
ideas."
As if talking to other humans was actually "meaningful
Kurzweil says that, while he believes his chatbots will allow for
interesting conversation, we'll have to wait until 2029 for AI to have
human-level language ability, "and everything that comes with it." At
that point, AI will be able to pass the Turing test, meaning they will
be indistinguishable from a real person in a blind test.
Of course, an AI that is simply up to the level of a modern human
isn't really that interesting to Kurzweil, who hopes to one day upload
his consciousness to the internet and dialog with super-intelligent
forms of AI. "If you think you can have a meaningful conversation with a
human, you'll be able to have a meaningful conversation with an AI in
2029. But you'll be able to have interesting conversations before that."
E ink, the company behind the
pigment-based, low-energy monochromatic displays found in many of
today’s popular readers finally figured out how to create up to 32,000
colors in what is almost the exact same technology. The company will unveil what it’s calling a breakthrough technology
on Tuesday at the annual SID Display Week conference in San Francisco.
“For the first time, we can create all colors at every pixel
location,” said E Ink Holding’s Head of Global marketing Giovanni
Mancini. “We have encapsulated four different things in one micro-cup.”
Those four things are actually four different pigments: yellow, cyan,
magenta and white. In traditional monochromatic E Ink, there were just
two colors: black and white. Both microcups work in similar ways, E Ink
changes the polarity to move the pigments around. For monochrome, the
white and black pigments basically switch places (you see white or black
on the reflective screen). However, for the new full-color
electrophoretic display, E Ink had to figure out a more sophisticated
way to manage the pigments in each tiny cup.
“The ability to control those pigments is significant,” said Mancini
In addition to being different colors, each pigment will have
additional properties that gives E Ink greater control over their
movement and position. This allows E Ink to move some or all together to
create combinations that result in up to 32,000 display colors. Since
each cup basically represents a controllable pixel, the results can be
pretty stunning.
Color that lasts
The new display, which E Ink will publicly demonstrate for the first
time, is a 20-inch, 2500 x 1600 resolution display that actually shares
monochrome E Ink’s impressive power capabilities. Mancini told Mashable
that it’s equally power-efficient. He explained that it could be used
in bus stop signage. “Bus stops are powered with solar cells, you could
power this with solar cells,” he said.
A color E Ink display rendering a full-color image.
Image: E Ink
E
Ink is not alone in the low-power color display market. Qualcomm’s
full-color Mirasol display technology has been around for more than six
years. Instead of pigments, capsules and polarity, it uses a fully
mechanical system to create a wide color gamut and uses almost zero
power to maintain the image once it’s set. Last year, Apple reportedly bought Qualcomm’s Mirasol plant. No word yet on if Apple plans to productize the low-power color display technology.
Not for everyone
While color E Ink is on the fast-track to commercialization, it does
have some significant limitations. For now, the resolution is 150 pixels
per inch (ppi), which is roughly half the resolution you find on a
typical, 6-inch, monochromatic E Ink display.
In addition, the full-color E Ink can’t come anywhere close to the
virtual instant refresh capabilities of today’s ereaders. Right now, it
takes a color E Ink display two seconds to fully resolve.
Here is how color E Ink recreates its 32,000 colors.
Image: E Ink
Mancini
doesn’t consider this an issue, though, because the company is
targeting commercial signage, which wouldn’t need to change that often
and is also designed to grab attention, an area where color E Ink may
excel. Mancini said color E Ink will feature highly saturated colors.
“Something close to what you would see on printed poster, paper type of
product.”
He also noted that even though the prototype will be less than 2-feet
wide, size is actually only limited by E Ink’s manufacturing
capabilities.
General availability for color E Ink displays is expected in two
years. Will color E Ink make it to future Amazon Kindles? Mancini would
only reiterate that E Ink is currently focused on the display market.
Using
Google while conducting searches on your mobile will now make for a
better visual experience as the company has launched a new format for
Search: rich cards.
The new rich cards search format
from Google will basically showcase the visual results in a card-like
format, like a carousel. The rich cards allow users to scroll through
the cards.
Introduced on May 17, rich cards are aimed at enhancing the Search
experience on the mobile platform. Moreover, the new format is looking
to further develop the existing rich snippets format from the company.
For the unfamiliar, rich snippets basically show off the image, as
well a summary of the information below the regular results. Both the
rich snippets and the new rich cards deploy Schema.org to enhance the
results. The rich cards are designed in a manner to make them more
suitable for mobile device search.
"Rich cards are a new Search result format building on the success of
rich snippets. Just like rich snippets, rich cards use schema.org
structured markup to display content in an even more engaging and visual
format, with a focus on providing a better mobile user experience," notes Googles.
- See more at:
http://www.techtimes.com/articles/160105/20160521/google-improves-mobile-search-with-rich-cards-offers-new-visual-experience.htm#sthash.SLaSqIB8.dpuf
e rich cards allow users to scroll through the cards.
Introduced on May 17, rich cards are aimed at enhancing the Search
experience on the mobile platform. Moreover, the new format is looking
to further develop the existing rich snippets format from the company.
For the unfamiliar, rich snippets basically show off the image, as
well a summary of the information below the regular results. Both the
rich snippets and the new rich cards deploy Schema.org to enhance the
results. The rich cards are designed in a manner to make them more
suitable for mobile device search.
"Rich cards are a new Search result format building on the success of
rich snippets. Just like rich snippets, rich cards use schema.org
structured markup to display content in an even more engaging and visual
format, with a focus on providing a better mobile user experience," notes Googles.
The search results are shown in the movable carousel format
horizontally. Users can scroll the results left or right, and even
scroll up to see more carousels. Each carousel can have cards for a
particular site or several sites.
Initially, Google is pushing out rich cards for two categories of content, namely movies and recipes.
The search results will appear only in English for now for mobile
searches. Google has revealed that it is "actively experimenting" with
more methods of extending the feature publishers' way.
The company asserts that it has developed a wide-ranging tool set, as
well as updated its developer documentation, to help both developers
and site owners transition from preliminary exploration to application
to performance monitoring.
Google is optimistic that the rich cards format will enable site
owners to get noticed in the crowd, in the Search results space. This
will enable them to lure more target users. Search users will also
benefit as the rich cards offer a more pictorial format, which will aid
them in finding the requisite information quicker.
Whether rich cards are successful in meeting Google's goal of
simplifying search on mobile devices - and making googling more visual -
remains to be seen.
- See more at:
http://www.techtimes.com/articles/160105/20160521/google-improves-mobile-search-with-rich-cards-offers-new-visual-experience.htm#sthash.SLaSqIB8.dpuf
e rich cards allow users to scroll through the cards.
Introduced on May 17, rich cards are aimed at enhancing the Search
experience on the mobile platform. Moreover, the new format is looking
to further develop the existing rich snippets format from the company.
For the unfamiliar, rich snippets basically show off the image, as
well a summary of the information below the regular results. Both the
rich snippets and the new rich cards deploy Schema.org to enhance the
results. The rich cards are designed in a manner to make them more
suitable for mobile device search.
"Rich cards are a new Search result format building on the success of
rich snippets. Just like rich snippets, rich cards use schema.org
structured markup to display content in an even more engaging and visual
format, with a focus on providing a better mobile user experience," notes Googles.
The search results are shown in the movable carousel format
horizontally. Users can scroll the results left or right, and even
scroll up to see more carousels. Each carousel can have cards for a
particular site or several sites.
Initially, Google is pushing out rich cards for two categories of content, namely movies and recipes.
The search results will appear only in English for now for mobile
searches. Google has revealed that it is "actively experimenting" with
more methods of extending the feature publishers' way.
The company asserts that it has developed a wide-ranging tool set, as
well as updated its developer documentation, to help both developers
and site owners transition from preliminary exploration to application
to performance monitoring.
Google is optimistic that the rich cards format will enable site
owners to get noticed in the crowd, in the Search results space. This
will enable them to lure more target users. Search users will also
benefit as the rich cards offer a more pictorial format, which will aid
them in finding the requisite information quicker.
Whether rich cards are successful in meeting Google's goal of
simplifying search on mobile devices - and making googling more visual -
remains to be seen.
- See more at:
http://www.techtimes.com/articles/160105/20160521/google-improves-mobile-search-with-rich-cards-offers-new-visual-experience.htm#sthash.SLaSqIB8.dpuf
Talks about Motorola bringing back the clamshell phone or flip phone this year have been making rounds on the web. However, a most recent report negates the speculation.
A video released by Motorola Mobility may have suggested that Motorola Razr flip phone might be revived. The clip showed several young people who may have been students in a high school halls setting using and talking on the flip phone. The description of the clip stated, "Flip back to the Razr days of yesteryear and get ready for the future."
According to TechPortal, the original Motorola Razr hit the market in 2004 and then sold over 130 million handsets in the following four years. Aside from the stylish and appealing look on the outside, Motorola Razr also featured qualities and highlighted aspects that turned the model into a success especially among the youth. This was practically the essence showcased in the Motorola Mobility teaser video.
What set Motorola Razr apart from its competitors then was the design. Its clamshell design was thinner as opposed to its contemporaneous mobile phone models, hence the name "Razr." It had a three-inch display, ran at a reasonable speed and on adequate storage.
In the event a Motorola Razr 2016 version launches, it may still carry its flagship flip and slender design. Looking back on the Razr clamshell phone may prove to be pleasing, nice and nostalgic but this may well remain a throwback in memory and will not be the actual case.
In Time's most recent report, it debunked all the previous speculations about the Motorola Razr comeback. It wrote with assurance that there are no plans for the brand to release a 2016 version of the extensively well-received handset from more than a decade ago.
The article quoted Motorola representative Kathryn Hanley explaining that the brand appreciates how the throwback video was generally welcomed with excitement as Razr was an iconic phone which demonstrated that a mobile phone could function excellently and at the same time be stylish. Hanley went on to clear the air saying, "While Moto is not re-releasing the RAZR, we will transform mobile again on June 9."
The date refers to the upcoming Lenovo Tech World event in San Francisco where Motorola is slated to introduce new products and announce other news.
Lenovo acquired Motorola Mobility from Google in 2014.
The year was 2004, and Motorola had just announced what was then an insanely thin flip phone, the RAZR V3. It was -- and still is -- a head-turner, and eventually over 130 million units were sold in total. Such were the glorious days of Motorola. Twelve years later, the now Lenovo-owned
brand appears to be prepping a relaunch of this legendary model,
according to its teaser video of a nostalgic walkthrough at a high
school. "Flip back to the Razr days of yesteryear and get ready for the
future." Well, our money's on an Android refresh of the RAZR flip phone,
and we're already quite stoked about that. The big unveil will take
place at Lenovo Tech World on June 9th, and we have a feeling that this
new RAZR may overshadow the new Moto X devices that are also expected there.
At Google I/O today, the company announced what they are calling "Google Daydream."
This is their upcoming virtual-reality platform that takes Google
Cardboard several steps further. Daydream is a platform that will
provide guidance for both hardware and software developers
to create truly immersive Android N compatible VR hardware, games and
experiences. The first hardware will be available this Fall. Companies
such as HTC, Samsung, ASUS and others are working on smartphone handsets
and VR headsets that are Daydream compatible. Game developers like EA
and Ubisoft are already working on compatible games. Google is also
releasing reference devices including a headset and controller to
encourage developers and companies to develop content for Daydream. In
addition to Daydream, Google announced Allo, Duo, Instant Apps, and a stable Android N developer preview.
Researchers in the US are creating half-human, half-animal embryos to help save
lives, particularly people with a wide range of ailments. The embryos
create better animal models to study the occurrence of human diseases
and its progression.
One of the aims of the experiment using chimeras
is to create farm animals with human organs. The body parts could then
be harvested and transplanted into very sick people, reports
Boisestatepublicradio.
However, a number of bioethicists and scientists frown on the
creation of interspecies embryos which they believe crosses the line.
New York Medical College Professor of Cell Biology and Anatomy Stuart Newman calls the use of chimeras as entering unsettling ground which damages “our sense of humanity.”
Ryan Troy and John Powers, from the University
of California-Davis (UCD), in a paper titled “Human-Animal Chimeras:
What are we going to do?” cites the mention of chimeras in Greek mythology as a monster made up of multiple parts of different animals such as a goat’s body, a lion’s head and a serpent’s tail.
The two admit that the use of chimeras is controversial, but believe that many scientists think it is not a threat
to human dignity. They add that on top of the list of fears on the use
of chimeras is that humans and animals should stay as separate entities.
Because of the question mark on the ethics of its use, the National
Institutes of Health has placed a moratorium on funding of chimera
experiments, but some researchers resort to alternative funding. In
defending the UCD’s creation of chimeras, reproductive biologist Pablo
Ross explains, “We’re not trying to make a chimera just because we want
to see some kind of monstrous creature … We’re doing this for a
biomedical purpose.”
Ross is attempting to create a pancreas which could be transplanted
into a diabetic. He uses new gene-editing techniques to remove the gene
which pig embryos need to make a pancreas. Ross injects the
human-induced pluripotent stem cells into the pig embryos. For the
embryo to develop and produce an organ Ross implanted the chimera
embryos into the wombs of adult pigs, injecting 25 embryos into each
side of the pig’s uterus.
After 28 days, Ross would retrieve the chimeric embryos and dissects
it to see what the human stem cells are doing inside, if it starts to
form a pancreas.
Arizona State University bioethicists warns that by inserting human
DNA into animals and giving the animas some human capacities, it would
be “kind of, maybe, even playing God.” But Ross replies, “I don’t
consider that we’re playing God or even close to that. We’re just trying
to use the technologies that we have developed to improve people’s life.”
A handful of scientists around the United States are trying to do
something that some people find disturbing: make embryos that are part
human, part animal.
The researchers hope these embryos, known
as chimeras, could eventually help save the lives of people with a wide
range of diseases.
One way would be to use chimera embryos to create better animal models to study how human diseases happen and how they progress.
Perhaps
the boldest hope is to create farm animals that have human organs that
could be transplanted into terminally ill patients.
But some
scientists and bioethicists worry the creation of these interspecies
embryos crosses the line. "You're getting into unsettling ground that I
think is damaging to our sense of humanity," says Stuart Newman, a professor of cell biology and anatomy at the New York Medical College.
The experiments are so sensitive that the National Institutes of Health has imposed a moratorium on funding them while officials explore the ethical issues they raise.
Nevertheless,
a small number of researchers are pursuing the work with private
funding. They hope the results will persuade the NIH to lift the
moratorium.
"We're not trying to make a chimera just because we want to see some kind of monstrous creature," says Pablo Ross, a reproductive biologist at the University of California, Davis. "We're doing this for a biomedical purpose."
The NIH is expected to announce soon how it plans to handle requests for funding.
Recently, Ross agreed to letme
visit his lab for an unusual look at his research. During the visit,
Ross demonstrated how he is trying to create a pancreas that
theoretically could be transplanted into a patient with diabetes.
The first step involves using new gene-editing techniques to remove the gene that pig embryos need to make a pancreas.
Working
under an elaborate microscope, Ross makes a small hole in the embryo's
outer membrane with a laser. Next, he injects a molecule synthesized in
the laboratory to home in and delete the pancreas gene inside. (In
separate experiments, he has done this to sheep embryos, too.)
After the embryos have had their DNA edited this way, Ross creates another hole in the membrane so he can inject human induced pluripotent stem cells, or iPS for short, into the pig embryos.
Like
human embryonic stem cells, iPS cells can turn into any kind of cell or
tissue in the body. The researchers' hope is that the human stem cells
will take advantage of the void in the embryo to start forming a human
pancreas.
Because iPS cells can be made from any adult's skin
cells, any organs they form would match the patient who needs the
transplant, vastly reducing the risk that the body would reject the new
organ.
But for the embryo to develop and produce an organ, Ross
has to put the chimera embryos into the wombs of adult pigs. That
involves a surgical procedure, which is performed in a large operating
room across the street from Ross's lab.
Pablo Ross of the University of California, Davis inserts
human stem cells into a pig embryo as part of experiments to create
chimeric embryos.
Rob Stein/NPR
The day Ross opened his lab to me, a surgical team was
anesthetizing an adult female pig so surgeons could make an incision to
get access to its uterus.
Ross then rushed over with a special
syringe filled with chimera embryos. He injected 25 embryos into each
side of the animal's uterus. The procedure took about an hour. He
repeated the process on a second pig.
Every time Ross does
this, he then waits a few weeks to allow the embryos to develop to their
28th day — a time when primitive structures such as organs start to
form.
Ross then retrieves the chimeric embryos to dissect them
so he can see what the human stem cells are doing inside. He examines
whether the human stem cells have started to form a pancreas, and
whether they have begun making any other types of tissues.
The
uncertainty is part of what makes the work so controversial. Ross and
other scientists conducting these experiments can't know exactly where
the human stem cells will go. Ross hopes they'll only grow a human
pancreas. But they could go elsewhere, such as to the brain.
"If
you have pigs with partly human brains you would have animals that
might actually have consciousness like a human," Newman says. "It might
have human-type needs. We don't really know."
That possibility
raises new questions about the morality of using the animals for
experimentation. Another concern is that the stem cells could form human
sperm and human eggs in the chimeras.
"If a male chimeric pig
mated with a female chimeric pig, the result could be a human fetus
developing in the uterus of that female chimera," Newman says. Another
possibility is the animals could give birth to some kind of part-human,
part-pig creature.
"One of the concerns that a lot of people
have is that there's something sacrosanct about what it means to be
human expressed in our DNA," says Jason Robert,
a bioethicist at Arizona State University. "And that by inserting that
into other animals and giving those other animals potentially some of
the capacities of humans that this could be a kind of violation — a kind
of, maybe, even a playing God."
Ross defends what his work. "I
don't consider that we're playing God or even close to that," Ross
says. "We're just trying to use the technologies that we have developed
to improve peoples' life."
Still, Ross acknowledges the
concerns. So he's moving very carefully, he says. For example, he's only
letting the chimera embryos develop for 28 days. At that point, he
removes the embryos and dissects them.
If he discovers the stem
cells are going to the wrong places in the embryos, he says he can take
steps to stop that from happening. In addition, he'd make sure adult
chimeras are never allowed to mate, he says.
"We're very aware
and sensitive to the ethical concerns," he says. "One of the reasons
we're doing this research the way we're doing it is because we want to
provide scientific information to inform those concerns."
Ross is working with Juan Carlos Izpisua Belmonte from the Salk Intitute for Biological Studies in La Jolla, Calif., and Hiromitsu Nakauchi at Stanford University. Daniel Garry of the University of Minnesota and colleagues are conducting similar work.
Dag Kittlaus and Adam Cheyer created the artificial intelligence
behind Siri, Apple's iconic digital assistant, and one of the first
modern apps to capably handle natural language queries on a smartphone.
Today the pair showed off their newest creation, Viv, a next generation AI assistant
that they have been developing in stealth mode for the last four years.
The goal was to create a better version of Siri, one that connected to a
multitude of services, instead of routinely shuffling queries off to a
basic web search. During a 20-minute demo onstage at Disrupt NYC, Viv
flawlessly handled a dozen complex requests, not just in terms of
comprehension, but by connecting with third-party merchants to purchase
goods and book reservations.
The major difference between Siri and Viv is that the latter is a far
more open platform. One of the biggest frustrations with Siri is that
it has only a small number of tasks it can complete. For the vast
multitude of requests or queries, Siri will default to a generic web
search. Viv's approach is much closer to Amazon's Alexa or Facebook's
Messenger bots, offering the ability to connect with third-party
merchants and vendors so that it can execute on requests to purchase
goods or book reservations. The company's tagline — intelligence becomes
a utility — nicely sums up its goal of powering the conversational AI
inside a multitude of gadgets and digital services.
Apple killed off Siri's partnerships
The critical distinction here is between broad horizontal AI and specialized vertical AI. A service like x.ai,
which shares investors with Viv, is focused on just one thing:
scheduling meetings. It does this task very well, but it can't do
anything else. Siri, Alexa, Cortana, and their ilk are meant to be broad
AI, able to handle a variety of different tasks. They act as the
command and control bot, forwarding on queries to the appropriate bot
for booking a hotel room or ordering flowers. So far Amazon and Facebook
have been clear that their approach will aim to integrate with as many
third-party services as possible. Siri and Google Now, on the other
hand, have remained more closed off.
Microsoft is also investing heavily in the idea of smart bots,
and showed off an engine for creating them during its recent developer
conference. At the heart of the paradigm shift from apps to bots was the
concept of "conversation as a platform," said CEO Satya Nadella. Viv
epitomizes this trend. Its creators have been working on the problem of
natural language comprehension for over a dozen years, starting with a
DARPA-backed AI project in the early 2000s. That has led to a very
nuanced and powerful system, capable of understanding and acting on
queries like: "On the way to my brother's house, I need to pick up some
cheap wine that goes well with lasagna."
We want a Samantha, not a Siri
"Viv is designed to be devices agnostic — think one platform, open to
all services, for all devices, personalized for you. Viv's goal is to
be ubiquitous so it will understand your preferences and history as you
engage with it on your mobile device, or in your car, or with your smart
device at home," said Adam Koopersmith, a partner with Pritzker Group
Venture Capital, one of Viv's investors. "Our sense is there will be a
move away from having hundreds of different apps that act independently.
These services will be integrated into everyday life. Viv will be the
platform to enable it."
Viv will be in competition with Microsoft's bot engine and the APIs
put out by Facebook to encourage developers to build bots for Messenger.
At today's demo it handled complex queries that sounded a lot like the
sort of thing Hound is good at.
Viv responded to things like: "Was it raining in Seattle three
Thursdays ago?" and "Will it be warmer than 70 degrees near the Golden
Gate Bridge after 5PM the day after tomorrow?" But she could also handle
more social queries like: "Send Adam 20 bucks" and "Send my mom some
flowers." Onstage, at least, these actions happened far more seamlessly
than the experience folks have been having with Facebook's bots.
Viv is making the same promise as Microsoft, Facebook, and other giants
So far, Hound hasn't been a huge hit with consumers. And, at least at
first, Viv certainly won't be able to offer merchants distribution to a
massive consumer base the way Facebook and Amazon can. Viv's founders
believe that third-party developers and merchants will choose them
because of the strength of their AI and because they are a neutral
party, not a tech giant. Onstage it showed off what it claims is a
breakthrough: "dynamic program generation." With every verbal request
Viv dynamically spit out code showing off how it understood and handled
the request. That would hypothetically allow developers to build out a
robust conversational UI for their services simply by speaking to Viv
and tweaking the code she generates in return.
"There is a fundamental language of systems that provides a way to
describe both structures and functions that is universal across any kind
of system... I am nearing completion of the basic specification of the
language and will be presenting my results at the next ISSS conference
in Boulder CO this July... This language, which I formally call SL, but
privately call "systemese", is like the machine language of the universe. Any system you choose to analyze and model can be described in this language...!
The beauty of the approach is that the end product of analysis is a
compilable program that is the model of the system. The language does
not just cover dynamics (e.g. system dynamics), or agents (agent-based),
or evolutionary (e.g., genetic algorithms) models. It incorporates all
of the above plus real adaptivity and learning (e.g. biological-like),
and real evolvability (as when species or corporations evolve in complex
non-stationary environments)... Systemese and mentalese (the language
of thought), a concept advanced by philosopher of mind Jerry Fodor, are
basically one in the same! That is, our brains, at a subconscious level,
use systemese to construct our models of how the world works.
--
Systems Science Ascending
After years of development in increasingly fracturing sub-disciplines it
seems that systems science as an integrated whole domain of knowledge
is rising again. For those familiar with the history of systems science
you will recall that in the early 1950's after some fairly spectacular
developments in fields such as cybernetics, information and
communications theory, computation and algorithms, and a more holistic
view of biology emerged the concept of general systems theory, or GST (von Bertalanffy,
1969). This theory advanced the notion that everything in the Universe
could be understood as having common patterns of organization,
structures, and functions regardless of the specific materials or
energies involved. Many researchers have investigated the nature of
these patterns (and a few more discovered since that time) and have
found they exist everywhere as suggested. The promise of a general
theory of systems was a way to actually understand phenomena in a deeper
way than disciplinary sciences had been able to achieve. But the
promise was not to be realized subsequently.
The nature of academia (PhD thesis and tenure requirements) tended to
push researchers into increasingly narrow sub-disciplines within the GST
domain. Complexity theorists gravitated toward their version of complex
systems. Information theorists did the same, as did cyberneticians.
Increasingly over the years the venues for sharing results among
sub-disciplines became fewer as those for sharing within became
dominant. Specialization, which seems to be a natural trait of our
species, pulled the rug from under what promised to be a grand way to
understand the world and organize our increasing knowledge (as a
system!). For many years, even while people still talked about systems
and even applied aspects of systems thinking to various disciplines
(e.g. systems biology) the field as an integrated domain took a back
seat to the disciplines themselves. How many degree programs in systems
science can you name? The only one I know of in the US is at Portland
State University (there are numerous others in the world but they tend
to occupy more peripheral realms of academia).
Even as this dissociation of central ideas in GST progressed, and as the
notion of a real science of systems as center to all other sciences
failed to take hold, there remained a group of “loyalists”, many of whom
descended from the giants of the 50's and 60's, who carried the banner
and maintained a vision of what could be if systems science were ever
recognized for its real potential more broadly (see below). I had been
aware of these groups and some of their work, but had not been able to
see the kind of work that I thought would be needed in order to
reconsolidate the field and provide a general scaffolding for the
knowledge of systems. My main concern was that many of the articles
published in some of their journals and other venues still carried a
special interest narrowness and I did not see many attempts to try to
integrate these into a coherent GST-like framework. Most of all I could
not find (and I looked very hard) any truly integrated textbooks on
systems science. There were plenty of books on systems science or
systems thinking (e.g. a whole series of excellent books by Fritjof Capra,
but written for general readers and, in my view, only hinting at
systems principles). So I decided to go the lone wolf route and develop a
textbook for undergraduate-level, but general science majors. I got a
great boost from my colleague Mike Kalton (co-author) and, together we
crafted a book that we believe covers the breadth of topics found in
systems science and explores many of these to sufficient depth that
readers in the sciences should be able to see how some general
principles of systemness apply.
With that book published by a major publishing house, I started meeting
people in these organizations and exploring collaborations. To my great
surprise and relief I discovered a fairly recent effort being carried on
in the UK by a group of researchers to revive GST and bring it up to
date with solidly grounded principles (I should point out that while I
have confidence that those I published are relatively solid, my methods
for identifying them were less rigorous than this group's approach).
They are looking for a more general theory, what they are calling GST*. We are now in conversations about how to push this agenda forward.
Meanwhile I have pushed ahead with two major book projects. The first is
tentatively titled “Understanding Systems: Systems Analysis, Modelling,
and Design” in which I lay out the formal language of systems (see
below) and the methodology of top-down functional/structural
deconstruction. The end result of this process is a complete
specification of a system of interest (at an appropriate level of
abstraction) in the language which can then be compiled into computer
codes and run in model simulation. If the system of interest is an
existing or desired human-built one, the process leads to a complete
specification for building, i.e. a systems engineering set of
specifications. I have completed the formal mathematical framework for
the language (actually a formal definition of systemness!) and the
introduction of the methodology. But there is a long way to go. More on
this later.
The second project is to apply the methods in the book just described to
a “starting” analysis of the human social system or HSS. I have already
described this in a previous post and will likely have a lot to say
about it in subsequent posts. Needless to say it is a massive scale
project. My objective is not to do a complete and detailed analysis but
merely to show how such an analysis would proceed if following the
systems analysis method. My plan is to go deeper into a few subsystems,
such as the “Science and Engineering” and “Economic” subsystems, ones
where I have some “expertise” or domain knowledge. These will serve as
examples to other domain experts on how to proceed in their various
areas. I could easily envision the project giving rise to a global
cooperative (open source) project, much like a Wikipedia, to collect and
organize all knowledge relating to the human condition — that is as
long as computers and the Internet are still working!
One of the most critical subsystems of the HSS is, of course, the
biological (and psychological) human subsystem. We are just one
subsystem integrated into our cultural cocoons. This one I have already
made a stab at. My whole series on the topic of “sapience”, and the
subsequent book that I have completed were an effort to analyze the
human psyche in an effort to discover why we consistently fail to learn
from prior experiences and fail to make good judgements in our
endeavours to flourish (e.g. global warming as a result of our
persistent demand for power). If you have read any of that work you know
that my conclusion was fairly discouraging. I have not yet decided what
to do with the book. I have shopped around for a publisher but most
responses include comments about how gloomy the conclusion is — where is
the hopeful message at the end? Moreover, it might be that it more
properly fits into a deeper analysis of the biological human subsystem,
that is I should hold it back until the context for it has been better
established. I still ponder that idea.
But several developments involving the stature and visibility of systems
science in the established science community gives me pleasure if not
hope. The systems organizations and research groups mentioned above are
gaining some leverage. And why not? Most people who spend any time at
all thinking about the major global problems that we face realize that
everything is connected (a system) and will have to be addressed
systemically. On the engineering side, many large complex product and
service producing organizations (except I suppose our governments) have
realized the need for a more informed version of systems engineering,
that is informed by actual systems science (as all other engineering
disciplines are informed by physics, chemistry, and/or biology, etc.)
The need for systems thinking is becoming quite prominent. But
systems thinking needs more than just casual thoughts about
connectedness. It needs an actual science of systems to make explicit
the nature of systemness and to provide the principles to guide methods
and organize knowledge gained. By the evidence I have seen over the last
year, it seems that an integrated systems science is ascending.
The NGSS is a comprehensive K-12 STEM education standard that seeks to
teach science, math, and engineering in a more holistic fashion,
especially recognizing issues that are common across fields to increase
students' grasping that science is a process rather than just a set of
separate subjects. To that end the standards include a call for the
inclusion of cross-cutting themes, topics that apply to all fields of
STEM. For example, quantitative thinking applies to all of these fields
and at many different levels. It is recognized that learning calculus,
for example, in the same way that arithmetic is learned — as a
standalone subject onto itself — is completely non-viable. However,
calculus learned and used in the context of all of the science subjects
is much more meaningful since the application is evident at the outset.
Recently the NSF has called for proposals to develop curriculum for some
of the cross-cutting themes that they champion and one of them is
explicitly systems science or systems thinking (the others are actually
implicitly systems related, such as the relations between the vertical
views of the sciences, i.e. from physics through chemistry through
biology, etc.) Several of the Linz team are working on two related
projects dealing with systems literacy in K-12 education. One of them is
a proposal to the NSF to incorporate the SS research work of the IFSR (International Federation of Systems Research) and the ISSS (International Society for Systems Science)
help inform the specific recommendations for cross-cutting themes in
SS. Kalton's and my book is named as using examples from many different
sciences to show how systems principles apply to all STEM subjects. It
may be that our book will one day see duty in K-12 teacher education
programs to prep them for teaching STEM.
Meanwhile the NSF has a strong interest directly in cross-cutting themes
across its various directorates. Normally these are silos and
administrate grants to disciplinary research with an occasional
interdisciplinary approach where two or more directorates cooperate on a
single grant program. It seems that these latter kinds of programs are
popping up more frequently, especially with the rise of global systemic
problems like climate change and energy. There was a meeting which many
of the directors attended specifically to hear about how systems science
is a transdisciplinary approach that could provide a backbone to
organizing research agendas. It is just at the discussion stage but it
seems many of the directors found it a very interesting idea.
The Language of Systems
In order for scientists of different disciplines to work in a truly
transdisciplinary way they will need a common language to speak. This is
a lot like the case of multiple kinds of computer languages being
compiled or interpreted on a single architecture, say C, Java, Python,
Fortran, etc. all being translated down to the machine code for a single
computer. They all look different on the surface, but they all have a
deeper sameness at the machine code level. Math is not sufficient
because it is not easily interpretable from one discipline to another.
It is not directly descriptive of structures, except, possibly for
geometry or topology, mostly only functions and relations. Powerful
enough if you already know the context of structures in which it is
being used, but not outside of that domain.
It turns out that there is a fundamental language of systems that
provides a way to describe both structures and functions (I briefly
describe some of it in our book) that is universal across any kind of
system and conforms to the principles outlined in our book. I am nearing
completion of the basic specification of the language and will be
presenting my results at the next ISSS conference in Boulder CO this
July. This might prove timely as it could help persuade the NSF and
others involved in the NGSS as well as educators, that there is hope for
finding ways to actualize those cross-cutting systems themes.
The language has a basic lexicon, syntax, and semantics (as well as
pragmatics provided by the principles of systems). I have been using
formal language theory to develop it so that it will be extensible as
future SS researchers may discover additional principles or nuances not
now recognized. This language, which I formally call SL, but privately
call “systemese”, is like the machine language of the universe. Any
system you choose to analyze and model can be described in this
language! I hope to find funding to start implementing a set of tools
for using the language for doing formal analysis. The beauty of the
approach is that the end product of analysis is a compilable program
that is the model of the system.
The language does not just cover dynamics (e.g. system dynamics), or
agents (agent-based), or evolutionary (e.g., genetic algorithms) models.
It incorporates all of the above plus real adaptivity and learning
(e.g. biological-like), and real evolvability (as when species or
corporations evolve in complex non-stationary environments).
There is one additional aspect of systemese that goes beyond its use in
system description and modelling. From my studies of brain functions
(regarding sapience) I have ascertained that systemese and mentalese (the language of thought), a concept advanced by philosopher of mind Jerry Fodor,
are basically one in the same! That is, our brains, at a subconscious
level, use systemese to construct our models of how the world works. Our
brains are designed to build models of the world based on our
experiences of it. Because the world is composed of systems, we all get a
similar result in terms of the models we construct, ergo we can agree
on much about the world. Even so we each have different perspectives and
very different experiences, even of the same phenomena, so tend to have
a high degree of variation in the details of our models. Plus our
wetware is notoriously prone to error — we make mistakes in perceptions
and judgements. Furthermore, our perspectives are modulated by our
affect system and our personalities. Sometimes the latter two can
distort our versions of reality. But by and large most people, most of
the time, can agree on some fundamental aspects of systems in the world.
That is, when looking at real systems in their environments, they see
pretty similar things, working in similar ways.
The problem is when they attempt to take into account much broader
environments and time scales beyond their sensory limits, when they
attempt to model things beyond their ken. When they contemplate things
like global warming or our own origins they are not really observing
experiential inputs to their models. Rather they are reduced to
accepting whatever authorities they have come to trust. So those models,
the ones that extend beyond ordinary experience, can be extremely
varied and all too often incorrect. In addition, when experience is
lacking but motivation is strong we have a tendency to just make stuff
up that “sounds” right.
But with that exception in mind, I think all human beings think
subconsciously in systemese. I am right now working on the translation
of systemese (mentalese) into natural language constructs (e.g., the
translation of system/process objects into nouns). If this goes well I
think I will be able to show that the natural language of Homo sapiens
has its roots in systemese that evolved from the earliest brains. Early
brains, as are found in cephalopods (squid and octopii) or arthropods
(insects, spiders, etc.) appear to be the first to be able to form
images of objects and detect relations between objects, including
possibly action relations. These brains are adapted to speak a very
primitive systemese internally, without conscious awareness. Over the
course of evolution systemese developed more nuance with increasing
brain complexity. The human brain with its extensive neocortex and
prefrontal lobes has an extensive system vocabulary and more refined
syntax that has evolved from the primitive versions. We also have an
elaborate verbal input/output subsystem which makes it possible to share
our thoughts (the conscious ones) with each othr. What I have called
systems thinking in my development of the concept of sapience reflects
the idea that human brains are in closer contact with this systemese as
it frames our other sapience constructs (judgement, moral sentiment, and
strategic thinking: See the category Sapience
for all that I have written on this subject). However, like all of
sapience, the strength of that capability is variable and most commonly
low in the population.
After I publish in the literature I will share these ideas with the readers here. We'll see if it flies.
Solutions
Over the years this blog has been dedicated to the analysis of the human
condition from a systems perspective. I have asked questions ranging
from biophysical realities (as in real economics) to how the mind works,
and what might consciousness be. Along the way I have offered a few of
the answers I have developed (my beliefs) to some of those questions. I
have spent no small amount of text analyzing the plight of the human
condition regarding the major global problems that confront us and have
concluded that in every case they are related to one another and are of
our own making. In a comment on my last post Don Stewart asked if maybe I
had the answers but just hadn't mentioned them. A few times I did
explore things like living styles (permaculture with low energy
footprint) that would be more sustainable than what the current BAU
style could be. But as often pointed out to me in comments and actually
quite obvious to me already, these cannot be solutions to the problems
everyone (or the vast majority) want to have solved. The problem they
want solved is for all the problems to go away through some magic of
technology (e.g. alternative energy) so that they are not inconvenienced
in their lifestyles. They do not want to give up their lifestyles,
dependent on lots of high power energy so they won't even explore the
options of living on far fewer resources (where the goal is still to
live as comfortably as possible). And so they are doomed to suffer the
loss of those resources the hard way, through depletion, and without
adequate preparation for how to live.
On several occasions I have stated that the solution is the one the
Universe has always had for systems that are out of control — the rise
of negative feedback loops amplified such that the system is brought low
(or destroyed completely) so that something new can evolve in its
place. The physical (material) resources aren't used up; the metals and
other minerals will still be available in abandoned cities. But the
energy resources in fossil fuels will be gone, requiring that any future
evolution must be based on real-time solar input as was the case for
billions of years prior. The biosphere and humanity in particular are
about to go through an evolutionary bottleneck event. And then the world
starts over, with or without a hominid ape population.
That isn't good news for us. It is just the way the Universe works. The
real problem we should be trying to solve is: Can our genus make it
through this bottleneck with a viable breeding population, and if so,
what will it take in the way of preparation? I can't easily address the
first part. But I can work on a little piece of the second part, namely
to consolidate our hard won knowledge of systems science and encode it
into a medium that stands a chance of making it through the bottleneck.
The book, my subsequent work on systemese, and leveraging on the current
interest in systems science/thinking by the NSF (and other
organizations) might be useful in this regard. The idea is that if there
is a future Homo, say 5,000 years from now, they might be able
to start from a better position of understand the world than we did
coming out of the Pleistocene. This is my version of acting on faith and
with purpose!
However I have begun to realize I cannot do this work alone. I could use
some help with both book projects and setting up an open source project
for the development of an SL compiler and graphic
front-end/knowledgebase back-end. I am lucky in starting to work with
some new colleagues in various parts of the world, at least on the
systems literacy and research questions. But I will need to enlist many
others, especially younger people, who can grasp this vision and are
ready to enlist in the effort. My email box is open!
References
Bertalanffy, L. von, (1969). General System Theory, George Braziller, New York.
My publication at Siggraph in '98 proposed conversion of
brainwave algorithms to create imagery and sound, plus AI to drive the imagery for guiding VR subject
towards a target brainwave state for deepening immersion into virtual
environments.
I've recently been writing ideas surrounding
creation of an image-based language to introduce a new communication
paradigm.
1. Matching mind images to an image/ video bank
2. Mapping image/impression-based communication forms which can be triggered by voice command
Highlights from the article (below):
Her vision is broad and sweeping: it runs from a new generation of
extremely high-resolution, affordable MRI machines for early detection
of cancer, heart disease, and more, to a far-out time (or maybe not so
far-out) when machines can read people’s minds and people can
communicate—with each other and maybe even with animals—via thoughts.
The idea “leverages the tools of our times,” Jepsen says, citing
advances in everything from physics to optoelectronics to consumer
electronics to big data and A.I. that can be combined to shrink the
size, improve the functionality, and lower the cost of MRI. “I could no
longer wait. I’m still writing up the patents. But I am incredibly
excited to strike off on this direction,” she says.“My big bet is we can use that manufacturing infrastructure to create
the functionality of a $5 million MRI machine in a consumer electronics
price-point wearable. And the implications of that are so big.” She
says every doctor’s office in the world could afford these wearable
devices and use them for early detection of neurodegenerative disease,
cancer, cardiovascular disease, internal bleeding, blood clots, and
more.
I had long planned a phone call with Mary Lou Jepsen for this
afternoon—a prep session for a chat I will be doing with her a week from
Monday night at Xconomy’s Napa Summit, where she is the featured dinner speaker. It was to be a normal prep chat until I got to work this morning and learned that CNET, Engadget, and Tech Insider
had all reported that the technology visionary was planning to leave
her post as executive director of engineering for Facebook and Oculus,
to focus on a new startup. It turned out she had talked about her plans
last night during a keynote speech at the Women of Vision Awards banquet
in Santa Clara, CA—and the media outlets had all seized on the news.
“I was actually really surprised anybody picked that up,” Jepsen told
me (showing she doesn’t fully understand what a big deal she is). So I
took advantage of the call to ask her more. Some of our talk was off the
record, but much of it was on the record, including quite a bit about
her new plans and the thinking behind them.
Her vision is broad and sweeping: it runs from a new generation of
extremely high-resolution, affordable MRI machines for early detection
of cancer, heart disease, and more, to a far-out time (or maybe not so
far-out) when machines can read people’s minds and people can
communicate—with each other and maybe even with animals—via thoughts.
The idea “leverages the tools of our times,” Jepsen says, citing
advances in everything from physics to optoelectronics to consumer
electronics to big data and A.I. that can be combined to shrink the
size, improve the functionality, and lower the cost of MRI. “I could no
longer wait. I’m still writing up the patents. But I am incredibly
excited to strike off on this direction,” she says.
The startup, whose name has not previously been released as far as I
can tell, is called Open Water (it could also be OpenWater, “not sure
yet…either is OK for now,” she says). “Peter Gabriel gave me the name.
He is a great advisor,” Jepsen says. In particular, she was inspired by
this article he wrote for Edge.org, called Open Water–The Internet of Visible Thought, in which he credited Jepsen for introducing him “to the potential of brain reading devices.”
Jepsen says she can’t talk about funding and more specific plans for
Open Water yet, and that she will remain at Facebook until August. But
here are some highlights of what she could say:
“What I try to do is make things that everybody knows are utterly,
completely impossible—I try to make them possible,” Jepsen sums up. She
does that by leveraging what she calls her “strange background” that
encompasses physics, computer science, media technology, art, electrical
engineering, and more. “That all comes together for me.” Indeed, you
can find more in this companion piece on that background,
which includes stints at Google X, One Laptop per Child (which she
co-founded), the MIT Media Lab, Intel, her own startups, and more.
In the case of Open Water, part of her motivation is her own health.
“I’m a brain tumor survivor,” she says. She had surgery to remove a
brain tumor in 1995, and since then has taken pills “twice a day every
day for the last 21 years to stay alive.” That has led her to read a lot
on the side about neuroscience—and think about how to advance the
field.
Part of the idea behind Open Water involves taking things at “the
hairy edge of what physics can do,” Jepsen says, and then “using my
substantial capability in consumer electronics” to make them possible at
consumer electronics price points. She says there is a huge potential
in the manufacturing plants in Asia that are primarily used to make
OLEDs, LCDs, and such. Jepsen adds that these consumer electronics
manufacturers have been mostly focused on smartphones for the past
decade or so. But, she says, we’ve reached saturation in mobile phones,
and sales are declining. “What I see,” she says, are “the subcomponent
makers being really hungry for what the new, new thing is.”
“My big bet is we can use that manufacturing infrastructure to create
the functionality of a $5 million MRI machine in a consumer electronics
price-point wearable. And the implications of that are so big.” She
says every doctor’s office in the world could afford these wearable
devices and use them for early detection of neurodegenerative disease,
cancer, cardiovascular disease, internal bleeding, blood clots, and
more.
“It’s such a big idea, it’s what I wanted to do for a decade. It’s
why I went to MIT [Media Lab]. It’s why I went to Google,” she says. “It
turned out that Google really needed me to do some other stuff that was
way more important to Google at the time. I’ve been incubating this
since 2005…and I clearly see how to do it and how to realize it in a few
short years.”
One factor in advancing her idea was work published about five years
ago by a group led by Jack Gallant at U.C. Berkeley, Jepsen says. The
research group used a functional magnetic resonance imaging scanner to
track blood flow and oxygen flow and image the brains of people shown
hundreds of hours of videos. You can read more about it here,
but the main point Jepsen stressed to me was that the work (and
subsequent work) has produced a library or database of sorts of how
brains react to different images. A computer using artificial
intelligence can then use such a database to basically look at MRI brain
images in real time and interpret what people are thinking about or
reacting to. This ability has been demonstrated at dozens of labs to
gauge the brain’s reactions to words, music, math equations, and more,
she says. But the resolution is poor and the process is expensive,
requiring people to lie still in big chambers inside a huge magnet.
“I was really struck by that, so I started thinking this is great,
but we need to up the resolution,” she says. “It’s in my head, I’ve got
this plan. I’ve got these inventions that I’m working on, and my next
step is to let myself pursue it full time.”
It is easy to see the power of these ideas to help make MRI far more
affordable and accessible. But for Jepsen, that is just Phase One. She
talks about the ability to image human thoughts in new ways, for
instance, by helping stroke sufferers who can’t talk find a new way to
communicate via their thoughts. Or for amputees to harness their
thoughts to move prosthetics more naturally.
And then she goes a step or two farther. “Can you imagine a movie
director waking up with an image of a new scene in her head, and just
being able to dump her dream” into a computer, she says. ”It could be so
much more efficient than the way we do it now.” For musicians, she
muses, this could be “a way to get the music out of your head.”
But that’s not all. “Maybe we can communicate with animals, maybe we
can scan animal brains and see what images they are thinking of,” Jepsen
says. “So little is known. Dolphins are supposed to be really
smart—maybe we can collaborate with them.”
It all sounds pretty far-out, I know, and she says so, too. But given
how long Jepsen has had these ideas in her head—and how much work has
been done in brain-machine interfaces—perhaps the world is finally ready
to receive her thoughts.
Reconstructing visual experiences from brain activity evoked by natural movies
Shinji Nishimoto, An T. Vu, Thomas Naselaris, Yuval Benjamini, Bin Yu & Jack L. Gallant (Current Biology 2011, PDF1.4M).
Quantitative modeling of human brain activity can provide crucial
insights about cortical representations and can form the basis for brain
decoding devices. Recent functional magnetic resonance imaging (fMRI)
studies have modeled brain activity elicited by static visual patterns
and have reconstructed these patterns from brain activity. However,
blood oxygen level-dependent (BOLD) signals measured via fMRI are very
slow, so it has been difficult to model brain activity elicited by
dynamic stimuli such as natural movies. Here we present a new
motion-energy encoding model that largely overcomes this limitation. The
model describes fast visual information and slow hemodynamics by
separate components. We recorded BOLD signals in occipitotemporal visual
cortex of human subjects who watched natural movies and fit the model
separately to individual voxels. Visualization of the fit models reveals
how early visual areas represent the information in movies. To
demonstrate the power of our approach, we also constructed a Bayesian
decoder by combining estimated encoding models with a sampled natural
movie prior. The decoder provides remarkable reconstructions of the
viewed movies. These results demonstrate that dynamic brain activity
measured under naturalistic conditions can be decoded using current fMRI
technology.
Frequently asked questions about this work
Could you give a simple outline of the experiment?
The goal of the experiment was to design a process for decoding
dynamic natural visual experiences from human visual cortex. More
specifically, we sought to use brain activity measurements to
reconstruct natural movies seen by an observer. First, we used
functional magnetic resonance imaging (fMRI) to measure brain activity
in visual cortex as a person looked at several hours of movies. We then
used these data to develop computational models that could predict the
pattern of brain activity that would be elicited by any arbitrary movies
(i.e., movies that were not in the initial set used to build the
model). Next, we used fMRI to measure brain activity elicited by a
second set of movies that were completely distinct from the first set.
Finally, we used the computational models to process the elicited brain
activity, in order to reconstruct the movies in the second set of
movies. This is the first demonstration that dynamic natural visual
experiences can be recovered from very slow brain activity recorded by
fMRI.
Can you give an intuitive explanation of movie reconstruction?
As you move through the world or you watch a movie, a dynamic,
ever-changing pattern of activity is evoked in the brain. The goal of
movie reconstruction is to use the evoked activity to recreate the movie
you observed. To do this, we create encoding models that describe how
movies are transformed into brain activity, and then we use those models
to decode brain activity and reconstruct the stimulus.
Can you explain the encoding model and how it was fit to the data?
To understand our encoding model, it is most useful to think of the
process of perception as one of filtering the visual input in order to
extract useful information. The human visual cortex consist of billions
of neurons. Each neuron can be viewed as a filter that takes a visual
stimulus as input, and produces a spiking response as output. In early
visual cortex these neural filters are selective for simple features
such as spatial position, motion direction and speed. Our motion-energy
encoding model describes this filtering process. Currently the best
method for measuring human brain activity is fMRI. However, fMRI does
not measure neural activity directly, but rather measures hemodynamic
changes (i.e. changes in blood flow, blood volume and blood oxygenation)
that are caused by neural activity. These hemodynamic changes take
place over seconds, so they are much slower than the changes that can
occur in natural movies (or in the individual neurons that filter those
movies). Thus, it has previously been thought impossible to decode
dynamic information from brain activtiy recorded by fMRI. To overcome
this fundamental limitation we use a two stage encoding model. The first
stage consists of a large collection of motion-energy filters that span
a range of positions, motion directions and speeds as the underlying
neurons. This stage models the fast responses in the early visual
system. The output from the first stage of the model is fed into a
second stage that describes how neural activity affects hemodynamic
activity in turn. The two stage processing allows us to model the
relationship between the fine temporal information in the movies and the
slow brain activity signals measured using fMRI. Functional MRI records
brain activity from small volumes of brain tissue called voxels (here
each voxel was 2.0 x 2.0 x 2.5 mm). Each voxel represents the pooled
activity of hundreds of thousands of neurons. Therefore, we do not model
each voxel as a single motion-energy filter, but rather as a bank of
thousands of such filters. In practice fitting the encoding model to
each voxel is a straightforward regression problem. First, each movie is
processed by a bank of nonlinear motion-energy filters. Next, a set of
weights is found that optimally map the filtered movie (now represented
as a vector of about 6,000 filter outputs) into measured brain activity.
(Linear summation is assumed in order to simplify fitting.)
How accurate is the decoder?
A good decoder should produce a reconstruction that a neutral
observer judges to be visually similar to the viewed movie. However, it
is difficult to quantify human judgments of visual similarity. In this
paper we use similarity in the motion-energy domain. That is, we
quantify how much of the spatially localized motion information in the
viewed movie was reconstructed. The accuracy of our reconstructions is
far above chance.
Other studies have attempted reconstruction before. How is your study different?
Previous studies showed that it is possible to reconstruct static
visual patterns (Thirion et al., 2006 Neuroimage; Miyawaki et al., 2008
Neuron), static natural images (Naselaris et al., 2009 Neuron) or
handwriting digits (van Gerven et al. 2010 Neural Computation). However,
no previous study has produced reconstructions of dynamic natural
movies. This is a critical step toward obtaining reconstructions of
internal states such as imagery, dreams and so on.
Why is this finding important?
From a basic science perspective, our paper provides the first
quantitative description of dynamic human brain activity during
conditions simulating natural vision. This information will be important
to vision scientists and other neuroscientists. Our study also
represents another important step in the development of brain-reading
technologies that could someday be useful to society. Previous
brain-reading approaches could only decode static information. But most
of our visual experience is dynamic, and these dynamics are often the
most compelling aspect of visual experience. Our results will be crucial
for developing brain-reading technologies that can decode dynamic
experiences.
How many subjects did you run? Is there any chance that they could have cheated?
We ran three subjects for the experiments in this paper, all
co-authors. There are several technical considerations that made it
advantageous to use authors as subjects. It takes several hours to
acquire sufficient data to build an accurate motion-energy encoding
model for each subject, and naive subjects find it difficult to stay
still and alert for this long. Authors are motivated to be good
subjects, to their data are of high quality. These high quality data
enabled us to build detailed and accurate models for each individual
subject. There is no reason to think that the use of authors as subjects
weakens the validity of the study. The experiment focuses solely on the
early part of the visual system, and this part of the brain is not
heavily modulated by intention or prior knowledge. The movies used to
develop encoding models for each subject and those used for decoding
were completely separate, and there no plausible way that a subject
could have changed their own brain activity in order to improve
decoding. Many fMRI studies use much larger groups of subjects, but they
collect much less data on each subject. Such studies tend to average
over a lot of the individual variability in the data, and the results
provide a poor description of brain activity in any individual subject.
What are the limits on brain decoding?
Decoding performance depends on the quality of brain activity
measurements. In this study we used functional MRI (fMRI) to measure
brain activity. (Note that fMRI does not actually measure the activity
of neurons. Instead, it measures blood flow consequent to neural
activity. However, many studies have shown that the blood flow signals
measured using fMRI are generally correlated with neural activity.) fMRI
has relatively modest spatial and temporal resolution, so much of the
information contained in the underlying neural activity is lost when
using this technique. fMRI measurements are also quite variable from
trial-to-trial. Both of these factors limit the amount of information
that can be decoded from fMRI measurements. Decoding also depends
critically on our understanding of how the brain represents information,
because this will determine the quality of the computational model. If
the encoding model is poor (i.e., if it does a poor job of prediction)
then the decoder will be inaccurate. While our computational models of
some cortical visual areas perform well, they do not perform well when
used to decode activity in other parts of the brain. A better
understanding of the processing that occurs in parts of the brain beyond
visual cortex (e.g. parietal cortex, frontal cortex) will be required
before it will be possible to decode other aspects of human experience.
What are the future applications of this technology?
This study was not motivated by a specific application, but was aimed
at developing a computational model of brain activity evoked by dynamic
natural movies. That said, there are many potential applications of
devices that can decode brain activity. In addition to their value as a
basic research tool, brain-reading devices could be used to aid in
diagnosis of diseases (e.g., stroke, dementia); to assess the effects of
therapeutic interventions (drug therapy, stem cell therapy); or as the
computational heart of a neural prosthesis. They could also be used to
build a brain-machine interface.
Could this be used to build a brain-machine interface (BMI)?
Decoding visual content is conceptually related to the work on
neural-motor prostheses being undertaken in many laboratories. The main
goal in the prosthetics work is to build a decoder that can be used to
drive a prosthetic arm or other device from brain activity. Of course
there are some significant differences between sensory and motor systems
that impact the way that a BMI system would be implemented in the two
systems. But ultimately, the statistical frameworks used for decoding in
the sensory and motor domains are very similar. This suggests that a
visual BMI might be feasible.
At some later date when the technology is developed further, will it be possible to decode dreams, memory, and visual imagery?
Neuroscientists generally assume that all mental processes have a
concrete neurobiological basis. Under this assumption, as long as we
have good measurements of brain activity and good computational models
of the brain, it should be possible in principle to decode the visual
content of mental processes like dreams, memory, and imagery. The
computational encoding models in our study provide a functional account
of brain activity evoked by natural movies. It is currently unknown
whether processes like dreaming and imagination are realized in the
brain in a way that is functionally similar to perception. If they are,
then it should be possible to use the techniques developed in this paper
to decode brain activity during dreaming or imagination.
At some later date when the technology is developed further, will it
be possible to use this technology in detective work, court cases,
trials, etc?
The potential use of this technology in the legal system is
questionable. Many psychology studies have now demonstrated that
eyewitness testimony is notoriously unreliable. Witnesses often have
poor memory, but are usually unaware of this. Memory tends to be biased
by intervening events, inadvertent coaching, and rehearsal (prior
recall). Eyewitnesses often confabulate stories to make logical sense of
events that they cannot recall well. These errors are thought to stem
from several factors: poor initial storage of information in memory;
changes to stored memories over time; and faulty recall. Any
brain-reading device that aims to decode stored memories will inevitably
be limited not only by the technology itself, but also by the quality
the stored information. After all, an accurate read-out of a faulty
memory only provides misleading information. Therefore, any future
application of this technology in the legal system will have to be
approached with extreme caution.
Will we be able to use this technology to insert images (or movies) directly into the brain?
Not in the foreseeable future. There is no known technology that
could remotely send signals to the brain in a way that would be
organized enough to elicit a meaningful visual image or thought.
Does this work fit into a larger program of research?
One of the central goals of our research program is to build
computational models of the visual system that accurately predicts brain
activity measured during natural vision. Predictive models are the gold
standard of computational neuroscience and are critical for the
long-term advancement of brain science and medicine. To build a
computational model of some part of the visual system, we treat it as a
“black box” that takes visual stimuli as input and generates brain
activity as output. A model of the black box can be estimated using
statistical tools drawn from classical and Bayesian statistics, and from
machine learning. Note that this reverse-engineering approach is
agnostic about the specific way that brain activity is measured. One
good way to evaluate these encoding models is construct a corresponding
decoding model, and then assess its performance in a specific task such
as movie reconstruction.
Why is it important to construct computational models of the brain?
The brain is an extremely complex organ and many convergent
approaches are required to obtain a full understanding of its structure
and function. One way to think about the problem is to consider three
different general goals of research in systems/computational
neuroscience. (1) The first goal is to understand how the brain is
divided into functionally distinct modules (e.g., for vision, memory,
etc.). (2) The second goal, contingent on the first, is to determine the
function of each module. One classical approach for investigating the
function of a brain circuit is to characterize neural responses at a
quantitative computational level that is abstracted away from many of
the specific anatomical and biophysical details of the system. This
helps make tractable a problem that would otherwise seem overwhelmingly
complex. (3) The third goal, contingent on the first two, is to
understand how these specific computations are implemented in neural
circuitry. A byproduct of this model-based approach is that it has many
specific applications, as described above.
Can you briefly explain the function of the parts of the brain examined here?
The human visual system consists of several dozen distinct cortical
visual areas and sub-cortical nuclei, arranged in a network that is both
hierarchical and parallel. Visual information comes into the eye and is
there transduced into nerve impulses. These are sent on to the lateral
geniculate nucleus and then to primary visual cortex (area V1). Area V1
is the largest single processing module in the human brain. Its function
is to represent visual information in a very general form by
decomposing visual stimuli into spatially localized elements. Signals
leaving V1 are distributed to other visual areas, such as V2 and V3.
Although the function of these higher visual areas is not fully
understood, it is believed that they extract relatively more complicated
information about a scene. For example, area V2 is thought to represent
moderately complex features such as angles and curvature, while
high-level areas are thought to represent very complex patterns such as
faces. The encoding model used in our experiment was designed to
describe the function of early visual areas such as V1 and V2, but was
not meant to describe higher visual areas. As one might expect, the
model does a good job of decoding information in early visual areas but
it does not perform as well in higher areas.
Are there any ethical concerns with this type of research?
The current technology for decoding brain activity is relatively
primitive. The computational models are immature, and in order to
construct a model of someone’s visual system they must spend many hours
in a large, stationary magnetic resonance scanner. For this reason it is
unlikely that this technology could be used in practical applications
any time soon. That said, both the technology for measuring brain
activity and the computational models are improving continuously. It is
possible that decoding brain activity could have serious ethical and
privacy implications downstream in, say, the 30-year time frame. As an
analogy, consider the current debates regarding availability of genetic
information. Genetic sequencing is becoming cheaper by the year, and it
will soon be possible for everyone to have their own genome sequenced.
This raises many issues regarding privacy and the accessibility of
individual genetic information. The authors believe strongly that no one
should be subjected to any form of brain-reading process involuntarily,
covertly, or without complete informed consent.
Posts
lives, particularly people with a wide range of ailments. The embryos
create better animal models to study the occurrence of human diseases
and its progression.
