Strange face images

The brain is extremely good at recognizing human faces and is believed to have a special module or part that is responsible for analyzing faces
(Carter, Rita; Frith, Christopher. (1999). Mapping the mind. University of California Press).

This module is obviously very handy, but it can also play nasty tricks with our perception.

Look at the image here to the right. You will most certainly recognize the famous singer and actress Madonna. The photo is upside down, but otherwise it seems normal to you, right?

Well, prepare yourself for a mind-boggler.

The image to the LEFT is is the same image as before, only rotated 180° to be "right". Not a pretty sight, is it? If you don’t believe me, print the image or copy it to a graphics editor and rotate it 180° again.

The image on the RIGHT is the original image for comparison. What a relief! (Original image courtesy of: WireImage.com)

What? How? In the funky image I have taken Madonna’s eyes and mouth and rotated them 180°. In addition I switched the right and left eyes (we are also extremely good at recognizing only eyes).

We are not used to faces being this way and therefore the face recognition module assumes all is in order and even recognizes the face. When we turn it around, the face recognition module instantly sees that something is terribly wrong.

As all of my time now is devoted to developing Spurl.net further, I realized that this book is not going to be finished. At the same time I have a lot of material that I have already written and it won’t do any good lying around on my computer, therefore I’ve decided to publish this material bit-by-bit on the Wetware blog (that has also been neglected due to Spurl).

Take it for what it is – more or less provisional work, unedited and not necessarily in the order it would have been in a completed work. The first chapter follows and other bits and pieces will come in during the next weeks.

There is a brilliant article on “idiot savants” – people that have autism or received serious brain damage so that they do not function in daily life, yet have acquired some incredible special talents. Among the individuals mentioned in the article is Kim Peek, the man who inspired Dustin Hoffman’s character in the Rain man. He knows the text of 7,500 books by heart! Another one is Ellen who can tell the current time to the second without a clock no matter what the season.

While the whole issue is most interesting, three other articles in the issue are worth special mentioning:

1. Anguish and Ethics (p. 10) that talks about how brain science is being used to better understand how humans deal with moral and ethical issues.
2. The quest to find consciousness (p. 32) on efforts to understand how consciousness works – or better yet, what it is. These questions have hitherto been left to philosophers, but are gradually becoming hard science.
3. The science of persuasion (p. 70). Consumer marketing – the persuasion techniques used to get us to buy certain products, elect a stupid president and other ways to make us sit and stand at “their” will.

I think most of you that read this blog will find this issue of Scientific American most amusing – and, no, there is nothing in it for me

Here are just a few interesting bits from the article:

“Take another look at those cables: They snake into the back of the computer and then out again, terminating in a cap on the monkey’s head, where they receive signals from hundreds of electrodes buried in its brain. The monkey is directing the robot with its thoughts.”

- – -

“To test this supposition, Chapin and Nicolelis inserted electrodes into a rat’s brain and began monitoring 46 neurons. They then trained the rat to press a lever to get a drink of water, and used the electrodes to record the pattern of signals the animal produced to move its arm. Then Chapin and Nicolelis disconnected the lever from the water supply, so that pressing the lever did nothing. The rat went on pressing the lever, but now the scientists gave the rat a drink of water when it simply produced the “press lever” command in its brain. After a while, the rat stopped bothering to lift its arm, and just thought about lifting it.” (page 3)

Much of the research is funded by DARPA. Wonder why?

“The brain does more than just move arms and legs — it also sends out complex commands that control muscles in the throat, tongue and mouth, creating speech. It’s conceivable that a computer could learn to recognize those commands before they leave the brain and then translate them into words. “You could imagine thinking about talking and having it projected into a room 2,000 miles away,” says Craig Henriquez. “I don’t see that that will be a problem. It’s very, very possible.”" (page 4)

But the current interface technique is – well – not entirely convenient:

“Take the equipment itself. Wires sprout from the implants in a monkey’s head and are jacked into a big signal processor, which in turn is plugged into a computer, which in turn is connected by cables to a robot arm.” (page 5)

While this sort of thing gives some people the creeps, several people have been experimenting with similar animal brained machinery, and we’ll no doubt see plenty more. Following are a few examples.

- – -

One of the debut articles for the new PLoS Biology magazine is “Learning to Control a Brain–Machine Interface for Reaching and Grasping by Primates“. If you prefer the shorter popular science version, Wired also had an article on it this week.

In short it is about a monkey that learned to control a robotic arm using thought alone. The monkey’s brain activity is picked up by an implanted electrode. In fact the research team, led by Dr. Miguel Nicolelis at Duke University had reported this achievement some three years ago. The breakthrough is that the monkey has now learned to move the robotic arm using thought alone, while three years ago he kept moving his real arm also the way he had originally been taught to control the robot arm.

- – -

The Potter Lab at Georgia Tech, responsible for the aforementioned robot artist has also done other similar projects, like the rat-brained robot mentioned in this Technology Review article.

- – -

And scientists at Northwestern University, Chicago – took a part of a lamprey’s brain and connected it to a robot, making it control the robot to move towards a light source. If you ask me the lamprey actually looks less creepy this way than under normal circumstances (I will not even discuss its diet).

- – -

The researchers doing these experiments usually refer to how it could help paralyzed patients. In a previous Wetware article, we discussed the case of quadriplegic Jim Jatich regained rudimentary control over his paralyzed arm aided by a brain interface. Breakthroughs in the field could certainly help countless paralyzed patients.

But what other use could it possibly have? Here are a few ideas:

• Using real neural tissue as a part of computer systems: If we learn to create intelligent interfaces between wetware and hardware, biological parts could be integrated with computer systems to provide functionality that has proven hard using traditional hardware and software. Think e.g. about using the visual interpretation from an animal’s brain as a layer in a computer vision system, or the nose and odor facilities of a dog’s brain to create devices for sniffing bombs or drugs.
• Using rats or other creatures with implants to aid search and rescue work or – less morally – as spying equipment. Implants would help both controlling them and to receive the feedback (smell, sight, sound). What the heck, I bet the military would love to have a whole army of such soldiers – armed and dangerous.
• Using “thought recognition” to interface with our computers, instead of a keyboard, mouse or voice commands.

At which point did you say: “Ok, hold it – we’re not going down that route!”. Well, whether you like it or not, we are. We will see this sort of gadgets as research continues. Some of them will in a few decades time seem no less moral than in vitro fertilization or farming chicken for that matter.

But some of them raise somehow deeper ethical questions. An army of remote controlled animals in warfare – that doesn’t seem right (killing each other in the first place doesn’t seem right but that’s another matter). And where will it stop? Once we start fiddling with the human brain, one must be careful about the security glitches you’ve got!

Personally I’m all for carefully exploring this route – in any case it cannot be stopped, any more than the production of nuclear bombs, human cloning or biological weapons research. And this for one will at least bring about some positive things to be sure.

So the question is: Which will we see first, a human-brained robot or a robot-brained human

These remarkable results are thanks to a system called vOICe (where the three middle letters stand for “Oh I See”), developed by Dr Peter Meijer, a senior scientist at Philips Research Laboratories in the Netherlands.

The method sounds simple enough. Currently “brighter areas sound louder [and] height is indicated by pitch” turning each camera pixel into a specific sound. A stream of pixels then makes up the whole image with a total resolution of up to several thousand pixels. That is not much, compared to the fact that even most rudimentary digital cameras have a few hundred thousand pixels. But as you can see in the examples below, it is surely enough to make out the content of the image.

Color is not represented in the vOICe soundscape, making the “soundscapes” grayscale images. A user can however activate a color identifier that speaks out color names.

Here are two examples of images turned into vOICe soundscapes. Even with an “untrained ear”, the sounds make perfect sense when compared to the images, although I would not claim I could make my way to the bathroom using this information alone. Try them:

44K WAV soundscape	88K WAV soundscape
Images and sound courtesy of the Seeing With Sound website

On the project website, you can even try loading your own images and turning them into soundscapes. You MUST try some of the examples there, they are quite amazing.

What vOICe effectively does is that it uses the user’s auditory input instead of the damaged optical input to get the visual information to the brain. I wonder if it is then processed in the same parts of the brain as it would have had it been “normal” vision?

This method has some obvious advantages over retinal and brain implants, in that it is non-intrusive. An article in Wired a couple of years ago this year discussed another method where the nerves in a user’s tongue was used to receive visual information. It is however not likely for either of these methods that the resolution will ever get close to the “real” thing. Bionic eyes (the implants) are much more likely to get to that level at some point, even though certainly they are not yet. This in no way undermines the importance and brilliance of the vOICe system and other “optical nerve bypassing” methods.

Thanks for the tip Magga.

At first I was a little skeptical of the importance of language in thinking, but the more I read and think about it, the more I believe in its central role in human thinking. Words are almost definitely used as references to memories. Try this for an example: Think of a dog. How do you do that without thinking of the word “dog”? Can you imagine? Of course a picture of a dog, hearing a bark or encountering a real live dog would get you there, but in the absence of those it is hard to imagine how you would call a dog to your mind without using the word “dog” as a reference. And what do we do to memorize things? We utter them over and over as if to imprint them thoroughly into our heads.

By nature we are extremely forgetful (for those of you who know me, I welcome the jokes . One of our excellences, as stated by Daniel Dennett lies in how “human beings offload as much of their minds as possible into the world”. By this he’s referring to how we write text and other scribbling or otherwise reminding means (tie a knot to the finger) to make references that we can then use to reload memories into our brain when needed.

A brief look at your own daily live may help to show how true this is. How much of your daily tasks would you be able to complete if you had to memorize every single thing? An organizer is an invaluable thing in many people’s lives. Even those of us that are not so organized write PostIt notes and stick them on things; write shopping lists instead of memorizing as we become aware of things that need to be bought; and scribble memo points about things that need to be done when we show up at work the next day.

We surely use text a lot to communicate with others, but we use it even more just to communicate with ourselves. It is fair to assume that this is due to the fact that the brain cannot keep track of all these things, at least not in a way that makes it accessible whenever we need it.

After noticing for the ten-thousandth time that I had in the morning forgotten the idea from the night before that was so brilliant it was simply unforgettable, I started carrying a notebook wherever I go and make an effort to write down things that I feel worth noting. I use paper, as nothing quite matches it in terms of usability and handiness. Once in a while I then take the last batch of notes, type them into a computer and arrange them to fit the different projects and subjects I’m working on. This makes them easily searchable which often comes in handy.

Despite several efforts to find good, specialized software to do this, I still haven’t found one that I like. This is probably because of my preconceptions of how such software should function and none of the systems I’ve tried meets them. The top wanted features are:

- Ability to make all notes searchable in one go while…
- notes are categorized and…
- every note can be put in multiple categories.
- Crossreferences can be made between notes.
- Everything is stored centrally on the net, prefereably accessible in a browser.

There are loads of other nice-to-have features, but these would pretty much do it for me. I have high hopes for Microsoft’s OneNote but there is still some wait.

In one way we can look at notes as an extension to our memory. But what do we really want to remember? I’ve seen several projects that intend to record everything we see and do to make it available for recalling later. The more futuristic ideas suggest a direct brain link, but tests so far are more camcorder / dictaphone / computer log combinations, like this project by MIT student Sunil Vemuri. Another interesting discussion of such system can be found in David Gelernter’s article “Tapping into the Beam” in The Next Fifty Years. There David Gelernter talks about the information beam, a logging feature that could be compared to the journal feature found in Microsoft Outlook, but extends it to cover our entire life.

Brain interfaces are obviously still quite far away and the recording solution puts demands on search abilities that out-Google anything even at the horizon of invention yet. Hence we do have to select the best bits and note them, making the daily note-taking at most a few kilobytes, rather than the 150 million Gbytes of data per minute, needed to store our entire experience! Let’s admit that most of it is junk anyway.

Taking notes may become easier with new technologies, but the selection, sorting and storing process is likely to be manual and limited rather than an unlimited store-all method.