Motion control & the simulation of touch

[Michaël Samyn]

We're currently prototyping two games that we're considering the Playstation Move controller for as main input device. But for one of these games, it is starting to feel inappropriate (in my mind, at least).

The interaction in the latter game is about touching. And you don't really have an avatar. The idea is more that you are actually touching the game world with your hands. A motion control interface seems appropriate in so far as it involves actual physical movement of the player's body. But it's the lack of feedback that bothers me.

With motion control you're basically waving your hands in the air, in a vacuum as a mater of speaking. The only way you know that you're actually doing something in the fictional world is by carefully observing the depiction on the screen.

In a way, this is also true for the traditional gamepad controller. Because the motions that you make are so minute that it's hard to feel any sort of correlation. I think this is one of the reasons why console games are almost always third person. Controlling a character by doing small symbolic actions, feels more acceptable than actually imagining doing the character's gestures through such a limited device. It's more like giving commands than direct control.

There's only two hardware control systems I can think of that allow for and encourage a feeling of direct control. The mouse and the touch screen.

The mouse has always felt like an extension of the arm to me. Through the mouse, my arm is extended into the virtual world. Clicking feels like tapping something with my finger, hovering over like touching with my hand, dragging like rubbing, and so on. Mouse interaction feels very physical and sensual to me. The interaction with the mouse object works as a substitute for interacting with a virtual object, thanks to the physicality of the mouse.

The touch screen is a more recent discovery. But it has an issue. While the mouse becomes almost invisible after a while, and it feels like my hand has entered the virtual space where it can actually touch things, the screen never goes away and remains impenetrable. I didn't have this problem with the iPod touch because the extreme small size of the screen gives one plenty of other things to worry about, plus the motions you make on such a small screen are very modest. But on the iPad, rubbing your finger on the larger screen continuously reminds you of the object you're interacting with. The glass is not very nice to touch (unlike for instance the plastic of a mouse), plus the constant awareness of your hand touching the hardware prevents the "virtualisation of the hand" that is so important to me when using the mouse.

That being said, the direct correspondence between the position of your finger and the element on the screen, does help believability. But only for objects that are somewhat similar to the screen: flat, two dimensional objects such as game boards and books.


I guess a touch screen is similar to a motion control system like Wii, Move and Kinect: they are all gesture based interfaces. Maybe that's what's wrong with them. The gesture is very physical. And as such it demands a physical response. But that physical response doesn't exist. Vibration in the controller can help this a little but. The iPad for instance vibrates when certain sounds are played. That feels very nice. And the Wii has a vibrator built in. I guess the Move does too. It's still a long ways off from actual force feedback that corresponds with the visual reality on the screen. But it's something. Kinect however has nothing of the kind. It's a purely visual system. And while it allows you to use your entire body for input, it gives you zero tactile feedback.


Anyway, I'd like to hear your thoughts because I want to know how subjective all this is. Does the mouse also feel like the most "sensual" input device for computers to you? Have you had experiences of "simulation of touch" with other devices?

[Thomas]

I also feel that the lack of feedback is a big negative on the modern motion controls. I think game controller and mouse is much better since (as you said) one can imagine the act of touching. But the motion controls there is no such thing and the bubble bursts very quickly.

Much better is proper haptic devices. I have been part of porting Penumbra to the Novint falcon and tried two different haptic devices when doing this. A haptic device is basically a robotic arm that stop your movements when a virtual object is in the way. The most immersive way of interacting like this is when the device is a below a screen and you where 3d glasses. The screen + 3d will then show the in-game pointer a (ball basically) in the actual world position behind the screen and you get proper feedback when interacting with objects. The feeling when doing this is extremely real and I find that I perceive the object not like a virtual one, but a physical one very quickly and easily. (this system is actually used when prototyping mobile phones)

One can also have this input device beside the computer and while you do not have the 1:1 mapping between the actual position of the hand and the in-game pointer, you get into it fairly quickly. The sensation of just touching a surface and being able to sense the texture is extremely exciting. You can feel if a surface is rough, slippery, spongy, etc.

However, there are some problems with this approach. First up, the adding a third (z) axis that goes into the screen is not always easy to handle. it is actually one of those things where 3D glasses really comes in handy, because it is hard to judge distances on a 2D screen. Also, this ball tends to get stuck all of the place. For example, in Penumbra you can feel the insides of a locker and then push a crate in front of, it trapping your cursor inside. This is very immersion breaking and the game has to have a sort of "move through walls"-mode in order to compensate.

I think is haptics is factors more exciting then anything on consoles, but it still has plenty of problems. I actually prefer having simple, coherent controls where some parts are left to the player imagination.

[Michaël Samyn]

I wouldn't want to be strapped into some kind of weird exoskeleton to play a game!

So I agree: imagination is crucial. But you have to give the imagination something. You have to stimulate it. I also believe imagination happens on many levels. I believe our hands have a memory, much like our nose does. And it is such memories that we as designers rely on to connect our fictional world to the imagination of the player. The visual can recall memories of touch. But even a little bit of actual touch (even if it's not realistic) can help a lot, in my experience.

I wonder how hard it would be to simulate something like the resistance you describe, Thomas, "in the air", to complement to motion controllers. Perhaps through a little engine in the Move controller or the WiiMote that can fly, pushing your hand in a certain direction? Sounds dangerous (what if you accidentally let go?).

Haptic response in touch screens seems more feasible. You're already touching the surface. Now the surface needs to either change shape or feel different (through tiny electrical charges perhaps?).

[Thomas]

There is a touch-pad for some laptops where you basically scroll a surface. There is a bit of plastic that wraps around in all directions in some manner and you need to physically move it. It would seem possible to add some sort of feedback there.

Propeller would be awesome Then you get a helicopter along with it I guess that simply having some kind of vibration might help though. But that is what they already have right?

The one thing I like most about motion controls is that it is more intuitive and inviting to an audience that has never played games. Showing a tennis game followed by a clip of something waving a a controller in the air is something that everybody gets. I think this has been the main draw of the Wii: it makes advertisement so much simpler. As we do want to expand the market, this is a good thing though and this is my biggest interest in the current motion controls.

[Andrew Tremblay]

The point is we've been fiddling around with vibrators for far too long, people are tired of them and are looking for the next big thing. Researchers, inventors, venture capitalists, all people really, are aware of the interest-hindering effect that full haptic suits and tacta vests have in their preparatory use. People don't want to take time to put it on so people don't want to buy it. The first company that figures out how to get people who are interested in more immersion what they want in the way they want it will be very, very rich, and that's making a lot of people flock to the area.

Haptic feedback is a bit of a research interest of mine, especially in the perception of textured surfaces. Take a look at what the Japanese are doing:
http://www.diginfo.tv/2010/05/21/10-0073-r-en.php
Actual texture differentiation. And this isn't technology that's ten years away, people are predicting this on the next iPhone and built into mousepads. There's a potential patent conflict with an American company, though, so we might not see it over here very soon.

What we'll be getting next is the 3D equivalent of a mouse, maybe just some finger clips.
The closest you can get to force feedback at the moment is the Novint. Oh look, it has a pistol attachment!

[Thomas]

Haptic feedback is a bit of a research interest of mine, especially in the perception of textured surfaces. Take a look at what the Japanese are doing:
http://www.diginfo.tv/2010/05/21/10-0073-r-en.php

Oh, that seems cool! I wonder at what fidelity this is. Even with novint, that is pretty rough in movement, surfaces have a very detailed feel.

[Michaël Samyn]

None of these technologies matter until they're available in the market, and successful too. As software makers, we need to work with the hardware that is out there.

I agree with Thomas that the accessibility of the motion controller definitely helps to broaden the audience. And I'm quite disappointed that Nintendo didn't do anything with its new audience. The Wii could have been an opening to the new interactive entertainment that we're all working on. Instead it became a fitness device.

The Playstation Move is by its nature limited to the Playstation console, a gamers' console. So the accessibility is less relevant here. Until Sony starts marketing the Move before the Playstation, to a new audience. But I don't see that happening any time soon.

[Michaël Samyn]

Anyway, can I conclude from this that we all agree that the mouse is a better device for simulating touch than any other readily available input device?

Does anyone know of any software tricks to increase the imagined sensation of touch? Because that's ultimately the only thing I'm interested in: the imagination of the player. I don't need people to actually feel something if they can imagine it.

[Thomas]

I think you can simulate touch a bit through the right kind of combination with audio, visuals and interaction.

Windowsill has certain parts where I imagine a surface being slippery, etc. There is a feedback loop between output and input that can make this very convincing.

[Kjell]

Coincidentally I recently did some research in this area ( using PlayStation®Move ) The most powerful solution I could find was ( simply said ) a combination of the following elements.

- During collision calculate the penetration direction + depth and move the camera relative to the inverse of this vector*
- During collision set the rumble intensity relative to the penetration delta + depth to represent the applied pressure.
+ Upon collision enable rumble for a single frame to let the player know collision has occurred ( state switch ).

* Similar to when you're pushing into a wall, causing your body to move / tilt into the opposite direction.

[Michaël Samyn]

Thanks for the tips!
I'll make sure to try this out as soon as I get my hands on a Move controller.

By the way, does anyone know of a way to use a Move controller with Unity without going through Sony?
(we are registered developers, but I'd like to experiment with this first before we order a devkit)

[QXD-me]

I think you can simulate touch a bit through the right kind of combination with audio, visuals and interaction.

Windowsill has certain parts where I imagine a surface being slippery, etc. There is a feedback loop between output and input that can make this very convincing.

Similar to this is the screen shaking effect that's in quite a few games (Torchlight is the one that springs to mind for me) which helps you feel the force of some impact. Perhaps this wouldn't scale too well to other sensations though.


Also, I wouldn't say that the mouse is the most immersive input to me. I actually find the gamepad more so (generally), although that's probably because I grew up on console games so using a mouse and keyboard isn't as comfortable for me. I'd imagine most people would prefer the mouse though, especially non-gamers.

[Michaël Samyn]

Also, I wouldn't say that the mouse is the most immersive input to me.

My question is not about immersiveness. It's about the simulation of touch. Not the same thing. For me, a gamepad is also fairly ideal in terms of immersion. Because of its ability to disappear. It helps me step into the world of the game, imagine being there. But that's not what I am after with this particular game. I would like to simulate in some way a physical sensation of touch, direct touch, with your own hands (not those of an avatar). And the mouse works better for me in this respect than any other common controller.

[QXD-me]

I see. Then yeah, I guess mouse would be better, though lack of the rumble does put it down quite a bit compared to gamepads in certain cases. (I think I was amalgamating ideas in my head and got lost as to the purpose along the way.)

[Michaël Samyn]

There are mice that have rumble feedback. They're just not very common.

Thinking some more about this, I was wondering if representing the hands on the screen in some way would help the player imagine that they are touching something.

There's still the issue of the pose the hands are in. If you're using a mouse, your hand is open and your fingers can "do the talking". I have always like the pointing finger cursor that you see when you hover over a link in a web browser. The image is an exact representation of the pose my hand is in. This helps imagining touch.

This is a potential problem for the Wii and Move controllers because they require that you close your hand to hold the device. A closed hand is a fist. And a fist is not the best pose for touching. And representing the hand as open on screen while it is closed in real life causes an undesirable disconnection. The Kinect could be more suitable for this sort of hand mimicry (providing it is sensitive enough to pick up on hand poses).