I met with Tim yesterday and we talked for a while about how to make the street view thing less like just a backdrop and more tied into the actual street view thing. And we also talked about transitions between the “normal” version and the portal version. And the idea of something somewhat participatory (like getting invited to join a play date). SO, the video above is sort of an attempt at mocking that up. The way it’s set up there’s a bit of a glitch where you get a peek at the Portal world for a split second first, but I actually sort of like that effect. It’s sort of like Tyler Durden’s subliminal flashes in Fight Club.
Also, thinking about how to tie the output back into the interface, I started looking into Panoramio, the user submitted geotagged photos that can appear on Google earth and sometimes Google Maps. I uploaded the picture of my accordian player, which apparently meets all the acceptance criteria since it now says “This photo is now selected for Google Earth” but I don’t know what the criteria is for acceptance to Google Maps. But I guess at least this is sort one step closer…
I’m starting to feel like the physical/virtual circle is starting to come together a little bit closer now.
And a sort of tangential side note, I read this article today about how magicians like Teller manipulate the human mind and thought it was super relevant to my work, considering a lot of it is about illusion and, as Tim called it, Constructed Confusion. Teller explains a few of the principles magicians employ when they want to alter your perception:
Exploit pattern recognition.
Make the secret a lot more trouble than the trick seems worth.
It’s hard to think critically if you’re laughing.
Keep the trickery outside the frame.
To fool the mind, combine at least two tricks.
Nothing fools you better than the lie you tell yourself.
If you are given a choice, you believe you have acted freely.
I think these are all really useful principles for designing engaging and uncanny hybrid experiences, especially ones that play in that space between the real and imaginary. So much of the experience is actually being completed in the person’s own mind. One of the things I’ve noticed when testing the teleportaling is that even I sometimes forget which object is in which box and reach for the virtual one instead of the real one. It may not be “user friendly” but I think that sort of constructed confusion is really fascinating cause it reveals just how much your subconsciously immersed in the experience.
She looks so much smaller outside in the real world. But I like being able to take her outside with me, though in her current stage she doesn’t do so well in the wind. I’d really like to have a bunch of these characters that I can place around town. Maybe the musical ones also play audio as well. All the characters right now are pretty rough sketches, I’m looking forward to developing the character design quite a bit more too. I’m potentially considering making them based on real people.
A few quick tests of the 2D robot/avatar thing. Basically just my character riding around on a roomba. It’s a pretty rough prototype but I feel like it gets the feeling/idea across pretty well. It’s sort of interesting to experience this autonomous avatar in my real world environment. Unfortunately, the roomba is particularly obsessed with getting close to edges.
Been working on some of the animation bits and getting everything correlated to the specific characters.. I only have 14 different marker objects right now, but I think that might actually be plenty. Keeping track of all of them and their individual values is sort of a lot to deal with right now.
I have also been thinking about the Real World equivalent of these things, and have begun to think about how to modify the real world space, through the portal space. Some initial ideas included projections but I would prefer to avoid using a projector. The current thought is to have large scale versions of these characters as big cardboard cut outs (which I’m calling Avatars) in the space and on wheels, which can be remotely controlled by the small scale markers within the box. I printed one out large yesterday to see what the scale shift would feel like. It’s definitely a different feeling being able to stand next to it as opposed to picking it up with two fingers. Below you can see the test print out of the character. The second pic shows the original character and me for scale.
The other thing I’ve been thinking about in terms of getting input from the real world is designing cameras that could be placed around the real space. I’m thinking I would like to put them inside a pyramid shape and hang them about (an obvious but fitting reference to the illuminati eye) it would also be sweet if they could glow as well.
So now I’m focusing on making my own marker objects, because the little model train people are alright but they’re just not my style. They all look like people from the 20′s. And I want to have custom things. And they’re expensive.
So yesterday I started by trying to sculpt my own little dudes out of fimo clay. This did not turn out as well as I had hoped, and they take forever to make each one. I’m generally pretty ok at sculpting stuff but the scale of these things makes it super hard. They just end up getting squished as I work with them. I think the guitarist turned out alright, but the other two, not so much. They’re ok, but not nearly as cute as I would like.
So I decided to just draw little 2D characters, because I’m much better at that. It’s easier to draw small things than sculpt small things. I like these guys a lot more than the clay ones. They just have so much more character and cuteness.
I also like what happens when they get put into the scene and the white gets keyed out, they just look like these line drawings in the space.
I started playing around with having the markers play related audio. In these tests the musicians play some music from a street carnival in Holland (as described in the geotag from freesound.org).
Good news folks! It works! At least in a basic version. I am able to teleportal both the live video feeds and the tracker data to both boxes. For a while yesterday I was playing with only having the trackers teleportaled with live audio instead of the video since I was having issues before. So I set up a skype account for each box and moved one box to the other side of the room and had some people test it with me. And while I was at it I set up twitter accounts too (@portalleft and @portalright) because every networked thing needs a twitter account, right? Not sure what I’ll do with the twitter accounts yet but you can follow them if you want. CamTwist has a little built in effect to pull in twitter streams so perhaps I’ll let people tweet to them.
Luckily, Bora, Matt, and Sal all talked me out of foregoing the video, because the live feed is sort of what makes it kind of special. So I went back to trying to make it work as planned. I hooked up one box to one of the studio iMacs and was able to get it working on one side, which was a huge relief. (An few interesting things I’ve noticed after seeing a few people play with it: People tend to like stacking/combining things from the two feeds. People also like to make voice-like noises while playing with the objects.)
At least I knew it was possible, and just a hardware issue. For a while though I was worried I would need to use 2 iMacs (which i dont have). But then I went back to the laptops, tweaked some settings, and moved them off the top of the boxes to the aluminum table. I’m not sure what made a bigger difference (settings or location) but since it works I’ll keep doing it like that. Elise had suggested putting the laptops on ice or something the other day, which at the time i thought was just kind of funny but I guess it would actually make a difference. I guess sometimes I forget that computers aren’t magic but are in fact just a pile of physical components that are affected by the physical world like any other real world thing.
So yes. Huge relief it works. Though you’ll notice in the video the tracker graphics move in real time while the video feed is delayed. This is because the feed is via the internets while the tracking data is being transfered via the local wifi. I’m sure there’s a way to make it more in synch But I’m actually ok with this lag and sort of like it.
And now to move on to more content related things for a while I think, mainly defining the objects and the animations to go with them. Although I also need to figure out a better mounting system for the cameras as the velcro is starting to fail me. I would also like to try and make both camera set ups as identical as possible. I’m thinking about taking the cases off the cameras so i can also get the lenses as close to each other as possible to reduce the perspective shift between boxes. Hopefully I don’t break them in the process.
So I built the second touch table / light box thing yesterday and got my second box all readjusted (lights, cams). I also got the remote hub feed thing working so each box is listening for markers from the other box as well.
Unfortunately, once I add this tracker layer my computer basically explodes when I try to include the video feed from the other box. Everything just comes to a screeching halt. So the best I can do is not see the other person’s hand but I see the effect their objects have on the tracked things (like in this video above). Or vise versa. Just not both simultaneously. I think there’s just way too much stuff going on which makes my computer’s brain hurt:
Reactivision is doing real time tracking of the markers, which gets sent to OSCulator. Flash takes tracking data from OSCulator and the second box. Camtwist layers the live screen cap of the Flash app with Webcam #1A. Ustream broadcasts webcam #2A. Meanwhile Camtwist needs to take a second live screen cap of the live Ustream from the second box.
My head sort of hurts just thinking about it all together. The thing about hacking together things that weren’t meant to go together is that everything is sort of bulky and not exactly optimized for performance. Plus I’m doing some weird things that all those apps probably weren’t meant to do and pushing the limits of my computer’s video processor. I think I’m going to give it a try on the big towers in the lab area just to see if it’s even possible, but I really don’t want to have to use the towers for the project. Mainly because I don’t have my own so it makes development a lot harder. Plus they’re big and bulky, so it doesn’t exactly make it portable.
The other thing I’ve been considering is perhaps foregoing the live video feed from the other box. Which sounds like it defeats the whole purpose, but perhaps it’s actually more interesting/mysterious? Because if I can at least see the effect the other person’s having on the space I get the feeling of a co-presence. And perhaps then that’s what differentiates the here from the there. So two people could collaborate locally (via the two holes). And the input from the network appears as a sort of ghostly effect with the added element of live audio.I imagine voice chat wouldn’t explode my computer nearly as bad as the video stream does.
It’s just really frustrating because I feel like I’m SOOOO close to getting it to do what I want and I’m just being held back by the limitations of my computer’s thinking power. I don’t want to be stuck on the technical stuff for so long. I want to move on to the content generation part of things.
Alright. Now we’re getting somewhere! Got my Unibrain Fire-i cameras with 2.1 mm wide angle lens. I really like these cameras. Everything just looks really sweet through the lens. I also rebuilt one of the light boxes. It has a taller base section to accomodate the camera though it actually helps the light diffuse more as well. New features of lightbox v3 include a hole in the back for wires, a door in the side for maintenance, and more solid side surface things. And the added camera of course.
I realized that the reactivision can actually track fingers as well, meaning the video sketch of my finger dragging the interface is quite possible with the new set up. Only thing is that the finger tips need to be really high contrast to show up in the camera. So I taped some little white dots on my finger tips. I actually kind of like the idea that you might have to dip your fingers in some white paint before being able to use the system.
Here’s what the computer “sees” from under the table (after futzing with the camera settings a bit). The “F” is for finger and the +10 shows its tracking the marker. It turns out I’m also able to scale the markers and use smaller versions of them when they’re well lit and I don’t have the layer of plexiglass.
The new light box is about 3 inches taller than the old one so it sticks up past the opening on the side a little bit right now. For the next iteration of the case I’ll have to adjust for that. I also adjusted the cameras higher and designed a new little lighting rig that allows the lights to be angled and closer to the subject. I like that there’s all this space in the front bottom, I’m thinking I can put some speakers in there. It would be great if I could make all the inner pieces fit together nice and neat, which is something Tim had mentioned last time.
And now after all that, finally a test of the tracking actually working fairly well… In this example I’m just tracking the ID and location (x,y) but I can have lots of other data as well including rotation and speed among other things. The object names are just placeholders. I don’t really want to use vector/CG looking graphics in the environment. All the augmented stuff will be shot stop motion or live action so it fits the environment and overall experience. But for now I’m super happy that at least this works.
So thinking about next steps for tomorrow and the rest of this week:
We went to the USC immersive/mixed reality lab thing on Friday. It was interesting to check it since it’s super related to what I’m working on. But at the same time I felt a little underwhelmed. Why do we need more military simulation VR head sets? I guess I just feel like I’ve been seeing these sort of “mixed reality” applications for the past few decades, so it doesn’t seem terribly new. I’m also pretty tired of people referencing the Star Trek holodeck, as if that’s the holy grail of a mixed reality experience.
I think the most interesting part about visiting is the fact that I have more things to compare to my version of mixed reality, which I feel is quite different. So at least it helps in positioning my project. The other big picture sort of thing I took away from the trip was that it seems most of these projects tend to be rather huge and over powering, and I realized I don’t want that sort of scale. I like small things. I’d like my experience to feel more intimate and human scale. And preferably cute.
I had a sort of tangential idea I wanted to test out, which I’m calling the Remote Flash Mob. Basically I was interested in taking the feeds of dancers in 2 different locations and combining them into one space as if they were dancing in the space together. I gathered some volunteers and did a quick test run of the idea last night. Unfortunately for whatever reason I wasn’t able to key out the green in the combined feed, which sort of defeats the purpose a little. But it was still a good experiment to test the fidelity of the network and the synchronicity. In the original idea I had imagined people being in completely different spaces but for my first test I just had them on opposite sides of the room with a projection in the middle so I could see the difference and the time delay. The lack of bandwidth causes a huge lag. And it was actually sort of inconsistent as well. So overall the test didn’t go quite as I had planned but it I thought it was still a useful experiment. I think bits of it may return later on…
I’ll admit I have a bit of an issue with QR codes and AR markers. My main problem with them is that they’re pretty ugly, and primarily designed to be looked at by computers and not people. I remember when I first saw an augmented reality demo. Printed out a little piece of paper, looked at it through my computer, and a little CG guy showed up on the marker. I think this was 2008 or so. I was pretty impressed then, but the novelty wore off pretty quickly. I think it comes back to this idea of immersion. From the Friends on Play podcast I listened to a few months ago there was this little nugget of insight: ”Things that don’t have a direct corollary in the real world kill immersion.” It’s not that everything has to be based on a real world thing exactly. But it’s when things seem especially out of place or particularly “un natural” within the context of the world we’re in that we sort of break from the experience, and I think that’s what happens when I see a lot of AR type things. Even if my “mirror” image (via the webcam) shows a little monster in my hand, when I look down at my hand all I see is a bunch of ugly squares.
Of course this is a mini rant that just leads up to the fact that I think I might have to use markers for my project too, but hopefully it’s in a way that doesn’t break that feeling of immersion as much. Basically after some of the earlier computer vision experiments and after talking about my dilemma with Andrew it seems that the best way to track things in the box is to basically build a mini “touch table” inside the box, where the markers can exist on the bottoms of my objects. This way the main cameras don’t see them, although in the current lighting, you can see the “base” under the object which sort of ruins the compositing. But when lit properly the white sort of gets keyed out.
I’ve been doing some material / scale tests. Unfortunately it seems that the markers need to be fairly large in order to be picked up by the camera, especially when there’s a layer of velum+plexi between the camera and the marker. The camera also needs to be quite a bit a way from the table top. And of course I have the added issue of having lighting on both sides of the “table top” whereas most “reactable” style things have projection on the bottom and the environment is best dark. I don’t mind this. In fact I’m glad that my project can exist in an environment with lights. (I’m a little tired of looking at dark silhouettes in front of projections..) It just means there’s another set of issues to be dealt with.
So this sort of means I’ll be rebuilding my light boxes to accomodate for more space between the “table” and the camera. Camera wise, I think I’ll be ordering the unibrain firewire cameras with the interchangeable lens. I got one of the fire-i web cams to play with first, thinking i might be able to use it for the “main” cameras. But the color and image quality just isn’t that great which means it doesn’t key out very cleanly. But as a camera that’s suppose to be an eye for a computer it’s pretty great, plus i can get the version with a remote ccd, making the camera footprint smaller. I can also control camera exposure and all those settings when I’m within the reactivision software. I’m learning a lot about cameras. Anyways, the video below is just me proving to myself I can use the fiducials…