Filthy RIA

Adobe Pixel Bender - First test

christian.giordano — Mon, 15 Dec 2008 18:13:00 GMT

With the release of Flash Player 10, Adobe introduced many innovations the Developer can take advantage from and the final user will enjoy. If you are a RIA fan, you know that big part is done by the graphic engine of the client. You might not know though, that an important part of the work is related to image processing. In facts, many fancy effects, like Apple dock Genie effect or even a simple spring effect, requires the creation, and eventual manipulation, of bitmaps on the fly.

From Flash 8, the developers have already access to some API which were already pretty low level (matrix filters, displacement map). Unfortunately few people took advantage of those, but when that happened, the results were impressive.

With CS4, Adobe introduced in different products (Flash, PhotoShop and AfterEffect) a scripting language for image processing called Pixel Bender. The script is processed at a very low level and despite in PhotoShop and AfterEffects it is processed by the GPU (with incredible performances), in Flash it is still processed by the processor and probably for this reason some functionalities are not available when deploying for the web. Another good aspect of the usage of PixelBender with Flash Player is that you can run scripts asynchronously taking advantages of multi-core architectures. A good example could be audio processing, but I'm sure that soon or later will be used for also other purposes.

As first test, I ported some scripts I use when doing computer vision to erode or dilate white spots in a black and white image. The scripts, as you can see, are quite approximate. This also to optimize peformances, but the speed how Flash processes this relatively big image is quite impressive. Increasing the size of dilation or erosion simply makes the same filter running more times. Right click to view or download the source.

I might look to Alchemy soon. So, stay tuned!

Time based 3D Charts

christian.giordano — Mon, 27 Oct 2008 10:38:00 GMT

Few months ago I have been researching with some colleagues new ways of visualizing data. Infovis has been a theme I've always been interested in because it is a field where rich interactions can be of real help on a functional level. The most worthy piece I did in the past is probably Tagged Colors. One of the aims of the research was to that the charts should be able to manage time browsing, so I thought straight away how common charts could handle it. I am not a big fan of pie charts but when imagining it in a 3D environment, it got more interesting. Of course I had to show to the others what I had in mind and for this reason I picked a tool I was pretty familiar with, Processsing. Processing has been created for rapid prototyping and it is especially useful if you need simple drawing API (lines, points, polygons), of course you can do much more complex things with it.

Below you can see the result after few hours with Processing and Eclipse.

Time based Pie and Bar Chart from nuthinking on Vimeo.

Future of browser's plugins

christian.giordano — Sat, 04 Oct 2008 12:05:00 GMT

With the arrival of new generation browsers and HTML 5 (which includes enhanced graphics and multimedia capabilities) people might start thinking that it is a big threat for browser plugins like Flash and thus for developers who live from those technologies. I will try to explain why it is not as it might look like.

First of all let's talk about Chrome, the choice by Google is clear, let's create a new browser using a mature and popular HTML rendering, so without reinventing the wheel and without creating hassles to UI developers, but focusing on the user experience. At the end Google got popular because of that, their search engine was the favorite from the users because their website and their queries' results were the fastest to load and the results were the most related, because better performance means better user experience. With Chrome they did some geek stuff behind, that's true, but from a user perspective you can simply say that you will be able to perform javascript faster than the others, you will have less memory problems (especially useful for large javascript applications) and you will have some interesting features like dynamic home and incognito browsing.

So if javascript and memory management will improve, and if HTML 5 will bring multimedia and enhanced graphic, why would you still develop RIA plugin based applications?

1) Expressiveness: Plugins will always be ahead in terms of features, this is because they are not restricted to standards so they don't have to wait anybody before implementing new functionalities. At the moment, for instance, with Flash you have 3D, access to bytes (there are already ftp and MySQL libraries), usage of webcams, last, but not least, good integration with tools creative people use. The web is not only text or RSS and if you want to compete with others, is better you try to differentiate yourself using at its best the newest feature available.

2) Performances: You can differentiate yourself also providing the most reactive experience and if you use Flash with its optimized binary communications and its fastest rendering engine, you are definitely at a good point.

3) Development environment: Using Object Oriented and strongly typed languages like ActionScript 3.0, Java or C# allows you to write more reusable code and to architect better large scale applications. Pre-compiling also helps your IDE to better support your writing. For this reason Google uses their web toolkit for their JavaScript based applications (where you basically write the application in pure Java), and other solutions like Objective-J and Script# are getting more popular. It's clear that new browsers will permit you to run downgraded RIA applications compare to the plugin based ones, but how these will be written? I'm sure more similar to how plugin based, and desktop applications in general, are, so no worries if you are a Flash developer.

4) No cross-browser/platform issues: The plugin vendor is responsible for making your application perform at the same way on different browsers and platforms. And most of Adobe clients (the developers who buy theirs IDEs) choose Flash to not to have to deal with those issues. For this reason when asked about Chrome, Adobe said that they are happy that there is a new browser in the market because of their nature of being cross-browser/platform. In the case of Chrome, since it is based on WebKit, doesn't seem a big advantage but, in general, it is clear that harder is the war across browsers and platforms, better it is for Adobe and I am sure Microsoft won't get so much along with Mozilla, Google or Apple.

The bigger threat for browser plugins I reckon is instead mobile browsers. In fact it might be very challenging to provide rich UI experiences on so many, very different and not powerful devices. As Adobe admits, in the future millions of people will have access for the first time to the Internet from a mobile skipping completely the PC step and mobile users will be the first target for websites, so web designers will have to design first for mobile and then eventually for PC. Interestingly enough we are seeing native mobile applications that are basically micro-websites, so maybe the GUI will get simpler but it will be customized based on the device you are using and taking advantage of its features. It is true that Adobe hasn't so much control on the browsers and platforms, but it has monopoly on the content generated by designers, so until that won't change I don't think Adobe will have many problems on being supported by browsers and platforms companies, and Flash plugin will stay the favorite.

To know about the future of Photoshop, After Effects and Flash IDE, stay tuned!

Multi-touch Flash application controlled by iPhone

christian.giordano — Sun, 21 Sep 2008 07:39:00 GMT

As mentioned on my previous post, my next experiment would have been connecting the iPhone to a Flash application. So here it is:

iPhone 2 Flash from nuthinking on Vimeo.

Design details

I managed to connect to the iPhone a couple of weeks ago but I wanted to create a decent demo to better show the technical achievement. Multi-touch interactions have still a long way to go and so far it's quite hard to see it useful for contexts different than editing, and of course multi-user experiences. In this experiment the user is able just to modify size and orientation of a carousel and its scrollbar, nothing more a part of course being able to scroll the carousel. I'm definitely interested on seeing how multi-touch interactions will be applied to standard UI components, as I showed also with my previous experiments.

Technical details

To create a bridge from iPhone to a Flash application I needed to use an iPhone application which would connect to a socket server the Flash application would receive data from. After I could try the efficiency of Multitouch Framework, I used their iPhone application and their framework for the Mac application, then I imported some classes from OpenFrameworks' addons to create the socket server (the same I used on one of my previous experiment), that's it. I would like to share though the most interesting parts of the process.

As you can see, after the finger starts to move a delay starts growing. This is because at any finger movement the iPhone sends a package of information to the application via network, and these packages are sent very frequently so that the network starts accumulating them. To solve a problem like this, you should avoid to send information about small movements. I implemented this on my mac application, but unfortunately the problem seems to be in the root, the iPhone application. The iPhone application seems to send to their framework all the information, without filtering them, the framework, in facts, returns all the events in basically the same way you would get them in an iPhone application, it would be nice to have a tolerance, which you can modify, for small movements so that you can optimize the communication, as I did on my Mac app. The only annoying part of this is that you have to identify the finger, thing that doesn't happen by default with iPhone SDK, but it's very easy to do by the way.

Other thing I wanted to mention is that despite the idea of using Multitouch Framework was to test multi-touch applications quickly, thus just using the iPhone as input device, I had to find an even more portable way to test it. In facts the Mac I usually use to write ActionScript doesn't have Leopard and thus is not the one I use for Mac development (with Objective-C 2.0), so I wanted to be able to test it without anything else than the usual input information a Flash application can give. The solution was pretty easy and combined perfectly with the controllers I was writing. I emulated the finger press just double clicking with the mouse, this would create a touch view which I can then drag or removing with another double click. Precisely the application can easily switch between the socket multi-touch controller, to the mouse emulated one. Here is a capture of this modality:

Multitouch testing in Flash from nuthinking on Vimeo.

When I started to test the iPhone as input device for network connected applications I saw instantly as obstacle the fact that the iphone can only track 5 fingers and that its surface is very small, comparing to common multi-touch surfaces. At the moment my emulated fingers can be dragged only one at the time, but it should be pretty easy to group some of them to move them together, in fact, as shown on my previous experiments, some interactions might required it and so it could be pretty handy to add this feature.

What's next? Multitouch framework is evolving quite rapidly, so I can't exclude I will experiment more with it soon, but at the moment I'm trying to be focused on an iPhone game, tilt based, I started designing and developing.

Stay tuned then!

Beta testing Multi-Touch Framework for Mac OS X

christian.giordano — Sat, 06 Sep 2008 07:51:00 GMT

Around two months ago some guys from The Media Computing Group at RWTH Aachen University started working on an interesting framework to develop multi-touch applications with Mac OS X. The idea is to develop applications which input could be easily switched between iPhone (or iPod-Touch), camera (FTIR / DI) or a TUIO server application.

The main problem is probably the fact that often the final set for a multi-touch installation is not very portable or replicable (ie. Microsof Surface, which is big, expensive and very rare). So you can imagine how problematic it could be when the development team is composed by more than a person. MultiTouch Framework tries to address it and I have been lucky enough to be accepted in the beta program. Lucky because Apple refused to add the application, which allows to use as input the iPhone, on the AppStore, but the guys were so kind enough to add the allowed developers in the provisioning file. So yesterday I received the required files and today in no time I was already testing the framework!

A gem from the release notes I would like to mention is: "We donʻt have a NDA, so you may talk and write about the framework if you like". It might be because the legal implications of having an NDA, but I like to think that they are pro openness, at the contrary of Apple.

The only input which can be used at the moment is the iPhone which communicates through the network to the Mac. They recommended to use the Mac as wireless spot and connect the device straight to it, but in my small wireless network at home it works pretty well also without this trick, and I reckon this might be enough for most of the tests.

The main problem of using an iPhone to test multi-touch applications is firstly the size. We know that in general these kind of applications are multi-user and are showed in big surfaces. As you can imagine the surface of the iphone is pretty small, and it could be quite hard to test an application on a 1:10, or even smaller, screen. Other issue is that iPhone handles maximum 5 fingers. It's true that they might be enough for most of the needs, but it can still be a limitation.

My next step will be to write a mac app which can communicate to Flash via binary socket so that I can use the iPhone as multi-touch input for flash applications, using the same technique I used here.

Using C++ with Objective-C (and iPhone SDK)

christian.giordano — Wed, 03 Sep 2008 06:08:00 GMT

Here a small tip which makes sense but wasn't necessary obvious. You can have your C++ class both as .h file plus .cpp file but also as .h file plush .mm file. The important part is that the Objective-C class which includes it NEEDS to have the .mm extension, not the default .m. Otherwise you get C++ header syntax errors in the include line in the .m file.

So after having your Objective-C class renamed properly (with .mm extension) you can include your C++ class and using it with the usual C++ syntax.

Give the Foo class:

#ifndef FOO_H_INCLUDED
#define FOO_H_INCLUDED

class Foo
    {
    public:
        void setId(int newId);
        int getId();
    private:
        int myId;
    };

#endif // FOO_H_INCLUDED

You can use it like this:

Foo *foo = new Foo;
foo->setId(16);
NSLog(@"foo id: %i", foo->getId()); // foo id: 16
delete foo;

I am still thinking how much to use Objective-C and how much C++ in my future projects, especially for libraries, using C++ might mean more reusable, accessible, and probably readable, code but I found in the Objective-C dynamism a powerfull tool which can allow me to implement easily design patterns which I use regularly in the ActionScript world.

iPhone fan carousel

christian.giordano — Mon, 01 Sep 2008 06:24:00 GMT

As I said on my previous post, it is pretty fun to play with multi-touch and, after I created some classes to handle it, it got pretty quick too!

In this experiment I recreated a navigation similar to the one use in a friend's project (fan like). I am pretty impressed how the device is performing (in the currrent demo all the images are animated everytime the fingers move). There is definitely margin for optimization, but still...

I presume next step for me might be to play with OpenGL ES and maybe port some of the stuff I did with OpenGL in the past. Lee is already working on a kind of porting of OF (OpenFrameworks) for iPhone SDK. I am curious to see how that goes. So far all my experiments are done 100% Objective-C, mainly for educational purposes, but I should definitely start looking on using more C++.

Stay tuned!

Scrolling experiments with the iPhone SDK

christian.giordano — Wed, 27 Aug 2008 14:30:00 GMT

Like many people, at least that I know, I am having a better look to the iPhone SDK. Not only because I have an iPhone and an iPod-Touch, but also because I was curious to see which is the Apple way on building applications, especially on mobile devices. In the past I made some experiments with J2ME, PalmOS and Flash Lite; each one with its own pros and cons. I am still on early stages to give a deep opinion about the language Apple uses (Objective-C) and their framework. But so far I have to admit I have been positively impressed. Because NDA I don't think I can say much else about this.

By the way, iPhone brought to us, user interface designers, new potentialities and of course new issues. I am not suprised that features like multi-touch, which is in a very hype moment recently (thanks also to Microsoft Surface and Co.), are used in small and isolated cases. New paradigms need to be made but, especially considering the "natural" approach that Apple had when designing iPhone OS, it gets tricky to decide which will be best.

So in these days where people are trying to understand how those kind of interfaces should behave, I took the liberty to experiment with the scrolling of entire screens as a power user like me could find easier. I know that having two different ways of doing the same thing is not ideal, but first time I saw how to scroll between the previous and the next photo, for instance, I hoped it was a joke. I am still far from the truth but this doesn't mean of course that the current UI behaviors cannot improve using different approaches.

From the video that will follow, you will be able to notice three modalities which can co-exist:

Normal
It is basically how the iPhone does currently, to scroll from a screen to another, you just have to drag it for more of than the half of the screen, in few words the screen that will be seen the most after releasing will be the next one.

Continuous
Why on earth the dragging should have inertia? Without it, if I wanted to drag it a bit I could just drag it slower. After releasing I leave the eventual speed the user set until the screens end.

Power
When two fingers are on the screen, they work as scrollbar thumb. This is a bit a tricky one, especially when you have a long list of screens (50 is probably a discreet number), because can be hard to use. It is also true that some easing could be applied to give a more smoother control, but less reactive. With this method you can scroll from the first to the last one at the speed of light.

It is clear that having 50 screens available at "the speed of light" can bring, especially on mobile devices, ram problems because they should be loaded at the start, but this is just an experiment and these behaviors could be easily apply in less demanding contexts.

It has been fun to play with the multi-touch and see how Apple framework handles them and I am looking forward to see which will be the behaviors that will become standard, at the moment let's enjoy the experiments!

Hacking a webcam for Computer-Vision

christian.giordano — Tue, 01 Jul 2008 13:59:00 GMT

In the previous post I mentioned how critical is to have the clearest images as possible to analyze and for this reason is important to know all the tricks to set up the environment. The most important trick I reckon is knowing when and how to use and InfraRed (IR) camera. In facts, these kind of camera react very differently than our eyes because they ignore some light spectrums. This allows you to get very readable images from contexts you would never expect. Not only IR cameras can be used for night-vision but in Computer-Vision context they can be used to ignore completely the visuals of projectors or normal displays so that you can clearly recognize people interacting with screens.

InfraRed cameras are usually used for security proposes and are usually combined with IR lamps. They also have in general as output a composite video that has then to be converted in digital to be analyzed. For this reason the most common solution is hacking a webcam and replacing its IR filter with something which can block visible light (nowadays there is an incredibly variety of webcams, from the very cheap to the ones with high resolution or with wide angles).

The process is very simple and you can follow it in this tutorial. I was a bit concerned on how to find a developed film with black parts but the solution was easy. If you don’t have it yet somewhere, go to a photo shop, buy a new color roll film for camera, open it so that it gets totally exposed (all the pictures would be white) and then make them develop it (being negative the film returned will be all black), it’s better you let the shop assistant know your aim so that you avoid them to generate the pictures and return you only the film.

The following picture shows a capture from my camera after being hacked. As you can see the images shown in the screen, probably my webmail, is totally ignored and with a simple background subtraction my finger can be identified clearly.

Introduction to Computer-Vision

christian.giordano — Mon, 30 Jun 2008 11:50:00 GMT

You might ask: “how on Earth computer-vision is related to RIA?”, my answer is “as Microsoft Surface is”.

Computer-vision is the technique used to make machines understand the physical environment through images. Used of course a lot by robots but getting more and more common, because of its affordability, as “input device” for computers allowing a more natural interaction (without mouse or keyboard) with the software. The recent boom of computer-vision is in facts due to multi-touch interfaces, like the ones built by Jeff Han and later by Microsoft (Surface and Touch Wall), which are making many users dreaming one day to browse their files like Tom Cruise in Minority Report.

With Surface, Microsoft is trying to create a solution where the computer not only tracks fingers but also items, this probably happens through fiducials presumably just added to the items as sticker (thing easily achievable with reacTIVision).

A big advantage of using cameras as input and projectors as output is indeed that your interactive surface is very scalable and the size is basically limited only by the projection/camera distance and angle.

The images that are retrieved by the camera have to be analyzed depending on the information we want to read from them. External factors like lighting or background can make the process very hard and having a good setup, when possible, is crucial. To analyze the images, general image processing algorithms have to be used, from blur or median filters to dilation and erosion, which can be very useful for instance when looking for connected components (here some fundamentals of Image Processing).

Luckily there are already libraries that hide many complex algorithms from you, but it is good to know how they work and how to better prepare the images for them.
THE C++ library is OpenCV, in the java world JMyron is quite popular.
With higher level front-ends, like Flash, the most common approach is to use a proxy application which analyzes the images and sends the results through socket. An example could be Touchlib.
Said that, since Flash can manage one webcam and allows you to perform low level image processing, it can be already used to create a complete solution, from input to output, by itself and I’m sure BitmapBlogDetection after a couple of bitmap filters can help you doing a multi-touch system in no time, and most important only ActionScript!

For further readings I would recommend to follow my related tag on del.icio.us.