During one of my classes at Carleton we were tasked with creating an interactive installation a few months ago. Though this project is a bit old now ( before the music visualizations ) I figured it might still be interesting to post as many of the lessons learned in this project were used to better optimize following projects.

The group I worked with decided to forgo Processing and use openFrameworks for speed considerations to create an environment that could interacted with by sound, motion, and blob-tracking ( people standing in front ).

The features we ended up implementing were:

• Particles fields that linked to camera-detected motion to move trees, flowers, the rain, and clouds (using optical flow)
• The ability to create more clouds by waving arms in front of screen
• Clouds dissipate and disappear when there is no motion in front of screen leading to the sun getting larger and killing all the trees and flowers. Conversely, as clouds get more numerous the sun gets smaller
• Once a certain number of clouds was reached they would start to rain which would then grow more trees and flowers
• Once there are enough clouds raining clapping or stomping in front of projection will create lightning
• The birds tend to flock towards the tallest person currently standing in front of the screen (using blob-detection)

And technologies/add-ons used in the building of this project:

• Optical flow for motion detection to move rain, birds, trees, and clouds
• Blob-tracking for determining where birds will fly and where grass will part
• Verlet physics used to define movement of trees and flowers
• Flocking algorithms used to control the birds movement
• Basic FFT analysis for determining when someone clapped or stomped to create lightning
• ruiThread for threading classes to better utilize the quad cores of Allan’s machine we used for demonstrations

The following video, uploaded by one of the group members Allan, demonstrates it being displayed at a pre-FITC party we were invited to show at. Though we had some trouble with lighting and sound levels, leading to difficult blob-tracking and sound detection, the motion detection and clarity of screen seemed to work relatively well.

A future presentation may also want to use some sort of pointers to guide users as to how to interact with it as the main difficulty was it not being immediately clear how it worked. Though, perhaps it could also be retooled to be more intuitive.

Group members involved in this project were:
Allan Yong
Anthony Scavarelli
Henri Kuschkovitz
Thomas Breffit

Also a special thanks to our teacher, James Acres, for helping with some, er okay, many of the problems we encountered ;)

Intro:

Recently I was asked to fill in as a “VJ/video jockey” for a local club this past Friday. Dabbling quite a bit with openFrameworks recently it seemed a natural fit to create an interactive experience, with some controls so that I could still claim to be a VJ and not just a programmer! The following motion detection/sound detection visualizer is what I came up with:

A clear video screencast of the project using my friend’s created cello track “Incidence”

The Beginning

While learning OpenFrameworks a while ago, and building an interactive environment project in school, a friend of mine asked to create a visualizer to go along with a cello concert he was playing near Toronto. At the time I was playing with particle and vector fields so using these tools seemed a natural fit. I came up with the following: trying to engage the audience by using their motion to move the falling particles ( which fall from the cellist playing at the top of the 8 foot stage/projection ).

Another Try

Again recently I was asked to create a music visualizer for a local club that would stretch across three rear-projected screens as opposed to just one front projection as above. Noticing how slow the Toronto gig projection was, with maybe only 300 particles, I knew I had a lot of optimizations to do! I decided to stick with particle fields but I also wanted to make sure the sound detection was much clearer, that I utilized some openGL effects for a more interesting and efficient look, and that there were controls built in so that one could control the visualizer during the performance as programming a computer to control everything by the sound is a difficult and expensive task. Also, I feel being part of the live performance in a more visceral sense can allow for greater creativity.

The live performance and setup of the three screens at the club

Features

• camera motion detection is transferred to a vector field which then is transferred to particles motion
• wandering particles ( the big glowing ones ) wander randomly
• ribbons are affected by camera motion
• some basic FFT ( Fast-fourier Analysis ) analysis is done on mic sound to get an overall feel for the volume of the sound to detect beats which affect the size of the ribbons and wandering particles
• colour change is affected by user which then translates smoothly between HSB hues using RGB to HSB and it’s reverse

Control

• ability to create or destroy the “wandering particles”
• ability to make wandering particles spit out other “fireworks” particles
• ability to change hues of background
• ability to toggle on or off the ribbon field

Optimizations

• Threading: This is huge as most modern processors utilize more than one core. Using threading to split up the workload more efficiently helped quite a bit. I used Rui Madeira’s thread classes to extend certain classes ( http://code.google.com/p/ruicode/downloads/list ).  Basically you put all your updating code into the updateThread() class and call updateOnce() during your main class’s applications update to keep everything relativily in sync with main applications set framerate. The classes I put in threads are as follows:
  • RibbonField ( the ribbons in the background )
  • ParticleField ( the “fireworks” particles that flow from the wandering particles )
  • WanderingParticleField
  • opticalFlow
• Optimizing opticalFlow. This includes:
  • putting it into a thread
  • changing the input camera resolution to 160×120 as opposed to 320×240 or 640×480 ( this makes a huge difference as there are less pixels to process )
  • blurring ofxCvImages not only looks cool but also decreases the amount of optical flow points while keeping everything relatively accurate
• Using direct openGL calls whenever possible such as called glBegin( GL_QUAD_STRIP ) to create trails as opposed to one quad at a time or creating my own function.
• Passing an image by reference to all particles that share a certain PNG ( This would be done within the “Field” classes wghen they create the individual particles ).
• Setting a max on the number of particles on the screen so that when I am controlling it I do not add too many and slow things down to a crawl. In this case the max number was especially important on the fireworks particles as they took quite a bit of processing to look the way they do and often were created together in very large clumps.
• FBO’s ( Frame Buffer Object ): using ofxFBOTexture I was able to create several FBO textures to try to lessen communication with GPU. It also provided useful for tinting the background image as one can use ofSetColor before drawing a FBO to tint it.

Equipment

• 3 rear-projection screens ( approximately 8 x 6 feet  )
• 3 projectors at 800×600 resolution each
• 1 Matrox Triple-Head-To-Go to bind all three projectors into one wide ( 2400×600) screen
• 1 MBP 2.4gHz to run everything on stage
• 1 PS3 EyeToy camera
• Another laptop to control laptop on stage near DJ station

Conclusion

This was definitely a great and enjoyable experience getting everything working using so much equipment. I was definitely a bit worried about getting three projectors running this piece smoothly but the optimizations above helped quite a bit! I think it is a simple example of a visualizer but I believe in its simplicity it worked well. The owner of the club liked it and my friend and girlfriend liked playing with during the show so much that I never actually controlled it all night lol. Anyways, I am nowhere close to an expert in openFrameworks, openGL, or C++ but if you have any questions please feel free to throw them in the comments section and I’ll do my best to answer them.

————————————————–

References

• Vector Fields / Particle Fields: http://www.makingthingsmove.org/blog/?cat=5
• Threading: http://code.google.com/p/ruicode/downloads/list
• Video Processing ( in samples folder ): http://www.openframeworks.cc/
• Intro to additive blending and OpenGL graphics: http://www.flight404.com/blog/?cat=4
• Badass cellist: http://www.kirkstarkey.com/
• Supporting Artist Community: http://artengine.ca/

I always have found mathematics and physics interesting …

Source code: Elliptical Orbits
Built with Processing

The fact that abstract equations full of several variables when used in programming can lead to truly beautiful results has always fascinated me ( just try changing any value (x) with sin(x) or cos(x) for example ). In fact using these equations are absolutely necessary when simulating more “realistic” results. The experiment above was a simple experiment ( for an assignment at school ) in creating elliptical orbits using elliptical equations.

float radius = (a * ( 1 - sq(e)))/( 1 + e * cos( angle )); //elliptical equation

The above code coming from the equation obtained from Wolfram MathWorld

where e refers to the “eccentricity” – a parameter that defines the “squished shape” of the ellipse ( a number between 0 and 1 ). An eccentricity of 0 will define a circle whereas an eccentricity of 1 will be so squished as to appear as a line. Go Ahead and play with the code linked above to experiment.

It’s fun to use equations to define motion. Instead of changing the position of an object linearly where each frame adds a discrete value such as one:

float position = position + 1;

changing it to ( for example ):

float position = position + position/sine(position) * 10.0;

can give a unique motion that can be both unexpected and perhaps interesting enough to be different ( I haven’t actually tried this yet haha). Not unlike an artist that sketches several different styles before settling on one in creating a new work it is important to treat this kind of programming as an art as well. Constantly try new things – one line of code can make a huge difference!

————————————————–

References

• Ellipse Mathematics! http://mathworld.wolfram.com/Ellipse.html
• Processing: http://processing.org/

I originally intended this blog to be about Flash and Flash-related posts such as Flex and such; but lately I have not been working with Flash at all. I don’t want to get into the Flash problems debate but I found it was a lot of work to get even the simplest of programs running smoothly. In the meantime, I have worked some co-ops at Fuel Industries and Adobe and continued with my university studies in Interactive Multimedia and Design – one year left!

Actually, after taking a class that touched on using Arduino microprocessors and Processing to create interesting interactive pieces I think I have found a new way to quickly create experimental pieces that involve input that doesn’t necessarily use keyboard and mouse as input. Specifically I have been doing a lot of work with a C++ framework called openFrameworks to create music visualizers for concerts around town here in Ottawa. Over the next week or so I will be posting abut what I have been doing over the last few months – from using Maya MEL script to create a downtown city, to using processing to simulate elliptical orbits, to several openFrameworks projects that use sound to create, hopefully, interesting visuals.

Well I have to run to my girlfriend’s for mother’s day dinner but expect new posts soon!

Well, I have been stuck working on some other things lately (like trying to fix the bottom bracket of my bike :( and am currently still working on my linkages tutorial. Just trying to determine how is the best way to present it (I’m new eh – Canadian for emphasis ;). In the meantime, I am going to give everyone interested in bringing 3D models into Papervision a present:

An air app that allows one to load and edit Collada animated/non-animated models!

This was a project I built in a few weeks between projects, while working as an intern at Fuel Industries. As it seems not all Collada models exported from 3D packages such as Blender, 3DMax, and Autodesk’s Maya load properly, I figured building a robust app that could install on any machine for 3D artists to test out their models before any flash/flex work would be a time saver.

Anyhow, I ran out of time and could not add some features such as undo states, loading other types of models (MD2, 3DX et cetera), and the ability to know why a model would fail to load properly. But I am going to include the entire Flex project so that if anyone wants to add any features to it go right ahead. Please feel free to comment back with the changes you have made and some code so others can build upon it, if you can.

This is the current list of features:

• load a Collada model from a local source
• scale, rotate, or orbit model using mouse controls
• ability to auto-texture
• manual texturing includes ability to turn off and on wireframe or flat colour
• ability to play through animations, frame-by-frame, or as would be expected
• ability to open up a secondary window that shows loaded models Collada .DAE data
• ability to edit and save .DAE data (in XML format).

Hopefully you can find this useful. If you do let me know.

Here is the air app with transparent windows enabled
Here is the air app with transparent windows disabled (runs better on slow machines)

Here is the entire Flex project (unzip and import existing project in Flex Builder)
And some sample Collada models

(And if you don’t have the Air framework installed already you will need it before you can install any Adobe Air applications. Get it here: http://get.adobe.com/air/ )

Enjoy! :)

NOTE: If you are having troubles building the project make sure to check out which version of Flex Builder you are using. I did not update FB3 while building this so the line in the ColladaAirViewer-app.xml is

<application xmlns="http://ns.adobe.com/air/application/1.1">

You may have to change it to:

<application xmlns="http://ns.adobe.com/air/application/1.5">

Hello,

Well, my first blog post ever. Figured it was about time I started posting some actionscript tips I have learned along my flash and flex learning journeys. It is a great way to share knowledge (both for you, the reader, hopefully, and myself as well. This as some of you may give your own take on the subjects). My goal is start with relatively basic tasks, and build up to more complex examples involving Papervision and such – that this blog could help one learn how to code in Flash, as I feel there are fundamentals that are glazed over when it comes to teaching the basics of Flash. The truth is, that all the coolest tricks in Flash go far beyond simple tweens and buttons and right into leveraging actionscript into creating and customizing your own effects. But there are fundamentals that will make yours, and others you may work with in the future, lives easier when working in Flash and Flex projects. I will try my best throughout all my blog posts to make the three following rules a point of focus in all my tutorials:

putting no more code than, maybe “stop();” on the Flash timeline. highly organized structure of all assets used in the creation of projects. object oriented programming (using an event-driven model).

It is important to stay away from the allure of putting code on the timelines as it is harder to change that code later, and Flex itself, will not recognize any timeline code on embedded movieclips (best to get used to it in Flash so there are no nasty surprises in Flex). Proper organization of all flash projects is essential in making everyone’s lives easier when it comes to re-tooling projects or changing things. And if all the coolest effects are programmatic then it would be best to follow, arguably, the cleanest and most easily understandable coding practises – this would be object-oriented programming or OOP for short. And seeing as how Flash and Flex are both built upon the concept of “something” triggering an event that will cause some other code to execute, it would be ideal to push the event-driven model of coding, as well, from the beginning.

I will not go into more detail about what these concepts are precisely at this time, as they can easily googled (here you go); but as I post more tutorials it will be easier to help explain them with examples.

Feel free to comment with some of your own ideas on what Flash and Flex are about. In the meantime I will start work on my first “actual” post about setting up Flash projects and linkages between flash library items (symbols) and actionscript classes that will define their behaviour. Hopefully I will get it up soon :)

Here are a couple of great Flash/Flex learning resources in the meantime:

• Kirupa.com
• Flash and Math
• Lee Brimelowe’s “The Flash Blog” (also covers Flex)

take care everyone, and happy flashing/flexing haha,
~ anthony

p.s. I am still working on tweaking the design of this blog so please bear with me :)