During one of my classes at Carleton we were tasked with creating an interactive installation a few months ago. Though this project is a bit old now ( before the music visualizations ) I figured it might still be interesting to post as many of the lessons learned in this project were used to better optimize following projects.

The group I worked with decided to forgo Processing and use openFrameworks for speed considerations to create an environment that could interacted with by sound, motion, and blob-tracking ( people standing in front ).

The features we ended up implementing were:

• Particles fields that linked to camera-detected motion to move trees, flowers, the rain, and clouds (using optical flow)
• The ability to create more clouds by waving arms in front of screen
• Clouds dissipate and disappear when there is no motion in front of screen leading to the sun getting larger and killing all the trees and flowers. Conversely, as clouds get more numerous the sun gets smaller
• Once a certain number of clouds was reached they would start to rain which would then grow more trees and flowers
• Once there are enough clouds raining clapping or stomping in front of projection will create lightning
• The birds tend to flock towards the tallest person currently standing in front of the screen (using blob-detection)

And technologies/add-ons used in the building of this project:

• Optical flow for motion detection to move rain, birds, trees, and clouds
• Blob-tracking for determining where birds will fly and where grass will part
• Verlet physics used to define movement of trees and flowers
• Flocking algorithms used to control the birds movement
• Basic FFT analysis for determining when someone clapped or stomped to create lightning
• ruiThread for threading classes to better utilize the quad cores of Allan’s machine we used for demonstrations

The following video, uploaded by one of the group members Allan, demonstrates it being displayed at a pre-FITC party we were invited to show at. Though we had some trouble with lighting and sound levels, leading to difficult blob-tracking and sound detection, the motion detection and clarity of screen seemed to work relatively well.

A future presentation may also want to use some sort of pointers to guide users as to how to interact with it as the main difficulty was it not being immediately clear how it worked. Though, perhaps it could also be retooled to be more intuitive.

Group members involved in this project were:
Allan Yong
Anthony Scavarelli
Henri Kuschkovitz
Thomas Breffit

Also a special thanks to our teacher, James Acres, for helping with some, er okay, many of the problems we encountered ;)

Intro:

Recently I was asked to fill in as a “VJ/video jockey” for a local club this past Friday. Dabbling quite a bit with openFrameworks recently it seemed a natural fit to create an interactive experience, with some controls so that I could still claim to be a VJ and not just a programmer! The following motion detection/sound detection visualizer is what I came up with:

A clear video screencast of the project using my friend’s created cello track “Incidence”

The Beginning

While learning OpenFrameworks a while ago, and building an interactive environment project in school, a friend of mine asked to create a visualizer to go along with a cello concert he was playing near Toronto. At the time I was playing with particle and vector fields so using these tools seemed a natural fit. I came up with the following: trying to engage the audience by using their motion to move the falling particles ( which fall from the cellist playing at the top of the 8 foot stage/projection ).

Another Try

Again recently I was asked to create a music visualizer for a local club that would stretch across three rear-projected screens as opposed to just one front projection as above. Noticing how slow the Toronto gig projection was, with maybe only 300 particles, I knew I had a lot of optimizations to do! I decided to stick with particle fields but I also wanted to make sure the sound detection was much clearer, that I utilized some openGL effects for a more interesting and efficient look, and that there were controls built in so that one could control the visualizer during the performance as programming a computer to control everything by the sound is a difficult and expensive task. Also, I feel being part of the live performance in a more visceral sense can allow for greater creativity.

The live performance and setup of the three screens at the club

Features

• camera motion detection is transferred to a vector field which then is transferred to particles motion
• wandering particles ( the big glowing ones ) wander randomly
• ribbons are affected by camera motion
• some basic FFT ( Fast-fourier Analysis ) analysis is done on mic sound to get an overall feel for the volume of the sound to detect beats which affect the size of the ribbons and wandering particles
• colour change is affected by user which then translates smoothly between HSB hues using RGB to HSB and it’s reverse

Control

• ability to create or destroy the “wandering particles”
• ability to make wandering particles spit out other “fireworks” particles
• ability to change hues of background
• ability to toggle on or off the ribbon field

Optimizations

• Threading: This is huge as most modern processors utilize more than one core. Using threading to split up the workload more efficiently helped quite a bit. I used Rui Madeira’s thread classes to extend certain classes ( http://code.google.com/p/ruicode/downloads/list ).  Basically you put all your updating code into the updateThread() class and call updateOnce() during your main class’s applications update to keep everything relativily in sync with main applications set framerate. The classes I put in threads are as follows:
  • RibbonField ( the ribbons in the background )
  • ParticleField ( the “fireworks” particles that flow from the wandering particles )
  • WanderingParticleField
  • opticalFlow
• Optimizing opticalFlow. This includes:
  • putting it into a thread
  • changing the input camera resolution to 160×120 as opposed to 320×240 or 640×480 ( this makes a huge difference as there are less pixels to process )
  • blurring ofxCvImages not only looks cool but also decreases the amount of optical flow points while keeping everything relatively accurate
• Using direct openGL calls whenever possible such as called glBegin( GL_QUAD_STRIP ) to create trails as opposed to one quad at a time or creating my own function.
• Passing an image by reference to all particles that share a certain PNG ( This would be done within the “Field” classes wghen they create the individual particles ).
• Setting a max on the number of particles on the screen so that when I am controlling it I do not add too many and slow things down to a crawl. In this case the max number was especially important on the fireworks particles as they took quite a bit of processing to look the way they do and often were created together in very large clumps.
• FBO’s ( Frame Buffer Object ): using ofxFBOTexture I was able to create several FBO textures to try to lessen communication with GPU. It also provided useful for tinting the background image as one can use ofSetColor before drawing a FBO to tint it.

Equipment

• 3 rear-projection screens ( approximately 8 x 6 feet  )
• 3 projectors at 800×600 resolution each
• 1 Matrox Triple-Head-To-Go to bind all three projectors into one wide ( 2400×600) screen
• 1 MBP 2.4gHz to run everything on stage
• 1 PS3 EyeToy camera
• Another laptop to control laptop on stage near DJ station

Conclusion

This was definitely a great and enjoyable experience getting everything working using so much equipment. I was definitely a bit worried about getting three projectors running this piece smoothly but the optimizations above helped quite a bit! I think it is a simple example of a visualizer but I believe in its simplicity it worked well. The owner of the club liked it and my friend and girlfriend liked playing with during the show so much that I never actually controlled it all night lol. Anyways, I am nowhere close to an expert in openFrameworks, openGL, or C++ but if you have any questions please feel free to throw them in the comments section and I’ll do my best to answer them.

————————————————–

References

• Vector Fields / Particle Fields: http://www.makingthingsmove.org/blog/?cat=5
• Threading: http://code.google.com/p/ruicode/downloads/list
• Video Processing ( in samples folder ): http://www.openframeworks.cc/
• Intro to additive blending and OpenGL graphics: http://www.flight404.com/blog/?cat=4
• Badass cellist: http://www.kirkstarkey.com/
• Supporting Artist Community: http://artengine.ca/

I originally intended this blog to be about Flash and Flash-related posts such as Flex and such; but lately I have not been working with Flash at all. I don’t want to get into the Flash problems debate but I found it was a lot of work to get even the simplest of programs running smoothly. In the meantime, I have worked some co-ops at Fuel Industries and Adobe and continued with my university studies in Interactive Multimedia and Design – one year left!

Actually, after taking a class that touched on using Arduino microprocessors and Processing to create interesting interactive pieces I think I have found a new way to quickly create experimental pieces that involve input that doesn’t necessarily use keyboard and mouse as input. Specifically I have been doing a lot of work with a C++ framework called openFrameworks to create music visualizers for concerts around town here in Ottawa. Over the next week or so I will be posting abut what I have been doing over the last few months – from using Maya MEL script to create a downtown city, to using processing to simulate elliptical orbits, to several openFrameworks projects that use sound to create, hopefully, interesting visuals.

Well I have to run to my girlfriend’s for mother’s day dinner but expect new posts soon!