Production Review

Prototype Update I

I tried using jsFeat to implement Lukas Kanade optical flow algorithm, but it became too time intenstive. So here is a hack:

Experiments in Computer Vision - OpenCV Optical Flow

I went down a rabbit-hole of using more complicated optical flow techniques.  My friend Tyler Henry on using OpenFrameworks for his project so I consulted with him on an approach:

For pre-recorded video use OpenCv:

1. get ofxCv add-on if you don’t have it.  it’s where all the magic happens.
Note: always include the ofxOpenCv add-on (comes with oF) in your project when you use ofxCv
(ofxOpenCv includes the OpenCv lib, which ofxCv needs)

2. blob tracking:   
ofxCv has a nice implementation.
Check out the example projects: example-contours-color and example-contours-tracking.  
Combining those will work for you to: a) get blobs by color and b) ID each blob and track it, so you can keep track of which blob is which - only necessary if you have multiple blobs (i.e. if 2 objects in your video are 35C).

3. optical flow

Process the video using Farneback flow.
Either try ofxCv’s example-flow project (you need Kyle M’s ofxControlPanel add-on for it to run).
Or try my version: optFlowTest
Tyler's version adds some processing to get the average flow, he reversed the flow (because ofxCv marks the flow as going towards the previous video frame - his version has flow going forward to the current frame).  

To do:

a. blob detection for your target objects

b. farneback optical flow on the whole scene -> spits out a vector<ofVec2f> with magnitude/direction of flow per pixel
that vector<ofVec2f> is your flow field

c. for loop through the vector<ofVec2f> and compare the pixel position of each flow point to the pixels positions in your blob.  Set every point in the flow field that’s not in the blob to (0,0), i.e. no flow.  Now you have just the flow inside your blobs.

d. throw particles on your blobs, and tell them to follow the flow of the pixel they’re at.  They should move with your blobs, according to the flow of the video.  you could add some randomness to their movement to make them more particle-y.  But when they get to a point that has no flow, they should either stop or bounce backwards, back into the flow field.

Caveats:

Farneback flow might ignore smooth (i.e no image texture) parts of your video.  For that reason you might want to use the non-heat-mapped version of your video, since the heat colors would probably reduce texture.  You could also try sharpening the video.

After step c. you might want to save the flow field as a csv (or json?) file, since calculating the flow is really expensive.  That way, you first calculate the flow and then use the calculated values later. Each row in the csv would correspond to a frame of video (keep track of the frames using ofVideoPlayer.getCurrentFrame() because it might skip frames).  Each two columns in the csv would contain ofVec2f.x and ofVec2f.y data for each pixel in your video.

Also try scaling the video down to something like 320x240 if you have issues with calculating the flow.  Farneback flow will choke on hi-res video.

Design Meeting II

Building the tool...

After working on some experiments with the thermal camera and mapping:

I am still working on the aesthetic quality and rendering performance, but all the pieces are there. 

Thermal Imaging Experiments

Using the below rig I filmed a NY city street using a thermal camera. 

Equipment:

  1. Camera
  2. Seek Thermal Camera iPhone extension
  3. iPhone (camera)
  4. Rig (courtesy of Chris Woebken)

I first take the thermal video and use an optical flow detection library called oflow (https://github.com/anvaka/oflow) to take the brightest points in each frame of the video to be the "hottest" points:

function isHot(color) {
    return color[0] > 150 && color[1] > 150 && color[2] > 150;
}

I then run this through three.js. Instead of rendering the "blobs" I use their x,y positions to map my particle system to thier locations each frame. 

Nanotronics Microscopy

Equipment: Nanotronics nSpec Microscope with Nikon Objectives and Allied Vision Technologies GT 2750 Camera

Full link to all the videos: http://microbiome.nyc/nanotronics/

Thoughts for Thesis Writing...

General thesis writing architecture:

30 pages

The paper has to explain: what is it, why are you doing it, who is it for, and how does it work. (an expanded version of the abstract). Kinds of writing that can go in the paper:

  1. Process of thinking is something you should keep a record of. Record and documentation of the project.
  2. Recording of a conversation with your friend / explanations to your grandmother
  3. Description in the 3rd person of a person who goes through your project — to change the perspective
  4. Research
  5. A show at the Met, MoMA, etc… write a description and how it relates to what you’re doing. 
  6. Precedents 
  7. How ideas developed and how they changed from last semester.
  8. Crisis - write out the obstacles that are facing you and think your way around them 
  9. The backdrop of what you’re doing is the reality we are living in today - how is your project relevant to this world…. why are you doing this?

The paper is a form of experiment and support - this paper can include anything — anything that was included in you working out your project. 

For my thesis:

  • history of the problem
  • Writing a paragraph on the juxtaposition between Sky Survey’s and Metagenomic Surveys…. 
  • Twin study - NASA - Mason Lab

 

Follow the ideas where they take you…. 

 

Notes:

There is the (1) thesis project and (2) a larger direction that involves the design of the city, health, the living environment.

  • Disfigurement of human life by industrialization and modern life. 
  • Homeostatic systems -- that which exits in balance 

 

Random links:

http://www.amazon.com/The-Machinery-Life-David-Goodsell/dp/0387849246

http://www.nytimes.com/2016/02/12/science/ligo-gravitational-waves-black-holes-einstein.html?_r=0

https://www.google.com/search?q=diatoms+in+the+ocean&espv=2&biw=1224&bih=539&source=lnms&tbm=isch&sa=X&ved=0ahUKEwiqzejw3rHLAhVCrD4KHaiFAwAQ_AUIBigB#imgrc=_

https://paleonerdish.wordpress.com/2013/06/10/an-introduction-to-diatoms/

https://en.wikipedia.org/wiki/Stromatolite