Developing Fast Multi-Object Tracking System

Client – Bong Wie

Advisor – Meng Lu

Sean Nichols

Chi Hoe How

Yishu Mei

 

Dec 14-12

Previous Work

  • Initial Plan : Turntable with meteorite
    • Simulates an asteroid in space with heat source
    • Show that IR camera can detect better than visible camera
  • FLIR camera used
    • Did not give individual pixel data (only point temp, min and max)
    • MATLAB was used to interpolate pixel temp data from image
  • EE 492 : We needed more electrical design
  • Decided to emulate Hypervelocity Asteroid Interceptor Vehicle (HAIV)
    • ​Optical camera and translational motion table
    • Design a system with fast multi-object detection
    • Process info and track an object of interest

Project Idea

  • Feedback loop
    • Camera image input controls future position of itself
    • Similar to algorithm outlined in client's paper

Project Breakdown

Hardware Component

  • Locates objects
  • Handles ALL image processing
  • K-Means Clustering
  • Centroids stored in registers

 

Software Component

  • Tracks objects
  • No image processing
  • Controls hardware

High Level Requirements

  • Minimize lag between object detection and frame rate
    • ​(HW processing + SW processing) < frame period(1/frame rate)
  • ​​Detect multiple objects
    • ​Determine how many objects to look for
      • ​Expected # of matched pixels
      • Expected pixels per object
      • Actual # of matched pixels
  • ​​Software must singulate object of interest
    • ​Keep last positions
    • Know current position
    • Predict future positions

Top Level System

Physical Implementation

  • Translational motion table​
    • Conveyor : x, y direction, controlled by stepper motor
    • Camera : placed on conveyor
  • HDMI Output                                                                               
    • User Feedback
    • Configuration
  • Camera Input
    • HD camera, 1920 x 1080 resolution
    • Bayer data output to FPGA

Hardware Implementation

  • Insert custom peripheral that goes into pipeline  
  • Interface with Xilinx AXI4-Stream Video Protocol
  • Register interface for control      

  • “K-Means Clustering” implemented in hardware to find centroids of each object

K - Means Clustering

  • K-Means Clustering
    • ​Object centroids found by averaging the positions of the nearest pixels for each pass.

                       Initialize                          Group Nearest                        Update Mean

K - Means Clustering

  • K-Means 2nd Pass

                                     Group Nearest                            Update Means

Hardware Implementation

Xilinx AXI4-Stream Interface 

  • Zero image processing pipeline latency
  • Ability to "paint" matched pixels
  • Counts Row and Column
  • Writes to Match RAM

Hardware Implementation

K-Means Clustering FSM

Hardware Implementation

K-Means Clustering Implementation

  • Grouping
    • Distance calculation
    • Minimum calculation
  • Averaging
    • Accumulation
    • Division

Hardware Implementation

Grouper

  • 16 Distance calculation cores
  • Runs through all matches 
  • Tags match with Object ID

Hardware Implementation

Grouper High Level Diagram

Hardware Implementation

Averager

  • 16 Accumulator cores
    • Only accumulate on ID match
    • Counts number of matches
  • 2 Divider modules
    • Separate for Row and Column
    • Large on resouces

Hardware Implementation

K-Means Clustering High Level Diagram

Worst Case Hardware Latency

Grouper

  • Max matches in RAM = 32400
  • Individual match processing = 17 cycles
  • Max processing time at 143 MHz= 3.6 ms per pass

Averager

  • Accumulation
    • Individual processing 1 cycle
    • Max processing time at 143 MHz = 226 us per pass
  • Division
    • Individual processing = 64 cycles
    • Max iterations = 16
    • Max processing time at 143 MHz = 7 us per pass

                       Absolute worst case K-Means processing = 11 ms

Stepper Motor

  • Implemented acceleration and deceleration control
  • Automatic handling switching of direction
  • Fully configurable via register set

Software Implementation

  • Handles single object tracking
    • ​No image processing
    • Tracks one when multiple are present
  • ​Determines # objects expected​
    • Can be function of expected pixel matches   per object
  • ​Calculates relative object speed
  • Updates motor speed to match calculated speed
  • P-D Controller
    • error correction to center camera over object of interest

Software Implementation

  • P-D Controller
    • error correction to center the camera over object of interest
    •  

Technical Challenge 1

  • Colorspace Challenges
  • Initially started with RGB colorspace
  • Not sufficient for filtering colors
    • Colors are subsets of grayscale
  • Solution : Moved to YCbCr colorspace
    • Based on difference of colors

Technical Challenge 2

  • Not enough Block RAM on Zynq
    • Worst Case - need 4,147,200 words
    • Would use more than 100% of BRAM resources                 
    • Solution : Sample on every 8th column and every 8th row
      • This reduces to a maximum of  64,800 words. 
        • Uses 35% of Zynq BRAM Resources
        • Still provides enough data samples

Technical Challenge 3

  • Camera Input to FPGA
    • ​Camera drives pixel clock and pixel values
    • FPGA special clock tree input buffers
    • ZedBoard limitation
      • These pins are only accessible through special FMC connector
    • ​Solution :  Use a different camera!!