CPS: Synergy: Distributed Sensing, Learning and Control in Dynamic Environments

Accomplishments

• Major Goals of the Project:

• Accomplishments under these goals:

Major Activities:

• Specific Objectives:

• Significant Results:

A) Multi­target Tracking and Data Association in a Sensor Network Considering Scene Constratins: Existing data association techniques mostly focus on matching pairs of data­point sets and then repeating this process along space­time to achieve long term correspondences. However, in many problems such as person re­identification, a set of data­points may be observed at multiple spatio­temporal locations and/or by multiple agents in a network and simply combining the local pairwise association results between sets of data­points often leads to inconsistencies over the global space­time horizons. In this work, we propose a novel Network Consistent Data Association (NCDA) framework formulated as an optimization problem that not only maintains consistency in association results across the network, but also improves the pairwise data association accuracies. The proposed NCDA can be solved as a binary integer program leading to a globally optimal solution and is capable of handling the challenging data­association scenario where the number of data­points varies across different sets of instances in the network. We have also achieved an online implementation of NCDA method that can dynamically associate new observations to already observed data­points in an iterative fashion, while maintaining network consistency. (Lead: Roy Chowdhury)

• Address consistent matching as a data-association task over network perspective
• Formulate an optimization problem to maximize overall pairwise similarity while abiding by network level constraints

  B) Online Adaptation of Learned Models in Sensor Networks: Most traditional multi­camera person re­identification systems rely on learning a static model on tediously labeled training data. Such a framework may not be suitable for situations when new data arrives continuously or all the data is not available for labeling beforehand. Inspired by the ‘value of information’ active learning framework, we propose a continuous learning person reidentification system with a human in the loop. The human in the loop not only provides labels to the incoming images but also improves the learned model by providing most appropriate attribute based explanations. These attribute based explanations are used to learn attribute predictors along the way. The overall effect of such a strategy is that starting with a few annotated images, the system begins to improve via a symbiotic relationship between the man and the machine. The machine assists the human to speed the annotation and the human assists the machine to update itself with more annotation so that more and more distinct persons are re­identified as more and more images come in. (Lead: Roy Chowdhury)

• Propose a sparse representative selection based approach
• Formulate as a convex optimization problem, which endorses selection of samples with the most variabilities

C) Managing and Querying Data Under Uncertainty: When objects travel so that their positions cannot be monitored continuously, and there are “blind” regions and periods, we have developed efficient techniques to determine whether groups of moving objects may have assembled within these blind regions. Our techniques are based on the widely­used approach of Contraction Hierarchies, and have proved to be very efficient. (Lead: Ravishankar)

• Introduce the novel and important class of assembly queries:
  1. “Assembly discovery”: determine whether two or more moving objects could have had a meeting within a region of interest
  2. “Assembly planning”: arrange for meetings for a group of friends visiting a city without violating their remaining schedules
• Provide efficient solutions given incomplete trajectory information, using the topology of the underlying transportation network
• Present a formal model for the general problem and prove the correctness of our algorithm
• Utilize a preprocessing method based on Contraction Hierarchies to gain orders of magnitude speed up over the naïve Dijkstra-based methods

  D) Information based Planner with Guarantees: Developed an optimal path planner which maximizes information collection, but with a probabilistic constraint of not getting lost. Formal proofs are defined, and empirical evaluations are conducted. A receding horizon approach to real time work is developed using a novel tail cost approximation. The work is verified with an indoor robot over multiple trials. (Lead: Campbell) Key results:

• General info gathering framework
  1. Maximizing information goals
  2. Probabilistic guarantees
• Asymptotic guarantees as the # of samples increases
• Bound on the reward for partially known environments
• Speeds computation (>1000x) to real time
• Validation by simulation/experiment

E) Joint exploration and tracking: Developed a formal solution to the multi­robot multiobject exploration and tracking problems simultaneously. A hierarchical architecture is used to coordinate robotic agents in the tracking of multiple Objects­of­Interest (OIs) while simultaneously allowing the task to remain computationally efficient. The primary contributions of this work are probabilistic guarantees on tracking performance, automatic discovery of new OIs, a seamless transition from exploration to tracking, and the automatic balancing of exploration and tracking. (Lead: Campbell) F) Attributes Co­occurrence Pattern Mining: Person re­identification has received considerable attention in the image processing, computer vision and pattern recognition communities because of its huge potential for video­based surveillance applications and the challenges it presents due to illumination, pose and viewpoint changes among nonoverlapping cameras. Being different from the widely used low­level descriptors, visual attributes (e.g., hair and shirt color) offer a human understandable way to recognize people. We have developed a new way to take advantage of them. First, convolutional neural networks are adopted to detect the attributes. Second, the dependencies among attributes are obtained by mining association rules, and they are used to refine the attributes classification results. Third, metric learning technique is used to transfer the attribute learning task to person re­identification. Finally, the approach is integrated into an appearance­based method for video­based person re­identification. Experimental results on two benchmark datasets indicate that attributes can provide improvements both in accuracy and generalization capabilities. (Lead: Bhanu)

• Visual attributes (e.g., hair and shirt color) offer a human understandable way to recognize people
• Design three ConvNets for attributes detection
• Exploit co-occurrence information to improve attributes’ descriptive capabilities
• Propose an attributes-based method and then combine it with an appearance-based model for final prediction
• Achieve better and more consistent results on two public datasets

   G) Multiperson Tracking by Online Learned Grouping: An online approach to learn elementary groups containing only two targets, i.e., pedestrians, for inferring high level context is introduced to improve multiperson tracking. In most existing data associationbased tracking approaches, only low­level information (e.g., time, appearance, and motion) is used to build the affinity model, and each target is considered as an independent agent. Unlike those previous methods, an online learned social grouping behavior model is used to provide more robust tracklet affinities. A disjoint grouping graph is used to encode social grouping behavior of pairwise targets, where each node represents an elementary group of two targets, and two nodes are connected if they share a common target. Probabilities of the uncertain target in two connected nodes being the same person are inferred from each edge of the grouping graph. Relationships between elementary groups are discovered by group tracking, and a nonlinear motion map is used for explaining nonlinear motion pattern between elementary groups. The proposed method is efficient, able to handle group split and merge, and can be easily integrated into any basic affinity model. The approach is evaluated on four data sets, and it shows significant improvements compared with state­of­the­art methods. (Lead: Bhanu) H) ­ Group Structure Preserving Pedestrian Tracking: In order to improve tracking performance, many ideas have been proposed, among which the use of geometric information is one of the most popular directions in recent research. We proposed a novel multicamera pedestrian tracking framework, which incorporates the structural information of pedestrian groups in the crowd. In this framework, first, a new crosscamera model is proposed, which enables the fusion of the confidence information from all camera views. Second, the group structures on the ground plane provide extra constraints between pedestrians. Third, the structured support vector machine is adopted to update the cross­camera model for each pedestrian according to the most recent tracked location. The experiments and detailed analysis are conducted on challenging data. The results demonstrate that the improvement in tracking performance is significant when a group structure is integrated. (Lead: Bhanu)

• A new cross-camera model is proposed, which enables the fusion of the confidence information from all camera views
• Group structures on the ground plane provide extra constraints between pedestrians
• The structured support vector machine is adopted to update model
• Excellent results are obtained on challenging data

• Key Outcomes or other Achievements:
  • Publications in major conferences and journals. In many cases the software has been released along with the papers. Three Special Issues of Journals have been published (IEEE Computer, IEEE Sensor and Computer Vision and Image Understanding). Overview articles have also been published in IEEE publications.
  • New dataset for multicamera person tracking and reidentification.
  • New dataset for object recognition in unconstrained environments
  • Algorithms and software for lidar/vision data fusion for tracking people from moving robots, and an information optimal exploration planner.
• What opportunities for training and professional development has the project provided?
  • More than six graduate students have been partially supported by this grant. Four of the students have completed their PhDs and two more are PhD candidates.
  • At Cornell university three graduate students have been partially supported by this grant, with one who finished an MS and the other two are still in the program.
• How have the results been disseminated to communities of interest?
  • Publications, software and datasets. For example, a paper describing results on handling uncertainty will appear in the 25th ACM SIGSPATIAL International Conference on Advances in Geographic Information Systems. One of the authors on this paper was an employee of ESRI, the leading provider of software systems for managing spatial and geographic data.
• What do you plan to do during the next reporting period to accomplish the goals?
  • We will focus on online learning approaches for adaptation of scene understanding models in unconstrained environments.
  • Refinement of attribute­based learning methods with uncertainty associated with objects.
  • Study the distributed version of the information exploration planner.
  • Exploitation of parallelism for the improvement of query processing.

Journals or Juried Conference Papers

• A. Das, R. Panda, A. Roy­Chowdhury (2015). Active Image Pair Selection for Continuous Person Re­identification. IEEE Intl. Conf. on Image Processing. Link
• Ivanov, A., & Campbell, M. (2018). Joint Exploration and Tracking: JET. IEEE Control Systems Letters. 1 (2), 43. Link
• Ivanov, A., & Campbell, M. (2018). Uncertainty Constrained Robotic Exploration: An Integrated Exploration Planner. IEEE Transactions on Control Systems Technology. Link
• N. Thakoor and B. Bhanu (2016). Selective Experience Replay in Reinforcement Learning for Reidentification. IEEE International Conference on Image Processing. Link
• R. Panda, A. Bhuiyan, V. Murino, A. Roy­Chowdhury (2017). Unsupervised Adaptation of Reidentification Models in Dynamic Camera Networks. In Preparation. Link
• R. Panda, A. Das, A. Roy­Chowdhury (2016). Video Summarization in a Multi­View Camera Network. International Conf. on Pattern Recognition. Link
• R. Theagarajan, F. Pala and B. Bhanu (2017). EDeN: Ensemble of Deep Networks for Vehicle Classification. Traffic Surveillance Workshop and Challenge (TSWC­2017) held in conjunction with IEEE Conference on Computer Vision and Pattern Recognition. Link
• Reaz Uddin, Vassilis Tsotras, Chinya Ravishankar. Indexing of Approximate Spatiotemporal Trajectories Using Hilbert Curves. Under Review.
• S. Yang, L. An, Y. Lei, M. Li, N. Thakoor, B. Bhanu and Y. Liu (2017). A Dense Flow­based Framework for Real­time Object Registration Under Compound Motion. Pattern Recognition. 63 279. Link
• X. Chen, Z. Qin, L. An and B. Bhanu (2016). Multi­person tracking by online learned grouping model with nonlinear motion context. IEEE Trans. on Circuits and Systems for Video Technology. 26 (12), 2226. Link
• X. Zhang, F. Pala and B. Bhanu (2017). Attributes Co­occurrence Pattern Mining for Video­based Person Re­identification. IEEE International Conference on Advanced Video and Signal­based Surveillance. Link
• Z. Jin, L. An and B. Bhanu (2017). Group Structure Preserving Pedestrian Tracking in Multicamera Video Network. IEEE Transactions on Circuits and Systems for Video technology. 27 (10), 2165. Link

• Reaz Uddin, Michael Rice, Chinya Ravishankar, and Vassilis Tsotras (2017). Assembly Queries: Planning and Discovering Assemblies of Moving Objects Using Partial Information. Proc. 25th ACM SIGSPATIAL International Conference on Advances in Geographic Information Systems. Redondo Beach, CA. Link

• Various Softwares Under Development.