Short infographic of our machine learning based approach to track fish and analyse behaviour.
The first sequence shows raw video input, of which a subset of frames was annotated as the training set. After training the neural network with these annotations, fish in the video were then able to be automatically segmented with Mask R-CNN; this segmentation is here represented with a colored mask and corresponding bounding-box applied to each successfully detected fish in each frame of the video (second sequence). Identities of each fish were maintained in subsequent frames using a nearest neighbor linking approach. The third sequence displays the trajectories and estimated spine positions of each individual. A simplified model (colored ‘fish’) was then overlaid on actual positions of fish to compute visual field connectivity using a ray-casting approach, considering the visual fields of each eye of a focal individual (fourth sequence, gray-shaded areas) and occlusions (black- shaded areas).