Knowledge extraction from video data is challenging due to its high complexity in both the spatial and temporal domain. Ground truth is crucial for the evaluation and the adaptation of algorithms to new domains. Unfortunately, ground truth annotation is inconvenient and time consuming. Common annotation tools mostly rely on simple geometric primitives such as rectangles or ellipses. Here we propose a novel, interactive and semi-automatic process, which actively asks for user input if the result of the automatic annotation appears to be incorrect. iSeg has been tested on two visual stimulus datasets for eye tracking experiments and on two surveillance datasets.

Reflecting the architecture of video visual analytics, our interactive annotation and segmentation tool iSeg focuses on a semi-automatic architecture putting the user in the loop. The main processing blocks can be seen in figure 1 and are in the following briefly explained.

Manual annotation of specific keyframes

The user has to outline the object on a few frames, where its pose is characteristic. The selection is a polygon, which can be drawn around the object.

Morphing polygon geometry

Subsequently the user can trigger the morphing process, which will interpolate the polygons between all given keyframes over the intermediate frames. To reduce the computational cost we implemented our own preliminary algorithm which satisfies our needs.

Automatic/Interactive keypoint-fitting

Since the morphing is only applied linearly, the resulting contours will most likely not be completely accurate, if the object moves non-linearly between the frames. To compensate this error we compute keypoints on the image patches inside the contours and track them over the frames. From the resulting movement a correction of the contour is calculated.

In some cases the algorithm might not be able to match the keypoints appropriately. Causes can be occlusion of the object or rapid movements. In these cases the system will recognize the error and actively query the user for a manual correction of the problematic frames.