AUTOMATIC SUMMARIZATION OF BROADCASTED SOCCER VIDEOS WITH ADAPTIVE FAST-FORWARDING

* Due to the file size limitation of supplemental material package, only low qualities videos are provided here, which are encoded using MPEG4. If you could not play the video, please download the free video player VLC Media Player to view the associated videos.


1. Resource Allocation Based Summarization

Interprete summarization as a resource allocation problem (Click to View). The whole process is envisioned in a divide and conqure paradigm. Video clips are grouped into video segments, where the set of candidate sub-summaries with various play speed and other narrative styles is built on each segment. Summarization is then implemented as a resource allocation problem, by finding the optimal subset of non-overlapped sub-summaries from those candidate sets.


2. Camera Motion Extraction (Click to View)

Click to View Video for Camera Motion Estimation from Optical Flow

*Explanation: Extraction of camera motion vectors by averaging motion vectors of grassland pixels computed from the optical flow analysis.The camera direction is represented by the green bold line starting from the center of the frame.


3. Computing Scene-changing Tempo (Click to View)

Click to View Video for Player-tracking Results

*Explanation: Scene-changing tempo serves as a clue for guiding adaptive fastforwarding. Rather than defining a scene-changing tempo to reflect the intensity of the real game (which hence makes scene-changing tempo different from the hot-spot), we are more interested in searching for a metric to evaluate the motion complexity of the scene, e.g., the flunctuation of the camera view or the diversified movement of multiple players. Given a clip, the flunctuation of its camera view is evaluated by the standard deviation of the camera motion vectors, while the complexity of diversified player movements is defined as the average of standard deviation of players’ moving speeds in each frame. The scene-changing tempo of this clip is then defined as a weighted sum of the above two terms.


4. Video segmentation based on view type structure (Click to View)

*Explanation: If there is no far view included in the above segment, this segment will be called a dependent segment, and will be merged into the previous one.


5. Refinement of Far-view clips (Click to View)

Click to View Video for Refined Far-view Clip Boundary


6. Solve this resource allocation problem by Lagrange relaxation

a) Benefit expansion of clip interest within each segment

* We set the base game relevance and emotional level to the confidence evalutated from hot-spots. We then spread them according to the view-type of each clip in the segment.

* For each consequent dependent segment (i.e, segment without far view), we will set the base game relevance and emotional level to 80% of its previous segment.

b) Extra benefit from local story organization

c) A generated convex-hull

* In the following two figures, we show benefit-cost convex-hulls computed for a sample segment. Below the convex-hull, a graph shows the boundaries of clips in this segment along with their corresponding viewtypes and scene-changing tempos. Six different duration constraints lead to six different optimal sub-summaries, plotted in the six subgraphs. In each sub-graph, the upper part shows the selection status of a clip, and the lower part shows its fastforwarding speed. (Play Speed: 1=1.0x, 2=1.5x, 3=2.0x).

Remarks 1: A global tendancy is that the overall play speed decreases when the duration constaint increases, so as to satisfy both a busy audience to fast-browse the game and an audience with more leisure to enjoy the video.

Remarks 2: The second observation is that a lower scene-changing tempo favors more on a higher play speed and a higher one takes in a lower play speed. This is refected, e.g., by 2378s-2387s in Opt2, 2367s-2378s in Opt3, and 2355s-2367s in Opt4.

Remarks 3: However, play speed does not always coincide with the scene-changing tempo, e.g., the result from 2387s-2392s in Opt1. The reason is that the play speed is not only affected by the scene-changing tempo, but is also determined by the base interest of a clip. The base interest of a clip is computed from its temporal distance to the end of the action. Hence, although the clip between 2391s-2393s has a lower scene-changing tempo than its preceding clip, any speed up of this clip will result in a higher benefit lost, which prevents it from taking a high fast-forwarding speed. We regard this consequence as a reasonable and positive effect, because a clip having a higher interest deserves more chances to be presented in a normal speed.


7. Global story organization

Results.

* we plot the summaries resulting from our fast-forwarding enabled method. The first subgraph presents the view-structure of segments, while the second gives the computed scene-changing tempos. Four cases are compared, i.e., our method with u^FF=0.40, 0.60 and 0.80 to the method without fast-forwarding (NO FF), under three duration constraints, i.e., u^LEN=10%,20% and 30% of the video length.

Remarks: It is easy to find that both a smaller u^FF and a smaller u^LEN intend to increase the overall play speed. Hence, it provides us a way to personalize the fast-forwarding functionality by tuning u^FF. Furthermore, by saving the duration resources through fast-forwarding, it is able to either include more actions, such as three actions within 2300s-2500s for u^LEN=10%, or complete an action, e.g., 2450s-2480s in the result of u^LEN=30%. All these demonstrate the advantage of including adaptive fast-forwarding into summarization.

Due to the file size limitation, we only provide two low-quality videos here.

*In order to include more actions for making comparisons, we choose to present the video results from u^LEN=20% rather than u^LEN=10% to present a long enough summary. Accordingly, we have to resize the original DVD quality video (PAL 720x576) to 360x288, and re-encode the video to a very low bitrate around 512kbps , so as to meet the limitation of 30MB. Although it is inavoidable that visual evaluation of these results will be affected by the low video quality, it is still a better option than reducing FPS or further reducing the video frame size.

Click to View Video for With FF (u^FF=0.60 u^LEN=20%)

Click to View Video for NO FF (u^LEN=20%)

 

 

Valid HTML 4.01 Strict