{"id":216992,"date":"2026-05-04T11:44:13","date_gmt":"2026-05-04T15:44:13","guid":{"rendered":"https:\/\/www.hajim.rochester.edu\/senior-design-day\/?p=216992"},"modified":"2026-05-04T11:44:13","modified_gmt":"2026-05-04T15:44:13","slug":"mosaic-multimodal-video-retrieval-for-ravenous-design-studios","status":"publish","type":"post","link":"https:\/\/www.hajim.rochester.edu\/senior-design-day\/mosaic-multimodal-video-retrieval-for-ravenous-design-studios\/","title":{"rendered":"Mosaic: Multimodal Video Retrieval for Ravenous Design Studios"},"content":{"rendered":"\n

A multimodal search system for dance video archives. <\/p>\n\n\n\n

This immediately makes your site look like an actual product. <\/p>\n\n\n\n

Project Overview<\/h2>\n\n\n\n

This project builds a multimodal system that transforms raw video into a structured, queryable data representation. Instead of treating video as a continuous stream, we break it into meaningful segments enriched with text, visual features, and semantic tags. This allows users to search video content using natural language queries and retrieve relevant moments instantly.<\/p>\n\n\n\n

The Problem<\/h2>\n\n\n\n
\n
\n

Unstructured Data<\/h3>\n\n\n\n

Videos are continuous streams with no structure.<\/p>\n<\/div>\n\n\n\n

\n

Disconnected Modalities<\/h3>\n\n\n\n

Text (Whisper) and vision (YOLO) operate independently, with no unified representation.<\/p>\n<\/div>\n\n\n\n

\n

No Temporal Alignment<\/h3>\n\n\n\n

Frame-based detection and time-based transcripts are not naturally aligned, preventing effective retrieval.<\/p>\n<\/div>\n<\/div>\n\n\n\n

This is not just a model \u2014 it is a system<\/h2>\n\n\n\n

Our goal is to build an infrastructure layer for video understanding by integrating vision, language, and time into a unified representation. We convert video into timestamped segments enriched with both textual and visual metadata, enabling powerful search capabilities across modalities.<\/p>\n\n\n\n

Overview of Method<\/h2>\n\n\n\n
\"\"<\/figure>\n\n\n\n

From Video to Structured Data <\/h2>\n\n\n\n
\"\"<\/figure>\n\n\n\n

Segmentation + tagging converts video into a searchable representation<\/p>\n\n\n\n

System Pipeline<\/h2>\n\n\n\n
\"\"<\/figure>\n\n\n\n

Each segment becomes a structured, searchable unit rather than part of a continuous video stream.<\/p>\n\n\n\n

This pipeline converts raw video into structured, searchable segments by aligning text, motion, and semantic tags.<\/p>\n\n\n\n

Dataset<\/h2>\n\n\n\n

We evaluated our system on a diverse dataset of 6\u20138 videos, including both sponsor-provided and external clips. The dataset contains over 20 minutes of footage, approximately 30,000 frames, and more than 6,000 object detections. It includes high-motion dance sequences designed to stress both object detection and temporal alignment challenges.<\/p>\n\n\n\n

Exploratory Results<\/h2>\n\n\n\n
\n
\n

Segments per Video<\/h3>\n\n\n\n

badbunny: ~75 <\/p>\n\n\n\n

single ladies: ~8 <\/p>\n\n\n\n

katseye touch: ~11<\/p>\n<\/div>\n\n\n\n

\n

Tag Distribution<\/h3>\n\n\n\n

solo: 23<\/p>\n\n\n\n

duo: 24<\/p>\n\n\n\n

group: 44<\/p>\n\n\n\n

high_energy: 10<\/p>\n\n\n\n

medium_energy: 15<\/p>\n\n\n\n

low_energy: 69<\/p>\n<\/div>\n<\/div>\n\n\n\n

Example Output<\/h2>\n\n\n\n

Example 1:<\/strong>
[badbunny] | 30\u201333 sec
Tags: solo, high_energy
Text: “This is the time when it’s dark.”
Description: solo dancer with fast, high-energy movement<\/p>\n\n\n\n

Example 2:<\/strong>
[singleladies] | 9\u201312 sec
Tags: duo, high_energy
Description: two dancers with fast, high-energy movement<\/p>\n\n\n\n

Each segment is converted into a structured unit that supports search and retrieval.<\/p>\n\n\n\n

Results<\/h2>\n\n\n\n

The system successfully processes raw video into structured data and generates over 100 timestamped segments across multiple videos. Each segment includes multimodal features such as text, motion characteristics, and semantic tags. This enables cross-video retrieval using natural language queries and motion-based filtering.<\/p>\n\n\n\n

Technical Challenges<\/h2>\n\n\n\n
    \n
  • Temporal alignment between Whisper (time-based) and YOLO (frame-based)<\/li>\n\n\n\n
  • Multimodal integration across vision, language, and time<\/li>\n\n\n\n
  • High computational cost of motion detection<\/li>\n\n\n\n
  • Complexity of capturing subtle and fast dance movements<\/li>\n<\/ul>\n\n\n\n

    Key Insight<\/h2>\n\n\n\n
    \n
    <\/blockquote>\n<\/blockquote>\n\n\n\n

    Make this bold-looking with a Quote block:<\/p>\n\n\n\n

    This is not a computer vision problem.
    This is a data coordination problem.<\/p>\n\n\n\n

    Value comes from structuring data, not just detecting it.<\/p>\n\n\n\n

    Next Steps<\/h2>\n\n\n\n

    Short Term<\/h3>\n\n\n\n
      \n
    • Improve timestamp alignment<\/li>\n\n\n\n
    • Reduce segmentation noise<\/li>\n<\/ul>\n\n\n\n

      Medium Term<\/h3>\n\n\n\n
        \n
      • Introduce vector embeddings for semantic search<\/li>\n\n\n\n
      • Improve ranking of results<\/li>\n<\/ul>\n\n\n\n

        Long Term<\/h3>\n\n\n\n
          \n
        • Scale to large video datasets<\/li>\n\n\n\n
        • Build an interactive search interface<\/li>\n<\/ul>\n\n\n\n

          Conclusion<\/h2>\n\n\n\n

          We built a working multimodal pipeline that converts video into structured, searchable data. By aligning vision, language, and time, our system enables efficient query-based retrieval across videos.<\/p>\n\n\n\n

          Final takeaway:<\/strong>
          We make video as searchable as text.<\/p>\n\n\n\n

          Acknowledgements<\/h2>\n\n\n\n

          Thank you to our sponsor Kurt Johnston and Professor Caliskan for their guidance and support throughout this project.<\/p>\n\n\n\n

          <\/p>\n","protected":false},"excerpt":{"rendered":"

          A multimodal search system for dance video archives.  This immediately makes your site look like an actual product.  Project Overview This project builds a multimodal system that transforms raw video…<\/p>\n","protected":false},"author":20892,"featured_media":0,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"_coblocks_attr":"","_coblocks_dimensions":"","_coblocks_responsive_height":"","_coblocks_accordion_ie_support":"","_monsterinsights_skip_tracking":false,"_monsterinsights_sitenote_active":false,"_monsterinsights_sitenote_note":"","_monsterinsights_sitenote_category":0,"footnotes":""},"categories":[4452,4512],"tags":[],"coauthors":[25922,25952,25962,25632],"class_list":["post-216992","post","type-post","status-publish","format-standard","hentry","category-current-year","category-dsc"],"acf":[],"yoast_head":"\nMosaic: Multimodal Video Retrieval for Ravenous Design Studios - Senior Design Day<\/title>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/www.hajim.rochester.edu\/senior-design-day\/mosaic-multimodal-video-retrieval-for-ravenous-design-studios\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"Mosaic: Multimodal Video Retrieval for Ravenous Design Studios - Senior Design Day\" \/>\n<meta property=\"og:description\" content=\"A multimodal search system for dance video archives.  This immediately makes your site look like an actual product.  Project Overview This project builds a multimodal system that transforms raw video…\" \/>\n<meta property=\"og:url\" content=\"https:\/\/www.hajim.rochester.edu\/senior-design-day\/mosaic-multimodal-video-retrieval-for-ravenous-design-studios\/\" \/>\n<meta property=\"og:site_name\" content=\"Senior Design Day\" \/>\n<meta property=\"article:published_time\" content=\"2026-05-04T15:44:13+00:00\" \/>\n<meta property=\"og:image\" content=\"https:\/\/www.hajim.rochester.edu\/senior-design-day\/wp-content\/uploads\/2026\/03\/image-14-1200x630.png\" \/>\n\t<meta property=\"og:image:width\" content=\"1200\" \/>\n\t<meta property=\"og:image:height\" content=\"630\" \/>\n\t<meta property=\"og:image:type\" content=\"image\/png\" \/>\n<meta name=\"author\" content=\"Britton Ambrose, James Kaprowski, Victoria Ritter, Audrey Clayton\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:label1\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data1\" content=\"Britton Ambrose, James Kaprowski, Victoria Ritter, Audrey Clayton\" \/>\n\t<meta name=\"twitter:label2\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data2\" content=\"3 minutes\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\\\/\\\/schema.org\",\"@graph\":[{\"@type\":\"Article\",\"@id\":\"https:\\\/\\\/www.hajim.rochester.edu\\\/senior-design-day\\\/mosaic-multimodal-video-retrieval-for-ravenous-design-studios\\\/#article\",\"isPartOf\":{\"@id\":\"https:\\\/\\\/www.hajim.rochester.edu\\\/senior-design-day\\\/mosaic-multimodal-video-retrieval-for-ravenous-design-studios\\\/\"},\"author\":{\"name\":\"Britton Ambrose\",\"@id\":\"https:\\\/\\\/www.hajim.rochester.edu\\\/senior-design-day\\\/#\\\/schema\\\/person\\\/4bff3f2e21b216d3438d55ec2de1f4e0\"},\"headline\":\"Mosaic: Multimodal Video Retrieval for Ravenous Design Studios\",\"datePublished\":\"2026-05-04T15:44:13+00:00\",\"mainEntityOfPage\":{\"@id\":\"https:\\\/\\\/www.hajim.rochester.edu\\\/senior-design-day\\\/mosaic-multimodal-video-retrieval-for-ravenous-design-studios\\\/\"},\"wordCount\":532,\"image\":{\"@id\":\"https:\\\/\\\/www.hajim.rochester.edu\\\/senior-design-day\\\/mosaic-multimodal-video-retrieval-for-ravenous-design-studios\\\/#primaryimage\"},\"thumbnailUrl\":\"https:\\\/\\\/www.hajim.rochester.edu\\\/senior-design-day\\\/wp-content\\\/uploads\\\/2026\\\/03\\\/image-14-1024x381.png\",\"articleSection\":[\"1. Current Year Programs\",\"DSC\"],\"inLanguage\":\"en-US\"},{\"@type\":\"WebPage\",\"@id\":\"https:\\\/\\\/www.hajim.rochester.edu\\\/senior-design-day\\\/mosaic-multimodal-video-retrieval-for-ravenous-design-studios\\\/\",\"url\":\"https:\\\/\\\/www.hajim.rochester.edu\\\/senior-design-day\\\/mosaic-multimodal-video-retrieval-for-ravenous-design-studios\\\/\",\"name\":\"Mosaic: Multimodal Video Retrieval for Ravenous Design Studios - Senior Design Day\",\"isPartOf\":{\"@id\":\"https:\\\/\\\/www.hajim.rochester.edu\\\/senior-design-day\\\/#website\"},\"primaryImageOfPage\":{\"@id\":\"https:\\\/\\\/www.hajim.rochester.edu\\\/senior-design-day\\\/mosaic-multimodal-video-retrieval-for-ravenous-design-studios\\\/#primaryimage\"},\"image\":{\"@id\":\"https:\\\/\\\/www.hajim.rochester.edu\\\/senior-design-day\\\/mosaic-multimodal-video-retrieval-for-ravenous-design-studios\\\/#primaryimage\"},\"thumbnailUrl\":\"https:\\\/\\\/www.hajim.rochester.edu\\\/senior-design-day\\\/wp-content\\\/uploads\\\/2026\\\/03\\\/image-14-1024x381.png\",\"datePublished\":\"2026-05-04T15:44:13+00:00\",\"author\":{\"@id\":\"https:\\\/\\\/www.hajim.rochester.edu\\\/senior-design-day\\\/#\\\/schema\\\/person\\\/4bff3f2e21b216d3438d55ec2de1f4e0\"},\"breadcrumb\":{\"@id\":\"https:\\\/\\\/www.hajim.rochester.edu\\\/senior-design-day\\\/mosaic-multimodal-video-retrieval-for-ravenous-design-studios\\\/#breadcrumb\"},\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\\\/\\\/www.hajim.rochester.edu\\\/senior-design-day\\\/mosaic-multimodal-video-retrieval-for-ravenous-design-studios\\\/\"]}]},{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\\\/\\\/www.hajim.rochester.edu\\\/senior-design-day\\\/mosaic-multimodal-video-retrieval-for-ravenous-design-studios\\\/#primaryimage\",\"url\":\"https:\\\/\\\/www.hajim.rochester.edu\\\/senior-design-day\\\/wp-content\\\/uploads\\\/2026\\\/03\\\/image-14.png\",\"contentUrl\":\"https:\\\/\\\/www.hajim.rochester.edu\\\/senior-design-day\\\/wp-content\\\/uploads\\\/2026\\\/03\\\/image-14.png\",\"width\":2048,\"height\":761},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\\\/\\\/www.hajim.rochester.edu\\\/senior-design-day\\\/mosaic-multimodal-video-retrieval-for-ravenous-design-studios\\\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Home\",\"item\":\"https:\\\/\\\/www.hajim.rochester.edu\\\/senior-design-day\\\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"Mosaic: Multimodal Video Retrieval for Ravenous Design Studios\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\\\/\\\/www.hajim.rochester.edu\\\/senior-design-day\\\/#website\",\"url\":\"https:\\\/\\\/www.hajim.rochester.edu\\\/senior-design-day\\\/\",\"name\":\"Senior Design Day\",\"description\":\"\",\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\\\/\\\/www.hajim.rochester.edu\\\/senior-design-day\\\/?s={search_term_string}\"},\"query-input\":{\"@type\":\"PropertyValueSpecification\",\"valueRequired\":true,\"valueName\":\"search_term_string\"}}],\"inLanguage\":\"en-US\"},{\"@type\":\"Person\",\"@id\":\"https:\\\/\\\/www.hajim.rochester.edu\\\/senior-design-day\\\/#\\\/schema\\\/person\\\/4bff3f2e21b216d3438d55ec2de1f4e0\",\"name\":\"Britton Ambrose\",\"url\":\"https:\\\/\\\/www.hajim.rochester.edu\\\/senior-design-day\\\/author\\\/jambros8\\\/\"}]}<\/script>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"Mosaic: Multimodal Video Retrieval for Ravenous Design Studios - Senior Design Day","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/www.hajim.rochester.edu\/senior-design-day\/mosaic-multimodal-video-retrieval-for-ravenous-design-studios\/","og_locale":"en_US","og_type":"article","og_title":"Mosaic: Multimodal Video Retrieval for Ravenous Design Studios - Senior Design Day","og_description":"A multimodal search system for dance video archives.  This immediately makes your site look like an actual product.  Project Overview This project builds a multimodal system that transforms raw video…","og_url":"https:\/\/www.hajim.rochester.edu\/senior-design-day\/mosaic-multimodal-video-retrieval-for-ravenous-design-studios\/","og_site_name":"Senior Design Day","article_published_time":"2026-05-04T15:44:13+00:00","og_image":[{"width":1200,"height":630,"url":"https:\/\/www.hajim.rochester.edu\/senior-design-day\/wp-content\/uploads\/2026\/03\/image-14-1200x630.png","type":"image\/png"}],"author":"Britton Ambrose, James Kaprowski, Victoria Ritter, Audrey Clayton","twitter_card":"summary_large_image","twitter_misc":{"Written by":"Britton Ambrose, James Kaprowski, Victoria Ritter, Audrey Clayton","Est. reading time":"3 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"Article","@id":"https:\/\/www.hajim.rochester.edu\/senior-design-day\/mosaic-multimodal-video-retrieval-for-ravenous-design-studios\/#article","isPartOf":{"@id":"https:\/\/www.hajim.rochester.edu\/senior-design-day\/mosaic-multimodal-video-retrieval-for-ravenous-design-studios\/"},"author":{"name":"Britton Ambrose","@id":"https:\/\/www.hajim.rochester.edu\/senior-design-day\/#\/schema\/person\/4bff3f2e21b216d3438d55ec2de1f4e0"},"headline":"Mosaic: Multimodal Video Retrieval for Ravenous Design Studios","datePublished":"2026-05-04T15:44:13+00:00","mainEntityOfPage":{"@id":"https:\/\/www.hajim.rochester.edu\/senior-design-day\/mosaic-multimodal-video-retrieval-for-ravenous-design-studios\/"},"wordCount":532,"image":{"@id":"https:\/\/www.hajim.rochester.edu\/senior-design-day\/mosaic-multimodal-video-retrieval-for-ravenous-design-studios\/#primaryimage"},"thumbnailUrl":"https:\/\/www.hajim.rochester.edu\/senior-design-day\/wp-content\/uploads\/2026\/03\/image-14-1024x381.png","articleSection":["1. Current Year Programs","DSC"],"inLanguage":"en-US"},{"@type":"WebPage","@id":"https:\/\/www.hajim.rochester.edu\/senior-design-day\/mosaic-multimodal-video-retrieval-for-ravenous-design-studios\/","url":"https:\/\/www.hajim.rochester.edu\/senior-design-day\/mosaic-multimodal-video-retrieval-for-ravenous-design-studios\/","name":"Mosaic: Multimodal Video Retrieval for Ravenous Design Studios - Senior Design Day","isPartOf":{"@id":"https:\/\/www.hajim.rochester.edu\/senior-design-day\/#website"},"primaryImageOfPage":{"@id":"https:\/\/www.hajim.rochester.edu\/senior-design-day\/mosaic-multimodal-video-retrieval-for-ravenous-design-studios\/#primaryimage"},"image":{"@id":"https:\/\/www.hajim.rochester.edu\/senior-design-day\/mosaic-multimodal-video-retrieval-for-ravenous-design-studios\/#primaryimage"},"thumbnailUrl":"https:\/\/www.hajim.rochester.edu\/senior-design-day\/wp-content\/uploads\/2026\/03\/image-14-1024x381.png","datePublished":"2026-05-04T15:44:13+00:00","author":{"@id":"https:\/\/www.hajim.rochester.edu\/senior-design-day\/#\/schema\/person\/4bff3f2e21b216d3438d55ec2de1f4e0"},"breadcrumb":{"@id":"https:\/\/www.hajim.rochester.edu\/senior-design-day\/mosaic-multimodal-video-retrieval-for-ravenous-design-studios\/#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/www.hajim.rochester.edu\/senior-design-day\/mosaic-multimodal-video-retrieval-for-ravenous-design-studios\/"]}]},{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/www.hajim.rochester.edu\/senior-design-day\/mosaic-multimodal-video-retrieval-for-ravenous-design-studios\/#primaryimage","url":"https:\/\/www.hajim.rochester.edu\/senior-design-day\/wp-content\/uploads\/2026\/03\/image-14.png","contentUrl":"https:\/\/www.hajim.rochester.edu\/senior-design-day\/wp-content\/uploads\/2026\/03\/image-14.png","width":2048,"height":761},{"@type":"BreadcrumbList","@id":"https:\/\/www.hajim.rochester.edu\/senior-design-day\/mosaic-multimodal-video-retrieval-for-ravenous-design-studios\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/www.hajim.rochester.edu\/senior-design-day\/"},{"@type":"ListItem","position":2,"name":"Mosaic: Multimodal Video Retrieval for Ravenous Design Studios"}]},{"@type":"WebSite","@id":"https:\/\/www.hajim.rochester.edu\/senior-design-day\/#website","url":"https:\/\/www.hajim.rochester.edu\/senior-design-day\/","name":"Senior Design Day","description":"","potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/www.hajim.rochester.edu\/senior-design-day\/?s={search_term_string}"},"query-input":{"@type":"PropertyValueSpecification","valueRequired":true,"valueName":"search_term_string"}}],"inLanguage":"en-US"},{"@type":"Person","@id":"https:\/\/www.hajim.rochester.edu\/senior-design-day\/#\/schema\/person\/4bff3f2e21b216d3438d55ec2de1f4e0","name":"Britton Ambrose","url":"https:\/\/www.hajim.rochester.edu\/senior-design-day\/author\/jambros8\/"}]}},"_links":{"self":[{"href":"https:\/\/www.hajim.rochester.edu\/senior-design-day\/wp-json\/wp\/v2\/posts\/216992","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.hajim.rochester.edu\/senior-design-day\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.hajim.rochester.edu\/senior-design-day\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.hajim.rochester.edu\/senior-design-day\/wp-json\/wp\/v2\/users\/20892"}],"replies":[{"embeddable":true,"href":"https:\/\/www.hajim.rochester.edu\/senior-design-day\/wp-json\/wp\/v2\/comments?post=216992"}],"version-history":[{"count":5,"href":"https:\/\/www.hajim.rochester.edu\/senior-design-day\/wp-json\/wp\/v2\/posts\/216992\/revisions"}],"predecessor-version":[{"id":239952,"href":"https:\/\/www.hajim.rochester.edu\/senior-design-day\/wp-json\/wp\/v2\/posts\/216992\/revisions\/239952"}],"wp:attachment":[{"href":"https:\/\/www.hajim.rochester.edu\/senior-design-day\/wp-json\/wp\/v2\/media?parent=216992"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.hajim.rochester.edu\/senior-design-day\/wp-json\/wp\/v2\/categories?post=216992"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.hajim.rochester.edu\/senior-design-day\/wp-json\/wp\/v2\/tags?post=216992"},{"taxonomy":"author","embeddable":true,"href":"https:\/\/www.hajim.rochester.edu\/senior-design-day\/wp-json\/wp\/v2\/coauthors?post=216992"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}