{"id":216892,"date":"2026-05-04T11:41:34","date_gmt":"2026-05-04T15:41:34","guid":{"rendered":"https:\/\/www.hajim.rochester.edu\/senior-design-day\/?p=216892"},"modified":"2026-05-04T11:41:34","modified_gmt":"2026-05-04T15:41:34","slug":"urmc-otolarnyology-predicting-sphenoidotomy-performance-in-residents","status":"publish","type":"post","link":"https:\/\/www.hajim.rochester.edu\/senior-design-day\/urmc-otolarnyology-predicting-sphenoidotomy-performance-in-residents\/","title":{"rendered":"URMC-Otolarnyology: Predicting Sphenoidotomy Performance in Residents"},"content":{"rendered":"\n<h2 class=\"wp-block-heading\">I. Introduction<\/h2>\n\n\n\n<p>Reliable assessment of surgical performance is essential for resident education, feedback, and progression within procedural training. Although attending-based evaluations such as the Objective Structured  assessment of Technical Skills (OSATS) provide clinically meaningful judgments of technical skill, these assessments remain partly subjective and may vary across evaluators and operative settings. As motion-<br>tracking systems become more available in surgical education, an important question is whether objective movement-derived metrics can help explain or predict expert ratings of resident performance. <\/p>\n\n\n<div class=\"wp-block-image\">\n<figure class=\"aligncenter size-large is-resized\"><img loading=\"lazy\" decoding=\"async\" width=\"688\" height=\"1024\" src=\"https:\/\/www.hajim.rochester.edu\/senior-design-day\/wp-content\/uploads\/2026\/05\/sphen-688x1024.jpg\" alt=\"\" class=\"wp-image-245982\" style=\"aspect-ratio:0.6718755785965562;width:402px;height:auto\" srcset=\"https:\/\/www.hajim.rochester.edu\/senior-design-day\/wp-content\/uploads\/2026\/05\/sphen-688x1024.jpg 688w, https:\/\/www.hajim.rochester.edu\/senior-design-day\/wp-content\/uploads\/2026\/05\/sphen-202x300.jpg 202w, https:\/\/www.hajim.rochester.edu\/senior-design-day\/wp-content\/uploads\/2026\/05\/sphen-768x1143.jpg 768w, https:\/\/www.hajim.rochester.edu\/senior-design-day\/wp-content\/uploads\/2026\/05\/sphen.jpg 929w\" sizes=\"auto, (max-width: 688px) 100vw, 688px\" \/><\/figure>\n<\/div>\n\n\n<p>In this study, we investigated whether motion-tracking and procedural features could predict OSATS-related outcomes in sphenoidotomy. By comparing baseline, motion-only, and engineered mixed models, this project aimed to determine whether objective motion and procedural data capture meaningful signal related to attending-rated technical skill. This project also investigated whether kinematic resident based<br>features can predict experience level of residents. Through comparison of baseline nonlinear models with more complex models, this study determines whether motion data can predict resident experience level. More broadly, this work explores the potential role of quantitative motion analysis as a complementary approach to expert surgical assessment.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\"><br>II. Data Overview<\/h2>\n\n\n\n<p>To analyze the performance and experience levels of Otolaryngology residents during sphenoidotomy, we utilized data from four distinct sources. A key characteristic of the collected data is its paired structure, where attending surgeons and residents operate on the same case on opposite nostrils. This enables direct comparison of performance under similar clinical conditions. This data is categorized into three primary domains:<br><br><strong>Survey Data: <\/strong>Includes two post-sphenoidotomy surveys, an attending survey evaluating technical features and resident performance, and a resident self-assessment survey.<br><br><strong>Motion Data: <\/strong>Objective kinematic data captured from the surgical device to measure movement and instrument efficiency.<br><br><strong>Experience Level Data: <\/strong>A longitudinal dataset tracking total surgical volume and specific Endoscopic Sinus Surgery (ESS) cases.<\/p>\n\n\n\n<figure class=\"wp-block-image size-full\"><img loading=\"lazy\" decoding=\"async\" width=\"854\" height=\"292\" src=\"https:\/\/www.hajim.rochester.edu\/senior-design-day\/wp-content\/uploads\/2026\/04\/image-45.png\" alt=\"\" class=\"wp-image-225692\" title=\"Screenshot 2026-04-19 at 19.19.24.png\" srcset=\"https:\/\/www.hajim.rochester.edu\/senior-design-day\/wp-content\/uploads\/2026\/04\/image-45.png 854w, https:\/\/www.hajim.rochester.edu\/senior-design-day\/wp-content\/uploads\/2026\/04\/image-45-300x103.png 300w, https:\/\/www.hajim.rochester.edu\/senior-design-day\/wp-content\/uploads\/2026\/04\/image-45-768x263.png 768w\" sizes=\"auto, (max-width: 854px) 100vw, 854px\" \/><\/figure>\n\n\n\n<p>Our data included analysis of <strong>8 residents and 2 attendings across ~45\u201351 surgical procedures<\/strong><\/p>\n\n\n\n<p>The data naturally had a paired structure (resident vs. attending on the same case). The would have attending one sphenoid and the resident would have the other sphenoid.<\/p>\n\n\n\n<p>Overall, the survey data provides subjective ratings, motion data captures objective kinematic metrics, and experience data defines proficiency levels based on surgical volume.<br><\/p>\n\n\n\n<h2 class=\"wp-block-heading\">III. Exploratory Data Analysis<\/h2>\n\n\n\n<p>The EDA focused on transforming raw surgical motion data into features that better captured operative efficiency and resident\u2013attending differences. Time metrics were decomposed into static, idle, and active times, path-based efficiency measures were created, and skewed variables such as time, distance, and jerk were log-transformed to reduce the influence of extreme cases. Because each resident and attending worked on the same case, the data was reorganized into a paired wide format, allowing direct comparison through raw and percent difference metrics.<\/p>\n\n\n\n<p>Overall, the exploratory analysis showed clear differences between resident and attending motion patterns. Residents generally required more time, had longer idle and non-static periods, and produced larger movement paths, suggesting lower motion efficiency and less direct instrument use. Some of the strongest differences appeared in mean static time, scope idle time, scope non-static time, and Hosemann total path length. Jerk metrics showed a more mixed pattern: attendings sometimes had higher jerk values, which may reflect faster and more decisive movements rather than poorer control.<\/p>\n\n\n\n<p>Experience and performance analyses suggested that sphenoidotomy-specific experience was more meaningful than total surgical experience. Residents with more sphenoidotomy cases tended to receive higher OSATS-related performance scores, especially for time-motion skills, while total case volume showed weaker and less consistent relationships with performance.<\/p>\n\n\n\n<figure class=\"wp-block-image size-large\"><img loading=\"lazy\" decoding=\"async\" width=\"1024\" height=\"319\" src=\"https:\/\/www.hajim.rochester.edu\/senior-design-day\/wp-content\/uploads\/2026\/05\/image-102-1024x319.png\" alt=\"\" class=\"wp-image-242352\" title=\"attVSexp.png\" srcset=\"https:\/\/www.hajim.rochester.edu\/senior-design-day\/wp-content\/uploads\/2026\/05\/image-102-1024x319.png 1024w, https:\/\/www.hajim.rochester.edu\/senior-design-day\/wp-content\/uploads\/2026\/05\/image-102-300x94.png 300w, https:\/\/www.hajim.rochester.edu\/senior-design-day\/wp-content\/uploads\/2026\/05\/image-102-768x239.png 768w, https:\/\/www.hajim.rochester.edu\/senior-design-day\/wp-content\/uploads\/2026\/05\/image-102-1536x479.png 1536w, https:\/\/www.hajim.rochester.edu\/senior-design-day\/wp-content\/uploads\/2026\/05\/image-102.png 1588w\" sizes=\"auto, (max-width: 1024px) 100vw, 1024px\" \/><\/figure>\n\n\n\n<figure class=\"wp-block-image size-large\"><img loading=\"lazy\" decoding=\"async\" width=\"1024\" height=\"319\" src=\"https:\/\/www.hajim.rochester.edu\/senior-design-day\/wp-content\/uploads\/2026\/05\/image-103-1024x319.png\" alt=\"\" class=\"wp-image-242362\" title=\"attVSTOTALexp.png\" srcset=\"https:\/\/www.hajim.rochester.edu\/senior-design-day\/wp-content\/uploads\/2026\/05\/image-103-1024x319.png 1024w, https:\/\/www.hajim.rochester.edu\/senior-design-day\/wp-content\/uploads\/2026\/05\/image-103-300x93.png 300w, https:\/\/www.hajim.rochester.edu\/senior-design-day\/wp-content\/uploads\/2026\/05\/image-103-768x239.png 768w, https:\/\/www.hajim.rochester.edu\/senior-design-day\/wp-content\/uploads\/2026\/05\/image-103-1536x478.png 1536w, https:\/\/www.hajim.rochester.edu\/senior-design-day\/wp-content\/uploads\/2026\/05\/image-103.png 1590w\" sizes=\"auto, (max-width: 1024px) 100vw, 1024px\" \/><\/figure>\n\n\n\n<p>Regression analysis further showed that performance was most closely tied to efficiency-related motion features rather than every motion metric equally. In particular, suction-related time inefficiency was negatively associated with performance, meaning residents who spent proportionally more time using suction relative to attendings tended to receive lower ratings. Normalized movement-efficiency measures were also meaningful predictors, suggesting that strong performance depends not only on how long instruments are used, but how efficiently motion is produced during active operative time.<\/p>\n\n\n\n<figure class=\"wp-block-image size-full\"><img loading=\"lazy\" decoding=\"async\" width=\"712\" height=\"205\" src=\"https:\/\/www.hajim.rochester.edu\/senior-design-day\/wp-content\/uploads\/2026\/05\/image-104.png\" alt=\"\" class=\"wp-image-242402\" srcset=\"https:\/\/www.hajim.rochester.edu\/senior-design-day\/wp-content\/uploads\/2026\/05\/image-104.png 712w, https:\/\/www.hajim.rochester.edu\/senior-design-day\/wp-content\/uploads\/2026\/05\/image-104-300x86.png 300w\" sizes=\"auto, (max-width: 712px) 100vw, 712px\" \/><\/figure>\n\n\n\n<h2 class=\"wp-block-heading\">IV. Statistical Hypothesis Testing<\/h2>\n\n\n\n<p>We tested whether resident performance was related to operative participation, case difficulty, experience level, and motion-tracking features. Because the data included ordinal ratings, paired resident-attending comparisons, and non-normal motion variables, we used Spearman correlations, Wilcoxon signed-rank tests, and Cohen\u2019s kappa.<\/p>\n\n\n\n<p><strong>Greater participation was linked to higher OSATS ratings<\/strong><br>Residents who completed a larger portion of the procedure tended to receive higher attending-rated technical skill, flow, and overall performance scores.<br><strong>Key result:<\/strong> \u03c1 = 0.44\u20130.52, p \u2264 0.004<\/p>\n\n\n\n<p><strong>Higher technical challenge was linked to lower performance<\/strong><br>More difficult cases were associated with lower attending-rated time\/motion skill, flow, and overall sphenoidotomy performance.<br><strong>Key result:<\/strong> \u03c1 = -0.325 to -0.385, p &lt; 0.05<\/p>\n\n\n\n<figure class=\"wp-block-image size-large\"><img loading=\"lazy\" decoding=\"async\" width=\"1024\" height=\"586\" src=\"https:\/\/www.hajim.rochester.edu\/senior-design-day\/wp-content\/uploads\/2026\/05\/tech_challenge-1024x586.png\" alt=\"\" class=\"wp-image-236812\" srcset=\"https:\/\/www.hajim.rochester.edu\/senior-design-day\/wp-content\/uploads\/2026\/05\/tech_challenge-1024x586.png 1024w, https:\/\/www.hajim.rochester.edu\/senior-design-day\/wp-content\/uploads\/2026\/05\/tech_challenge-300x172.png 300w, https:\/\/www.hajim.rochester.edu\/senior-design-day\/wp-content\/uploads\/2026\/05\/tech_challenge-768x439.png 768w, https:\/\/www.hajim.rochester.edu\/senior-design-day\/wp-content\/uploads\/2026\/05\/tech_challenge.png 1100w\" sizes=\"auto, (max-width: 1024px) 100vw, 1024px\" \/><\/figure>\n\n\n\n<p><strong>OSATS domains were internally consistent<\/strong><br>Technical skill, flow, and overall performance were strongly correlated, suggesting that these rating categories captured closely related aspects of operative competency.<br><strong>Key result:<\/strong> \u03c1 = 0.68\u20130.75, p &lt; 0.001<\/p>\n\n\n\n<p><strong>Residents also recognized performance patterns<\/strong><br>Resident self-ratings showed similar trends: greater participation was associated with higher self-rated performance, while higher technical challenge was associated with lower self-rated performance.<br><strong>Key result:<\/strong> participation \u03c1 = 0.52, p &lt; 0.001; challenge \u03c1 = -0.41, p = 0.008<\/p>\n\n\n\n<p><strong>Motion differences were strongest in efficiency metrics<\/strong><br>Paired resident-attending comparisons showed significant differences in total time, idle time, and static time, indicating that the clearest motion-based gap was procedural efficiency.<br><strong>Key result:<\/strong> total time p = 0.000008; idle time p = 0.000024; static time p = 0.000029<\/p>\n\n\n\n<h2 class=\"wp-block-heading\">V. Experience Level Prediction<\/h2>\n\n\n\n<p>A resident-only machine learning model was built to classify sphenoidotomy procedures as Advanced or Non-Advanced using motion-based features. To keep the model generalizable and avoid leakage, we removed attending-only variables, resident\/attending ratios, identifiers, administrative fields, and direct experience labels. The final dataset included 40 procedures from 8 residents, split evenly into 20 Advanced and 20 Non-Advanced cases after combining the single beginner case with the intermediate group.<\/p>\n\n\n<div class=\"wp-block-image\">\n<figure class=\"aligncenter size-full is-resized\"><img loading=\"lazy\" decoding=\"async\" width=\"614\" height=\"411\" src=\"https:\/\/www.hajim.rochester.edu\/senior-design-day\/wp-content\/uploads\/2026\/05\/image-98.png\" alt=\"\" class=\"wp-image-242252\" style=\"width:728px;height:auto\" srcset=\"https:\/\/www.hajim.rochester.edu\/senior-design-day\/wp-content\/uploads\/2026\/05\/image-98.png 614w, https:\/\/www.hajim.rochester.edu\/senior-design-day\/wp-content\/uploads\/2026\/05\/image-98-300x201.png 300w\" sizes=\"auto, (max-width: 614px) 100vw, 614px\" \/><\/figure>\n<\/div>\n\n\n<p>Starting from 111 motion features, LASSO logistic regression was used to narrow the feature set to 12 predictors. Next, highly correlated\/redundant variables were removed to produce a final set of four resident-focused features capturing motion efficiency, smoothness, and performance-related behavior. These features were used in a tuned RBF-kernel SVM with median imputation, standardization, and GridSearchCV hyperparameter tuning. Because the dataset was small, performance was evaluated using leave-one-out cross-validation.<\/p>\n\n\n<div class=\"wp-block-image\">\n<figure class=\"aligncenter size-full\"><img loading=\"lazy\" decoding=\"async\" width=\"483\" height=\"111\" src=\"https:\/\/www.hajim.rochester.edu\/senior-design-day\/wp-content\/uploads\/2026\/05\/image-96.png\" alt=\"\" class=\"wp-image-242232\" srcset=\"https:\/\/www.hajim.rochester.edu\/senior-design-day\/wp-content\/uploads\/2026\/05\/image-96.png 483w, https:\/\/www.hajim.rochester.edu\/senior-design-day\/wp-content\/uploads\/2026\/05\/image-96-300x69.png 300w\" sizes=\"auto, (max-width: 483px) 100vw, 483px\" \/><\/figure>\n<\/div>\n\n\n<p>The observed accuracy of 0.85 corresponded to a p-value of 0.0020, meaning that only 0.2% of the permuted runs achieved similar or better performance. This suggests the model is capturing a real relationship between the kinematic features and surgical expertise, rather than fitting to noise, even with<br>the small sample size.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\">VI. OSATS<\/h2>\n\n\n\n<p>We tested whether objective motion-tracking and procedural variables could predict attending-rated OSATS performance. The main outcome was attending-rated resident time-motion skill, while overall sphenoidotomy performance was used as a secondary outcome.<\/p>\n\n\n\n<figure class=\"wp-block-table\"><table class=\"has-fixed-layout\"><thead><tr><th>Model<\/th><th>Target<\/th><th>Main Predictors<\/th><th>R\u00b2<\/th><th>RMSE<\/th><th>MAE<\/th><\/tr><\/thead><tbody><tr><td>Elastic Net baseline<\/td><td>Time-motion skill<\/td><td>Experience, completion %, idle-time difference<\/td><td>0.552<\/td><td>0.694<\/td><td>0.564<\/td><\/tr><tr><td>Gradient Boosting<\/td><td>Time-motion skill<\/td><td>Experience, completion %, idle-time difference, engineered gap features<\/td><td>0.634<\/td><td>0.628<\/td><td>0.500<\/td><\/tr><tr><td>Elastic Net baseline<\/td><td>Overall performance<\/td><td>Experience, completion %, idle-time difference<\/td><td>0.532<\/td><td>0.660<\/td><td>0.551<\/td><\/tr><\/tbody><\/table><\/figure>\n\n\n\n<p><strong>Baseline model showed meaningful prediction<\/strong><br>The Elastic Net model used prior sphenoidotomy experience, attending-rated sphenoidotomy completion percentage, and mean idle-time difference.<br><strong>Key result:<\/strong> R\u00b2 = 0.552 for predicting time-motion skill<\/p>\n\n\n\n<p><strong>Gradient Boosting performed best<\/strong><br>Adding engineered resident-attending motion-gap features improved prediction of attending-rated time-motion skill.<br><strong>Key result:<\/strong> R\u00b2 = 0.634, RMSE = 0.628, MAE = 0.500<\/p>\n\n\n\n<p><strong>Top predictors were clinically interpretable<\/strong><br>The strongest predictors were prior sphenoidotomy experience, mean idle-time difference, and attending-rated sphenoidotomy completion percentage.<\/p>\n\n\n\n<figure class=\"wp-block-image size-large\"><img loading=\"lazy\" decoding=\"async\" width=\"1024\" height=\"550\" src=\"https:\/\/www.hajim.rochester.edu\/senior-design-day\/wp-content\/uploads\/2026\/05\/osats_features-1024x550.png\" alt=\"\" class=\"wp-image-236762\" srcset=\"https:\/\/www.hajim.rochester.edu\/senior-design-day\/wp-content\/uploads\/2026\/05\/osats_features-1024x550.png 1024w, https:\/\/www.hajim.rochester.edu\/senior-design-day\/wp-content\/uploads\/2026\/05\/osats_features-300x161.png 300w, https:\/\/www.hajim.rochester.edu\/senior-design-day\/wp-content\/uploads\/2026\/05\/osats_features-768x413.png 768w, https:\/\/www.hajim.rochester.edu\/senior-design-day\/wp-content\/uploads\/2026\/05\/osats_features.png 1063w\" sizes=\"auto, (max-width: 1024px) 100vw, 1024px\" \/><\/figure>\n\n\n\n<p><br><strong>Main takeaway:<\/strong> performance was most strongly linked to experience, procedural completion, and efficiency.<\/p>\n\n\n\n<p><strong>Overall performance was also predictable<\/strong><br>The same baseline predictor set performed well for attending-rated overall sphenoidotomy performance.<br><strong>Key result:<\/strong> R\u00b2 = 0.532, RMSE = 0.660, MAE = 0.551<\/p>\n\n\n\n<p><strong>Main conclusion<\/strong><br>OSATS ratings can be meaningfully predicted using a combination of resident experience, procedural completion, and time-efficiency features. This supports motion-tracking data as an objective supplement to traditional attending-based surgical assessment.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\">VII. K-Means Clustering<\/h2>\n\n\n\n<p>We used unsupervised clustering to explore whether surgical motion data naturally formed meaningful groups without relying on performance labels. Three K-means clustering approaches were tested: resident-only motion features, combined resident and attending motion features, and paired resident\u2013attending difference features. Across these analyses, the clusters reflected distinct motion patterns involving scope path length, jerk, active time, idle time, and non-scope movement behavior.<\/p>\n\n\n\n<p>The most informative results came from clustering the paired resident\u2013attending difference features, which preserved the matched structure of the dataset. These clusters were strongly separated by scope-based motion differences, including active path length, jerk, active time, idle time, and path efficiency measures. Unlike the resident-only clusters, the difference-based clusters showed a significant relationship with sphenoidotomy-specific experience and a near-significant relationship with attending-rated overall performance. This suggests that the motion gap between a resident and the attending on the same case may capture clinically meaningful variation better than resident motion alone.<\/p>\n\n\n\n<p>We also explored an attending-based similarity approach, where PCA and distance-based scoring were used to compare each resident\u2019s motion profile to attending motion patterns. More experienced residents generally had higher similarity scores, while less experienced residents showed larger improvement over time as their motion patterns moved closer to attending behavior. Overall, the clustering analysis showed that unsupervised learning can reveal hidden surgical motion patterns and may provide a useful foundation for automated, objective feedback in surgical training.<\/p>\n\n\n<div class=\"wp-block-image\">\n<figure class=\"aligncenter size-full\"><img loading=\"lazy\" decoding=\"async\" width=\"601\" height=\"414\" src=\"https:\/\/www.hajim.rochester.edu\/senior-design-day\/wp-content\/uploads\/2026\/05\/image-99.png\" alt=\"\" class=\"wp-image-242272\" srcset=\"https:\/\/www.hajim.rochester.edu\/senior-design-day\/wp-content\/uploads\/2026\/05\/image-99.png 601w, https:\/\/www.hajim.rochester.edu\/senior-design-day\/wp-content\/uploads\/2026\/05\/image-99-300x207.png 300w\" sizes=\"auto, (max-width: 601px) 100vw, 601px\" \/><\/figure>\n<\/div>\n\n<div class=\"wp-block-image\">\n<figure class=\"aligncenter size-full\"><img loading=\"lazy\" decoding=\"async\" width=\"628\" height=\"439\" src=\"https:\/\/www.hajim.rochester.edu\/senior-design-day\/wp-content\/uploads\/2026\/05\/image-101.png\" alt=\"\" class=\"wp-image-242302\" srcset=\"https:\/\/www.hajim.rochester.edu\/senior-design-day\/wp-content\/uploads\/2026\/05\/image-101.png 628w, https:\/\/www.hajim.rochester.edu\/senior-design-day\/wp-content\/uploads\/2026\/05\/image-101-300x210.png 300w\" sizes=\"auto, (max-width: 628px) 100vw, 628px\" \/><\/figure>\n<\/div>\n\n\n<h2 class=\"wp-block-heading\">VIII. Conclusions<\/h2>\n\n\n\n<p>This study investigated how well surgical motion-tracking data could predict resident performance during sphenoidotomy and support more objective feedback for surgical training. Exploratory analysis, hypothesis testing, and preliminary supervised and unsupervised modeling showed that time-efficiency metrics were the strongest indicators of attending OSATS ratings.<\/p>\n\n\n\n<p>Residents with more experience generally received higher OSATS scores, especially in time-motion skills. Motion analysis further showed that resident-attending differences were most consistent in temporal efficiency, including higher scope idle time, static time, and mean idle time for residents. In contrast, path length and jerk metrics were less consistently informative, suggesting that performance gaps were driven more by how efficiently residents used time than by overall movement magnitude or smoothness.<\/p>\n\n\n\n<p>Experience-level classification also showed promising results. Using only four resident motion features, an SVM classifier achieved 85% accuracy under LOOCV in distinguishing advanced from non-advanced residents. This suggests that motion features alone may support scalable proficiency assessment without requiring paired attending observations.<\/p>\n\n\n\n<p>The main limitation was the small sample size, so future work should validate these findings with more data. Additional integration of computer vision could help capture visual and spatial aspects of performance, supporting the development of an automated resident feedback tool to complement existing surgical assessment methods.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\">IX. Acknowledgements<\/h2>\n\n\n\n<p>We would like to acknowledge the URMC Otolaryngology team\u2014particularly Dr. Schmale and Dr. Ryan\u2014for their sponsorship and support throughout this project. We also sincerely thank Professor Anand of the Goergen Institute for Data Science for his guidance and expertise.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\">X. References<\/h2>\n\n\n\n<ol class=\"wp-block-list\">\n<li>Lindgren, F., Bj\u00f8rn Gunnar Hansen, Karcher, W., Sj\u00f6str\u00f3m, M., &amp; Eriksson, L. (1996). Model validation by permutation tests: Applications to variable selection. Journal of Chemometrics, 10(5-6), 521\u2013532. <a href=\"https:\/\/doi.org\/10.1002\/(sici)1099-128x(199609)10:5\/6%3C521::aid-cem448%3E3.0.co;2-j\">https:\/\/doi.org\/10.1002\/(sici)1099-128x(199609)10:5\/6%3C521::aid-cem448%3E3.0.co;2-j<\/a><\/li>\n\n\n\n<li>Lam, K., Cheng, A., Haider, H., Jafferji, M., Dasgupta, P., Ahmed, K., &amp; Khan, M. S. (2022). Machine learning for technical skill assessment in surgery: A systematic review. <em>NPJ Digital Medicine, 5<\/em>, Article 24.<\/li>\n<\/ol>\n","protected":false},"excerpt":{"rendered":"<p>I. Introduction Reliable assessment of surgical performance is essential for resident education, feedback, and progression within procedural training. Although attending-based evaluations such as the Objective Structured assessment of Technical Skills&hellip;<\/p>\n","protected":false},"author":20682,"featured_media":0,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"_coblocks_attr":"","_coblocks_dimensions":"","_coblocks_responsive_height":"","_coblocks_accordion_ie_support":"","_monsterinsights_skip_tracking":false,"_monsterinsights_sitenote_active":false,"_monsterinsights_sitenote_note":"","_monsterinsights_sitenote_category":0,"footnotes":""},"categories":[4452,4512],"tags":[],"coauthors":[25672,25682,25692,25702,25712],"class_list":["post-216892","post","type-post","status-publish","format-standard","hentry","category-current-year","category-dsc"],"acf":[],"yoast_head":"<!-- This site is optimized with the Yoast SEO plugin v27.7 - https:\/\/yoast.com\/product\/yoast-seo-wordpress\/ -->\n<title>URMC-Otolarnyology: Predicting Sphenoidotomy Performance in Residents - Senior Design Day<\/title>\n<meta name=\"description\" content=\"Research analysis on objective assessment in surgical education. This project utilizes motion-tracking data and machine learning to predict resident performance and experience levels during sphenoidotomy, offering a quantitative complement to traditional attending-based evaluations.\" \/>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/www.hajim.rochester.edu\/senior-design-day\/urmc-otolarnyology-predicting-sphenoidotomy-performance-in-residents\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"URMC-Otolarnyology: Predicting Sphenoidotomy Performance in Residents - Senior Design Day\" \/>\n<meta property=\"og:description\" content=\"Research analysis on objective assessment in surgical education. This project utilizes motion-tracking data and machine learning to predict resident performance and experience levels during sphenoidotomy, offering a quantitative complement to traditional attending-based evaluations.\" \/>\n<meta property=\"og:url\" content=\"https:\/\/www.hajim.rochester.edu\/senior-design-day\/urmc-otolarnyology-predicting-sphenoidotomy-performance-in-residents\/\" \/>\n<meta property=\"og:site_name\" content=\"Senior Design Day\" \/>\n<meta property=\"article:published_time\" content=\"2026-05-04T15:41:34+00:00\" \/>\n<meta property=\"og:image\" content=\"https:\/\/www.hajim.rochester.edu\/senior-design-day\/wp-content\/uploads\/2026\/05\/sphen-929x630.jpg\" \/>\n\t<meta property=\"og:image:width\" content=\"929\" \/>\n\t<meta property=\"og:image:height\" content=\"630\" \/>\n\t<meta property=\"og:image:type\" content=\"image\/jpeg\" \/>\n<meta name=\"author\" content=\"Nicole Cassara, Jayla Hall, Marie Kinchington, Soumya Konar, Calvin Tian\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:label1\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data1\" content=\"Nicole Cassara, Jayla Hall, Marie Kinchington, Soumya Konar, Calvin Tian\" \/>\n\t<meta name=\"twitter:label2\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data2\" content=\"11 minutes\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\\\/\\\/schema.org\",\"@graph\":[{\"@type\":\"Article\",\"@id\":\"https:\\\/\\\/www.hajim.rochester.edu\\\/senior-design-day\\\/urmc-otolarnyology-predicting-sphenoidotomy-performance-in-residents\\\/#article\",\"isPartOf\":{\"@id\":\"https:\\\/\\\/www.hajim.rochester.edu\\\/senior-design-day\\\/urmc-otolarnyology-predicting-sphenoidotomy-performance-in-residents\\\/\"},\"author\":{\"name\":\"Nicole Cassara\",\"@id\":\"https:\\\/\\\/www.hajim.rochester.edu\\\/senior-design-day\\\/#\\\/schema\\\/person\\\/e63d995331056db171f33e127fd4dedb\"},\"headline\":\"URMC-Otolarnyology: Predicting Sphenoidotomy Performance in Residents\",\"datePublished\":\"2026-05-04T15:41:34+00:00\",\"mainEntityOfPage\":{\"@id\":\"https:\\\/\\\/www.hajim.rochester.edu\\\/senior-design-day\\\/urmc-otolarnyology-predicting-sphenoidotomy-performance-in-residents\\\/\"},\"wordCount\":1804,\"image\":{\"@id\":\"https:\\\/\\\/www.hajim.rochester.edu\\\/senior-design-day\\\/urmc-otolarnyology-predicting-sphenoidotomy-performance-in-residents\\\/#primaryimage\"},\"thumbnailUrl\":\"https:\\\/\\\/www.hajim.rochester.edu\\\/senior-design-day\\\/wp-content\\\/uploads\\\/2026\\\/05\\\/sphen-688x1024.jpg\",\"articleSection\":[\"1. Current Year Programs\",\"DSC\"],\"inLanguage\":\"en-US\"},{\"@type\":\"WebPage\",\"@id\":\"https:\\\/\\\/www.hajim.rochester.edu\\\/senior-design-day\\\/urmc-otolarnyology-predicting-sphenoidotomy-performance-in-residents\\\/\",\"url\":\"https:\\\/\\\/www.hajim.rochester.edu\\\/senior-design-day\\\/urmc-otolarnyology-predicting-sphenoidotomy-performance-in-residents\\\/\",\"name\":\"URMC-Otolarnyology: Predicting Sphenoidotomy Performance in Residents - Senior Design Day\",\"isPartOf\":{\"@id\":\"https:\\\/\\\/www.hajim.rochester.edu\\\/senior-design-day\\\/#website\"},\"primaryImageOfPage\":{\"@id\":\"https:\\\/\\\/www.hajim.rochester.edu\\\/senior-design-day\\\/urmc-otolarnyology-predicting-sphenoidotomy-performance-in-residents\\\/#primaryimage\"},\"image\":{\"@id\":\"https:\\\/\\\/www.hajim.rochester.edu\\\/senior-design-day\\\/urmc-otolarnyology-predicting-sphenoidotomy-performance-in-residents\\\/#primaryimage\"},\"thumbnailUrl\":\"https:\\\/\\\/www.hajim.rochester.edu\\\/senior-design-day\\\/wp-content\\\/uploads\\\/2026\\\/05\\\/sphen-688x1024.jpg\",\"datePublished\":\"2026-05-04T15:41:34+00:00\",\"author\":{\"@id\":\"https:\\\/\\\/www.hajim.rochester.edu\\\/senior-design-day\\\/#\\\/schema\\\/person\\\/e63d995331056db171f33e127fd4dedb\"},\"description\":\"Research analysis on objective assessment in surgical education. This project utilizes motion-tracking data and machine learning to predict resident performance and experience levels during sphenoidotomy, offering a quantitative complement to traditional attending-based evaluations.\",\"breadcrumb\":{\"@id\":\"https:\\\/\\\/www.hajim.rochester.edu\\\/senior-design-day\\\/urmc-otolarnyology-predicting-sphenoidotomy-performance-in-residents\\\/#breadcrumb\"},\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\\\/\\\/www.hajim.rochester.edu\\\/senior-design-day\\\/urmc-otolarnyology-predicting-sphenoidotomy-performance-in-residents\\\/\"]}]},{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\\\/\\\/www.hajim.rochester.edu\\\/senior-design-day\\\/urmc-otolarnyology-predicting-sphenoidotomy-performance-in-residents\\\/#primaryimage\",\"url\":\"https:\\\/\\\/www.hajim.rochester.edu\\\/senior-design-day\\\/wp-content\\\/uploads\\\/2026\\\/05\\\/sphen.jpg\",\"contentUrl\":\"https:\\\/\\\/www.hajim.rochester.edu\\\/senior-design-day\\\/wp-content\\\/uploads\\\/2026\\\/05\\\/sphen.jpg\",\"width\":929,\"height\":1383},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\\\/\\\/www.hajim.rochester.edu\\\/senior-design-day\\\/urmc-otolarnyology-predicting-sphenoidotomy-performance-in-residents\\\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Home\",\"item\":\"https:\\\/\\\/www.hajim.rochester.edu\\\/senior-design-day\\\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"URMC-Otolarnyology: Predicting Sphenoidotomy Performance in Residents\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\\\/\\\/www.hajim.rochester.edu\\\/senior-design-day\\\/#website\",\"url\":\"https:\\\/\\\/www.hajim.rochester.edu\\\/senior-design-day\\\/\",\"name\":\"Senior Design Day\",\"description\":\"\",\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\\\/\\\/www.hajim.rochester.edu\\\/senior-design-day\\\/?s={search_term_string}\"},\"query-input\":{\"@type\":\"PropertyValueSpecification\",\"valueRequired\":true,\"valueName\":\"search_term_string\"}}],\"inLanguage\":\"en-US\"},{\"@type\":\"Person\",\"@id\":\"https:\\\/\\\/www.hajim.rochester.edu\\\/senior-design-day\\\/#\\\/schema\\\/person\\\/e63d995331056db171f33e127fd4dedb\",\"name\":\"Nicole Cassara\",\"url\":\"https:\\\/\\\/www.hajim.rochester.edu\\\/senior-design-day\\\/author\\\/ncassara\\\/\"}]}<\/script>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"URMC-Otolarnyology: Predicting Sphenoidotomy Performance in Residents - Senior Design Day","description":"Research analysis on objective assessment in surgical education. This project utilizes motion-tracking data and machine learning to predict resident performance and experience levels during sphenoidotomy, offering a quantitative complement to traditional attending-based evaluations.","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/www.hajim.rochester.edu\/senior-design-day\/urmc-otolarnyology-predicting-sphenoidotomy-performance-in-residents\/","og_locale":"en_US","og_type":"article","og_title":"URMC-Otolarnyology: Predicting Sphenoidotomy Performance in Residents - Senior Design Day","og_description":"Research analysis on objective assessment in surgical education. This project utilizes motion-tracking data and machine learning to predict resident performance and experience levels during sphenoidotomy, offering a quantitative complement to traditional attending-based evaluations.","og_url":"https:\/\/www.hajim.rochester.edu\/senior-design-day\/urmc-otolarnyology-predicting-sphenoidotomy-performance-in-residents\/","og_site_name":"Senior Design Day","article_published_time":"2026-05-04T15:41:34+00:00","og_image":[{"width":929,"height":630,"url":"https:\/\/www.hajim.rochester.edu\/senior-design-day\/wp-content\/uploads\/2026\/05\/sphen-929x630.jpg","type":"image\/jpeg"}],"author":"Nicole Cassara, Jayla Hall, Marie Kinchington, Soumya Konar, Calvin Tian","twitter_card":"summary_large_image","twitter_misc":{"Written by":"Nicole Cassara, Jayla Hall, Marie Kinchington, Soumya Konar, Calvin Tian","Est. reading time":"11 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"Article","@id":"https:\/\/www.hajim.rochester.edu\/senior-design-day\/urmc-otolarnyology-predicting-sphenoidotomy-performance-in-residents\/#article","isPartOf":{"@id":"https:\/\/www.hajim.rochester.edu\/senior-design-day\/urmc-otolarnyology-predicting-sphenoidotomy-performance-in-residents\/"},"author":{"name":"Nicole Cassara","@id":"https:\/\/www.hajim.rochester.edu\/senior-design-day\/#\/schema\/person\/e63d995331056db171f33e127fd4dedb"},"headline":"URMC-Otolarnyology: Predicting Sphenoidotomy Performance in Residents","datePublished":"2026-05-04T15:41:34+00:00","mainEntityOfPage":{"@id":"https:\/\/www.hajim.rochester.edu\/senior-design-day\/urmc-otolarnyology-predicting-sphenoidotomy-performance-in-residents\/"},"wordCount":1804,"image":{"@id":"https:\/\/www.hajim.rochester.edu\/senior-design-day\/urmc-otolarnyology-predicting-sphenoidotomy-performance-in-residents\/#primaryimage"},"thumbnailUrl":"https:\/\/www.hajim.rochester.edu\/senior-design-day\/wp-content\/uploads\/2026\/05\/sphen-688x1024.jpg","articleSection":["1. Current Year Programs","DSC"],"inLanguage":"en-US"},{"@type":"WebPage","@id":"https:\/\/www.hajim.rochester.edu\/senior-design-day\/urmc-otolarnyology-predicting-sphenoidotomy-performance-in-residents\/","url":"https:\/\/www.hajim.rochester.edu\/senior-design-day\/urmc-otolarnyology-predicting-sphenoidotomy-performance-in-residents\/","name":"URMC-Otolarnyology: Predicting Sphenoidotomy Performance in Residents - Senior Design Day","isPartOf":{"@id":"https:\/\/www.hajim.rochester.edu\/senior-design-day\/#website"},"primaryImageOfPage":{"@id":"https:\/\/www.hajim.rochester.edu\/senior-design-day\/urmc-otolarnyology-predicting-sphenoidotomy-performance-in-residents\/#primaryimage"},"image":{"@id":"https:\/\/www.hajim.rochester.edu\/senior-design-day\/urmc-otolarnyology-predicting-sphenoidotomy-performance-in-residents\/#primaryimage"},"thumbnailUrl":"https:\/\/www.hajim.rochester.edu\/senior-design-day\/wp-content\/uploads\/2026\/05\/sphen-688x1024.jpg","datePublished":"2026-05-04T15:41:34+00:00","author":{"@id":"https:\/\/www.hajim.rochester.edu\/senior-design-day\/#\/schema\/person\/e63d995331056db171f33e127fd4dedb"},"description":"Research analysis on objective assessment in surgical education. This project utilizes motion-tracking data and machine learning to predict resident performance and experience levels during sphenoidotomy, offering a quantitative complement to traditional attending-based evaluations.","breadcrumb":{"@id":"https:\/\/www.hajim.rochester.edu\/senior-design-day\/urmc-otolarnyology-predicting-sphenoidotomy-performance-in-residents\/#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/www.hajim.rochester.edu\/senior-design-day\/urmc-otolarnyology-predicting-sphenoidotomy-performance-in-residents\/"]}]},{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/www.hajim.rochester.edu\/senior-design-day\/urmc-otolarnyology-predicting-sphenoidotomy-performance-in-residents\/#primaryimage","url":"https:\/\/www.hajim.rochester.edu\/senior-design-day\/wp-content\/uploads\/2026\/05\/sphen.jpg","contentUrl":"https:\/\/www.hajim.rochester.edu\/senior-design-day\/wp-content\/uploads\/2026\/05\/sphen.jpg","width":929,"height":1383},{"@type":"BreadcrumbList","@id":"https:\/\/www.hajim.rochester.edu\/senior-design-day\/urmc-otolarnyology-predicting-sphenoidotomy-performance-in-residents\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/www.hajim.rochester.edu\/senior-design-day\/"},{"@type":"ListItem","position":2,"name":"URMC-Otolarnyology: Predicting Sphenoidotomy Performance in Residents"}]},{"@type":"WebSite","@id":"https:\/\/www.hajim.rochester.edu\/senior-design-day\/#website","url":"https:\/\/www.hajim.rochester.edu\/senior-design-day\/","name":"Senior Design Day","description":"","potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/www.hajim.rochester.edu\/senior-design-day\/?s={search_term_string}"},"query-input":{"@type":"PropertyValueSpecification","valueRequired":true,"valueName":"search_term_string"}}],"inLanguage":"en-US"},{"@type":"Person","@id":"https:\/\/www.hajim.rochester.edu\/senior-design-day\/#\/schema\/person\/e63d995331056db171f33e127fd4dedb","name":"Nicole Cassara","url":"https:\/\/www.hajim.rochester.edu\/senior-design-day\/author\/ncassara\/"}]}},"_links":{"self":[{"href":"https:\/\/www.hajim.rochester.edu\/senior-design-day\/wp-json\/wp\/v2\/posts\/216892","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.hajim.rochester.edu\/senior-design-day\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.hajim.rochester.edu\/senior-design-day\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.hajim.rochester.edu\/senior-design-day\/wp-json\/wp\/v2\/users\/20682"}],"replies":[{"embeddable":true,"href":"https:\/\/www.hajim.rochester.edu\/senior-design-day\/wp-json\/wp\/v2\/comments?post=216892"}],"version-history":[{"count":17,"href":"https:\/\/www.hajim.rochester.edu\/senior-design-day\/wp-json\/wp\/v2\/posts\/216892\/revisions"}],"predecessor-version":[{"id":246002,"href":"https:\/\/www.hajim.rochester.edu\/senior-design-day\/wp-json\/wp\/v2\/posts\/216892\/revisions\/246002"}],"wp:attachment":[{"href":"https:\/\/www.hajim.rochester.edu\/senior-design-day\/wp-json\/wp\/v2\/media?parent=216892"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.hajim.rochester.edu\/senior-design-day\/wp-json\/wp\/v2\/categories?post=216892"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.hajim.rochester.edu\/senior-design-day\/wp-json\/wp\/v2\/tags?post=216892"},{"taxonomy":"author","embeddable":true,"href":"https:\/\/www.hajim.rochester.edu\/senior-design-day\/wp-json\/wp\/v2\/coauthors?post=216892"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}