Multimodal 3D Fusion and In-Situ Learning for Spatially Aware AI
Chengyuan Xu, Radha Kumaran, Noah Stier, Kangyou Yu, Tobias Höllerer
ISMAR 2024
code / arXiv

We develop a vision-language 3D reconstruction system, anchoring CLIP features to the reconstructed mesh to support two augmented reality applications on the Magic Leap 2: spatial search with free-text natural language queries, and intelligent object inventory tracking.

Smoothness, Synthesis, and Sampling: Re-thinking Unsupervised Multi-View Stereo with DIV Loss
Alex Rich, Noah Stier, Pradeep Sen, Tobias Höllerer
ECCV 2024 (Oral Presentation)

We re-structure the core training objective for unsupervised MVS, allowing our networks to learn far better object boundaries for clean and coherent 3D reconstructions without requiring depth/3D ground truth.

FineRecon: Depth-aware Feed-forward Network for Detailed 3D Reconstruction
Noah Stier, Anurag Ranjan, Alex Colburn, Yajie Yan, Liang Yang, Fangchang Ma, Baptiste Angles
ICCV 2023
code / arXiv / supplementary

We improve the reconstruction of high-frequency content by fixing a fundamental sampling error for volumetric TSDF ground truth. We also leverage depth guidance from MVS, and a new point back-projection architecture, to achieve highly accurate and detailed reconstructions.

LivePose: Online 3D Reconstruction from Monocular Video with Dynamic Camera Poses
Noah Stier, Baptiste Angles, Liang Yang, Yajie Yan, Alex Colburn, Ming Chuang
ICCV 2023 (Oral Presentation)
data / arXiv

We propose neural de-integration to handle SLAM pose updates during online reconstruction from monocular video. To study this problem, we introduce the LivePose dataset of online SLAM pose estimates on ScanNet, the first to include full pose streams with online updates.

Interactive Segmentation and Visualization for Tiny Objects in Multi-megapixel Images
Chengyuan Xu, Boning Dong, Noah Stier, Curtis McCully, D. Andrew Howell, Pradeep Sen, Tobias Höllerer
CVPR 2022, demo track

We present an open-source software toolkit for identifying, inspecting, and editing tiny objects in multi-megapixel HDR images. These tools offer streamlined workflows for analyzing scientific images across many disciplines, such as astronomy, remote sensing, and biomedicine.

Weakly-Supervised Convolutional Neural Networks for Vessel Segmentation in Cerebral Angiography
Arvind Vepa, Andrew Choi, Noor Nakhaei, Wonjun Lee, Noah Stier, Andrew Vu, Greyson Jenkins, Xiaoyan Yang, Manjot Shergill, Moira Desphy, Kevin Delao, Mia Levy, Cristopher Garduno, Lacy Nelson, Wandi Liu, Fan Hung, Fabien Scalzo
WACV 2022

We present a high-quality, annotated DSA vessel segmentation dataset, the largest to-date that is publicly available. We identify semi-automated annotation methods that significantly reduce the labeling cost while preserving the ability to train accurate segmentation models.

VoRTX: Volumetric 3D Reconstruction with Transformers for Voxel-wise View Selection and Fusion
Noah Stier, Alex Rich, Pradeep Sen, Tobias Höllerer
3DV 2021 (Oral Presentation)
code / arXiv

We introduce a 3D reconstruction model with a novel multi-view fusion method based on transformers. It models occlusion by predicting projective occupancy, which reduces noise and leads to more detailed and complete reconstructions.

3DVNet: Multi-View Depth Prediction and Volumetric Refinement<
Alex Rich, Noah Stier, Pradeep Sen, Tobias Höllerer
3DV 2021

We introduce a deep multi-view stereo network that jointly models all depth maps in scene space, allowing it to learn geometric priors across entire scenes. It iteratively refines its predictions, leading to highly coherent reconstructions.

DeepOSM-3D: Recognition in Aerial LiDAR RGBD Imagery
Abhejit Rajagopal, Noah Stier, William Nelson, Shivkumar Chandrasekaran,
Andrew P Brown
SPIE DCS 2020

We present a framework for wide-area detection and recognition in co-incident aerial RGB images and LiDAR, and we source ground-truth semantic labels from OpenStreetMap.

Towards Deep Iterative-Reconstruction Algorithms for Computed Tomography (CT) Applications
Abhejit Rajagopal, Noah Stier, Joyoni Dey, Michael A King, Shivkumar Chandrasekaran
SPIE Medical Imaging (MI) 2019

We propose deep neural networks for CT that are equivalent to classical methods at initialization, but can be trained to optimize performance while maintaining convergence properties and interpretability.

Deep Learning of Tissue Fate Features in Acute Ischemic Stroke
Noah Stier, Nicholas Vincent, David S Liebeskind, Fabien Scalzo
IEEE Bioinformatics and Biomedicine (BIBM) 2015

We predict the extent of tissue damage following stroke, using the cerebral hypoperfusion feature observed in MRI after stroke onset. We show improved performance using a CNN vs. the standard single-voxel regression model.

Detection of Hyperperfusion on Arterial Spin Labeling Using Deep Learning
Nicholas Vincent, Noah Stier, Songlin Yu, David S Liebeskind, Danny JJ Wang, Fabien Scalzo
IEEE Bioinformatics and Biomedicine (BIBM) 2015

We train a CNN to classify regions of hyperperfusion based on cerebral blood flow (CBF) observed in MRI after stroke onset, approaching the accuracy of human experts.