intuitively, when a person is relaxed and has no task to perform, one tends to look at salient objects in the field of view (bottom-up). As tasks are introduced and workload increases, one usually tends to select a more task-oriented gaze behavior (top-down) and a shift from salient objects-oriented gaze patterns to important objects-oriented gaze patterns can be observed. In the automotive environment, this shift between gaze pattern types and their linkage to the driver or passengers’ states suggests that modeling a gaze pattern can lead to an understanding of one’s state and vice versa.

Gaze patterns were modeled by training both deep learning networks and statistical models. Deep learning networks were trained to digest effectively larger datasets, and statistical models were selected for their simplicity and explainability. A set of experiments was conducted both in real-world setups and in simulated environments. The real-world experiments took place in Israel and the USA while modeling the behavior of drivers and passengers. Overall, our results supported our assumptions and can be divided into two types: prediction of expected gaze patterns given the environment and establishing a linkage between gaze patterns and the driver’s and passenger’s state.

Lane detection is crucial for autonomous driving, furnishing indispensable data for safe navigation. Modern algorithms employ anchor-based detectors, followed by a label assignment process to categorize training detections as either positive or negative instances. However, the existing methods might be limited and not necessary optimal due to relying on predefined classic cost functions with not many calibration parameters.
Our research introduces MatchNet, a deep learning-based approach aimed at optimizing the label assignment process. Interwoven into a SOTA lane detection network such as CLRNet, MatchNet replaces the conventional label assignment process with a submodule network. This integration yields significant enhancements, particularly in challenging scenarios such as curve detection (+1.82%), shadows (+1.1%), and non-visible lane markings (+0.88%). Notably, this method boosts lane detection confidence level, enabling a 3% increase in the confidence threshold.

Integrating medical imaging with Electronic Health Records (EHRs) is crucial for comprehensive patient analysis. Deep Neural Networks excel in multimodal tasks in the medical domain, yet, the complex endeavor of effectively merging medical imaging with clinical, demographic and genetic information represented as numerical tabular data remains a highly active and ongoing research pursuit. We propose a novel hypernetwork-based framework for tabular-imaging fusion, where the image processing is conditioned on EHR values and measurements, thereby leveraging them to enhance the predictive results. Tested on brain MRI tasks, including age prediction and Alzheimer's classification, our method shows generality and outperforms single-modality models and existing fusion methods.


 

American Football, a complex sport of strategy, is the most popular sport in the US, engaging tens of millions of fans every week. Amazon owns exclusive broadcast rights for the Thursday Night Football (TNF) and is working to create a unique viewing experience, presenting new analytic features and enhanced graphics to help fans get more out of the game. This lecture will give a peek into the new features of the 2023 season, covering two ML powered features that are based on player tracking data collected in low latency in the NFL venues

In this talk, I will show that generative models can learn the dynamics of interacting quantum particles on disordered chains, a general scenario underlying a wide range of physical problems, from many-body quantum physics to quantum computation. Our algorithm learns complex quantum correlations from unlabeled examples and can then generate new physically valid instances with tunable physical parameters. This enables post-training exploration of the problem space, revealing underlying physical phenomena and accelerating the learning of more complex problems. These results suggest a general framework for generative AI in physical analysis and discovery. 

Though the huge breakthrough in vision-language models, they are still far from achieving human-level scene-understanding and have several fundamental limitations.
We show that these models are not able to perform simple tasks such as questions regarding locations and relations between objects. We suggest a model which can
naturally answer such questions and achieve scene-understanding even for complex scenes. It does this by using an iterative goal-driven approach that resembles
human vision. Our model is able to focus its attention in each iteration on the relevant parts of the scene and thus iteratively build a complex understanding of the scene.

In the lip-to-speech task, the objective is to accurately generate the missing speech for a soundless video of a person talking. It is required, e.g., when the speech signal is completely obfuscated by background noises. In this talk, I will present LipVoicer, a novel approach for producing high-quality speech for in-the-wild silent videos. LipVoicer leverages the transcription of the speech we wish to generate as predicted by a lip-reading model, and a diffusion model conditioned on the video to generate mel-spectrograms. LipVoicer achieves exceptional results, and the generated speech sounds natural and synchronized to the lip motion. 

XR devices immerse users in augmented realities, seamlessly merging digital and physical realms. Achieving this demands advanced perception technology that is resilient across environments, low in power usage, and has minimal latency. Yet, existing solutions struggle to meet this demanding combination of requirements, even with Apple’s Vision Pro setting a new standard for XR glasses’ 3D perception.

VoxelSensors’ Active Event Sensors (AES) enable robust, low-power, low-latency 3D sensing using laser triangulation. This innovation enhances SLAM, odometry, gesture recognition, and tracking, potentially revolutionizing augmented reality experiences. Boris will outline this groundbreaking approach and perspectives in XR 3D perception.

This talk will demonstrate how Generative AI can be adapted to augment datasets for computer vision training.  Utilizing diffusion models, we enhance dataset quality by seamlessly implanting concepts into images, improving downstream model performance. We provide an algorithmic framework for localizing the appropriate place for implanting a concept and for the actual generation of the concept given the background. To address the stochastic nature of diffusion models, which may generate images that do not contribute to the improvement of downstream models, we employ clustering and filtering, maximizing dataset relevance. We demonstrate the methods on both public and real world datasets.

Matching points across images from different cameras is commonly used in vision systems for a variety of purposes such as 3D reconstruction and field calibration. Systems doing so over time are often designed to achieve high consistency. Meaning that matches may change but do not jitter between solutions or errors. The presented method estimates consistent matches in adynamic environment using dense optical flows between the sequent images of each individual camera.

Lane detection is crucial for autonomous driving, furnishing indispensable data for safe navigation. Modern algorithms employ anchor-based detectors, followed by a label assignment process to categorize training detections as either positive or negative instances. However, the existing methods might be limited and not necessary optimal due to relying on predefined classic cost functions with not many calibration parameters.
Our research introduces MatchNet, a deep learning-based approach aimed at optimizing the label assignment process. Interwoven into a SOTA lane detection network such as CLRNet, MatchNet replaces the conventional label assignment process with a submodule network. This integration yields significant enhancements, particularly in challenging scenarios such as curve detection (+1.82%), shadows (+1.1%), and non-visible lane markings (+0.88%). Notably, this method boosts lane detection confidence level, enabling a 3% increase in the confidence threshold.

Osteoporotic vertebral compression fractures (VCFs) in the elderly pose significant quality-of-life challenges. a fraction of the VCF are occult, and cannot be distinguished from normal vertebrae by Traditional imaging like x-rays and CT scans. Tc99 bone scans is usually utilized to identify occult fractures. We propose a data-driven solution employing machine learning and computer vision to identify unique radiological patterns in occult VCFs. Our method, using only CT scans data of 24 vertebrae, successfully segments vertebrae into clusters, revealing distinct volume ratios that distinguish normal vertebrae from those with occult fractures. Importantly, we identified that vertebral posterior element volumes aid occult fracture identification and may play a role in the pathology of VCF. This approach highlights the potential of machine learning in enhancing skeletal condition diagnosis, bridging inter-modality gaps.

Osteoporotic vertebral compression fractures (VCFs) in the elderly pose significant quality-of-life challenges. a fraction of the VCF are occult, and cannot be distinguished from normal vertebrae by Traditional imaging like x-rays and CT scans. Tc99 bone scans is usually utilized to identify occult fractures. We propose a data-driven solution employing machine learning and computer vision to identify unique radiological patterns in occult VCFs. Our method, using only CT scans data of 24 vertebrae, successfully segments vertebrae into clusters, revealing distinct volume ratios that distinguish normal vertebrae from those with occult fractures. Importantly, we identified that vertebral posterior element volumes aid occult fracture identification and may play a role in the pathology of VCF. This approach highlights the potential of machine learning in enhancing skeletal condition diagnosis, bridging inter-modality gaps.

Large language models (LLMs) have emerged as a breakthrough technology, exhibiting remarkable performance across a wide range of tasks. Until recently, the development of LLMs seemed constrained by high barriers, resulting in a few companies dominating the field. However, recent advancements in the field have significantly lowered these barriers, enabling the development of high-quality LLMs with a limited amount of effort and computation resources.

In this tutorial, we will explore the challenges involved in building LLM models, the development that allows building such high-performance custom models with a small amount of resources, and the new possibilities it unlocks, including multimodal extension and expanded context windows.

T1 mapping is a quantitative MRI technique that has emerged as a valuable tool in the diagnosis of diffuse myocardial diseases. However, prevailing approaches have relied heavily on breath-hold sequences to eliminate respiratory motion artifacts. This limitation hinders accessibility and effectiveness for patients who cannot tolerate breath-holding. We address this limitation by introducing PCMC-T1, a physically-constrained deep-learning model that accounts for the signal decay along the longitudinal relaxation axis for motion correction in free-breathing T1 mapping. PCMC-T1 demonstrated superior results compared to baseline methods using a 5-fold experimental setup on a publicly available dataset of 210 patients.

Deep Neural Networks (DNNs) have excelled in many fields, largely due to their proficiency in supervised learning tasks. However, the dependence on vast labeled data becomes a constraint when such data is scarce.
Self-Supervised Learning (SSL), a promising approach, harnesses unlabeled data to derive meaningful representations. Yet, how SSL filters irrelevant information without explicit labels remains unclear.
In this talk, we aim to unravel the enigma of SSL using the lens of Information Theory, with a spotlight on the Information Bottleneck principle. This principle, while providing a sound understanding of the balance between compressing and preserving relevant features in supervised learning, presents a puzzle when applied to SSL due to the absence of labels during training.
We will delve into the concept of 'optimal representation' in SSL, its relationship with data augmentations, optimization methods, and downstream tasks, and how SSL training learns and achieves optimal representations.
Our discussion unveils our pioneering discoveries, demonstrating how SSL training naturally leads to the creation of optimal, compact representations that correlate with semantic labels. Remarkably, SSL seems to orchestrate an alignment of learned representations with semantic classes across multiple hierarchical levels, an alignment that intensifies during training and grows more defined deeper into the network.
Considering these insights and their implications for class set performance, we conclude our talk by applying our analysis to devise more robust SSL-based information algorithms. These enhancements in transfer learning could lead to more efficient learning systems, particularly in data-scarce environments.
Joint work with Yann LeCun,  Ido Ben Shaul, and Tomer Galanti.

 

In time-series analysis, nonlinear temporal misalignment is a major problem that forestalls even simple averaging. An effective learning-based solution for this problem is the Diffeomorphic Temporal Alignment Net (DTAN), that, by relying on a diffeomorphic temporal transformer net and the amortization of the joint-alignment task, eliminates the drawbacks of traditional alignment methods. Unfortunately, existing solutions for the joint alignment problem crucially depend on a regularization term whose optimal hyperparameters are dataset-specific and usually searched via a large number of experiments. Here we propose a regularization-free DTAN that obviates the need to perform such an expensive, and often impractical, search. Concretely, we propose a new well-behaved loss that we call the Inverse Consistency Averaging Error (ICAE), as well as a related new triplet loss and support for variable-length signals joint alignment. Our code is available at https://github.com/BGU-CS-VIL/RF-DTAN.

Risk management is paramount in all enterprizes, and particularly in Healthcare where patient safety takes precedence. Flaws in design or product functionality can result in treatment delays, patient harm, and reputational damage. AI, exemplified by models like ChatGPT and Dall-E, has the potential to revolutionize digital Healthcare, enabling more patient-focused care and informed interactions. However, AI's inherent uncertainty and the risk of generating inaccurate or misleading outputs pose challenges. This discussion explores the delicate balance between leveraging AI's capabilities and safeguarding patient safety, highlighting the need for responsible implementation in healthcare settings.

Autonomous driving and advanced driver-assistance systems rely on sensors and algorithms to perform appropriate actions. Typically, the sensors include color cameras, radar, lidar and ultrasonic sensors. Strikingly however, although light polarization is a fundamental property of light, it is seldom harnessed for perception tasks. Here, we analyze the potential for improvement when using an RGB-polarimetric camera for the tasks of monocular depth estimation and free space detection, as compared to using a standard RGB-only camera. We show that quantifiable improvement can be achieved using state-of-the-art neural networks, with minimum architectural changes. Additionally, we introduce an open dataset with RGB-polarimetric images, lidar scans, GNSS / IMU readings and free space segmentations that can be used by the community for new research.

The Iron Swords War brought profound tragedy, notably in identifying countless victims under challenging conditions. Iron Sword Artificial Intelligence Victim
Identification (IS_AI4VI) is developing an AI tool to facelifted victim identificatiom leveraging the expertise of over 80 volunteers from academia, the medical sector, and industry. This AI-driven initiative focuses on matching post-mortem CT scans with ante-mortem records to identify victims accurately. Operating under stringent data integrity and confidentiality protocols on a secure AWS cloud endorsed by the Ministry of Health, IS_AI4VI prioritizes ethical data handling. By streamlining the collection, anonymization, and analysis of medical data, and seeking Helsinki approval from Israeli hospitals, IS_AI4VI is dedicated to refining the process of victim identification.

Two mostly separate fields of machine learning -- computer vision and natural language processing -- have gradually become closer in recent years.

Advancements in each field have greatly influenced the other, driven in part by the abundance of weakly annotated data in the form of image-text pairs. These advancements brought focus to the multimodal Vision-Language models (VL) which jointly process images and free-text.

At IBM Research we have focused our research on multimodal learning, the limitations, applications and the adaptation of these models to the world of business documents.

In this talk, we will cover our latest work in the VL field, with topics such as Foundation Models for Expert Task Applications, Understanding Structured Vision and Language Concepts, and more.

The biggest and most challenging problems when using this amazing technology in a commercial setting are not necessarily algorithmic in nature. This talk suggests the biggest problems are related to trained data and responsible AI, to models accessibility by the community, and lastly, how to use Visual Generative AI to create a commercial impacta. Specifically, we will focus on solving one hard problem, attribution and transparency of the v-gen-ai. 

Can a diffusion process be corrected after taking a wrong turn? We present Attend-and-Excite, a novel method that guides a text-to-image diffusion model to attend to all subjects in a text prompt and strengthen — or excite — their activations, encouraging the generation of all subjects in the prompt.

Sightful is committed to moving AR beyond the hype and to create solutions that are immediately valuable and intuitive. In this talk, join Dori Peleg, Sr. Director of algorithms, as he present the first AR laptop and discuss how we build perception algorithms by synthesizing and balancing Artificial Intelligence (AI), hardware (HW) engineering and user experience (UX) design

To enhance deep network performance, the precision and efficiency of optimizers in recognizing gradient trends are crucial. Existing optimizers primarily rely on first-order Exponential Moving Averages, resulting in noticeable delays and suboptimal performance. We introduce the Fast-Adaptive Moment Estimation (FAME) optimizer. FAME leverages a higher-order Triple Exponential Moving Average (TEMA, inspired by the financial domain) to improve gradient trend identification. Here, TEMA actively influences optimization dynamics, unlike its passive role in finance. FAME excels in identifying gradient trends accurately, reducing lag and offering smoother responses to fluctuations compared to first-order methods. Results showed FAME’s superiority. It minimizes noisy trend fluctuations, enhances robustness, and boosts accuracy in significantly fewer training epochs than existing optimizers.

In this talk, we focus on the layout generation process, an essential ingredient of graphic design applications. We discuss the main challenges in the field, describe the different solution approaches, and introduce our recently proposed method, DLT, which consists of a novel joint discrete-continuous diffusion process, and highlight its effectiveness for conditioned layout generation.

In this talk, I will introduce SubTuning, a novel parameter-efficient finetuning method for neural networks that selectively trains a subset of the layers. This approach is based on the observation that the utility of layers in a pretrained model varies when adapting to a target task, influenced by factors such as model architecture, pretraining tasks, and data volume. By leveraging this observation, SubTuning carefully chooses the layers to finetune, providing a flexible method that outperforms conventional methods in scenarios with scarce data while also enabling efficient inference in multi-task settings.

In contrast to the computer vision field, where large open-access databases are abundant, the medical AI field suffers from data scarcity. Specifically, datasets of raw magnetic resonance imaging (MRI) measurements are small and quite limited. This poses a challenge for training AI algorithms for certain tasks, e.g. image reconstruction from MRI measurements. A common workaround is to download non-raw datasets that were published for other tasks, such as tumor segmentation, and use them for synthesizing “raw” MRI data and training reconstruction algorithms. Nevertheless, this could lead to biased results. In this talk I will describe how the bias emerges from such naïve workflows, and how it leads to fantastic, overly-optimistic results, which are too good to be true. Moreover, I will show that algorithms trained on synthesized data could later fail in clinical settings and miss important details. Next, I will introduce a new framework, titled “k-band”, which our team developed to address this challenge. The k-band framework enables training MRI reconstruction algorithms using only limited data, in a self-supervised manner, and hence reduces the need for massive datasets.

In today's landscape, engineering design remains mostly manual, posing significant challenges in translating market and product requirements into engineering concepts, technical specifications, and 3D computational (CAD) models. This labor-intensive process results in extended Time to Market (TTM), often causing organizations to lag behind their competitors. While the potential of Generative Artificial Intelligence (GenAI) is promising, existing generally-trained Language Models (LLMs) and Deep Learning models struggle to comprehend the complexity of engineering systems. A tailored engineering-specific solution is essential.  

 

Creating and updating HD maps for autonomous driving is critical yet prohibitively expensive at scale. This presentation delves into our scalable solution, relying on crowd-sourced vision data from AI-powered dashcams, while addressing mixed-fleet localization accuracy challenges, and without the use of expensive lidars. Utilizing deep learning in computer vision and structure-from-motion components, we generate precise 3D point clouds with dense 3D representations of various road assets, and provide high levels of asset localization. Join us as we explore the technical intricacies of this approach, offering insights into its potential to revolutionize autonomous navigation. 

Defect Detection in the Semiconductor industry is an extreme case of Anomaly Detection comprised of very small anomalies often well-harmonized in the background pattern.

Although very similar, no reference sample is perfect for comparison due to production variation.

We propose to generate this perfect reference – a generated image counterpart that is identical to the input sample everywhere, but the defective area.

We are using a novel fusion of stochastic and deterministic learning to train a conditional generative deep VAE model.

We demonstrate perfect reference generation for MVTec dataset and silicon manufacturing Scanning Electron Microscope (SEM) images achieving industry SOTA results.

After the atrocities of Oct 7^th, 2023, there was a need to identify victims whose conditions made traditional forensic methods ineffective. In this talk, we will show how RTC Vision, in collaboration with Mazor Robotics (now part of Medtronic), pivoted Computer-Vision technology originally designed for robot-assisted spine surgery to meet this urgent need.

Our approach leverages the technology to match postmortem CT scans of vertebrae with pre-existing CT scans, achieving a remarkable 100% identification success rate in our cases. Furthermore, I'll show how, even in the absence of prior CT scans, we can utilize antemortem X-ray images for identification.

In this groundbreaking talk, Amit Svarzenberg, CTO of Microsoft for Startups, explores the crucial role of AI multimodality in advancing the capabilities of autonomous agents. He discusses how leveraging multimodal AI systems—capable of processing and understanding diverse data types including text, images, audio, and sensor data—significantly enhances the autonomy, adaptability, and operational efficiency of these agents. Through an examination of recent advancements and practical case studies, Svarzenberg demonstrates how multimodal AI not only improves agents' perception and decision-making skills but also enables more seamless and natural interactions between humans and agents. By incorporating multimodal AI, autonomous agents can achieve a more comprehensive understanding of their surroundings, allowing for the execution of complex tasks with remarkable accuracy and dependability. This presentation highlights the transformative impact of multimodal AI on the evolution of autonomous agents towards truly intelligent and self-sufficient entities, marking a pivotal advancement in our pursuit of an AI-empowered future.

Address the challenges and opportunities of AI training in developing Vision AI solutions. Hear about the latest capabilities from NVIDIA TAO Toolkit and learn how TAO brings the power of Vision Transformers, Foundational models and Gen AI models to help developers build their AI solutions.

The field of computer vision is in the midst of a generative revolution, demonstrating groundbreaking image synthesis results, portraying highly complex visual concepts such as objects’ interaction, lighting, 3D shape, and pose. Expanding this progress to videos introduces two key challenges: (i) the distribution of natural videos is vast and complex, requiring orders of magnitude more training data than images, and (ii) raw video data is extremely high dimensional, requiring extensive computation and memory. In this talk, I’ll present different methodologies aimed at overcoming these challenges and advancing our capabilities to synthesize and edit visual content across both space and time. These methods range from layered video representations tailored to a specific video, to leveraging generative image priors for video synthesis tasks, and finally, designing and harnessing large-scale text-to-video models, which provides us with powerful motion priors. I’ll demonstrate how these methods unlock a variety of novel content creation applications, such as transferring motion across distinct object categories, image-to-video synthesis, video inpainting, and stylized video generation.

Transformers are neural networks that learns context from relationships in sequential data using a mechanism called attention.

The modern transformer was proposed in the 2017 paper titled 'Attention Is All You Need' by Ashish Vaswani et al., Google Brain team.

While transformer models are basically large encoder/decoder blocks that process data they also have an attention ingredient that allows them to detect patterns in data.

In this session, a brief introduction to the foundations of transformers will be given.  

Although the mapping between sound and meaning in human language is assumed to be largely arbitrary, research in cognitive science has shown that there are non-trivial correlations between particular sounds and meanings across languages and demographic groups, a phenomenon known as sound symbolism. Among the many dimensions of meaning, sound symbolism is particularly salient and well-demonstrated with regards to cross-modal associations between language and the visual domain. In this work, we investigate vision-and-language models such as CLIP and Stable Diffusion and find strong evidence that they do display sound symbolic patterns, paralleling the well-known kiki–bouba effect in psycholinguistics.

Have you ever wondered why when you ask Midjourney for pictures of drivers, only images of men appear? Or why Siri's default voice is female?
The lecture explores the existence of gender biases in artificial intelligence and how they impact our understanding of reality.

As AI continues to permeate our lives, it's crucial to recognize that it is not always neutral. From Google Translate to cutting-edge generative AI platforms, this lecture will examine the gender biases present in various technologies. Through engaging examples, attendees will gain a deeper understanding of the issue and learn about available tools to address and create a more equitable future.

We address the problem of surface reconstruction from unoriented point clouds. Implicit neural representations (INRs) have become popular for this task, but when information relating to the inside versus outside of a shape is not available optimization relies on heuristics and regularizers to recover the surface. These methods can be slow to converge and easily get stuck in local minima. We propose a two-step approach, OG-INR, where we (1) construct an octree and label what is inside and outside (2) optimize for a continuous and high-fidelity shape using an INR that is initially guided by the octree's labelling. To solve for our labelling, we propose an energy function over the discrete structure and provide an efficient move-making algorithm that explores many possible labellings.Our results show that the exploration by the move-making algorithm avoids many of the bad local minima reached by purely gradient descent optimized methods .

With the meteoric rise of AI-image generators, fake images of public figures - such as 'Trump's arrest' - have recently become viral sensations. The risks of synthetic media being utilized to spread misinformation and undermine democracy, were brought into the public’s attention, raising an interesting question - can we use AI to catch AI-generated images before they become the next viral hit? Zefr, the global leader in brand suitability, is introducing advanced vision models to detect AI-generated images and counter misinformation. The talk will cover real world examples, the challenges of detecting fakes and practical tips for training and deploying specialized vision models at scale.

In this talk, we present two methods for Text-to-Video generation: i) Make-A-Video (MAV), video generation from textual prompts, and ii) Make-A-Video3D (MAV3D), three-dimensional dynamic scenes generation from text descriptions. MAV introduces a paradigm for directly translating the tremendous recent progress in Text-to-Image generation to Text-to-Video. MAV3D leverages a 4D dynamic Neural Radiance Field (NeRF) optimized for scene appearance, density, and motion consistency through the MAV model. The dynamic video output generated from the provided text can be viewed from any camera location and angle, and can be composited into any 3D environment. Both methods rely only on text-image pairs and unlabeled videos. To the best of our knowledge, MAV3D is the first to generate 3D dynamic scenes given a text description.

We present a new method for partial annotations of MR images that uses a small set of consecutive annotated slices from each scan with an annotation effort equal to few annotated cases. The training is performed by using only annotated blocks, incorporating information about slices outside the structure, and modifying a batch loss function. For fetal body segmentation of in-distribution data, the use of partial annotations resulted in decrease in Standard Deviations of Dice scores by 22% and 27.5% for the FIESTA and TRUFI sequences respectively. For TRUFI out-of-distribution data, the method increased average Dice scores from 0.84 to 0.9.

Object detection is a pivotal component in the realm of computer vision, instrumental in facilitating machines to discern and localize objects within visual data. Recent years have seen significant advancements in object detection through the evolution of potent neural network architectures, notably the YOLO (You Only Look Once) family. This talk introduces a novel YOLO-based architecture, YOLO-NAS, developed via a proprietary neural architecture search (NAS) algorithm, AutoNAC. By optimizing accuracy and efficiency, YOLO-NAS redefines state-of-the-art object detection, paving the way for increased precision and performance in applications like autonomous vehicles, robotics, and video analytics.

WSC Sports is developing an AI platform for generating automatic sports highlights. To make the storytelling of our content more compelling, our system is also programmed to add to the highlight video, short transitions such as, team lineup graphics,  close-ups of reactions, etc.
Unlike regular plays in the game, these transitions tend to change from one broadcaster to the other, and over time. Although plain supervised algorithms can recognize and classify these transitions, it can be quite costly and difficult to keep their performance at a high level. In this presentation, we will present our Concept Detector, a framework for creating concepts - a set of textual and visual queries combined with a set of rules to retrieve those events. 

3D human pose estimation (HPE) is a fundamental task in human-computer interaction. Monocular 3D HPE is a challenging task due to a lack of in-the-wild annotated data, high computational load, and accessibility to depth observations. Despite their success on 2D HPE, end-to-end DNN approaches to 3D HPE hardly generalize to in-the-wild scenes with multiple self-occlusions. Recent approaches suggested optimizing 3D humanoid model parameters to minimize a 2D objective, however, as they optimize in 2D, they suffer from depth ambiguities.

We propose a two-stage depth-based solution to monocular 3D HPE. We start by using a deep neural network to predict 2D body-joint locations and to classify joints as occluded or visible. Then, having valid depth for the visible joints, we solve a Quadratically Constrained Quadratic Program enforcing skeletal and temporal-continuity constraints and thereby solving the self-occlusion problem. We demonstrate our method effectiveness on Gentex’s in-cabin 180 degrees fisheye depth camera and show that it can reconstruct reliable 3D human pose in complex situations.

Azure Video Indexer is an analytical tool to generate insights from videos while indexing them. Person tracking is a crucial aspect of video analysis and plays a significant role in Azure Video Indexer. However, it poses several algorithmic and computational challenges, particularly in real-world scenarios such as the media industry. These challenges include the need for efficient and scalable algorithms, handling multiple camera switches, dealing with different angles and poses, occlusions and more.

In this talk, I will present our novel pipeline for person tracking, with significant improvements for media and entertainment videos. Our approach addresses the above mentioned challenges and significantly reduces the computational cost and runtime required for person tracking. It combines Neural Network Models together with a novel tracking algorithm, all running fast and efficiently on a CPU.

Vision and Language (VL) models have demonstrated remarkable zero-shot performance in a variety of tasks. However, some aspects of complex language understanding still remain a challenge. We introduce the collective notion of Structured Vision&Language Concepts (SVLC) which includes object attributes, relations, and states which are present in the text and visible in the image. Recent studies have shown that even the best VL models struggle with SVLC. A possible way to fix this issue is by collecting dedicated datasets for teaching each SVLC type, which might be expensive and time-consuming. Instead, we propose a more elegant data-driven approach for enhancing VL models' understanding of SVLCs that makes more effective use of existing VL pre-training datasets and does not require any additional data. While automatic understanding of image structure remains largely unsolved, language structure is much better modeled and understood, allowing for its effective use in teaching VL models. We propose various techniques based on language structure understanding that can be used to manipulate the textual part of off-the-shelf paired VL datasets. VL models trained with the updated data exhibit a significant improvement of up to 15% in their SVLC understanding with only a mild degradation in their zero-shot capabilities both when training from scratch or fine-tuning a pre-trained model.

Simulation is a critical tool for ensuring the safety of autonomous driving. However, traditional simulation methods can be labor-intensive and struggle to scale.
In this talk, I will discuss an innovative neural simulation approach that learns to simulate driving scenarios from data. Specifically, I will focus on my latest research in three key areas: scene reconstruction in both appearance and geometry, motion generation of humans and vehicles, and LiDAR view synthesis.

Globally, an estimated 40M diagnostic reading errors occur annually; approximately 62% can be attributed to cognitive/perceptual issues associated with complacency, underreading, and search satisfaction. AI expert systems are critical to address the exponential growth in the volume of medical images generated and help alleviate workforce:workflow inefficiencies. But outsourcing clinical decision-making can exacerbate existing errors and introduce FN/FP reporting issues. We will present image enhancement methods that transform early disease detection capabilities, applying a “Deep Learning for ALL” approach that advances pixel-level “Image Intelligence” for both humans and AI and cooperatively promotes knowledge-building, pattern recognition, and attribute extraction

This talk introduces the concept of joint optimization in computational imaging using deep learning, demonstrating its practical benefits for improved performance. By showcasing real-world examples from Magnetic Resonance Imaging (MRI) and Multiple Input Multiple Output (MIMO) radar imaging, we reveal how this approach can positively impact the end performance. Join us for an exploration into the world of computational imaging, where simple yet effective techniques can make a meaningful difference.

Many of the tasks we face as data scientists or machine-learning researchers relate to categorization in one way or the other. In the words of David Mumford: "The world is continuous, but the mind is discrete." We often define categories when breaking down a real-world problem into an ML-based solution. However, actual target values may be continuous or at least ordered. This is something to consider and even leverage in the design of your ML model.

Using case studies from real-world data domains, we will see how acknowledging the inner relations of our target labels can boost the knowledge we provide in the training phase, better model the world, reduce overconfidence, and improve robustness. From classical concepts to state-of-the-art, this talk will walk you through regression-based approaches for what may seem like classification problems. Unlock the true potential of your labels and boost your classifiers!

As the use of generative AI becomes more prevalent in the tech industry, it can be difficult to understand how to effectively implement this new technology in your company's products.In this talk, Ofir Bibi will provide a general overview of how we implemented generative AI at Lightricks and showcase our latest developments in image transformation technology. He will also share thoughts on the impact that generative AI will have on the industry. 

Generative AI has revolutionized various domains, including art and design. However, the success of generative  models heavily relies on high-quality, extensive image datasets for effective training. Whether you're an AI enthusiast, researcher or a student, you've likely encountered challenges associated with untidy image datasets in Generative AI or other visual data-focused AI applications.

These challenges, including issues like duplicated images, mislabeled data, and outliers, can severely impact model reliability, waste computational resources and storage, and demand significant manual cleanup efforts. In our research project, LION-1B, we uncovered quality issues in approximately 105 million images. Notably, more than 90 million images were identified as duplicates, over 7 million images were blurry or of low quality, and more than 6 million images were deemed outliers.

To address these challenges, we have released a set of refined versions of popular visual datasets, namely LAION-1B and ImageNet-21K. We have named these refined datasets VL Datasets, and they are freely accessible through the visuallayer Python SDK or the free user-friendly VL Profiler UI. By utilizing VL Datasets, AI practitioners and researchers can enhance the development of more robust and reliable AI models.

Connecting traditional algorithms to ensure a successful lazy-labeling strategy.

In order to improve safety in autonomous driving it is important to use multiple sensors. In this talk we will discuss how to perform perception from 3D data acquired by the Innoviz lidar. Specifically we will describe how to identify obstacles or moving objects and perform detection in 3D. 

We will show how combining AI with classic thence provide us both with high accuracy and robustness to diverse conditions and scenarios.

Amit will discuss FETA, NeurIPS 2022 main conference paper. FETA: Towards Specializing Foundation Models for Expert Task Applications. While Foundation Models (FMs) have demonstrated unprecedented capabilities, FMs still have poor out-of-the-box performance on expert tasks. We offer an automatic system and method to adapt FMs to expert data using raw documents only, without requiring any annotations. FETA can be easily used on any document. We also propose a benchmark built around the task of teaching FMs to understand technical documentation. Our FETA benchmark focuses on text-to-image and image-to-text retrieval in public car manuals and sales catalogue brochures. 

Image retrieval are essential for managing, searching and making sense of the ever-growing volume of visual data, across diverse fields and applications. In this talk I will present several of our research works in interactive image retrieval, including a new architecture that leverages few shot learning, a greedy active learning approach for image retrieval, a new method that combines textual and visual search and finally introducing a system that leverages the emerging chat capabilities for the benefit of image retrieval.

In today's construction industry, precise knowledge of underground infrastructures is of paramount importance in project planning. However, the current scenario lacks a reliable and up-to-date map of underground infrastructure.
We have developed a groundbreaking mapping technology based on the processing of multiple sources, including engineering records which are extremely accurate and rich in information.
Engineering records need to be geolocated, extracted, and digitized. This is particularly challenging due to the very high variability in documents and the density of the sketches.
In this talk, we will present the innovative solution we have developed to process this data at scale.

Deep learning networks have emerged as a prominent technology for the accurate detection and identification of objects on the road, empowering the way to fully autonomous cars. In the pursuit of cost-effective and power-efficient solutions, this presentation delves into the challenges and strategies associated with optimizing deep learning networks for real-time execution in the context of vehicular environments.

Given the stringent constraints of cost-efficiency and low power consumption, a key challenge lies in achieving high performance without compromising accuracy.

Several optimization techniques are elucidated contribute to network efficiency, with a focus on minimizing computational redundancy and encouraging resource-efficient execution.

Through a compelling case study, the effectiveness of these optimization techniques is demonstrated in the real-world scenario of running a transformer network within an automotive system.

Ultrasound and radar signals are beneficial for medical imaging due to their non-invasive and low-cost nature. Quantitative medical imaging can display various physical properties of the scanned medium, in contrast to traditional imaging techniques. This broadens the scope of medical applications including fast stroke imaging. However, current quantitative imaging techniques are time-consuming and tend to converge to local minima. We propose a neural network based on the physical model of wave propagation, to achieve real time multiple quantitative imaging for complex and realistic scenarios,using data from only eight elements, demonstrated for diverse transmission setups using either radar or ultrasound signals.

We introduce PD-DWI, a machine-learning model for early prediction of pathological complete response (pCR) in breast cancer patients undergoing neoadjuvant chemotherapy (NAC). Leveraging decomposed diffusion-weighted MRI (DWI) and clinical data, our model outperforms conventional methods in the BMMR2 challenge, achieving an area under the curve (AUC) of 0.8849 versus 0.8397. PD-DWI has the potential to enhance pCR prediction accuracy, reduce MRI acquisition times, and eliminate the need for contrast agents.