Personalising sound over loudspeakers

Philip Jackson
Institute of Sound Recording, University of Surrey, UK

Philip Coleman
Centre for Vision Speech & Signal Processing, University of Surrey, UK

Filippo Fazi
Institute of Sound & Vibration Research, University of Southampton, UK

Background and Scientific Context

In our information-overloaded daily lives, unwanted sounds create confusion, disruption and fatigue in what do and experience. Taking control of your own sound environment, you can design what information to hear and how. Providing personalised sound to different people over loudspeakers enables communication, human connection and social activity in a shared space, meanwhile addressing the individuals’ needs. Recent developments in object-based audio, robust sound zoning algorithms, computer vision, device synchronisation and electronic hardware facilitate personal control of immersive and interactive reproduction techniques. Accordingly, the creative sector is moving towards more demand for personalisation and personalisable content.

Aims of the Tutorial

This tutorial offers participants a novel and timely introduction to the increasingly valuable capability to personalise sound over loudspeakers, alongside resources for the audio signal processing community. Presenting the science behind personalising sound technologies and providing insights for making sound zones in practice, we hope to create better listening experiences. The tutorial attempts a holistic exposition of techniques for producing personal sound over loudspeakers. It incorporates a practical step-by-step guide to digital filter design for real-world multizone sound reproduction and relates various approaches to one another thereby enabling comparison of the listener benefits.

Target Audience and Assumed Knowledge

The tutorial is aimed at academic and industrial researchers with a practical interest in acoustical engineering, and specifically in developing systems for personalising sound that is rendered over loudspeakers. It would suit postgraduate research students and practising research engineers who wish to gain an appreciation of the capabilities and limitations of current technology alongside an understanding of the principles underpinning the state of the art. We will assume a basic familiarity with vector and matrix algebra, as well as fundamental acoustics, such as the Greens function of a monopole sound source with spherical wave front and delay-and-sum beamforming.

Contents Outline

The provisional structure of the tutorial is as follows:

• Introduction to personalising sound over loudspeakers
• Definition of the personal sound zone and spatial audio problems
• Overview of key approaches to controlling the sound field to deliver audio to one or more listeners, including least-squares and spherical-harmonic approaches
• Fundamental limitations and performance measures that are relevant for users: loudness, target quality, interference, spatial quality
• Relationships between design choices and performance measures: order, geometry, cost function
• Regularisation for robust solutions that handle measurement uncertainty
• Dealing with the effects of room reverberation
• Making practical sound zone filters, in anechoic and reflective environments
• Sound zones for binaural audio over loudspeakers
• Summary and conclusion
• Future perspectives

The main features of the tutorial are:

• Overview of the fundamentals of personal sound field theory
• Review of methods from panning and ambisonics to multi-point, including the popular mode-matching and pressure-matching techniques
• Characterisation of the various approaches, including their strengths and limitations
• Formulations for single-/multi- zone for mono vs. spatial audio
• Robust MIMO system design including regularisation
• Step-by-step guide for making sound zone filters
• Experimental results under anechoic and reflective conditions
• Illustration of applications in the car and at home
• Future perspectives for connectivity and interactivity

The three presenters, all academic faculty staff members at UK universities, collectively have over thirty years’ experience in delivering lectures, giving invited talks and participating in workshops, conference and summer schools. They have been collaborating together for 7 years through AES sound field control conferences and the S3A project on object-based spatial audio.

Data augmentation techniques for deep learning

David ROUSSEAU
Université d’Angers, France
Sotirios A. TSAFTARIS
University of Edinburgh, UK

Background and Scientific Context: Deep learning is currently used world-wide in almost all domains of image analysis as an alternative to traditional purely handcrafted tools. With their high capacity of tuning, deep neural networks are seemingly capable of solving informational tasks of any level of complexity provided enough annotated data are provided to them. However, this requested amount of data, typically 10000 instances, are overpassing most capacity of isolated human scientific consortium. A possible pathway though to apply deep learning without running the risk of overfitting is to proceed to data augmentation from an original small amount of significant, i.e. representative of the complexity of the expected data.

Aims of the Tutorial: In this tutorial we propose an overview of the different techniques of data augmentation.

Target Audience and Assumed Knowledge: The Audience is assumed to have basic knowledge about supervised machine learning and image processing.

Contents Outline: We will provide the description of the following data augmentation techniques * Data augmentation by transfer learning

• Replication of instances after geometrical transformation
• Simulation of instances based on expert-based knowledge
• Simulation of instances based on neural networks via the so-called Generative adversarial networks.
• Illustration of these techniques are given in various bio imaging problems including biology, medical imaging and plant science.
• Typical expected performance gain will be discussed.

Unlocking new Dimensions in Radio-based Positioning “5G Localization”

Henk Wymeersch
Chalmers University of Technology

Gonzalo Seco-Granados
Universitat Autònoma de Barcelona

Background and Scientific Context:

The use mmWave technology is one of the novel elements in 5G systems, and it offers the potential to satisfy the rate requirements in mmWave thanks to the large available bandwidth and directional interference-free transmissions. The community is highly interested in the fact that mmWave signals are useful for very accurate ranging and angle estimation both between devices, and between devices and base stations. In turn, this is giving rise to a potential new positioning technology: 5G localization. This tutorial will cover the basics of 5G localization: models, performance bounds, algorithms for localization, tracking and mapping, and applications.

Aims of the Tutorial:

The objective of the tutorial is to describe models, theoretical bounds, and algorithms that serve to exploit the 5G technologies also for positioning services. Moreover, 5G systems will offer a natural and tight connection between localization and communication, where each of the two functionalities will benefit from the other. This tutorial will also contribute to creating links between a broad spectrum of topics, from those more related to communications such as throughput, beam alignment, channel estimation, etc., to those closer signal processing and positioning such as channel modeling for positioning, compressed sensing, estimation bounds for positioning, tracking and mapping.

Target Audience and Assumed Knowledge:

The target audience includes researchers in signal processing interested in learning about 5G localization models, methods and bounds; tracking and mapping techniques; as well as about the synergies between communication and localization. Researchers on GNSS (GPS, Galileo, Glonass, Beidou, ...) also form part of the target audience since they may be interested to understand the possibilities of 5G positioning since its hybridization with GNSS gives rises to a practically ubiquitous localization solution. The emphasis of the tutorial is to provide insights and hands-on experience through small exercises. The tutorial presumes basic knowledge on wireless communication and statistical signal processing.

Contents Outline:

The tutorial will start with stating the “5G Positioning Challenge”: How can a user, from the downlink signal of a single base station, estimate its position, orientation, as well as clock bias, even the presence of an unknown propagation environment? This challenge will be addressed, relying on the application of several timely techniques from signal processing, and communication theory, including beamforming design, sparse channel estimation, data association, and simultaneous localization and mapping.

In addition, we describe novel positioning and mapping methods, tailored to the 5G scenario.

• 1. We will first give an overview of measurements derived from radio signals that can convey position-related information. Techniques for fusing measurements and providing snapshot positioning will be covered (including optimization-based methods, geometric methods, and inference-based methods).
• 2. Then, we present the potential of 5G technologies for positioning. In particular, we will describe, both qualitatively and quantitatively, how large bandwidths, high carrier frequencies, large arrays, and D2D are each individually beneficial for positioning. Each aspect will be supported with mathematical analysis and indicative performance results. We cover fundamental performance bounds, positioning algorithms as well as methods for radio mapping the environment for simultaneous localization and mapping. We will discuss one-way and two-way transmission schemes for localization, and also tracking methods.
• 3. We will conclude the tutorial with a list of research challenges, useful for researchers wishing to embark on this topic. Finally, we will also deliver the overall conclusions and an extensive list of references from the technical literature.

Inverse problems in image restoration: from basic principles to current deep-learning trends

Filip Sroubek
UTIA, Czech Academy of Sciences, Pod Vodarensou vezi 4, 182 08 Prague, Czech Republic

Jan Flusser
UTIA, Czech Academy of Sciences, Pod Vodarensou vezi 4, 182 08 Prague, Czech Republic

Barbara Zitova
UTIA, Czech Academy of Sciences, Pod Vodarensou vezi 4, 182 08 Prague, Czech Republic

Background and Scientific Context:

Signals acquired by real devices suffer from various types of degradation such as noise, blur, aliasing and geometric warping. We face this degradation phenomenon in acquisition of acoustic, electrical, speech, and (namely) image signals of any dimensions. In many applications the presence of degradation makes data processing problematic. Understanding the source of degradation is essential for correctly handling the connected processing problems. From the mathematical perspective, the common characteristic of degradation is ill-posedness, which leads to complicated inverse problems. Various mathematical approaches have been proposed that strive for finding a stable estimator.

Aims of the Tutorial:

Image processing is a prominent research field focusing on restoration and recognition of degraded images. It has been an active field for several decades and still attracts a considerable interest of the community. Guiding the tutorial audience through the basic principles, important details and different mathematical approaches will help the tutorial participants to understand the key points and to concentrate in their future work on the core aspects of this complex problem. This tutorial is focused on handling of degraded images but the presented approaches and methods can be applied in all areas of digital signal processing.

Target Audience and Assumed Knowledge:

The target audience of the tutorial are: 1) Researchers from all application areas who need to analyze distorted images; 2) Software professionals, industry researchers, and application developers of Computer Vision or Image Processing software; 3) Graduate students of computer science, artificial intelligence, signal processing, image analysis, pattern recognition, and related areas. The tutorial is self-contained and there is no specific requofed knowledge of the participants except standard undergraduate courses of image processing and pattern recognition.

Contents Outline:

Various types of degradation such as noise, blur, aliasing and geometric deformation are present in digital images and hamper further processing of images. Heavy degradation may corrupt images to such an extent, that neither automatic analysis nor visual interpretation of the content are possible. Restoration and invariants are two major approaches to handle degradation and both are covered by the tutorial. They are more complementary rather than concurrent; each of them is appropriate for different tasks and employs different mathematical methods and algorithms. The tutorial consists of four parts: modeling degradation, image restoration, invariants to degradation, and applications. We start with modelling all types of degradation, analyzing their sources and providing recommendation to avoid them in practical applications. In the image restoration part of the tutorial, we review traditional as well as modern techniques for tackling problems of denosing, superresolution, image rectification and blind deconvolution in various forms. In the invariant part, we show invariants to noise, blur and geometric transformations. The tutorial is enriched with numerous demonstrations and practical examples. The content of the tutorial originates from speakers’ 20-year experience in image restoration, deconvolution, invariants, and related fields. Please see further details about the tutorial at here.

Quantum Communications: Signal Processing meets Entanglement

Angela Sara Cacciapuoti
University of Naples Federico II

Marcello Caleffi
University of Naples Federico II

Background and Scientific Context:

Very recently, researchers worldwide have started to devote massive efforts in designing and implementing quantum computers, with 20-qubit computing processors already prototyped and several groups making very fast progress towards the 50-qubit regime. Nevertheless, to fully unleash the ultimate vision of the quantum revolution, it is necessary to design the Quantum Internet, a communication network able to share quantum information among remote quantum computers. But the Quantum Internet is governed by the laws of quantum mechanics. Hence, phenomena with no counterpart in classical communication networks impose a major network-paradigm shift from a communication perspective to harness the quantum mechanics specificities.

Aims of the Tutorial:

To fully unleash the ultimate vision of the quantum revolution, i.e., the Quantum Internet, it is necessary to design effective quantum communication strategies in order to “share” quantum information among remote quantum processors. However, despite the tremendous progress of the quantum technologies, quantum communications still constitute a key issue. This tutorial aims at providing the participants with a wide view about quantum communications and the unique challenges for transmitting quantum information.

Target Audience and Assumed Knowledge:

Given the novelty of the topic, the tutorial is targeted to audience new to the field. A graduated knowledge on signal processing, with emphasis on signal processing for digital communications, is assumed.

Contents Outline:

First, the importance of quantum communications will be highlighted, by reviewing the very last efforts, from both Industry and Academia, toward the design of quantum computing and quantum networks. Then, the main principles/postulates of quantum mechanics for quantum information processing (such as entanglement, no-cloning theorem, teleportation, etc.), as well as the quantum bra-ket notation will be gently introduced. Furthermore, the unique implications of the quantum mechanics on the quantum communications design will be presented and discussed. Finally, a review of the existing open problems and possible research directions concludes the tutorial.

Signal processing for vehicular systems

Robert W. Heath Jr.
The University of Texas at Austin, USA

Nuria González Prelcic
The University of Texas at Austin, USA, and University of Vigo, Spain

Background and Scientific Context

Signal processing is playing an important role in automated driving. The reason is that automated driving exploits multiple sensing modalities including millimetre wave radar, computer vision, not to mention wireless communication and satellite navigation. The objective of this tutorial is to provide an overview of vehicular systems from a signal processing perspective. The emphasis will be placed on introducing new topics to the signal processing community and explaining how existing and new tools may be applied for such problems.

Aims of the Tutorial

This tutorial will present an overview of signal processing in vehicular systems. The main objective is to explain the connections between core technical disciplines in signal processing and applications of those disciplines in vehicular systems. There will be high-level introductions to important concepts in vehicular systems coupled with specific examples of how signal processing tools are used to solve key problems.

Target Audience and Assumed Knowledge

Given the current trend towards automation of driving, the topic is timely and relevant for many members of the signal processing community. The topic is important for students because of the increasing number of jobs at automotive equipment manufacturers and the suppliers of automotive equipment. The topic is relevant for Professors as it exposes new research direction and funding opportunities. Finally, the topic is valuable for the industry especially those looking to transition their skills into a new application domain. No specific technical expertise is required aside from a general knowledge of statistical signal processing.

 
Contents Outline

Material for the tutorial will be presented in three main sections. The first section will consist of an overview of the state of vehicle automation and connectivity. This part will include a description of the key sensing components and their requirements. It will also discuss the state-of-the-art in automotive communication. The second section will overview the state-of-the-art and challenges in sensing for vehicles, focusing on applications that involve sensor fusion and collaborative perception of the environment. The application of mathematical tools such as collaborative filtering and machine learning to automotive problems will be discussed. A specific important application example will be on how to use sensors for localization using simultaneous localization and mapping. The third section will overview the state-of-the-art and challenges in communication for vehicles. Emphasis will be placed on the challenges associated with high data rate and low latency communication methods like millimetre wave communication. Preliminary results that show how millimetre wave can be rapidly configured for mobile settings will be provided. The idea of using millimetre wave communication signals for high accuracy position and orientation estimation will also be presented. Each section will highlight key contributions in the literature (not just by the tutorial authors) and will also point out directions for future work.

Cross-Modal Music Retrieval and Applications

Meinard Müller
International Audio Laboratories Erlangen, Erlangen, Germany

Andreas Arzt
Institute of Computational Perception, Johannes Kepler University, Linz, Austria

Stefan Balke
Institute of Computational Perception, Johannes Kepler University, Linz, Austria

Background and Scientific Context:

There has been a rapid growth of digitally available music data including audio recordings, images of
scanned sheet music, album covers, and an increasing number of video clips. In this tutorial, we cover
general signal processing and machine learning concepts designed to bridge the gap between these
different music representations. In particular, we discuss traditional approaches based on musically
motivated features, generalized audio fingerprinting, as well as recent data embedding techniques based
on deep learning. These technologies form the building blocks for many exciting music navigation and
browsing applications including the classical problem of automated score following.

Aims of the Tutorial: 

Music not only connects people but also relates to many different research disciplines. Adopting an
interdisciplinary perspective, our aim is to show that music is an attractive, rich, and challenging problem
domain that has many things to offer to the signal processing community. Considering cross-modal music
retrieval tasks, we demonstrate that these scenarios are well suited to discuss signal processing and
machine learning techniques (comparing traditional feature extraction and deep learning approaches).
Furthermore, we want to give some examples of fascinating music retrieval applications of academic,
educational, and commercial relevance.

Target Audience and Assumed Knowledge: 

The main goal of this tutorial is to give an exciting and easy-to-understand introduction to music
processing appealing to a wide audience in academia and industry. By providing many illustrative audio
examples and by working with pictures (rather than with formulas), we will make an effort to convey the
main ideas, in particular to non-experts and to researchers who are new to the field of music processing.
By doing so, the tutorial should appeal to a wide and interdisciplinary audience working in different fields
including signal processing, information retrieval, and machine learning.

Contents Outline: 

It is our primary goal to give an exciting tutorial that ranges from basic to advanced techniques used in
music processing. Our tutorial consists of three one-hour sessions, where we make sure that in each
session is enough room for questions and interaction with the audience. To have some variety throughout
the tutorial, we will discuss theoretical as well as practical aspects in each of the three sessions. Recent
techniques based on deep learning will play a role throughout the tutorial, even though they are
mentioned explicitly only in the third session of the following overview.

1. Session. General introduction and overview; music representations; sequence-based retrieval and mid-
   level music representations; shingle-based retrieval; recent approaches and discussion.
2. Session. Introduction to score following; music transcription including approaches based on machine
   learning; symbolic fingerprinting; indexing and efficiency issues; applications/demos.
3. Session. General introduction to deep learning approaches in music processing; cross-modal music
   embedding; data augmentation; recent trends using deep reinforcement learning; final round of questions
   and discussion.

Literature:
Meinard Müller, Andreas Arzt, Stefan Balke, Matthias Dorfer, and Gerhard Widmer: Cross-Modal Music
Retrieval and Applications: An Overview of Key Methodologies. Signal Processing Magazine, Issue 1,
January 2019, pages 52-62.

See also
SPM Special Issue on Recent Advances in Music Signal Processing
Supplementary Materials
Recent Advances in Music Signal Processing

Learning Nonlinear and Dynamic Connectivity and Processes over Graphs

Georgios B. Giannakis
University of Minnesota, Twin Cities

Yanning Shen
University of Minnesota, Twin Cities

Background and Scientific Context:
Learning the topology of graphs as well as processes evolving over graphs emerges in various application domains dealing with e.g., gene-regulatory, brain, power, and social networks. Key graph-aware learning tasks include regression, classification, subspace clustering, anomaly identification, and dimensionality reduction. Scalable approaches to deal with such high-dimensional tasks experience a paradigm shift to address the unique modeling and computational challenges associated with data-driven science in the modern era of big data analytics. Albeit simple and tractable, linear time-invariant models are limited since they fall short when it comes to modeling generally evolving topologies, as well as nonlinear and dynamic dependencies among nodal processes.

Aims of the Tutorial:
The goal of this tutorial is to overview contemporary advances, and develop a unifying framework to capture nonlinearities and dynamics. The approaches leverage nonlinear counterparts of partial correlation and partial Granger causality, as well as nonlinear structural equations and vector auto-regressions, along with attributes such as low rank, sparsity, and smoothness to capture even directional dependencies with abrupt change points, as well as time-evolving processes over possibly time-evolving topologies. Extensive real data experiments highlight the impact of the nonlinear and dynamic models on consumer and financial networks, as well as gene-regulatory and brain networks.

Target Audience and Assumed Knowledge:
The target audience includes graduate students and researchers with general interest in data science, network science, graph signal processing, machine learning and particular interests in applying nonlinear models, and online learning tools to various real-world networks, e.g., gene-regulatory, brain, power, financial and social networks. The audience only need basic knowledge in statistical learning and optimization.

Contents Outline:

Part I: Introduction and background
Context and timeliness for learning and topology inference over graphs (10 mins)

Part II: Identification of graph connectivity (75 mins)
(i) (Partial) correlation models and graphical Lasso;
(ii) Linear structural equation models (SEMs) and multi-layer networks;
(iii) Dynamic SEMs and tensors for changing topologies;
(iv) Kernel-based partial Granger causality; and
(v) Nonlinear structural vector autoregressive models.

Part III: Learning processes over graphs: (75 mins)
(i) Prediction from partially-observed processes on graphs;
(ii) Kernel Kriged Kalman Filtering for inference of dynamic processes;
(iii) Nonlinear SEMs for top-N recommender systems; and
(iv) Joint ID of topologies and signals on graphs.

Part IV - Open issues: Challenging yet promising research directions. (10 mins)

Detection and Classification of Acoustic Scenes and Events

Tuomas Virtanen
Tampere University

Annamaria Mesaros
Tampere University

Toni Heittola
Tampere University

Background and Scientific Context:

Sounds in our everyday environments contain an abundance of information about physical events that take place in it. This information has great potential in various applications such as context-aware devices, multimedia information retrieval, and acoustic monitoring, to name a few. Furthermore, the development of deep learning and the availability of new environmental sound datasets has enabled detection and classification of environmental sounds with accuracy that was not possible previously, opening up new application possibilities, and also interesting research problems.

Aims of the Tutorial:

This tutorial will cover basic concepts of detection and classification of acoustic scenes and events. It will introduce how these tasks are solved by state-of-the-art approaches such as convolutional recurrent neural networks, with some task-specific tailoring. Since the methods are heavily based on machine learning and results are dramatically affected by the data used to train systems, we will also present ways to acquire data for the development. We will introduce existing tools, including audio signal processing and machine learning libraries, and available datasets, which will allow a newcomer to easily start working in the field.

Target Audience and Assumed Knowledge:

The tutorial is aimed at researchers and engineers with background in computer science, electrical engineering or other related fields, who know the basics of machine learning and signal processing.

Contents Outline:

Introduction to detection and classification of acoustic scenes and events

• Problem formulation and overview of applications
• Defining categories of acoustic scenes and events
• Scientific and technical challenges

The machine learning approach to detection and classification of acoustic scenes and events

• General system structure for audio content analysis
• Machine learning perspective on classification, segmentation, tagging, and detection
• Acoustic features
• Convolutional and recurrent neural networks
• Measuring performance of systems: metrics, cross-validation, and reproducible results
• Examples of state-of-the-art approaches

Datasets: planning, recording and annotating audio data

• Data collection and annotation process; DCASE datasets case studies
• Open data and reproducibility; available datasets
• Alternative ways to produce annotations: crowdsourcing, active learning
• Data augmentation techniques

Tools we use and how we use them (Python tools)

• Feature extraction
• Machine learning
• Evaluation
• Tools for data handling and visualization

DCASE Challenge and other venues

• Introduction of DCASE community, workshop and challenge
• DCASE 2019 Challenge tasks
• Other channels for finding and publishing related research

Sketching Tools for Big Data Signal Processing

Alexandr Andoni
Department of Computer Science, Columbia University

Malcolm Slaney
Google AI

Background and Scientific Context

The modern scale and uses of data require new algorithmic design principles. Over the past couple of decades, the computer science community has developed many such principles. Core among them is the sketching tools, which has already impacted the design of fast algorithms in a variety of settings. The notion of sketching is a computational generalization of the classic dimension-reduction method, originally used for the purpose of reducing space usage. For example, sketching yielded some of the first compressed sensing algorithms. In recent years, sketching has been increasingly used to also speed-up algorithms for, e.g., similarity search, numerical linear algebra, machine learning, and iterative methods.

Aims of the Tutorial 

This tutorial will focus on algorithmic techniques to tackle massive datasets efficiently. We will cover 1) some novel computational models for how to think of massive data processing, and 2) core algorithmic techniques. A common thread will be how to summarize complex data into a smaller core “sketch”, via sampling and sketching techniques, so that later processing is more efficient since the dataset is now smaller, or perhaps in a more convenient form.

Target Audience and Assumed Knowledge

The goal of this tutorial is to give you a flavour of appropriate computational tools and why they work. As background, it is beneficial to have exposure to the following math/CS topics: basic probability theory, linear algebra; analysis of algorithms, and some basic algorithms, such as hash tables and binary search.

Contents Outline

There are a range of topics which we will cover, as well as their applications to signal processing and machine learning.

Sketching as a generalization of dimension reduction. Dimension reduction is typically envisioned in the Euclidean space, used to preserve Euclidean distance/dot-products, but for many applications, other geometries are more appropriate (e.g., l_1 for Lasso regression). Sketching provided an alternative, more powerful way to “compress” data.

Similarity search (high-dimensional nearest neighbour search). Problem: preprocess a dataset of objects (e.g., images, represented as Euclidean vectors), such that, given a new object, the data structure can quickly find all the similar objects from the dataset. This problem is a fundamental primitive for many related problems, such as closest pairs, variants of clustering, etc. Particularly effective solutions are obtained via Locality-Sensitive Hashing (LSH), a notion closely related to sketching. We’ll touch upon more recent data-dependent LSH as well.

Numerical linear algebra. A typical problem here is linear regression: to find a vector x minimizing ||Ax-b|| for some n-by-d matrix A. It turns out one can speed-up standard algorithms for this problem using sketching techniques. The general idea is to reduce the size of matrix A while preserving the solution x (approximately). The similar approach gives improved algorithms for problems such as approximate SVD and matrix multiplication.

Applications including:

• Machine learning
• Feature representations
• Initialization and streaming
• Recognition and Search
• Similarity search
• Embeddings
• Similarity
• Fast retrieval
• Alternate approaches
• Compressed sensing

Signal Processing and Beyond Using Deep Generative Models

David Wipf
Microsoft Research

Background and Scientific Context

Although definitions and perspectives vary, we can view generative modelling as the unsupervised process of estimating the probability density of some observed data of interest, or alternatively, learning a model capable of generating novel samples that are indistinguishable in distribution from these observations. Within the past five years, the landscape of generative modelling paradigms has been dramatically altered by the emergence of scalable deep neural networks. The resulting development typically involves a neural network architecture that has been trained to accept random input samples from a standardized Gaussian distribution and then convert them into output samples mirroring a complex target distribution. The variational autoencoder (VAE) represents a powerful representative example, with close ties to many commonly-used signal processing tools designed, broadly speaking, to extract low-dimensional structure from high-dimensional data.

Aims of the Tutorial

This tutorial will focus on the core principles of deep generative models, both in theory and practice. Special consideration will be given to the VAE framework and its natural precursors among a wide array of signal processing algorithms and dimensionality reduction methods. Moreover, against this backdrop, VAEs will be interpreted as furnishing a more powerful class of nonlinear inlier model, capable of both reconstructing a contaminated training dataset drawn from an unknown low-dimensional manifold, while implicitly learning the distribution of samples within this manifold. This functionality has widespread relevance to speech/image processing applications, computer vision, graphics, and semi-supervised learning.

Target Audience and Assumed Knowledge

Although not a requirement, this tutorial will likely have a larger impact among those who have at least some prior exposure to established signal processing techniques such as matrix completion/recovery, robust PCA, subspace clustering, or ICA. Beyond this, a rudimentary knowledge of deep learning models will be assumed, although the basic foundations of deep generative models will be presented beginning from an introductory level. Even so, as the tutorial proceeds to the latest VAE developments, analyzing a sequence of special cases with increasing complexity, the content should engage even relative experts.

Contents Outline

****** Background and Motivation ******

• The high-level motivation for unsupervised learning via deep generative models
• Brief introduction to arguably the two most influential variants: \emph{variational autoencoders} (VAEs) and generative adversarial networks (GANs)
• Strengths and weaknesses of VAEs and GANs
• Relevant applications from signal and image processing

****** Details of the Variational Autoencoder ******

• Background on latent variable models and variational inference
• Derivation of the VAE energy function
• Interpretations of the constituent encoder and decoder networks
• Stochastic training and reparameterization tricks

****** Connecting VAEs with Traditional Signal Processing Models and Algorithms ******

• Finding low-dimensional structure within high-dimensional signals
• Linear and bilinear inlier models (or unions thereof) for signal reconstruction in the presence of outliers and/or missing data
• Examples: Robust PCA, matrix completion, factor analysis, subspace clustering, independent component analysis
• Signal reconstruction versus generative modelling: the former is a prerequisite for the latter, and the VAE can do both unlike standard GAN models
• Leveraging VAEs as outlier-robust nonlinear extensions of numerous linear/bilinear signal reconstruction methods
• VAEs versus deterministic autoencoders for nonlinear signal reconstruction
• From signal recovery to the generation of new samples from a target distribution
• Practical examples using natural images

****** The Latest Analysis and Theoretical Developments ******

• Explanations for subtle, underappreciated VAE behaviours
• Properties of VAE global optima
• Sufficient conditions for perfect signal reconstruction and the recovery of ground-truth manifolds
• Variants/Enhancements of the basic VAE model inspired by the theory
• Sufficient conditions for extracting correct generative distributions from within a ground-truth manifold

****** Extensions and Applications ******

• Beyond generative modeling: Finding disentangled/semantically-meaningful latent representations using the VAE encoder
• Identifiability issues and weak supervision
• Conditional VAE models
• Semi-Supervised learning with deep generative models
• Practical usage issues and applications from signal/image processing