Optimal navigation in complex flows

Michele Buzzicotti, Department of Mechanical Engineering, Visiting Scholar, Spring 2020

Hopeman 224

The search for an optimal navigation strategy in a complex environment is a key problem
in several applications, with potential breakthroughs in the open challenge of lagrangian
data assimilation, crucial for both meteorology and climate modeling [1, 2]. Here, we
use a Reinforcement Learning (RL) approach to identify optimal strategies able to accomplish 

different complex navigation problems. As first example, we focus on smart inertial
autonomous drifters able to actively change their size, hence to modify their inertia and
density, to reach their navigation goal. In short, the smart drifter explores the interplay
between its choices of size and its dynamical behaviour in the flow environment to reach
precise regions of the flow domain [3, 4, 5]. The second example consists in finding the
path that minimizes the time to navigate between two given points in a fluid flow, known
as the Zermelo’s problem. Here, we focus on the case of a vessel which has a slip velocity
with fixed intensity, Vs, but variable direction and navigating in a 2D turbulent sea. We
use an Actor-Critic RL algorithm, and compare the results with strategies obtained analytically 

from continuous Optimal Navigation (ON) protocols. We show that for our application, 

ON solutions are unstable for the typical duration of the navigation process and
are therefore not useful in practice. On the other hand, RL solutions are much more robust
with respect to small changes in the initial conditions and to external noise and are able to
find optimal trajectories even when Vs is much smaller than the maximum flow velocity.
Furthermore, we show how the RL approach is able to take advantage of the flow properties 

in order to reach the target, especially when the steering speed is small [6]. These
works illustrate the potential of RL algorithms to model adaptive behavior in complex
flows and paves the way towards the engineering of smart unmanned autonomous vehicles
that solve difficult navigation problems.

[1] Di Leoni, P. C., Mazzino, A., & Biferale, L. (2018). Inferring flow parameters and
turbulent configuration with physics-informed data assimilation and spectral nudging.
Physical Review Fluids, 3(10), 104604.
[2] Di Leoni, P. C., Mazzino, A., & Biferale, L. (2019). Synchronization to big-data:
nudging the Navier-Stokes equations for data assimilation of turbulent flows. arXiv
preprint arXiv:1905.05860.[3] Gustavsson, K., Biferale, L., Celani, A., & Colabrese, S. (2017). Finding efficient
swimming strategies in a three-dimensional chaotic flow by reinforcement learning.
The European Physical Journal E, 40(12), 110.
[4] Colabrese, S., Gustavsson, K., Celani, A., & Biferale, L. (2017). Flow navigation by
smart microswimmers via reinforcement learning. Physical review letters, 118(15),
158004.
[5] Colabrese, S., Gustavsson, K., Celani, A., & Biferale, L. (2018). Smart inertial parti-
cles. Physical Review Fluids, 3(8), 084301.
[6] Biferale, L., Bonaccorso, F., Buzzicotti, M., Di Leoni, P. C., & Gustavsson, K. (2019).
Zermelo’s problem: Optimal point-to-point navigation in 2D turbulent flows using
Reinforcement Learning. Chaos: An Interdisciplinary Journal of Nonlinear Science 29.10 (2019): 103138.

Bio Sketch:

Michele is a visiting scholar in MechE at UofR during 2020, working on turbulence and machine learning. He obtained his PhD in theoretical physics in 2017 at the University of Rome Tor Vergata, where he has an appointment as a researcher. His main research activity is in the study of turbulent flows using numerical simulations. Working from both Eulerian and Lagrangian points of view, he is interested in the development of non-linear, out of equilibrium models, such as Large-Eddy-Simulation closures for the small-scale dynamics of high Reynolds or magnetohydrodynamic flows. He is responsible for the development of High-Performance Computing (HPC) codes for state-of-the-art Direct Numerical Simulation (DNS), which are typically run on supercomputers and scale up to tens of thousands of processing cores. He is also involved in the application and development of Artificial Intelligence (AI) tools to fluid flow problems such as: deep learning for the data analysis of turbulent flows and reinforcement learning/policy gradient methods for the development of smart vehicles which are able to solve optimal navigation tasks in complex environments.