## Optimal navigation in complex flows

Michele Buzzicotti, Department of Mechanical Engineering, Visiting Scholar, Spring 2020

*Friday, January 24, 20201:30 p.m.*

Hopeman 224

The search for an optimal navigation strategy in a complex environment is a key problem

in several applications, with potential breakthroughs in the open challenge of lagrangian

data assimilation, crucial for both meteorology and climate modeling [1, 2]. Here, we

use a Reinforcement Learning (RL) approach to identify optimal strategies able to accomplish

different complex navigation problems. As first example, we focus on smart inertial

autonomous drifters able to actively change their size, hence to modify their inertia and

density, to reach their navigation goal. In short, the smart drifter explores the interplay

between its choices of size and its dynamical behaviour in the flow environment to reach

precise regions of the flow domain [3, 4, 5]. The second example consists in finding the

path that minimizes the time to navigate between two given points in a fluid flow, known

as the Zermelo’s problem. Here, we focus on the case of a vessel which has a slip velocity

with fixed intensity, Vs, but variable direction and navigating in a 2D turbulent sea. We

use an Actor-Critic RL algorithm, and compare the results with strategies obtained analytically

from continuous Optimal Navigation (ON) protocols. We show that for our application,

ON solutions are unstable for the typical duration of the navigation process and

are therefore not useful in practice. On the other hand, RL solutions are much more robust

with respect to small changes in the initial conditions and to external noise and are able to

find optimal trajectories even when Vs is much smaller than the maximum flow velocity.

Furthermore, we show how the RL approach is able to take advantage of the flow properties

in order to reach the target, especially when the steering speed is small [6]. These

works illustrate the potential of RL algorithms to model adaptive behavior in complex

flows and paves the way towards the engineering of smart unmanned autonomous vehicles

that solve difficult navigation problems.

[1] Di Leoni, P. C., Mazzino, A., & Biferale, L. (2018). Inferring flow parameters and

turbulent configuration with physics-informed data assimilation and spectral nudging.

Physical Review Fluids, 3(10), 104604.

[2] Di Leoni, P. C., Mazzino, A., & Biferale, L. (2019). Synchronization to big-data:

nudging the Navier-Stokes equations for data assimilation of turbulent flows. arXiv

preprint arXiv:1905.05860.[3] Gustavsson, K., Biferale, L., Celani, A., & Colabrese, S. (2017). Finding efficient

swimming strategies in a three-dimensional chaotic flow by reinforcement learning.

The European Physical Journal E, 40(12), 110.

[4] Colabrese, S., Gustavsson, K., Celani, A., & Biferale, L. (2017). Flow navigation by

smart microswimmers via reinforcement learning. Physical review letters, 118(15),

158004.

[5] Colabrese, S., Gustavsson, K., Celani, A., & Biferale, L. (2018). Smart inertial parti-

cles. Physical Review Fluids, 3(8), 084301.

[6] Biferale, L., Bonaccorso, F., Buzzicotti, M., Di Leoni, P. C., & Gustavsson, K. (2019).

Zermelo’s problem: Optimal point-to-point navigation in 2D turbulent flows using

Reinforcement Learning. *Chaos: An Interdisciplinary Journal of Nonlinear Science* 29.10 (2019): 103138.

Bio Sketch:

Michele is a visiting scholar in MechE at UofR during 2020, working on turbulence and machine learning. He obtained his PhD in theoretical physics in 2017 at the University of Rome Tor Vergata, where he has an appointment as a researcher. His main research activity is in the study of turbulent flows using numerical simulations. Working from both Eulerian and Lagrangian points of view, he is interested in the development of non-linear, out of equilibrium models, such as Large-Eddy-Simulation closures for the small-scale dynamics of high Reynolds or magnetohydrodynamic flows. He is responsible for the development of High-Performance Computing (HPC) codes for state-of-the-art Direct Numerical Simulation (DNS), which are typically run on supercomputers and scale up to tens of thousands of processing cores. He is also involved in the application and development of Artificial Intelligence (AI) tools to fluid flow problems such as: deep learning for the data analysis of turbulent flows and reinforcement learning/policy gradient methods for the development of smart vehicles which are able to solve optimal navigation tasks in complex environments.