Closure problems are critical in predicting complex dynamical systems, e.g., turbulence or cloud dynamics, for which numerically resolving all degrees of freedom remains infeasible in the foreseeable future. Although researchers have been advancing traditional closure models of those systems for decades, the performance of existing models is still unsatisfactory in many applications, mainly due to the limited representation power of existing models and the associated empirical calibration process. Recently, the rapid advance of machine learning techniques shows great potential for improving closure models of dynamical systems. In this talk, I will share some progress in data-driven closure modeling for complex dynamical systems. More specifically, I will demonstrate the use of derivative-free Kalman methods to learn closure models from indirect and limited amount of data. In addition to deterministic closures, examples of sparse identification of dynamical systems and the learning of stochastic closures will also be presented.