Abstract: A stochastic optimization method, the multi-generation Gaussian process optimizer (MG-GPO), was developed for design optimization of complex systems such as accelerators. The method builds up and updates posterior Gaussian process models of the physical system to be optimized and uses the models to filter for good solutions in a multi-generation stochastic optimization process. It results in substantial improvement in convergence speed in comparison to traditional stochastic optimization algorithms, such as NSGA-II and MOPSO. The method has been tested with real life accelerator design problems. Application of the method in online optimization of accelerators was also demonstrated.