On the Statistical Challenges of Echo State Networks and Some Potential Remedies.

Echo state networks are powerful recurrent neural networks. However, they areoften unstable and shaky, making the process of finding an good ESN for aspecific dataset quite hard. Obtaining a superb accuracy by using the EchoState Network is a challenging task. We create, develop and implement a familyof predictably optimal robust and stable ensemble of Echo State Networks viaregularizing the training and perturbing the input. Furthermore, severaldistributions of weights have been tried based on the shape to see if the shapeof the distribution has the impact for reducing the error. We found ESN cantrack in short term for most dataset, but it collapses in the long run.Short-term tracking with large size reservoir enables ESN to perform strikinglywith superior prediction. Based on this scenario, we go a further step toaggregate many of ESNs into an ensemble to lower the variance and stabilize thesystem by stochastic replications and bootstrapping of input data.