This paper follows the 25 years of development of methods and systems for knowledge-based neural network systems and more specifically the recent evolving connectionist systems (ECOS). ECOS combine the adaptive/evolving learning ability of neural networks and the approximate reasoning and linguistically meaningful explanation features of symbolic representation, such as fuzzy rules. This review paper presents the classical now hybrid expert systems and evolving neuro-fuzzy systems, along with new developments in spiking neural networks, neurogenetic systems, and quantum inspired systems, all discussed from the point of few of their adaptability, model interpretability and knowledge discovery. The paper discusses new directions for the integration of principles from neural networks, fuzzy systems, bio- and neuroinformatics, and nature in general.