Deep Neural Networks (DNNs) are becoming an increasingly interesting, valuable and efficient machine learning paradigm with implementations in natural language processing, image recognition and hand-written character recognition. Application of deep architectures is increasing in domains that contain feature hierarchies (FHs) i.e. features from higher levels of the hierarchy formed by the composition of lower level features. This is because of a perceived relationship between on the one hand the hierarchical organisation of DNNs, with large numbers of neurons at the bottom layers and increasingly smaller numbers at upper layers, and on the other hand FHs, with comparatively large numbers of low level features resulting in a small number of high level features. However, it is not clear what the relationship between DNNs hierarchies and FHs should be, or whether there even exists one. Nor is it clear whether modelling FHs with a hierarchically organised DNN conveys any benefits over using non-hierarchical neural networks. This study is aimed at exploring these questions and is organized into two parts. Firstly, a taxonomic FH with associated data is generated and a DNN is trained to classify the organisms into various species depending on characteristic features. The second part involves testing the ability of DNNs to identify whether two given organisms are related or not, depending on the sharing of appropriate features in their FHs. The experimental results show that the accuracy of the classification results is reduced with the increase in ‘depth’. Further, improved performance was achieved when every hidden layer has the same number of nodes compared with DNNs with increasingly fewer hidden nodes at higher levels. In other words, our experiments show that the relationship between DNNs and FHs is not simple and may require further extensive experimental research to identify the best DNN architectures when learning FHs.