3.1. Model Predictions for Crystal System Classification
We developed several ANN models using code written in the programming language C to estimate the class of the crystal structures. The classification capabilities of the ANN models were tested on a dataset of 267 samples, with 40 samples reserved for testing. We obtained a minimum of seven misclassifications by varying ANN parameters with the 6-22-22-22-1 architecture, a momentum term of 0.3, and a learning rate of 0.6.
Figure 4 illustrates the weight distribution in the ANN model with a 6-22-22-22-1 architecture at both the initial and optimal stages of training. Initially, the weights are randomly initialized within a narrow range of −0.5 to 0.5 (shown as light gray points), reflecting the untrained state of the model. After 40,000 iterations, the final weights (depicted in red) show a much broader range, spanning from −47.85 to 45.76. This significant expansion indicates the adjustments made during training to capture the complex relationships between input features and crystal structure classification. The wider distribution and increased dispersion of the weights demonstrate the model’s enhanced complexity and optimization, enabling improved classification performance and predictive accuracy.
Monoclinic samples were labeled as 0, and orthorhombic samples as 1, with a cut-off value of 0.5 used for classification.
Figure 5 illustrates the ANN model’s classification performance for monoclinic and orthorhombic crystal structures in both training and testing datasets using distance metrics.
Figure 5a shows the distribution of 118 monoclinic samples from the training set, which were clustered within a narrow distance range and resulted in zero misclassifications. Similarly,
Figure 5b shows the distribution of 109 orthorhombic samples, which were also clustered within a narrow range and resulted in zero misclassifications.
For the testing dataset, slight classification challenges are observed.
Figure 5c shows the distribution of 21 monoclinic samples, with most correctly classified near the origin, but four samples are misclassified because they fall outside the main cluster.
Figure 5d shows the distribution of 19 orthorhombic samples, with most correctly classified, although three samples are misclassified because they deviate from the primary cluster.
Overall, these graphs highlight the ANN model’s high classification accuracy in training data, with perfect classification for both monoclinic and orthorhombic crystal structures. While minor misclassifications are observed in the testing data, they reveal areas for potential model refinement, as shown in
Table 1. This classification performance is promising and comparable to or better than the results reported in the literature, demonstrating the robustness of the ANN model for crystal structure prediction.
In 40 test datasets, the seven bold datasets are indicated as misclassified data.
The structural and chemical similarities between monoclinic and orthorhombic crystal systems make distinguishing them difficult. Minor inaccuracies or overlapping patterns in the data can impact the model’s predictions. As ANNs are often considered “black boxes”, it can be difficult to understand exactly how they make these distinctions. However, sensitivity or feature importance analysis could help identify which features are most influential in distinguishing between these two structures, potentially revealing areas for data enhancement.
Table 2 highlights specific samples that illustrate these challenges. For instance, samples 5 and 12, both monoclinic, were misclassified as orthorhombic because their volumes closely match the average volumes of orthorhombic structures. Similarly, sample 14 was misclassified from monoclinic to orthorhombic due to its energy above the hull being close to orthorhombic averages. In contrast, samples 33 and 35, which are orthorhombic, were misclassified as monoclinic because their formation energy, site count, and density values resemble those typically observed in monoclinic structures. Lastly, samples 34 and 39, both orthorhombic, were misclassified as monoclinic due to similarities in bandgap and unit cell volume, which are close to typical monoclinic values.
These misclassifications arise from overlapping structural and physical parameters in volume, energy above the hull, formation energy, site density, and bandgap between monoclinic and orthorhombic systems. Such overlaps make it difficult for the ANN to differentiate between the two structures accurately. Although ANNs identify complex patterns well, they struggle with subtle, overlapping features. Thus, improving accuracy may involve introducing discriminative features like interatomic distances, bond angles, or symmetry-specific descriptors or applying ensemble methods and decision rules to address overlaps in monoclinic and orthorhombic systems.
3.2. Evaluation of ANN Model Performance Using Confusion Matrix
Table 3 presents the confusion matrix for the ANN model’s predictions. This matrix details the model’s performance, including true positives (TPs), false negatives (FNs), false positives (FPs), and true negatives (TNs). The ANN correctly predicted the crystal structure for 135 samples, classified as TPs, while four samples were misclassified as FNs despite being correct according to DFT data. Additionally, there were three FPs, where the ANN predicted a positive match incorrectly, and 125 TNs, where the model accurately identified samples that did not match the specified class in the DFT data. In total, 139 samples were predicted as positive (135 TPs + 4 FNs) and 128 as negative (3 FPs + 125 TNs).
The confusion matrix demonstrates the ANN model’s high accuracy and reliability in mirroring the DFT-calculated crystal structures, as reflected by the substantial number of true positive and true negative classifications. Based on the confusion matrix in
Table 3, we calculated various performance metrics to evaluate the ANN model’s predictive accuracy. These metrics include accuracy, which reflects the overall correctness of the model’s predictions; Matthews correlation coefficient (MCC), which provides a balanced measure of the quality of binary classifications; recall, which indicates the model’s ability to identify positive cases correctly; specificity, assesses the model’s effectiveness in recognizing negative cases; F-score, which balances precision and recall to evaluate the model’s reliability. These metrics provide a comprehensive understanding of the ANN model’s performance and its capability to accurately distinguish between the crystal structures.
The ANN model demonstrated strong performances across various evaluation metrics, indicating its robustness and reliability in predicting the crystal structure. An accuracy of 0.973 shows that the model correctly predicted 97.3% of the instances. A specificity of 0.978 measures the proportion of true negatives correctly identified by the model. A specificity of 0.978 means that the model accurately identified 97.8% of the negative cases, demonstrating its effectiveness in avoiding false positives.
A Matthews correlation coefficient of 0.951 is a comprehensive metric that considers true and false positives and negatives. An MCC of 0.951 suggests a strong correlation between the observed and predicted classifications, reflecting the model’s robustness and balanced performance across all classes. The F-score is the harmonic mean of precision and recall, providing a single metric that balances both. An F-score of 0.974 indicates that the model maintains high precision (correctly identifying positive cases) and recall (correctly identifying all relevant cases), making it highly effective overall. Recall (sensitivity) (0.971) measures the proportion of true positive cases correctly identified by the model. A recall of 0.971 means the model successfully identified 97.1% of the actual positive cases, ensuring minimal false negatives. Sensitivity (0.97), often used interchangeably with recall, also measures the model’s ability to identify actual positive cases. A sensitivity of 0.97 confirms the model’s high capability in detecting positive instances accurately. These metrics collectively demonstrate the ANN model’s high reliability and effectiveness in predicting the class of the crystal structures of materials.
The proposed ANN model achieved an impressive prediction accuracy of 97.3%, as shown in
Figure 6, significantly outperforming traditional machine learning methods depicted in the comparative analysis. Meanwhile, models such as k-nearest neighbors (kNN), neural network (NN), random forest (RF), and extremely randomized trees (ERTs) demonstrated respectable accuracy levels ranging from 75% to 83% at 85% training data; the performance of the ANN model highlights its ability to handle complex, nonlinear relationships in the dataset effectively. This high accuracy underscores the robustness and reliability of the 6-22-22-22-1 ANN architecture, optimized with a learning rate of 0.6 and a momentum term of 0.3. The ANN model’s superior classification capability and minimal misclassifications demonstrate its potential as a powerful tool for predicting crystal structures in orthosilicate cathode materials for lithium-ion batteries. Such exceptional performance makes it a valuable resource for accelerating advancements in battery materials research and development.
While the proposed ANN model achieves high accuracy (97.3%) and demonstrates superior capability in capturing nonlinear relationships in the dataset, it also has potential disadvantages, including risks of overfitting and computational complexity. Overfitting occurs when the model memorizes the training data but fails to generalize to new data. Cross-validation was employed during model training to mitigate overfitting and ensure robust performance. Regularization techniques, such as weight decay and dropout, are additional strategies that can be incorporated to enhance model generalization further.
Another challenge is the computational complexity associated with the 6-22-22-22-1 ANN architecture, which demands significant resources for training. However, this is counterbalanced by the model’s ability to effectively handle complex datasets where simpler models like kNN or RF may fall short. Future work will explore the use of ensemble methods and lightweight architectures to reduce computational requirements while maintaining high performance. These considerations present a fair assessment of the model’s advantages and limitations, providing a pathway for its continued refinement and application in material science research.
3.4. Identification of Feature Importance
Figure 8 illustrates the Index of Relative Importance (I
RI) [
15,
16] for input variables in predicting crystal structures within the Li–Si–Fe–O system, distinguishing between monoclinic and orthorhombic forms. Each input variable (V, D, N
S, E
g, E
H, E
F) represents a distinct property influencing structural configuration. The I
RI values quantify each variable’s influence, highlighting which factors dominate different structures.
Figure 8a presents I
RI values for predicting the monoclinic structure, where N
S and E
F show vital positive contributions essential for identifying monoclinic structures in this system.
Figure 8b also pertains to monoclinic classification but under a different condition, where D and E
F show notable negative contributions, indicating their inverse relationship with this classification.
Figure 8c displays I
RI values for orthorhombic structure predictions in the bottom row. Here, E
H and E
F contribute notably, though with less consistency in direction than the monoclinic classification.
Figure 8d reveals a contrasting condition where E
F has an overwhelmingly positive I
RI value, suggesting it is the primary determinant for orthorhombic classification in this scenario. Overall, the figure highlights how E
F, N
S, and E
H emerge as highly influential variables, with impacts varying between monoclinic and orthorhombic structures. This insight provides guidance on the critical atomic or electronic characteristics in structural determinations within complex oxide systems.
