Discussion#
This research project was about predicting psychotic disorder with machine learning based on macro- and microstructural MRI data. For the macrostructural data cortical thickness and for the microstructural data mean diffusivity and fractional anisotropy were used. For the machine learning algorithms, logistic regression and support vector machine were used. The main research question addressed in this project was: Which of the MRI metrics (CT, MD, FA) are most informative for achieving accurate machine classification of psychotic disorder?.
1. Macrostructural data: Cortical Thickness#
As shown in the data exploration section, there is overall reduced cortical thickness in all except for one brain region in patients compared to controls. In line with previous research (see Introduction), there is a consistent pattern of reduced cortical thickness in psychotic patients across the brain. When it comes to predict psychotic disorder with machine learning based on cortical thickness, the results are not as clear. The logistic regression model with cortical thickness as the only feature has an accuracy of 68.18%, whereas the support vector machine model with cortical thickness as the only feature has an accuracy of 90.91%. A potential explanation for this difference could be that the SVM model might be overfitting. While the logistic regression model were iterated over 5000 runs, for the SVM model the default 5-fold cross-validation was used. However, in another study by Davatzikos et al. (2005) a sMRI-based classificaiton model was conducted, testing the performance of a SVM reaching an accuracy of 81% via leave-one-out cross-validation. For an overview of SVM models predicting psychotic disorder, consult Table 1 in Zarogianni et al. (2013), indicating accuracy ranges from 70.4% to 81.1%. In general, it should be noted that accuracy measures should be taken with caution as the dataset is imbalanced regarding the number of controls and patients.
2. Microstructural data: Mean Diffusivity and Fractional Anisotropy#
In general, the microstructural data aligns with research findings that indicate abnormalities in both FA and MD in patients with psychotic disorder compared with healthy subjects (Ardekani et al., 2003,Kubicki et al., 2007; Lee et al., 2009; Narr et al., 2009; Szeszko et al., 2005). For example, as the data exploration section showed, MD was higher in patients for the majority of brain regions except for 18 regions, mainly consisting left and right hemisphere lateralocciptal regions. However, the differences are still really small. This is also in line with previous findings, indicating increased MD in patients (see Introduction, also Lee et al., 2009). On the contrary, however, as shown for FA in the data exploration section, the differences are not as clear since they seem not to be consistent across the brain but exhibiting a little tendency towards decreased FA in patients, noting that the differences again are small. This contradicts previous findings stating that brain-wide reductions in fractional anisotropy (FA) (Kubicki et al., 2007). On the other hand, for example Caan et al. (2006) also found increasements in FA in some regions and decreases in others.
When it comes to predict psychotic disorder with machine learning based on mean diffusivity and fractional anisotropy, the logistic regression model in this project showed an accuracy of 75.27% for MD and 66.97% for FA. In comparison, the support vector machine model showed an accuracy of 82.86% for MD and 77.14% for FA. Looking at the F1-Score, the logistic regression model and SVM model bnoth performed better for MD (63.52%, 62.5% respectively). The accuracy results for FA in SVM models are comparable to the findings in the study by Caan et al. (2006). Using FA, authors report classification accuracy of 75% using a fivefold cross validation. Another study, however, achieved an 80% classification accuracy based on a leave-one-out cross validiation approach (Caprihan et al.,2008). An even higher accuracy of 90.62% in distinguishing schizophrenia patients and controls was reported by Ingalhalikar et al. (2010) They trained a non-linear SVM model following a feature extraction step that involves creating a feature by concatenating average anisotropy and diffusivity values in anatomically meaningful regions. With another method using Fisher’s linear discriminant analysis (LDA), Ardekani et al. (2003) achieved classification accuracies of 94% and 98% for FA and MD, respectively, in a test cohort of 25 patients with schizophrenia and 25 healthy volunteers. According to the authors, combining the information from FA and MD, however, did not improve the classification accuracy due to the high degree of correlation between these measures.
3. Limitations and Conclusion#
When intepreting the results, there are some limitations to keep in mind. First, as already mentioned above, accuracy scores should be taken with caution due to unbalanced groups. Also, the sample size is relatively small. The dataset consists of 28 patients and 80 healthy controls for the macrostructural data and 33 patients and 82 controls for the macrostructural data. Furthermore, there might an overfitting problem, since every brain region was used as a feature. This means that the model might have learned the noise in the data. Feature selection could be used to reduce the number of features and to avoid overfitting which did not happen in this project, but was tried to be tackled with multiple iterations which for both models show significant different runs. Also, future research could focus on multimodal dataintegration in terms of partial last squares or similarity network fusion.
All in all, regarding the research question the results of this project show that MD provides more accuracy (75.27%) in the logistic regression model while CT provides more accuracy (90.91%) in the SVM model. It can be said with caution, that both macrostructural and microstructural MRI metrics seem to be useful in predicting psychotic disorder and serve as sensitive biomarkers for differentiating between psychotic patients and controls indicating structural and structural connectivity abnormalities. In fact, machine learning can provide relief regardig the diagnosis process but still the implementation in clinical practice remains elusive. However, the results of this project are not conclusive, further research is needed to confirm the findings.
Reflection on the research project#
With almost no to little experience in computational neuroscience and basic coding skills, I decided to take the research module for the following reasons: my enthusiasm for the brain, the methodology with which the brain can be studied and artificial intelligence. If only I had known what I was getting myself into … I would have done it again, but this time differently.
The whole process was characterised by ups and downs. Thanks to the very well prepared material (at this point, a huge thank you to Peer), I had the opportunity to read up and dive on the variety of topics. I didn’t always understand everything straight away, the topics weren’t and aren’t easy fare, but with continuous engagement with the topic, I felt comfortable with the various concepts over time. It was also helpful to know that I could ask my tutors at any time (at this point, a huge thank you to Peer, Michael and Jack). I am grateful to have gained valuable experience, be it related to what really matters when designing a research project from scratch, considering various factors that make it transparent in terms of open science, or with computational neuroscience topics in general. I have learned that it is a process that takes a long time and is not linear. You have to reckon with setbacks because maybe a code doesn’t work or you can’t put an idea into practice or does not understand what is happening. But the process continues and you learn whether you want to or not.
In retrospect, I would have done many things differently. First, I would have tried to develop a much more solid foundation for myself. By this I mean that I would have liked to invest more time in researching the literature before finally deciding on a topic and spending more time in understanding the basics to the fullest. The same goes for the selection of the data set. My initial idea was to find a dataset that explores many different MRI metrics for the “full experience”. While this is the case for the dataset I used for this project, I underestimated how much time it would take to understand this data. Understanding how it was measured, what it says and how to process it in code. Unfortunately, I was not able to work with all types of data as originally planned. The short-term acceptance for my semester abroad also contributed a lot to this, which is why I was not able to spend the time on this project that I would have liked to. Many other things that were planned I could not accomplish be it working with the other metrics and also in terms of multimodal dataintegration or specifying the brain regions to only a few. What I mean to say is that I’m sure I could have made more of the project with a more solid foundation, spending more time in understanding the basics to have it easier in implementing them in practice. I would also have liked to use the opportunity to contact our tutors more often afterwards and ask for help.
Nevertheless, it was a very educational experience. I liked the combination of theoretical input and practical implementation. I am very happy to have learned how useful it is to work with version control and what it actually means to set up a machine learning algorithm and the functionality behind these methods and having gained more knowledge regarding micro- and macrostructural data. I would like to take this opportunity to thank the people from the Fiebach Lab once again for this great module and the wonderful time!
References#
Ardekani, B. A., Bappal, A., D’Angelo, D., Ashtari, M., Lencz, T., Szeszko, P. R., Butler, P. D., Javitt, D. C., Lim, K. O., Hrabe, J., Nierenberg, J., Branch, C. A., & Hoptman, M. J. (2005a). Brain morphometry using diffusion-weighted magnetic resonance imaging: application to schizophrenia. Neuroreport, 16(13), 1455–1459. https://doi.org/10.1097/01.wnr.0000177001.27569.06
Caan, M. W., Vermeer, K. A., Van Vliet, L. J., Majoie, C. B. L. M., Peters, B. D., Heeten, G. J. D., & Vos, F. M. (2006). Shaving diffusion tensor images in discriminant analysis: A study into schizophrenia. Medical Image Analysis, 10(6), 841–849. https://doi.org/10.1016/j.media.2006.07.006
Caprihan, A., Pearlson, G. D., & Calhoun, V. D. (2008). Application of principal component analysis to distinguish patients with schizophrenia from healthy controls based on fractional anisotropy measurements. NeuroImage, 42(2), 675–682. https://doi.org/10.1016/j.neuroimage.2008.04.255
Davatzikos, C., Shen, D., Gur, R. C., Wu, X., Liu, D., Fan, Y., Hughett, P., Turetsky, B. I., & Gur, R. E. (2005). Whole-Brain Morphometric Study of Schizophrenia Revealing a Spatially Complex Set of Focal Abnormalities. Archives of General Psychiatry, 62(11), 1218. https://doi.org/10.1001/archpsyc.62.11.1218
Ingalhalikar, M., Kanterakis, S., Gur, R. C., Roberts, T. P., & Verma, R. (2010). DTI Based Diagnostic Prediction of a Disease via Pattern Classification. Lecture Notes in Computer Science, 558–565. https://doi.org/10.1007/978-3-642-15705-9_68
Kubicki, M., McCarley, R. W., Westin, C., Park, H., Maier, S. E., Kikinis, R., Jolesz, F. A., & Shenton, M. E. (2007). A review of diffusion tensor imaging studies in schizophrenia. Journal of Psychiatric Research, 41(1–2), 15–30. https://doi.org/10.1016/j.jpsychires.2005.05.005
Lee, K., Yoshida, T., Kubicki, M., Bouix, S., Westin, C., Kindlmann, G., Niznikiewicz, M. A., Cohen, A. F., McCarley, R. W., & Shenton, M. E. (2009). Increased diffusivity in superior temporal gyrus in patients with schizophrenia: A Diffusion Tensor Imaging study. Schizophrenia Research, 108(1–3), 33–40. https://doi.org/10.1016/j.schres.2008.11.024
Narr, K. L., Hageman, N., Woods, R. P., Hamilton, L. S., Clark, K. A., Phillips, O. R., Shattuck, D. W., Asarnow, R. F., Toga, A. W., & Nuechterlein, K. H. (2009). Mean diffusivity: A biomarker for CSF-related disease and genetic liability effects in schizophrenia. Psychiatry Research: Neuroimaging, 171(1), 20–32. https://doi.org/10.1016/j.pscychresns.2008.03.008
Szeszko, P. R., Robinson, D., Ashtari, M., Vogel, J. P., Betensky, J. D., Sevy, S., Ardekani, B. A., Lencz, T., Malhotra, A. K., McCormack, J., Miller, R. L., Lim, K. O., Gunduz-Bruce, H., Kane, J. M., & Bilder, R. M. (2008). Clinical and Neuropsychological Correlates of White Matter Abnormalities in Recent Onset Schizophrenia. Neuropsychopharmacology, 33(5), 976–984. https://doi.org/10.1038/sj.npp.1301480