Incorporating real-world data (RWD) into clinical trials can enhance trial efficiency, diversity, and generalizability. This paper introduces the Framework for Research in Synthetic Control Arms (FRESCA), which uses a novel Recommender System combined with Equity Adjustment strategies to design and evaluate Representative Hybrid Clinical Trials (HCTs). FRESCA employs a novel matching algorithm through its recommendation system to select suitable patients from RWD while ensuring that the selected population is representative of the target demographic. This dual approach improves both patient selection and trial outcomes by balancing statistical appropriateness and equity. Simulations based on data from two existing randomized clinical trials (RCTs) show that using FRESCA to recommend patients from RWD and apply equity adjustments enhances internal validity and generalizability. Our analysis indicates that combining matching and equity adjustments yields more accurate treatment effect estimates and fair population representation, even with reduced RCT control group sizes. In contrast, using either method alone may result in biased outcomes. The flexibility of FRESCA to simulate various HCT scenarios makes it a valuable tool for advancing equitable and efficient clinical trial designs.
arXiv
CTBench: A Comprehensive Benchmark for Evaluating Language Model Capabilities in Clinical Trial Design
Nafis Neehal, Bowen Wang, Shayom Debopadhaya, and 4 more authors
We introduce CTBench, a benchmark to assess language models (LMs) in aiding clinical study design. Given metadata specific to a study, CTBench examines how well AI models can determine the baseline features of the clinical trial (CT) which include demographic and relevant features collected at the start of the trial from all participants. The baseline features, typically presented in CT publications (often as Table 1), are crucial for characterizing study cohorts and validating results. Baseline features, including confounders and covariates, are also required for accurate treatment effect estimation in studies involving observational data. CTBench consists of two datasets: "CT-Repo", containing baseline features from 1, 690 clinical trials sourced from clinicaltrials.gov, and "CT-Pub", a subset of 100 trials with more comprehensive baseline features gathered from relevant publications. We develop two LM-based evaluation methods for evaluating the actual baseline feature lists against LM-generated responses. “ListMatch-LM” and “ListMatch-BERT” use GPT-4o and BERT scores (at various thresholds), respectively, to perform the evaluation. To establish baseline results, we apply advanced prompt engineering techniques using LLaMa3-70B-Instruct and GPT-4o in zero-shot and three-shot learning settings to generate potential baseline features. We validate the performance of GPT-4o as an evaluator through human-in-the-loop evaluations on the CT-Pub dataset, where clinical experts confirm matches between actual and LM-generated features. Our results highlight a promising direction with significant potential for improvement, positioning CTBench as a useful tool for advancing research on AI in CT design and potentially enhancing the efficacy and robustness of CTs.
@article{neehal2024ctbench,title={CTBench: A Comprehensive Benchmark for Evaluating Language Model Capabilities in Clinical Trial Design},author={Neehal, Nafis and Wang, Bowen and Debopadhaya, Shayom and Dan, Soham and Murugesan, Keerthiram and Anand, Vibha and Bennett, Kristin P},journal={arXiv preprint arXiv:2406.17888},year={2024},}
2023
AMIA
Framework for Research in Equitable Synthetic Control Arms 🏆
Randomized Clinical Trials (RCTs) measure an intervention’s efficacy, but they may not be generalizable to a desired target population if the RCT is not equitable. Thus, representativeness of RCTs has become a national priority. Synthetic Controls (SCs) that incorporate observational data into RCTs have shown great potential to produce more efficient studies, but their equity is rarely considered. Here, we examine how to improve treatment effect estimation and equity of a trial by augmenting “on-trial” concurrent controls with SCs to form a Hybrid Control Arm (HCA). We introduce FRESCA – a framework to evaluate HCA construction methods using RCT simulations. FRESCA shows that doing propensity and equity adjustment when constructing the HCA leads to accurate population treatment effect estimates while meeting equity goals with potentially less “on-trial” patients. This work represents the first investigation of equity in HCA design that provides definitions, metrics, compelling questions, and resources for future work.
@inproceedings{neehal2023framework,title={Framework for Research in Equitable Synthetic Control Arms 🏆},author={Neehal, Nafis and Anand, Vibha and Bennett, Kristin P},booktitle={AMIA Annual Symposium Proceedings},volume={2023},pages={530},year={2023},organization={American Medical Informatics Association},}
SCT
EquiSCAT: Strategies for Equity Considerations in Synthetic Control Arm Design 🏆
@article{neehalequiscat,title={EquiSCAT: Strategies for Equity Considerations in Synthetic Control Arm Design 🏆},author={Neehal, Nafis and Anand, Vibha and Bennett, Kristin P},year={2023},}
2022
IEEE BIBM
Subpopulation Analysis in Causal Inference: A Healthcare Case Study
Georgios Mavroudeas, Nafis Neehal, Jason Kuruzovich, and 2 more authors
In 2022 IEEE International Conference on Bioinformatics and Biomedicine (BIBM), 2022
Treatment interventions are usually targeted to improve a specific outcome on as elected group of patients who are eligible to receive the treatment. The success of such treatments is determined by the post-intervention treatment effect on the population under consideration. There are cases when the treatment group contains multiple categories of eligible populations, with various effects, especially when the study’s criteria are loosely defined. In such studies (non-targeted trials) non-eligible subjects may be treated, producing heterogeneous treatment effects within the treated group. Inferring the effectiveness of the treatment under this scenario is difficult since the average treatment effect on the treated is a combination of multiple effect levels. This can bias the resulting conclusion of the causal studies. We propose an end-to-end framework based on matching and unsupervised clustering for extracting population sub-groups based on their effect levels. We demonstrate our methods on a real-world healthcare application, highlighting the value of subpopulation analysis for recovering multiple effect groups.
@inproceedings{mavroudeas2022subpopulation,title={Subpopulation Analysis in Causal Inference: A Healthcare Case Study},author={Mavroudeas, Georgios and Neehal, Nafis and Kuruzovich, Jason and Bennett, Kristin P and Magdon-Ismail, Malik},booktitle={2022 IEEE International Conference on Bioinformatics and Biomedicine (BIBM)},pages={1673--1676},year={2022},organization={IEEE},}
HIMS
Hybrid matching methods for treatment program evaluation: A case study
Nafis Neehal, Georgios Mavroudeas, M Magdon-Ismail, and 2 more authors
In International Conference on Health Informatics and Medical Systems, 2022
We study a type-2 diabetes (T2D) health management pro- gram (HMP) using causal methods for treatment effect estimation on electronic health records. We use matching and survival analyses to assess T2D onset and acute care usage (emergency room or inpatient visits). To account for bias and healthcare usage changes due to the COVID-19 pandemic, we developed a hybrid matching approach that first identifies the set of potential controls based on time and other critical features and then applies matching methods. We compare results across seven state-of-the-art methods including expert-informed approaches. We find that HMP potentially improved subject health by more rapidly identi- fying patients with undiagnosed T2D at enrollment, allowing for timely treatment. After the initial two months, no significant differences are observed in time to T2D onset. We also found that HMP patients were less likely to seek acute care indicating improved health outcomes. We highlight practical challenges in observational health studies.
@inproceedings{neehal2022hybrid,title={Hybrid matching methods for treatment program evaluation: A case study},author={Neehal, Nafis and Mavroudeas, Georgios and Magdon-Ismail, M and Kuruzovich, J and Bennett, KP},booktitle={International Conference on Health Informatics and Medical Systems},year={2022},}
2021
IEEE BIBM
Predictive modeling for complex care management
Georgios Mavroudeas, Nafis Neehal, Xiao Shou, and 3 more authors
In 2021 IEEE International Conference on Bioinformatics and Biomedicine (BIBM), 2021
Complex care management (CCM) or hot spotting programs identify and manage high-need/high-cost patients, improving long-term health quality and medical costs. Typically, physicians refer patients to CCM. Despite strict guidelines to ensure that eligible patients are placed in appropriate programs, such a provider-based approach is limited by provider-capacity and the narrow view of a patient that a provider sees. We propose an ML workflow to augment the provider-based approach, that can flag patients who are suited to CCM. Our predictor uses a global view of a patient’s entire history across multiple providers and time to identify high-risk individuals from among all the individuals in a matter of seconds. On a monthly basis, we evaluate our predictions against physician referrals. In the test dataset, 41% of the top-500 highest risk individuals found by our model were referred to CCM by a physician at some point in the 6-month window following our prediction (top-500 is a parameter that can be set to match the CCM program’s capacity). Of those who were not referred in the 6-month window, 30% were referred at some time in their trajectory. The remaining false positives had a greater than 95% similarity when compared to true positive physician referrals in terms of cost profiles (both prior to referral and after referral) and patient profile. This remarkable similarity suggests that our machine learning predictor can identify new candidates for complex care management and/or predict referrals before a physician has an opportunity to do so.
@inproceedings{mavroudeas2021predictive,title={Predictive modeling for complex care management},author={Mavroudeas, Georgios and Neehal, Nafis and Shou, Xiao and Magdon-Ismail, Malik and Kuruzovich, Jason N and Bennett, Kristin P},booktitle={2021 IEEE International Conference on Bioinformatics and Biomedicine (BIBM)},pages={1443--1448},year={2021},organization={IEEE},}
2020
Springer IJCCI
Olympic sports events classification using convolutional neural networks
Shahana Shultana, Md Shakil Moharram, and Nafis Neehal
In Proceedings of International Joint Conference on Computational Intelligence: IJCCI 2018, 2020
The analysis of different sports data to get valuable insight has become immensely important nowadays. Profuse application of Artificial Intelligence in different sectors has become a very popular trend as well. However, the application of AI in sports analytics is still a new research domain left for exploration. With a view to applying AI in sports analytics, we have deployed Inception V3 and MobileNet which are Google’s most popular Convolutional Neural Networks to successfully recognize five different sports events from a huge image dataset of these events. Both of the models have achieved a very high performance in terms of accuracy, precision, recall, and f-measure while applied to the target dataset for successful classification.
@inproceedings{shultana2020olympic,title={Olympic sports events classification using convolutional neural networks},author={Shultana, Shahana and Moharram, Md Shakil and Neehal, Nafis},booktitle={Proceedings of International Joint Conference on Computational Intelligence: IJCCI 2018},pages={507--518},year={2020},organization={Springer Singapore},}
2019
Springer RTIP2R
Shot-Net: A convolutional neural network for classifying different cricket shots
Md Ferdouse Ahmed Foysal, Mohammad Shakirul Islam, Asif Karim, and 1 more author
In Recent Trends in Image Processing and Pattern Recognition: Second International Conference, RTIP2R 2018, Solapur, India, December 21–22, 2018, Revised Selected Papers, Part I 2, 2019
Artificial Intelligence has become the new powerhouse of data analytics in this technological era. With advent of different Machine Learning and Computer Vision algorithms, applying them in data analytics has become a common trend. However, applying Deep Neural Networks in different sport data analyzing tasks and study the performance of these models is yet to be explored. Hence, in this paper, we have proposed a 13 layered Convolutional Neural Network referred as “Shot-Net” in order to classifying six categories of cricket shots, namely Cut Shot, Cover Drive, Straight Drive, Pull Shot, Scoop Shot and Leg Glance Shot. Our proposed model has achieved fairly high accuracy with low cross-entropy rate.
@inproceedings{foysal2019shot,title={Shot-Net: A convolutional neural network for classifying different cricket shots},author={Foysal, Md Ferdouse Ahmed and Islam, Mohammad Shakirul and Karim, Asif and Neehal, Nafis},booktitle={Recent Trends in Image Processing and Pattern Recognition: Second International Conference, RTIP2R 2018, Solapur, India, December 21--22, 2018, Revised Selected Papers, Part I 2},pages={111--120},year={2019},organization={Springer Singapore},}
IEEE ECCE
Runtime optimization of identification event in ECG based biometric authentication
Nafis Neehal, Dewan Ziaul Karim, Sejuti Banik, and 1 more author
In 2019 International Conference on Electrical, Computer and Communication Engineering (ECCE), 2019
Biometric Authentication has become a very popular method for different state-of-the-art security architectures. Albeit the ubiquitous acceptance and constant development in trivial biometric authentication methods such as fingerprint, palm-print, retinal scan etc., the possibility of producing highly competitive performance from somewhat less-popular methods still remains. Electrocardiogram (ECG) based multi-staged biometric authentication is such a method, which, despite its limited appearance in earlier research works, are currently being observed as equally high-performing as other trivial popular methods. The identification stage of this method suffers from requiring a high runtime, due to cross-matching of the new data with every single template data stored in the database, especially when dealing with huge amount of data. To solve this unaddressed problem, in this paper, we have proposed a K-means clustering based novel method where all the template data are clustered based on similarity, and only the most similar data cluster is searched instead of whole dataset during the identification event. Using our method we have achieved a maximum of 79.26% time reduction with 100% accuracy.
@inproceedings{neehal2019runtime,title={Runtime optimization of identification event in ECG based biometric authentication},author={Neehal, Nafis and Karim, Dewan Ziaul and Banik, Sejuti and Anika, Tasfia},booktitle={2019 International Conference on Electrical, Computer and Communication Engineering (ECCE)},pages={1--5},year={2019},organization={IEEE},}
Springer RTIP2R
ExNET: deep neural network for exercise pose detection
Sadeka Haque, AKM Shahariar Azad Rabby, Monira Akter Laboni, and 2 more authors
In Recent Trends in Image Processing and Pattern Recognition: Second International Conference, RTIP2R 2018, Solapur, India, December 21–22, 2018, Revised Selected Papers, Part I 2, 2019
Pose detection estimate human activity in images or video frames using computer vision technique. Pose detection has many applications, such as body to augmented reality, fitness, animation etc. ExNET represents a way to detect human pose from 2D human exercises image using Convolutional Neural Network. In recent time Deep Learning based systems are making it possible to detect human exercise poses from images. We refer to the model we have built for this task as ExNET: Deep Neural Network for Exercise Pose Detection. We have evaluated our proposed model on our own dataset that contains a total of 2000 images. And those images are distributed into 5 classes as well as images are divided into training and test dataset, and obtained improved performance. We have conducted various experiments with our model on the test dataset, and finally got the best accuracy of 82.68%
@inproceedings{haque2019exnet,title={ExNET: deep neural network for exercise pose detection},author={Haque, Sadeka and Rabby, AKM Shahariar Azad and Laboni, Monira Akter and Neehal, Nafis and Hossain, Syed Akhter},booktitle={Recent Trends in Image Processing and Pattern Recognition: Second International Conference, RTIP2R 2018, Solapur, India, December 21--22, 2018, Revised Selected Papers, Part I 2},pages={186--193},year={2019},organization={Springer Singapore},}
Springer RTIP2R
Incept-N: a convolutional neural network based classification approach for predicting nationality from facial features
Masum Shah Junayed, Afsana Ahsan Jeny, Nafis Neehal, and 2 more authors
In Recent Trends in Image Processing and Pattern Recognition: Second International Conference, RTIP2R 2018, Solapur, India, December 21–22, 2018, Revised Selected Papers, Part II 2, 2019
Nationality of a human being is a well-known identifying characteristic used for every major authentication purpose in every country. Albeit advances in application of Artificial Intelligence and Computer Vision in different aspects, its’ contribution to this specific security procedure is yet to be cultivated. With a goal to successfully applying computer vision techniques to predict a human’s nationality based on his facial features, we have proposed this novel method and have achieved an average of 93.6% accuracy with very low mis-classification rate.
@inproceedings{junayed2019incept,title={Incept-N: a convolutional neural network based classification approach for predicting nationality from facial features},author={Junayed, Masum Shah and Jeny, Afsana Ahsan and Neehal, Nafis and Ahmed, Eshtiak and Hossain, Syed Akhter},booktitle={Recent Trends in Image Processing and Pattern Recognition: Second International Conference, RTIP2R 2018, Solapur, India, December 21--22, 2018, Revised Selected Papers, Part II 2},pages={466--475},year={2019},organization={Springer Singapore},}
IEEE ICTS
AcneNet-a deep CNN based classification approach for acne classes
Masum Shah Junayed, Afsana Ahsan Jeny, Syeda Tanjila Atik, and 4 more authors
In 2019 12th International Conference on Information & Communication Technology and System (ICTS), 2019
Skin diseases are very common and nowadays easy to get remedy from. But, sometimes properly diagnosing these diseases can be quite troublesome due to the stiff hard-to-discriminate nature of the symptoms they exhibit. Deep Neural Networks, since its recent advent, has started outperforming different algorithms in almost every sectors. One of the problem domains, where Deep Neural Networks are really thriving today, is Image Classification and Object and Pattern Discovery from images. A special type of Deep Neural Network is Convolutional Neural Networks (CNN), which are being extensively used for different sorts of computer vision and image classification related problems. Hence, we have proposed a novel approach, where we have developed and used a Deep Residual Neural Network model for classifying five classes of Acnes from images. Our model has achieved an approximate accuracy as much as 99.44% for one class, and the rest were also above 94% with fairly high precision and recall score.
@inproceedings{junayed2019acnenet,title={AcneNet-a deep CNN based classification approach for acne classes},author={Junayed, Masum Shah and Jeny, Afsana Ahsan and Atik, Syeda Tanjila and Neehal, Nafis and Karim, Asif and Azam, Sami and Shanmugam, Bharanidharan},booktitle={2019 12th International Conference on Information \& Communication Technology and System (ICTS)},pages={203--208},year={2019},organization={IEEE},}
Springer RTIP2R
A comparative study of different cnn models in city detection using landmark images
Masum Shah Junayed, Afsana Ahsan Jeny, Nafis Neehal, and 2 more authors
In Recent Trends in Image Processing and Pattern Recognition: Second International Conference, RTIP2R 2018, Solapur, India, December 21–22, 2018, Revised Selected Papers, Part I 2, 2019
Navigation assistance using different local Landmarks is an emerging research field now-a-days. Landmark images taken from different camera angles are being vividly used alongside the GPS (Global Positioning System) data to determine the location of the user and help user with navigation. However, determining the location of the user by recognizing the landmarks from different images, without the help of GPS, can be a worthy research trend to explore. Hence, in this paper, we have conducted a comparative study of 3 different popular CNN models, namely - Inception V3, MobileNet and ResNet50, and they have achieved an overall accuracy of 99.7%, 99.5% and 99.7% respectively while determining cities using landmark images.
@inproceedings{junayed2019comparative,title={A comparative study of different cnn models in city detection using landmark images},author={Junayed, Masum Shah and Jeny, Afsana Ahsan and Neehal, Nafis and Atik, Syeda Tanjila and Hossain, Syed Akhter},booktitle={Recent Trends in Image Processing and Pattern Recognition: Second International Conference, RTIP2R 2018, Solapur, India, December 21--22, 2018, Revised Selected Papers, Part I 2},pages={547--560},year={2019},organization={Springer Singapore},}
IEEE ECCE
Prediction of preferred personality for friend recommendation in social networks using artificial neural network
Nafis Neehal, and MA Mottalib
In 2019 International Conference on Electrical, Computer and Communication Engineering (ECCE), 2019
Social Networking sites have nowadays become the most common way to communicate over online for people around the world. For making friends in social network, there remains an underlying friend recommendation framework which suggests friends to the users. However, most of the existing friend recommendation frameworks consider only the number of mutual friends, geo-location, mutual interests etc. to recommend one person as a friend to another. But, in real life, people, who have similar personalities, tend to become friends to each other, application of which is completely missing in the modern friend recommendation frameworks. Hence, we have proposed a personality based friend recommendation framework in this paper, which consists of a 3-Layered Artificial Neural Network for friend preference classification and a distance-based sorted subset selection procedure for friend recommendation. Our model tends to achieve a fairly high precision, recall, fl-measure and accuracy of around 85 %,85%,82% and 83% respectively in the friend choice classification task.
@inproceedings{neehal2019prediction,title={Prediction of preferred personality for friend recommendation in social networks using artificial neural network},author={Neehal, Nafis and Mottalib, MA},booktitle={2019 International Conference on Electrical, Computer and Communication Engineering (ECCE)},pages={1--6},year={2019},organization={IEEE},}
IEEE ICASERT
An empirical study of cervical cancer diagnosis using ensemble methods
Enamul Karim, and Nafis Neehal
In 2019 1st International Conference on Advances in Science, Engineering and Robotics Technology (ICASERT), 2019
Cervical Cancer, being one of the most pressing issues now-a-days, needs to be addressed properly. With a view to achieving an accurate diagnosis method for Cervical Cancer by screening the risk factors, different machine learning approaches have been taken over time. But by analyzing the performances of most of state-of-the-art approaches, it was inferred that there is still room for improvement by developing a more accurate model. Hence, in this paper an approach using ensemble methods with SVM as the base classifier has been taken. The ensemble method with Bagging technique achieved an accuracy of 98.12% with very high precision, recall and f-measure value.
@inproceedings{karim2019empirical,title={An empirical study of cervical cancer diagnosis using ensemble methods},author={Karim, Enamul and Neehal, Nafis},booktitle={2019 1st International Conference on Advances in Science, Engineering and Robotics Technology (ICASERT)},pages={1--5},year={2019},organization={IEEE},}
2018
Elsevier Procedia
InceptB: a CNN based classification approach for recognizing traditional bengali games
Mohammad Shakirul Islam, Ferdouse Ahmed Foysal, Nafis Neehal, and 2 more authors
Sports activities are an integral part of our day to day life. Introducing autonomous decision making and predictive models to recognize and analyze different sports events and activities has become an emerging trend in computer vision arena. Albeit the advances and vivid applications of artificial intelligence and computer vision in recognizing different popular western games, there remains a very minimal amount of efforts in the application of computer vision in recognizing traditional Bangladeshi games. We, in this paper, have described a novel Deep Learning based approach for recognizing traditional Bengali games. We have retrained the final layer of the renowned Inception V3 architecture developed by Google for our classification approach. Our approach shows promising results with an average accuracy of 80% approximately in correctly recognizing among 5 traditional Bangladeshi sports events.
@article{islam2018inceptb,title={InceptB: a CNN based classification approach for recognizing traditional bengali games},author={Islam, Mohammad Shakirul and Foysal, Ferdouse Ahmed and Neehal, Nafis and Karim, Enamul and Hossain, Syed Akhter},journal={Procedia computer science},volume={143},pages={595--602},year={2018},publisher={Elsevier},}
arXiv
Crick-net: a convolutional neural network based classification approach for detecting waist high no balls in cricket
Md Harun-Ur-Rashid, Shekina Khatun, Mehe Zabin Trisha, and 2 more authors
Cricket is undoubtedly one of the most popular games in this modern era. As human beings are prone to error, there remains a constant need for automated analysis and decision making of different events in this game. Simultaneously, with advent and advances in Artificial Intelligence and Computer Vision, application of these two in different domains has become an emerging trend. Applying several computer vision techniques in analyzing different Cricket events and automatically coming into decisions has become popular in recent days. In this paper, we have deployed a CNN based classification method with Inception V3 in order to automatically detect and differentiate waist high no balls with fair balls. Our approach achieves an overall average accuracy of 88% with a fairly low cross-entropy value.
@article{harun2018crick,title={Crick-net: a convolutional neural network based classification approach for detecting waist high no balls in cricket},author={Harun-Ur-Rashid, Md and Khatun, Shekina and Trisha, Mehe Zabin and Neehal, Nafis and Hasan, Md Zahid},journal={arXiv preprint arXiv:1805.05974},year={2018},}
Sequence alignment in bioinformatics and compu-tational biology has always been a challenging task. With Next Generation Sequencing (NGS) techniques in hand, researchers are now capable of studying biological systems at a level never been possible before. Scientists now have billions of bytes of biological data to work with, trillions of sequences to align. But this comes at a cost of requiring computing machines having a tremendous amount of computational and analytical power. Purchasing this huge amount of hardware and setting up a standalone infrastructure would not only cost an unnecessarily massive amount of money and labor but also would become troublesome to maintain. Moreover, for aligning a huge number of DNA or Protein sequences a scalable multiple sequence alignment (MSA) algorithms is needed with decent accuracy. In such context, this paper presents a novel implementation of Partial Order Alignment (POA) algorithm on a multi-node Hadoop Cluster running on MapReduce framework. The implementation was done in Amazon AWS platform with multiple EC2 instances. It is a map-only implementation with Hadoop Streaming. The result of this implementation shows a drastic reduction in runtime with no accuracy degradation.
@inproceedings{neehal2017cloud,title={Cloud-poa: A cloud-based map only implementation of po-msa on amazon multi-node ec2 hadoop cluster},author={Neehal, Nafis and Karim, Dewan Ziaul and Islam, Ashraful},booktitle={2017 20th International Conference of Computer and Information Technology (ICCIT)},pages={1--6},year={2017},organization={IEEE},}
Incept-N: a convolutional neural network based classification approach for predicting nationality from facial featuresIn Recent Trends in Image Processing and Pattern Recognition: Second International Conference, RTIP2R 2018, Solapur, India, December 21–22, 2018, Revised Selected Papers, Part II 2, 2019
