Evaluation and comparison of hereditary Cancer guidelines in the population

Background Family health history (FHx) is an effective tool for identifying patients at risk of hereditary cancer. Hereditary cancer clinical practice guidelines (CPG) contain criteria used to evaluate FHx and to make recommendations for genetic consultation. Comparing different CPGs used to evaluate a common set of FHx provides insight into how well the CPGs perform, the extent of agreement across guidelines, and how well they identify patients who should consider a cancer genetic consultation. Methods We compare the American College of Medical Genetics and Genomics (ACMG) and the National Comprehensive Cancer Networks (NCCN) (2019) CPG criteria for FHx collected by a chatbot and evaluated by ontologies and web services in a previous study. Collected FHx met criteria from seven groups: Gene Mutation, Breast and Ovarian, Li-Fraumeni syndrome (LFS), Colorectal and Endometrial, Relative Meets Criteria, ACMG Only Criteria, and NCCN Testing. CPG Criteria were coded and matched across 12 ACMG sub-guidelines and 6 NCCN sub-guidelines for comparison purposes. Results The dataset contains 4915 records, of which 2221 met either ACMG or NCCN criteria and 2694 did not. There was significant overlap—1179 probands met both ACMG and NCCN criteria. The greatest similarities were for Gene Mutation and Breast and Ovarian criteria and the greatest disparity existed among Colorectal and Endometrial criteria. Only 156 positive gene mutations were reported and of the 2694 probands who did not meet criteria, 90.6% of them reported at least one cancer in their personal or family cancer history. Conclusion Hereditary cancer CPGs are useful for identifying patients at risk of developing cancer based on FHx. This comparison shows that with the aid of chatbots, ontologies, and web services, CPGs can be more efficiently applied to identify patients at risk of hereditary cancer. Additionally this comparison examines similarities and differences between ACMG and NCCN and shows the importance of using both guidelines when evaluating hereditary cancer risk.


Family health history indicates risk for hereditary cancer
Family health history (FHx) is an effective tool for identifying individuals at risk for hereditary cancer [1][2][3]. An estimated 5-10% of all cancers are hereditary and individuals with a hereditary cancer syndrome are often at risk of developing cancer at a young age and more than one type [4][5][6]. Despite FHx being an effective way to estimate a patient's risk of developing hereditary cancer, this important information is often not collected by clinicians resulting in many patients not being referred for genetic counseling. As many as 50% of patients with accurate FHx highly suggestive of hereditary risk for breast and ovarian cancer syndrome (HBOC) or Lynch syndrome (LS) are not being referred for genetic counseling [7,8]. Additionally, an estimated 20% of primary care patients have family histories that indicate increased risk of developing a hereditary cancer [5,9]. To improve overall cancer outcomes for patients and their families, it is important to conduct an effective evaluation of hereditary cancer clinical practice guidelines (CPG). The National Comprehensive Cancer Network (NCCN) and the American College of Medical Genetics and Genomics (ACMG) have each curated sets of CPGs based on FHx designed to support medical professionals in assessing FHx risk for hereditary cancer [10][11][12]. Each criterion in a CPG establishes thresholds based on elements of a patient's FHx, such as number and types of cancers and their age of onset, to determine whether or not an individual is at risk of developing hereditary cancer and should consider receiving genetic counseling. Unfortunately, referral rates are low for cancer genetic consultations because providers lack training and confidence in assessing FHx for hereditary cancer risk [4,13,14]. Additionally, hereditary cancer CPGs are complicated and frequently updated. Another obstacle is that providers have insufficient time to collect and analyze FHx during a patient visit [15,16].
Collecting and evaluating patient FHx using chatbots, APIs, and ontologies In 2019 we deployed an ad campaign which invited users to engage with a user-friendly chatbot designed to collect patient FHx. It reached over 14,000 participants, a quarter of which completed a full FHx [17]. The results demonstrated that chatbots are an effective tool for collecting FHx and that there is an interest in the population for understanding hereditary cancer risk. The chatbot eases the burden of collecting patient data associated with traditional surveys and webforms by allowing users to engage with a human-like texting interface. However, collecting FHx is only half the task and evaluating CPG criteria for FHx is challenging and time-consuming. Ontologies and web application programming interfaces (APIs) are used to assess patient FHx by formalizing the CPG criteria and providing access to them Ritchie JB, et al: Enabling Patients to Receive Clinical Practice Guideline Recommendations for Hereditary Cancer Risk Using Chatbots, Family History, Application Programming Interfaces (API), Ontologies, and Owlready2: System Description, under review. CPG criteria have two crucial information components: logical and clinical. Logical information, such as how many cancers run in a family and the age of onset for meeting a criteria, describe the thresholds that a patient's FHx must meet in order to warrant a recommendation to consider genetic counseling. Clinical information, such as disease factors and distinguishing between different cancer subtypes, determines which thresholds are and are not met by a patient's FHx. Ontologies are designed to represent this kind of knowledge in a machine-readable format so that the logical and clinical elements of CPG criteria can be understood together and interpreted by the computer. The chatbot can send a patient's FHx to the ontology through web APIs, which apply CPG criteria and return the appropriate recommendations. Recommendations returned to the patient can also be sent to their provider to determine whether a cancer genetic consultation should be considered. Chatbots combined with webbased, ontology-oriented programming not only provide patients access to understanding CPG criteria and the implications they might have for their health with respect to cancer, but also can alleviate the strain on clinicians to collect and analyze FHx during patient visits.
This study reports on referral recommendations for 4915 users who completed the ItRunsInMyFamily chatbot workflow and received a recommendation to consider a genetics consultation (positive screen) or not (negative screen) based on CPG criteria [17]. The primary aim of this report is to understand how many of these 4915 patients met criteria, which criteria they met, how many received recommendations to receive a cancer genetic consultation, what types of cancers they had, and which genetic mutations were reported. For probands who screen negative, we also report those with a history of cancer in their family. The secondary aim of this study is to provide a side-by-side comparison of ACMG and NCCN hereditary cancer CPGs with respect to the collected patient FHx. Characterizing the patients who engaged the chatbot and collected their FHx demonstrates the utility of chatbots, ontologies, and web services in assessing hereditary cancer risk. Comparing ACMG and NCCN highlights agreement and disparity between the two CPGs as well as provides insight into the percentages and types of cancer criteria that are met by the patients.

Data collection and analysis
In November 2019, Welch et al. ran an online marketing campaign that ultimately collected FHx for 4915 users [17]. The majority of users during this campaign were 40-59 year old women and about half reported European ancestry. On average, patients had just under 20 family members in their family history. Welch et al. provide an in-depth analysis of the patient cohort including lifestyle (smoking, alcohol consumption, etc.), screening (mammogram, colonoscopy, etc.), and other personal and family health history metrics. In the present study, we considered only patients whose full FHx had been collected and performed descriptive analysis in Google Sheets to compare the frequency and type of criteria met for ACMG and NCCN CPGs.
We used criteria from the ACMG 2015 guideline; the Genetic/Familial High-Risk Assessment: Breast and Ovarian, Version 1.2020; and the Genetic/Familial High-Risk Assessment: Colorectal Version 3.2019, NCCN Clinical Practice Guidelines in Oncology [10][11][12]. Not all criteria represented in the ontology are included in this analysis-just those met by patients based on their FHx.
To compare and evaluate criteria from both ACMG and NCCN guidelines, we coded and numbered the criteria met by probands for each guideline included in the hereditary cancer ontology used to assess FHx. For the purposes of this paper, we divided each guideline into sub-guidelines to enable comparison of similar sets of criteria. ACMG criteria from 12 sub-guidelines-Syndro-mePatient, Brain, Breast, Ovarian, Colorectal, Endometrial, Leukemia, Gastric, Prostate, Melanoma, Thyroid, and AtRiskFDR-and NCCN criteria from six subguidelines-CRIT1, CRIT4, HRS3, POLYP1, LS1, and GENE1-were met by patients. Individual criteria are coded based on the sub-guidelines they are from and the order of their sequence in said sub-guideline e.g. Breast.01, Breast.02 for the first two criteria in the ACMG Breast sub-guideline. The SyndromePatient subguideline criteria refer to patients with a positive personal or familial gene mutation and AtRiskFDR subguideline criteria refer to patients at risk because a family member meets ACMG criteria. CRIT1, CRIT4, and GENE1 are sub-guidelines of the NCCN Genetic/Familial High-Risk Assessment: Breast, Ovarian, and Pancreatic guideline and include criteria covering breast cancer, ovarian cancer, pancreatic cancer, positive gene mutations, LFS, and risk conveyed by FDRs who meet criteria. HRS3, POLYP1, and LS1 are sub-guidelines of the NCCN Genetic/Familial High-Risk Assessment: Colorectal guideline and include criteria covering colorectal cancer, endometrial cancer, polyps, and LS. Figure 1 maps coded criteria from ACMG and NCCN guidelines that are either word for word the same or include the same clinical elements with slightly different thresholds. ACMG Breast.07, Colorectal.07, Leukemia.02, and Brain.03 represent ACMG LFS criteria and are each mapped to the corresponding group of NCCN LFS criteria CRIT4.02-06. Criteria in ACMG and NCCN with no sufficiently similar counterpart in the other CPG are mapped to No Match. We grouped matching or similar criteria into the following groups: Gene Mutation; Breast and Ovarian; LFS, Colorectal and Endometrial; Relative Meets Criteria; ACMG Thyroid, Gastric, Melanoma, Prostate (TGMP); and NCCN Testing (Fig. 2). In many cases, a criterion was common across both guidelines. By matching similar criteria across ACMG and NCCN into specific groups, we were able to compare patients who met criteria from one, both, or neither guideline.

Results
Of the patients who engaged with the chatbot to collect their FHx, 4915 completed the chatbot workflow and received a risk assessment report. Of these, 2221 (45.2%) met ACMG or NCCN criteria for a genetic cancer consultation referral and 2694 (54.8%) did not meet criteria. Twenty-four percent (1179/4915) met both ACMG and NCCN criteria for a genetic cancer consultation. Only 765 (15.6%) met NCCN criteria and not ACMG criteria while 277 (5.6%) patients met AMG criteria and not NCCN criteria.
Probands who met criteria often met criteria from more than one sub-guideline. Figure 2 shows the main overlaps among several of the different groups of guideline criteria we studied. Not all overlapping criteria groups could be displayed because the full set of overlapping CPGs was impossible to render as a Venn diagram. Instead, homologous criteria groups for both CPGs were intersected, and all groups were intersected with NCCN testing. The majority of probands who met criteria that recommended genetic counseling from either ACMG or NCCN met an NCCN criteria that recommended genetic testing as well (1957/2221; 88%). Tables 1 and 2 contain the codes, criteria text, and the number of probands that met individual criteria and overall sub-guidelines.
Of the 4915 patients who entered their FHx, 1969 (40.1%) met NCCN criteria of which 1957 (99.4%) met NCCN Testing criteria, 204 (10.4%) patients met Breast, Ovarian, and Pancreatic CRIT1, 47 (2.4%) met Breast, Ovarian, and Pancreatic CRIT4, and 563 (28.6%) met Colorectal HRS3. The 12 patients who met NCCN criteria but did not receive a recommendation for genetic testing had already reported a positive genetic mutation in their family and therefore did not receive a recommendation for testing per NCCN guidelines. However, they did receive a recommendation to consider a cancer genetic consultation per ACMG and NCCN. Of the 1969 patients who met NCCN criteria, a total of 1662 (84.4%) had FHx that met NCCN criteria for counseling or testing, and 833 (42.3%) had personal cancer history that met NCCN criteria for counseling or testing. Twentyeight percent of the 1969 patients who met NCCN criteria had personal and FHx that met NCCN criteria for genetic counseling or testing.
Nearly a third of the 4915 patients who collected their FHx-1456 (29.6%)-met ACMG criteria for a cancer genetics consultation. ACMG has criteria only for cancer genetic consultations and none for genetic testing. Of the 1456 patients who met ACMG criteria, 308 (21.2%) met criteria for ACMG breast and ovarian, 65 (4.5%) Fig. 1 Criteria map. ACMG coded criteria mapped to NCCN coded criteria based on similar or matching clinical information and thresholds. Criteria codes are named for the sub-guideline they come from and the order with which they appear in the sub-guideline e.g. ACMG Breast.01 is the first criteria in the breast cancer sub-guideline in ACMG and CRIT1.01 is the first criterion in the breast, ovarian, and pancreatic guidelines subguideline CRIT1 in NCCN 3). Breast cancer accounted for the largest proportion of cancers for which CPG criteria was implemented regardless of the number of cancers in the family. There were four probands with one 'other' cancer in their family who met criteria because a family member reported a positive cancer gene mutation. More families with three or fewer cancers did not meet CPG criteria than those that did, and more families with four or more cancers met criteria than those that did not (Fig. 3).

Discussion
Using a chatbot, web APIs, and hereditary cancer ontologies, we collected complete FHx for 4915 probands and coded and analyzed the hereditary cancer CPG criteria they met from ACMG and NCCN. Collecting FHx for at-risk patients and applying hereditary cancer guidelines allowed us to examine performance of ACMG and NCCN in identifying patients at risk and highlight important differences and similarities among them (Fig. 2). Genetic Mutation, Breast and Ovarian, and LFS guideline criteria groups for ACMG and NCCN captured similar sized sets of probands. Genetic Mutation criteria was exactly the same for ACMG and NCCN and therefore captured identical sets of probands. ACMG captured a moderately larger set of probands than NCCN for Breast and Ovarian criteria and the set of probands who met NCCN Breast and Ovarian criteria were almost completely subsumed by the set of probands who met ACMG Breast and Ovarian criteria. LFS criteria captured similar-sized sets of probands for both guidelines but had less overlap than did Breast and Ovarian criteria. Probands who met Colorectal and Endometrial criteria had the greatest disparity. NCCN Colorectal and Endometrial guideline criteria captured a much larger set of probands and almost completely subsumed the set of probands who met the analogous ACMG criteria. NCCN had no analog to ACMG Thyroid, Gastric, Melanoma, or Prostate cancer criteria and NCCN alone had genetic testing criteria which vastly subsumed the majority of probands who met ACMG or NCCN criteria.

Guideline comparison
ACMG and NCCN share overlap in guideline criteria with minor differences. The biggest difference between the two is ongoing updates for NCCN whereas ACMG was published in 2015 and has not been revised since. Additionally, ACMG includes criteria for a number of cancers that NCCN does not. These differences are strengths and weaknesses that influence how effectively the guidelines identify patients at risk. Frequent updates to NCCN keep the guideline criteria current, and ACMG catches patients at risk for cancers that NCCN does not publish criteria for. Together they balance each other in providing the best indication of risk for a patient based on their FHx.

Breast and ovarian
Not surprisingly, the criteria groups with the most congruent criteria were Breast and Ovarian cancer (Fig. 1). ACMG and NCCN each had a single criterion for Ovarian, which, though worded differently, effectively states that any history of personal or familial ovarian, fallopian tube, or peritoneal cancer meets the criterion. Breast cancer criteria had very close mapping between ACMG and NCCN. Primary differences were in the thresholds and cancers that made up the logical component of the criteria. The differences that resulted in notable  (Table 1). Additionally, NCCN restricts the set of cancers required to meet the criteria, resulting in fewer probands meeting the criteria (CRIT1.09) and accounting for why ACMG Colorectal.05 Colorectal cancer and two additional cases of any LS-associated cancer in the same person or in close relatives 2 Endometrial.05 Endometrial cancer and 2 additional cases of any LS-associated cancer in the same person or in close relatives 9 Relative meets criteria 1306 ACMGAtRiskFDRPatient First degree relative (mother father or sibling) meets risk criteria 1306 Remaining ACMG 68 Gastric.02 ≥2 cases of gastric cancer one dx at age < 50 in close relatives 9 Gastric.04 ≥3 cases of gastric cancer in close relatives 1 Melanoma.02 ≥3 cases of melanoma and/or pancreatic cancer in close relatives 3 Prostate.02 ≥2 cases of prostate cancer dx at age ≤ 55 in close relatives 1 Thyroid.01 Medullary thyroid cancer 11 Thyroid.04 Papillary thyroid cancer (cribriform-morular variant) 46 These are the criteria included in the hereditary cancer ontology used to evaluate FHx from ACMG that at least one proband met. Each section (Gene Mutation, LFS, Colorectal and Endometrial, Relative meets criteria, and Remaining ACMG) groups related criteria together. For each criterion in a section the total number of probands who met that criteria are recorded. Given that a proband can meet more than one criteria, the section totals are the total number of probands who met at least one criteria from the criteria in that section captured approximately 100 more probands than NCCN. Despite this difference, the Breast and Ovarian guideline criteria still bore the closest resemblance in content and thresholds other than the Gene Mutation guideline criteria.

LFS
LFS guideline criteria identified similar-sized sets of patients but had less overlap than Breast and Ovarian criteria. The lack of overlap makes sense because the LFS criteria between ACMG and NCCN guidelines had the least in common in terms of wording and structure in their respective CPGs. LFS criteria from ACMG come from four separate sub-guidelines-Breast, Colorectal, Leukemia, and Brain-and all have the same text with the respective cancer substituted in. The NCCN CRIT4 sub-guideline is dedicated to LFS and has highly specific sets of criteria that cover all LFS cancers. Despite these differences, the sets of criteria displayed considerable overlap in their respective sets of probands.

Colorectal and endometrial
Sets of probands identified by the Colorectal and Endometrial guideline criteria represented the greatest disparity in size between ACMG and NCCN. Like Breast and Ovarian, the Colorectal and Endometrial criteria for NCCN and ACMG were similar in wording and structure with a few exceptions that account for the large difference in the number of probands captured by these criteria (Fig. 1)   respectively. The looser thresholds for these NCCN criteria account for almost 100 probands who meet criteria compared to less than 10 probands who meet analogous ACMG criteria. Probands met twice as many LS associated criteria in NCCN (HRS3.03, 05, 08, and 09) than they did in ACMG (Colorectal.05 and Endometrial.05). Additionally, NCCN LS criteria had lighter restrictions on cancers and a wider range of relatives that contributed to the logical thresholds required to meet criteria, resulting in over 400 probands who met NCCN criteria compared to about 10 probands who met analogous ACMG criteria (Tables  1 and 2). These two sets of Colorectal and Endometrial criteria from ACMG and NCCN demonstrate how very small changes in thresholds can have enormous effects on which patients meet criteria and receive recommendations to consider genetic counseling and testing.
Thyroid, gastric, melanoma, and prostate ACMG Thyroid, Gastric, Melanoma, and Prostate (TGMP) sub-guidelines have no obvious analog in NCCN for the criteria we implemented. However, 69 probands met criteria from one of these four subguidelines. The majority of them met Thyroid cancer criteria for medullary or papillary thyroid cancer. These cancers, and others, are less commonly focused on in the hereditary cancer guidelines but should not be forgotten.

NCCN testing
The vast majority of patients who met either ACMG or NCCN criteria for genetic counseling also received a genetic testing recommendation from NCCN. It should be noted that ACMG did not have analogous genetic testing recommendations, only counseling. Importantly, less than one-tenth (156/2221; 7.0%) of patients who met criteria for hereditary cancer risk had received a genetic test. This statistic underscores the urgency for more effectively identifying those at risk for cancer based on FHx.

Screened negative but had family history of cancer
The relatively high prevalence of cancers designated as "other" in this set of probands makes sense because these cancers would have little impact when evaluating guidelines for specific hereditary cancers. Indeed the guideline criteria from either ACMG or NCCN can say little about a family's hereditary cancer risk based on, for example, two unknown cancer diagnoses in family members. The logical components of the CPGs are ineffective without the clinical components provided by the patient. Probands with breast cancer, colorectal cancer, and/or melanoma account for the next largest trends of probands with cancer prevalent in their FHx that did not meet CPG criteria. For families with large numbers of cancer for which specific guideline criteria are implemented, either age of onset was too great or cancers were shared across both sides of the family in such a way as to fail to meet criteria.

Limitations and future directions
This study shows that FHx can be collected with userfriendly chatbots and that web services can provide access to risk assessment based on CPG criteria. We had a high screen positive rate of 45.2% which was likely due to participation bias. The dataset utilized here was collected via an ad campaign [17] and focused primarily on middle-aged women. Therefore, the results are skewed towards cancers more commonly found in women from that age group. Eligibility for genetics consultation and meeting clinical practice criteria are not perfectly aligned. The primary purpose of this study was to determine the criteria met and the patterns of cancer in patients who recorded their FHx. There very likely are patients who do not strictly meet criteria but are probably eligible for genetic consultation and testing. However, evaluating the suitability of the clinical practice guidelines for accurately identifying the patients who are eligible for counseling and testing lies outside the scope of this study. Furthermore, not every subguideline and criteria from ACMG and NCCN was implemented and not every implemented subguideline and criteria is represented in this analysis. Because the primary aim of this study was to demonstrate the feasibility of automating application of CPG criteria to probands with potential risk of hereditary cancer conveyed by reported FHx, only the subguidelines and criteria that probands met were included in the analysis. Therefore, the criteria met by probands reflects the demographic surveyed by skewing towards cancers most common in middle-aged women.

Conclusion
The primary aim of this research study was to demonstrate the feasibility of automating hereditary cancer risk assessment using chatbots, ontologies, and web services by comparing patients who participated and the CPG criteria they met. These results show hereditary cancer CPGs can be automated using chatbots and ontology-driven risk assessment. Additionally, the secondary aim of this research study was to provide a side-by-side comparison for ACMG and NCCN hereditary cancer CPGs. ACMG and NCCN Breast and Ovarian guideline criteria are the most similar across all criteria we compared, with the exception of the Gene Mutation criteria, which is an exact match. Colorectal and Endometrial guideline criteria have the greatest disparity in the number of probands who met these criteria, showing that despite their similar wording and structure, small changes to thresholds-including cancers and the range of family members the criteria apply to-can vastly affect the number of patients who meet criteria and receive recommendation to consider genetic counseling/testing consultation. Overall, probands who met criteria generally met more than one and often met criteria from multiple sub-guidelines within ACMG and NCCN.
Identifying patients who meet criteria from these guidelines is challenging and nuanced, but with the help of technological solutions, this burden can be lifted from off the shoulders of frontline care providers. More efficient and effective identification of patients who should consider genetic counseling and testing consultations will lead to earlier interventions and preventive measures that can improve outcomes among individuals at risk for hereditary cancer.