Development and evaluation of verbal fluency cut-off scores for the Brisbane Evidence-Based Language Test (EBLT) for use in clinical practice

ABSTRACT

The Brisbane Evidence-Based Language Test (EBLT) incorporates verbal fluency tasks (animals and words starting with ‘F’) although it does not currently provide cut-off scores for the verbal fluency subtest. The aim is to gain speech pathologists’ perceptions of the current scoring of the verbal fluency subtest (i.e., initial survey) and develop new verbal fluency cut-offs. An online survey study design was implemented and involved three online survey stages (1) initial survey, (2) weekly surveys and (3) final survey. Forty-two speech pathologists (age range: 23–66 years) participated in the initial survey and 23 clinicians participated in the weekly surveys. The verbal fluency cut-offs were calculated through diagnostic receiver operating characteristic sensitivity and specificity analysis comparing verbal fluency test scores with the binary (yes/no) language reference standard result. A cut-off score of ≤6 words on each task (animal task and words starting with ‘F’ task) was indicative of impaired performance. When not stratified by age, the combined animal and words starting with ‘F’ task cut-off score had a lower cut-off of ≤15 indicative of impairment. Overall diagnostic accuracy was greater when the animal and ‘F’ words tasks were combined. Participants reported that the new cut-offs were helpful, appropriate, in line with their clinical impression, patient presentation and met their needs as clinicians. The findings of this study indicate speech pathologists found the new verbal fluency cut-offs to be a useful addition in informing their decision-making in their clinical language assessments.

KEYWORDS:

• Aphasia
• stroke
• verbal fluency
• cut-off scores
• Brisbane Evidence-Based Language Test (EBLT)

Introduction

Stroke is a leading cause of mortality and morbidity globally and often results in ongoing impairments (Strilciuc et al., 2021). Approximately 30% of stroke survivors experience aphasia, which may affect the production and comprehension of written, spoken and/or signed language (Berg et al., 2020; Engelter et al., 2006; Grönberg, Henriksson, Stenman, & Lindgren, 2022). Language production, specifically word retrieval issues, commonly occur in people with aphasia (PWA) and are predominately assessed through picture naming, narrative production, and/or verbal fluency (Faroqi-Shah & Milman, 2018; Schuchard & Middleton, 2018). Verbal fluency tests require the participant to produce as many words relevant to a pre-determined category as possible in a set time frame, typically one minute (Kim, Kim, Kim, & Heo, 2011). These verbal fluency measures are often incorporated within aphasia assessments as they are a quick and efficient method of providing clinically relevant information about linguistic and executive functions (Bolla, Lindgren, Bonaccorsy, & Bleecker, 1990). Verbal fluency is incorporated in the Brisbane Evidence-Based Language Test (EBLT), which is an assessment that can be used to diagnose the presence or absence of post-stroke aphasia (Rohde et al., 2020). This language assessment assesses acquired language disorders in adult populations and is used in speech pathology clinical practice (Arnold, Wallace, Ryan, Finch, & Shrubsole, 2020). Currently, the EBLT does not provide cut-off scores for the verbal fluency subtest but only total scores (/45 for animals and /35 for /f/ words) are provided. The total scores of /45 for animals and /35 for /f/ words are maximum scores only and not cut-off scores indicating impaired/non-impaired language. These maximum scores were selected with the specific intention that they will not be achieved by any patient. These maximum scores were determined by the Brisbane EBLT psychometric dataset (n = 100 patients) where the highest patient score achieved for animals was 38 and 29 for /f/ words. These higher maximum scores (45 and 35) are therefore comfortably over what any patient achieved. If the maximum scores were lower (e.g., 25), and a patient could name 28 animals, they would be awarded 25 and 3 animal scores would essentially be ‘lost’. The current absence of clinical cut-off scores is limiting the clinical applicability of test results. Therefore, the purpose of this study was to develop and evaluate verbal fluency cut-off scores for the EBLT according to speech pathologists.

Word retrieval difficulties in PWA may present as slow responses, semantic and/or phonological errors and/or an inability to produce the target word (Galletta & Goral, 2018). Difficulties in word retrieval are theorised to result from damage to cognitive or language processes, which causes impaired lexical-semantic representations (Riès, Dronkers, & Knight, 2016). Lexical activation and selection are the overarching processes for word retrieval, where lexical activation is subconsciously achieved by identifying features that correlate to both semantic memory and the intended word (Riès et al., 2016). This process allows the formulation of lexical representations through semantic feature mapping (Faroqi-Shah & Gehman, 2021). Lexical selection occurs when the target word’s phonological form and articulatory plan are retrieved (Riès et al., 2016). Verbal fluency tasks are completed via this word retrieval process, although verbal fluency measures provide clinically relevant information on additional cognitive processes (Barry, Bates, & Labouvie, 2008).

Verbal fluency measures typically involve two tasks: phonemic fluency (i.e., words starting with the letter ‘F’) and semantic categories (i.e., animals) (Sarno, Postman, Cho, & Norman, 2005). The EBLT incorporates both of these tasks. Several cognitive processes are involved in verbal fluency tasks, including semantic memory, selective inhibition, attention, self-monitoring, processing speed, effort and atypical word searches dictated by phonemic and or semantic categories (Barry et al., 2008; Bolla et al., 1990; Gladsjo et al., 1999). Verbal fluency measures have been found to have clinically significant implications for measuring patient outcomes and determining intervention targets. A study by Sarno and Levita (1979) found that phonemic fluency tasks were the most sensitive measure of determining a change in patients in their first-year post-stroke. Four tasks were implemented with the study’s participants: visual naming, the Token Test, sentence repetition and verbal fluency, where the verbal fluency task was most successful at distinguishing the most and least improved patients (Sarno & Levita, 1979). Similarly, a study by Adamovich and Henderson (1984) found verbal fluency measures to be clinically significant in creating an intervention for PWA when patients’ verbal fluency responses were analysed. Responses on verbal fluency tasks indicated that PWA used semantic association strategies significantly less than language-intact and healthy control groups, whereas PWA instead relied on phonemic associations (Adamovich & Henderson, 1984). Verbal fluency assessments are reported to be widely used in clinical practice and useful not only in identifying lexical difficulties but also examining multiple elements associated with executive function (Jansson, Ortiz, & Barreto, 2020). The development of new cut-off scores specifically for this verbal fluency sub-test will therefore assist in the clinical interpretation of these executive features, which few other language sub-tests focus specifically on. The large number of responses obtained in this subtest (often 20–25 responses) also enables the calculation of clinically useful cut-offs. Other subtests (e.g., naming objects) often have limited response types (e.g., single words or phrases) and therefore clinically-useful cut-offs not able to be similarly calculated.

Verbal fluency performance is influenced by demographic factors including age, education, sex and ethnicity (Peña-Casanova et al., 2009). Age and education have been found to significantly impact verbal fluency performance (Acevedo et al., 2000; Barry et al., 2008; Kave, 2005). Higher age is associated with poorer performance on verbal fluency tasks, and higher education is associated with improved performance on verbal fluency tasks (Barry et al., 2008). Due to cerebral changes that occur with ageing, verbal fluency performance typically reduces (Barnes & Burke, 2006). This reduction in performance occurs because verbal fluency tasks involve executive ability in conjunction with effort and the ability to accurately retrieve and organise information, which are factors that are impacted by age-related cerebral changes (Barnes & Burke, 2006; Bryan & Luszcz, 2000; Plumet, Gil, & Gaonac'h, 2005). A study by Barry et al. (2008) reported that neurologically intact older adults perform more poorly and are more variable on verbal fluency tasks than younger populations. This study was conducted on healthy, neurologically intact participants; hence these results are not clinically relevant to stroke survivors. Tombaugh, Kozak, and Rees (1999) used neurologically intact participants and found that age had a greater impact on performance on semantic fluency tasks than education, and education was reported to have a greater impact on phonemic fluency tasks. Age was the factor used in this study to create more specific and clinically relevant cut-off scores due to the reported impact of age on verbal fluency performance. Stroke patients are typically older adults; hence there may be a reduction in verbal fluency performance which occurs with age in conjunction with potential stroke-related language and executive function difficulties.

While verbal fluency tasks are accurate indicators of linguistic and executive functions in PWA, there is limited research exploring the diagnostic accuracy of verbal fluency tasks in distinguishing between PWA and language intact stroke patients. A longitudinal study by Sarno et al. (2005) examined the changes in phonemic verbal fluency performance in 18 PWA stroke survivors. These participants were assessed on the letters ‘F’, ‘A’, and ‘S’ at three-month intervals while receiving post-stroke intervention for three months to one-year post-stroke (Sarno et al., 2005). Their verbal fluency results demonstrated an improvement in access to grammatical categories (e.g., nouns, verbs, modifiers, and function words) and significantly improved frequency of phonemic clusters when generating word lists at the end of the intervention (Sarno et al., 2005). This study's verbal fluency task results were not used for diagnostic purposes but rather to recognise the importance of qualitative aspects of verbal fluency (Sarno et al., 2005). This study also compared phonemic vs semantic tasks and performance in fluent vs non-fluent PWA (Sarno et al., 2005). Sarno et al. (2005) attempted to establish prevailing norms for an F-A-S verbal fluency task, reporting that a typical cut-off score would be 36 words given that 12 words would, on average, be produced per letter. The use of the neurotypical controls limited this study as it reduced the clinical application of the results within stroke populations. Another limitation was that there was no comparison of age on verbal fluency achievement, hence the likely decline of performance on verbal fluency tasks due to age was not accounted for in the cut-offs.

The Western Aphasia Battery-Revised (WAB-R) incorporates semantic verbal fluency in the assessment battery (termed ‘Word Fluency’); however, the verbal fluency task does not provide cut-off scores that distinguish between PWA and language-intact stroke patients (Kertesz, 2006). A maximum score of 20 is provided in the scoring guidelines of the WAB-R (Kertesz, 2006). The WAB-R incorporates a section detailing that semantic fluency is more important in assessing aphasia than phonemic fluency as the semantic fluency task allows further exploration of lexical access (Bolla et al., 1990). Furthermore, phonemic fluency tasks involve more executive functioning and are typically more difficult for neurotypical people and PWA. This task is deemed more sensitive for frontal deficits or dementia; however, a limitation of the study was that only neurologically intact participants were involved (Bolla et al., 1990).

A case control study by Faroqi-Shah and Milman (2018) comparing animal, phonemic, and action fluency in aphasia identified the relevance of qualitative analysis on verbal fluency and provided data for interpreting verbal fluency scores. This study involved 28 PWA and 40 age-matched neurotypical adults, all of whom were administered one phonemic (the letters ‘F’, ‘A’, ‘S’) and two semantic (actions and animals) verbal fluency tasks (Faroqi-Shah & Milman, 2018). Faroqi-Shah and Milman (2018) clearly outlined their aims and the methods for describing the implementation of the verbal fluency task and included a script for administering the task to the participant. This study found that verbal fluency tasks are impaired in PWA, particularly the animal verbal fluency task, and PWA produced lexically simpler answers compared to their neurotypical peers (Faroqi-Shah & Milman, 2018). Despite these findings, the study used a neurotypical control group; hence the results cannot be applied to identifying aphasia in stroke populations in clinical settings. Another limitation of this study was that sensitivity and specificity scores were not presented and verbal fluency performance was not compared with age.

A study by Kim et al. (2011) differentiated PWA and language intact stroke patients through a semantic verbal fluency task and created cut-off scores between the patient groups. It was determined that a 30 s administration time has discriminative validity to differentiate between the 53 stroke patients and 28 neurotypical control participants (Kim et al., 2011). This finding is important to the population as a reduced assessment time, from 60 to 30 s, could reduce frustration levels and administration time whilst gaining diagnostically valuable information (Kim et al., 2011). The PWA were compared to neurotypical control peers in conjunction with language-intact stroke patients, improving the study results’ clinical application (Kim et al., 2011). The cut-offs between the PWA and language-intact groups were 7.0 and 6.0 at the 60 and 30 s periods respectively (Kim et al., 2011). Similar to the previously discussed studies, this study did not consider age when creating the cut-offs, although the mean ages and years of education between the patient and control groups did not significantly differ (Kim et al., 2011). This study found that PWA performed poorer than language-intact stroke patients, and language-intact stroke patients performed poorer than the neurotypical controls (Kim et al., 2011). The authors attributed this pattern of performance to the involvement of executive functions and memory in verbal fluency tasks (Kim et al., 2011). Overall, there is a lack of research examining the verbal fluency cut-off scores stratified by age in conjunction with a lack of use of language-intact stroke survivors as controls.

Verbal fluency cut-off scores need to be developed for the EBLT to improve the clinical applicability of the verbal fluency subtest, thereby enhancing assessment and intervention planning post-stroke. The study’s first aim was to gain speech pathologists’ perceptions of the current scoring of the verbal fluency subtest and recommended areas for improvement (i.e., initial survey). The second aim was to develop new verbal fluency cut-offs that included age stratification and severity ratings. The final aim was to pilot and review the feedback to determine if the new cut-offs were appropriate for use in practice (i.e., weekly surveys).

Methods and methods

Study design

A repeated online survey study design was used to explore participant perspectives of the verbal fluency cut-off scores. There was a co-running study that used the same surveys but focussed on overall severity scores of the EBLT, those results were reported elsewhere. The Checklist for Reporting of Survey Studies (CROSS) and the Checklist for Reporting Results of Internet E-Surveys (CHERRIES) reporting guidelines were used in this study to guide the content and format of information (Eysenbach, 2004; Sharma et al., 2021).

Survey administration and study preparation

Participants were recruited through snowball sampling of the speech pathology email network: Speech Pathology Email Chats (SPECS). Clinicians registered with the Brisbane Test (brisbanetest.org) website were also informed of the study and potential participants were asked to circulate the study within their networks. The researchers involved in the study also contacted their professional contacts whom they thought might be interested in study participation.

Ethical considerations

The study was approved by De-identified research ethics committee.

Data collection methods

Data for participant experience was collected through three online survey stages, (1) initial survey, (2) weekly surveys and (3) final survey, using de-identified survey links through Qualtrics from May to July 2022. Survey administration involved a three-week recruitment period where the initial survey was sent to potential participants, followed by the first weekly survey sent three weeks later.

Participants were required to input a unique code into the initial survey to label their subsequent surveys. This technique preserved participant anonymity while simultaneously allowing researchers to link their responses over time.

Stage 1: initial survey

Participants were sent an initial 5-to-10-min online customised mixed methods survey. This survey had 13 questions with multiple-choice options or free-text boxes. The questions included general demographic questions such as gender, age, clinical caseload and work environment, the number of years working as a speech pathologist and participants’ general usage and perceptions of the scoring of the EBLT and verbal fluency subtest (see Appendix 1). Verbal fluency cut-off scores were provided at the end of the survey.

Calculation of the verbal fluency diagnostic accuracy cut-off scores. The original diagnostic-accuracy data set from the Rohde et al. (2020) cross-sectional validation study was used to calculate the verbal fluency cut-off scores. The EBLT, including the verbal fluency subtest, was conducted with 100 consecutive acute stroke patients (mean age of 66.49 years) admitted to two tertiary hospitals in Brisbane, Australia. Each eligible stroke patient was administered two assessments, randomised in order of administration. The first assessment was the EBLT (including the verbal fluency subtest) and the second was a reference standard measure which was based on a composite of different language and other clinical measures which together formed a binary yes/no result determining whether or not the patient presented with aphasia (Rohde et al., 2020). The resulting 100 stroke patients were therefore categorised into two groups depending upon the results of this reference measure: (1) stroke patients with aphasia and (2) stroke patients without aphasia. These clinical groups were selected as they are reflective of the populations seen in stroke care and assists in differentiating between the lexical and cognitive processes that influence verbal fluency impairment with language-intact stroke populations. Healthy (neurologically intact) participants were not included as the control group as comparison of performance between healthy populations against PWA post-stroke also evaluates the impact of stroke in general (and not aphasia specifically). The performance of these language-intact groups does not assist in identifying language impairment in stroke (Rohde et al., 2018).

The verbal fluency cut-off scores were calculated to determine the presence or absence of impairment (i.e., diagnostic accuracy cut-off scores) based on performance on this language task. These cut-off scores were calculated for the semantic task (i.e., animal naming), the phonemic task (i.e., words starting with the letter ‘F’) and the combined semantic and phonemic tasks. Additional age-specific cut-offs were also calculated through stratifying the data into age categories for the combined ‘animals’ and ‘F’ words score. This was achieved by analysing the 100 stroke participant data from the Rohde et al. (2020) study into three distinctive age categories (25–50, 51–75 and 76–100). The development of severity categories indicating if a verbal fluency impairment is ‘mild’, ‘moderate’ or ‘severe’ was also undertaken for these scores stratified by age. The limitations of the sample size (only 100 stroke patients) meant the data was able to be stratified according to one variable (age) only. Stratification of the data according to additional variables (sex, education etc.) would have resulted in too few datapoints within each category. Age was selected as the variable of interest due to the body of research indicating age to be a strong predictor of verbal fluency performance (Barnes & Burke, 2006; Barry et al., 2008).

Diagnostic cut-off scores were created through sensitivity and specificity receiver operating characteristic (ROC) curve analysis. The method of analysis for diagnostic accuracy in this study was the same method used by the Rohde et al. (2020) study. As per Rohde et al. (2020), Stata/IC 13.0 was used for diagnostic analysis. Verbal fluency test scores were used to calculate diagnostic accuracy by comparing these scores with the binary (yes/no) language reference standard result (composite reference of multiple language tests and clinical opinion) determining whether or not the patient presented with aphasia (Rohde et al., 2020). This comparison was used to determine test specificity and sensitivity and positive (+LR) and negative likelihood ratios (−LR) (Rohde et al., 2020). The specificity and sensitivity scores were evaluated at each cut-off threshold to determine the diagnostic verbal fluency cut-off scores (i.e., scores used to discriminate between the presence or absence of impairment) (Rohde et al., 2020).

As per the full version of the EBLT, two cut-off scores were provided, which helped optimise the test's clinical utility and assisted in determining the presence or absence of a language condition. The lower verbal fluency threshold scores were determined from the cut-off score which produced the highest sensitivity for a specificity ≥90%. A score at or below these cut-offs indicated the likely presence of impairment (Rohde et al., 2020). Higher cut-offs were created to determine the likely absence of impairment (Rohde et al., 2020). The higher cut-offs were created from threshold scores that yielded sensitivities of ≥90% (Rohde et al., 2020). A score at or above these cut-offs indicated the likely absence of impairment. At-risk scores were those that fell in the range between the higher and lower cut-offs and indicated a risk of impairment.

Calculation of the severity range verbal fluency cut-off scores. Severity range cut-off scores were created to quantify verbal fluency scores into mild, moderate and severe categories. These severity ranges were calculated by dividing the lower cut-off score (discriminating between impaired and non-impaired language performance) into three equal categories to achieve the severe, moderate and mild severity ranges. The scores were rounded into whole numbers to ensure they were clinically relevant and no overlap in score categories was achieved.

Stage 2: weekly surveys

The aim of the weekly surveys was to obtain feedback on participants’ experiences with the newly-developed verbal fluency cut-off scores in practice. The weekly survey contained 15 questions which remained the same over the six weeks. Seven of the 15 questions specifically surrounded verbal fluency and included the frequency of use of the verbal fluency subtest, the populations this subtest was used on, perceptions of the cut-off scores, if they were meeting the needs of the clinician, and if any concerns had arisen or changes could be made. The remaining eight of the 15 questions were focussed around the severity guidelines for the whole EBLT and asked about how participants were finding the overall guidelines, whether the guidelines were in line with their clinical impression, meeting their needs as clinicians, and whether there were any concerns or changes to be made. The questions were a combination of short answer and multi-choice formats (see Appendix 2).

Stage 3: final survey

In the final week of data collection, participants were asked to complete a 5-minute survey to give their final perceptions of the verbal fluency cut-off scores and their intended use in clinical practice. Five questions were specific to the verbal fluency subtest in multi-choice and open-ended questions (see Appendix 3).

Survey development and piloting

The three surveys were pretested twice each by the research team and feedback was used to ensure the functionality and usability of the surveys to maximise completion rates before distribution to the participants. The research team included final year speech pathology honours students and registered speech pathologists. The participants used for pretesting were employed in metropolitan hospitals in Australia.

Sample characteristics of survey responses

To be eligible for inclusion in the study, participants had to be practicing speech pathologists who worked in an adult caseload and used the EBLT in their clinical practice. Participants were informed that there would be weekly surveys as part of the study, although they were able to withdraw at any point. One female participant withdrew from the study following the release of the first weekly survey as a result of infrequent use of the EBLT. Exclusion criteria included people who were not speech pathologists and speech pathologists who did not use the EBLT in their clinical practice.

Statistical analysis

The survey data were screened for complete and incomplete responses, where nine respondents with one or fewer questions answered were removed from the data set. Furthermore, only logical and interpretable data was included. An additional entry was excluded from the study as the participant did not meet inclusion criteria (i.e., they were a student). Data analysis of quantitative survey data was completed using Microsoft Excel and SPSS software. Quantitative data was analysed using descriptive statistics, specifically counts and means. Qualitative survey data was analysed using Graneheim and Lundman’s (2004) qualitative content analysis to determine categories and sub-categories. This was achieved by firstly reading through the responses several times to gain an understanding of potential sub-categories. Responses were divided into condensed meaning units and were labelled into codes. The codes were sorted into sub-categories and then categories which held the overall meaning of the responses. All researchers discussed the codes and sub-categories to maximise the study's rigour.

Results

Respondent characteristics

Initial survey

Forty-two speech pathologists (41 females and one male, aged 23–66 years) participated in the initial survey. Participants had variable years of experience in the profession, ranging from 2 to 43 years and worked in countries including Australia, New Zealand, the United States of America, Canada, India, and the United Kingdom. Participants worked in numerous workplaces including acute and rehabilitation hospital settings in metropolitan areas, private practices, rural community hospitals and outpatient settings. The participant demographics for all participants who responded to the initial survey are reported in Table 1.

Table 1. Initial and weekly survey – participant demographics.

Characteristic	Initial Survey (n = 42)	Weekly Survey (n = 23)
Age	Mean 37.6 years (range 23–66)	Mean 37.2 years (range 23–66)
Gender	Females 97.6% (41) Males 2.4% (1)	Females 95.6% (22) Males 4.4% (1)
Identify as Indigenous	Indigenous 4.7% (2) Non-Indigenous 92.8% (39) Prefer not to say 2.4% (1)	Indigenous 8.7% (2) Non-Indigenous 86.9% (20) Prefer not to say 4.4% (1)
Years of experience	11.95 years (range 2–43)	11.15 years (range 2–43)
Current country	Australia 42.8% (18) United States of America 33.3% (14) Canada 4.8% (2) England 4.8% (2) New Zealand 2.4% (1) Scotland 2.4% (1) Wales 2.4% (1) United Kingdom 2.4% (1) India 2.4% (1) Not reported 2.4% (1)	Australia 47.8% (11) United States of America 26% (6) Canada 4.4% (1) England 8.7% (2) Wales 4.4% (1) United Kingdom 4.4% (1) India 4.4% (1)
Current area of work	Acute hospital setting 27.2% (25) Metropolitan area 14.1% (13) Inpatient Rehabilitation/Rehabilitation hospital setting 18.5% (17) Rural area 6.5% (6) Community/ hospital 6.5% (7) Out-patient clinic 12% (11) Private practice 5.4% (5) In-patient clinic 2.2% (2) Home Health 1.1% (1) Skilled Nursing Facility 1.1% (1) Residential aged care 1.1% (1) Medical college programme 1.1% (1) Not otherwise specified 2.2% (2): NHS, regional area	Acute hospital setting 23.1% (12) Metropolitan area 19.2% (10) Rehabilitation hospital setting 17.3% (9) Rural area 1.9% (1) Community hospital 7.7% (4) Out-patient clinic 11.6% (6) Private practice 7.7% (4) In-patient clinic 1.9% (1) Home health 1.9% (1) Residential aged care 1.9% (1) Medical college programme 1.9% (1) Not otherwise specified 3.8% (2): NHS, regional area
Patient caseload (n = 23)	Stroke patients 42.9% (18) Traumatic Brain Injury 16.7% (7) Neurodegenerative conditions − Dementia 7.1% (3) − Parkinson’s disease 4.8% (2) Withdrawal 2.4% (1) Failure to thrive 2.4% (1) Cancer patients 2.4% (1) Generalist caseload 7.1% (3) Unspecified neurodegenerative conditions 2.4% (1) Unspecified neurological patients 11.9% (5)

Note: Numbers have been rounded to the nearest whole number.

Weekly surveys

Of the 42 participants who completed the initial survey, twenty-three clinicians (22 females and 1 male, aged 23–66 years) then participated in the weekly surveys. These 23 participants had a range of work experience as speech pathologists from 2 to 43 years. Participants reported that they worked in Australia, the United States of America, Canada, England, the United Kingdom and India (Table 1). Clinician participation was variable for each weekly survey (Figure 1). Participation had an overall decline from the initial survey to the final survey. The lowest participation rate was for the week 6 survey where two clinicians participated.

Figure 1. Flow chart of the number of participants for each survey.

Main findings

Current use and perceptions of the verbal fluency subtest

Initial survey. Twenty-three of the 42 participants reported their use and perceptions of the verbal fluency subtest. The verbal fluency subtest was used at varying frequencies in a typical week, ranging from no use to being implemented daily. Of the 23 participants, 19 responded to how often they used the verbal fluency subtest in a typical week, where 17 used the verbal fluency subtest on a weekly basis, with the majority using the subtest approximately one to two times per week (range: 1–7). Three participants did not provide numerical equivalents to their weekly use with one reporting they did not use the test ‘often’, another reporting they use the test ‘almost daily’ and another stating they were unaware of a separate verbal fluency component. Results of these frequencies are summarised in Figure 2.

Figure 2. Initial survey: frequency of use of verbal fluency subtest in a typical week.

Survey questions which sought qualitative information on participants’ experiences and perceptions of the verbal fluency scores were analysed using content analysis. Each participant was assigned a number from one to 23 to ensure consistency in reporting throughout the weekly surveys. Table 2 reports the results from the initial survey question: ‘What are your current thoughts on scoring the verbal fluency component of the EBLT? Is it specific enough for your needs?’. Four participants (P4, P12, P19, P22) reported that the scoring was appropriate; however, the majority (n = 18) of participants reported that the current scores should be modified due to various reasons: norms would be beneficial (n = 2), the scores are not specific enough (n = 2), scores are too high (n = 3), scores underestimate functional abilities (n = 3), are a poor measure of verbal output (n = 1) or do not capture strengths (n = 1). One participant (P16) provided a perspective which was not relevant to the verbal fluency subtest and instead commented on the picture description subtest; this comment was removed from content analysis.

Table 2. Initial survey: content analysis of current perceptions of scoring the verbal fluency subtest with categories, subcategories and quotes.

Categories	Subcategories	Quotes
Unsupportive perceptions and constructive feedback	Scores are not specific enough	P1: ‘No. I think the high scores throw out the overall expressive score (which then confuses doctors when we rate severity)’ P8: ‘Horrible. Does not capture a good spectrum of my patients. Need more categories’
	Scores are too high	P2: ‘What you score it out of seems too high.’ P20: ‘No – it's unrealistic. I also feel it doesn’t account for typical WFDs in the aging population’ P21: ‘The cut off always seems very high. I tend to qualitatively comment on performance and also use TPCH (R) hem screen norms as rough guide’
	Participant uses alternative measures	P3: ‘I use it, but I often refer to the norms utilised for this within other assessments.’ P15: ‘We use noun fluency, phonemic fluency, COWA, verb fluency. However not many slps practice this.’
	Scores underestimate functional abilities	P7: ‘I find that patients often score low on this section but are more fluent in conversation than this score alone predicts’ P13: ‘No – I have never met a patient who scored above 20–30 in total across the two verbal fluency tasks and this massively reduced the overall score, when in function there are few issues’ P14: ‘Usually makes patients appear worse than they are. In the acute setting, patients often just give up.’
	Poor measure of verbal output	P10: ‘Not at all, generally if it is below 11 or 12 it is more concerning but generally it is not a good measure of verbal output.’
	Norms would be beneficial	P11: ‘No. Norms would be useful’ P23: ‘No – norms would be advantageous’
	Scores do not capture strengths	P18: ‘Doesn't specify semantic networks or capture strengths’
Supportive feedback	Scores are appropriate	P4: ‘For a verbal fluency task I find it appropriate – I wouldn't regularly use it in isolation’ P12: ‘Yes’ P19: ‘Y but only part of assessment’ P22: ‘Yes – however I do on occasions come back and do further in-depth assessment of this component but for the purposes of the assessment tool I feel there is enough information.’

Verbal fluency cut-off scores

New verbal fluency scores were created for the participants to pilot in their clinical practices. To reflect the existing format of the diagnostic cut-offs for the EBLT, two cut-off scores for the animal task and words starting with ‘F’ task were developed: a lower cut-off (indicating likely presence of impairment) and a higher cut-off (indicating likely absence of impairment). An ‘at-risk zone’ between these two cut-offs indicates possible impairment (Table 3). Where possible, severity ranges (indicating mild, moderate or severe impairment) were also calculated.

Table 3. Diagnostic accuracy characteristics for animals and words starting with ‘F’ (separate task analysis).

Verbal Fluency Tasks	Possible score range	Patient scores	Age range (years)	Cut-off score (≥90% specificity)	Sensitivity	Specificity	LR+	LR−	At risk zone	Cut-off score (≥90% sensitivity)	Sensitivity	Specificity	LR+	LR−	Area Under Curve, SE (95% CI)
Animals	0–45	0–39	26–97	≤6*	60.27%	96.30%	16.27	0.41	7–18	≤18	91.78%	55.56%	2.06	0.15	0.867, 0.038 (0.793–0.941)
‘F’ Words	0–35	0–29	26–97	≤6*	71.23%	92.59%	9.62	0.31	7–10	≤10	93.15%	66.67%	2.79	0.10	0.89, 0.038 (0.815–0.965)

*Range too small to quantify meaningfully into severity categories.

Animal and ‘F’ word separate task analysis. Cut-offs that indicate the presence of verbal fluency impairment were created by determining the scores with the highest sensitivity for a specificity of ≥90%. These cut-off parameters were selected to achieve the highest sensitivity as possible (optimise the test’s ability to accurately identify aphasia) while still maintaining a high level of specificity (≥90%) and ensure that the test is also able to accurately identify when the person does not present with language-impairment (Rohde et al., 2020). This method was used for the animal verbal fluency and words starting with ‘F’ tasks. A cut-off score of ≤6 words for the animal task yielded a sensitivity of 60.27% and a specificity of 96.30% with LR + 16.27 and LR − 0.41. A cut-off of ≤6 words for the words starting with ‘F’ task had a sensitivity of 71.23% and a specificity of 92.59% with LR + 9.62 and LR − 0.31. Severity analysis (the creation of severity score ranges ‘mild’, ‘moderate’ and ‘severe’) was also attempted for both animals and ‘F’ word tasks however, as both generated a lower cut-off score of 6, the score range between 0 and 6 was too small to be meaningfully quantified into three separate severities. For these separate tasks no mild, moderate, or severe ranges were consequently able to be provided. The separate cut-off scores for animals and ‘F’ words are reported in Table 3.

Combined animal and ‘F’ word analysis. Cut-off scores were also created for the combined animals and ‘F’ words total task scores. This analysis combined scores across the two tasks to create a single verbal fluency score and the combined animal and words starting with ‘F’ task was separated into mild, moderate and severe categories. Cut-offs were calculated when not stratified for age and also when stratified by age (Table 4). The combined task cut-off score when not stratified by age had a lower cut-off of ≤15 indicative of impairment and had a sensitivity of 68.49% and a specificity of 92.59% with LR + 9.25 and LR − 0.34.

Table 4. Diagnostic accuracy characteristics – combined task analysis (animals and ‘F’ words).

Verbal Fluency Tasks	Patient scores	Severe	Moderate	Mild (lower cut-off score) (≥90% specificity)	Sensitivity	Specificity	LR+	LR−	At risk zone	Cut-off score (≥90% sensitivity)	Sensitivity	Specificity	LR+	LR−	Area Under Curve, SE (95% CI)
Animals & ‘F’ Words combined (26–97 years)	0–68	0–5	6–10	11–15	68.49%	92.59%	9.25	0.34	16–29	≤29	90.41%	62.96%	2.44	0.15	0.897, 0.031 (0.835–0.959)
Combined score by age: 25–50 years	0–54	0–9	10–19	20–29	84.62%	100%	N/A	0.15	30–36	≤36	92.31%	60%	2.31	0.13	0.946, 0.05 (0.848–1)
Combined score by age: 51–75 years	0–50	0–8	9–16	17–24	86.49%	100%	N/A	0.13	25–30	≤30	91.89%	84.31%	2.57	0.13	0.941, 0.03 (0.881–1)
Combined score by age: 76–100 years	0–34	Range too small to quantify meaningfully into severity categories		≤7	65.22%	100%	N/A	0.35	8–25	≤25	91.30%	50%	1.83	0.17	0.84, 0.072 (0.699–0.98)

Note. The possible score range was 0–80 across each age range.

Cut-offs were stratified into three age categories (from age 25 to 100 years) grouping into approximately 25 years in each. The 25–50 years age range for the combined task had a lower cut-off of ≤29 which had a sensitivity of 84.62% and a specificity of 100% with LR − 0.15. A lower cut-off of ≤24 was determined for the 51–75 years age range for the combined task which yielded a sensitivity of 86.49% and a specificity of 100% with LR − 0.13. For the 76–100 years group there was a small lower cut-off of seven that could not be quantified into severity categories. The 76–100 age range had a sensitivity of 65.22% and a specificity of 100% with LR − 0.35. Overall, combining the animal and words starting with ‘F’ tasks creates a marginal greater overall diagnostic accuracy as determined by a greater AUC curve (0.897) compared to the individual tasks (Table 4).

Results and perceptions of the verbal fluency cut-off scores

Week 1 survey. The verbal fluency subtest was used by participants throughout the first week between 0 and 5 times (Appendix 4). Five participants (P7, P9, P10, P19, P23) reported they did not use the cut-off scores, two (P16, P17) implemented them twice, one (P4) implemented them three times, and another (P1) used them five times. Participants perceived the cut-offs to be ‘very helpful’ (P1, P4, P16, P17) (Appendix 5). Participants who used the cut-offs found the scores beneficial and reported that the scores met their needs (Appendix 6). Relative to clinical impression and presentation of the clients, 33.3% of participants (n = 3) found the cut-offs strongly in line with their clinical impression, 11.1% (n = 1) found the cut-offs somewhat in line and 55.6% (n = 5) neither agreed nor disagreed (Appendix 6). Three participants (P4, P7, P16) did not have concerns with the cut-offs and one participant (P1) suggested that it would be beneficial to separate the severity scores and commented that more information about the ‘at-risk’ category would be helpful (Appendix 5). Participants (P1, P16, P17) commented that no changes were necessary (Appendix 5).

Week 2 survey. The cut-off scores were implemented from 0 to 3 times during the week. Five participants did not use the cut-off scores (P2, P4, P16, P19, P23), one participant (P1) used the scores once, and two (P17, P11) used the scores three times. Three participants (P1, P4, P17) reported that the cut-offs were appropriate and useful in practice and one (P11) raised a concern that the scores were ‘a bit high’ (Appendix 5). The cut-off scores met the needs of five participants (62.5%), while 37.5% (n = 3) reported they were unsure as to whether the cut-offs met their needs as clinicians (Appendix 6). Three participants (42.9%) reported that they strongly agreed the cut-offs were in line with their clinical impression and client presentation, 28.6% (n = 2) somewhat agreed, and neither agreed nor disagreed (Appendix 6).

Week 3 survey. The verbal fluency cut-offs were used from 0 to 3 times during week 3. Two participants (P2, P16) did not implement the cut-offs over the week, one (P21) used the cut-offs once, one (P1) used the cut-offs twice and one participant (P17) used the cut-offs three times. Two participants (P1, P17) reported that the cut-offs were appropriate and useful in clinical practice (Appendix 5). All participants believed that the cut-offs met their needs and 75% (n = 3) of participants strongly agreed that the cut-offs were in line with their clinical impression and client presentation and 25% (n = 1) somewhat agreed (Appendix 6). No changes were suggested by participants (Appendix 5).

Week 4 survey. Participants implemented the cut-off scores between 0 and 3 times with three participants (P1, P4, P16) not implementing the cut-offs in the past week and two participants (P17, P21) implementing the cut-offs three times. The two participants (P17, P21) that implemented the scores reported that the cut-offs were appropriate and good for documentation (Appendix 5). One participant (P17) reported that the cut-offs were meeting their needs and strongly agreed that they were in line with their clinical impression and client presentation and one (P21) was unsure that the cut-offs met their needs and somewhat agreed that the cut-offs were in line with their clinical impression (Appendix 6). No changes were reported (Appendix 5).

Week 5 survey. The participants (P4, P17, P21) did not implement the cut-off scores in week 5. There were no reports of perceptions of the cut-offs, although one participant (P17) believed the cut-offs met their needs as a clinician and neither agreed nor disagreed that the cut-offs were in line with their clinical impression and client presentation (Appendix 6). No changes were reported (Appendix 5).

Week 6 survey. Two participants (P16, P17) completed the survey with one (P16) not implementing the cut-offs and one (P17) implementing the cut-offs once. One participant (P17) reported that the cut-offs were appropriate, believed they were meeting their needs as a clinician, and strongly agreed that the cut-offs were in line with their clinical interpretation and client presentation (Appendix 6). There were no reported changes (Appendix 5).

Final survey. Two participants (P4, P17) reported that the verbal fluency cut-offs were useful in clinical practice and one participant (P19) did not implement the cut-offs over the week due to an inappropriate caseload. Two participants (P4, P17) reported that they would use the new cut-offs routinely in clinical practice and perceived the cut-offs to help with clinical decision-making (Table 5). One participant (P19) was unsure if they would routinely use the cut-offs and if the new cut-offs helped with clinical decision-making (Table 5). All participants (P4, P17, P19) would encourage other speech pathologists to use the cut-offs in their clinical practice (Table 5). There was no relevant feedback for the verbal fluency cut-off scores and one response was removed from the data as it was not relevant to the verbal fluency subtest.

Table 5. Final survey: analysis of quantitative data for question 3, 4 and 5.

Questions	Scale responses	Percentages and participant numbers
Q3. Do you think you would use the new verbal fluency cut-offs routinely in your clinical practice?	Yes	66.7% (n = 2) P4, P17
	Unsure	33.3% (n = 1) P19
Q4. Do the new verbal fluency cut-offs help inform your clinical decision-making?	Yes	66.7% (n = 2) P4, P17
	Unsure	33.3% (n = 1) P19
Q5. Would you encourage other speech pathologists to use these verbal fluency cut-offs in their clinical practice?	Yes	100% (n = 3) P4, P17, P19

Discussion

This study was the first to evaluate current and newly developed cut-off scores for the verbal fluency subtest in the EBLT. The majority of participants perceived the current scoring as not specific enough, too high, underestimated functional abilities and a poor measure of verbal output. In response to participant feedback new verbal fluency cut-off scores were developed to provide clinically relevant information to speech pathologists and multidisciplinary teams that may assist with intervention planning and decision-making. This study also evaluated the newly developed cut-off scores through clinician perspectives. The majority of those that implemented the new cut-off scores reported that they were helpful, appropriate, in line with their clinical impression and presentation of the patient and were meeting their needs as clinicians.

A finding of this study was that speech pathologists wanted more specific information from the verbal fluency cut-off scores in the EBLT. While this is the first study to find that speech pathologists are seeking more specific cut-offs for the EBLT, similar findings have been found by Sarno et al. (2005) who determined that qualitative and quantitative information derived from verbal fluency tasks have been used to help achieve definite and individualised hypotheses surrounding prognosis in PWA. This finding supports that clinicians are seeking more specific and definite information on recovery from language impairments and highlights the importance of clinicians gaining in-depth information on patient abilities. Hersh, Wood, and Armstrong (2018) determined that a comprehensive language assessment was crucial in post-stroke rehabilitation as it facilitates intervention through language baselines, diagnosis, monitoring progress, goal setting and intervention and discharge planning. The study by Hersh et al. (2018) provides potential reasons why speech pathologists want more specific cut-offs to inform their clinical decision-making.

As a result of speech pathologists’ perceptions towards the current verbal fluency cut-offs, new cut-off scores were created for the animal and words starting with ‘F’ tasks which were divided into severity categories. The at-risk and higher cut-off scores were lower for the words starting with ‘F’ task, suggesting that this phonemic task was more difficult. Various studies have calculated the average scores for phonemic fluency tasks although these studies were conducted on neurologically intact adults and hence did not provide cut-offs indicating the presence or absence of impairment (Ruff, Light, Parker, & Levin, 1996; Spreen & Risser, 2003). Kim et al. (2011) compared PWA stroke patients to language-intact stroke patients on a semantic fluency task and created a cut-off score of 7.0 that indicated impairment. This cut-off was similar to the cut-off created for the semantic verbal fluency task in this study (cut-off score of ≤6) which indicates impairment. In the current study, combined animal and words starting with ‘F’ cut-offs were created and stratified for age and severity categories. The cut-off scores had varying degrees of diagnostic accuracy which varied between tasks and age categories. Overall diagnostic accuracy was greater when the animal and words starting with ‘F’ tasks were combined as determined by a greater AUC curve. These developed cut-offs are able to be implemented in speech pathology practice, particularly since the cut-offs were created using data from stroke patients as opposed to neurologically intact participants.

Another study finding was that the phonemic verbal fluency task was likely to be more difficult than the semantic fluency task as the at-risk and higher cut-off scores were lower for the words starting with ‘F’ task. This finding has been reported in other studies. A study by Shao, Janse, Visser, and Meyer (2014) noted that the task demands were different where semantic fluency tasks allow the participant to use existing links between concepts to elicit responses. This is compared to phonemic fluency tasks which require the participant to retrieve the words from a phonemic category which is not often completed in speech production and is hence a novel retrieval strategy (i.e., more so a measure of executive ability) (Luo, Luk, & Bialystok, 2010; Shao et al., 2014). A study by Sarno et al. (2005) similarly found that phonemic fluency tasks are unnatural linguistic tasks as words are typically retrieved through semantic association instead of through sounds. This overall finding that phonemic verbal fluency tasks are more difficult and indicate executive abilities as opposed to verbal ability (more associated with semantic tasks) may impact clinicians’ qualitative and quantitative interpretations of the individual task scores (Shao et al., 2014).

This study stratified the verbal fluency cut-off scores by age due to the influence age has on verbal fluency performance. Increased age is associated with poorer performance on verbal fluency tasks, hence by stratifying the new cut-off scores by age the cut-offs are more specific and clinically relevant (Barry et al., 2008). This allows the cut-off scores to be integrated into speech pathology clinical practice, thereby providing clinicians with a more specific diagnostic measure of impairment. The majority of participants involved in this study found that the cut-offs were helpful (e.g., P1: ‘Very helpful in clearly documenting the severity of aphasia associated with stroke’) and were appropriate and useful in practice (e.g., P21: ‘Good descriptors/guide for documentation’). One participant (P11) was concerned that the cut-offs were too high and another participant (P1) suggested that more information about the ‘at-risk’ category would be beneficial. Similarly to verbal fluency tasks, age effects have been reported to influence other post-stroke language tasks including picture naming, whereby reaction times to name pictures were notably reduced with increased age (Gordon & Cheimariou, 2014). Simos, Kasselimis, Potagas, and Evdokimidis (2014) reported significant effects of age on sentence-level auditory comprehension where performance decreased as age increased.

The cut-off scores for the combined animal and words starting with ‘F’ task were divided into severity categories (e.g., mild, moderate and severe). By dividing the scores into severity categories, the clinical applicability of the scores was increased as clinicians would be able to gain more valuable and specific information on diagnostic accuracy which may aid with areas of further assessment or intervention planning. Incorporating severity categories in the cut-offs are useful for clinicians as part of their assessments as it provides additional information that may be used in documentation or description of assessment findings (Khadka, Gothwal, McAlinden, Lamoureux, & Pesudovs, 2012). Severity categories may also be used to facilitate the planning of additional assessment items or aid with intervention targets (Khadka et al., 2012). The severity cut-off scores for the combined animal and words starting with ‘F’ task may be implemented by clinicians in their clinical practice which may benefit clinical documentation, further assessment and intervention planning and may provide a method for clinicians to describe results to patients effectively.

Strengths and limitations

A key strength of this study was the participant diversity. Participants ranged in age, gender, race and country (e.g., from six countries) and area of work (e.g., from acute to residential care facilities). This enabled feedback from clinicians practicing in a wide range of settings. This diversity influenced another key strength of the study, namely the study’s generalisability. As a result of implementing the cut-off scores in a range of settings, populations and by multiple clinicians, the findings of this study could be applied to a broader clinical context. Another strength of this study was the number of times participants got the opportunity to provide feedback, where participants were provided with six weekly surveys over two months. This enabled participants to provide their perceptions at any time during the survey if any concerns were raised following clinical implementation.

A study limitation was the low response rate and high attrition rate. Forty-two clinicians participated in the initial survey but of those participants, 23 participated in the weekly surveys. The highest response rate for a weekly survey was for the first survey with 10 participants which decreased throughout the weeks, with the lowest response rate at two participants for the week 6 survey. The high attrition rate reduced the number of perspectives received for the cut-off scores, particularly as the study progressed. While this attrition may be associated with the high frequency of the surveys (weekly), this rate of attrition as the study progressed may also indicate that participants did not have concerns or that their concerns were resolved as there were numerous opportunities for participants to provide feedback throughout the study duration. Further research is warranted in this area including additional re-testing of the cut-off scores in a larger sample size. Another limitation was that the cut-off scores were stratified by age only and did not consider other factors that may influence performance on verbal fluency tasks such as education and ethnicity.

Potential areas for future research

Areas for future research would be to explore the potential differences that different dialects may have on verbal fluency cut-off score interpretation (e.g., Aboriginal English dialect) and to analyse other factors influencing verbal fluency performance, as this study focused on age. However, other factors, such as education, were not examined.

Verbal fluency tasks can be used post-stroke to provide qualitative and quantitative information on a patient’s language abilities. The existing EBLT cut-off scores provided clinicians only with diagnostically accurate scores to determine the likely absence or presence and severity of language impairment. However, this study found that clinicians were wanting verbal fluency specific cut-off scores to inform their clinical decision-making for their patient populations. The new cut-off scores were created to rectify this clinical gap. The combined animal and words starting with ‘F’ verbal fluency tasks were found to be slightly more diagnostically accurate than the individual verbal fluency tasks. Following implementation of the new scores in clinical practice, study participants reported that the new cut-offs were appropriate, met their needs as clinicians, and aligned with their clinical interpretation and client presentation. The findings of this study could be used by clinicians to provide information on language impairments to speech pathologists to guide clinical decision-making and intervention planning post-stroke.

Acknowledgements

The authors thank the study participants for their contribution to this research and are grateful to (De-identified for review) for helping create and distribute the surveys.

Disclosure statement

Alexia Rohde is the creator of the Brisbane Evidence-Based Language Test.

Additional information

Funding

The author(s) reported there is no funding associated with the work featured in this article.

Appendices

Appendix 1. Initial survey questions and response options

Table

Questions	Response option/s
(Initial information message and consent) Development and evaluation of subtest and severity cut-offs for the Brisbane Evidence-Based Language Test You are invited to participate in an online survey exploring the development of subtest and severity cut-off scores for the Brisbane Evidence-Based Language Test. The project involves completing a single, short 13 item survey. It is anticipated that the survey will take you 5–10 min to complete. Then you will be asked to trial the new cut-offs in your clinical practice and complete a series of 5 min weekly email surveys sharing your perspectives of the new cut-offs, and a final 5 min survey exploring your perspectives of the cut-offs. You will trial the cut-offs for 8 weeks in your clinical practice if you choose to participate. The study has ethical approval through (De-identified research ethics committee), and is being conducted as part of two 4th year speech pathology honours research projects. Please forward this survey link to any colleagues you think might be interested in participating in the project. Participation in the survey is entirely voluntary. By selecting ‘yes’ below, you are consenting to be involved in this project. If you do not wish to participate, please select ‘no’ and follow the prompts to exit the survey. Thank you for your support, (Author names de-identified for review)	Multiple-choice options: Yes/No
Q1. For this study you will create a unique code by combining: the last number of your birth year, the initial of your first name and the first letter of your birth month. (For example: 1AJ). You will enter this 3-digit code into Qualtrics before completing the weekly and final surveys.	Text box
Q2. What gender do you identify as?	Multiple-choice options: Female/Male/(Non-binary/gender diverse)/My gender identity isn’t listed. I identify as: (Text box)/Prefer not to say
Q3. What is your age?	Text box
Q4. Do you identify as Indigenous?	Multiple-choice options: Yes/No/Prefer not to say
Q5. How many years have you worked as a speech pathologist?	Text box
Q6. What country do you currently work in?	Text box
Q7. Where are you currently working? Please select all that apply.	Multiple-choice options: Rural area/Metropolitan area/Acute hospital setting/Rehabilitation hospital setting/Community hospital/In-patient clinic/Out-patient clinic/Private practice/Other: Please specify (text box)
Q8. What caseload are you working in? Please be as specific as possible and include your current diagnostic groups. Example response: ‘rehabilitation ward, primarily working with stroke patients.’	Text box
Q9. How often do you use the Brisbane Evidence-Based Language Test (EBLT) in a typical week?	Text box
Q10. How do you find the overall scoring of the EBLT? Is it specific enough for your needs? If yes, why? Or If not, why not?	Text box
Q11. How frequently do you use the verbal fluency component of the EBLT in a typical week?	Text box
Q12. What are your current thoughts on scoring the verbal fluency component of the EBLT? Is it specific enough for your needs? If yes, why? If not, why not?	Text box
Q13. Please provide your email below. In doing so, you will have access to the new cut-off scores and it will allow us to send the weekly and final survey links to you. Your email will remain confidential and will not be shared or distributed.	Text box
(Message provided following an input of an email address) Please keep the below cut-off scores confidential until after publication. Click on the link below to download the cut-off scores. For further information or inquiries, please contact (Author de-identified for review) via email: (De-identified for review).	–

Appendix 2. Weekly survey questions and response options

Table

Questions	Response option/s
(Initial information message) This survey will give you an opportunity to share your perspectives and use of the new EBLT severity guidelines and subtest cut-off scores during the past week. Thank you for your participation, (Authors de-identified for review)	–
Q1. Please enter your unique 3-digit code in the textbox below. Reminder: The last number of your birth year, the initial of your first name and the first letter of your birth month. (For example: 1AJ).	Text box
Q2. How many times did you use the EBLT during the week?	Text box
Q3. What populations did you use the EBLT on during the week?	Text box
Q4. How many times did you use the verbal fluency component of the EBLT during the week?	Text box
Q5. What populations did you use the verbal fluency component of the EBLT on during the week?	Text box
Q6. How are you finding the new verbal fluency cut-off scores?	Text box
Q7. Are the new verbal fluency cut-offs in line with your clinical impression and presentation of the clients (e.g., age, presence/severity of language deficits)?	Likert scale: Strongly agree/Somewhat agree/Neither agree nor disagree/Somewhat disagree/Strongly disagree
Q8. Are the new verbal fluency cut-offs meeting your needs as a clinician? Please explain your choice in the text boxes provided.	Multiple-choice options with text boxes: Yes/No/Unsure
Q9. So far, do you have any concerns with the new verbal fluency cut-offs? Please outline them here (being as specific as possible). Please also include any suggestions you have for resolving these concerns.	Text box
Q10. Would you make any changes to the new verbal fluency cut-offs based on the findings from the previous week? Please outline the changes here.	Text box
Q11. How are you finding the new scoring guidelines to indicate severity using the EBLT?	Text box
Q12. Are the new severity scoring guidelines in line with your clinical impression and presentation of the clients?	Likert scale: Strongly agree/Somewhat agree/Neither agree nor disagree/Somewhat disagree/Strongly disagree
Q13. Are the new severity scoring guidelines meeting your needs as a clinician? Please explain your choice in the text boxes provided.	Multiple-choice options with text boxes: Yes/No/Unsure
Q14. So far, do you have any concerns with the new severity scoring guidelines? Please outline them here (being as specific as possible). Please also include any suggestions you have for resolving these concerns.	Text box
Q15. Would you make any changes to the new severity scoring guidelines based on the findings from the previous week? Please outline the changes here.	Text box

Appendix 3. Final survey questions and response options

Table

Questions	Response option/s
(Initial information message) This is the final survey where you have the opportunity to share your perspectives of the new EBLT severity guidelines and subtest cut-off scores. Thank you for your participation, (Authors de-identified for review)	–
Q1. Please enter your unique 3-digit code in the textbox below. Reminder: The last number of your birth year, the initial of your first name and the first letter of your birth month. (For example: 1AJ).	Text box
Q2. What is your final perception of the new verbal fluency cut-off scores?	Text box
Q3. Do you think you would use the new verbal fluency cut-offs routinely in your clinical practice? Please explain your choice in the text boxes provided.	Multiple-choice options with text boxes: Yes/No/Unsure
Q4. Do the new verbal fluency cut-offs help inform your clinical decision-making? Please explain your choice in the text boxes provided.	Multiple-choice options with text boxes: Yes/No/Unsure
Q5. Would you encourage other speech pathologists to use these verbal fluency cut-offs in their clinical practice? Please explain your choice in the text boxes provided.	Multiple-choice options with text boxes: Yes/No/Unsure
Q6. Any other feedback on the new verbal fluency cut-offs?	Text box
Q7. What is your final perception of the new severity scoring guidelines?	Text box
Q8. Do you think you would use the new severity scoring guidelines routinely in your clinical practice? Please explain your choice in the text boxes provided.	Multiple-choice options with text boxes: Yes/No/Unsure
Q9. Do the new severity scoring guidelines help inform your clinical decision-making? Please explain your choice in the text boxes provided.	Multiple-choice options with text boxes: Yes/No/Unsure
Q10. Would you encourage other speech pathologists to use these new severity scoring guidelines in their clinical practice? Please explain your choice in the text boxes provided.	Multiple-choice options with text boxes: Yes/No/Unsure
Q11. Any other feedback on the new severity scoring guidelines?	Text box

Appendix 4. Weekly surveys: number of times the verbal fluency subtest was used in the past week for each week

Table

Number of times used	Number of participants
	Week 1	Week 2	Week 3	Week 4	Week 5	Week 6
0	3	4	1	3	3	1
1	3	2	2	–	–	1
2	2	–	1	–	–	–
3	1	2	1	2	–	–
4	–	–	–	–	–	–
5	1	–	–	–	–	–

Appendix 5. Content analysis of qualitative questions from weekly surveys with categories, subcategories and quotes

Table

Categories	Usefulness of the developed cut-offs		Perceptions of concerns and changes		Constructive feedback
Sub-categories	Cut-offs are helpful	Cut-offs are appropriate and useful in practice	No concerns	No changes	Suggestions for improvement	Cut-offs are too high
Quotes	P1: ‘Very helpful in clearly documenting severity of aphasia associated with stroke’ P4 and P16: ‘Very helpful’ P4: ‘Helpful’	P17: ‘They seemed appropriate. I liked that they got specific into mild, moderate, severe.’ P1: ‘Useful for diagnosis and clear documentation’ P17: ‘Appropriate, accurate’ P17: ‘Appropriate’ P21: ‘Good descriptors/guide for documentation’	P4: ‘No specific concerns’ P7: ‘N/A’ P4, P16, P17: ‘No’ P1 and P2: ‘Nil’	P1, P4, P16, P17: ‘No’ P1: ‘Nil’	P1: ‘It would be helpful to be able to separate the severity scores out (Mild, Mod, Severe) however I can see this would be difficult given the small data range. It would also be helpful to have more information about the “at risk” category,’	P11: ‘I feel they are a bit high’

Appendix 6. Analysis of quantitative data for question 7 and 8 for the weekly surveys

Table

Questions	Scale responses	Week 1	Week 2	Week 3	Week 4	Week 5	Week 6
		Percentages and participant numbers
Q7. Are the new verbal fluency cut-offs in line with your clinical impression and presentation of the clients (e.g., age, presence/severity of language deficits)?	Strongly agree	33.3% (n = 3) P1, P16, P17	42.9% (n = 3) P1, P16, P17	75% (n = 3) P1, P16, P17	50% (n = 1) P17	–	100% (n = 1) P17
	Somewhat agree	11.1% (n = 1) P4	28.6% (n = 2) P4, P11	25% (n = 1) P2	50% (n = 1) P21	–	–
	Neither agree nor disagree	55.6% (n = 5) P7, P9, P10, P19, P23	28.6% (n = 2) P19, P23	–	–	100% (n = 1) P17	–
Q8. Are the new verbal fluency cut-offs meeting your needs as a clinician?	Yes	50% (n = 4) P1, P4, P16, P17	62.5% (n = 5) P1, P2, P4, P11, P17	100% (n = 5) P1, P2, P16, P17, P21	50% (n = 1) P17	100% (n = 1) P17	100% (n = 1) P17
	Unsure	50% (n = 4) P7, P10, P19, P23	37.5% (n = 3) P16, P19, P23	–	50% (n = 1) P21	–	–
Participants who did not implement the cut-offs during the week	–	(n = 3) P7, P9, P10, P19, P23	(n = 5) P2, P4, P16, P19, P23	(n = 2) P2, P16	(n = 3) P1, P4, P16	(n = 3) P4, P17, P21	(n = 1) P16