“The sun’ll come out, tomorrow, betcha bottom dollar…”

“The sun’ll come out, tomorrow”

Whenever people end a Facebook post – or a tweet – with “do your own research,” there’s a chance that they’ve done no such thing themselves. Or to be more specific, they’re unlikely to have gone through a set of procedures that are what folks who actually do research for a living would go through. There are hundreds of books out there that explain, in excruciating detail, what “research” is but I’d bet money on the probability that the number of folks who have read ONE book on the research process is significantly lower than the number of people who believe the world is secretly governed by lizard people [1], that aliens regularly visit the earth [2], and that if there is no evidence for X, that in itself is proof that X exists [3].

Research is essentially a tool used by scientists to test the veracity of hypotheses. Or in words of one syllable, a way to find out if what you think is true is, in fact, true. But let’s go down the rabbit hole a little more here and try to avoid the words “true” and “truth” because like “commonsense,” so many people claim to know what it is but often turn out to be as subjective in their thinking as everyone else. For many people, “truth” is simply what they believe to be true and may have zero bearing on any actual, observable, demonstrable, measurable quality. The majority of people reading this will believe in some supernatural deity (or deities) with absolute conviction that this is “true,” yet given the opportunity to provide any supporting evidence that is objective and replicable, they will fail stunningly. Faith is not Truth, and Truth is not Belief. Anyone can have faith in something or someone, and anyone can believe in any ridiculous things they like. As I’ve said for years, the First Amendment of the US constitution pretty much guarantees your right to be a complete idiot, and it’s a testament to its power that so many people are making the most of it.

Deciding that something is “true” turns out to be a devilishly difficult and intellectually irritating thing. Even when Little Orphan Annie sings, “The sun’ll come out, tomorrow, betcha bottom dollar that tomorrow, they’ll be sun,” the scientist in you should be thinking, “Oh yeah, how do you know that’s true?” Annie is definitely on my list of “irritating imps” from literature, with the number one spot reserved for Dickens’ Tiny Tim, but she can’t be blamed for at least attempting to make a statement that has all the appearance of being unassailably truthful.

To be more accurate, we’d first have to make sure we all shared the same definitions of the hypotheses she’s espousing; that “The sun will come out tomorrow.” In relation to “the sun,” it would be churlish to suggest that someone might interpret what she’s referring to as Canis Major or Corona Borealis but if we want to be as accurate as we can, we’d make explicit that “the sun” is the star closest to Earth some 93 million miles away. Then, by “come out,” we mean that it will take a position in the sky above the horizon, regardless of whether it is obscured by clouds, smoke, overhanging branches, or a roof. That way is someone tries argue that, “I was in my cellar and didn’t see the sun so it didn’t ‘come out'” you can happily sneer and smack them on the head with a large dead cod and remind them of the definition. This also infers that we must have some way of measuring the act of “coming out” that is objective, which means it doesn’t really on belief, opinions, points-of-view, or any other internal mental phenomena), and repeatable by others, which means anyone else could do the same measurement. Finally, “tomorrow” also needs to be defined as being a point 24 hours after an initial point. If it’s 48 hours before it appears again, then that’s not tomorrow but “every two days.”

But it doesn’t stop here, does it? The location in which I take the measure may also affect the truth of the statement. It may work for Annie who’s in New York city but what about Pippi Longstocking on her visit to grandma’s house in Svalbard, Norway. Being inside the Arctic circle means she’s singing, “The sun’ll come out, in April, betcha bottom Krone that in April, they’ll be sun.” Clearly there is still a sense that “the sun will come up” at some point but it’s not as clean cut as it appears in the Tomorrow song.

Now, given that we’re happy with defining our terms and restrict our experiments to places closer to the equator, what happens when we find that we see the sun rise on Jan 1 at 7:20 AM, then again on Jan 2 at 7:21 AM. Can we say, “Yes, it’s true, the sun’ll come out tomorrow?” You could try and maybe many people would believe you. But the problem is that this is based on ONE sample and it’s a very poor piece of research that takes one sample and concludes it’s the truth. That’s like watching one video on YouTube of someone checking ballot papers and then concluding that the entire election was a scam (unless it was a vote for your candidate, in which case it was true!) No, you’d want to take another sample 24 hours later on Jan 3. Then Jan 4. Then Jan 5. And keep doing that until… well, when?

This is another feature of research that people can misunderstand; that there is an “end” to research [4]. Their understanding is that “research” and “scientific analysis” provide truth and “an answer” when, in reality, that’s not quite right! Back with the rising sun, if you were to say it’s true that the sun will come up tomorrow, what you’re really saying is “given that we’ve observed a sunrise since records began, and that the laws of physics which apply to the universe and how planets revolve around suns have not changed, there is probability close to one that the sun will come up tomorrow.” Is there a chance that the sun may NOT come up? Could there be an unforeseen cosmic accident (black hole hitting the sun) that may wipe out sun and the Earth during the night? Well, maybe. And that is where all research ends up.

Research is about creating hypotheses – ideas that may or may not be true – and then dreaming up ways to test them. You determine in advance what measurements you consider will support the hypotheses and after testing, see if those measures were achieved. If they were, then you can say, “yeah, the objective measure support my hypothesis and it’s therefore worth considering as ‘useful’ or ‘important.'” The more measures you take and the more support you find, then the more meaningful the results. Better still, if lots of other people do the same measures and get the same results, then your hypothesis becomes tougher, stronger, and develops a bit of a cocky swagger.

But although thousands upon thousands of repeated experiments can support a hypothesis, it only takes ONE discrepancy to shatter it into tiny pieces. This notion is sometimes called the “Black Swan” approach. If you set up the theory that “All swans are white” you can spend years out in the world looking at swans and see that every swan you comes across is white. You can then have hundreds of other bird watchers report back that every swan they’ve seen is white. But as soon as ONE person person provides evidence that they’ve found a black swan, the “All swans are white” is now wrong; it’s downgraded at best to “Not all swans are white” or “Most swan are white” or “All swans in the world, except for in the village of Little Turdington that are black, are white.”

Of course, this also introduces the issue of determining what constitutes valid data. The ornithologist from Little Turdington may have taken a photo of a white swan but, armed with a copy of PaintShop Pro 2021, changed the color to black in an attempt to discredit the original hypothesis. Real researchers wouldn’t take that one photo and accept it as proof but more likely send out lots of other researchers to LIttle Turdington to see if they, too, can observe and measure black swans. As more evidence of black swans appears, then the truth, or veracity, of the original claim is diminished.

Science progresses step by step by step. One hypothesis leads to another and depends on what has gone before. In the development of a vaccine for COVID-19, one of the claims by vaccine-skeptics is that it was developed too quickly and not tested enough, which completely ignores that fact that work on similar vaccines has been going on for years, and that all the work that has gone before always sets the scene for what comes next. Sir Isaac Newton, whose “Theory of Gravity” is still a theory [5], is credited with saying that, “If I have seen further, it is by standing upon the shoulders of giants,” and that’s pretty much what research is all about.

And if you don’t believe me, do your own research 🙂

Notes

[1] According to Wikipedia, the increasingly deranged ex-sports commentator, David Icke in the UK, believes that “tall, blood-drinking, shape-shifting reptilian humanoids from the Alpha Draconis star system, now hiding in underground bases, are the force behind a worldwide conspiracy against humanity.” He is, sadly, not alone. Proof, should you need it, includes videos of famous people (the Queen, Hillary Clinton, the Pope) all blinking like lizards, so it must be true.

[2] Given that (a) reports of aliens from outer space visiting earth have been around for as long as folks have been writing and (b) not one has yet gone through the “take me to your leader” process, it makes me wonder just how “alien” their thinking must be. Imagine investing unimaginable amounts of time, money, and energy into building interstellar spaceships that can carry you anywhere in the universe, traveling millions and millions of miles -perhaps over immense periods of time – and then landing in some backwoods to pick up a random person so you can stick an anal probe up their arse before flying back home. Now that IS some alien psychology!

[3] The best conspiracy theories are the ones that evolve in such a way that they become completely impervious to any evidence against them, and where all and any statements are “proof” of the theory. Any “proof” of election fraud in the US 2020 election is seen by conspiracy theorists as “clear evidence” for the stolen election; any absence of evidence of fraud is “proof” that the fraud was effective and therefore also evidence for the stolen election! According to one of my favorite philosophers of science, Karl Popper, this type of thinking is referred to a “metaphysical” because it is incapable – by design – of being tested by any objective measurement and is, ergo, non-scientific. Popper was no fan of Psychoanalysis and always deemed it a metaphysical philosophy rather than a science. Religion, as a phenomenon, is also metaphysical in nature in that there are no measurements we can do to test for the existence of a supernatural entity, or entities (it seems unfair to throw Odin, Thor, and Loki into the “myth basket” while allowing Jehovah and Allah special status as “real”).

[4] All research article end with “much more research is needed,” which is (a) probably true and (b) setting the scene for the next grant application. If anyone can cite a paper that concludes with “and, ergo, there is no need to look at this topic anymore because we now have all the answers” I’d love a copy.

[5] Just because something is labeled a “theory” doesn’t imply that is is somehow wrong. Both “gravity” and “evolution” are spoken of as “theories” but if you chose to believe gravity isn’t “real,” I suggest throwing yourself off a high building and seeing how well your disbelief stops you from hitting the ground at 150 miles per hour and leaving you intact enough to stand up and say, “See, I told you it was just a theory.” In this case, you’d be well advised to treat the “theory of gravity” as a practical reality.

Countdown to Christmas Quiz: Question 9 – Monday 9th December

What is the three-letter acronym we use to describe the integration of:

(a) clinical expertise/expert opinion,
(b) external scientific evidence, and
(c) client/patient/caregiver perspectives to provide high-quality services reflecting the interests, values, needs, and choices of the individuals we serve?

ANSWER: EBP – Evidence-Based Practice

Sometimes, evidence-based practice seems – and sounds – complicated, especially when you’re trying to make sense of statistical data presented in peer-reviewed articles. However, the basic principles are simple and all you need to ask yourself is one question: “Is what I am doing with my client based on the best information available to me?” If the answer is “yes,” then you’re on the right track; if the answer is “no,” then your next question is “Where do I get the best information about what I am doing?”

Links

Position paper on EBP in Communication Disorders from ASHA’s Joint Committee on Evidence-based Practice.

The Handbook for Evidence-Based Practice in Communication Disorders by Christine Dollaghan published by Brookes Publishing.

First Baby Step to Thinking of Evidence-Based Practice: Be Skeptical

At the recent 2012 conference of the International Society for AAC (ISAAC) there was some robust discussion about the technique know as facilitated communication. It’s a controversial technique and surprisingly one on which ISAAC does not have a position paper – which is an endeavor currently underway with a view to something being published soon. I say “surprisingly” because many other professional organizations have had position papers for many years, from the American Academy of Child and Adolescent Psychiatry (1993) through to the Victorian Advocacy League for Individuals with Disability [1]. ASHA has had a statement since 1994, so it does seem a little tardy for the group whose raison d’être is AAC to be publishing a statement on an AAC technique. But never mind, at least there is action being taken, which is better than continuing to say nothing.

But this isn’t about the pros and cons of FC. It’s about the development of a mindset that allows people to think about FC – and Non-Oral Motor Speech Exercises, Equine Therapy, Canine Therapy, Sensory Integration, and other such debatable practices. The reason I started with the reference to FC was simply because during the discussion, one person actually said, “But there’s more to this than Science.”

Is there? Is there really? I can appreciate that things in the world can be difficult to measure, and that there are times when measurement seems unfeasible and even intractable, but that doesn’t mean we stop trying.

Evidence-based practice can be tough. When you get into the nitty-gritty of the scientific method – which is a big chunk of what EBP is about – it’s easy to get overwhelmed by talk of variables, pre-tests, post-tests, levels of confidence, skewed distributions, ANOVA, one- versus two-tailed hypothesis, Bayesian, Cartesian, and the whole catastrophe that is experimental design. Even the most readable of books, such as the excellent The Handbook for Evidence-Based Practice in Communication Disorders by Christine Dollaghan [2], can be hard to read and even more challenging to digest. The potential complexity of designing ways to measure clinical practice is, to put it bluntly, off-putting. When you have a caseload of 200 clients and only 24 hours in a day, the idea of setting up formal measurement procedures is about as welcome as a bacon sandwich at a Bar Mitzvah.

Nil desperandum! Like any other skill in life, becoming a more effective practitioner of EBP doesn’t require you to be an expert all at once. You can improve your practice simply by sharpening your mindset to be more in tune with the concepts of EBP. And the first thing you can learn to do is become a Skeptic.

First, let me shovel out of the way that huge mound of steaming objection that being a skeptic is just an excuse for rejecting everything and believing in nothing. That’s a cynic, or a nihilist. In a 2010 interview with Skeptically Thinking, philosopher and author Massimo Pigliucci [3] said;

I think that a crucial aspect of being skeptical, of engaging in critical thinking, is not the idea that you reject claims because they seem absurd. That’s not being a skeptic, that’s just being a cynic. It’s just denying things for the sake of denying it. The idea of skepticism is that you inquire — that you do the work.

“Doing the work” is obviously a tough one because in our world of Wikipedia and endless cable shows about ghost hunters, psychics, celebrity hauntings, and quick-fix psychology, it’s easy to let someone else do the work for you – and that work may be of stunningly poor quality and accuracy. However, a little “critical thinking” is not that hard.

So here are my Top Three Critical Questions to help you become a baby Skeptic. And feel free to be skeptical about whether my three are a good three!

1. If someone claims X causes Y because they did Z, can the claim be tested independently? If I tell you that I can stop an interdental lisp by pushing the tip of a client’s tongue with a wooden spoon, while simultaneously saying “go back, tongue, go back,” you’d be right to ask if anyone else can do it, and you may even try it yourself. But if I claim that the reason no-one else can do it is because they don’t have the same spoon, or that my intonation pattern is very specific, you’d also be right to call bullshit on me.

2. If someone claims X causes Y because they did Z, are there any other simpler explanations as to why Y may have happened? When TV ghost hunters use a drop in temperature to “prove” the presence of a ghost, could something simpler have caused it? When a child appears to speak more after an hour with a dolphin, was it actually the dolphin’s presence causing it or just that the kids was happy?

3. If someone claims X causes Y because they did Z, what change was actually measured and how? “My kid talks more to my therapy dog, so therapy dogs work.” More than what? More than if there was a cat? More than 6 months ago? More than when he walked in the door? I had a client many years ago who swore blind that his stammer was much better after a few pints of beer and he wondered if he could get a prescription! Although I never took the opportunity to spend a night out at the bar with him, his measure of “better” was that he felt he was more fluent. But after a few pints of ale, I’m not sure my client was particularly accurate in his measurement techniques.

Oddly enough, I’m not going to suggest you use your common sense because this can be less “common” and “sensible” than you might believe. A recent book by Duncan Watts takes the notion of common sense to task. In Everything is Obvious: How Common Sense Fails Us, he argues that;

Common sense is “common” only to the extent that two people share sufficiently similar social and cultural experiences. Common sense, in other words, depends on what the sociologist Harry Collins calls collective tacit knowledge, meaning that it is encoded in the social norms, customs, and practices of the world.

Anyone who feels that common sense is in some sense the truth may want to spend at least 30 minutes listening to the discussions that go on in your country’s government, with folks in the US now facing 2 months of pre-election “common sense” being thrust down their throats. If sense were really that common, all parties in the political divides would cease to exist because their would only be one truth.

So common sense is less helpful in making evidence-based judgements than the basic science of testing and measuring. Even minimal measurement is better than no measurement because it gets you ever closer to an improved metric. You don’t have to subscribe to the “all or nothing” fallacy that some folks promote. Remember that there are different levels of measurement you can use, and each one has its pros and cons.

So let’s invent an example based on Dolphin Therapy. I can ask my client to tell me as much as possible about a picture of a busy street and record what is said, then repeat the task 5 minutes after spending a half-hour with a dolphin. If I simple count the number of words before and after the swim, then find the post-dolphin condition has twice as many words, is that a “good” measure? Well, the safest answers is “it’s a measure” but the notion of “goodness” is more complex. But here’s the valuable thing; you’ve at least created for yourself a methodology that you can use with the rest of your swimming clients. You can also do it again next time you client has another dolphin session. And the next.

Of course, don’t be surprised if someone else comes along and pokes holes in your methodology and results. The good news is you actually have some results to talk about, rather than a blanket statement about how “good for the kids” this dolphin fun is. Nor should you be surprised if someone uses the second question in my list to suggest an alternative explanation such as “the kid was just relaxed and would have done just as well if you’d given him a massage, or a bowl of ice-cream, or a flight in a helicopter.” This will help you go back and think of a better way to measure and test (or try to get a grant for “Helicopter Therapy” sponsored by folks who like flying in helicopters!) [3]

Enough for now. Once an article passes the 1500-word mark, it ceases to qualify as “baby steps.” So take those three critical questions and start trying them out. If you want some homework, try them out while watching a TV show about UFO’s or Bigfoot – it’s kinda fun.

Notes
[1] No, the “Victorian League” is not a group of steam-punk enthusiasts who yearn for a return to the values of the 19th century but an organization (VALID) based in the Australian state of Victoria, the capital of which is Melbourne.

[2] Dollaghan, C. A. (2007). A Handook of Evidence-Based Practice for Communication Disorders. Baltimore: Paul H. Brookes Publishing. This is great book and if you wanted to buy just one reference for EBP, I’d go for thisl But be warned; it is so full of excellent one-liners and summaries that if you use a yellow highlighter, there’s a fair chance you’ll end up with a banana-colored book. I use sticky tags and I think I went though three packs of them! And if you don’t want to spend the money – and time – on the book, you can read Christine’s 2004 ASHA Leader article entitled Evidence-Based Practice: Myths and Realities.

[3] Often the people promoting the benefits of animal therapy are animal lovers who appear to want to somehow “prove” that there’s something special about their dog/cat/dolphin/horse/lizard/three-toed sloth/whippet etc. I have no doubt that research shows how stroking a cat can reduce your blood pressure temporarily, but I can get the same effect from drinking beer, riding my motorcycle, or having sex. However, unlike the animal therapy folks, I am not promoting Drunken Biker Orgy therapy, or DBO as it would be referred to in the academic literature. Which may turn out to be a spectacular loss of revenue for me as a future project…

The Dudes Do ATIA Orlando 2012: Day 3. Of Data and Describbling

The great thing about starting a day on a bad note is that usually, it can only get better. This morning’s “bad note” was to make a pot of coffee using the water from the faucet in the hotel room. Big mistake. Huge. I’m pretty convinced that all hotels have special filters installed that turn perfectly acceptable water into something that tastes as if it’s just dripped off an industrial sludge compactor, and to which is added a selection of chemical cleaners that are banned by the United Nations. Call me cynical, but the fact that they also strategically place bottles of water right next to the faucet  and charge more than a cost of gas. Seriously, $5.00 for 1/2 gallon of water. This also explains why the “water-powered car” has yet to be invented – it’d be too expensive to run!

Must... try... to... swallow...

After at least two mouthfuls of the warm, brown liquid that was clinging hopefully to the side of my cup, I decided to use it to descale the sink and poured it down the drain. There! Better now.

The day did, indeed, improve because I’d managed to schedule more sessions than meetings, so I had the pleasure of sitting back to listen to presenters and describbling notes in my notebook.

Describble is a real word, even though my spell checker disagrees. Honestly. It’s in the Oxford English Dictionary, and that, for me, is a working definition of “real” as far as words are concerned. [1] It is, however, not a particularly popular or well-used item. In fact, it is credited as being a nonce word. And for those for whom nonce is new, it means “a word apparently used only ‘for the nonce’, i.e. on one specific occasion or in one specific text or writer’s works.” [2]

Geeky as I may sometimes seem to be (although technically I’m more of a nerd than a geek) I still find that my favorite piece of technology for a conference is a pen and a notebook. I can still write faster than I can type or swype, and there’s also something vaguely satisfying about seeing brown ink dry quickly on cream paper, watching the light momentarily flash off of the wet blaze as you describble quickly across the parchment.

Low-tech "tablet" or "notebook"

And I describbled a lot at the presentation by Deborah Witkowski entitled Data Collection in AAC: Gathering Performance and Outcomes Evidence. Debbie talked about the difference between standardized evaluations and profiles/inventories, and how it makes sense to be eclectic in your choices. In AAC, she stressed that “measurement” is not just counting words but looking at access, vocabulary, syntax, semantics, and pragmatics. She also encouraged people to do two basic things before implementing any therapeutic intervention; (a) take a baseline measure and (b) determine you measure of success. This may seem obvious but once of the current issues with the iApp approach to AAC is that there is little to no data being collected. The “baseline” is typically “we downloaded this app” and the measure of success is “Bob is now communicating” but with NO specification as to what “communicating” is; and I have seen YouTube videos where folks push the iPad against the child’s hand and say “Look, he said it!”[3]

Debbie provided her slides as a PDF so we have them available for you at the Speech Dudes Box account:

Witokowski 2012 Data Collection in AAC

Today was an 8:00 to 5:00 day, which is not different from a normal day, but somehow at a conference, by the end of several sessions, you feel a little drained. Sitting down and listening seems like an easy thing to do but oddly it uses energy. Well, something must be using up energy because by the time I was back at my room around 5:45 p.m. I was pretty much ready for a nap. Or maybe I’m just old.

…but not too old. By 7:00 p.m. I was revitalized and ready to eat, although not ready to take a long trip somewhere. So, we went to the hotel’s Tropicale restaurant for a delicious Shrimp Bisque, followed by a Seafood Pasta, generously loaded with chunks of salmon, swordfish, scallops, and crab claws, all washed down with ice-cold, lime stuffed Coronas. On the way back, we called in at our friends’ villa and spent the rest to the evening chatting, laughing, and helping to clear the refrigerator of beer.

I’d forgotten all about the morning coffee.

Notes
[1] I’m also one of those who recommends the Urban Dictionary as a source for current slang and entertainment, but a word’s presence in the UD doesn’t classify it yet as a “real” word for me – more of a “hopeful monster” ready for being tested against lexical natural selection.

[2] It’s also been used more recently (since the 1970’s) to mean a sexual pervert, particularly a child molester. It may derive from the earlier nance or nancy, a word to describe a homosexual male, which in turn comes from 19th century slang for buttocks. Once again, the Dudes have access to facts you didn’t even know you wanted to know!

[3] There’s a whole minefield of issues out there with regard to “over-the-counter” AAC, where parents skip any evaluation of their child and use YouTube, iTunes, and “the nice man at Best Buy” to decide what app is best. One step toward improving the situation is to determine measures of performance, preferably built into apps. Of course, many app “designers” won’t want this because measurement not only shows success but also failure. The importance attached to success makes us loath to see failure as a critical measure; and knowing something is NOT working is as valuable as knowing it is.

The Dudes Do ASHA 2011: Day 4. On Bondage, Booze, and Bayesian Statistics

It’s a cheap trick I know, but the “bondage” referred to in the title refers to the binding of a laptop with a smartphone, otherwise known as tethering. Casual visitors who have landed here via a Google search hoping for some images of people tied up and engaging in fetishistic practices should leave now because there are none. Tethering is a little trick you can use if you find yourself without an easily accessible WiFi connection – such as at the San Diego Convention Center. Interestingly, the brochure in my hotel room touts the city as being one of the most “wired” places on the world but it conveniently leaves out the phrase “at a price.” There is a WiFi connection to be had inside the Convention Center but it is a fee-for-use option that, as always, seems steep for 10 minutes, which is about the time I needed to upload the latest blog posts.

Computer bondage

However, with a cable and a piece of free software, it’s possible to tether a computer to a smartphone and use the phone’s Internet connectivity as a router. Specifically, I’m using a Dell Latitude, a Droid 3, a USB cable to connect the two, and a piece of software called PdaNet – one version running on the phone and its partner-in-crime running on the laptop. [1]

I say “in crime” because the phone company don’t really want you to do this. What they want is for you to pay an extra $20 per month on top of the $30 per month you are currently using in order to download through your phone. This strikes me as odd because whether I download data to my phone or my laptop via the phone, it’s the same data use. And seeing as I also have an unlimited data plan, what’s the difference? So, the outcome of this is that tethering is frowned upon and can, if the phone company finds out, can lead to a stiff warning followed by being kicked off the system.

In truth, I never tether if I can possibly avoid it, simply because it’s actually much, much easier (and more stable) to use WiFi or an ethernet connection. I reserve tethering to emergency situations. Such as the SDCC.

The tethering took place immediately following my last ASHA session for 2011; a short course entitled Evidence-based Statistics for Measuring Strength of Evidence, a title that sounds benign enough but turned out to be a wolf in sheep’s clothing. The presenters, David Maxwell and Eiki Satake, are both Professors at Emerson College and clearly know their stuff.

EBP Pyramid

I became an SLP partly because the part of the brain that deals with numbers is clearly non-functioning in my case. I know enough math to keep me away from the casinos [2] and overpaying at the bar, but when equations and formulae come onto the scene, I suffer from math blindness. Which is why this particular course had me cranking up my brain volume to eleven in order to follow some of the arguments.

And that’s where the Bayesian [3] statistics came in and I learned the difference between a Bayesian  and a Frequentist approach to statistical analysis. Let me try to summarize what the presenters were suggesting and how it relates to EBP. Essentially, they say that;

(a) Current research practice in Speech and Language Pathology uses a Frequentist model that relies on creating a null hypothesis and testing it against a P-value.

(b) This model inherent works on a “pass” or “fail” basis but tells you little about the strength or power of a treatment.

(c) Such p-values are not the best way to deal with clinical outcomes.

(d) A Bayesian model uses a modified P-value – the “precise P” as opposed to the “imprecise P” – which is more meaningful for clinical data.

(e) Using a Bayesian precise P-values tells us more about the strength of experimental observations and is, therefore better for evidence-based practice.

(f) There is a difference between statistical significance and clinical significance, and the latter is better evaluated using a Bayesian method.

This summary is probably hugely unfair because condensing a 3-hour short course into five sentences is hardly “best practice,” and there was a 150+ slides handout to accompany it, and that’s tough to condense! Nevertheless, if there’s a single take-away idea, it’s that we should be considering Bayesian methods for EBP. [4]

Fortunately, there is a book available that explains Bayesian statistics as applied to EBP; The Handbook of Statistical Methods: Single Subject Design (2008) [5]. I have the sneaky feeling that this is unlikely to be light, bedtime reading but I’ll see if I can track a copy down in the library before splashing out on a copy for the bookshelf.

Satake et al. 2008

Which brings us to the booze and perhaps the second most useful piece of information to come out of the short course; beer and statistics do mix!

At some point in their training, SLP’s will have come across the “Student’s t-test,” which is not, as it suggests, a “t-test” for a “Student” to use but one originated by someone called “Student.” It turns out that “Student” was, in fact, the pseudonym of William Gosset, a statistician who worked on a project with folks at the Guinness corporation to test the quality of the beer. The t-test provided a way for beer testers to take a few small samples, measure them, and then use the test to get a quality score. Now there’s a job! Unfortunately for Gosset, as a Guinness employee, he couldn’t publish what was proprietary information. Nor did he want to face any ribbing from his colleagues because he was working for a private company – a beer maker at that. However, he was eventually allowed to write papers but under the nickname of “Student,” hence the ultimate name of “Student’s t-test.”

Guinness

At this point, I recommend you’ve had enough statistics for the day and head for a local bar to do your own research into how effective the t-test is at quality control. The sample size is up to you.

Notes
[1] Setting up your phone and laptop for tethering is not rocket science and Google is your friend for using the PdaNet 3.0 software. As I said, it’s not illegal but if the Cell Network Mafia come a-knocking, I take no responsibility if you’re found at the bottom of a river with concrete overshoes. If you’re using a Mac and an iPhone, I can’t help you, but undoubtedly “there’s an app for that” and it’s so intuitive you can give it to a 6-month-old to set it up for you.

[2] Gambling (or as the industry prefers to call it in a miserable effort to make it sound more friendly, gaming) is an example, par excellence, of applied probability. All you really need to know in order to avoid living your life under bridges and drinking rubbing alcohol hidden inside a brown bag are two things; (a) the gambling industry only exists because the odds are always in their favor, and (b) your odds of becoming extremely wealthy by gambling are so small as to be, in most cases, negligible to zero. Casinos always tell you about the big winners but don’t provide you with a list of losers. By all means go out with $100 in your pocket for a night of fun, at the end of which you expect to have lost it all, but when you start thinking, “OK, maybe just another hundred and then I’ll call it quits,” start practicing how to make the sentence “Honey, I lost the house” acceptable.

[3] It’s pronounced /’bəɪziˌjʌn /. I add this because I was saying it wrong for years!

[4] I found a short summary of Satake’s critique of Frequentist statistics in an article he wrote in 2010. Entitled Moving Forward to Evidence-Based Statistics: What Really Prevents Us?, he explains why the traditional P-value is misleading and why. http://www.pluralpublishing.com/web_flyer/web_flyer_community_august/web_flyer_community_august.htm 

[5] Satake, E., Jagaroo, V., and Maxwell, D.L. (2008). Handbook of Statistical Methods: Single Subject Design. San Diego, CA: Plural Publishing.

The Dudes Do ASHA 2011: Day 1. On Traveling, Truth, and EBP

journey, (n). / /. A day’s travel; the distance traveled in a day or a specified number of days.

According to the English writer Oliver Goldsmith, “Life is a journey that must be traveled no matter how bad the roads and accommodations.” Well, in all honesty, when the journey is to the ASHA 2011 conference in San Diego and the accommodations are at the Hilton San Diego Bayfront, it’s hard to imagine anyone feeling any sympathy for our having to travel for hours and across thousands of miles to get here. When you’re sitting back in a swanky, high-priced bar drinking White Russians and nibbling on a $26 selection of gourmet meatballs, you keep glancing at the hotel entrance to see if the “Occupy San Diego” people are about to storm the building with flaming torches, dragging off “the rich” to a hastily constructed gibbet.

The Odysea Bar

Still, it’s not as if this is the regular lifestyle for the Speech Dudes. No sir! For every luxury hotel we stay in there are tales we can tell of others that involve insect infestations, bullet-proof check-in desks, ear-splitting air-conditioners, no air-conditioners (in a Coloradan summer), and one just on the edge of a European red-light district that rented by the hour. So rest assured the Dudes are immensely appreciative of their current locations, split as they are between the Hilton and the Marriott, also with a view of the bay.

Despite having traveled individually from the East Coast, Mid-West, and Canada, we are, after all, Speechies, and within minutes of ordering food and drinks at 10:30 pm PST, the conversation  shifted from how which is the worst airport in the US (Philadelphia  is currently in the running, says one of us) to Speech Pathology, specifically the issue of outcomes. Incredible as it may seem that a group of guys at a bar would want to get into Evidence-based Practice rather than the sad state of the NBA and the prospect of not being able to spend some quality time over Christmas secretly hoping LeBron James will twist an ankle and bruise his ego, we ploughed into the current fad for dropping iPads [2] on every kid with a pair of hands as the “miracle cure” from St. Stephen of Jobs and the Angels at Apple.

The messianic zeal of evangelical Jobbites [3] is such that the answer to life, the Universe, and everything, is quite simply “there’s an App for that.” And if there isn’t, then someone will make one and all will be well. It’s no use talking about “levels of evidence” or “controlled experiments” or even “proof” because there is, of course, lots of “proof” on YouTube, and, of course, sales of iPads outstrip the gross domestic product of several South American countries so they must be useful.

Already, Jobbites reading this are spluttering and trembling, their wobbling fingers poised to launch into a tirade of near-religious rhetoric in defense of the New World Order, where Apple will save the world by the benevolent use of the “One store to bring them all and in the darkness bind them.” But take a deep breath and hold back for just a few minutes.

Whether using technology for teaching articulation skills, sentence construction, fluency, AAC, literacy, and so on, the glamour [2] of the machine can cloud our judgment. Our pads, tablets, smart phones, and laptops are such magical totems that we really want them to be special. The trick is for us to make sure we are open-minded enough to see the benefits of new technology (or just new approaches in general) but not so open-minded that our brains fall out.

Sackett, Strauss, Richardson, Rosenberg, and Haynes (2000) [3] talked about using a foreground question to determine the quality of external evidence that can help us make informed decisions about using a new approach. The use the acronym PICO as a memory aid that helps us formulate a testable question;

• A Patient or Problem
• An Intervention (a treatment or evaluation)
• A Comparison or Contrast
• An Outcome (measurable, of course)

Thus, a good question to ask would be “If I use this app for a month (I) with my 4-year-old client with a hearing loss (P) will the improvement in selecting new images on the screen (O) be better than if I’d used simple picture books (C).”

OK, so you can fine tune this in a few ways but we’re writing a blog post not an EBP textbook. The point is that we do have the clinical tools to evaluate the use of flashy new technology if we ask the right questions. Just using an app for a month and noticing “change” tells you nothing; the client may have “changed” if you’d let them watch SpongeBob SquarePants with you because the interaction was the cause, not the app.

Healthy skepticism is not a rejection of change but a necessary perspective to evaluate the extent of change. What is dangerous about new approaches, technological or social, is when claims are made to efficacy that are based purely on anecdote and a wish to see things happen.

And who would have thought White Russians and meatballs could lead to this.

Time for bed…

Time for bed

Notes

[1] Curiously it is iPads and not tablets in general. A recent report asked schools if they were considering buying new tablet technology for the classrooms and 100% said iPads against 0% for any other device. Now, doesn’t that strike anyone else as odd if the education system is supposed to be evaluating things on merit? Or is this a triumph for Apple’s marketing department? Just a thought.

 [2] The word glamour not only means “fascination” or “allure” but also “a magical spell cast over a person to hold them in thrall.” Literary types might want to take a read, or re-read, of Tennyson’s Idylls of the King and the verses entitled The Beguiling of Merlin. This demonstrates poetically how the glamorous Vivienne enchants and defeats the Arthurian mythological wizard.

[3] A Jobbite is an individual who considers Steve Jobs and all his works as beyond criticism. Followers of Ayn Rand have a similar perspective, which ultimately can result in the emergence of cults. Although Jobs was unquestionably influential in the world of technology, it’s easy to forget that Apple succeeded because of the creativity and hard-work of its workers. Jobs himself didn’t actually sit down and build iAnythings – he facilitated it, and there was his skill.

[4] Sackett, D.L., Strauss, S.E., Richardson, W.S., Rosenberg, W. and Haynes, R.B. (2000) Evidence-based medicine: How to practice and each EBM. Edinburgh, Scotland: Churchill Livingstone.

Quackery, Hokum, Baloney: Separating Science from Stupidity

Suppose I told you that somewhere between Earth and Mars there is an invisible teapot that orbits the sun once every 666 days. The teapot is invisible because it is cloaked using technology developed by space aliens, who left it there to monitor our progress. They believe that once we make contact with the teapot, an alarm will sound and they will return to see if we are truly worthy of being galactic citizens.

The Orbiting Teapot

The question you need to ask is; “Is that true?” and if so, “How do I know it’s true?” This is, of course, the fundamental question for Science. What do we know and how do we know it.

The invisible teapot was created by the philosopher Bertrand Russell back in 1952 and went like this:

If I were to suggest that between the Earth and Mars there is a china teapot revolving about the sun in an elliptical orbit, nobody would be able to disprove my assertion provided I were careful to add that the teapot is too small to be revealed even by our most powerful telescopes. But if I were to go on to say that, since my assertion cannot be disproved, it is intolerable presumption on the part of human reason to doubt it, I should rightly be thought to be talking nonsense.

But this is the 21st century, after all, and we are all sophisticated, intelligent people, and we have a wealth of scientific knowledge and instrumentation to help us test for the presence of the teapot. The “powerful telescopes” of 1952 have been replaced by much more sophisticated technology and we can now see much more of our solar system.

Bertrand Russell

In principle, therefore, we could focus all the world’s telescopes along the elliptic plane and search for the pot. We already know what it looks like, we know it is between Earth and Mars, and we know it is cloaked. The cloaking business may make it trickier but we also know from current research that “cloaking” is little more than deflecting light around a mass. We could spot the teapot by looking to see if there is a teapot-sized region of space that makes stars behind it appear to change position; this is because the mass of the pot will cause light to bend ever so slightly (it why in a solar eclipse we can see stars on the edge of the sun that are actually behindit).

The key thing to note here is that we TEST for the presence of the teapot and refuse to accept it on faith. I may be able to spin the most wonderful story about the pot, about how beautiful and splendiferous it can be, and how much it has changed my life, but if all I have is my personal perceptions and ideas, you would be right to treat what I say as bullshit of the highest order.

The only way for me to prove that I am right is to provide evidence of the pot. If the telescopes suddenly reveal a sea-green piece of revolving pottery, orbited by teacups (hey, there may be more to the teapot than I knew!) then you should start taking me more seriously. When several independent observatories have pictures, and all independently identify its location by numerical coordinates, and spectrograph analyses all show its chemical structure, then I’m pretty much vindicated. And if after a few years NASA’s latest “Pot Probe” reaches that location and scoops up the teapot into its gaping maw, then that’s likely to be as much proof as any reasonable person would require to be able to say, “Yes, there IS a teapot in outer space.”

Testability is a cornerstone of Science. And the thing that has to be tested is a HYPOTHESIS[1], which is defined as;

A proposition or principle put forth or stated (without any reference to its correspondence with fact) merely as a basis for reasoning or argument, or as a premiss from which to draw a conclusion.

The aim of Science is to test a hypothesis, that is, to see if it is true or false. Now in reality, you don’t actually prove something to be “true,” you “support” it. Truth and support are two very different things. If my hypothesis is “All swans are white,” I can test it by sitting by a river bank photographing every swan than lands on the water in front of me. If I have several friends across the world do the same thing, we might find that all the pictures we have turn out to be white swans. Does this mean that “All swans are white” is true? Nope, it just means that there is overwhelming support, based on many observations and measurements by many people, that swans are white. However – and here’s the kicker – if we find just ONE black swan, the hypothesis is dead in the water. Gone. No amount of evidence can make a hypothesis true, but just one observation can make it false.

This is the principle of falsification, promoted and discussed at great length by the great philosopher of Science, Karl Popper, whose Logic of Scientific Discovery[2] is  a classic in the field. For a more relaxed read (and by “relaxed” I mean “requires a little concentration” as opposed to “Oh, my frickin’ head’s about to explode!”) you might prefer Objective Knowledge: An Evolutionary Approach[3], published in 1972.

Karl Popper

It’s also explained eloquently by another Carl, Carl Sagan, in his 1996 book The Demon-Haunted World: Science as a Candle in the Dark. I can’t recommend this book enough to students who are eager to learn about the scientific method in an enjoyable and entertaining fashion. It is, perhaps, his best and most lucid book, and it beats me why this isn’t recommended as a high-school text or at least an undergraduate offering to all students. Many people have a woeful understanding of what science and the scientific method are all about and this one book explains it so well.

One particularly practical offering is Chapter 12: The Fine Art of Baloney Detection, where Sagan offers a number of ways to check whether a proposition or hypothesis is valid. Here’s the list for Baloney Detection:

• Wherever possible you need to find independent confirmation of the facts. One person or test does not a hypothesis prove!
• Encourage and engage in debate on the evidence by knowledgeable proponents of all points of view.
• Don’t fall for arguments from authority alone; I may have a PhD in Astrophysics but that doesn’t mean there IS a teapot.
• Be prepared to try multiple hypotheses.
• Try not to get overly attached to a hypothesis just because it’s yours, THis is harder than you might think.
• Measure, measure, measure. Objective numbers always trump personal beliefs, no matter how many folks share that belief.
• If there is a chain of argument every link in the chain must work.
• Sharpen up Occam’s Razor[3] – if there are two hypothesis that could explain the data equally well, choose the simpler.
• Check to see if the hypothesis can, at least in principle, be falsified: Is it testable? If it isn’t testable, it isn’t science![4]
• Can other people replicate the experiment and get the same result?
• Conduct control experiments, especially “double-blind” experiments where the person taking measurements is not aware of the test and control subjects.
• Check for confounding factors; make sure you separate as many of the variables as you can.

This is why evidence-based practice is so important. It separates the speculative from the scientific. The current rush to buy iDevices as a blanket solution for those individuals who need an AAC device is a good example of where hypotheses precede evidence. When someone turns up at the clinic doors with a kid, an iPad, and a recommendation from a video on YouTube that “this is the answer,” what do you say? There are many purported “evidential” video clips on the Internet that are well-meaninged attempts by parents to show how their kids have “improved” by using technology, but with no pre-testing and no measure of what “improvement” is, it’s impossible to call this evidence.

In their desire to help people with communication problems, it’s sometimes easier to believe in orbiting teapots than measure performance.

Notes
[1] The word hypothesis comes directly from the Greek ὑπόθεσις and means “placing under.” ὑπό is “under” and you see this in words such as hypodermic (under the skin), hypothalamus (under the thalamus), and hypochondria (under the breast-bone). The θέσις part orignal referred to the action of placing a foot or hand down to beat time in poetry or music, and it became, by extension, the putting down of a proposition or idea.

[2] Popper, K.R. (1935) Logik der Forschung (The Logic of Research) , Vienna: Springer; trans. The Logic of Scientific Discovery, London: Hutchinson, 1959.

[3] Popper, K.R. (1972) Objective Knowledge, Oxford: Clarendon Press. If you just want to focus on just one chapter, try Chapter 6: Of Clouds and Clocks, which can be read somewhat independently of the book as a whole, and is less dense than some of the earlier chapters. Popper isn’t the easiest of folks to read and in truth, I still have a hard time with much of his stuff on probability because of the math and logic involved, but he’s well worth the effort.

[4] “Pluralitas non est ponenda sine neccesitate” or “plurality should not be posited without necessity.” This is attributed to William of Ockam (1285-1349), an English Franciscan Monk and philosopher, who used this premise in much of his work and thinking, although the notion was actually a common principle in medieval thought. The actual phrase, Occam’s Razor, appeared first in 1852 and was used by the astronomer and physicist, William Rowen Hamilton. No mention of his looking for a teapot…

[5] The difference between Science and Pseudoscience often comes down to this rule of Testability. An idea that is inherently untestable is called metaphysical or speculation. You may well believe passionately that there are fairies at the bottom of your garden but unless you can subject them to testing, they are no more real than my orbiting teapot.