Select your font size  A  A  A 
 
about us products & services consulting & support news & events contact us
Centuries-old techniques developed by Thomas Bayes find modern applications because they are simple and effective.

Bayesian Techniques Assist Automated Decision Tools - Nova Scotia

print this article 
 

Centuries-old techniques developed by Thomas Bayes find modern applications because they are simple and effective.

(1) Bayesian techniques are employed in the automated filtering of unwanted spam, the formation of medical diagnoses, the detection of viruses, and in several other ways that advance compatible business objectives.

Bayes Theorem uses Conditional Probability to calculate the probability of A given B, provided that the probability of B given A, the probability of A and the probability of B are all known.

For example, suppose an automated program could determine that a particular phrase is present in 70% of spam and 50% of non-spam emails, and that an email is 90% likely to be spam, and suppose that A means "the email is spam", B means "the email is not spam", and C means "the phrase is present". Then P(A), the probability of A, is 90%. P(B) is 10%. P(C|A), the probability of C given that A is true, is 70%. P(C|B) is 50%. P(C) = (0.70 * 0.90) + (0.50 * 0.10) = 0.68 = 68%. Some useful probabilities for classifying the email would then be P(A|C), the probability that the email is spam given that the phrase is present, and P(B|C), the probability that the email is not spam given that the phrase is present. Using conditional probability, P(A|C) = P(A) * P(C|A) / P(C). P(A|C) = 0.90 * 0.70 / 0.68 ~= 0.92647 ~= 93%. P(B|C) = P(B) * P(C|B) / P(C) = 0.10 * 0.50 / 0.68 ~= 0.07353 ~= 7%. Therefore, the probability that the email is spam, based on that one data point, is 93%, and the probability that it is not spam is 7%.

To improve the accuracy of this technique, a computer program could analyze thousands of data points in less than a tenth of a second, which is approximately how long it takes to download an email message. The program could test for phrase D, phrase E, phrase F, etc., and use data about each one to modify the overall confidence that the email is spam or not spam. In hand-waving mathematical terms, if C, D and E are found to be true, the program can automatically determine P(A|C,D,E) (the probability that the email is spam given that C, D, and E are true) using an extended version of Bayes Theorem, applied to a Bayesian network. By using good tokens (i.e. by asking the right questions, using all available information), this technique can be up to 99.5% accurate with 0.3% false positives.

A related algorithm is state-based, so that rather than using a directed acyclic graph of conditional probabilities (a Bayesian network), one could automatically produce a set of known states, along with transition matrices (stochastic matrices) showing the probability of moving from a given state to another (transition probabilities). These stochastic matrices fall naturally from statistics coming from a large enough set. CRM114 uses this technique to improve spam detection accuracy beyond naive Bayesian techniques, and suggests innovative approaches to potentially bring 99.999% reliability.

The applications of statistical techniques such as described above go beyond spam detection. For instance, one could imagine the same or similar techniques being used in the medical field to classify DNA, in industry to properly direct calls that come through automated call systems, in elections to predict how a particular message may affect polls, etc.

To learn more about how statistical methods can be used to improve your business, please contact us.

Works Cited

  1. J J O'Connor and E F Robertson. "Thomas Bayes." From The MacTutor History of Mathematics archive.
  2. William S. Yerazunis. "The Spam Filtering Plateau at 99.9% Accuracy and How to Get Past It." From
    3. CRM114 - the Controllable Regex Mutilator.
  3. Eric W. Weisstein. "Bayes' Theorem." From MathWorld--A Wolfram Web Resource.
  4. Eric W. Weisstein. "Conditional Probability." From MathWorld--A Wolfram Web Resource.
  5. Paul Graham. "Better Bayesian Filtering." From
    6. Paul Graham.

Most Recent Website and Regional Updates

 Research Tools
Measure human resource allocation and collect data with the goal of determining patterns that will bring forward actionable insights which may lead to policy changes, saving money and improving quality of service.
 
 Process Evaluation Questions
Questions to help focus discussion about process improvement
 
 Operations Research
Operations Research (frequently called OR), is the methodical study of how to do things better. It is also called Optimization Theory.
 
 Our Role in Operations Optimization
Meet objectives more efficiently by improving operational effectiveness and profitability.
 
 Monte Carlo Method
Short description of the Monte Carlo Method in optimization theory.
 
 Introduction to Markov Process
Introduction to Markov Chains and Markov Processes, with a link to an introductory PDF which provides essential details about Markov analysis, along with helpful examples and exercises.
 
 With new website, community news is just a click away
Dear Readers: We're pleased to announce the launch of our new website www.
 
 Crowd wants it built right the first time: Build It Right, supporters gather against secrecy
Protesters marched across the site of the Mainland Common last Saturday to demonstrate against what they call the secrecy of the plans.
 
 Boy has a 'helping out' spirit: IWK telethon organizers praise nine-year-old's accomplishment
Ben Ryan is a young man with a plan. This weekend, the nine-year-old Basinview Drive Elementary School student, his younger brother Alex and at least one friend will be featured on the IWK Telethon for Children where they'll hand over $1,041.
 
 Local Sparks hope to keep the flame alight
Despite an increase in the number of girls signed up in the program, the 2nd Dartmouth Sparks unit may have to close its doors unless new leaders are found next year.
 
 Whalen slams premier's 'paternalistic' attitude: Clayton Park MLA calls for full disclosure of Mainland Common plans
Fed up with the secrecy surrounding the Mainland Common plans, and fearing a rerun of the Commonwealth Games debacle, MLA Diana Whalen is calling on the Progressive Conservatives to pass a bill that would force governments to be open about major projects they are funding.
 
 If voters have to choose ...: While both have declined, Kelly holds majority over Fougere
When given the choice between Peter Kelly and Sheila Fougere as mayor of Halifax Regional Municipality, just under one-half of residents choose Kelly, and two in 10 support Fougere, a recent Corporate Research Associates poll suggests.
 
 Massive new development sparks little public interest
Last week's public hearing about a large development proposed for Middle Sackville may have garnered more than two hours of debate, but local Coun.
 
 Plaque to honour a fallen soldier
There were tears and laughter as people gathered last week to honour Pte. Braun Woodfield by dedicating a plaque to his memory at the Royal Canadian Legion (Branch 164) in Eastern Passage.
 
 Respite care offers much-needed break: Workshop offers help for parents of special-needs children
6,210. That's the number of children under the age of 15 in this province who have some form of disability.
 
 Have your say in a greener community
Your chance to have a say in how we treat our natural environment is coming to a conference room near you.
 
 Car crashes through house
The RCMP Halifax District and RCMP Metro Traffic Services responded to a vehicle, which crashed into a residence at 5:15 p.m. Monday afternoon on Prospect Road.
 
 Central library plans announced
After months of public consultations and planning, Halifax Public Libraries and the HOK Planning Group are excited to present the chosen building concept for the new Central Library for HRM.
 
 Website celebrates HRM Bike Week
A new website for HRM's Bike Week activities has gone live and Coun. Sheila Fougere is urging residents to visit the site to learn about the many events planned for this year.
 
 Province invests money in playground equipment
School children will soon get to spend their recesses and lunch hours on new playground equipment thanks to $192,000 worth of investments announced by the province this week.
 
 Community program funding helps prevent crime
More than 30 communities are receiving one-time grants this week from the Department of Justice for projects that support Nova Scotia's crime prevention strategy, Time to Fight Crime Together.
 
 Do you know someone who should be awarded for bravery?
Nominations are being sought for Nova Scotia's new Medal of Bravery.
 
 Don't burn that brush without a fire permit
Halifax Regional Fire and Emergency reminds residents that it is illegal to burn brush without a burning permit in Halifax Regional Municipality and in many areas of HRM (including Bedford, Halifax, Dartmouth, Cole Harbour, Timberlea and parts of Sackville), burning is not permitted at all.
 
 Nova Scotian youth get hooked on fishing
Youth from across Nova Scotia are baiting their hooks and heading to the province's fishing spots as part of the Learn to Fish (L2F) program.
 
 Website makes researching older homes easier
Nova Scotians interested in researching the history of their homes and properties now have a new Internet resource on their side.
 
 'I run for hope and a cure': HEALTH Relay for Life raises money for cancer
The fifth anniversary of the Canadian Cancer Relay for Life in Sackville is fast approaching and with more than 50 teams registered and a first-ever event sponsor, organizers are confident this relay will create more awareness and raise more funds than ever before.
 
 Cancer Care invites you to 'Celebrate Life 2008' festivities
For the seventh year in a row, Cancer Care Nova Scotia will present Celebrate Life 2008, an afternoon of music and festivities as Nova Scotians mark National Cancer Survivors Day.
 
 Wilderness areas targeted by joint forces' operation
Officers from provincial and federal agencies joined forces through early May to increase awareness of rules affecting back country angling and proper use of Nova Scotia wilderness areas.
 

Google
 
Web transparen.com

Contact Information

Related Information

   
 
E C M | © 2003-2007 Transparen Corp.      

Standardized Services: Data Recovery Service / Creative Services / Premium Web Hosting Services / System Administration Tech Support Services
Recent Projects: Full-Service Mortgage and Financing Company / System to manage flights from Vancouver to Tofino / Photo exchange verification service
Our Vancouver BC Server Proudly Hosts: automated parking and revenue control systems, leafside lane at southlands, cost effective alternative power sources, Higher Grade Learning Centres, pacific forage bag supply, sunburst medical, neosonic design, roger mahler photography - passionate, intriguing, desirable, the connection between east and west, affordable flights to victoria and tofino, low interest mortgage brokers in vancouver, richmond, surrey, toronto, Toronto Calgary and Vancouver IT staffing and talent search
* Annapolis County, Nova Scotia * Antigonish County, Nova Scotia * Argyle, Nova Scotia * Barrington, Nova Scotia * Cape Breton Regional Municipality, Nova Scotia * Chester, Nova Scotia * Clare, Nova Scotia * Colchester County, Nova Scotia * Cumberland County, Nova Scotia * Digby, Nova Scotia * Guysborough, Nova Scotia * Halifax Regional Municipality, Nova Scotia * East Hants, Nova Scotia * West Hants, Nova Scotia * Inverness County * Kings County * Lunenburg, Nova Scotia * Pictou County, Nova Scotia * Queens Regional Municipality, Nova Scotia (see also Queens County, Nova Scotia) * Richmond County, Nova Scotia * Shelburne, Nova Scotia * St. Mary's, Nova Scotia * Victoria County, Nova Scotia * Yarmouth, Nova Scotia * Amherst, Nova Scotia * Annapolis Royal, Nova Scotia * Antigonish, Nova Scotia * Berwick, Nova Scotia * Bridgetown, Nova Scotia * Bridgewater, Nova Scotia * Canso, Nova Scotia * Clark's Harbour, Nova Scotia * Digby, Nova Scotia * Hantsport, Nova Scotia * Kentville, Nova Scotia * Lockeport, Nova Scotia * Louisdale, Nova Scotia * Lunenburg, Nova Scotia * Mahone Bay, Nova Scotia * Middleton, Nova Scotia * Mulgrave, Nova Scotia * New Glasgow, Nova Scotia * Oxford, Nova Scotia * Parrsboro, Nova Scotia * Pictou, Nova Scotia * Port Hawkesbury, Nova Scotia * Port Joli, Nova Scotia * Port Mouton, Nova Scotia * Shelburne, Nova Scotia * Springhill, Nova Scotia * Stellarton, Nova Scotia * Stewiacke, Nova Scotia * Trenton, Nova Scotia * Truro, Nova Scotia * Westville, Nova Scotia * Windsor, Nova Scotia * Wolfville, Nova Scotia * Yarmouth, Nova Scotia * Bear River 6, Nova Scotia * Bear River 6B, Nova Scotia * Cambridge 32, Nova Scotia * Chapel Island 5, Nova Scotia * Cole Harbour 30, Nova Scotia * Eskasoni 3, Nova Scotia * Fisher's Grant 24, Nova Scotia * Gold River 21, Nova Scotia * Horton 35, Nova Scotia * Indian Brook 14, Nova Scotia * Membertou 28B, Nova Scotia * Merigomish Harbour 31, Nova Scotia * Millbrook 27, Nova Scotia * New Ross 20, Nova Scotia * Pennal 19, Nova Scotia * Pomquet and Afton 23, Nova Scotia * Sheet Harbour 36, Nova Scotia * Shubenacadie 13, Nova Scotia * Summerside 38, Nova Scotia * Wagmatcook 1, Nova Scotia * Whycocomagh 2, Nova Scotia * Wildcat 12, Nova Scotia * Yarmouth 33, Nova Scotia * Aylesford, Nova Scotia * Baddeck, Nova Scotia * Bible Hill, Nova Scotia * Canning, Nova Scotia * Caribou, Nova Scotia * Chester, Nova Scotia * Collingwood, Nova Scotia * Cornwallis Square, Nova Scotia * Dover, Nova Scotia * Freeport, Nova Scotia * Greenwood, Nova Scotia * Havre Boucher, Nova Scotia * Hebbville, Nova Scotia * Kingston, Nova Scotia * Lawrencetown, Annapolis County, Nova Scotia * New Minas, Nova Scotia * Port Williams, Nova Scotia * Pugwash, Nova Scotia * River Hebert, Nova Scotia * St. Peter's, Nova Scotia * Tatamagouche, Nova Scotia * Tiverton, Nova Scotia * Westport, Nova Scotia * Weymouth, Nova Scotia * Advocate Harbour, Nova Scotia * Arichat, Nova Scotia * Arisaig, Nova Scotia * Arlington, Nova Scotia * Aspen, Nova Scotia * Atlanta, Nova Scotia * Bangor, Nova Scotia * Barrachois, Cape Breton County, Nova Scotia * Barrachois, Colchester County, Nova Scotia * Barrington, Nova Scotia * Barrington Passage, Nova Scotia * Bass River, Nova Scotia * Bay St. Lawrence, Nova Scotia * Bear River, Nova Scotia * Bedford, Nova Scotia (was a town) * Beinn Bhreagh, Nova Scotia * Beinn Scalpie, Nova Scotia * Benn Eoin, Nova Scotia * Borneo, Nova Scotia * Brighton, Nova Scotia * Brookfield, Nova Scotia * Brooklyn, Nova Scotia * Caledonia, Nova Scotia * Cambridge, Hants County, Nova Scotia * Cambridge, Kings County, Nova Scotia * Canaan, Kings County, Nova Scotia * Canaan, Lunenburg County, Nova Scotia * Canaan, Yarmouth County, Nova Scotia * Cape North, Nova Scotia * Capstick, Nova Scotia * Cannes, Nova Scotia * Chéticamp, Nova Scotia * Cheverie, Nova Scotia * Church Point, Nova Scotia * Clementsport, Nova Scotia * Clementsvale, Nova Scotia * Conquerall Mills, Nova Scotia * Country Harbour, Nova Scotia * Creignish, Nova Scotia * Cumberland, Nova Scotia * Dartmouth, Nova Scotia (was a city) * Debert, Nova Scotia * Deep Brook, Nova Scotia * Dingwall, Nova Scotia * Dominion, Nova Scotia * Donkin, Nova Scotia * Economy, Nova Scotia * East Berlin, Nova Scotia * Ecum Secum, Nova Scotia * Elmsdale, Nova Scotia * Enfield, Nova Scotia * Falmouth, Nova Scotia * Fauxburg, Nova Scotia * Five Islands, Nova Scotia * Glace Bay, Nova Scotia * Glen Haven, Nova Scotia * Goshen, Nova Scotia * Grand Etang, Nova Scotia * Granville Ferry, Nova Scotia * Great Village, Nova Scotia * Guinea, Nova Scotia * Guysborough, Nova Scotia * Halifax, Nova Scotia (former city) * Halls Harbour, Nova Scotia * Harpellville, Nova Scotia * Hebron, Nova Scotia * Hebb's Cross, Nova Scotia * Hectanooga, Nova Scotia * Hilden, Nova Scotia * Hopewell, Nova Scotia * Hubbards, Nova Scotia * Indian Harbour Lake, Nova Scotia * Indian Point, Nova Scotia * Ingonish, Nova Scotia * Ingonish Beach, Nova Scotia * Inverness, Nova Scotia * Joggins, Nova Scotia * Kingsburg, Nova Scotia * L'Ardoise, Nova Scotia * Larrys River, Nova Scotia * Lawrencetown, Nova Scotia in Halifax Regional Municipality * Little Brook, Nova Scotia * Liverpool, Nova Scotia (was a town) * Londonderry, Nova Scotia * Louisbourg, Nova Scotia * Louisdale, Nova Scotia * Lucas, Nova Scotia * Mabou, Nova Scotia * Maccan, Nova Scotia * Marion Bridge, Nova Scotia * Maitland, Nova Scotia * Martinique, Nova Scotia * Meat Cove, Nova scotia * Melrose, Nova Scotia * Meteghan, Nova Scotia * Meteghan River, Nova Scotia * Middle Musquodoboit, Nova Scotia * Milford Station, Nova Scotia * Milton, Nova Scotia * Miramichi, Nova Scotia * Mount Denson, Nova Scotia * Mount Uniacke, Nova Scotia * Musquodoboit Harbour, Nova Scotia * Necum Teuch, Nova Scotia * New Canada, Nova Scotia * New Germany, Nova Scotia * New Grafton, Nova Scotia * New Ross, Nova Scotia * New Waterford, Nova Scotia * Newcombville, Nova Scotia * Nictaux, Nova Scotia * Nineveh, Lunenburg County, Nova Scotia * Nineveh, Victoria County, Nova Scotia * North Alton, Nova Scotia * North Preston, Nova Scotia * North Sydney, Nova Scotia * Oakland, Nova Scotia * Ohio, Nova Scotia * Oregon, Nova Scotia * Passchendaele, Nova Scotia * Petite Riviere, Nova Scotia * Pleasant Bay, Nova Scotia * Pockwock, Nova Scotia * Port Bickerton, Nova Scotia * Port Hastings, Nova Scotia * Port Hood, Nova Scotia * Port Maitland, Nova Scotia * Port Medway, Nova Scotia * Port Morien, Nova Scotia * Pubnico, Nova Scotia * Renfrew, Nova Scotia * River John, Nova Scotia * Selma, Nova Scotia * Sheet Harbour, Nova Scotia * Sherbrooke, Nova Scotia * Sherose Island, Nova Scotia * Shinimicas Bridge, Nova Scotia * Ship Harbour, Nova Scotia * Shubenacadie, Nova Scotia * Sonora, Nova Scotia * Springfield, Nova Scotia * Spryfield, Nova Scotia * Sydney, Nova Scotia * Sydney Mines, Nova Scotia * Terra Nova, Nova Scotia * Three Mile Plains, Nova Scotia * Trafalgar, Nova Scotia * Tremont, Nova Scotia * Troy, Nova Scotia * Tusket, Nova Scotia * Upper Musquodoboit, Nova Scotia * Upper Stewiacke, Nova Scotia * Valley, Nova Scotia * Victoria, Nova Scotia * Walden, Nova Scotia * Waterville, Nova Scotia * Waverley, Nova Scotia * Wedgeport, Nova Scotia * West Arichat, Nova Scotia * West Berlin, Nova Scotia * Western Shore, Nova Scotia * Whycocomagh, Nova Scotia * Wine Harbour, Nova Scotia * Yankeetown, Nova Scotia