|
|
| Line 1: |
Line 1: |
| {{Refimprove|date=May 2011}}
| | Myrtle Benny is how I'm known as and I really feel comfy when people use the complete name. Hiring has been my occupation for some time but I've currently applied for another 1. Minnesota has usually been his house but his wife desires them to transfer. Doing ceramics is what my family and I enjoy.<br><br>Feel free to visit my web page :: at home std testing ([http://Moodle.Kspu.Karelia.ru/user/view.php?id=23302&course=1 linked here]) |
| {{Expert-subject|date=May 2011}}
| |
| | |
| Given a population whose members can be potentially separated into a number of different sets or [[Class (set theory)|classes]], a '''classification rule''' is a procedure in which the elements of the population set are each assigned to one of the classes.<ref>[http://mathworld.wolfram.com/StatisticalTest.html Mathworld article for statistical test]</ref> A perfect test is such that every element in the population is assigned to the class it really belongs. An imperfect test is such that some errors appear, and then [[statistics|statistical analysis]] must be applied to analyse the [[statistical classification|classification]].
| |
| | |
| A special kind of classification rule are [[binary classification]]s.
| |
| | |
| == Testing classification rules ==
| |
| [[File:binary-classification-labeled.svg|thumb|220px|right|The [[confusion matrix]] can be used to derive four basic measures]]
| |
| | |
| Having a dataset consisting in couples ''x'' and ''y'', where ''x'' is each element of the population and ''y'' the class it belongs to, a '''classification rule''' can be considered as a function that assigns its class to each element. A binary classification is such that the label ''y'' can take only a two values.
| |
| | |
| A classification rule or '''classifier''' is a function ''h'' that can be evaluated for any possible value of ''x'', specifically, given the data <math>\{(x_1,y_1),\dots,(x_n, y_n)\}</math>, ''h''(''x'') will yields a similar classification <math>\hat{y}=h(x),</math> as close as possible to the true group label ''y''.
| |
| | |
| The true labels ''y<sub>i</sub>'' can be known but will not necessarily match their approximations <math>\hat{y_i}=h(x_i)</math>. In a binary classification, the elements that are not correctly classified are named false positives and false negatives.
| |
| | |
| Some classification rules are static functions. Others can be computer programs. A [[Statistical classification (machine learning)|computer classifier]] can be able to learn or can implement static classification rules. For a training data-set, the true labels ''y<sub>j</sub>'' are unknown, but it is a prime target for the classification procedure that the approximation :<math>\hat{y_j}=h(x_j) \approx y_j</math> as well as possible, where the quality of this approximation needs to be judged on the basis of the statistical or probabilistic properties of the overall population from which future observations will be drawn.
| |
| | |
| Given a classification rule, a '''classification test''' is the result of applying the rule to a finite sample of the initial data set.
| |
| | |
| ==Binary and multiclass classification==
| |
| Classification can be thought of as two separate problems - [[binary classification]] and [[multiclass classification]]. In binary classification, a better understood task, only two classes are involved, whereas multiclass classification involves assigning an object to one of several classes.<ref>Har-Peled, S., Roth, D., Zimak, D. (2003) "Constraint Classification for Multiclass Classification and Ranking." In: Becker, B., Thrun, S., Obermayer, K. (Eds) ''Advances in Neural Information Processing Systems 15: Proceedings of the 2002 Conference'', MIT Press. ISBN 0-262-02550-7</ref> Since many classification methods have been developed specifically for binary classification, multiclass classification often requires the combined use of multiple binary classifiers.
| |
| | |
| Binary classification methods include [[probit regression]] and [[logistic regression]]. Multiclass classification methods include [[multinomial probit]] and [[multinomial logit]].
| |
| | |
| ==Table of Confusion==
| |
| | |
| When the classification function is not perfect, false results will appear. The example confusion matrix below, of the 8 actual cats, a function predicted that three were dogs, and of the six dogs, it predicted that one was a rabbit and two were cats. We can see from the matrix that the system in question has trouble distinguishing between cats and dogs, but can make the distinction between rabbits and other types of animals pretty well.
| |
| | |
| {| border="1" cellspacing=0 cellpadding="2"
| |
| |+Example confusion matrix
| |
| !
| |
| !
| |
| !colspan="3"| Predicted
| |
| |-
| |
| !
| |
| !
| |
| !Cat
| |
| !Dog
| |
| !Rabbit
| |
| |-
| |
| !rowspan="3"| Actual
| |
| !Cat
| |
| |5
| |
| |3
| |
| |0
| |
| |-
| |
| !Dog
| |
| |2
| |
| |3
| |
| |1
| |
| |-
| |
| !Rabbit
| |
| |0
| |
| |2
| |
| |11
| |
| |}
| |
| | |
| When dealing with binary classifications these concepts are simpler
| |
| | |
| | |
| ===False positives===
| |
| | |
| [[Type I and type II errors|False positive]]s result when a test falsely or incorrectly reports a positive result. For example, a [[medical test]] for a [[disease]] may return a positive result indicating that patient has a disease even if the patient does not have the disease. We can use Bayes' theorem to determine the probability that a positive result is in fact a false positive. We find that if a disease is rare, then the majority of positive results may be false positives, even if the test is accurate.
| |
| | |
| Suppose that a test for a disease generates the following results:
| |
| | |
| * If a tested patient has the disease, the test returns a positive result 99% of the time, or with probability 0.99
| |
| * If a tested patient does not have the disease, the test returns a positive result 5% of the time, or with probability 0.05.
| |
| | |
| Naively, one might think that only 5% of positive test results are false, but that is quite wrong, as we shall see.
| |
| | |
| Suppose that only 0.1% of the population has that disease, so that a randomly selected patient has a 0.001 prior probability of having the disease.
| |
| | |
| We can use Bayes' theorem to calculate the probability that a positive test result is a false positive.
| |
| | |
| Let ''A'' represent the condition in which the patient has the disease, and ''B'' represent the evidence of a positive test result. Then, the probability that the patient actually has the disease given the positive test result is
| |
| | |
| : <Math>\begin{align} P(A | B) &= \frac{P(B | A) P(A)}{P(B | A)P(A) + P(B |\text{not }A)P(\text{not }A)} \\ \\
| |
| &= \frac{0.99\times 0.001}{0.99 \times 0.001 + 0.05\times 0.999} \\ ~\\ &\approx 0.019. \end{align}</math>
| |
| | |
| and hence the probability that a positive result is a false positive is about 1 − 0.019 = 0.98, or 98%.
| |
| | |
| Despite the apparent high accuracy of the test, the incidence of the disease is so low that the vast majority of patients who test positive do not have the disease. Nonetheless, the fraction of patients who test positive who do have the disease (0.019) is 19 times the fraction of people who have not yet taken the test who have the disease (0.001). Thus the test is not useless, and re-testing may improve the reliability of the result.
| |
| | |
| In order to reduce the problem of false positives, a test should be very accurate in reporting a ''negative'' result when the patient does not have the disease. If the test reported a negative result in patients without the disease with probability 0.999, then
| |
| | |
| : <Math>P(A|B) = \frac{0.99\times 0.001}{0.99 \times 0.001 + 0.001\times 0.999} \approx 0.5, </math> | |
| | |
| so that 1 − 0.5 = 0.5 now is the probability of a false positive.
| |
| | |
| ===False negatives===
| |
| | |
| On the other hand, [[Type I and type II errors|false negative]]s result when a test falsely or incorrectly reports a negative result. For example, a medical test for a disease may return a negative result indicating that patient does not have a disease even though the patient actually has the disease. We can also use Bayes' theorem to calculate the probability of a false negative. In the first example above,
| |
| | |
| : <Math>\begin{align} P(A |\text{not }B) &= \frac{P(\text{not }B | A) P(A)}{P(\text{not }B | A)P(A) + P(\text{not }B |\text{not }A)P(\text{not }A)} \\ \\
| |
| | |
| &= \frac{0.01\times 0.001}{0.01 \times 0.001 + 0.95\times 0.999} \\ ~\\ &\approx 0.0000105.\end{align}</math>
| |
| | |
| The probability that a negative result is a false negative is about 0.0000105 or 0.00105%. When a disease is rare, false negatives will not be a major problem with the test.
| |
| | |
| But if 60% of the population had the disease, then the probability of a false negative would be greater. With the above test, the probability of a false negative would be
| |
| | |
| : <Math>\begin{align} P(A |\text{not }B) & = \frac{P(\text{not }B | A) P(A)}{P(\text{not }B | A)P(A) + P(\text{not }B |\text{not } A) P(\text{not }A)} \\ \\ | |
| | |
| &= \frac{0.01\times 0.6}{0.01 \times 0.6 + 0.95\times 0.4} \\ ~\\ &\approx 0.0155.\end{align}</math>
| |
| | |
| The probability that a negative result is a false negative rises to 0.0155 or 1.55%.
| |
| | |
| ===Worked example===
| |
| {{SensSpecPPVNPV}}
| |
| | |
| ==Measuring a classifier with sensitivity and specificity==
| |
| | |
| In training a classifier, one may wish to measure its performance using the well-accepted metrics of sensitivity and specificity. It may be instructive to compare the classifier to a random classifier that flips a coin based on the prevalence of a disease. Suppose that the probability a person has the disease is <math>p</math> and the probability that they do not is <math>q=1-p</math>. Suppose then that we have a random classifier that guesses that the patient has the disease with that same probability <math>p</math> and guesses that he does not with the same probability <math>q</math>.
| |
| | |
| The probability of a true positive is the probability that the patient has the disease and the probability that the random classifier guesses this correcty, or <math>p^2</math>. With similar reasoning, the probability of a false negative is <math>pq</math>. From the definitions above, the sensitivity of this classifier is <math>p^2/(p^2+pq)=p</math>. With more similar reasoning, we can calculate the specificity as <math>q^2/(q^2+pq)=q</math>.
| |
| | |
| So, while the measure itself is independent of disease prevalence, the performance of this random classifier depends on disease prevalence. The classifier may have performance that is like this random classifier, but with a better-weighted coin (higher sensitivity and specificity). So, these measures may be influenced by disease prevalence. An alternative measure of performance is the [[Matthews correlation coefficient]], for which any random classifier will get an average score of 0.
| |
| | |
| The extension of this concept to non-binary classifications yields the [[confusion matrix]].
| |
| | |
| ==See also== | |
| * [[Statistical classification]]
| |
| * [[Medical test]]
| |
| * [[Diagnostic test]]
| |
| * [[Bayesian inference]]
| |
| * [[Sensitivity and specificity]]
| |
| * [[Gold standard (test)]]
| |
| | |
| ==References==
| |
| <references />
| |
| | |
| [[Category:Statistical classification]]
| |
Myrtle Benny is how I'm known as and I really feel comfy when people use the complete name. Hiring has been my occupation for some time but I've currently applied for another 1. Minnesota has usually been his house but his wife desires them to transfer. Doing ceramics is what my family and I enjoy.
Feel free to visit my web page :: at home std testing (linked here)