[WEB SECURITY] How to evaluate a CAPTCHA

[Jeremiah Grossman <jeremiah@xxxxxxxxxxxxxxx>]

Since the topic of defeating CAPTCHA's came up, I thought I'd lend my  
thoughts to the conversation about how to evaluate them.

Today's website are battered by a number of automated attacks by web  
robots. The most common attacks we see are blog and message board  
spam, mass account registration, username/password brute force (DoS),  
and web-based game automation.

First, what's a CAPTCHA and how are they used? The acronym means...
"Completely Automated Public Turing Test to Tell Computers and Humans  
Apart"

CAPTCHA's, specifically the image variety we're used to seeing, are  
one of several methods available to defend against the attacks above.  
The other methods we have available are request/response throttling,  
action limit triggers, embedded javascript code, etc.  Image  
CAPTCHA's have become highly popular for stopping the web robots  
because they're simple and effective (to a varied degree).

As other posts in this thread indicate, Image CAPTCHA's may be  
vulnerable to optical character recognition (OCR) attacks. In  
addition, these system commonly have implementation weaknesses, but  
thats another topic all together. From a UI perspective, complaints  
against Image CAPTCHA's are that people don't like filling out the  
random code and the system discriminates against those with vision  
impairments.

In response, other styles of CAPTCHA's have been suggested. Its  
sometimes difficult to determine whether a recommended CAPTCHA is  
going to work or scale well on the Web. To evaluate the effectiveness  
of a new CAPTCHA, I've been working on a list of characteristics to  
use as a guide. These are characteristics which I've found are  
important in order for the solution to function securely in the Web  
medium.

Lets call it the CATCHA Effectiveness Test.

--------
1) The test must be able to be administered where the human and the  
server are remote to each other over the network.

2) The test must be easy for humans to pass.
- Less than 0.01% of humans should fail the test on the first attempt.

3) The test must be hard for computer to pass
- Computers should have less than a 1 in 10,000,000 chance of  
guessing the correct answer. (Even after a pre-determined amount of  
analysis time)

4) The test must be able to be completed by a human in less than a  
several seconds.

5) Knowledge a test question, answer, or result (or combination  
thereof) must not impact the predictability of following tests.

6) The test should not discriminate against the blind or the deaf. Or  
provide a solution to address the issue.

7) The test should not possess a geographic, cultural, or language bias.
--------

Hopefully others on the list will have some good ideas on how to  
improve the model. Or know of existing resources.



Regards,

Jeremiah Grossman



---------------------------------------------------------------------
The Web Security Mailing List
http://www.webappsec.org/lists/websecurity/

The Web Security Mailing List Archives
http://www.webappsec.org/lists/websecurity/archive/