86
$\begingroup$

Deep convolutional neural networks are very good at computer vision related tasks. Optical Character Recognition (OCR) is one important branch of computer vision. In fact, the convolution neural network architecture is pioneered by Yann LeCun in the OCR task of handwritten digits recognition.

So is it possible to use a neural network to recognize characters in CAPTCHA images? These images usually contain multiple characters and have varies distractions.

For example, is it possible use a neural network to recognize the characters as "AKNC" in the following image?

enter image description here

$\endgroup$
  • $\begingroup$ Given the answer below, the short answer to the final question is yes. Note however that nowadays, captcha and re-CAPTCHA have many ways of challenging people to prove that they are not machines. $\endgroup$ – masterxilo Sep 4 '17 at 16:20
97
$\begingroup$

Here is one example using a convolutional neural network (CNN) to crack the CAPTCHA. We will use a CAPTCHA library to generate sample CAPTCHA images and then train a neural network to decode these images. The trained network can be downloaded from crack_CAPTCHA.wlnet.

In the following, we will describe how to prepare the training data, construct and train the CNN.

Data preparation

The CAPTCHA library we will use to generate the images is Captcha. Using this library, We generate about 50K images and use them as our dataset. For simplicity, we constrain ourselves to images with exactly four characters. The code for generating the image will be included at the end of this post. Here are some sample images from the dataset

enter image description here

A few more sample images can be downloaded from here.

After generation our dataset, we can load them into Mathematica.

This is the data directory

dir = "/Users/xslittlegrass/Downloads/captcha/";

The labels of all the images in the dataset

labels = 
  StringSplit[Import[dir <> "labels.txt", "String"], "\n"];

and the total number of samples

nsample = Length[labels]
(* 51200 *)

Load the images as File object

We load the images as File objects, this allows out-of-core training.

fileNameIndex = Sort[FromDigits /@ FileBaseName /@ FileNames["*.jpg", dir]];
imagefiles = Table[File[dir <> ToString[i] <> ".jpg"], {i, fileNameIndex}];

The image has 170X80 dimensions

imgdim = Import[imagefiles[[1, 1]]] // ImageDimensions
(* {170, 80} *)

We use 90% of these sample as training data and 10% as test data

data = <|"Image" -> #1, "digit1" -> StringTake[#2, {1}], 
     "digit2" -> StringTake[#2, {2}], "digit3" -> StringTake[#2, {3}],
      "digit4" -> StringTake[#2, {4}]|> & @@@ 
   Transpose[{imagefiles, labels}];
trainingData = data[[1 ;; Round[0.9*Length[data]]]];
testData = Complement[data, trainingData];

Construct neural network

The neural network contains two parts, the feature extraction network and the classification network. For the feature extraction network, we use a simple CNN resembles the VGG network.

convnet = NetChain[{
   ConvolutionLayer[32, {3, 3}],
   ElementwiseLayer[Ramp],
   ConvolutionLayer[32, {3, 3}],
   ElementwiseLayer[Ramp],
   PoolingLayer[{2, 2}, {2, 2}],
   ConvolutionLayer[64, {3, 3}],
   ElementwiseLayer[Ramp],
   ConvolutionLayer[64, {3, 3}],
   ElementwiseLayer[Ramp],
   PoolingLayer[{2, 2}, {2, 2}],
   ConvolutionLayer[128, {3, 3}],
   ElementwiseLayer[Ramp],
   ConvolutionLayer[128, {3, 3}],
   ElementwiseLayer[Ramp],
   PoolingLayer[{2, 2}, {2, 2}],
   ConvolutionLayer[256, {3, 3}],
   ElementwiseLayer[Ramp],
   ConvolutionLayer[256, {3, 3}],
   ElementwiseLayer[Ramp],
   PoolingLayer[{2, 2}, {2, 2}],
   FlattenLayer[],
   DropoutLayer[0.2]
   }, "Input" -> NetEncoder[{"Image", imgdim}]
  ]

enter image description here

For the classification network, we use 4 independent linear layers for each of our four characters

classes = Join[ToString /@ Range[0, 9], ToUpperCase@Alphabet[]];
net = NetGraph[{convnet, 36, SoftmaxLayer[], 36, SoftmaxLayer[], 36, 
   SoftmaxLayer[], 36, 
   SoftmaxLayer[]}, {NetPort["Image"] -> 
    1 -> 2 -> 3 -> NetPort["digit1"], 
   1 -> 4 -> 5 -> NetPort["digit2"], 1 -> 6 -> 7 -> NetPort["digit3"],
    1 -> 8 -> 9 -> NetPort["digit4"]}, 
  "digit1" -> NetDecoder[{"Class", classes}], 
  "digit2" -> NetDecoder[{"Class", classes}], 
  "digit3" -> NetDecoder[{"Class", classes}], 
  "digit4" -> NetDecoder[{"Class", classes}]]

enter image description here

Training

We use the default setting for the hyper-parameters and train 3 rounds. The total training time is about 2.5 hours on the Nvidia 750M in use. Since Mathematica 11.1 doesn't support GPU training on Mac, the network is trained in Mathematica version 11.0.

trained = 
 NetTrain[net, trainingData, ValidationSet -> testData, 
  MaxTrainingRounds -> 3, TargetDevice -> "GPU"];

The trained network can be downloaded from here

Testing

After 2.5 hours of training, the network seems to converge well. Here are some test on the samples from the test set

Grid@Partition[#, 4] &@
 With[{re = RandomSample[testData, 12]}, 
  Labeled[#1, 
     Column[{"   true:" <> #2, 
       "predict:" <> StringJoin[Values@trained[#1]]}], Top] & @@@ 
   Transpose[{Import /@ First /@ re[[All, "Image"]], 
     StringJoin /@ 
      Transpose[{re[[All, "digit1"]], re[[All, "digit2"]], 
        re[[All, "digit3"]], re[[All, "digit4"]]}]}]]

enter image description here

It seems that the network can predict most of the characters correct, with a few confusions about a "0" and an "O", which is understandable. We can have a more detailed look at the accuracy:

testPredict = trained[testData[[All, 1]]];

testlabels = 
  Transpose[Values /@ 
    testData[[All, {"digit1", "digit2", "digit3", "digit4"}]]];

The character level accuracy is about 95% which is quite good

Count[Flatten[Values@testPredict - testlabels], 0]/(4.*Length[testData])
(* 0.955664 *)

And the image level accuracy (get all four characters correct) is about 84%

Count[#1 == #2 & @@@ 
   Transpose[{Transpose[Values@testPredict], Transpose[testlabels]}], 
  True]/(1.*Length[testData])
(* 0.839648 *)

We see that most of these mistakes are due to confusion between "0"'s and "O"'s

Counts[Sort /@ 
   Select[Transpose[{Flatten[Values@testPredict], 
      Flatten[testlabels]}], #[[1]] != #[[2]] &]] // Short

(*
<|{0,O}->475,{D,O}->12,{I,T}->15,{N,W}->2,<<199>>,{B,C}->1,{G,S}->1,{F,P}->1,{H,L}->1|>
*)

Python code for generating CAPTCHA

from captcha.image import ImageCaptcha
import matplotlib.pyplot as plt
import string
import random
import numpy as np

def generate(width,height,num_char):
    characters = string.digits + string.ascii_uppercase
    rand_str = ''.join(random.sample(characters,num_char))
    generator = ImageCaptcha(width=width, height=height)
    image = generator.generate_image(rand_str)
    return (image,rand_str)


# images and labels will be in saved to save_dir folder
save_dir = '/Users/xslittlegrass/Downloads/captcha/'
labels = []
for i in range(4):
    img, char = generate(140,80,4)
    plt.imsave(save_dir+str(i)+".jpg",np.array(img))
    labels.append(char)
with open(save_dir + 'labels.txt','w') as f:
    for lb in labels:
        f.write('%s\n' % lb)

Reference

  1. A captcha library that generates audio and image CAPTCHAs, https://github.com/lepture/captcha/
  2. captcha_break, https://github.com/ypwhs/captcha_break
  3. Breaking captchas using torch, https://github.com/arunpatala/captcha
$\endgroup$
  • 3
    $\begingroup$ They said that they would restore GPU training support on OS X. $\endgroup$ – Szabolcs Apr 16 '17 at 9:07
  • 1
    $\begingroup$ @Szabolcs That's great news, I hope the update will come soon. Even for my 4-year-old 750M, training is 7X faster on GPU than on CPU. $\endgroup$ – xslittlegrass Apr 16 '17 at 13:28
  • 4
    $\begingroup$ Good job! Considering your experience, any sort of captcha can be trained and learned by a neural network (no matter the size of digits). The question now is how ensure web apps security as Deep Learning overpass the current techniques? Even if the captcha is generated by a neural network it could be learned by another one. So ... $\endgroup$ – Jean-Claude Apr 26 '17 at 11:17
  • $\begingroup$ What should the variable "classes" be? I'm trying your code, and it seems not to be defined. $\endgroup$ – Eric Brown Apr 29 '17 at 19:04
  • 1
    $\begingroup$ @EricBrown I just add the definition, thanks for pointing it out. $\endgroup$ – xslittlegrass Apr 29 '17 at 19:12

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.