Here is one example using a convolutional neural network (CNN) to crack the CAPTCHA. We will use a CAPTCHA library to generate sample CAPTCHA images and then train a neural network to decode these images. The trained network can be downloaded from crack_CAPTCHA.wlnet.
In the following, we will describe how to prepare the training data, construct and train the CNN.
Data preparation
The CAPTCHA library we will use to generate the images is Captcha. Using this library, We generate about 50K images and use them as our dataset. For simplicity, we constrain ourselves to images with exactly four characters. The code for generating the image will be included at the end of this post. Here are some sample images from the dataset
A few more sample images can be downloaded from here.
After generation our dataset, we can load them into Mathematica.
This is the data directory
dir = "/Users/xslittlegrass/Downloads/captcha/";
The labels of all the images in the dataset
labels =
StringSplit[Import[dir <> "labels.txt", "String"], "\n"];
and the total number of samples
nsample = Length[labels]
(* 51200 *)
Load the images as File object
We load the images as File objects, this allows out-of-core training.
fileNameIndex = Sort[FromDigits /@ FileBaseName /@ FileNames["*.jpg", dir]];
imagefiles = Table[File[dir <> ToString[i] <> ".jpg"], {i, fileNameIndex}];
The image has 170X80 dimensions
imgdim = Import[imagefiles[[1, 1]]] // ImageDimensions
(* {170, 80} *)
We use 90% of these sample as training data and 10% as test data
data = <|"Image" -> #1, "digit1" -> StringTake[#2, {1}],
"digit2" -> StringTake[#2, {2}], "digit3" -> StringTake[#2, {3}],
"digit4" -> StringTake[#2, {4}]|> & @@@
Transpose[{imagefiles, labels}];
trainingData = data[[1 ;; Round[0.9*Length[data]]]];
testData = Complement[data, trainingData];
Construct neural network
The neural network contains two parts, the feature extraction network and the classification network. For the feature extraction network, we use a simple CNN resembles the VGG network.
convnet = NetChain[{
ConvolutionLayer[32, {3, 3}],
ElementwiseLayer[Ramp],
ConvolutionLayer[32, {3, 3}],
ElementwiseLayer[Ramp],
PoolingLayer[{2, 2}, {2, 2}],
ConvolutionLayer[64, {3, 3}],
ElementwiseLayer[Ramp],
ConvolutionLayer[64, {3, 3}],
ElementwiseLayer[Ramp],
PoolingLayer[{2, 2}, {2, 2}],
ConvolutionLayer[128, {3, 3}],
ElementwiseLayer[Ramp],
ConvolutionLayer[128, {3, 3}],
ElementwiseLayer[Ramp],
PoolingLayer[{2, 2}, {2, 2}],
ConvolutionLayer[256, {3, 3}],
ElementwiseLayer[Ramp],
ConvolutionLayer[256, {3, 3}],
ElementwiseLayer[Ramp],
PoolingLayer[{2, 2}, {2, 2}],
FlattenLayer[],
DropoutLayer[0.2]
}, "Input" -> NetEncoder[{"Image", imgdim}]
]
For the classification network, we use 4 independent linear layers for each of our four characters
classes = Join[ToString /@ Range[0, 9], ToUpperCase@Alphabet[]];
net = NetGraph[{convnet, 36, SoftmaxLayer[], 36, SoftmaxLayer[], 36,
SoftmaxLayer[], 36,
SoftmaxLayer[]}, {NetPort["Image"] ->
1 -> 2 -> 3 -> NetPort["digit1"],
1 -> 4 -> 5 -> NetPort["digit2"], 1 -> 6 -> 7 -> NetPort["digit3"],
1 -> 8 -> 9 -> NetPort["digit4"]},
"digit1" -> NetDecoder[{"Class", classes}],
"digit2" -> NetDecoder[{"Class", classes}],
"digit3" -> NetDecoder[{"Class", classes}],
"digit4" -> NetDecoder[{"Class", classes}]]
Training
We use the default setting for the hyper-parameters and train 3 rounds. The total training time is about 2.5 hours on the Nvidia 750M in use. Since Mathematica 11.1 doesn't support GPU training on Mac, the network is trained in Mathematica version 11.0.
trained =
NetTrain[net, trainingData, ValidationSet -> testData,
MaxTrainingRounds -> 3, TargetDevice -> "GPU"];
The trained network can be downloaded from here
Testing
After 2.5 hours of training, the network seems to converge well. Here are some test on the samples from the test set
Grid@Partition[#, 4] &@
With[{re = RandomSample[testData, 12]},
Labeled[#1,
Column[{" true:" <> #2,
"predict:" <> StringJoin[Values@trained[#1]]}], Top] & @@@
Transpose[{Import /@ First /@ re[[All, "Image"]],
StringJoin /@
Transpose[{re[[All, "digit1"]], re[[All, "digit2"]],
re[[All, "digit3"]], re[[All, "digit4"]]}]}]]
It seems that the network can predict most of the characters correct, with a few confusions about a "0" and an "O", which is understandable. We can have a more detailed look at the accuracy:
testPredict = trained[testData[[All, 1]]];
testlabels =
Transpose[Values /@
testData[[All, {"digit1", "digit2", "digit3", "digit4"}]]];
The character level accuracy is about 95% which is quite good
Count[Flatten[Values@testPredict - testlabels], 0]/(4.*Length[testData])
(* 0.955664 *)
And the image level accuracy (get all four characters correct) is about 84%
Count[#1 == #2 & @@@
Transpose[{Transpose[Values@testPredict], Transpose[testlabels]}],
True]/(1.*Length[testData])
(* 0.839648 *)
We see that most of these mistakes are due to confusion between "0"'s and "O"'s
Counts[Sort /@
Select[Transpose[{Flatten[Values@testPredict],
Flatten[testlabels]}], #[[1]] != #[[2]] &]] // Short
(*
<|{0,O}->475,{D,O}->12,{I,T}->15,{N,W}->2,<<199>>,{B,C}->1,{G,S}->1,{F,P}->1,{H,L}->1|>
*)
Python code for generating CAPTCHA
from captcha.image import ImageCaptcha
import matplotlib.pyplot as plt
import string
import random
import numpy as np
def generate(width,height,num_char):
characters = string.digits + string.ascii_uppercase
rand_str = ''.join(random.sample(characters,num_char))
generator = ImageCaptcha(width=width, height=height)
image = generator.generate_image(rand_str)
return (image,rand_str)
# images and labels will be in saved to save_dir folder
save_dir = '/Users/xslittlegrass/Downloads/captcha/'
labels = []
for i in range(4):
img, char = generate(140,80,4)
plt.imsave(save_dir+str(i)+".jpg",np.array(img))
labels.append(char)
with open(save_dir + 'labels.txt','w') as f:
for lb in labels:
f.write('%s\n' % lb)
Reference
- A captcha library that generates audio and image CAPTCHAs, https://github.com/lepture/captcha/
- captcha_break, https://github.com/ypwhs/captcha_break
- Breaking captchas using torch, https://github.com/arunpatala/captcha
