# Automatic image recognition of Lissajous figures on oscilloscope

I have thousands of oscilloscope images showing Lissajous figures for example like below 4x5 image collection

Define a function that can generate various Lissajous figures

ClearAll[Lissajous];
Lissajous[freqRatio_, ampRatio_, θ_, opts : OptionsPattern[]] :=
ParametricPlot[{Sin[100 t],
ampRatio*Sin[freqRatio*100 t + θ]}, {t, 0, 2π}, opts,
AspectRatio -> Automatic, Frame -> True, PlotRange -> All]

The goal is to determine freqRatio, ampRatio, θ of each oscilloscope image. For example, the last row of previous image corresponds to

The images are taken rather sloppy, but should not be a big problem for recognition:

1. The backgrounds are dirty. But the oscilloscope traces are bright to the human eye.
2. The oscilloscope screens are often tilted, but the borders of the screen should be served as a good reference

I have no clue how to tackle this problem, but I believe this recognition problem can be solved by Mathematica. Could someone help give a solution?

Update:

I share three zips of separate oscilloscope images 1, 2, 3

To make the problem more easily handled. We can assume:

1. the freqRatio only take simple numbers like: 1,2,3,1/3,2/3...etc. That is umerator and denominator are no more than 3.
2. the θ only take 0,π/4,π/2,3π/4,π
• Is it fair to assume that all three parameters (freqRatio, ampRatio, θ) are "nice" numbers, ie. small rational numbers or rational multiplies of $\pi$? Also, can you please provide a few separate images of screen in the original resolution? Feb 5, 2023 at 11:52
• @Domen Thank you very much for your comment. Very good suggestion. I made edits to my post. Please take a look. Feb 5, 2023 at 16:16
• You could probably also do this with Classify or LeNet Feb 6, 2023 at 3:38
• @VitaliyKaurov That should need a lot of tagged data for training right? Feb 7, 2023 at 4:23
• @matheorem, I was also playing a bit with the idea of using neural nets. You don't need tagged data, you can generate them yourself (the same way as in my answer), then use data augmentation (adding noise, slight deformations ...). Feb 7, 2023 at 8:04

This is not a particularly efficient solution, but it works since you have a very limited range of possible parameters, and the screen images are not too distorted. It also determines only freqRatio and θ, but not the ampRatio, since this would require some calibration.

The main idea is to pre-generate images of all possible Lissajous curves, then correlate preprocessed screen images with all curves and pick the one with the highest correlation.

## 1. Generate all possible Lissajous curves

Clear[Lissajous];

Lissajous[freqRatio_, ampRatio_, θ_] :=
Table[{Sin[t], ampRatio*Sin[freqRatio* t + θ]}, {t, 0, 10 π/freqRatio, .2}]

(* Image size should be fairly low for better speed performance *)
imageSize = 100;

freqs = Flatten@Table[n/m, {m, 1, 3}, {n, 1, 3}] // DeleteDuplicates;
θs = {0, 1/4, 1/2, 3/4, 1};
amps = {-1, 1};

lissajousAll =
Catenate@Catenate@
Table[{ColorNegate@Binarize@
Rasterize[Graphics[{Thickness[.03], Line@Lissajous[freq, amp, θ Pi]}],
ImageSize -> {imageSize, imageSize}], freq, amp, θ},
{freq, freqs}, {θ, θs}, {amp, amps}];

Since I don't know how to generate only unique Lissajous curves, we can simply remove duplicated images.

(* Image distance threshold depends on imageSize,
and should be set manually so that all unique
curves are returned. *)
lissajous = DeleteDuplicates[lissajousAll, ImageDistance[First@#1, First@#2] < 20 &]

## 2. Preprocess screen images

Preprocessing takes multiple stages, including binarization of brightness channel with manually set thresholds, some morphological operations (thinning, dilation), and final cropping to a square image. The parameters and operations can be tuned if they perform poorly for other sets of images. Generally, the processed images should have distinct curve shapes without much speckles or holes.

origImg = Import["https://i.stack.imgur.com/n1NtY.jpg"];

(* Cutting the image into separate screens *)
imgs = ImagePartition[origImg, {385, 260}] // Flatten;

screens =
ImageCrop[
ImageResize[
ImageCrop[
Dilation[
DeleteSmallComponents[
Thinning[
DeleteBorderComponents@
MorphologicalBinarize[
Last@ColorSeparate[#, "HSB"], {.7, .99}], 20], 5],
3]], {imageSize, Automatic}], {imageSize, imageSize}] & /@ imgs

## 3. Find best match

Processed image is correlated with all possible Lissajous curves, and the one with the highest correlation is chosen. Note that for a meaningful correlation between binary images, they should be centered to zero (subtraction of 1/2).

findParameters[img_] :=
Drop[First@
ReverseSortBy[{Max[ImageCorrelate[#[[1]] - 1/2, img - 1/2]],
ColorNegate@#[[1]], #[[2]], #[[3]], #[[4]]} & /@ lissajous, First], 1]

fits = Table[findParameters[img], {img, screens}];

TableForm[fits,
TableHeadings -> {imgs, {"Best Lissajous", "freqRatio", "ampRatio", "θ"}},
TableAlignments -> Center]

• Thank you so much for your solution. Learned a lot. I found there are some drawbacks. The process of extracting oscilloscope curves is unstable. It relies on MorphologicalBinarize parameters and details around the screen for removing them. I tested other images, it fails badly. I add share more image data in the post. Feb 7, 2023 at 3:10
• I also do not understand ImageCorrelate. There is no detail in the documentation. Why should we subtract img by 1/2? What is more, since the screen is often tilted. I found sometimes it leads to similar but wrong result. I think adjust screen to rectangular is important for correct ImageCorrelate. I found Binarize@First@ColorNegate@ColorSeparate[img, "HSB"] can indicate screen shape clearly, but I do not know how to reshape the screen region to rectangular. Feb 7, 2023 at 3:10
• Well, if your other images are significantly different, the preproceessing has to be changed accordingly. Regarding ImageCorrelate for binary images: $0\times0=0$ and $0\times 1 = 1$, which means that correlation would compare only the white parts of the images (the curves) and not the black parts (the background). This can lead to false results, like Lissajous curve $(1/3, 1, 0)$ being matched to $(1/3, -1, \pi/4)$. If you subtract 1/2, then also the background is being compared. As for the tilt correction, I did not implement my method worked without it. There are a lot functions for ... Feb 7, 2023 at 8:45
• ... image processing and computer vision in Mathematica, which can help you in identifying the screen location and perform the neccessary transformation. See for example the first application under ImagePerspectiveTransformation. Feb 7, 2023 at 8:46
• Argh, obviously I meant to write $0\times1 = 0$ :) Feb 7, 2023 at 9:20