Theoretical concepts of watermark manipulation (part of version 3.2 of the “C + Executables” version of “Release That Watermark!”)

Download link for the programs: https://www.mediafire.com/folder/qkxpmudgvszm4/ReleaseThatWatermark!

Preamble

This document’s purpose is to give an overview of the basic concepts behind translucent watermarks, their manipulation and their detection.

I – How are watermarks added to images?
The most common digital images are defined as a set of pixels. In the case of coloured images, pixels are made of three values, representing the colours red, green and blue (also known as RGB). In the case of 8-bit images, there are 256 possible values (from 0 to 255) for each colour of each pixel.

Some image formats (like PNG) also support transparency. It is just an additional value of the pixel, as if it were a fourth colour. So the pixels of an 8-bit coloured image with transparency are made of four values each: red, green, blue and opacity (also known as RGBA, A standing for “alpha channel”), each value being in the 0 to 255 range.

Translucent watermarks are RGBA images.

The watermarked images, however, are usually RGB images. They are made by mathematically “mixing” the values of the watermark’s pixels with the values of the image’s pixels, using the opacity to know how much of each value is used.

So if a colour of a pixel in an 8-bit image has the value X before watermarking, and Y after watermarking, and if the same colour of the matching pixel of the 8-bit watermark has the value W, and if this pixel’s transparency is A, then we have the following equality:

Y=(1-A/255)X+(A/255)W

Keep in mind that A/255 and (1-A/255) are both values between 0 and 1. This insures that the pixel value remains between 0 and 255, and allows the watermark to be visible on both bright and dark backgrounds.

However, when the watermarked images are saved in the JPEG format, the compression process results in a loss of quality. Therefore, the values change slightly, and the above formula is no more exact. Still, it remains approximate.

II – How is it possible to extract watermarks from samples?

Since the pixels of a watermark are made of colours and an opacity, extracting watermark means finding these values. This document will give a one-sample method, a two-sample method, and a statistical multi-sample method.

1) Using one sample with a known background (partial extraction)

a) Generic extraction method

With only one sample, provided the background is perfectly known, the watermark above can be partially extracted. The watermark cannot be fully extracted, since some pixels may be identical to the background in the watermarked image even when they are an actual mix of the background and the watermark. There is no way to tell these troublesome pixels apart and to interpret them correctly with just one sample.

Colours and opacity can be calculated when a given pixel is different from the original image’s pixel:

(RY,GY,BY) = (1-A/255)(RX,GX,BX)+(A/255)(RW,GW,BW)

Which is equivalent to:

(RY,GY,BY) – (RX,GX,BX) = (A/255)((RW,GW,BW) – (RX,GX,BX))

To express A directly, it is convenient to use a norm on the colour space, for instance the euclidean norm:

||(R,G,B)||=SquareRoot(R²+G²+B²)

Mathematical discussion: Although the euclidean norm is an obvious candidate, the colour space is not geometric, so the right choice is not necessarily the obvious one. Current experiments suggest that the extraction process is a bit more efficient with this norm than it is with ||(R,G,B)||=|R|+|G|+|B|. Finding the best norm for this extraction method is currently an open question.

For the pixels of the extracted watermark to remain in the 8-bit range, it is convenient to calculate the maximum possible distance (with the meaning given by the chosen norm) that a pixel can have to the background in the RGB colour space. For 8-bit values, if RX>127.5, then the farthest value is 0, and if RX<127.5, then the farthest value is 255. We can name RF this farthest value in the red range, and define similarly GF and BF in the green and blue range. If we name DF the distance to the farthest pixel, then we have:

DF=||(RF,GF,BF) – (RX,GX,BX)||

The norm can be used with the earlier equalities, and we can write:

||(RY,GY,BY) – (RX,GX,BX)|| = (A/255)||(RW,GW,BW) – (RX,GX,BX)||

A can be defined as the following:

A=255||(RY,GY,BY) – (RX,GX,BX)||/DF

Without going through all the details, this definition insures the various values remain in the valid ranges. Nonetheless, it is worth saying that this is just one possible definition. Finding the best definition for A is still an open question too.

If A is equal to 0 (full transparency), RW, GW and BW can be set to any value. If it is not, then it is possible to calculate RW, GW and BW like this:

RW=RX+(255/A)(RY-RX)

GW=GX+(255/A)(GY-GX)

BW=BX+(255/A)(BY-BX)

Thus, a single sample makes the extraction of a watermark at least partially possible. This is useful when good samples with perfectly known backgrounds are especially hard to find.

Nevertheless, even with a perfect sample, the quality of watermarks extracted with this method cannot compete with the two-sample extraction, provided the second samples has a sufficient quality too.

Moreover, two mathematical questions (the choice of the norm and the choice of A) might leave room for optimisation of the one-sample extraction method.

b) Alternate method for black background

If the background colour happens to be black, then the background colour part of the watermarking equation conveniently becomes equal to zero, resulting in a simpler equation:

(RY,GY,BY) = (A/255)(RW,GW,BW)

We can choose A so it is minimal, resulting in high transparency and high colour intensity. Once again, this is not the only possible definition for A. It is convenient to define the maximum of RY, GY and BY:

M=max(RY,GY,BY)

If M=0, then it means A must be set to 0 and RW, GW and BW can be set to any values.

If M is not equal to zero, then it is possible to define A, RW, GW and BW as follows:

A=M

RW=(255/M)RY

GW=(255/M)GY

BW=(255/M)BY

This method is thus another example of the one-sample extraction can be made differently simply by choosing a different definition for A. Although this ambiguity makes it harder to extract watermarks, it also leaves room for other uses of the one-sample extraction technique, such as creating an alpha layer to make other kinds of images translucent (for example logos). This particular extraction method is good at making bright colours especially vivid and dark colours especially translucent. It also conserves the hue. It can also be used for dark backgrounds that are only approximately black.

c) Alternate method for white background

Similarly, if the background is white, it is possible to define A differently to perform an extraction. The watermarking equation is the following:

(RY,GY,BY) = (1-A/255)(255,255,255)+(A/255)(RW,GW,BW)

It is convenient to re-write it to work on the “negative image” instead. For that purpose, let us define:

NRY=255-RY

NGY=255-GY

NBY=255-BY

NRW=255-RW

NGW=255-GW

NBW=255-BW

The equation first becomes:

(255-NRY,255-NGY,255-BY)=(1-A/255)(255,255,255)+(A/255)(255-NRW,255-NGW,255-NBW)

which can be expanded:

(255,255,255)-(NRY,NGY,NBY)=(255,255,255)-(A,A,A)+(A,A,A)-(A/255)(NRW,NGW,NBW)

and simplified nicely:

(NRY,NGY,NBY)=(A/255)(NRW,NGW,NBW)

We can thus use the black-background extraction method on the negative image by first looking for the maximum value of NRY, NGY, and NBY:

M=max(NRY,NGY,NBY)

If M=0, then A must be set to 0 and RW, GW and BW can be set to any values.

If M is not equal to zero, then A, NRW, NGW and NBW can be defined as:

A=M

NRW=(255/M)NRY

NGW=(255/M)NGY

NBW=(255/M)NBY

So:

A=M

RW=255-(255/M)(255-RY)

GW=255-(255/M)(255-GY)

BW=255-(255/M)(255-BY)

Once again, this is only one possible definition, making watermark extraction harder, but with other possible uses in making images translucent.

d) Alternate method to extract the middle range of luminance

This method is a modification of the method a). Its purpose is to make both very bright pixels and very dark pixels transparent, while keeping the most opacity in the middle range of luminance.

If a “virtual background colour” is set to (127.5, 127.5, 127.5), if DF is defined as DF=127.5, and if A is defined as A=255(1-||(RY,GY,BY) – (RX,GX,BX)||/DF), then the same equations as in a) can be used to define RW, GW and BW:

RW=RX+(255/A)(RY-RX)

GW=GX+(255/A)(GY-GX)

BW=BX+(255/A)(BY-BX)

This method is definitely not an all-purpose method for watermark extraction, but it can be convenient to make an image translucent for other uses.

2) Using two samples with known different backgrounds (full extraction)

If two samples can be found, and if each pixel of one background is different from the matching pixel of the other background, and if they are perfectly known, then it is possible to extract the troublesome pixels that could not be extracted from one sample. Indeed, they can be told apart from one sample’s background by comparing to the matching pixel on the other sample.

This time, the situation can be represented by two equalities:

(RY1,GY1,BY1) = (1-A/255)(RX1,GX1,BX1) + (A/255)(RW,GW,BW)

(RY2,GY2,BY2) = (1-A/255)(RX2,GX2,BX2) + (A/255)(RW,GW,BW)

The subtraction of these equalities conveniently allows the calculation of A without worrying about W yet:

(RY2-RY1,GY2-GY1,BY2-BY1) = (1-A/255)(RX2-RX1,GX2-GX1,BX2-BX1)

To express A directly, it is convenient to use a norm on the colour space, for instance the norm:

||(R,G,B)||=|R|+|G|+|B|

Mathematical discussion: As in the other extraction method, the euclidean norm is also a possible choice. The maximum of R, G and B is also a possible choice. Once again, the best definition for the norm is currently still an open question.

Then, it is possible to write:

||(RY2,GY2,BY2)-(RY1,GY1,BY1)||=(1-A/255)||(RX2,GX2,BX2)-(RX1,GX1,BX1)||

So:

(1-A/255)=||(RY2,GY2,BY2)-(RY1,GY1,BY1)||/||(RX2,GX2,BX2)-(RX1,GX1,BX1)||

In the end:

A=255(1-||(RY2,GY2,BY2)-(RY1,GY1,BY1)||/||(RX2,GX2,BX2)-(RX1,GX1,BX1)||)

This way of defining A, rather than using an equality on one of the components like RY2-RY1= (1-A/255)(RX2-RX1), avoids the edge cases where A can not be calculated because RX2=RX1 . It allows support for a larger type of backgrounds, since it is enough that for each pixel, at least one colour doesn’t have the same value in the two backgrounds, but not necessarily all the three colours, and not necessarily the same colour for each pixel.

However, one thing is known for sure: if the two backgrounds have at least one pixel with the same exact three colours, it will crash the program.

A workaround could be to define a specific value for A in this edge case (with the one-sample method or an arbitrary value). Aside from satisfying a dire need to use “almost usable” samples because usable ones are too scarce, this feature is not very interesting, and might confuse users if a sample they thought good leads to bad quality of extraction if it does not even send them a warning. Therefore, it might be appropriate to either implement a warning system, or to lock this feature inside of a “low-quality-background-tolerant mode”, or to do both.

With the value of A calculated, one can rewrite one of the two equalities representing the situation (both will give the same result). The second equality, for instance, can be rewritten as:

(RY2,GY2,BY2) + ((A/255)-1)(RX2,GX2,BX2) = (A/255)(RW,GW,BW)

and then as:

(RW,GW,BW)=(255/A)(RY2,GY2,BY2)+(1-255/A)(RX2,GX2,BX2)

So in the end:

RW=(255/A)RY2+(1-255/A)RX2

GW=(255/A)GY2+(1-255/A)GX2

BW=(255/A)BY2+(1-255/A)BX2

This allows the extraction of all three colours and the opacity of the watermark, with excellent results, provided excellent samples are available. The choice of the norm is still an open question (perhaps the results do not even depend on this choice), but the results are satisfying enough for this concern to be mostly theoretical.

3) Using statistics on a set of samples (approximate extraction)

The one-sample and the two-sample extractions have one common requirement: the availability of watermark samples on a perfectly known background. It sometimes happens that such perfect samples are hard to find, but the watermark is available on other backgrounds that cannot be known completely.

It is possible, at each given position, to calculate statistical properties of the samples’ pixels at this position.

It is important here not to confuse these “sample statistics” (analysis of the pixels at one position in all the samples), with the “spatial statistics” (analysis of the pixels in one sample at all the positions). This extraction method only relies on “sample statistics”, and not on “spatial statistics” at all.

The statistical average of a colour of a given pixel in the set of N samples is defined as:

Av(C) = (C1 + C2 + … + CN)/N

The statistical standard deviation will be defined as:

StDev(C) = SquareRoot( (C1-Av(C))² + (CN-Av(C))² +…+ (CN-Av(C))² )

For convenience, it is useful to also have such definitions for three-colour pixels:

Av((R,G,B)) = ( Av(R) , Av(G) , Av(B) )

StDev((R,G,B)) = ( StDev(R) , StDev(G) , StDev(B) )

Once again, keep in mind we are neither comparing pixels inside of one sample, nor colours inside of one pixel. Theses statistics compare a pixel at a given position in all the samples.

The use of a norm on the RGB colour space is useful too. The best choice (if any) is currently not determined, be it experimentally or theoretically.

Two simple candidates are the following: ||R,G,B||=|R|+|G|+|B| and ||R,G,B||=SquareRoot(R²+G²+B²)

Back to our samples, if a same pattern is common at the same position in several samples, then the statistical properties of the pixels may be very different from other pixels. Such a behaviour allows to more or less extract the watermark.

In the set of samples, the pixels at a given position share the same watermark values and opacity, on different backgrounds. Thus, if (RX,GX,BX) and (RY,GY,BY) are the values of a pixel before and after watermarking, and A and (RW,GW,BW) the opacity and colour values of the watermark, then for each pixel at a given position:

(RY,GY,BY) = (1-A)(RX,GX,BX)+A(RW,GW,BW)

Hence:

Av((RY,GY,BY)) = (1-A)Av((RX,GX,BX))+A(RW,GW,BW)

StDev((RY,GY,BY)) = (1-A)StDev((RX,GX,BX))

If we name NSDX = ||StDev((RX,GX,BX))|| and NSDY = ||StDev((RY,GY,BY))||, then for a watermarked pixel we have:

A = 1- (NSDY / NSDX)

It can also be written as:

A = (NSDX- NSDY) / NSDX

Hence, if we also name AvX=Av((RX,GX,BX) and AvY=Av((RY,GY,BY):

(RW,GW,BW) = ( NSDX AvY – NSDY AvX ) / (NSDX – NSDY)

However, if we can measure the (RY,GY,BY) values, we cannot do the same for (RX,GX,BX). Thus, AvX and NSDX will be manually set as parameters to ensure a good watermark extraction.

AvX and NSDX can be understood as a model of the background’s statistical properties.

NSDX directly controls the opacity of the extracted watermark. Any position with NSDY being superior or equal to the NSDX can be considered not watermarked.

AvX then controls the luminance of the extracted watermark.

Thus, an approximate watermark can be extracted by using the statistical properties of a set of samples, even is none of these sample can be used for other methods of extraction.

Nevertheless, it is important to remember that statistics are not magic. Extracting a watermark with a satisfying quality may require a large set of samples, but even in that case the result is approximate. That is why before performing the statistical extraction, the set of samples should be skimmed to find potentially good samples for a two-sample full extraction.

III – How to remove a watermark from an image?

With an extracted watermark the original image’s pixels can be restored. It once again relies on the “mixing” formula:

(RY,GY,BY) = (1-A/255)(RX,GX,BX)+(A/255)(RW,GW,BW)

If it is rewritten, it will look like this:

(RX,GX,BX)=((RY,GY,BY)-(A/255)(RW,GW,BW))/(1-A/255)

So:

RX=(RY-(A/255)RW)/(1-A/255)

GX=(GY-(A/255)GW)/(1-A/255)

BX=(BY-(A/255)BW)/(1-A/255)

Is it possible that the result won’t be absolutely perfect. Since digital images need integer values, some ratios be rounded to the nearest integer during the calculations. But the real deal are the quality losses that happened if the image was ever saved in the JPEG format. Perfect watermark removal would require a way to deal with these losses. But although restoring compression losses is a very interesting question and might lead to great progress in information technology, it is well beyond the scope of this document.

Further improvement of image quality can be made by “denoising”, using tools such as Waifu2X. Denoising can reduce a little more the remains of watermarks.

IV – How to auto-detect watermarks (experimental)

Un-watermarking an image requires a correct alignment of the un-watermarking filter with the watermark on the image. Auto-detection can allow to reduce the user’s workload in finding the position parameters for un-watermarking, but requires much more computational power than a manual search.

A simple detection technique can rely on colour ranges: watermarked pixels can not have any value, because the watermarking process restricts the possible range.

Let us start with the equality representing the watermarking process for an 8-bit image:

(RY,GY,BY) = (1-A/255)(RX,GX,BX)+(A/255)(RW,GW,BW)

For the purpose of this explanation, let us first have a look only at the red components. We do not know the value of RX before un-watermarking. However, we know that RX is still in the [0, 255] range. Thus, we have the two following inequalities:

(1-A/255)0 + (A/255)RW <= RY

RY <= (1-A/255)255+(A/255)RW

If we rewrite the first inequality and merge it with the second one, then we get the following double inequality:

(A/255)RW <= RY <= (1-A/255)255+(A/255)RW

To make an interesting use of the above double inequality, it is important to notice two properties:

0 <= (A/255)RW

(1-A/255)255+(A/255)RW <= 255

So the double inequality defines the range of possible values for the red component of a watermarked pixel, this range is included in the [0, 255] range, and if A is greater than 0, then this range is smaller than [0, 255].

The green and the blue colour components will give similar relationships.

To sum up:

A pixel Y of the image is possibly watermarked by a pixel W of the watermark if:

RY is in the range [(A/255)RW, (1-A/255)255+(A/255)RW]

and GY is in the range [(A/255)GW, (1-A/255)255+(A/255)GW]

and BY is in the range [(A/255)BW, (1-A/255)255+(A/255)BW]

Thus, it is possible to perform pixel-per-pixel tests to check if pixels of the image match the colour ranges defined by pixels of the watermark.

This technique is successful to detect full-quality watermarks in a full-quality image.
However, a complex system of tolerance to invalid values is required if the watermark and/or the image do not have full quality (typically because of JPEG compression). Thus, a generic watermark detector suited for real-world examples is currently still a very difficult technical challenge.

Moreover, testing every possible position for the watermark requires a significant amount of computational power, scaling up very fast with the sizes of the image and the watermark.

To sum up, generic auto-detection of watermarks is very slow, and it is likely to fail if you do not have the original full-quality watermark and image.

Human experience is currently more cost-effective tool to find the correct position parameters for un-watermarking.

Conclusion

The current methods in this document provide a way to remove most of a translucent watermark (provided nice samples are available for watermark extraction), or at least to make it less visible. Thus, they can provide a noticeable increase in quality, that can be deemed either satisfying as-is or at least helpful to ease a manual redrawing.

Some experimental techniques are also explained, although they are currently not recommended in practice.

Afterword

Other methods might be researched in the future to extract watermarks differently. Spectral analysis could be a path to explore, and it could get along well with statistical extraction.

AI-based watermark removal with machine learning might be possible.

Non-translucent (completely opaque) watermarks and other writings, shapes and symbols, cannot be removed by the method explained in this document. The pixels they overwrote need to be redrawn manually. That’s why the methods here are more likely to slightly upgrade redrawers’ toolboxes rather than to reduce the need and the praise for their skills and creativity.