TAMPERE IMAGE DATABASE 2008 TID2008, version 1.0
TID2008 is intended for evaluation of full-reference image visual
quality assessment metrics. TID2008 allows estimating how a given
metric corresponds to mean human perception. For example, in
accordance with TID2008, Spearman correlation between the metric
PSNR (Peak Signal to Noise Ratio) and mean human perception (MOS,
Mean Opinion Score) is 0.525.
How to test your metric? Step 1. Download the TID2008. Step 2. Calculate values of your metric
for each distorted image of TID2008. Step 3. Use included software
("spearman.exe" or "kendall.exe") for calculate a correlation between your metric and MOS.
In case of publishing results obtained by means of TID2008 please
refer to one of the following papers:
(full description of TID2008)
[1] N. Ponomarenko, V. Lukin, A. Zelensky, K. Egiazarian, M. Carli,
F. Battisti, "TID2008 - A Database for Evaluation of Full-Reference
Visual Quality Assessment Metrics", Advances of Modern
Radioelectronics, Vol. 10, pp. 30-45, 2009.
(a short description of TID2008)
[2] N. Ponomarenko, F. Battisti, K. Egiazarian, J. Astola, V. Lukin
"Metrics performance comparison for color image database", Fourth international
workshop on video processing and quality metrics for consumer electronics,
Scottsdale, Arizona, USA. Jan. 14-16, 2009, 6 p.
WinRAR archive with TID2008 (~550 MB):
tid2008.rar
The TID2008 contains 25 reference images and 1700 distorted images
(25 reference images x 17 types of distortions x 4 levels of distortions).
Reference images are obtained by cropping from Kodak Lossless True
Color Image Suite.
All images are saved in database in Bitmap format without any compression.
File names are organized in such a manner that they indicate a number of
the reference image, then a number of distortion's type, and, finally, a
number of distortion's level: "iXX_YY_Z.bmp".
The file "mos.txt" contains the Mean Opinion Score for each distorted image.
The MOS was obtained from the results of 838 experiments carried out by
observers from three countries: Finland, Italy, and Ukraine (251 experiments
have been carried out in Finland, 150 in Italy, and 437 in Ukraine). Totally,
the 838 observers have performed 256428 comparisons of visual quality of
distorted images or 512856 evaluations of relative visual quality in image
pairs. Higer value of MOS (0 - minimal, 9 - maximal, MSE of each score
is 0.019) corresponds to higer visual quality of the image.
Ranking of compared metrics in accordance Ranking of compared metrics in accordance
with Spearman correlation with MOS with Kendall correlation with MOS
Rank Measure Spearman correlation Rank Measure Kendall correlation
1 MSSIM 0.853 1 MSSIM 0.654
2 SSIM 0.808 2 SSIM 0.605
3 VIF 0.750 3 VIF 0.586
4 VSNR 0.705 4 VSNR 0.534
5 VIFP 0.655 5 VIFP 0.495
6 NQM 0.624 6 PSNR-HVS 0.476
7 UQI 0.600 7 NQM 0.461
8 PSNR-HVS 0.594 8 PSNR-HVS-M 0.449
9 XYZ 0.577 9 UQI 0.435
10 IFC 0.569 10 XYZ 0.434
11 PSNR-HVS-M 0.559 11 IFC 0.426
12 PSNRY 0.553 12 PSNRY 0.402
13 PSNR 0.525 13 WSNR 0.393
14 MSE 0.525 14 LINLAB 0.381
15 SNR 0.523 15 SNR 0.374
16 WSNR 0.488 16 DCTUNE 0.372
17 LINLAB 0.487 17 PSNR 0.369
18 DCTUNE 0.476 18 MSE 0.369
Types of distortion used in TID2008
N Type of distortion
1 Additive Gaussian noise
2 Additive noise in color components is more intensive than additive noise in the luminance component
3 Spatially correlated noise
4 Masked noise
5 High frequency noise
6 Impulse noise
7 Quantization noise
8 Gaussian blur
9 Image denoising
10 JPEG compression
11 JPEG2000 compression
12 JPEG transmission errors
13 JPEG2000 transmission errors
14 Non eccentricity pattern noise
15 Local block-wise distortions of different intensity
16 Mean shift (intensity shift)
17 Contrast change
The following files contain values of some quality metrics calculated for
the TID2008 images:
"psnr.txt" - peak signal to noise ratio;
"psnry.txt" - peak signal to noise ratio calculated for the luminance component;
"snr.txt" - signal to noise ratio [3].
"mse.txt" - inverted values of mean square error [3].
"dctune.txt" - inverted values of the DCTune metric [4];
"uqi.txt" - values of the UQI metric [5];
"ssim.txt" - values of the SSIM metric [6];
"mssim.txt" - vaules of the MSSIM metric [7,3];
"linlab.txt" - inverted values of the LinLab metric [8];
"xyz" - inverted values of the YCxCz2XYZ metric [9];
"psnrhvs.txt" - values of the PSNR-HVS metric [10];
"psnrhvsm.txt" - values of the PSNR-HVS-M metric [11];
"vif.txt" - values of the VIF metric [12,3];
"vifp.txt" - pixel domain version VIF [12,3];
"nqm.txt" - values of the NQM metric [13,3];
"wsnr.txt" - values of the WSNR metric [14,3];
"ifc.txt" - values of the IFC metric [15,3];
"vsnr.txt" - values of the VSNR metric [16,3];
[3] Matthew Gaubatz, "Metrix MUX Visual Quality Assessment Package: MSE,
PSNR, SSIM, MSSIM, VSNR, VIF, VIFP, UQI, IFC, NQM, WSNR, SNR",
http://foulard.ece.cornell.edu/gaubatz/metrix_mux/
[4] A. B. Watson, "DCTune: A technique for visual optimization of DCT
quantization matrices for individual images," Soc. Inf. Display Dig.
Tech. Papers, vol. XXIV, pp. 946-949, 1993.
[5] Z. Wang, A. Bovik, "A universal image quality index", IEEE Signal
Processing Letters, vol. 9, pp. 81-84, March, 2002.
[6] Z. Wang, A. Bovik, H. Sheikh, E. Simoncelli, "Image quality assessment:
from error visibility to structural similarity", IEEE Transactions on
Image Proc., vol. 13, issue 4, pp. 600-612, April, 2004.
[7] Z. Wang, E. P. Simoncelli and A. C. Bovik, "Multi-scale structural
similarity for image quality assessment," Invited Paper, IEEE Asilomar
Conference on Signals, Systems and Computers, Nov. 2003.
[8] B. Kolpatzik and C. Bouman, "Optimized Error Diffusion for High Quality
Image Display", Journal Electronic Imaging, pp. 277-292, 1992.
[9] B. W. Kolpatzik and C. A. Bouman, "Optimized Universal Color Palette
Design for Error Diffusion", Journal Electronic Imaging, vol. 4,
pp. 131-143, 1995.
[10] K. Egiazarian, J. Astola, N. Ponomarenko, V. Lukin, F. Battisti,
M. Carli, "New full-reference quality metrics based on HVS", CD-ROM
Proceedings of the Second International Workshop on Video Processing
and Quality Metrics, Scottsdale, USA, 2006, 4 p.
[11] N. Ponomarenko, F. Silvestri, K. Egiazarian, M. Carli, J. Astola,
V. Lukin "On between-coefficient contrast masking of DCT basis
functions", CD-ROM Proc. of the Third International Workshop on Video
Processing and Quality Metrics. - USA, 2007. - 4 p.
[12] H.R. Sheikh.and A.C. Bovik, "Image information and visual quality,"
IEEE Transactions on Image Processing, Vol.15, no.2, 2006, pp. 430-444.
[13] Damera-Venkata N., Kite T., Geisler W., Evans B. and Bovik A. "Image
Quality Assessment Based on a Degradation Model", IEEE Trans. on Image
Processing, Vol. 9, 2000, pp. 636-650.
[14] T. Mitsa and K. Varkur, "Evaluation of contrast sensitivity functions
for the formulation of quality measures incorporated in halftoning
algorithms", ICASSP '93-V, pp. 301-304.
[15] H.R. Sheikh, A.C. Bovik and G. de Veciana, "An information fidelity
criterion for image quality assessment using natural scene statistics",
IEEE Transactions on Image Processing, vol.14, no.12, 2005, pp. 2117-2128.
[16] D.M. Chandler, S.S. Hemami, "VSNR: A Wavelet-Based Visual Signal-to-Noise
Ratio for Natural Images", IEEE Transactions on Image Processing,
Vol. 16 (9), pp. 2284-2298, 2007.
The programs "spearman.exe" and "kendall.exe" calculate values of Spearman
and Kendall rank correlations for entire set of the TID2008 images as well
as for particular subsets given in the following Table:
Subsets of TID2008 definded by default
N Type of distortion Noise Noise2 Safe Hard Simple Exotic Exotic2 Full
1 Additive Gaussian noise + + + - + - - +
2 Noise in color comp. - + - - - - - +
3 Spatially correl. noise + + + + - - - +
4 Masked noise - + - + - - - +
5 High frequency noise + + + - - - - +
6 Impulse noise + + + - - - - +
7 Quantization noise + + - + - - - +
8 Gaussian blur + + + + + - - +
9 Image denoising + - - + - - - +
10 JPEG compression - - + - + - - +
11 JPEG2000 compression - - + - + - - +
12 JPEG transm. errors - - - + - - + +
13 JPEG2000 transm. errors - - - + - - + +
14 Non ecc. patt. noise - - - + - + + +
15 Local block-wise dist. - - - - - + + +
16 Mean shift - - - - - + + +
17 Contrast change - - - - - + + +
The command line is "spearman data1 data2" or "kendall data1 data2".
Command line examples:
spearman mos.txt ssim.txt
kendall mos.txt dctune.txt
spearman linlab.txt xyz.txt
kendall psnr.txt psnr-hvs.txt
An example of usage:
kendall.exe mos.txt uqi.txt
Noise : 0.363
Noise2 : 0.419
Safe : 0.454
Hard : 0.568
Simple : 0.586
Exotic : 0.214
Exotic2: 0.405
Full : 0.438
We plan to regularly update the versions of this database. New versions
will include new types of distortion and take into account results of
additional experiments.
We will highly appreciate authors of other metrics if they will inform
us (please, mail to karen@cs.tut.fi or nikolay@ponomarenko.info) how to get
executable files (e.g., Matlab codes) of their metrics. We guarantee
that we will not pass them to other users and will include future
results obtained for such metrics in analysis for our database.
Back to main page...
Last changed: 2010-02-22.