A Revisit on Image Files, Filetypes, Filesizes, Duplicates, and Revisions

# A Revisit on Image Files, Filetypes, Filesizes, Duplicates, and Revisions Hello, Mikaeri here. Let's go ahead and talk about this topic, since I understand that there's considerable confusion over what makes an acceptable vs. unacceptable identical image to upload, whether that be a duplicate, revision, etc... I'll try to make this as simple as possible by simply going over the guidelines outlined in the [Parent and Child Relationships wikipage](https://danbooru.donmai.us/wiki_pages/21859) about parenting identical images and give reasons as to why they're ordered as they are. But before that, we should really outline some things. ## Filetypes Images come in a large variety of filetypes, but the ones we're going to talk about mainly are JPG and PNG. You won't need to care about other file formats like GIF or BMP as they're fairly rare, especially in the case of the latter (which you should skip uploading altogether or convert if you really have to). ### JPG JPG is a lossy compression algorithm and raster file format for image files. Its strength is in providing an *accurate* representation of an image rich with color information in an acceptable storage size; this includes things like photographs and drawings. Most images encoded in JPG use the YUV colorspace; they don't strictly encode bit values per pixel like PNG might. Instead, the image is *sampled* and converted into that colorspace from its' raw encoding. That's done through a mathematical transformation where values are stored as coefficients of a luma component (Y') and two chroma components (Cb', Cr'). If you want to read more about this, feel free to visit [this question on StackExchange](https://photo.stackexchange.com/questions/8310/how-is-the-ycbcr-color-space-represented-in-a-jpeg-image). But all you need to know that it is not the original image; it's just an accurate representation of it, which varies depending on % compression quality, chroma subsampling and other export settings. JPG does not support alpha transparency -- this is something important I'll mention later (as it has to do with a very specific site). Regarding metadata, most JPG files embed [EXIF metadata](https://en.wikipedia.org/wiki/Exif). This isn't really important for drawings though. But oftentimes it is embed nonetheless by various image editors. The main reason why JPG is a preferred file format for images is because its *web-friendly*. Its takeaway is that you can easily share a JPG image online within reason; even at the highest compression quality you're still only ever going to see small dimples of image variance compared to a PNG export or the raw project file itself (PSD, KRA, GIMP, etc). ### PNG PNG is a lossless raster file format that supports lossless compression. As PNG is lossless, images in PNG files are bit-for-bit accurate representations of the original image (as long as, well, they haven't been downscaled by the artist purposefully). PNG supports a variety of different colorspaces primarily using the archetypal RGB color model, the two of which you'll be seeing most is RGB24 or RGBA32. RGB24 supports 8 bits per channel (hence the 24), but does not support alpha transparency. RGBA32, however, does (as an extra 8 bits is reserved for the alpha channel per pixel). The conversion from the RGB24 color space to the RGBA32 color space is *lossless* as all that happens is the addition of a pretty much empty alpha layer to the whole image. The opposite is not true, however; a conversion from RGBA32 to RGB24 is lossy as all alpha information is lost. This is sort of misleading though, since an RGBA32 image that doesn't encode any *meaningful* alpha information doesn't necessarily lose anything in its conversion to RGB24. PNG doesn't necessarily support EXIF metadata; instead, blocks can be assigned with information, analogous to a key-value table. This explains the rather huge variance in image metadata from file to file, as many editors will insert basically their own format of metadata into such a field. You can read more about how they're assigned [here](http://dev.exiv2.org/projects/exiv2/wiki/The_Metadata_in_PNG_files), but rest assured you don't need to know all that (unless, of course, you're curious). In unique circumstances such metadata can be fairly cumbersome, but most PNG metadata will fit into storage space that's usually no more than 1-2kb in size, aside from if an IEND chunk happens to be corrupted. But, as a result of this "freedom" of metadata, PNG images can encode a LOT of info. Here's an example. This is metadata extracted from [id 2845594](https://danbooru.donmai.us/posts/2845594): ``` File Name : id_2845594.png Directory : . File Size : 334 kB File Modification Date/Time : 2017:09:12 14:19:23-07:00 File Access Date/Time : 2017:09:12 14:19:23-07:00 File Creation Date/Time : 2017:09:12 14:19:23-07:00 File Permissions : rw-rw-rw- File Type : PNG File Type Extension : png MIME Type : image/png Image Width : 834 Image Height : 1000 Bit Depth : 8 Color Type : RGB Compression : Deflate/Inflate Filter : Adaptive Interlace : Noninterlaced Pixels Per Unit X : 11811 Pixels Per Unit Y : 11811 Pixel Units : meters Profile Name : Photoshop ICC profile Profile CMM Type : Lino Profile Version : 2.1.0 Profile Class : Display Device Profile Color Space Data : RGB Profile Connection Space : XYZ Profile Date Time : 1998:02:09 06:49:00 Profile File Signature : acsp Primary Platform : Microsoft Corporation CMM Flags : Not Embedded, Independent Device Manufacturer : IEC Device Model : sRGB Device Attributes : Reflective, Glossy, Positive, Color Rendering Intent : Perceptual Connection Space Illuminant : 0.9642 1 0.82491 Profile Creator : HP Profile ID : 0 Profile Copyright : Copyright (c) 1998 Hewlett-Packard Company Profile Description : sRGB IEC61966-2.1 Media White Point : 0.95045 1 1.08905 Media Black Point : 0 0 0 Red Matrix Column : 0.43607 0.22249 0.01392 Green Matrix Column : 0.38515 0.71687 0.09708 Blue Matrix Column : 0.14307 0.06061 0.7141 Device Mfg Desc : IEC http://www.iec.ch Device Model Desc : IEC 61966-2.1 Default RGB colour space - sRGB Viewing Cond Desc : Reference Viewing Condition in IEC61966-2.1 Viewing Cond Illuminant : 19.6445 20.3718 16.8089 Viewing Cond Surround : 3.92889 4.07439 3.36179 Viewing Cond Illuminant Type : D50 Luminance : 76.03647 80 87.12462 Measurement Observer : CIE 1931 Measurement Backing : 0 0 0 Measurement Geometry : Unknown Measurement Flare : 0.999% Measurement Illuminant : D65 Technology : Cathode Ray Tube Display Red Tone Reproduction Curve : (Binary data 2060 bytes, use -b option to extract) Green Tone Reproduction Curve : (Binary data 2060 bytes, use -b option to extract) Blue Tone Reproduction Curve : (Binary data 2060 bytes, use -b option to extract) White Point X : 0.31269 White Point Y : 0.32899 Red X : 0.63999 Red Y : 0.33001 Green X : 0.3 Green Y : 0.6 Blue X : 0.15 Blue Y : 0.05999 Image Size : 834x1000 Megapixels : 0.834 ``` Talk about *insane*! There's a variety of stuff that matters and stuff that doesn't, depending on who you talk to. One of the things someone might care about is the [ICC color profile](https://en.wikipedia.org/wiki/ICC_profile), which basically describes how an image's color should be described to a color input or output device. Other parts someone might also care about is the software used, the encoder used, or even the thumbnail embedded in the image. But to put it shortly, this information is non-trivial in many circumstances, and the presence of metadata usually signifies that the image has *not* been modified or stripped in any manner outside of what the artist has provided. We won't talk about PNG interlacing (albeit that also increases filesize), since most images come exported non-interlaced by default. It's an option rarely seen. ## Filesize (and why bigger doesn't mean better) **Filesize does not accurately describe image quality.** This is where I'll describe why that is. ### JPG filesize Remember what I said about JPG being lossy? Well, that counts double if you re-encode a JPG by way of transforming an image or simply just resaving it in your editor. This is known as *lossy-lossy* compression. **When you save a JPG that uses JPG information, you legitimize the JPG artifacts in that image.** Your editor simply doesn't know how to re-encode a bit-by-bit representation of the image you've saved because all it sees is *the* image. Doing this means you re-encode even the lossy artifacts already present in the original image, in addition to introducing more artifacts. And not to mention when you do that, you're saving a new image entirely. It may even be a larger filesize than the original image you were working with. And it's not as if you can do much to make it smaller; JPG compression only works best when you're working with an image that's lossless to begin with. **A JPG image isn't automatically better just because it has a higher filesize.** There are way too many factors to account for that; Yes, it could mean that such an image has underwent _less_ sampling (perhaps a higher quality compression or better chroma subsampling options) but it says nothing about what we know from the original image; we can only observe the raw image we have. Just as a small note, Twitter always resamples JPG images uploaded to the service, as do other social networks like Pawoo and Weibo. If I recall correctly, it's something like 85% compression with 4:2:0 chroma subsampling, [but this comes from a comment I read some time ago by fireattack](http://danbooru.donmai.us/forum_posts/133431). Artists can be naive at times. Sometimes they may not have the original project file anymore and still want to revise an image, in which case they'll go straight to editing the JPG file they have online because it's all they have at hand. As an uploader, you're going to have to keep an eye out for such cases. There are cases where a file is simply smaller because it's been stripped of all its metadata. But aside from that, if the image is still the same compared to its metadata-present alternative, then it shouldn't matter. ### PNG filesize This is where the statement especially rings true. Bigger doesn't mean better, *especially* not in the case of PNGs. Remember, **PNG is lossless**. Most PNGs that are uploaded to pixiv, deviantart, etc. are bit-by-bit accurate representations of the original image that the artist wanted to provide. But sometimes, they may provide images that are even better compressed *while still representing the same image*. In that case, PNGs with smaller filesizes are **better** than their larger counterparts, assuming that the metadata is still present. Of course, you could always copy it over if it isn't but... Since examples do a much better job at explaining this, let's take a look at some of these karutamo images: * [id 2779976](https://danbooru.donmai.us/posts/2779976) * [id 2779979](https://danbooru.donmai.us/posts/2779979) * [id 2779980](https://danbooru.donmai.us/posts/2779980) * [id 2779982](https://danbooru.donmai.us/posts/2779982) See how these images from his official portfolio are better compressed than *both* the images he uploaded to Twitter *and* Pixiv? And they still have the metadata present too. Take a look: ``` ExifTool Version Number : 10.61 File Name : id_2779982.png Directory : . File Size : 1725 kB File Modification Date/Time : 2017:09:12 15:11:48-07:00 File Access Date/Time : 2017:09:12 15:11:48-07:00 File Creation Date/Time : 2017:09:12 15:11:48-07:00 File Permissions : rw-rw-rw- File Type : PNG File Type Extension : png MIME Type : image/png Image Width : 1158 Image Height : 1637 Bit Depth : 8 Color Type : RGB with Alpha Compression : Deflate/Inflate Filter : Adaptive Interlace : Noninterlaced Gamma : 2.2 White Point X : 0.3127 White Point Y : 0.329 Red X : 0.64 Red Y : 0.33 Green X : 0.3 Green Y : 0.6 Blue X : 0.15 Blue Y : 0.06 Background Color : 255 255 255 Pixels Per Unit X : 13779 Pixels Per Unit Y : 13779 Pixel Units : meters Datecreate : 2017-03-23T05:58:12+00:00 Datemodify : 2017-03-23T05:58:12+00:00 Software : Celsys Studio Tool Image Size : 1158x1637 Megapixels : 1.9 ``` Celsys Studio Tool is the same as CLIP STUDIO PAINT, indicating that that was the software used. But what about cases where an image is better compressed on Twitter than on Pixiv or another site? Well, interestingly enough, this has happened before. Take a look at these two images: * [id 2712109](https://danbooru.donmai.us/posts/2712109) * [id 2844376](https://danbooru.donmai.us/posts/2844376) At first you might believe that the first is obviously better. It is, but just in one respect -- it preserves metadata. Twitter never preserves metadata. However, in this case, the Twitter version is actually compressed better. Let's take a look at these filesizes: * id 2712109 with metadata present: 1.69 MB (1,781,726 bytes) * id 2712109 with metadata stripped: 1.69 MB (1,781,705 bytes) (only 21 bytes lost) * id 2844376: 1.63 MB (1,718,810 bytes) That's a difference of 62,916 bytes from the pixiv image! Strange, isn't it. And when you do a raw diff with those two images, you will find that they represent the *exact same image*. The size is somewhat trivial, but it's a decrease nonetheless. **Cases like these are indicative of when uploading a duplicate is actually preferred because of better lossless PNG compression**. You could even "frankenstein" an ideal copy for archival by simply copying the metadata from id 2712109 *into* id 2844376. The same can be said of another identical pair of images, which I'll let you explore if you need further proof: * [id 2818071](https://danbooru.donmai.us/posts/2818071) * [id 2811322](https://danbooru.donmai.us/posts/2811322) In most cases, though, pixiv will keep the original compression whereas Twitter will often introduce a round of *bad* compression through the use of some pngcrush-like software at upload time. There are plenty of examples of this, but if you need one: * [id 2810165](https://danbooru.donmai.us/posts/2810165) * [id 2706474](https://danbooru.donmai.us/posts/2706474) *Reminder to self: find more SFW examples...* ## Choosing the Best Parent for Identical Images Finally. Although with all the explanations from above you should have enough information as to what, how and why these are listed in the order they are. Let's go through them. * **Unsampled**. Image samples should always be parented to their original image (and subsequently flagged and/or deleted). This is pretty obvious, no further explanation needed. * **Most recent active revision**. Also self-explanatory. * **Least image artifacts**. This guideline is sort of played around with; oftentimes if an image is large enough to offset the amount of artifacting in an image (as is the case with many of Yoshida Iyo's images), then that gets the parent instead. * **Most lossless file format**. * **Highest resolution**. If a picture has been upscaled, the upscaled version should be the child. We prefer the most original image first and foremost, as intended by the artist and not artifically upscaled with waifu2x and the like. * **Largest colorspace**. RGBA32 encodes more data per pixel than RGB24, *even if the alpha channel has absolutely no useful information in it*. * **Most recent active post**. There are images with extremely subtle differences between each other that it could go either way. But sometimes an artist deletes their old work and reuploads it with *very* minute changes. * **Most unaltered metadata** for exactly identical images. Pixiv no longer uses the jpegtran optimizer, so that's not something you need to worry about. But either the metadata is present or it isn't, and that's what this refers to. * **Best compression**. Whichever post has a smaller filesize while retaining the exact same image data is the parent. The PNG filesize subsection should explain this all, but tl;dr better lossless compression is always more preferable. ## What "Duplicates" You Should Up (and What We Will Approve) Note that if you're going to be uploading "duplicates", they should all be from a legitimate source **by the artist or publisher**. This criteria does not apply to third-party edits -- that includes optimizations. * **Identical PNG images with better compression than an existing image on the server**. This is regardless of metadata present -- as I mentioned before with [id 2844376](https://danbooru.donmai.us/posts/2844376). * **JPG images with larger resolution than an existing image on the server**. This is regardless of whether the existing image is JPG or PNG. In the case of PNG, you can always parent it to its' PNG counterpart, but some artists export their PNG images to pixiv in a smaller resolution compared to what they share on Twitter... unknowingly, assumably. Note that these images should **NOT** be artificially upscaled by way of waifu2x or the like. You must always try to upload from a trusted source (whether by the artist or publisher). * **Pre-revisions**. Obviously, this is way more confusing, but if you can confirm an image is a pre-revision by tagging the subsequent image revision as such, then we're also obliged to accept that. See [id 2851989](https://danbooru.donmai.us/posts/2851989). ## Tools Some of the tools you might need, to help understand stuff: ### Metadata Tools * [ExifTool](https://www.sno.phy.queensu.ca/~phil/exiftool/) -- The easiest metadata utility in my opinion. It's rather simplistic. * [Exif2](http://www.exiv2.org/) -- another metadata utility. Also a valid option. ### Images * [ImageMagick](https://www.imagemagick.org/script/index.php) -- specifically the [compare](https://www.imagemagick.org/script/compare.php) function. You'll probably be using it a lot. * [Krita](https://krita.org/en/) -- Any editor works, really, but I like Krita best for everything (image editing and drawing). * [Honeyview](http://www.bandisoft.com/honeyview/) -- You can view basic EXIF metadata by simply pressing the Tab key. Convenient. Plus it's a fast image viewer to begin with. ## Some Notes Regarding Certain Sites ### Twitter Twitter will **always** recompress an image uploaded to the service... *except in certain cases!* All Twitter PNGs are RGBA32. If you upload a PNG with alpha transparency information, it will retain that information; it will just do the classic *introduce a round of bad compression* to the image and strip out the metadata as usual... which is a mini-bundle of fun, but whatever, right. However, take care that this does not apply to PNGs with RGB24 information. Instead, those are converted and sampled to JPG as per the usual. This is all (hopefully) explained in [howto:twitter](https://danbooru.donmai.us/wiki_pages/83965) also, so in case you forget you can refer to there. * [And this site kind of describes it all](http://ravenworks.ca/twitimagefix/) ### Pawoo Pawoo is pretty much the same as Twitter except Pawoo retains image metadata. It also carries image dimension maxima; images can only be 1280 x 1280 at their largest. One thing to note about Pawoo, however, is that Pawoo samples at a *considerably* lower rate than Twitter does (which makes uploading from Pawoo worth it). They still sample images uploaded, but it's much harder to tell if the images are the same dimensions. ## Questions & Comments If you have any questions or comments about this guide, bump [the topic](https://danbooru.donmai.us/forum_topics/14426) and we can have a discussion about it. I'm seeking to make this guide better and more user-friendly, so all comments are welcome. And if you feel like your question is too specific or you feel it's too inappropriate to make public, feel free to message me on Danbooru. My handle is Mikaeri ([user #470449](https://danbooru.donmai.us/users/470449)).