# Illuminarty API v0 *(private alpha version)*
<div style="text-align:center">
<img src="https://i.imgur.com/t3CKfJK.jpg" alt="hero" width="500"/>
</div>
<br>
[Illuminarty](https://www.illuminarty.ai/)
The performance of the API is subject to change, we expect it to become better as we collect more data and include additional features.
The following analysis has been done using the trial version for the API deployed on 11/4/2022.
## Performance Analysis
### TLDR
* *We assume that no image in the "Human Images" has been generated by AI without disclosure*
* If we use a threshold of 70%, we can filter about **74.2%** of the AI drawings, but about **5.2%** of the human drawings will be misclassified.
* If we use a threshold > 85%, we can filter about 32% of the AI drawings, and 0% of the human drawings will be misclassified.
* The performance is the worst for Dalle2, but it can be handled by its watermarks.
### Dataset
We conducted a simple analysis for the current performance of Illuminarty using the images from [Reddit](https://www.reddit.com).
We crawled a total of 1,000 images from 10 subreddits, 100 images for each subreddit. We tried to pick 5 subreddits that contain only the human-drawn images and the other 5 that contain the AI generated images. Here is the list of the subreddits that we chose:
* Human Images
* https://www.reddit.com/r/Art/
* https://www.reddit.com/r/DeviantArt/
* https://www.reddit.com/r/DigitalArt/
* https://www.reddit.com/r/Illustration/
* https://www.reddit.com/r/Pixiv/
* AI Images
* https://www.reddit.com/r/aiArt/
* https://www.reddit.com/r/dalle2/
* https://www.reddit.com/r/midjourney/
* https://www.reddit.com/r/NovelAi/
* https://www.reddit.com/r/StableDiffusion/
We assume that every single post from subreddits in the "Human Images" is indeed from human artists and the same for the "AI Images".
### Overall Description
The following figure shows the distribution of the probabilities obtained from the Illuminarty API.
As you can see from the figure, there are some human-drawn images that show high probabilities and also some AI images that show low probabilities. These images can be misclassified depending on the threshold (e.g. above 50%, or above 70%) we choose for deeming an image as generated from an AI.
The following figure shows the [ROC Curve](https://en.wikipedia.org/wiki/Receiver_operating_characteristic), which shows how the true positive rate (correctly guessing the AI images) and the false positive rate (wrongly guessing a human-drawn image as AI) vary as we change the threshold probability.
Since increasing the true positive rate is at a trade-off with decreasing the false positive rate, one should choose the threshold comparing the costs of having a false positive and having a false negative (deeming an AI image as human-drawn).
Let us assume that we prefer to keep the false positive rate low, i.e., around 5%. According to the data of 1k images we crawled from Reddit, the threshold probability should be about 72%. If we choose the threshold value of 70%, we can obtain the following [confusion matrix](https://en.wikipedia.org/wiki/Confusion_matrix):
Thus in this case 26 out of 500 images (5.2%) that were human-drawn will be misclassified, and 142 out of the 500 images (28.4%) that were AI generated will be misclassified, thus filtering about 71.6%. 28.4% seems like a quite high number, however as mentioned earlier, some of the posts from the AI subreddits contain human intervention, e.g. screenshots or source images.
From the observations that will be made in the next sections, we assume that we can remove the following errors:
* 53 images from r/dalle2
* 9 images from r/NovelAi
* 13 images from r/StableDiffusion
a total of 75 images can be removed from the false negatives and thus we get this new confusion matrix:
which now gives us the false negative rate of 15.8%, thus filtering 74.2% of the AI images.
Lastly, one can see that the maximum value for the human drawings were 84.57%. If we set the threshold to be bigger than this value, there would be 0 false positives. Here is the confusion matrix assuming that we can subtract the above 75 images:
but this would allow a false negative rate of 68%, filtering only about 32% of the AI images.
We share the examples of the misclassified images using 70% as the threshold in the next sections, for each subreddit.
**WARNING!! Some of the images in the following section are NSFW**
### r/Art
The following 8 out of 100 images would have been misclassified when using the threshold of 70%.
<img src="https://i.redd.it/dsfifef3vqx91.jpg" alt="drawing" width="400"/>
Probability: 80.21%
https://www.reddit.com/r/Art/comments/yl3ycf
<img src="https://i.redd.it/tbu3alawjrx91.jpg" alt="drawing" width="400"/>
Probability: 78.86%
https://www.reddit.com/r/Art/comments/yl7eb0
<img src="https://i.redd.it/1i2j657n8tx91.png" alt="drawing" width="400"/>
Probability: 78.52%
https://www.reddit.com/r/Art/comments/yl8e6o
<img src="https://i.redd.it/zmwp4kun6sx91.jpg" alt="drawing" width="400"/>
Probability: 74.63%
https://www.reddit.com/r/Art/comments/yl35z7
<img src="https://i.redd.it/4m0t5x8visx91.jpg" alt="drawing" width="400"/>
Probability: 73.34%
https://www.reddit.com/r/Art/comments/yl4se6
<img src="https://i.redd.it/qx722i7pisx91.jpg" alt="drawing" width="400"/>
Probability: 73.34%
https://www.reddit.com/r/Art/comments/yl4rl5
<img src="https://i.redd.it/lmt48oogfqx91.jpg" alt="drawing" width="400"/>
Probability: 73.12%
https://www.reddit.com/r/Art/comments/yl1yk4
<img src="https://i.redd.it/fsxvrjpvesx91.jpg" alt="drawing" width="400"/>
Probability: 71.34%
https://www.reddit.com/r/Art/comments/yl495l
### r/DeviantArt
The following 3 out of 100 images would have been misclassified when using the threshold of 70%.
<img src="https://i.redd.it/2apjht9uq2s91.png" alt="drawing" width="400"/>
Probability: 77.90%
https://www.reddit.com/r/DeviantArt/comments/xwrvo8
<img src="https://i.redd.it/awhs4nolryv91.png" alt="drawing" width="400"/>
Probability: 77.85%
https://www.reddit.com/r/DeviantArt/comments/yd6i2s
<img src="https://i.redd.it/4znyhpctk6w91.jpg" alt="drawing" width="400"/>
Probability: 76.86%
https://www.reddit.com/r/DeviantArt/comments/ye303z
### r/DigitalArt
The following 7 out of 100 images would have been misclassified when using the threshold of 70%.
<img src="https://i.redd.it/7q73h6eprnx91.jpg" alt="drawing" width="400"/>
Probability: 84.57%
https://www.reddit.com/r/DigitalArt/comments/ykrghc
<img src="https://i.redd.it/084l068tunx91.jpg" alt="drawing" width="400"/>
Probability: 84.05%
https://www.reddit.com/r/DigitalArt/comments/ykrsmf
<img src="https://i.redd.it/4cg2pyh3lox91.jpg" alt="drawing" width="400"/>
Probability: 81.59%
https://www.reddit.com/r/DigitalArt/comments/ykonk1
<img src="https://i.redd.it/ec7jh0x9arx91.jpg" alt="drawing" width="400"/>
Probability: 80.82%
https://www.reddit.com/r/DigitalArt/comments/yl614t
<img src="https://i.redd.it/2kdsgehxmqx91.png" alt="drawing" width="400"/>
Probability: 76.25%
https://www.reddit.com/r/DigitalArt/comments/yl2zbn
<img src="https://i.redd.it/qbqgxo86vnx91.jpg" alt="drawing" width="400"/>
Probability: 75.27%
https://www.reddit.com/r/DigitalArt/comments/ykru7q
<img src="https://i.redd.it/0ortfb8cdrx91.jpg" alt="drawing" width="400"/>
Probability: 72.04%
https://www.reddit.com/r/DigitalArt/comments/yl6hg8
### r/Illustration
The following 6 out of 100 images would have been misclassified when using the threshold of 70%.
<img src="https://i.redd.it/lefsqk09dox91.jpg" alt="drawing" width="400"/>
Probability: 84.57%
https://www.reddit.com/r/Illustration/comments/yktntt
<img src="https://i.redd.it/jt10fwt5vsx91.jpg" alt="drawing" width="400"/>
Probability: 84.41%
https://www.reddit.com/r/Illustration/comments/yl6ive
<img src="https://i.redd.it/3ty7xkzj4kx91.jpg" alt="drawing" width="400"/>
Probability: 83.52%
https://www.reddit.com/r/Illustration/comments/yk8faz
<img src="https://i.redd.it/dhpo9lzdspx91.jpg" alt="drawing" width="400"/>
Probability: 82.42%
https://www.reddit.com/r/Illustration/comments/yktfhv
<img src="https://i.redd.it/mevorswc1kx91.jpg" alt="drawing" width="400"/>
Probability: 80.32%
https://www.reddit.com/r/Illustration/comments/yk7y8e
<img src="https://i.redd.it/sp4e4a6ikpx91.jpg" alt="drawing" width="400"/>
Probability: 74.47%
https://www.reddit.com/r/Illustration/comments/ykxsg1
### r/Pixiv
The following 2 out of 100 images would have been misclassified when using the threshold of 70%.
<img src="https://i.redd.it/lf3n9ormujx91.png" alt="drawing" width="400"/>
Probability: 75.44%
https://www.reddit.com/r/Pixiv/comments/yk70gz
<img src="https://i.redd.it/uz3upjx5gwn91.jpg" alt="drawing" width="400"/>
Probability: 70.77%
https://www.reddit.com/r/Pixiv/comments/xef554
### r/aiArt
20 out of 100 images would have been misclassified when using the threshold of 70%.
We share the first 5 of them that have the lowest probabilities.
<img src="https://i.redd.it/05ahhkv1alx91.png" alt="drawing" width="400"/>
Probability: 11.26%
https://www.reddit.com/r/aiArt/comments/ykeh6t
<img src="https://i.redd.it/wnmrh94neqx91.jpg" alt="drawing" width="400"/>
Probability: 11.79%
https://www.reddit.com/r/aiArt/comments/yl1u2q
<img src="https://i.redd.it/8ktt2c7ablx91.png" alt="drawing" width="400"/>
Probability: 11.86%
https://www.reddit.com/r/aiArt/comments/ykepib
<img src="https://i.redd.it/3ot9lrbawnx91.jpg" alt="drawing" width="400"/>
Probability: 12.06%
https://www.reddit.com/r/aiArt/comments/ykry6k
<img src="https://i.redd.it/13vodsw8fgx91.png" alt="drawing" width="400"/>
Probability: 12.37%
https://www.reddit.com/r/aiArt/comments/yju8u3
### r/dalle2
We would like to note that in fact every image shared in the r/dalle2 subreddit can be correctly classified by its watermark. Here we share the result when we do not use this information, assuming the cases when somebody intentionally removes them from the images.
53 out of 100 images would have been misclassified when using the threshold of 70%.
We share the first 5 of them that have the lowest probabilities.
<img src="https://imgur.com/3h39fwJ.jpg" alt="drawing" width="400"/>
Probability: 5.62%
https://www.reddit.com/r/dalle2/comments/ykar51
<img src="https://i.redd.it/6fwwb59aenx91.jpg" alt="drawing" width="400"/>
Probability: 6.96%
https://www.reddit.com/r/dalle2/comments/yki9o5
<img src="https://i.redd.it/tp3cb9p56px91.jpg" alt="drawing" width="400"/>
Probability: 8.77%
https://www.reddit.com/r/dalle2/comments/ykw9pr
<img src="https://i.redd.it/1mt2gy8bjox91.png" alt="drawing" width="400"/>
Probability: 9.22%
https://www.reddit.com/r/dalle2/comments/yku8ap
<img src="https://i.redd.it/w1yzgw0ogpx91.png" alt="drawing" width="400"/>
Probability: 9.49%
https://www.reddit.com/r/dalle2/comments/ykxcj4
### r/midjourney
19 out of 100 images would have been misclassified when using the threshold of 70%.
We share the first 5 of them that have the lowest probabilities.
<img src="https://i.redd.it/ig1g7zg74gx91.jpg" alt="drawing" width="400"/>
Probability: 10.96%
https://www.reddit.com/r/midjourney/comments/yjshm5
<img src="https://i.redd.it/psfh7lgl7ex91.png" alt="drawing" width="400"/>
Probability: 12.18%
https://www.reddit.com/r/midjourney/comments/yjagje
<img src="https://i.redd.it/m4d8kjbtpnx91.jpg" alt="drawing" width="400"/>
Probability: 17.65%
https://www.reddit.com/r/midjourney/comments/ykk7d6
<img src="https://cdn.discordapp.com/attachments/1036927824763817997/1037567890305589278/underthebug_modern_bike_incorrectly_assembly_b32e1cda-cef2-4f6b-8958-d049ba396ab7.png" alt="drawing" width="400"/>
Probability: 19.47%
https://www.reddit.com/r/midjourney/comments/ykr1ai
<img src="https://i.redd.it/upx83khjhdx91.png" alt="drawing" width="400"/>
Probability: 26.87%
https://www.reddit.com/r/midjourney/comments/yjemx4
### r/NovelAi
16 out of 100 images would have been misclassified when using the threshold of 70%.
However, screenshots and memes also seem to be shared quite often in r/NovelAi.
The following 8 images do not seem to be AI-generated:
* https://i.redd.it/wvsyab93oxw91.png
* https://i.redd.it/rj71e7hbknw91.png
* https://i.redd.it/x8rdkfomcqx91.png
* https://i.redd.it/kgq7sv7uvtw91.png
* https://i.redd.it/lifjgsi40tw91.jpg
* https://i.redd.it/o4rkhosc78x91.jpg
* https://i.redd.it/o0hq99acftw91.png
* https://i.redd.it/iw0tsj05k7x91.jpg
And this image contains a human picture:
<img src="https://i.imgur.com/7f0eVHi.jpg" alt="drawing" width="400"/>
Probability: 40.37%
https://www.reddit.com/r/NovelAi/comments/yl28lp
Thus if we exclude the above cases, 7 out of 100 images would have been misclassified.
We share the 5 images that have the lowest probabilities, except for the ones mentioned above.
<img src="https://i.redd.it/4mgfcpcly3x91.png" alt="drawing" width="400"/>
Probability: 33.15%
https://www.reddit.com/r/NovelAi/comments/yi5vxg
<img src="https://i.imgur.com/f4G2DWT.png" alt="drawing" width="400"/>
Probability: 43.93%
https://www.reddit.com/r/NovelAi/comments/yk16qu
<img src="https://i.redd.it/xa5r19izprw91.jpg" alt="drawing" width="400"/>
Probability: 46.86%
https://www.reddit.com/r/NovelAi/comments/ygne3d
<img src="https://i.redd.it/4hg6iqytn6x91.png" alt="drawing" width="400"/>
Probability: 51.39%
https://www.reddit.com/r/NovelAi/comments/yik2dr
<img src="https://i.redd.it/c9tw798wvix91.png" alt="drawing" width="400"/>
### r/StableDiffusion
34 out of 100 images would have been misclassified when using the threshold of 70%.
Similarly to r/NovelAi, r/StableDiffusion shares a lot of non-AI generated images as well.
The following 8 images do not seem to be AI generated:
* https://i.redd.it/cqonp8l3dpx91.jpg
* https://i.redd.it/7t7fs6obkjx91.png
* https://i.redd.it/ppoo9opsghx91.png
* https://i.redd.it/91txjrufwgx91.png
* https://i.redd.it/81yqcs3uhrx91.png
* https://i.redd.it/w0g5vqdrxmx91.jpg
* https://i.redd.it/w7sp493igix91.png
* https://i.redd.it/rmpa82bk7sx91.jpg
The following 3 images are collages, not sure whether this should be seen as AI generated:
* https://i.redd.it/h8nyy4jzcpx91.png
* https://i.redd.it/ctdbg8h3thx91.jpg
* https://i.redd.it/pnvf35xfdrx91.jpg
The following 2 images seem to include a human-drawn image:
<img src="https://i.redd.it/mf5gvw4u6kx91.jpg" alt="drawing" width="400"/>
Probability: 8.54%
https://www.reddit.com/r/StableDiffusion/comments/yk8r34
<img src="https://i.redd.it/fpvvie669kx91.png" alt="drawing" width="400"/>
Probability: 29.60%
https://www.reddit.com/r/StableDiffusion/comments/yk936k
Thus if we exclude the above cases, 21 out of 100 images would have been misclassified.
We share the 5 images that have the lowest probabilities, except for the ones mentioned above.
<img src="https://i.redd.it/7wbdlfukfmx91.jpg" alt="drawing" width="400"/>
Probability: 11.17%
https://www.reddit.com/r/StableDiffusion/comments/yklezn
<img src="https://i.redd.it/dqxj1ufumkx91.jpg" alt="drawing" width="400"/>
Probability: 11.38%
https://www.reddit.com/r/StableDiffusion/comments/ykb2pw
<img src="https://i.redd.it/sa05gfm3knx91.png" alt="drawing" width="400"/>
Probability: 12.95%
https://www.reddit.com/r/StableDiffusion/comments/ykqn94
<img src="https://i.redd.it/o3sjdxraxkx91.png" alt="drawing" width="400"/>
Probability: 13.26%
https://www.reddit.com/r/StableDiffusion/comments/ykcmnu
<img src="https://i.redd.it/ppjjr2ujbnx91.png" alt="drawing" width="400"/>
Probability: 13.46%
https://www.reddit.com/r/StableDiffusion/comments/ykpmtj
## How to Use
The current endpoint for the API is `https://api.illuminarty.ai/v0/is_ai`.
You can send a POST request to the endpoint with the following two variables:
* `apiKey`: please contact us through [email](mailto:art@illuminarty.ai) or [Discord](https://discord.gg/DbBKDNVGSY) if you do not have one yet
* `image`: a single image file you want to verify
both variables are required. You should add the `apiKey` to the `Authorization` header using the authentication scheme of `ApiKey-v1`, and you should add the `image` as a multipart form-data.
So an example using cURL would be the following:
```bash
curl -H "Authorization: ApiKey-v1 <YOUR API KEY>" \
-F image=@<YOUR IMAGE> \
https://api.illuminarty.ai/v0/is_ai
```
If the request was successful, it will return a JSON with a single key named `probability`:
```
{
"probability": 0.9893
}
```
this number represents the probability of being *generated by an AI*. Thus, ideally an image made from an AI should show a probability close to 1.0, and an image drawn by a human artist should show a probability close to 0.0.
If the request was not successful, it will return an error message:
```
{
"message": "Internal Server Error"
}
```
The trial version for the API has a rate limit of 30 requests / minute. We will try to provide better options in the following updates.
Sign in with Wallet
Connect another wallet