Fixing AI generated pixelart
Fixing AI generated pixelart
I’ve wanted to create a game in a pixelart style for quite some time now, but just didn’t have a lot of time on my hands to really dive into the topic. There’s a lot to know to create a game - starting with technicalities like picking an engine and ending with the artstyle - like creating even the simplest of assets(and a lot more than that!).
I can get past the technical stuff, but I’m not particularily artsy.
Coincidentally, I’ve been also playing around with Midjourney. I’ve tried generating a lotta stuff, pixelart being one of them. I was mildly happy with the result, but there were some artifacts that needed some fixing.
The artifacts varied a lot - but it looked like a vast majority of them could be fixed by some simple image math. So I’ve created fixelart, which is a simple library that fixes up a lot of those artifacts.
Let’s see an example of what this simple utility can do:
This is an image generated with the prompt 2d pixelart girl portrait pixelart pixelart --v 5(those pixelart repetitions are deliberate, to ensure the most pixel-arty output). You can see that it’s not exactly pixelart - but it’s close.
Ad.1 Original Midjourney-generated image
Here’s some of the “fixed” output, based on a couple of simple algorithms
Ad.2 Majority algorithm
Ad.3 Average of pixel colors(downscale)
Ad.5 Harmonic mean
Ad.6 Geometric mean
Ad.7 5% top color algorithm
Ad.8 10% top color algorithm
Too see the differences better, here’s a version of preview that overlaps images
All of this looks quite good, right? Here’s a handful of information:
- Midjourney’s output is a png of 1024x1024.
- Output is a 128-bit pixelart(8x8 of input pixels are used for computing an output pixel(1024/8=128))
- All the code in available in this repo
If you want to play around with this tool yourself - I’ve created a playground for it
Let’s talk about how all of this works. It’s actually very simple:
- The image is split into
outPixWidthbyoutPixHeightchunks - Then a simple algorithm(
strategy) is being applied to each chunks - Then the image is de-chunkified
There are a couple of algorithms currently implemented:
Strategies.AVERAGE- take the arithmetic mean of colors in the blocks(which is basically downscaling the image)
Strategies.HARMONIC- take the harmonic mean of colors in the block
Strategies.GEOMETRIC- take the geometric mean of colors in the block
Strategies.MIDRANGE- take the midrange of colors in the block
Strategies.QUADRATIC- take the quadratic mean of colors in the block
Strategies.CUBIC- take the cubic mean of colors in the block
Strategies.MAJORITY- take the color that occurs the most often in the block(a maximum)Strategies.ALG(05|10|20|30|40|50|60|70|80|90)- a mix; if a color is making up above X%, then take it, otherwise take the average
This set of strategies could easily be expanded by algorithms that take into account not only pixels in one block, but also the surrounding blocks, which could provide different(possibly better) results.
The library also provides a way to shrink the output to match the pixelart pixel to match the actual pixel size(yes, it’s confusing)
More examples
Let’s see more examples with different input styles
Pixel width/height: 3
A shotgun - notice how harmonic/geometric/quadratic means differ in color tones
Pixel width/height: 6
Pixel width/height: 8
Sometimes Midjourney generates almost pixel perfect stuff with weird outlines - fixelart can fix that too. Notice how 5% top majority algorithm can fix the art while preserving detail - taking just the majority has some artifacts. This image really shows that for each different style, different algorithms can shine.
Pixel width/height: 8
Pixel width/height: 4
Ending notes
Just as an ending note, I want to say that all of this isn’t particularily innovative, but I wanted to show how well all of this works in combination with Midjourney and different rounding strategies. I’ve also noticed that other libraries which are supposed to “pixel-art” your stuff are not giving you any options to customize the behavior.
Since the output AI-tools give are close enough to actual pixelart, the “pixelarted”(or fixelarted) output is not a pixel mess not resembling anything, but something that can look fairly good. Or atleast my untrained eye sees it as pretty good.
I also wanted to show that with the rise of AI GAN tools, having a generate => convert to pixelart pipeline as a part of your prototyping might be viable, even if only to create a moodboard to get inspired.
I’ll probably experiment with something more than just means in the future - I think taking into account neighbouring blocks could really make a great algorithm.
Lastly, If you enjoyed any of what I’ve written here, or you think the library or playground is cool, I’d appreciate a star on the fixelart repo, so I know if anyone’s interested in seeing any further development of this tool. I’d also highly appreciate any feature requests - If through reading this at any point you thought “if only this library could also do XXX” - feel free to create an issue!
Now, if you’re still hungry for examples - try fixelart yourself in the playground!