Development Blog

I guess concept artist are a thing of the past. 🤷‍♀️

I've been using Dall-e. Similar concept, type in a description of what you want and it tries to generate that picture.  You can also specify what style you want. Oil painting ,3D graphic or pixel art etc. The misspelling  you are getting is deliberate. Its to stop people creating offensive texts  or ask for a  famous person to hold a sign saying "I am stupid" or worse.

Below is Dall-es' attempt at "UltraEngine" title graphic.

 

These images all came from dall-e:

 

The power of these programs is amazing. 

It really will change the way we create art.

Here is a Rabbit walking sprite sheet from Dall-e.

Normally this would take ages to create in some paint program, now you just type a one line sentence.

More startlingly good images. If you could just train it on a set of game textures the output would be much better:

Midjourney seem to make better artistic images then dalle.

I think is just trained with billions of images and has some kind of image recognition and classifications and is just copy pasting, deforming images in a clever way.Maybe im wrong.

Probably is also using input from the people generating the images to learn what good image means.

 

Anyway this is good news for indies by the look of it

I plan to use midjourney for some 2d game experiment.

 

old man sitting on a mountain near the sea with a maelstrom swirling in front of him , providence , cthulhu monster , H P Lovecraft world , the color from the sky

 

I've been playing with these for a few months now ... these image generators will put a lot of people out of a job. Including me, to some extent.

anyway: Dall-e and midjourney gave me mixed results, and I got threatned to get banned for trying to get some gory textures. Stable Diffusion is the new kid in town, and it can be run locally, on colab, or on their website, and the code is open source, so you can just bypass the NSFW filter and get all the blood and guts you want. also boobs, if you are okay with them having three nipples, or no nipples, occasionally. Texture generation has also been disappointing, so far.

but then there's centipede diffusion, a predecessor of stable diffusion. runs on colab and creates a bunch of small images, quickly, and uses those then as input images for the diffusion, and finally does upscaling. here, you can actually get very good textures, - you'll have to make them tilable afterwards, and the process is very slow. But since it doesn't run on your GPU, but on one of Google's. It's also much easier to control the style - if you really want a specific quality in your image - it helps to know the specific painterly qualities of individual artists, though, you can ask for that. Here's some meat and fat and skin, with the painterly qualities of a bunch of 19th and 16th century painters.

 

[image removed]

This is all extremely entertaining and all but I've not yet seen a single image actually useful for any production of anything to be hones.

@chsch Can you train centipede diffusion with your own images?

I also tried training playform.io with all the Gothic textures from Quake 3 Arena. The results are interesting, and clearly have a similar feel to the source material. However, I would not call these images useful for anything:

I didn't try midjourney yet, but it really interests me. I saw some really cool concept images made with it. However, I've been playing with Dall-e recently for some basic concept images.

Here's some test for some Horror Lovecraftian realistic images (I can't remember the actual keywords I've inputed)

 

However, just like @chsch because I wanted some horror related images, I got threaten of being banned from using DALL-E, even though I didn't ask for gory stuff.

Generative Adversarial Networks (or GAN's for short) are the approach taken to get this kind of generative model using deep learning techniques. Supervised learning is the most common method of training GAN's which are usually made up of two sub-models, a discriminator and a counterfeiter. The counterfeiter has the role of generating fake images, where the discriminator then must classify those images as either real or fake. The ultimate goal is to converge training to a solution where the counterfeiter gets so good at generating fake examples the discriminator can then no longer classify the examples as real or fake.

My masters thesis (which I finally finished a few days ago) explored some pretty interesting areas of training supervised learning problems in the computer vision field. Most datasets are typically composed of real-world examples of images, its often an expensive and time-consuming task to create these datasets of images as it not only requires hardware in terms of camera equipment but also heavy investment of valuable time to annotate the images (supervised learning for object detection involves labelling images which is the practice of drawing bounding boxes around the objects within the image in order to provide the model labelled data to learn from). Automating the entire process of data collection is the pursuit of happiness. Applying technologies such as game engines to render the images and also program the logic for automatic annotation of objects in a scene provides an end-to-end approach for data collection. The possibilities to simulate abstract imagery within a game engine opens up the doors for high-quality data that is collected in an efficient way for building large-scale datasets. Synthetic data could improve these kinds of models as we can simulate almost anything we like within a game engine. Feeding the resulting image data that is correctly formatted for model input could really push the results these types of models can achieve.

Check out differentiable rendering, that is crazy stuff and the generalization power is improving over the years 

 

On 9/9/2022 at 6:06 PM, Josh said:

@chsch Can you train centipede diffusion with your own images?

technically, yes- but realistically, no. There was a person who offered custom trained models though, which resulted in a few very weird image generators . Ukiyo-e diffusion was relatively "useful", but she also released a thing called "textile diffusion" which would create everything as stitched into fabric. She was doing this fo a few hundred bucks if you let her release the model publicly, and a few hundred more if you wanted to keep the model to yourself. - She doesn't do that anymore, though, she got hired by Stable Diffusion.

Stable DIffusion, as I found out by now, can do tiled textures, and it can do image to image. Meaning, you can use it as a filter that creates variations of a given image- or texture. Say, you have one texture of rocks from texturehaven - you can make variations of that, photorealistic or stylized.

Stable diffusion also offers a thing called textual inversion, where you can train it on a handful of images to "learn" a concept, and assign a variable nameto that concept, so you can then ask it to make images of 'variable name' in the style of something else, or to create whatever, in the style of 'variable name'.

It's not quite training a whole model, and I haven't tried out how well it works - but right now, the only thing that makes this not absolutely perfect for the job is that generating images larger than 768x768 takes more VRAM than google colab offers me.

On 9/9/2022 at 9:22 PM, Josh said:

I also tried training playform.io with all the Gothic textures from Quake 3 Arena. The results are interesting, and clearly have a similar feel to the source material. However, I would not call these images useful for anything:

 

I do have a fine art education, and honestly, as far as capital A Art is concerned, these are great.

Here is another one. This one is used for making textures including maps ( height, normal, emission etc).

https://barium.ai/

A few months later, and this is how the technology has progressed:

Not 100% those will tile correctly, but these should be fine for a base texture. I'm sure eventually we'll see plugins for Stable Diffusion that'll make the all the textures for a PBR material and be accurate. 

tiling is a feature of SD, though and has been a mere checkbox away for a while. PBR is likely going to be messy, though, but I was quite impressed by a model trained to create depth maps from pictures of trees.... in the meantime: SD and materialize are now all you need.

Also, there's a blender plugin to create tiling textures from within blender.

18 hours ago, chsch said:

tiling is a feature of SD, though and has been a mere checkbox away for a while. 

Where? I don't see this anywhere on their site:
https://stablediffusionweb.com/#demo

I have a Stable Diffusion server on my machine with a RTX 3060. Since I installed Easy Diffusion and not something like Automatic111, I don't seem to have much flexibility, but Easy Diffusion had an automatic installation. 

On 4/23/2023 at 5:02 PM, Josh said:

Where? I don't see this anywhere on their site:
https://stablediffusionweb.com/#demo

it's there in the AUtomatic 1111 webui, together with A LOT of other options.

I don't see any such option. It says "Advanced options are unavailable right now".

My conclusion on this stuff is it's useless. At best you can create something that looks worse than stock PBR materials you can buy or download for free.