Artshiftr Paintings
With our media project Artshiftr we're exploring the potential for transforming location photography into different formats to create assets for creators, designers, and decorators.
In this post we discuss how we combined cutting-edge AI techniques with our own drone photography to create painterly prints rooted in real landscapes. Every piece starts with a genuine photograph and ends as a vibrant, gallery-quality print.
We've been using machine learning in our work for many years. These systems usually involve some form of computer vision for gesture-recognition, pose-detection, or image classification. In recent years there's been a new wave of AI image processing tools which have opened many new doors of possibility in the field of media production.
From AI up-resolution to depth estimation and masking, these tools have made it possible to process images and video in powerful ways. With the rise of ComfyUI we can now easily combine these tools into custom AI workflows to build streamlined systems we can use in our work.
Media Ethics
Amongst these new AI techniques is diffusion, which has become a contentious subject in our field. As digital creators, and contributors to popular stock media libraries, we've seen as our media became a part of the datasets used to train AI diffusion systems, so we personally understand the ethical implications of this technology. After substantial research and consideration we've determined what we believe is a grounded approach to leveraging AI diffusion in our work which is both legally and ethically sound.
In this post we'll talk about the system we created to transform crops of some of our favorite drone photographs into digital paintings. We'll discuss some of the ethical considerations, and where we see this aspect our industry going in the future.
The Source Material
We believe that authenticity is one of the most valuable qualities in art, and every effort should be made to capture and preserve this quality in a work. With Artshiftr, all of our works are grounded in real photographs we've taken of nature, binding the works to real places with genuine stories behind them. For this project we opted to take four of our panoramic drone shots and use them as the basis for these digital paintings.
The Workflow
Users of popular AI tools like ChatGPT are likely familiar with their powerful image processing capabilities; individual images can easily be uploaded and iteratively processed in amazing ways. But these systems have clear limitations, most notably their resolution. While these tools may currently work for processing images as large as 1920x1080px resolution, they will not give you anywhere close to the ~10,000px detail needed to support a large format print.
For our digital paintings, we opted to create custom workflows in ComfyUI which allow us to process the images locally on our workstations using StableDiffusion. Here's some key details from that workflow:
-SDXL : We use the Fenris SDXL Lightning model which enables quick diffusion at 2048px x 1024px and is built on SDXL 1.1 which has a license which permits commercial use. We prompt for the Impressionism style [ 1860-1886 ] or Watercolor, and sometimes try out different material-centric LORAs depending on the image.
We then use depth and line control-nets to guide the diffusion across multiple layers to keep it adhering to our original photo.
-We use an ip-adapter sampling a close up of some brush-detail combined with specific prompting about the materials to create as much detail as is necessary for a large format painting.
-We feed the original panoramic photo into the sampler's latent image, then process it through the system in multiple layers, never more than 50% so the essence of the original image is preserved.
-We use multiple layers of a latent upscale through the detail demon node to increase the resolution of the image while preserving the important details.
Additional Processing
This process allows us to create 4k stylized versions of the images which largely match with the aesthetic we're going for, but many issues still exist which require manual post-processing for us to finalize the image. In this case there were a number of issues with repetitious patterns in the clouds as well as a kink in the sunlight reflection which required re-working in Photoshop. After cleanup, the 4k image is still short of the 10-12k resolution needed for prints, so we run the image through the Topaz Gigapixel AI to increase the resolution by 2.5x into the needed range.
The Result
The resulting image captures an accurate look at the vast beauty of the Georgian Bay translated through the impressionist style for those who prefer an emotive painting style over a photo for their decorative needs.
If you're interested in viewing our digital painting collection, check out our Artshiftr Etsy store.
