## Idea generation system for Visual Creative worker

### Motivation
When an artist start their works, it's crucial for them to brainstorm different design elements.
Most of the time, the designer will try to gather enough visual reference, including photograph, other people's work for inspiration.
This process is the hardest part in the design pipeline. It often takes few hours of even few days to finish. And most of the designer struggled to generate good design idea.
Currently, generative AI,such as image generation model, has been widely use in entertainment industry to produce realistic art. However, the image generation model only focus on the completion of the artwork, and its lacking of fine grained control. which make it difficult to integrade into current artists' workflow.

We proposed a system that can integrate the current generative AI into artist's work flow, especially in the preproduction part. We aim to increase both efficiency and creativity on idea generation process.

This issue becomes particularly evident during the initial stages of artistic projects in the entertainment industry, where brainstorming various design elements is a critical first step. Typically, artists gather a wide array of visual references, such as photographs and examples of others' work for inspiration from websites such as Pinterest or Google Image. This stage is often the most challenging part of the design process, requiring several hours or even days to complete, and many designers find it difficult to generate compelling design ideas.

In our pilot study

The system aim to
1. Greatly increase the efficiency
2. Greatly improve current user experience during the workflow
3. Integrate generative AI into visual worker's workflow. (Which is very important because a lot of artist / designer hate AI work)

Why not just use AI generative image?
1. Just like Navipath for doctors, AI generative image is not part of a workflow for the artists/ designers. We aim to integrate this with their original workflow
2. AI generative images are lacking of fine grained control
3. AI generative images are sometime inaccurate for designer/artist to use, especially when the reference have to be structural correct and clear.

Our system aim to provide a much faster process to help designer generate multiple designer ideas, and each idea with its own reference images.
### User
All of the visual designer / some artist
### Scenerio
Entertainment designer: design scene for movie/video games/ TV show
Graphic designer: design main visual in multiple places

### Current workflow
Given a design topic, most visual designers will first use google or pinterest to direct search for related image, when they find the images that interest them, they will grab the image to their reference folder or board. After this process, the designer generate some design idea, and do the research, visual searching again. With this iterative process, the designer finally decide a design direction, and picks some reference to help them finish the design.

### System Design
Our system will combine both LLM ,Visual generative AI, and Image captioning model to assist designer to iterate this process.
**The system aims to provide not only visual reference but also idea generation assistant for designer.**

#### iterative idea generation and reference gathering
Begin with a main design topic given by the designer, we use LLM as a backend to generate few ideas as options.
The user can then select a subtopic given by LLM, and based on the subtopic, LLM will generate another few options along with some images

After few iteration, the user will can choose their final design idea arrange by LLM
And Based on the design idea in text, the system will search online those image, and show top N image to users, and let them pick their favorite into the reference board. The Top N will use CLIP similarity model to decide.
There will also be some AI generative image for user to choose if they want

##### Slider of creativity
Sometimes artist need to choose between conservative of unconventional visual design given a theme, we design a slider so the user can decide if the idea generated and references should be more wild or not
- Conservative: follow the theme
- Unconventional: Mix ideas with different theme

**After more iteration, the users will get one of the thorough design idea with a set of reference image based on the design idea.**
If user want more design idea, they can repeat the process

#### Smart reference gathering system
During the process, the user will have multiple ways to gather reference
1. Keyword approach
2. Exploration by topics
- User can browse different reference images by a topic (layout, color, theme)
3. Using existing image and a given condition
- User can use images they have or those given by the process to search for more reference by color, reference or theme
4. Recommendation base on current selections
- Our system will provide some reference image based on user's current reference board
5. Combination of reference
- With generative AI approach, the user can combine different reference images' element, such as color, layout, or theme
6. Smart classification system for references
- Automatic Hashtag for searching

### Compare to current solution
As my experience in the animation industry, I'm quite confident that there isnt a efficient method for designer to generate Idea along with reference gathering using AI solution. With the system above, the visual worker will save huge tons of time, and have higher quality of design idea and references.

### Value to the user
The idea generation process for artist is hard
The reference searching/gathering process is annoying
The system will
1. Greatly increase the efficiency
2. Greatly improve current user experience during the workflow
3. Integrate generative AI into visual worker's workflow. (Which is very important because a lot of artist / designer hate AI work)

## 視覺創意工作者的概念生成系統
### 動機
當設計師開始他們的工作時，腦力激盪不同的設計元素是至關重要的。大多數情況下，設計師會試圖收集足夠的視覺參考資料，包括照片、其他人的作品以獲得靈感。這個過程是設計流程中最困難的部分。它通常需要幾個小時甚至幾天才能完成。而且大多數設計師都難以產生好的設計概念。

這個系統旨在：
大幅提高效率
大幅改善當前工作流程中的用戶體驗
將生成式AI整合到視覺工作者的工作流程中（這一點非常重要，因為許多藝術家/設計師討厭AI作品）

為什麼不直接使用AI生成圖像？
就像Navipath給醫生那樣，AI生成圖像並不是藝術家/設計師工作流程的一部分。我們旨在將其與他們原有的工作流程整合
AI生成圖像缺乏細微的控制
AI生成圖像有時對於設計師/藝術家來說不夠精確，特別是當參考資料必須結構正確且清晰時。
我們的系統旨在提供一個更快的過程，以幫助設計師生成多個設計概念，每個概念都有其自己的參考圖像。

### 用戶
所有視覺設計師/部分藝術家

### 情境
娛樂設計師：為電影/視頻遊戲/電視節目設計場景
平面設計師：在多個地方設計主視覺

### 現有工作流程
給定一個設計主題，大多數視覺設計師首先會使用Google或Pinterest直接搜索相關圖像，當他們找到有興趣的圖像時，他們會將這些圖像保存到他們的參考文件夾或看板中。完成這個過程後，設計師會產生一些設計概念，並再次進行研究和視覺搜索。通過這個反覆的過程，設計師最終決定一個設計方向，並挑選一些參考資料來幫助他們完成設計。

### 系統設計
我們的系統將結合大型語言模型(LLM)、視覺生成AI和圖像標注模型來協助設計師迭代這一過程。
該系統旨在為設計師提供不僅是視覺參考，而且是概念生成助手。
從設計師給出的主要設計主題開始，我們使用LLM作為後端生成幾個作為選項的想法。
用戶可以選擇LLM給出的一個子主題，基於該子主題，LLM將生成另外一些選項以及一些圖像。
經過幾次迭代後，用戶可以選擇LLM整理過後的最終設計概念。根據這個設計概念，系統將在線搜索這些圖像，向用戶顯示前N張圖像，讓他們選擇喜歡的放入Reference Board。前N張圖像將使用CLIP相似度模型來決定。如果用戶想要，也會有一些AI生成的圖像供他們選擇。
經過更多的迭代後，用戶將得到一個徹底的設計概念，並根據設計概念有一套參考圖像。
如果用戶想要更多的設計概念，他們可以重複這個過程。

### 與現有解決方案的比較
根據我在動畫行業的經驗，我相當有信心說目前沒有一種高效的方法讓設計師使用AI解決方案來生成概念以及收集參考資料。通過上述系統，視覺工作者將節省大量時間，並獲得更高質量的設計概念和參考資料。

### 對用戶的價值
藝術家的概念生成過程是困難的
參考資料的搜索/收集過程是煩人的
該系統將：
大幅提高效率
大幅改善當前工作流程中的用戶體驗
將生成式AI整合到視覺工作者的工作流程中（這一點非常重要，因為許多藝術家/設計師討厭AI作品