# Jan: User Journey (Feb 2024) ## How are our products used? I'd like to articulate the vision for our 18 month roadmap in a User Journey, so that Jan Core Team and Contributors understand what we are setting out to achieve. - Jan Desktop - Jan Home Server - Jan Mobile - Jan Hub - Jan Enterprise Server This is meant as a thought exercise to provoke discussion. Please use the comments to agree and critique. ## Ideal Customer Persona **Background** Robert* is an `Product Manager` at `Databricks`, a team lead on ML pipelines that sells to Enterprise. Robert has an engineering background (6 years as a Java Engineer), and while he no longer writes production code, he is still able to handle basic technical setup and write straightforward Python and Javascript scripts. **Interest** Robert is keenly aware of the latest developments in open source AI: - Youtube channels: [WorldofAI](https://www.youtube.com/@intheworldofai), [Fahd Mirza](https://www.youtube.com/@fahdmirza), [Mervin Praison](https://www.youtube.com/@MervinPraison), [Kaparthy](https://www.youtube.com/@AndrejKarpathy) and [sentdex](https://www.youtube.com/@sentdex) - Reads [r/localllama](https://www.reddit.com/r/LocalLLaMA/) - AI Newsletters: [Big Brain](https://www.bigbraindaily.com/) and [AI Breakfast](https://aibreakfast.beehiiv.com/). Robert wants to tinker and experiment with these AI tools in his personal life. Even though he nominally works in AI, Databricks' version of AI isn't relevant to him and he doesn't see any of it improve his own life. He's sure that a lot of these AI tools can go a long way towards optimizing his own workflow. **Constraints** However, Robert just had a new baby and he's not able to devote the time to AI. He's always said he'll play with Mistral locally over the weekend, but has never gotten to it. Robert has daily use computer is a Macbook Pro with 16gb of RAM, and an iPhone with 1TB of storage and [8gb LPDDR5 RAM](https://9to5mac.com/2023/11/18/iphone-ram-list/). _*Inspired by a real person_ ## Month 1 ### Week 1: Robert Downloads Jan Desktop **Discovery** Robert hears about Jan from one of his engineers who shared it in a Slack channel, and downloads Jan onto his Macbook. ![image](https://hackmd.io/_uploads/S1qC_nsja.png) **Onboarding** - Jan has a bunch of initial screens like [Raycast's Onboarding Screens](https://pageflows.com/post/desktop-web/onboarding/raycast/) - Robert understands that Jan is a Copilot-like tool, that he can customize and use anywhere - Robert downloads a recommended local model (mirror download from Jan Hub) - Robert adds an API key for OpenAI, Mistral or Bard - Robert opts in to add his email to receive new product updates & research insights - Robert follows a few tooltips on how to personalize Jan, e.g. - adding docs/instructions to Jan Assistant - customizing Jan Data Folder - customizing the keyboard shortcut / inline cmd to invoke Jan ![image](https://hackmd.io/_uploads/HkBI5hoo6.png) ### Weel 1-2: Robert uses Jan daily, in his natural workflows. **Jan's Copilot with Inline Copywriting** - D1: Robert opens his Mac Mail app to check his email - Robert responds to an email by pressing `Alt-Space`, and prompting Jan to draft an email reply for him - Robert responds to an email written in Turkish by highlighting, right-click menu -> Jan, and tells Jan to translate it for him - Robert reads a website by right-clicking a page and asking Jan to summarize it for him ![image](https://hackmd.io/_uploads/r1pzshsop.png) *Inspiration: ChatGPT wrappers, e.g. ChatNBX* **Jan's Copilot with Prompts library** - Robert opens Obsidian, where he's has a note with a braindump of text he's been collecting for a blog post - Robert presses `Alt-Space` to summon Jan's launcher, with the copied text already in the context window - Robert finds a good prompt library for a good blog post prompt and lets Jan generate a draft blog post - Jan allows him to paste the drafted blog post in Microsoft Word ![image](https://hackmd.io/_uploads/SyjVshsoa.png) *Inspiration: Perplexity.ai* **Jan's Copilot with Web Search** - Robert turns on Jan's Web Search capability by adding his [SerpAPI Key](https://serpapi.com/duckduckgo-search-api) - He is able to have a Perplexity-like experience that searches the web to answer his queries ### Week 2-4: Robert customizes Jan ![image](https://hackmd.io/_uploads/rJyl9Uhja.png) *Alfred's UI for dragging files into Alfred's launcher* **Customizing Jan's Copilot with custom data** - Robert sees the value of Jan and wants to customize it to his needs - He experiments with Jan's memory capability by turning on memory, which enables Jan to remember interactions between threads - He experiments by adding documents to Jan who "learns" them - Behind the scenes, Jan ingests documents into a local RAG database - Ideally, this should work with both OpenAI or local models - ie. Allow user to switch models while still using same RAG database ![image](https://hackmd.io/_uploads/HkX_I8hjT.png) *Data Connectors, revisited* **Customizing via Action Connectors** - Robert decides to connect Jan to his personal systems so Jan is able to understand his context - Jan integrates Integration-as-a-Service providers like Zapier or Microsoft Fabric - Immediately connect it to thousands of tools - LLMs have some sort of guidance library or function calling that can learn how to call these APIs - Robert connects Jan to Gmail: - Email: Jan is email-aware and Jan can function call into Gmail to pull up relevant emails - Calendar: Jan is calendar-aware and can function call into GCal to pull up relevant events - Robert connects Jan to Obsidian by pointing Jan to his Obsidian data folder - Jan ingests Obsidian into a RAG database at a certain time interval - Jan is able to function call into Obsidian to create notes - Jan has human-in-the-loop approval for function calls - Will seek permission before making function calls to Gmail, Google Calendar or Obsidian create note ![image](https://hackmd.io/_uploads/SysriUnoT.png) *[Promptly](https://www.trypromptly.com/) or [LLMStack's](https://llmstack.ai/) UI for configuring an Agent* **Customizing via Custom RAG** - Robert starts to customize Jan even more by tinkering with his Copilot - He opens up Jan Copilot settings, and adds a folder of work documents to Jan so it has work-specific knowledge - Behind the scenes, this is indexed into a local RAG database - Robert is able to customize ingestion on a per-file basis by using Settings to configure ingestion strategy - Each extension (e.g. `.pdf`) has a default ingestion script that can be overridden - He can define a custom ingestion script for specific files - Can implement custom Ingestion pipelines by writing a Llamaindex script ![image](https://hackmd.io/_uploads/HkgX9U2jp.png) *Amazon's current fine-tuning UI. Can we build a more elegant interface?* **Customizing via Fine-tuning and Micro-training** - Robert also constantly "micro-trains" his model with positive and negative Q&A pairs - Robert also experiments with the fine-tuning function in Jan - Creates a synthetic Q&A dataset based on knowledge documents - Robert reviews the Q&A dataset before starting the fine-tuning process - Robert reviews the finetuned model and can revert to the previous model if he doesn't like it - Robert then uses his customized Jan in the natural flow of his work - He then uses Jan to answer some customer support emails, and write a Product Spec for his team - He also uses Jan to draft a couple of replies in Slack and Whatsapp **"Incognito" mode for Copilots** - Robert is able to use "Incognito" mode for his Copilot - This turns off Jan's ability to remember interactions between threads - #censored ## Month 2 ### Week 4: Robert sets up Jan Home Server and Jan Mobile **Using Jan on the go** - Robert wants to bring his personalized Jan on mobile - Robert uses Jan Desktop's server mode and exposes it to the internet via the [ngrok](https://ngrok.com/) - He downloads the Jan Mobile app and connects it to his Jan Desktop's ngrok address - He is able to access his customized Jan on his phone ![image](https://hackmd.io/_uploads/ry4xv3os6.png) *Inspiration: Google Keyboard's Translate* **Using Jan Mobile's Keyboard Accessory View** - Jan Mobile ships with a [Keyboard Accessory View](https://reactnative.dev/docs/inputaccessoryview) called Jan Keyboard - Robert switches to Jan Keyboard to use AI in apps, prompting and getting a response directly in-line - Robert uses Jan Keyboard to draft and reply to Whatsapp, Telegram and Zalo messages ![image](https://hackmd.io/_uploads/ByW4v3soT.png) *Inspiration for Jan Home Server* **Setting up Jan Home Server** - Robert decides that keeping his laptop on 24/7 is not the right way to use Jan, and wants to set up Jan Home Server instead - Robert reads Jan's Hardware Guide in its docs and buys a ready-made 4090 AI rig off Amazon (Jan gets 6% in affiliate fees) - Robert installs Jan Home Server on the 4090 cluster - Robert follows the instructions to push his customized Jan from his Macbook to his Jan Home Server (ie. copy-and-paste files) - Robert follows the instructions on Jan's site on how to expose his Jan Home Server to the internet via ngrok - Robert uses his Jan Mobile to connect to his Jan Home Server ## Month 3 ![image](https://hackmd.io/_uploads/SJJt6I3j6.png) *OS-Copilot paper and demo implementation* **Robert lets Jan use the computer** - Robert turns on Jan's experimental feature which allows Jan to control his computer - [Mac Accessibility Apps](https://support.apple.com/en-sg/guide/mac-help/mh43185/mac) - [Windows Automation API](https://www.codemag.com/article/0810042/Windows-Automation-API-3.0-Overview) - Jan is able to "use" the computer with human-in-the-loop - Robert watches Jan take control of the computer - Inspired by [OS-Copilot](https://www.youtube.com/watch?v=IVapbjYqCIA) - Inspired by [MultiON](https://www.youtube.com/watch?v=nU3OulGPvH8) - Jan is able to "observe" Robert using the computer and record macros - Similar to [Playwright Test Generator](https://playwright.dev/docs/codegen) - Robert is able to record macros and give them names and descriptions - Macros as registered as callable functions in LLMs ![image](https://hackmd.io/_uploads/S18vWD3op.png) *Voyager: Agents that self-learn* **Jan "self-reflects" and self-trains using spare processor cycles** - Jan actually has an "automatic tuning" that happens when there are spare MLX or GPU cycles - Jan self-reflects and improves Robert's customized model automatically - This is more powerful than RAG - This is more powerful than - This builds on Research discoveries from the: - [Voyager Paper](https://openreview.net/forum?id=ehfRiF0R3a) - [Eureka Paper](https://arxiv.org/abs/2310.12931) ## Open Questions - How does Foundry fit into this?