---
# System prepended metadata

title: UI copilot - Reading Group

---

# UI copilot - Reading Group

## Template

- tldr;
- key takeaways
- implications on the project
    - questions
    - TODOs
- (optinal) details 


## AgentTuning: Enabling Generalized Agent Abilities For LLMs

- tldr;
    - introduces AgentTuning
        - 1) AgentInstruct dataset, an instruction tuning dataset for agents built on 6 AgentBench tasks 
            - high-quality ReACT (CoT + decision making) GPT4 trajectories
        - 2) an hybrid tuning strategy
            - mix between training on AgentInstruct and a general domain instructions (filtered ShareGPT dataset)
    - masterful evaluation framework
        - there's 6 held-in tasks/domain which you finetune on
        - there's 6 helf-out domain for more evaluation (includes MiniWob and WebArena)
        - there's 3 General tasks to evaluated general capabilities
    - OS AgentLM, a suite of finetuned LLAMA2
- key takeaways:
    - AgentBench prescribes 1-shot (vs 0-shot) bc of the stringent requirements for the output format in agent tasks
    - trajectory filtering (for successful ones) is important for both held-in and -out performance
    - AgentInstruct drastically increases held-in performance, almost matching GPT4
    - AgentInstrcut also increases held-out performance, highlighting some general agent capabilities
    - AgentInstruct dramatically decreases elementary errors (aligns w/ Allac's hope/intuition)
    - bigger held-out boost in 70B vs 13-7b
        - the general capabilites emerging from scale are important for general agents 
    - Mixing Agent and General instruction is important, espcially to get the 70B held-out boost
- implications on the project
    - questions:
        - should we generalize the scope to "LLM-as-agents" in the spirit of maximizing impact?
            - should we consider having helf-out domains?
        - is it fair game to finetune on GPT4 traces, for commerciability purposes?
        - how does the environement provide feedback in Mind2Web? ins't it a dataset?
        - Should we use their AgentInstruct dataset for finetuning?
        - Should we add AgentLM to our baselines (it's another LLAMA2, so easy to support)
        - could we do error analysis like them, i.e. programatically categorize the errors into `invalid actions`, `function call error`, `function repetition`, `refuse to answer`
    - TODOs:
        - checkout their codebase, we might want to reuse some stuff
        - checkout how they prompt for MiniWob and WebArena
        - checkout what is the observation and action space for MiniWob and WebArena
 



