Paper link | Note link | Code link | ICLR 2024 :::success Thoughts In this work, they have presented the first comprehensive study of encoding graph-structured data as text for consumption by LLMs. ::: Abstract This paper performs the first comprehensive study of encoding graph-structured data as text for consumption by LLMs.

Paper link | Note link | Code link | AAAI 2024 :::success Thoughts They propose Graph Neural Prompting (GNP), a novel plug-and-play method to assist pre-trained LLMs in learning beneficial knowledge from KGs. ::: Abstract To reduce the limitation of the large language model, the existing work enhances pre-trained LLMs using grounded knowledge.

Paper link | Note link | Code link | EMNLP 2023 :::success Thoughts They proposed a general framework for improving the zero-shot reasoning ability of LLMs over structured data. ::: Abstract

Paper link | Note link | Code link | AAAI 2024 :::success Thoughts Graph of Thoughts (GoT) enhances LLMs’ capabilities through networked reasoning. It enables a fine-grained control over individual thoughts. They illustrate several use cases for GoT. :::

Paper link | Note link | Code link | ICLR 2024 :::success Thoughts This paper purposes reasoning on graphs (RoG). Synergizing LLMs with KGs to conduct faithful and interpretable reasoning. :::

Paper link | Note link | Code link | ICLR 2024 :::success Thoughts ::: Abstract Background Method image

Paper link | Code link | EMNLP 2023 :::success Thoughts This study purposed on method called MolCA. It is a molecular language modeling, which aims to enable LMs to perceive 2D graphs for molecule-totext generation. :::

Paper link | Note link | Code link | ICML 2023 :::success Thoughts They attempt to develop an embodied language model by integrating multimodal information, including images. ::: Abstract

Feb Study Group 02/15 March Study Group 03/02

Paper link | Code link | ICML 2023 :::success Thoughts This study uses human-in-the-loop rationalization approach for multimodal transformers. ::: Abstract Visual-langauge models (VLM) rarely align with user’s rationales for specific answers.

Paper link | Note link | Code link | NeurIPS 2023 :::success Thoughts This study demonstrated the effectiveness of visual instruction tuning. They introduced an automatic pipeline for generating language-image instruction-following data, which was used to train LLaVA, a multimodal model capable of following human intent to complete visual tasks. :::

Paper link | Code link | EMNLP 2023 :::success Thoughts This study introduces the first joint study on the compression and debiasing problems of Vision Language Pre-trained Model (VLP) for the Visual Question Answering (VQA) task. ::: Abstract Despite their strong performance on Visual Question Answering (VQA) tasks, Vision Language Pre-trained Models (VLPs) struggle with two issues:

Paper link | Note link | Code link | EMNLP 2023 :::success Thoughts This study conducted evaluation experiments on several LVLMs to examine their susceptibility to object hallucination. ::: Abstract Large vision-language models (LVLMs) can suffer from issues like object hallucinations, where generated descriptions include objects that don't match the target images.

Paper link | Code link | CVPR 2022 :::success Thoughts This paper introduces a new task called Audio-Visual Question Answering (AVQA). They build a dataset with 45,867 question-answer pairs covering various audio-visual modalities and question types. Additionally, they propose a spatio-temporal grounding model to enhance fine-grained scene understanding and reasoning.

Paper link | ICML 2023 :::success Thoughts This study presents a simple yet effective representation learning method, RLEG, guided by diffusion-based embedding generators. Diffusion models generate embeddings online to aid in learning effective vision-language representations. Pretrained generators transfer embeddings between vision and language domains, and these generated embeddings serve as augmented samples for contrastive learning.

Paper link | Code link | AAAI 2024 :::success Thoughts Reinforcement Learning (RL) for sequence generation models can be computationally expensive. This study proposes an efficient RL method (ESRL) using two-stage and dynamic sampling approaches. :::

Paper link | Code link | AAAI 2024 :::success Thoughts This study tackles the primary challenges associated with the factuality evaluation of large language models (LLMs) and introduce OntoFact—an adaptive framework designed for detecting inaccuracies through a sophisticated Ontology-level Relation Linking (ORL) mechanism. This framework aims to uncover knowledge deficiencies at the ontology level. :::

Paper link ｜ EMNLP 2023 :::success Thoughts This study uses reinforcement learning (RL) and identifies relevant and irrelevant tags to enhance the process of retrieving FAQ questions. ::: Abstract In online customer service, efficiently retrieving FAQ questions is crucial.

Paper link | Code link | EMNLP 2023 :::success Thoughts This study use offline RL to improve the quality and utility of open-domain dialogue system. ::: Abstract In open-domain dialogue systems, maintaining a consistent persona is crucial.

Paper link | Note link | Code link | ICML 2023 :::success Thoughts This article proposes the ELLM method, which uses a large language model to generate goals based on the current state of the agent, guiding reinforcement learning to effectively explore in an environment lacking external rewards. However, this method is dependent on the accuracy of the environment characteristics and language model. :::