A study on question answering in the ceramics domain using large language models with retrieved triples and generated textual knowledge prompts

Abstract

The propensity of Large Language Models (LLMs) to hallucinate poses a significant challenge in specialized domains with fragmented knowledge, such as ceramic question answering. A critical yet underexplored factor is how the format of retrieved knowledge—structured versus unstructured—affects model accuracy. In this study, we develop a Retrieval-Augmented Generation (RAG) framework for the Chinese ceramic domain. A domain-specific knowledge graph was constructed from heterogeneous sources, and a BERT-based module was employed for hop prediction and path reasoning. The retrieved subgraphs were provided to LLMs either as structured triples or converted textual descriptions. We evaluate four LLMs (GPT-3.5, GPT-4, ChatGLM2-6B, LLaMA-2-7B) on a Chinese ceramic QA benchmark. Across all models, structured triples yield higher accuracy, with ChatGLM2-6B achieving the best performance (97.24%). Smaller models benefit more from structured representations, while larger models process textual descriptions more effectively. These results highlight the importance of knowledge representation for reducing hallucination in domain-specific QA. However, as all experiments are conducted on Chinese datasets within a single domain, the generalizability of our findings across languages and other vertical domains remains an open question. We discuss this limitation and outline directions for future multilingual and cross-domain validation.