"We Need Structured Output": Towards User-centered Constraints on Large Language Model Output
Citations Over TimeTop 1% of 2024 papers
Abstract
Large language models can produce creative and diverse responses. However, to integrate them into current developer workflows, it is essential to constrain their outputs to follow specific formats or standards. In this work, we surveyed 51 experienced industry professionals to understand the range of scenarios and motivations driving the need for output constraints from a user-centered perspective. We identified 134 concrete use cases for constraints at two levels: low-level, which ensures the output adhere to a structured format and an appropriate length, and high-level, which requires the output to follow semantic and stylistic guidelines without hallucination. Critically, applying output constraints could not only streamline the currently repetitive process of developing, testing, and integrating LLM prompts for developers, but also enhance the user experience of LLM-powered features and applications. We conclude with a discussion on user preferences and needs towards articulating intended constraints for LLMs, alongside an initial design for a constraint prototyping tool.
Related Papers
- → Workflow Reuse in Practice: A Study of Neuroimaging Pipeline Users(2014)15 cited
- A Workflow Event Logging Mechanism and Its Implications on Quality of Workflows(2010)
- Workflow Analysis using Graph Kernels.(2010)
- → Utilizing Tags for Scientific Workflow Recommendation(2019)2 cited
- Configurable and Collaborative Scientific Workflows(2014)