Harnessing AI for Business Insight: Deploying LLM Summarization Pipelines

Large Language Models Human-AI Interaction GPT-4 Case Study Book Chapter
Project Overview:
This chapter proposes a business-oriented framework for building and evaluating LLM-powered text-summarization systems. It critiques single-number metrics such as ROUGE and BERTScore, arguing for multi-dimensional evaluation (relevance, factuality, fluency, coherence) and explicit consideration of human–computer-interaction factors and domain context. A detailed GPT-4 case study shows how the framework surfaces actionable insights from 1000 + YouTube comments on a smartphone launch, using sentiment-split summarizers, topic alignment, and a rich multi-dimensional data visualization.
📄
Read the Forthcoming Chapter (Oxford UP)
In The Oxford Handbook on the Foundations and Regulation of Generative AI (2024)
With Kartik Hosanagar & Ramayya Krishnan
SSRN preprint →
Key Contributions & Technical Execution
🛠️ Pipeline Engineering
Built a two-stage structured UGC pipeline (sentiment-split summarizers + comment-to-topic labeler) powered by GPT-4 and rendered insights in a color-coded dashboard.
📚 Research Review
Critically reviewed the state of LLM summarization, highlighting limitations of single-number metrics like ROUGE and BERTScore, and proposing a richer evaluation framework .
Key highlights
The “Myth of the Single Number”
ROUGE & BERTScore correlate weakly with business-facing quality dimensions, necessitating richer, multi-objective evaluation.
Human–AI-Business Interaction Matters
Trust, explainability, and workflow placement strongly mediate the value managers glean from LLM outputs. Effective summarization pipelines must consider these factors explicitly and understand not just the NLP properties of a their LLM applications, but also the human and business context in which they operate.
Proposed context ontology for product feedback summarization pipelines
Can be leveraged to guide the development of summarization pipelines in new domains by providing a set of common context types and prompts. This could help strike a balance between the flexibility and power of LLMs and the nuance and context needed to leverage them in task-specific settings.