Enterprise AI Analysis
Using Large Language Models to Generate, Validate, and Apply User Intent Taxonomies
Analyzing user intents in information access, especially for emerging AI-driven chat, is a complex challenge. Our novel methodology leverages Large Language Models (LLMs) with Human-in-the-Loop (HITL) to create, validate, and apply user intent taxonomies. This approach delivers cost-effective, scalable, and verifiable insights, significantly reducing bottlenecks in intent-focused research and enabling deeper understanding of user behavior across diverse modalities.
Quantifiable Impact & Efficiency Gains
Our LLM-HITL methodology not only streamlines the complex process of user intent analysis but also delivers measurable improvements in reliability and efficiency for enterprise applications.
0.76κ
Human-Human Agreement (Kappa)
0.72κ
Human-GPT-4 Agreement (Kappa)
0.85κ
GPT-4 Internal Consistency (Fleiss' Kappa)
50%+
Efficiency Gains in Taxonomy Development
0%
'Other' Labels on Human Annotation
Deep Analysis & Enterprise Applications
Select a topic to dive deeper, then explore the specific findings from the research, rebuilt as interactive, enterprise-focused modules.
Our innovative methodology combines the generative power of LLMs with critical human oversight to ensure robust, validated user intent taxonomies for complex data analysis.
Enterprise Process Flow
Taxonomy Generation (Phase 1)
→
Taxonomy Validation (Phase 2)
→
Taxonomy Application (Phase 3)
This iterative Human-in-the-Loop approach ensures that LLM-generated taxonomies are both scientifically rigorous and practically applicable for enterprise needs.
The rigorous validation process ensures that LLM-generated taxonomies meet high standards of accuracy, consistency, and comprehensiveness, making them reliable for critical enterprise decision-making.
0.72 - 0.76 κ
High Inter-Rater Agreement (Human-Human & Human-LLM)
The study demonstrates strong inter-coder reliability (0.7620 Kappa human-human, 0.7212 Kappa human-GPT-4) and high internal consistency (0.8516 Fleiss' Kappa for GPT-4), validating the robust and objective nature of LLM-generated taxonomies with human oversight.
| User Intent | Search Requests (%) | Chat Requests (%) |
|---|---|---|
| Information Retrieval | 54.5% | 45.5% |
| Leisure | 51.1% | 48.9% |
| Ask for Advice or Recommendation | 43.3% | 56.7% |
| Create | 41.3% | 58.7% |
| Learn | 31.1% | 68.9% |
| Figure 4 reveals a distinct shift in user intent preferences between search and chat modalities. While 'Information Retrieval' and 'Leisure' are relatively balanced, 'Create', 'Learn', and 'Ask for Advice or Recommendation' intents show a significant tilt towards AI chat, highlighting its emerging role in complex and generative tasks. | ||
Implementing LLM-powered taxonomies allows enterprises to quickly adapt to evolving user behaviors in new modalities like AI chat, driving more relevant and personalized information access.
Case Study: Analyzing User Intents in Search vs. Chat with LLMs
Our end-to-end pipeline leveraged LLMs with Human-in-the-Loop to generate and apply a new user intent taxonomy to heterogeneous log data from Bing Search and Bing Chat. This revealed a significant shift: users increasingly prefer AI chat for complex tasks like 'Create,' 'Learn,' and 'Ask for Advice or Recommendation,' while traditional search remains strong for information retrieval and leisure. This demonstrates LLMs' ability to uncover critical, modality-specific user behavior insights.
This rapid analysis capability helps developers adapt systems to evolving user intents quickly and cost-effectively, ensuring AI tools remain relevant and effective.
Quantify Your Enterprise AI Efficiency Gains
Use our interactive calculator to estimate the potential time and cost savings your organization could achieve by implementing LLM-powered data analysis solutions.
Your AI Taxonomy Implementation Roadmap
Our proven 6-phase methodology ensures a systematic, validated, and efficient deployment of LLM-powered user intent taxonomies within your organization.
Identify Application & Data
Clearly define the purpose of the taxonomy, what user intent means for your application, how it will be used, and prepare clean log data, considering input requirements for your chosen LLM.
Build & Fine-Tune Taxonomy
Craft a detailed prompt for the LLM, specifying application context, good taxonomy criteria, and constraints. Bootstrap multiple taxonomy versions to test sensitivity and consolidate.
Measure Comprehensiveness & Consistency
Use the LLM to label training data, checking for a low percentage of 'Other' labels. Assess the LLM's consistent application of definitions across samples, possibly with multiple runs.
Improve Taxonomy Clarity
Request the LLM to expand category definitions and examples (both positive and negative) to enhance clarity and ensure unambiguous understanding for human and machine annotators.
Measure Validity & Accuracy
Apply the refined taxonomy to the data used for generation. Manually check a random sample to confirm assigned labels follow definitions, ensuring internal validity and accuracy.
Perform Annotations & Measure Conciseness
Run the final taxonomy on new test data, ensuring all important concepts are covered (low 'Other' rate) and no categories have insufficient samples. Compute Inter-Coder Reliability (ICR) with human annotators.
Ready to Transform Your Data Analysis with AI?
Leverage the power of LLMs and human expertise to unlock unprecedented insights from your data. Our proven methodology ensures accuracy, scalability, and efficiency.
Ready to Get Started?
Book Your Free Consultation.
Let's Discuss Your AI Strategy!
Lets Discuss Your Needs
Select a Date
Sun
Mon
Tue
Wed
Thu
Fri
Sat