Pipeline Methodology — Agent Intelligence Documentation

Pipeline Overview

The pipeline processes raw customer data through 4 specialized AI agents in sequence. Each agent adds a layer of intelligence, passing enriched outputs downstream. The core design principle is NO HARDCODING — every metric, threshold, and decision is derived from the actual data.

DAKA

v8.1.0

Data Analysis & Knowledge Acquisition

"Let me understand the DATA"

→

SEKI

v7.3.0

Segmentation Expert & Knowledge Intelligence

"Let me understand WHO"

→

LOKI

v7.2

Persona Discovery & Behavioral Psychology

"Let me understand WHY"

→

STRATEGI

v9.0

Strategic Growth Intelligence & Wave Planner

"Let me plan WHAT to do"

DAKA — Data Analysis & Knowledge Acquisition

Version 8.1.0 · Enterprise-Grade Universal BI Intelligence Agent

"Before we act, let me deeply understand your data — its quality, structure, risks, and hidden signals."

What It Does

Validates file integrity, schema, and data quality
Profiles every column — types, distributions, outliers
Detects industry, market, currency, and entity type from data patterns
Identifies root causes of business problems via dual correlation analysis
Assesses data readiness for campaign execution (Grade A–F)
Computes segmentability (Hopkins Statistic) and CLV trajectory
Performs waterfall decomposition (Pyramid Principle)
Detects PII, data leakage, and security risks

Libraries & Tools

pandas DataFrame operations & profiling
numpy Numerical computations
scipy.stats t-distribution for p-values
sklearn StandardScaler + NearestNeighbors (Hopkins)
regex PII pattern detection
LLM Router Goal-specific BI interpretation
Knowledge Layer Industry KPI templates

Statistical Methods

Method	Purpose	Key Parameters
Pearson Correlation	Detect linear relationships between features	r ≥ 0.5 (moderate), r ≥ 0.7 (strong)
Spearman Rank Correlation	Detect monotonic/nonlinear relationships	Divergence > 0.15 flags nonlinear
Dual Correlation Selection	Takes stronger of Pearson vs Spearman per pair	Best-of-two approach (v8.1)
Bonferroni Correction	Control family-wise error rate across multiple tests	α × n_tests
Benjamini-Hochberg FDR	Control false discovery rate	Rank-based threshold at α = 0.05
Hopkins Statistic	Assess whether data has natural cluster structure	H > 0.7 = segmentable, H ≈ 0.5 = random
IQR Outlier Detection	Identify statistical outliers per numeric column	Fences: Q1 − 1.5×IQR, Q3 + 1.5×IQR
Skewness Analysis	Detect highly skewed distributions	\|skew\| > 2.0 = highly skewed
Coefficient of Variation	Detect low-variance (uninformative) features	CV < 0.01 = low variance
CLV Proxy (Tenure × ARPU)	Estimate customer lifetime value trajectory	GROWING / STABLE / DECLINING

Methodology Flow

Load & Validate — Load CSV data, check file integrity, schema (no duplicate columns), and data completeness
Profile & Detect Types — Profile every column (dtypes, nulls, cardinality). Auto-detect phone numbers, emails, booleans, currencies, ZIP codes
Context Detection — Infer industry (10 industries), entity type, market/geography, and currency from data patterns with confidence scoring
Semantic Inference — Classify columns by role (identity, revenue, engagement, value, usage). Identify KPI candidates per industry
Quality Analysis — Missing data (nulls + placeholders), duplicates, outliers (IQR), consistency (date order, negative values), cardinality
Correlation Network — Dual Pearson+Spearman with p-values, Bonferroni + FDR correction, trivial pair filtering, redundancy detection (r ≥ 0.95)
Goal Investigation — Check requirements (target column present?), compute baseline metrics, identify root causes via correlations
Waterfall Decomposition — Revenue tree, demographic cuts, digital footprint analysis, feature impact assessment, crosstab generation
Readiness Scoring — Weighted 6-component score (validation 15%, completeness 25%, quality 20%, type cleanliness 10%, consistency 10%, goal alignment 20%) → Grade A–F
Advanced Metrics — Hopkins statistic (segmentability), CLV proxy (tenure×ARPU trajectory), effort estimation (hours to fix issues)

Key Thresholds

Missing Data Critical

> 50%

Redundant Correlation

r ≥ 0.95

Hopkins Segmentable

H > 0.70

Root Cause Min Corr

r ≥ 0.30

Outlier IQR Multiplier

1.5

Max Correlation Edges

200

Grade A Threshold

≥ 90%

Grade F Threshold

< 40%

SEKI — Segmentation Expert & Knowledge Intelligence

Version 7.3.0 · Goal-Aligned Intelligent Customer Segmentation (Pipeline v10.0)

"I don't just cluster — I create segments aligned to your business goals with stability guarantees."

What It Does

Creates goal-aligned customer segments using ML clustering
Selects optimal number of segments (k) via silhouette optimization
Compares multiple clustering models and selects the best
Validates segment stability across 20 runs (Adjusted Rand Index)
Calculates goal propensity with bootstrap confidence intervals
Auto-splits dominant segments (>40% of base) into sub-groups
Generates campaign-ready labels ("Retain: HV Active")
Computes segment migration paths with feasibility scoring

Libraries & Tools

sklearn.cluster MiniBatchKMeans (memory-efficient)
sklearn.metrics silhouette_score, adjusted_rand_score
sklearn.preprocessing StandardScaler
pandas + numpy Data ops
joblib Model serialization
LLM Router Segment naming & scoring
Knowledge Layer Industry model configs

Statistical & ML Methods

Method	Purpose	Key Parameters
MiniBatchKMeans	Primary clustering algorithm (memory-efficient for 200K+ rows)	batch=2048, n_init=10, max_iter=200
Silhouette Score	Select optimal k — measures cluster cohesion vs separation	Tests k ∈ [3,4,5,6,7,8,9]
Adjusted Rand Index (ARI)	Measure segment stability across 20 random seeds	Campaign block if ARI < 0.50
Bootstrap Resampling	95% confidence intervals for propensity scores	100 bootstrap samples
Feature Interaction Detection	Find feature products that better predict goal KPI	Must exceed individuals by ≥ 0.05
Collinearity Removal	Drop highly correlated features before clustering	Threshold: r ≥ 0.95
Multi-Model Comparison	Compare behavioral, value, churn, lifecycle, RFM models	Composite score with silhouette weight=50
L2 Centroid Distance	Migration path feasibility between segments	HIGH < 1.5, MEDIUM < 3.0, LOW ≥ 3.0
Goal-Aware Feature Weighting	Boost features aligned to business goal	Target KPI: ×1.8, Model-specific: ×2.0

Methodology Flow

Load DAKA Outputs — Unified dataset, domain config, KPI candidates, goal requirements, baseline metrics
Feature Selection — Filter numeric columns, apply semantic role mapping, exclude IDs. Detect feature interactions (product terms that better predict goal)
Goal-Aware Weighting — Boost target KPI (×1.8), supporting features (×1.3), model-specific features (×2.0). Clamp to [0.5, 5.0]
Data Preparation — Sample up to 100K rows, standardize with StandardScaler, apply per-feature weights, drop collinear features (r > 0.95)
Multi-Model Comparison — Run up to 6 model types (behavioral, value, churn, lifecycle, RFM, kmeans) with composite scoring. Silhouette delta override if gap ≥ 0.12
Stability Validation — Run best model 20× with different seeds, compute pairwise ARI. Hard block if stability < 0.50. Campaign-ready flag
Dominant Segment Split — If any segment > 40% of base, auto-split into 3 sub-segments using re-clustering
Propensity & Risk Tiers — Calculate goal propensity per segment with bootstrap CIs. Assign tiers: HIGH_RISK / HIGH_OPPORTUNITY / NEEDS_ACTION
Migration Paths — Compute centroid distances, boundary users (midpoint ratio < 0.55), revenue uplift per path. Top 10 most feasible
Campaign Labels — Auto-generate "Action: Who" labels (max 25 chars). Actions: Retain, Reactivate, Upsell, Reward, Nurture, Grow

v10.0 Pipeline Enhancements

ARI Consensus with Mapped-Label Mode Assignment
Maps bootstrap run labels to common basis before computing mode. Improves consensus robustness across multiple clustering runs.

Silhouette 0.480 Achieved via Behavioral Model
Behavioral model optimization produces silhouette coefficient of 0.480, delivering improved segment cohesion and separation without sacrificing interpretability.

Key Thresholds

Stability Hard Block

ARI < 0.50

Stability Warning

ARI < 0.70

Dominant Segment

> 40% share

Bootstrap Samples

100 (95% CI)

Stability Runs

20 seeds

High Propensity

≥ 0.30

Silhouette Excellent

> 0.50

Max Features

100K rows sampled

LOKI — Persona Discovery & Behavioral Psychology Engine

Version 7.2 · LLM-First Intelligent Persona Discovery (Pipeline v10.0)

"I don't match templates — I discover personas by reasoning about data patterns and human psychology."

What It Does

Discovers personas via LLM reasoning (not template matching)
Models psychology: motivations, barriers, triggers, emotional state
Statistically grounds psychology claims via z-score testing
Infers communication channels from actual data signals
Consolidates similar personas using trait-weighted similarity
Computes multi-source confidence (closeness × certainty × completeness)
Calculates persona evolution windows with urgency levels
Graceful degradation: LLM → 8 fallback archetypes → deterministic rules

Libraries & Tools

LLM Router Persona discovery & psychology reasoning
numpy Z-score computation, sigmoid functions
pandas User-level enrichment & aggregation
joblib SEKI model artifact loading
itertools Pairwise similarity computation
Knowledge Layer Industry persona templates
math Exponential decay for confidence

Methods & Techniques

Method	Purpose	Key Parameters
LLM Persona Discovery	Reason about segment data to discover psychology	Temp: 0.3, max 2 retries
Z-Score Grounding	Statistically validate psychology claims against data	STRONG: \|z\| ≥ 2.0, MODERATE: \|z\| ≥ 1.5
Trait-Weighted Jaccard Similarity	Merge similar personas (adaptive threshold)	Motivations: 3.0, Barriers: 3.0, Triggers: 2.0
Sigmoid Centroid Certainty	Measure segment separation quality	sigmoid(min_dist / scale)
Exponential Decay Closeness	User-level confidence from centroid distance	exp(−dist / median_dist)
Composite Confidence	Combine 3 signals into per-user confidence	45% closeness + 35% certainty + 20% completeness
Tenure-Based Evolution Windows	Optimal intervention timing per persona	p25/p50/p75 tenure quartiles
Data-Driven Channel Inference	Infer channels from actual app/SMS/USSD signals	Keyword matching on column names

v10.0 Pipeline Enhancements

Iterative Jaccard Enforcement
Up to 10 iterations eliminating overlapping persona pairs with Jaccard similarity > 0.5, ensuring distinct and non-overlapping psychological profiles.

DIFFERENTIATION_POOL
12 motivations, 10 barriers, 10 triggers available for forced diversity. Ensures psychological profiles across personas maintain maximum distinction.

Goal Overlay Protection
Only fills MISSING psychology fields without overwriting enforced fields. Preserves critical psychological constraints while enriching incomplete profiles.

4-Component Tiered Fallback Naming
Persona names follow {Engagement} {Value} {Behavioral} {Psychology} structure with deterministic fallback to archetypes when data is insufficient.

Barrier Enforcement with BARRIER_POOL
Systematically assigns barriers from a curated pool of 10 barrier types, ensuring realistic and grounded psychological constraints for each persona.

Methodology Flow

Load Upstream Data — SEKI segments with users, domain config, segment profiles (with propensity), optional model artifact
Intelligent Discovery — For each segment: call LLM with key features, propensity, root causes. Discover persona name, goals, psychology, channels, messaging tone
Statistical Grounding — For each psychology claim: map concept → keywords → find matching data columns → compute z-score. Tier: STRONG / MODERATE / WEAK / UNGROUNDED
Channel Inference — Detect app engagement (moa_*, pwa_*), SMS signals, USSD signals. Merge data-driven channels with LLM suggestions (data gets priority)
Persona Consolidation — Compute trait-weighted Jaccard similarity between all persona pairs. Merge if similarity > adaptive threshold (0.80 + 0.01×segments)
Confidence Computation — Per user: closeness (45%, exp decay from centroid), certainty (35%, sigmoid of centroid separation), completeness (20%, KPI coverage)
Bounded Overrides — Non-MOA + high revenue (90th pctl) → conversion target. MOA + low usage (10th pctl) → fragile starter. Adds reason codes
Evolution Windows — Per persona: compute tenure quartiles. Map goal type to urgency (CRITICAL for churn, HIGH for dormant, MEDIUM for growth/engagement)

8 Fallback Archetypes

Dormant Mass — HIGH_RISK + Not Engaged + Large segment

Non-Digital At-Risk — HIGH_RISK + Not Engaged + Small

Fragile Starter — HIGH_RISK + Engaged (at risk of leaving)

Transitional — MEDIUM_RISK + Not Engaged

Core Mainstream — MEDIUM_RISK + Engaged

Digital Power User — LOW_RISK + Engaged

Stable Non-Digital — LOW_RISK + Not Engaged

Unclassified — Default fallback

STRATEGI — Strategic Growth Intelligence & Wave Planner

Version 9.0 · Data-First Campaign Orchestration with Competitive Intelligence (Pipeline v10.0)

"I turn insights into action — multi-wave campaign plans with realistic scenarios and rigorous A/B/C testing."

What It Does

Generates multi-wave campaign plans from root causes
Scores strategies on confidence × impact × feasibility
Creates 3-dimension A/B/C variants (channel + offer + tone)
Calibrates conversion rates against industry benchmarks
Models wave carryover effects (fatigue + conversion removal)
Plans 4 competitive scenarios (Conservative → Best Case)
Aligns campaign timing to billing/recharge cycles
Optimizes budget allocation via linear programming

Libraries & Tools

scipy.optimize linprog for budget optimization
LLM Client Wave plan enhancement
collections Counter for keyword extraction
math Scoring & calibration functions
json + pathlib Persistence
Knowledge Layer Industry learnings & CIG
re Pattern parsing

Methods & Techniques

Method	Purpose	Key Parameters
Composite Strategy Scoring	Rank strategies by data-backed quality	30% confidence + 45% impact + 25% feasibility
Industry Benchmark Calibration	Normalize propensity rates to realistic campaign rates	Telecom retention: 15–25%, reactivation: 3–8%
3-Dimension A/B/C Variants	Test channel, offer size, and message tone independently	34/33/33 split ratio
Wave Carryover Modeling	Adjust later waves for fatigue and prior conversions	10% fatigue/wave, 30% max cap
Competitive Scenario Projections	Range of outcomes for executive planning	Floor: −40%, Competitive: −25%, Best: +20%
Linear Programming (scipy)	Mathematically optimal budget allocation	5% min, 60% max per wave
Billing Cycle Alignment	Time campaigns to user recharge/renewal cycles	Retention: days −5 to −1 from cycle
Control Group Allocation	10% holdout for lift measurement	Per-strategy holdout
Strategy Grounding	Validate psychology claims against data signals	Score 0.0–1.0 = grounded/total
Segment Overlap Detection	Identify double-targeting across strategies	Warn if overlap > 30%

Methodology Flow

Extract Inputs — Root causes from DAKA, segments from SEKI, personas from LOKI, CIG patterns, learnings. Parse goal, budget, timeline
Data-Derived Configuration — Build grounding keywords from actual metrics (no hardcoding). Derive channels from persona consensus. Derive rates from SEKI propensity
Generate Wave Plan (Data-First) — For each root cause × segment: create strategy with persona-matched channels. Assign to waves by ROI ranking. Dynamic wave names from content
LLM Enhancement (Optional) — Improve descriptions, messaging, rationale. Retry up to 2× with backoff. Falls back to data-driven plan if fails
Score Strategies — Confidence (data signals, propensity, grounding) × Impact (segment size, alignment, conversion) × Feasibility (channels, wave position). Rating: HIGH / MEDIUM / LOW
Generate A/B/C Variants — For HIGH-rated strategies: test 3 dimensions (channel ↔ alternative, standard ↔ aggressive offer, urgency ↔ benefit ↔ social proof tone). 34/33/33 split
Enrich & Calibrate — User counts from SEKI. Delivery rates per channel. Industry benchmark calibration (raw propensity → realistic rate). Control group holdout (10%)
Billing Cycle + Carryover — Align timing to recharge cycles. Remove converted users from later waves. Apply fatigue (10%/wave, max 30%). Prevents 15–30% overstatement
Budget & Scenarios — scipy linprog optimization (or proportional fallback). 4 competitive scenarios: Conservative Floor (−40%) → Best Case (+20%). Segment overlap detection
Validate & Ground — ≥1 wave, 1–5 strategies/wave, ≥60% root cause coverage. Ground claims against data signals. Final grounding score + ungrounded claim list

v10.0 Pipeline Enhancements

Lifecycle-Bespoke Strategy Descriptions
Strategy descriptions are regenerated AFTER lifecycle phase assignment, ensuring messaging is optimized for customer journey stage (New, Mature, Veteran).

Grounding Audit at 85.7%
Merged psychology and data-derived keyword taxonomy achieves 85.7% strategy grounding rate, validating strategy claims against actual data signals.

Wave Number Metadata Propagation
Wave assignment metadata is propagated through all downstream strategy specifications, ensuring consistency in campaign sequencing and timing.

Lifecycle-Specific Offers
New → welcome bonus | Mature → VIP loyalty rewards | Veteran → exclusive retention programs. Tailored incentive structure per customer lifecycle stage.

Industry Benchmark Rates (Telecom)

Reactivation

3% – 8%

Retention

15% – 25%

Upsell

10% – 15%

Education

8% – 12%

Quick Win

12% – 20%

Fatigue Cap

30% max

Control Group

10% holdout

Budget Bounds

5%–60% per wave

Data Flow Between Agents

From	To	Key Data Passed
DAKA	SEKI	Unified CSV (200K rows × 83 cols), domain config (KPIs, semantics, goal requirements), root causes, baseline metrics
DAKA	LOKI	Domain config (industry, KPI candidates, semantic groupings), root causes for persona reasoning
DAKA	STRATEGI	Root causes (strategy basis), baseline metrics (conversion rates), billing cycle data
SEKI	LOKI	Users with segment IDs, segment profiles (propensity, goal rank, key features), model artifact (centroids)
SEKI	STRATEGI	Segment profiles (sizes, propensity, alignment, campaign labels), migration paths
LOKI	STRATEGI	Persona summaries (psychology, preferred channels, messaging tone), evolution windows

Core Design Principles

NO HARDCODING — Every metric, threshold, channel, and conversion rate is derived from actual data. No hardcoded segment names, no template matching, no assumed constants.

DATA-FIRST, LLM-SECOND — Complete data-driven plans are always built first. LLM enhances descriptions and messaging but never gates execution. Graceful degradation guaranteed.

GOAL ALIGNMENT — Every agent is driven by the business goal. Segments are goal-aligned, personas reflect goal psychology, strategies target goal-specific root causes.

STATISTICAL RIGOR — Multiple testing corrections (Bonferroni + FDR), stability validation (20 runs), bootstrap CIs, z-score grounding, industry benchmark calibration.

GRACEFUL DEGRADATION — Every agent has fallback paths. LLM fails → rule-based. Missing data → proxy detection. No model → minimal output. Pipeline never crashes.

CAMPAIGN READINESS — Output is designed for direct CVM execution — campaign labels, A/B/C test specs, billing cycle timing, budget allocation, control groups.

Pipeline Methodology & Agent Documentation

Pipeline Overview

DAKA

SEKI

LOKI

STRATEGI

DAKA — Data Analysis & Knowledge Acquisition

What It Does

Libraries & Tools

Statistical Methods

Methodology Flow

Key Thresholds

SEKI — Segmentation Expert & Knowledge Intelligence

What It Does

Libraries & Tools

Statistical & ML Methods

Methodology Flow

v10.0 Pipeline Enhancements

Key Thresholds

LOKI — Persona Discovery & Behavioral Psychology Engine

What It Does

Libraries & Tools

Methods & Techniques

v10.0 Pipeline Enhancements

Methodology Flow

8 Fallback Archetypes

STRATEGI — Strategic Growth Intelligence & Wave Planner

What It Does

Libraries & Tools

Methods & Techniques

Methodology Flow

v10.0 Pipeline Enhancements

Industry Benchmark Rates (Telecom)

Data Flow Between Agents

Core Design Principles