The SIFT Likelihood

What this notebook is for

This notebook builds the likelihood for the SIFT instrument and checks it. The code lives in R/sift_model.R; the checks are in R/test_sift_model.R. Everything here runs at deliberately small sample sizes — the point is to establish that the machinery is correct, not that it is fast.

It draws on three sources:

• @Yavorsky_2021 for the utility decomposition, the closed-form reservation utility of @Kim_2010, the kernel-smoothed logit accept–reject simulator, and the argument that an exogenous, product-specific search-cost shifter lets us estimate the match-value standard deviation instead of fixing it to one.
• @Ursu_Seiler_Honka_2024 for notation and the smoothing convention.
• GHK, for the block of the likelihood where it legitimately applies.

Setup

In [1]:

source("R/sift_model.R")
ginv <- make_ginv()

Respondent \(i\) faces task \(t\) with \(J\) alternatives on a landing page, plus a no-purchase outside option. Writing \(\mathcal{S}\) for the ordered set of clicks:

\[ \delta_{ijt} = \mathbf{X}_{ijt}'\boldsymbol{\beta} + \eta_{ijt}, \qquad u_{ijt} = \delta_{ijt} + \varepsilon_{ijt}, \qquad u_{i0t} = a_0 . \]

\(\delta\) is what the respondent knows from the landing page. \(\varepsilon\) is what a click reveals. The search cost depends on randomized position, \(c_{ijt} = \exp\{\gamma_0 + \gamma_p\,\mathrm{pos}_{ijt}\}\), and the reservation value follows @Weitzman_1979 via @Kim_2010:

\[ z_{ijt} = \delta_{ijt} + \zeta_{ijt}\,\tilde\sigma, \qquad g(\zeta) \equiv \phi(\zeta) - \zeta\,[1-\Phi(\zeta)] = c_{ijt}/\tilde\sigma . \]

Where SIFT departs from the field settings

Four things differ from the environments @Yavorsky_2021 and @Ursu_2018 study.

1. It is a panel. Each respondent completes \(T\) tasks, so we observe many search spells per person. @Morozov_2021 needs panel structure to identify preference heterogeneity in a search model; conjoint gets it by design, and so does SIFT.
2. Position is randomized, so the search-cost shifter is exogenous by construction rather than by argument. @Yavorsky_2021 had to defend the exogeneity of driving distance at length (their §3.3.1); we do not.
3. The product page is designed. A click reveals \(\mathbf{L}_{ijt}'\boldsymbol{\kappa} + \xi\) where \(\mathbf{L}\) is drawn by us. From the respondent’s viewpoint the revealed component is a mixture of normals, not a normal. We use the variance-matched normal \(\tilde\sigma^2 = \boldsymbol{\kappa}'\mathrm{Var}(\mathbf{L})\boldsymbol{\kappa} + \sigma_\xi^2\) in the reservation value.
4. Zero clicks are possible, and non-purchase is an outcome we model rather than condition away — @Yavorsky_2021 conditions on search and purchase.

The DGP

In [2]:

d <- sim_sift(N = 60, T = 8, J = 6, seed = 3, ginv = ginv)
table(rowSums(d$cons_set != 0))     # clicks per task

At the default parameters this produces roughly 5% zero-click tasks, 74% one-click, and a thin tail beyond — a click distribution with the same shape as the field data in @Yavorsky_2021 (their Fig. 3: 91% one search), but with more mass in the tail because we can make clicking cheap by design.

Two likelihoods

(A) Kernel-smoothed logit accept–reject

This is the @Yavorsky_2021 Appendix C simulator: form the violation margins \(\nu\) for each Weitzman condition and smooth the indicator with a logit kernel,

\[ \tilde L^q = \Big(1 + \textstyle\sum_j e^{-\lambda_1\nu_{1j}} + \sum_j e^{-\lambda_2\nu_{2j}} + e^{-\lambda_3\nu_3} + e^{-\lambda_4\nu_4}\Big)^{-1}. \]

It is general and easy to extend. Its weakness is well known and stated plainly in @Yavorsky_2021 (their footnote 21): the estimator is only asymptotically unbiased, the bias falls as the scale factors \(\lambda\) grow, and large \(\lambda\) destroys the smoothness the optimiser needs.

(B) GHK on the \(\eta\) block, analytic on the \(\varepsilon\) block

The construction rests on one observation.

NoteConditioning on the observed choice turns a union into a conjunction

The stopping rule \(\max_{h\le K} u_h \ge \max_{l>K} z_l\) is a statement about a maximum, so as written it is a union of events — any one of the searched alternatives could be the one that clears the bar — and unions are not orthants. But the choice rule tells us which alternative attains the maximum. Conditional on the observed purchase, stopping becomes \(u_{j^*} \ge z_l\) for every unsearched \(l\): a conjunction of linear inequalities, which is exactly what GHK evaluates.

Two further simplifications follow:

• Continuation collapses to a single binding constraint. Continuing past step \(k\) requires \(u_h < z_k\) for all \(h<k\). Since \(z\) decreases along the search order, the binding constraint is \(z_K\) alone: continuation \(\iff \max(u_0, u_{s_1},\dots,u_{s_{K-1}}) < z_{s_K}\).
• Conditional on \(\boldsymbol{\eta}\) and on \(\varepsilon_{j^*}\), the remaining \(\varepsilon\)’s are independent one-sided constraints and integrate out in closed form as a product of normal CDFs. Only a one-dimensional integral remains.

So GHK draws \(\boldsymbol{\eta}\) from the truncated selection region — smooth, and low variance — and the \(\varepsilon\) block is quadrature rather than simulation.

In [3]:

d2   <- sim_sift(N = 25, T = 6, J = 5, seed = 7, ginv = ginv)
info <- prep_sift(d2)
dr   <- make_draws_sift(d2, nd = 200, seed = 21)
nll_sift_ks (d2$true_par, d2, info, dr, ginv)
nll_sift_ghk(d2$true_par, d2, info, dr, ginv)

The trap this construction sets

The continuation constraint applies to every \(h<K\) — including the chosen alternative when it is not the last click. If the respondent clicked three things and bought the first, then continuing past it required \(u_{j^*} < z_{s_K}\), which caps \(\varepsilon_{j^*}\) from above. Omitting that cap is an easy mistake and a silent one: the likelihood stays positive, finite, and smooth, and the optimiser converges happily on a model that is simply wrong.

It is caught immediately by the enumeration test below. Before the fix, every outcome in which the purchase was the last click had ratio 1.00 against brute force, and every outcome in which it was not was inflated by 8–33×. That signature named the bug precisely.

A second trap: the ex ante and realized views of a click

The post-search component appears twice in the model and means something different each time. Before a click the respondent does not know \(\mathbf{L}\), so from their viewpoint what a click will reveal has the ex ante spread \(\tilde\sigma\) — that is the object belonging in the reservation value. After a click, \(\mathbf{L}\) is revealed to the respondent and observed by us, because we designed it, so realized utility carries \(\mathbf{L}'\boldsymbol{\kappa}\) in the mean and only \(\xi\) stays random.

Collapsing the two — using \(\tilde\sigma\) for realized utility as well — is tempting because it shortens the likelihood, and it is wrong in a way that is invisible from the likelihood value. It leaves \(\boldsymbol{\kappa}\) entering only through \(\tilde\sigma^2 = \boldsymbol{\kappa}'\mathrm{Var}(\mathbf{L})\boldsymbol{\kappa} + \sigma_\xi^2\), where it trades off exactly against \(\sigma_\xi\) and is not identified at all.

We found this the direct way. A recovery run returned \(\hat\kappa\) sitting at exactly its starting value of zero on both coordinates, with \(\sigma_\xi\) inflated from 0.70 to 1.47 so as to hold \(\tilde\sigma\) near its true 1.29 — the signature of a parameter the likelihood cannot see. Everything else (\(\boldsymbol{\beta}\), \(a_0\), \(\gamma_0\), \(\gamma_p\)) recovered fine, which is what makes the failure easy to miss.

Folding the realized \(\mathbf{L}\) into the mean for searched alternatives is therefore not a refinement. It is what makes designing the product page pay off: \(\boldsymbol{\kappa}\) is identified by which alternative gets bought given the attributes that the click revealed.

Checks

R/test_sift_model.R runs five. The decisive one enumerates every reachable (ordered click sequence, purchase) outcome at \(J=3\) — 49 of them — and checks that the model probabilities sum to one and match a brute-force simulation of the DGP written independently of the likelihood.

check	result
DGP click distribution sensible	✓
both likelihoods finite, per-task \(\log L \le 0\)	✓
simulation variance, KS vs GHK	GHK lower; see the comparison study
truth beats perturbations	✓ both
49 outcomes sum to one	1.0009

Per-outcome agreement with brute force is within Monte Carlo error everywhere (ratios 0.97–1.02).

In [4]:

# the decisive check
source("R/test_sift_model.R")

A mini recovery run, and what it does not show

R/mini_recovery.R fits 60 respondents \(\times\) 8 tasks (\(J=5\), \(n_d=120\)) from a cold start. It takes 5.5 minutes and converges.

	true	before the \(\mathbf{L}\) fix	after
\(\beta_1\)	1.000	1.058	1.031
\(\beta_2\)	−0.800	−0.784	−0.743
\(\kappa_1\)	0.900	0.000	1.259
\(\kappa_2\)	−0.600	0.000	−1.111
\(\log\sigma_\xi\)	−0.357	0.388	0.194
\(a_0\)	0.400	0.371	0.085
\(\gamma_0\)	−1.050	−0.887	−0.207
\(\gamma_p\)	0.120	0.101	0.052
\(-\log L\)		1080.5	950.7

The point of the table is the \(\kappa\) rows and the log-likelihood, not the errors. Before the fix \(\kappa\) sat at exactly its starting value; after it, \(\kappa\) moves and the fit improves by 130 log-likelihood points on the same data. That is the identification result.

WarningThis is not a parameter-recovery claim

480 tasks is far too small, roughly three quarters of them have a single click, and \(\gamma_0\) and \(\log\sigma_\xi\) are known to lie on a ridge — @Yavorsky_2021 reports a correlation of 0.95 between the search-cost intercept and \(\log\sigma\) even in field data with 6,511 consumers. The errors on \(\gamma_0\) (+0.84) and \(a_0\) (−0.32) are consistent with that ridge and with the sample size; they are not evidence that the likelihood is right, and they are not evidence that it is wrong.

The evidence that the likelihood is correct is the enumeration test: 49 outcomes summing to 1.0024, each matching an independently written brute-force simulation of the DGP within Monte Carlo error. Parameter recovery is a question for @sec-simstudy, at commercial task counts, with a proper replication design.

Where this leaves the estimation plan

GHK’s advantage is real but it is not the dramatic constant an earlier draft of this work claimed. Measured properly — 20 independent draw sets per cell, rather than the six that produced a headline of “399×” that did not survive re-measurement — the variance ratio against the kernel-smoothed simulator is 100× at 150 tasks and 21× at 320 tasks. It moves around with the design, because the kernel-smoothed estimator’s spread is driven by a handful of awkward tasks rather than by an average.

The number that actually matters is time to a given precision, since runtime is linear in draws. On that basis GHK is 3–21× cheaper than kernel smoothing across the two designs measured, and crude frequency is not competitive at all — 175–260× more expensive, and badly biased besides, because zero cells have to be floored. Kernel smoothing also carries a visible upward bias in the negative log-likelihood (305 against GHK’s 295 at 150 tasks), exactly the asymptotic-only unbiasedness that @Yavorsky_2021 flags in his footnote 21.

Full detail, including the crude-frequency arm, is in the simulator comparison.

That matters for the 12-hour budget. It also reframes the choice recorded in sections/05_identification.qmd: the binding constraint on SIFT is estimating heterogeneity, and the cheaper route there is hierarchical Bayes, which never integrates over the heterogeneity distribution at all.

Still to build

• Random coefficients on \(\boldsymbol{\beta}\) and \(\boldsymbol{\kappa}\), and the HB sampler.
• Correlated \(\mathbf{L}\) across attributes; the variance-matched approximation already accommodates it through \(\mathrm{Var}(\mathbf{L})\), but the DGP currently draws \(\mathbf{L}\) independently.
• A proper simulation study at commercial task counts, for sections/06_simstudy.qmd.