Funnel Experiments
Part of Lead generation experiments
Deciding when a landing-page result is inconclusive
Check assignment, lead outcomes and uncertainty before declaring a landing-page winner. Learn what an inconclusive result permits you to decide.
A landing-page test is inconclusive when its evidence cannot settle the decision it was designed to make. Distinguish three cases: a broken comparison, outcomes still awaiting assessment, and a valid comparison whose estimated effect is too uncertain. A small observed difference alone does not make a test inconclusive; “no clear winner” does not prove the versions equivalent.
Check whether the comparison worked
Confirm eligible visitors were assigned as planned and saw the intended page. Compare observed group sizes with the planned allocation, taking the number assigned into account; ordinary random variation by itself does not signal a fault. Investigate an unexplained sample-ratio mismatch before interpreting outcomes.
Check for a variant that failed to load, duplicated events, different traffic reaching the versions, or requests missing from the receiving record.
Check that both groups used the same sales-acceptance rule and had comparable time for assessment. Keep pending enquiries visible; do not treat them as rejected. Note campaign, staffing or service changes that could have affected the comparison.
A tracking or delivery fault may require a new test after repair. Adding visitors to a faulty comparison does not repair the earlier data.
Ask what the estimate rules out
For a valid comparison, show eligible visitors and accepted outcomes in each group, the observed difference, and an uncertainty interval or the output of the analysis chosen before launch. Compare that range with the smallest gain or loss that would change the business decision.
If plausible effects include both a worthwhile gain and a meaningful loss, the test has not settled the choice.
A significance test that fails to reject a no-difference hypothesis does not prove no effect. The test may have too few outcomes to reveal a difference that matters. Future sample planning depends on the baseline rate, the difference of interest and the analysis suited to the actual test; there is no universal visitor count.
If the full uncertainty range lies within a pre-agreed band of differences too small to matter, the result can support that narrower practical conclusion. The band and analysis must suit the decision. Merely seeing a large p-value cannot establish equivalence.
Record the next action
- Assignment or tracking fault:repair and verify the setup, then rerun if the question still matters.
- Assessment backlog:complete comparable reviews before judging accepted-enquiry outcomes.
- Valid but imprecise result:decide whether more eligible traffic is feasible and worth waiting for; state the operating choice in the meantime.
- Sufficiently precise, immaterial difference:document the bounded conclusion and the threshold used.
Follow the stopping rule selected for the test. Repeatedly inspecting an ordinary fixed-end analysis and stopping when it first looks favourable can mislead; methods designed for ongoing monitoring have different rules.
Record the counts, the reason the result remains uncertain and the next action so a later team can revisit the decision.
