User testing (sample testing)

Send a physical sample to a real person, have them agree to your requirements and answer a questionnaire on a hosted form (no login), then optionally sell them the sample with a Shopify draft order.

Where: setlist → User testing Related setting: Settings → Questionnaires

⚠️ "User testing" vs "Tests" — two different features. User testing (this page) = samples sent to real people for feedback. Tests = internal QA you run yourself. Similar names, different jobs.

What it is

User testing runs a small sample program. For each test you:

1. Pick a tester (a contact) and the sample under test (a product + a specific variant).
2. Optionally attach a questionnaire.
3. Generate a hosted form link and share it with the tester.
4. The tester opens the link (no login), agrees to your requirements, and fills out the questionnaire + optional notes/photos.
5. Once they submit, you can sell them the sample via a Shopify draft order.

The lifecycle

A test moves through five statuses:

Invited → Acknowledged → Submitted → Converted (plus Cancelled, off to the side)

Set it up: step by step

1. (Optional) Build a questionnaire

In Settings → Questionnaires, create a reusable question set (up to 10 questions). You can also run "acknowledgment only" with no questionnaire.

2. Create the test

User testing → New test. Fill four sections:

• Tester — choose an existing contact or add a new contact (Name + Email required; Phone/Notes optional). A new contact is also synced to a Shopify customer (best-effort, once commerce access is granted) so a draft order later ties into your store CRM.
• Sample under test — choose the Product (required) and the Variant (the specific sample).
• Questionnaire — pick a saved one, or "No questions (acknowledgment only)". New questionnaire opens a builder right here (name + up to 10 questions, each with a type, Required/Optional, and dropdown options where the type needs them); Create & select saves it as a real questionnaire, attaches it, and it stays editable later in Settings → Questionnaires. The questionnaire is snapshotted onto this test at creation, so editing it later never changes this tester's questions or answers.
• Requirements & timing — edit the requirements the tester must acknowledge (prefilled with sensible default copy), and set a Due date (required, no past dates).

Click Create test & generate link → you land on the test's detail page with the link ready to copy.

Adding a new contact writes to Shopify. On that path the button opens the standard Shopify-write confirmation — "Create this test — and a Shopify customer?" — spelling out that it adds the contact as a Shopify customer only if no customer with that email exists yet. Picking an existing contact writes nothing to Shopify and submits straight away.

3. Share the link

On the detail page, copy the Tester form link (.../t/<token>) and send it to your tester. Anyone with the link can open the form — there's no tester login.

4. Watch results come in

As the tester acts, the status advances and their responses, notes, and photos appear on the detail page.

5. Sell the sample (optional)

Once the test is Submitted, the Sell the sample section offers Create draft order (unit price prefilled from the variant — edit it to comp or discount; set quantity). The button opens a Shopify-write confirmation listing exactly what will be created, and reminding you that a draft order is not a sale — nobody is charged and reports are untouched until the invoice is sent and paid in Shopify. On success the test flips to Converted with Open draft order in Shopify and Invoice link.

Selling the sample needs commerce access. The first time you create a draft order (or sync a tester to a Shopify customer), setlist asks Shopify for commerce access (read_customers, write_customers, write_draft_orders) via a "Commerce access needed" banner with a Request commerce access button. Grant it once and Create draft order becomes clickable. Everything else — invites, acknowledgment, questionnaires — works without it. See Shopify access & permissions.

The pages

List view

Four metrics — All tests, In progress (invited/acknowledged), Submitted (ready to sell), Converted — and a table: Tester · Product · Sample · Status · Completion (answered/total) · Due.

Detail view

Sections:

• Status — status badge, due date, tester info (with a note when they aren't linked to a Shopify customer yet), the sample, and the Tester form link with Copy link. The link is hidden once a test is Cancelled.
• Acknowledgment — the requirements text and who signed/when (or "Not acknowledged yet").
• Responses — per-question answers once submitted (photo questions show a thumbnail gallery).
• Tester notes & photos — general feedback/photos the tester added. Only shown once the test is Submitted or Converted and the tester actually left something.
• Sell the sample — the draft-order step (see above). Before submission it just reads "Available once the tester has submitted the questionnaire."
• Danger zone — Cancel this test (disables the link). Hidden once a test is Converted or already Cancelled.

What the tester sees (the public form)

The link opens a clean, branded setlist page (not the Shopify admin, no login). Every actionable screen carries an at-a-glance panel — Product, Sample (when it isn't the default), From (your store name), and Feedback due. It walks the tester through:

1. Step 1 of 2 — Acknowledge: read the requirements, check the box, and type their full name to sign.
2. Step 2 of 2 — Questionnaire (or a short Confirm & finish if acknowledgment-only): answer the questions. Question types include short text, long text, single-select, multi-select, and photo upload.
3. Notes & photos: an always-available spot for extra feedback and up to 6 photos (e.g. "it tarnished" / "arrived damaged").

When they submit, the test becomes Submitted, everything flows to your detail page, and the tester sees a thank-you screen. A cancelled or unknown token gets its own plain message rather than a form. Photo limits: images only, under 10 MB each, up to 6 per field and 12 per submission.

Tips & gotchas

• Snapshotting protects in-flight tests. Editing a questionnaire never changes a test that's already created — each test keeps its own copy of the questions.
• The link is the key. There's no tester account; anyone with the link can open the form. Cancel a test to disable its link.
• Submitted ≠ sold. "Sell the sample" is an explicit step that creates the draft order and moves the test to Converted.
• Unsynced variant? If the variant isn't in Shopify yet, the draft order uses a custom line item (title + price) instead of the real variant.

See also

• Settings → Questionnaires
• Tests — the other testing feature (internal QA)
• Products