Creating your first automated test flow with Uptime

Frequently asked questions

Here are some of the most frequently asked test flow questions

• What tests should I record?

  For most merchants, the following tests provide solid testing coverage:

  • Home > Collection page
  • Collection page > Quick add to cart
  • Product page > Add to cart > Checkout
  • Adding various upsells/recommendations across product pages and cart

  For more test flow ideas check out this blog post.

• How often do these test flows run?

  We want to give you peace of mind knowing whenever a change is made to your online store that it’s thoroughly tested. That’s why your test flows will run whenever you make an update to your online store theme. This includes edits made via the Shopify admin editor (eg. editing headlines, imagery, or moving sections around), a developer partner pushing code, or if you install a third-party app. All of your tests will also run by default every 24hrs.

• What should I do if a test fails?

  If one of your tests flows fail you will receive an email alert (and Slack if you have it connected) with useful information to help start debugging the issue.

  Here are some helpful tips to assess a test flow failure:

  • Check the test flow screenshot: when a test fails we will take a screenshot at the point of failure. This can be viewed by clicking the ‘View screenshot’ button on the test flow that failed. This image will often make it obvious to you in terms of what’s going wrong.
  • Check the advanced debugging information: above the screenshot image in the app, you will see advanced debugging information, which for the technically inclined will provide some insights into what event in the test failed.
  • Test the flow yourself live: you should double-check the test by clicking through the steps live on your online store. This will help you confirm if it’s just a passing issue, or if something is currently broken.
  • Re-record if necessary: if an adjustment on your store caused a test to fail, but the actual functionality is still working, it may be necessary to simply re-record the test flow. This can be done by following the simple steps above
• Does this impact my store metrics?

  Nope! While your automated test flows are running we block events to Google Analytics, Shopify Analytics, and Amplitude. So there won’t be any impact on your store's analytics.

• Does this impact the performance of my store?

  Nope! Uptime does not make any changes or performance impact to your online store – it runs completely separately on our own servers. That means it's safe to install anytime.

• What happens if a product sells out?

  If for example, you have a test that adds a product to the cart and that product sells out, that test flow will fail. This is because the test will be unable to add that product to the cart.

  This is why we recommend recording test flows on products that generally remain in stock.

• Can I test end-to-end checkout?

  Due to privacy reasons (collecting payment details) and since Shopify doesn’t provide the ability to create test checkouts, we do not currently support end-to-end checkout test flows.

  You can record tests as far as the ‘Payment details’ checkout step (keep in mind this will create an abandoned checkout in Shopify).

  Checkout is handled by Shopify and is generally covered by Uptime’s Shopify status page monitoring, so going beyond the first step of checkout is not necessary for most merchants.

• Do you support headless stores?

  We do! For headless stores, since the theme itself is hosted outside of Shopify, you can leverage our ‘Flow run webhook’ to trigger your test flows to run whenever you deploy code to your online store.

  You can find your ‘Flow run webhook’ on the Uptime settings page. Headless support is currently only available on Enterprise plans. Email us for more details at [email protected]

• What location do these test flows run from (geographically)?

  Our tests run from a server based in the United States (Iowa to be specific) – meaning each test flow will simulate a customer residing in the US.

  Therefore it’s important when you’re recording tests to simulate the US customer experience, whether via a VPN or setting store location/currency preferences before recording.

• What if my store is password protected?

  Not a problem! Make sure to record your tests in incognito so you are forced to enter the password before accessing the store.

• Can I record tests that interact with an <iframe>?

  Unfortunately at the moment, we are unable to run tests that interact with an element nested within an <iframe>. We hope to support <iframe>s in a future update!