Setting scheduled or recurring website audits

Sitebulb Pro users are able to set up Projects to run scheduled or recurring audits, which can be done at the initial audit setup or retrospectively added at a later date.

This article will walk you through how scheduling works and how to do it. Don't miss the last section, which describes how you can pair this with automatic exporting to set up a fully automated workflow.

You can jump to a specific area of the guide using the jumplinks below:

Video Demonstration

For those that prefer video guides, we have a complete demonstration of the scheduling feature on the video below:

Scheduled vs Recurring

Before we start, a note on the lexicon. A scheduled audit simply relates to a single audit that is scheduled to run at a particular date, whereas a recurring audit is one that automatically runs at the same time each month/week/day.

Scheduled audits are particularly useful if you need to audit websites during specific time periods. For example, consider a busy ecommerce store, the development team may insist that the site is only ever crawled in the middle of the night when traffic is at its lowest. Scheduling allows you to avoid that 2am alarm clock!

Recurring audits are particularly useful if you need to run regular audits on the same websites. For example, if part of your client contract is to do a technical health check each month on the website. Setting a recurring monthly audit ensures that you have all the data you need, when you need it.

How to set up a scheduled website audit

When setting up a new audit, once you have finished customising the other elements, you should notice an option at the bottom entitled Scheduling.

Add Scheduling

If you wish to add a schedule, enable the Schedule Audit box;

Enable scheduling

This will open up the Scheduling panel and options. By default, the 'Once' radio button is selected, which essentially means you are setting up a scheduled audit to run only once, but not regularly recur.

All that is left to do is select a date and time, which can be specified to a 5 minute interval:

Set Scheduled Date and Time

You may notice that the green button at the bottom right, which normally says Start Now, has switched out to Save Audit, since the audit is scheduled for the future and will not actually run immediately.

How to set up a recurring website audit

A recurring audit is effectively a special case of a scheduled audit, so the initial steps as above are all the same.

To set up a recurring audit, this time you need to select the Recurring radio button, which will also open up a set of new options underneath.

Select Recurring Audit

Select how often you would like this audit to run

The next set of options are self-explanatory, but I will explain them nonetheless:

  • Monthly - the audit will run every month
  • Weekly - the audit will run every week
  • Daily - the audit will run every day

Select when to start the audit

This section is a little less intuitive. The default option is that the audit will start now and will run in its entirely, and then Sitebulb will schedule the audit to recur every month/week/day (based on the option you selected above).

Using the time selector, you can pick a time in the day for the recurring audits to run:

Set recurring audit time

In the example above, we've set up an audit to run now (10:42) and then recurring every month at 23:25. So the first audit will run right now and the second audit will run in one month, at 11:25pm. Then all future audits will run every month at 11:25pm.

Since the selected option is to start the audit now, the green button bottom right stays as Start Now:

Start Scheduled Audit now

The alternative option here is to select a specific time to start the first audit, and at the same time set up a recurring schedule. This is effectively a combination of the two concepts - we set up a single scheduled audit (without running one right away) and then set this to be recurring with a specific periodicity.

Here is an example setup:

Set recurring audit for different time

So, in this example we switched to a weekly recurring audit, then set the first audit to run on May 5th at 14:30.

Now, if you check the screenshot below, you can see that the green button has changed to Save Audit. For clarity, this means that the first audit will not run now, but will automatically run on May 5th at 2:30pm, and then it will run every Tuesday thereafter, also at 2:30pm.

Save Audit

How to view scheduled audits

Once you have some scheduled audits set up, you can quickly and easily view them by navigating to the Scheduled tab from the main Projects screen. You will see your scheduled audits listed below, along with the 'Next Audit' run date, and the recurring interval, if the audit is set to recur.

Scheduled audits dashboard

Then, to navigate to a specific project, click the View Project button for the corresponding audit:

Navigate to specific project

As an aside, you can also differentiate any Projects that have scheduling set up from the main Projects list, delineated by a little calendar symbol:

Calendar Symbol

Editing Project settings

Let's say you have a recurring audit set up to run every week, and your client calls up to let you know that they have uncerimoniously implemented AMP (without telling you first!!!). You may now wish to include AMP validation as part of your weekly audit in Sitebulb. By editing the project settings, you can go in and add AMP to the crawl settings, so that Sitebulb will also check this every week.

In order to do this, navigate to the project in question (see instructions above), and hit the blue button Edit Settings and Schedule:

Edit settings

This will open up the familiar 'Audit setup' page, which you can then adjust as required. In this case, all we wanted to do was add Structured Data validation, so it is just a case of checking the 'Structured Data' tickbox and then hitting the Save Changes button. However, you can go in and change any element of the audit setup, including Advanced Settings on the left. Just make sure to hit save!

Structured Data

Editing Project schedule

An extension of the above, you can choose to edit the scheduling of an audit as well. Simply navigate to the project, and open up the audit setup page using the blue button Edit Settings and Schedule.

Then, scroll down and press the Schedule Audit thumbnail button, to open up the scheduling panel. You can change any element of the schedule, including the recurrence interval and when the next audit should run. Make sure to hit the Save Changes button!

Change Schedule

Starting an ad hoc audit

Let's say you have a recurring audit set up to run every month, but your client calls and tells you they have just made multiple changes to the robots.txt file on a whim ([email protected]$ards!), so you figure 'I'd better audit the site now!'.

Well, you can do this too, and setup Sitebulb to run an ad hoc audit, without interfering with your pre-scheduled recurring monthly audit.

As before, navigate to the project page, and hit the green button Start and Configure a New Audit, which will allow you to start a new audit right away, and will not change the recurring or scheduled date you have already put in place.

Ad hoc audit

How to cancel scheduled audits

If you realise that you no longer want a specific scheduled or recurring audit to run, you can simply cancel the schedule off the project. All this will do is remove the scheduled element - all scheduled or recurring audits will be removed, and it will revert back to a normal project. Note that this will not delete the project, just the schedule.

In order to do this, navigate to the project in question and hit the Cancel Schedule text link.

Cancel schedule

Accessing and adjusting scheduling settings

The default behaviour of scheduled audits works on the basis of two core assumptions:

  1. That your computer is actually on, when scheduled audits need to run
  2. That Sitebulb is actually running on your computer, when scheduled audits need to run

If both of these assumptions are met, Sitebulb will handle scheduled audits just fine. If either of them is not, then it is not possible for Sitebulb to carry out the scheduled audit.

However, Sitebulb has a number of in-built settings that help mitigate these issues, which are accessible via this link on the Scheduled Audits page:

Find scheduling settings

This will bring you to a settings page with specific scheduling settings:

Scheduling settings

These tickboxes offer the following options:

  • Disable scheduling - stop Sitebulb running any scheduled audits at any time, while this box is ticked.
  • Load on Startup - enable Sitebulb to load up when you start up your machine, and run silently in the background.
  • Run missed audits - if Sitebulb has not been running for a period of time, and has missed some scheduled audits, force Sitebulb to queue up and run these audits the next time it is running.

These options come with a degree of nuance, so you may wish to read below for a more complete explanation of why they exist and how you might wish to use them.

Disable scheduling

If you tick this option, it will over-ride all scheduled Audits so that none of them run until this box is un-ticked. Let's say you are going on holiday for 2 weeks and know you won't be around to look at any scheduled audits that come in during the time you are away, it might be suitable to stop scheduled audits from running until you are back.

Once you un-tick this box, any recurring Audits will be rescheduled for the next available date.

Load on Startup

If you tick this option, Sitebulb will load up whenever you switch on your machine, and run silently in the background, without a visible user interface. It will sit in your system tray, waiting for when it is needed.

Sitebulb running in background

Then, when scheduled Audits are due to start, Sitebulb will run these in the background. You'll get a notification message when it does:

Sitebulb has started notification

Note: On Mac, Sitebulb will need to be given permission to do this, for which you will need to grant by clicking OK;

Mac Permission

But once you do, you'll also see the notification in the top right of your screen, whenever Sitebulb starts running a scheduled audit in the background:

Audit started on Mac message

Run missed audits

If you tick this option, Sitebulb will queue and run any scheduled Audits that have been missed because Sitebulb was not running.

By default, if you missed an Audit for today’s date, Sitebulb will queue and run the Audit the next time you open the software. However, if you miss an Audit for any other day, Sitebulb will not run them by default. If you tick the box, it will run them regardless of the date.

As an example, let's say you had some audits set up to run like this:

  • Daily audits for client A
  • Weekly audits for client B
  • Monthly audits for client C

If you went on holiday for a month and your machine was off the entire time, when you came back into the office and opened up Sitebulb it would do the following:

  • Schedule and run 1 audit for client A, then set the schedule back to tomorrow
  • Schedule and run 1 audit for client B, then set the schedule back to a week from today
  • Schedule and run 1 audit for client C, then set the schedule back to a month from today

Automatic notification when audit is complete

Since scheduled audits are often run in the background or on a remote machine, it is easy to forget that you sent Sitebulb off to do this task, and so adding an email alert when the crawl is finished can be a helpful reminder.

You can set this up in Sitebulb via the 'Crawler Settings' options on the left-hand menu in the audit setup:

Send email when finished

To set this up, toggle the checkbox on, enter your email address in this box (or multiple addresses, comma separated) and Sitebulb will send you a friendly email whenever your audit is finished:

Send Email When Finished

Automating exports and saving to a custom folder

This is the cherry on the cake of the scheduling feature. It allows you to build a fully automated workflow where Sitebulb can run audits in the background and automatically save the export files in a location of your choosing - either on your local machine or in a shared cloud drive that your whole team can access.

You simply pick a location for the exports and/or the audit share file, and then at the end of the audit Sitebulb will save the data there for you.

The most important thing to note is that this is a project setting, so you set the folder location for a specific project - it will NOT automatically set this for every project you create.

You can set the custom save folder either when setting up a new audit, or when you edit the project settings. 

Access the Data Exports section towards the bottom of the left-hand menu when setting up your project.

Save Data to Custom Folder

This page offers two save locations, by clicking Select you can choose a local save folder OR a shared save folder, such as a cloud storage service like Dropbox - which allows other team members to pick up the files once they've been generated.

CSV/Excel export files

These are all the export files that Sitebulb produces at the end of an audit, including all the Hint exports and exports for all the URL data, such as 'Internal' and 'Indexability'. Mostly these are .csv files, but there are also some formatted .xlsx files that span multiple worksheets.

Saving CSV and Excel Files

So the data gets saved in the folder like this:

Data saved in file

Audit project file

This is specifically the audit project share file (.sbp), which you would normally get by going into the Audit Overview and hitting the Share Audit button. This file can be imported by any other Sitebulb user, and they would then have access to the entire audit as if they carried it out on their own machine. Check out this guide to learn more about importing audit files.

Save Audit Project Files

Fully automated audits

While the custom save location works for any audit, it is most powerful when you use it alongside recurring audits.

The features combine to allow you to set up fully automated workflows, such as:

  1. Set up a project to run weekly recurring audits on a website
  2. Set up export files to save to a shared folder on Dropbox
  3. Give other team members access to the shared folder
  4. Receive automatic email notifications when audits are completed

Every week, new export files will appear in the shared folder, allowing team mates to grab the new files and build their reports with all the fresh data.