I've been a Qualtrics customer/user for a little over a year. How does it measure up and has my opinion changed over time?
Background
In a past blog post I discussed a little more about experiences of building operational process (you can find that blog here). For this particular post I want to talk about how Qualtrics has fitted into that process and my evaluation as a Qualtrics user a year or so down the line. The key drivers for us assessing a new data collection platform were:
- We were moving away from an existing platform with its own underlying data collection system.
- We wanted a survey design process that was not dependent on specialist setup knowledge. Specifically we wanted project managers to be able to set up and mange their jobs in their entirety which included creating the online survey and working with the data from an online survey.
- We needed a platform with strong API credentials in order to be able to create seamless integration between applications and our own internal processes and portal developments.
After testing a number of potential candidates we eventually settled on Qualtrics. That's going to rightly suggest to you that this review of Qualtrics will, for the most part, be positive. What I additionally hope to provide though is my opinion on the platform over a year on from first implementation. How was the on-boarding and implementation? Has my opinion changed over time?
This is not a comprehensive review of Qualtrics - I am only focusing on the elements that I use within the business. Namely the Research Suite and Target Audience. I'll only very lightly touch on data analysis and reporting.
Initial Contact
I wanted to be certain we would be able to do everything that we required within Qualtrics and right from the start they were engaged in understanding these requirements and giving the support and, crucially, the time needed to properly evaluate. During this time sales support and technical support was exceptional. I had direct support contacts and their ability to comprehend and answer my many questions was admirable. Being a new user to Qualtrics these answers were often supplemented with precise step by step instruction and even video solutions created specifically to respond to my queries. It was a promising start but I did have concerns whether that support effort might drop off once we'd committed to that platform. I'll discuss further experiences with sales and support later on.
The Survey Design interface
This was what I was primarily interested in. We had a specific requirement that our Project Managers would be able to set up their own online surveys. Of course we would expect to provide a level of training and that there would be a period of learning before fully up to speed but we needed them to get there quickly and with no prior experience of online survey setup.
The other personal interest I had in this is that I've had close ties to survey scripting throughout my career. I started in Market Research doing DP but very quickly moved to survey setup. Initially scripting in Quancept and then moving over to Dimensions IOM (and TOM for analysis). I've scripted every type of electronic survey of every level of complexity. I knew the technology could be capable to provide a simple GUI with advanced functionality but I was sceptical whether there were products that were yet there. I'm happy to report that any scepticism was mis-guided.
The Qualtrics interface is online. It is modern, clean, and straightforward. All expected question types are available with further advanced options either available through sub menus or available as properties on the right hand side of the design screen.
Image 1: Qualtrics Survey Design Interface
The survey design process is split into two main elements. The design view which is where you create the survey text/questions/responses and the "Survey Flow" which acts as the advanced survey routing. Here you can specify branching/skip logics, add survey variables (known in Qualtrics as "Embedded data"), add conjoint sections, web services etc. Overall it adds up to a very capable system presented in a sensible and engaging layout. Whilst it would not be the ideal platform for some of the more complex survey designs I've been involved in, I was very pleasantly surprised to discover that it would cover the vast majority - and with ease. It may not be the most powerful/flexible data collection platform I've worked with but it's certainly been the most pleasing and accessible.
Image 2: Qualtrics Survey Flow Interface
Options for survey look and feel are in another menu with a selection of provided "themes" and additional menu options for quota setups, survey triggers, translations, are in another. Survey Options cover things such as the presence of a back button, whether surveys are open access, by invitation only, or password protected. "Ballot Box Stuffing" prevents surveys from being taken more than once. Survey Expiration allows the input of a specified date range for when the survey is open. There are settings to determine what happens to any partial data records - e.g. delete them after a certain time or else move them into the main dataset.
All surveys can be shared with the rest of the team for collaboration.
The Survey Experience
So the survey design experience is good. How about the actual survey experience itself? Well let's first look at the testing/preview phase. As every good scripter knows you should test, test, and test some more. One of the great features of Qualtrics preview is that you are immediately provided with both a desktop and a mobile view. It might seem a small touch but it's incredibly useful to make sure that your design translates for all users and it points to the seriousness that Qualtrics takes this. From a personal perspective it has been a real frustration that in 2018 I continue to see online surveys that do not adapt well to different devices. Even more annoying is survey design choices that simply do not suit mobile devices. Qualtrics is truly device agnostic. And even more than that will adapt survey design to better fit the device it is being used on (you can override this behaviour). An example of that is a grid question. Qualtrics will explode the grid question out to a much more mobile friendly layout where required.
Additionally to this, whilst there is limited control over the page, it is possible to add headers and footers which can include JavaScript code and to include JavaScript code on given questions/pages. We take advantage if this flexibility to add an instant chat option to our surveys so our respondents have a simple mechanism to get in touch with a project manager should they have any queries, or experience any difficulties with our surveys (a similar example is in action on this page).
Image 3: Qualtrics Survey Experience
Panel Management and Distribution
When we bought into the Qualtrics Research Suite we additionally bought into their panel management solution, Target Audience. Our panel management needs were fairly modest. We needed a database repository with basic control, ability to pull relevant samples and send mailings, and provide us with key performance metrics. We chose Target Audience not because it was best in class but because it was good enough for our need and was immediately integrated with the research suite. We could have opted for a more fully featured database/panel management system and easily connected this to Qualtrics but Target Audience was good enough. And it remains good enough. Certainly nothing spectatcular and considering how good the rest of the platform is I would have to say it's a weaker component of the Qualtrics suite. It is the one component that I would consider swapping out for an alternative but it does do the job we ask of it for now.
Image 4: Target Audience Interface
From a distribution perspective the tool works fine. We have available templates and translated templates, We select a sample and survey and the system sends out the emails. We receive back stats on email delivery, emails opened, surveys stats, incidence rates. All as expected, all relatively straightforward. No complaints.
Image 4: Qualtrics Distribution through Target Audience
Data Reporting
Given the nature of our business our data reporting requirements are typically fairly limited. Whilst we do run full quantitative surveys, we're mostly recruiting respondents for further in depth interviews so we're assessing data on an individual level rather than aggregated results. For those specific needs the default data view works perfectly for us. We can see respondent data line by line and tailor which fields are important to us.
Image 5: Qualtrics Data View
Additionally to this there is an option for online reports. It's more structured towards aggregated data reporting but it is a viable route to provide a live view of the data. The provided link is either public or password protected and does not require a Qualtrics login. It's a viable client update mechanism, if not the ideal solution.
There are cross tabulation options and text analytics but I won't go into detail in this review simply because they're not commonly in use in or business. Data can be weighted as required. Again something we don't often have cause to use but reassuring it's there all the same!
Data export options are as expected. CSV, XML, and SPSS formats are all catered for. Through the API you can easily pull data in a JSON format. More on the API below.
API integrations
Qualtrics has a REST API. It is a paid feature but one that we are reliant upon. The API is reasonably extensive but we primarily use it to pull data down from Qualtrics and to push data back into Qualtrics data records (thus giving us a consolidated data source despite a reliance on a few different applications outside of Qualtrics). The API is central our internal processes, project management reporting tools, client reporting tools and portals.
Onboarding
I don't have much to say on the onboarding process other than it was smooth and had us in a position to be running live projects almost with immediate effect. A Qualtrics internal team helped us to create our survey look and feel.
Implementation
The Qualtrics documentation includes a number of tutorials to bring you up to speed quickly. Additionally to this I created documentation and training instruction specifically to our internal process. Credit to Qualtrics and credit to our PMs - they were engaged and setting up dummy projects in no time. Within a week one of our PMs had scripted and launched her own study with minimal hand holding - and she loved it compared to the legacy process of having to book the work in with a scripter, test it, wait for small changes and amendments. I could not have dreamed of a smoother and faster implementation.
Sales & Support
Sales were, unsurprisingly, enthusiastic. To be fair to them they were engaged throughout the initial sign up period, interested to understand our business needs and work with us to tailor the proposal and cost to our business need, size and budget.
Support is the stand out area. They have been first class in all regards. There are multiple routes to access - either phone, email, or online chat. When it comes to anything related to the Research Suite the response is quick but more importantly the product knowledge and understanding of the query is exceptionally good. The answers provided are accurate. I can honestly say that I've never had better support experience than I have through Qualtrics for their Research Suite. Support for Target Audience is a slightly different story. It's seemingly not considered a core product so usually requires specialist support. The support has still been very good but usually requires that I'm passed through to a specialist so typically requires a phone call.
Image 5: Qualtrics Support Options. Two Thumbs Up!
Given the model of Project Managers running studies it's important that they are well supported. We provide internal support but queries that are outside of our ability or during times when internal support is not available it's a comfort to know that the support is high quality and available.
Documentation and help videos/tutorials is comprehensive, clear, and up to date. Previous platforms where this has not been the case has highlighted just how easily this can taken for granted.
Summary and opinion over time
So far so good. Over a year on and have my opinions changed? Not hugely. I'm still immensely positive about Qualtrics and how it was a good fit for our business. If anything my opinions have gone up as we've experienced situations outside of the norm that Qualtrics has handled with relative ease e.g. diary type studies, complex segmentation calculations (which we ran through a web service, returning the results back into Qualtrics - all in real time and invisible to the respondent of course) etc.
Quality of support hasn't tailed off as our enquiries become more informed and therefore more complex.
Is Qualtrics a good fit for your business? I guess that depends on your expected model and whether the pricing model is not inhibitive. It's not a cheap product. Our own license covers the required number of users and some additional functionalities but also, importantly, a response limit. The response limit covers our anticipated yearly usage and because of the type of business and the small universes we work with is fine. That may be less the case if requiring very high numbers of responses.
Overall I'm happy. I'm happy with the decision we made and I'm happy for the industry that the technology has now caught up to democratize the survey setup capability and change the dynamic. If you're interested then I really recommend giving some of these products a try (Qualtrics or others) - they have come on leaps and bounds in the past few years and I suspect, as I was, you'd be pleasantly surprised at the capability.
