Analysis: Getting to know your data

Delib has 3 articles related to data analysis in Citizen Space

  1. Preparation and survey design
  2. Getting to know your data (this article)
  3. Producing reports

We also have a quick start guide if you need to jump into the process straight away.

In this article, we're looking at delving into your response data. The previous article on preparation and survey design was about structuring your questions and getting ready to do analysis. This is more about understanding your data when you get there.

To better look into you response data, we’ll be covering the following in this article:

Getting started and some examples

When to start?

You may want to start carrying out analysis before your activity closes, which is absolutely fine. However it's a good idea to wait until you have a good number of responses so that you can get a decent indication of what your respondents are saying.

What are you trying to find out?

You should be aware of why you are interrogating the data; what do you need to know? Understanding your response data enables you to tell the story about it in the manner that best suits the next stage of your process; be that creating a consultation report, delivering evidence for it to be actioned (and so on).  

Many Citizen Space activities have a fairly open objective — to seek people's views on a particular issue. This would normally take the form of a mixture of open and closed questions, but with a lot of room for comments to gather opinion/evidence. Understanding your data here is generally so you can deliver a narrative around the themes and topics emerging from the responses.

Alternatively, a more closed style of activity commonly asks people to choose from sets of pre-determined options. This can take the form of many more quantitative questions i.e. Which of these plans do you prefer? Please rank these options from 1 to 5; and so on. Understanding your data in this case will be more about showing how many people chose each particular option and how the responses to questions relate to one another. And this approach tends to be more common in surveys than formal consultations.

In practice, most activities we see in Citizen Space have elements of both, and by understanding your data you can create something useful that will contain information about how many people chose the closed options they were given and why, as well as telling a story about the themes and evidence submitted in comment fields. The best outcomes also say what will happen next in the process.

Related to this, let’s look at some examples of different types of consultation and engagement activity and what the aims were.

Examples

A number of years ago, the Department for Business, Energy and Industrial Strategy (BEIS) ran a consultation on the Sharing Economy. It centred around gathering people's views with the aim of producing a review of the sharing economy: a document exploring the issue which could inform future policy.

The independent review of the sharing economy overview page which included a statement from a partnered organisation's CEO and a couple of sharing economy examples (property sharing, time and skill sharing, and shared transport).

Qualitative questions included in the survey. 1. Please describe your business and how it fits within the sharing economy. 2. In what ways does your service compete with or complement traditional business models?

This is an example of an open objective — it did not have an expectation in advance of what people might say, and was simply seeking the unique experience and views of respondents. A report from the consultation would highlight some of the most prominent themes from respondents’ answers in its headlines, and then support this by breaking down some of the data behind these themes – for example: what type of respondent was more likely to make what point?

Transport for London (TfL) consulted on the plans for new ‘cycle superhighways’ going through Central London. The broad objective was to seek public approval for the plans on a stage-by-stage basis. In contrast to the BEIS consultation, their objective was quite closed: how many people approve of each part of the plans and how many disapprove? Quantitative analysis can then reveal correlations in the data: Which groups were more likely to approve the plans? Were there correlations between approval of one part of the plan and disapproval of another? These headline figures can then be supported by any qualitative data collected – people’s reasons for approving or disapproving. 

Survey question asking whether people support proposals for a new North-South Cycle Superhighway with radio buttons answer component and a qualitative text box for more comments on the proposals.

The tools in Citizen Space to help you make sense of your data

Where to start?

It can sometimes be tricky to know where to begin interpreting your responses. Citizen Space allows you to either view responses by respondent (all of the questions that one respondent answered) or responses by question (showing the way all respondents answered one of the questions).

However, going to 'Analyse responses' allows you to go through the responses one by one, and there isn't really a better initial way of getting to know your data than actually reading what people have said. In Analyse Responses you are also able to tag (code) the responses, to add your own notes to each question, to pull out quotes or useful chunks of comments, and to answer any analyst only questions you have set up. This article tells you all about using the Analyse Responses page and is well worth a read.

Making sense of text (qualitative) response data

Understanding and making sense of people's text comments in order to turn them into something tangible (like a consultation report) is generally the longest part of the analysis process. At the start, it's often a large jumble of lots of opinions or evidence and - as we said at the beginning of this article - the goal is to turn that cloud of information into something which makes sense, something that can form a narrative. If, like BEIS, you want to create a useful document of evidence upon which policy (or other) decisions could be made, just bunging all those un-interrogated comments in a document for people to read is unlikely to be very helpful.

We need to:

  • Know what people are saying to us in their responses
  • Identify any common themes and actions
  • Identify how people might feel about our proposals; and then
  • Deliver that in an easy to understand way.

Coding (called 'tagging' in Citizen Space) can help turn qualitative information into quantitative, this can help you to:

  • Track any themes and topics emerging in comments from respondents
  • Track any sentiments emerging in comments from respondents (i.e. broadly supports plans; unhappy; etc)
  • Tidy up poorly designed questions (i.e. when a quantitative set of options should have been given instead of a text box)

However, tagging/coding is not the same as collecting quantitative information. Coding each response into these categories will give you an idea of the numbers in each group, but as these were originally comments/evidence, it might help you to use the analyst notes field to pull out any quotes or useful supporting phrases from these comments to use in your report. This will help to add context to your themes and numbers. This article explains more about tagging/coding and how to do it.

Building a process around coding/tagging

You can either decide which themes to look for in your data in advance, or explore your data to see what themes emerge in response to your questions, or you could combine both approaches. A few of our customers take a combined approach, reviewing the first 10-20% of responses first before deciding on the relevant codes/tags to add. Using a google doc which is shared with colleagues to create a code book can help with this, it also helps ensure the codes added are less subjective. Planning some codes in advance can also ensure you keep the number to a sensible limit and don't duplicate.

For example:

Question:  What improvements would you like to see in the surgery?

If the responses in the comments field broadly fall into 3 categories: a) those proposing shorter waiter times b) improvements to the waiting room area and c) longer opening times, you could use the following codes: 

  • Tag 1 - Shorter waiting times
  • Tag 2 – Waiting room improvements
  • Tag 3 – Longer opening times
  • Tag 4 – Other response

Making sense of quantitative data

By comparison with text responses, working with the answers to quantitative questions is a whole lot easier. Here the work has been done for you because you've given your respondents the options to choose from in the survey itself and they've made their selections. Citizen Space then aggregates those answers and delivers them as charts and tables. Fairly simple. However: you do still want to understand this data so that you can do something with it, such as tell its story, too. Sure, you could just put the numbers into your report as they are, but there's real value in interrogating this information and looking at how the numbers relate to one another. 

Use grouping (cross-tabulation) and filtering to query your data

Cross tabulation (called 'grouping' in Citizen Space) provides an excellent way of comparing groups of information, such as comparing the response data from two questions to determine any relationship between them. There's more about the practicalities of doing this in Citizen Space here. Here you might want to see, for example, where people said they live against what they think is the most valuable service your organisation provides. You'd then see which areas value which services the most (or least) which could help you to share that story in your report. 

Unlike a cross tab, which compares two questions, a filter allows you to think of a query like: 'I'd like to see all the responses from people who said they live in <area>'. This allows you to look at just those respondents from <area>, to see how they answered other questions. By looking at the same question with different filters applied, differences between the various respondents represented by the filter can be quickly seen. You can even filter on multiple questions and criteria e.g. I'd like to see all respondents who live in <area>, who also said they like cheese, who also want to hear back from us about this activity.

By interrogating the data in this way, you may be able to find patterns or trends that help you to understand why a person answered the way they did, or to add real insight into your report. 

Top tip: Always remember to clear filters or groupings from your previous query as they do not clear automatically.

This kind of querying (and many other reasons) is why survey design and working out upfront what you need to know is so important, because it means you can ensure you ask those questions in your survey to allow you to get the information you need. Survey design is covered in our article on preparation and design.

What about responses from elsewhere? 

If you're getting to know your response data, then it isn't always just the data collected online you have to think about. Your response data might include responses collected offline, through focus groups, email and other formats. 

All of this is evidence going into your activity and could be fed into your consultation report or the next part of your process. How you choose to work with this data is down to you, but do think about the forms these other responses take.

If you're getting back paper versions of your online survey questions then - as they'll largely be in the same format as your online questions - you could add these to your Citizen Space dataset using the 'manually add a response' function. This will add those to your dataset in Citizen Space so that you can analyse them alongside all your online responses. Citizen Space will also log that these came in elsewhere by giving them a slightly different response ID, which starts 'BHLF', so you can track how many you have received and can report on that.

For responses taking a different form, like emails, postcards, letters, notes on carrier pigeons, and so on: after you've read them you could maybe consider using the same tag themes and sentiment codes you used for your qualitative analysis in Citizen Space to allow you to log the trends for these as well. You might choose to note how many you have received as well because all of this will be useful detail to include in your report. Working with these types of responses will be harder as they aren't uniform nor are they automatically added to a central place for analysis and exporting as online responses will be, so it's worth thinking about how you can pull useful information out of them.

Here's a bit more information about manually adding in offline responses.

What if I prefer working with data in Excel/SPSS/something else

That is absolutely fine! You don't need to do analysis in Citizen Space at all if you have a different way of working. Citizen Space provides full data exports for your responses, so you can take these and work wherever you are most comfortable. When you export the response data from an activity from Citizen Space, they'll be in the form of an .xlsx file. These can be opened and worked with in Excel or other spreadsheet/analysis tools like SPSS and others. This article describes how to export your responses.

Some tips on exports:

  • In Citizen Space, you can download the full response data at any stage, whether the activity is still open or not. Make sure that the data you are analysing is up to date and the latest version.
  • Once you have downloaded your data, keep a ‘clean' copy of the data so that this first sheet is ‘read only’ - this ensures you always have a copy of your raw data to hand, which can be useful if you accidentally make changes or delete the data as you manipulate it.
  • If you are not familiar with Excel, or need a source for refreshing your knowledge, Microsoft has its own guidance and YouTube has a bunch of videos for carrying out specific tasks. 
  • If you're using SPSS, there's also a host of good tutorials online: this one from the LSE's website is useful.

A note on analysis and campaign responses

By looking at and understanding your data, you may find in some activities that you have received a number of standard (non-unique) responses, for example as part of a campaign. It's important to include all responses in your analysis. However, you may also wish to carry out additional analysis excluding the standard responses in order not to skew the results just by the numbers alone. You may also wish to highlight the views of particular groups separately. For example, if your policy change affects hill farmers more than other types of farmers, you could provide analysis of responses for all respondents, and also analyse responses of hill farmers against those of other farmers.

Be aware of which stakeholders/groups/people did not or have not yet responded to your activity, and consider whether they need to be approached through other means.

Including a table of campaign responses as an appendix - as Department of Health did with their report on standardising tobacco packaging - can help. Referencing the different campaigns which were run and by whom can also be useful insight to include in a report. 

Example of a table to include in reporting for campaign responses.