Piloting the Digital Experience Insights Service (2018)
19 November 2018
DEI-CS-UoQ.jpg
Credit: © The University of Queensland.

Data from over 9,000 students provides a solid evidence base

Overall, using the Insights survey has been fabulous. There are so many wins to using this survey: it has a good history of use and provides a solid evidence base to support what we are doing at the university.

Project lead: Dr Sam McKenzie, project manager, Institute for Teaching and Learning Innovation, University of Queensland

Project aims: Informing innovation and checking assumptions

The University of Queensland used the digital experience insights service for the first time in 2017-2018. The student survey ran alongside key development initiatives that were already underway and the aim was to get reliable and up-to-date information to support:

  • The implementation of a bring your own device (BYOD) policy. There was an expectation that students would have wifi portable devices to bring to campus and that they would use these both in and out of class to support their learning. The university wanted to better understand and test this expectation and the issues around BYOD
  • The university was looking at how they could change from a centrally invigilated paper-based examination system to make use of e-assessment opportunities and to provide more authentic assessment instruments

Refurbishment of learning spaces through a campus wide strategy that includes student use of own devices and developing multi-functional study areas

Strategies for engaging students

The university decided to invite all higher education students to participate in the survey to get the full gamut of student experiences and reflections. With a student population of approximately 55,000 students, the university was delighted to receive 9,987 – an 18% response rate.

Early discussions with colleagues led to the survey being launched as the 2018 UQ Student Technology Survey. Naming it by year signifies intent to run the survey in future years. Some basic customisations were made to the survey, for example, naming the learning management system used by the university as Learn.UQ (Blackboard) as virtual learning environment (VLE) is not a term the students would recognise.

The University of Queensland uses the Quality Indicators of Learning and Teaching (QILT) scheme. This is a national scheme supported by a website which provides prospective students with relevant, transparent and up-to-date information on the study experience and employment outcomes data from Australian higher education institutions. Students are able to compare statistics from different fields of education, universities and programs. A small student focus group was used to test out the relevance of different potential fields with students. This revealed that study area was the field students identified with most, rather than a school or faculty.

The final questions were open-ended and deliberately designed for ease of completion. Students were asked ‘Have we missed something?’ and ‘What else would you like to tell us?’

Students received targeted communications via several means:

  • A student email was sent out to all 55,000 students using the customer relation management (CRM) system. These were individually addressed and were sent from the academic registrar. It was interesting to note that the survey was launched at 12:30am and the first response was received just nine minutes later. 4,316 replies were received on day one. A week later, as the response rate naturally started to dwindle, follow up emails were sent which resulted in a second spike in responses with 2,369 respondents on day eight.
  • The project team of seven staff included a social media expert so platforms such as Facebook, Twitter and Instagram were used to help promote the survey, maintain the survey profile and to send out reminders to participate. The Office of Marketing and Communication (OMC) retweeted and reposted the social media postings along with team members from the Library and Institute for Teaching and Learning Innovation. Getting the OMC onboard was instrumental to achieving a high response rate and also meant that the project team were able to leverage further support.
  • Digital signage channels such as announcement screens across the university and at entrance points were used. While different screens were owned by many different owners, a significant success was securing approval from the owner of 70 centrally-controlled screens broadcasting announcements and information in teaching and learning spaces.
  • Communication materials produced by the insights team were also used.

A prize draw incentivised students to complete the survey with the opportunity to win donated coffee vouchers, gift cards and the main prize of an iPad.

Data analysis process

The high response rate surprised the project team and has presented challenges in data analysis. The initial expectation was that approximately 600 to 1000 responses would be received so receiving nearly 10,000 responses has given the small, grass roots project team a bigger than expected challenge. Initial data analysis was done using the standard output from Jisc online surveys but the university evaluations unit has agreed to take on more detailed data analysis. This brings in considerable statistical expertise that is enabling the university to make best use of the valuable data.

Data from previous IT services surveys about wifi provision and library services provides historical knowledge and context. Various surveys conducted in 2008, 2010, 2012 and 2015 showed that a high number of students have laptops but don’t bring them to campus because they don’t feel they have to, they are heavy, or the batteries don’t last and they can’t find a secure power source to charge them safely.

Key findings

The overall global ratings are one thing but nuances are key. The quantitative data gave us a quick overview of the story, but the qualitative really brought to light the full story that explained the numbers.

Project lead: Dr Sam McKenzie, project manager, Institute for Teaching and Learning Innovation, University of Queensland
  • Students rated digital provision very highly at 92%, but the survey provided more nuanced data and identified areas that required further investigation such as wifi (satisfaction rate of 84%), how well the university is supporting use of personal devices and the timeliness of access to support
  • 75% of students said that they know they need digital skills in the workplace but only 44% felt prepared. This disparity provides a useful discussion point when engaging others across the university. With potential government plans to align funding policy to metrics such as student satisfaction ratings, retention and employability, this perception requires further investigation
  • The project team were surprised by the wifi response rate of 84%. It was thought that deadspots and patchy wifi provision had been identified and remedied after an internal survey conducted in 2012. The qualitative comments provided by survey respondents are highlighting wifi issues that the university can address. As in the UK, Australian universities have EduRoam so no matter what campus or university students are on they should be able to access wifi
  • The university now knows that 95% of students have laptops which they use for learning and 98.8% have smartphones or tablets or laptops. This is a key question not always found in other surveys and the emphasis is important: it moves the focus from ownership of personally owned devices to specifically reference use for learning. It also helps the university to address equity issues and perhaps indicates that a scholarship scheme to enable learners to buy a laptop may be more effective than a short-term loan scheme (current loan scheme does not allow students to take laptops off campus)

Acting on the findings

A key benefit of the insights survey is that it has eliminated some hurdles for the projects going on at the university. We can now say that we have reliable and current data from 10,000 students that will enable us to focus on improving things across the university.

Project lead: Dr Sam McKenzie, project manager, Institute for Teaching and Learning Innovation, University of Queensland

Data from the digital experience insights survey is being used to inform other projects and initiatives.

The university evaluations unit have cleaned the data (for example, to remove nonsensical answers) and are producing the first draft of reports for each curriculum area and level of study. These draft reports reports have been presented to a strategic group whose members are connected with a wide range of the university’s faculties and service departments. The strategic group, in collaboration with the project team, has developed short and mid-term actions and will help guide the dissemination of the results that are relevant to their areas.

The reports will then be formalised and disseminated via each of the 30 teaching and learning committees that support the individual schools at the university. Dissemination activities will include opportunities for the teaching and learning committees to discuss the findings and to see how they compare within the university and against international benchmarking data, although the high sample size from the University of Queensland and different sampling methods makes a direct comparison with other institutions difficult.

The findings will also be shared with the university’s learning designers’ forum and other relevant committees. Learning designers are a key audience as they are able to prompt and support changes to academic practice.

The OMC would like to publish the findings too and share these through their media channels.

Tips for others and lessons learned

  • Getting senior management approval to run the survey was important. The approval of the academic registrar to send out an email to all students was instrumental and the support from the OMC with communication was valuable
  • It is relatively easy to implement the survey but can be harder to manage the data once the survey closes. The higher than expected response rate caused capacity issues for the project team who recommend considering the data analysis process and how this fits into work schedules early on in the implementation process. If you have data analysis expertise within the university, their help is likely to be invaluable.

The involvement of our evaluations unit in analysing our data is enormously beneficial and a significant bonus. This seems to be a measure of the importance the data is being afforded.

Project lead: Dr Sam McKenzie, project manager, Institute for Teaching and Learning Innovation, University of Queensland

Next steps

The survey findings will feed into strategic activities from the university’s Strategic Plan 2018-2021 and the Student Strategy Initiative.

Data from the student experience insights survey is also integral to the work of a task force that has been established to target digital capabilities development. Using the survey results as part of their evidence base, the task force launched Digital Essentials, a series of online modules for students to quickly build digital skills so that they can succeed in study and work

The university is considering using the Jisc discovery tool to encourage students and staff to self-reflect on and develop their digital capabilities.

Project team and contact details

The project team included:

  • Dr Simon Collyer, manager, eLearning Systems and Support, University of Queensland
  • Mr Nick Fitt, manager UX services, Library, University of Queensland
  • Associate Professor Pedro Isaias, associate professor of higher education innovation, Institute for Teaching and Learning Innovation, University of Queensland
  • Dr Sam McKenzie, project manager, Institute for Teaching and Learning Innovation, University of Queensland.
  • Dr Christine Slade, lecturer of higher education, Institute for Teaching and Learning Innovation, University of Queensland
  • Mr Jacob Tilse, online content coordinator, Library, University of Queensland
  • Mrs Noela Yates, acting manager information and digital literacy team, Library, University of Queensland

Contact:

Dr Sam McKenzie (email: sam.mckenzie@uq.edu.au), project manager, Institute for Teaching and Learning Innovation, University of Queensland.