The University of South Australia (UniSA) is a public research university in the Australian state of South Australia. It is a founding member of the Australian Technology Network of universities, and is the largest university in South Australia with approximately 32,000 students. UniSA was founded in its current form in 1991 with the merger of the South Australian Institute of Technology and the South Australian College of Advanced Education. It’s currently ranked at No. 29 in the World's Top 50 Under 50 universities.
The challenge of unprecedented demand
In early September 2020, UniSA’s vice chancellor David Lloyd told the country’s leading news analysis program, ABC TV’s 7.30, that applications to UniSA Online are up 70 percent.
The high demand was driven by increased applications from onshore, domestic students.
UniSA already had a strong track record in online learning, with an established, dedicated mobile app and a portfolio of mobile classrooms. Still, when the COVID-19 pandemic hit, regular lectures, classroom teaching, and exams went from physical to virtual.
The learning management system had been designed to service approximately 3,000 concurrent students and staff doing a variety of online activities. However, COVID-19 changed that, with all students additionally having to undertake exams online and creating further load.
The growth meant the Learning and Teaching Systems team had to ensure its systems could handle the greatly increased concurrent load.
"If all the students were going to be in the quiz module, which is quite resource-heavy, we needed to ensure it could handle up to 2,000 simultaneous users for the online exams alone, as well as supporting other concurrent activities. We had to do a lot of load testing of Moodle [learning management system] to see if it could cope," explains Richard Lamb, UniSA Manager, Learning and Teaching Systems, Information Strategy and Technology Services.
Clear visibility and proactive remediation
In total, UniSA has more than 32,000 students. The learning and teaching interfaces have to be working 100 percent of the time, as any faults can have severe implications on students and staff. Today, UniSA has the right tools and processes in place to meet surges in demand, but this wasn’t always the case.
UniSA has been using New Relic for approximately six years, but prior to that, a help desk call would involve delving through cryptic error logs without being able to spot patterns, and exporting data to a CSV file while the clock was ticking.
"With every deployment, we would be completely blind whenever there were problems … until New Relic," Lamb says.
By leveraging the power of New Relic, Lamb’s team can “see exactly what’s going on” and identify bottlenecks or any queries that might be causing system issues.
"With New Relic, our team can immediately see errors as well as throughput. Instead of three hours of bug-fixing, an error can be instantly alerted, and a deployment rolled back, Richard Lamb, UniSA Manager, Learning and Teaching Systems, Information Strategy and Technology Services.
New Relic has come to the rescue multiple times. Lamb recalls a recent incident in particular where New Relic identified a bottleneck with too many students all logging on at the same time—one minute before 9 o’clock—to sit for an exam.
“There was a bottleneck and a 30-second wait for students to get in. New Relic detected this, and we were able to advise students to log in much earlier in the future to avoid any issues,” Lamb says, describing New Relic as one of the university’s business critical applications.
UniSA added more servers and resources to each site, based on predicted demand. Throughout the month-long exam period the team maintained "eyes on glass" with someone constantly rostered to monitor the system using New Relic’s dashboards, he adds.
"Our alerting ecosystem has been set up so that if we have a problem at 3 o’clock in the morning, it will automatically notify someone on mobile, escalate accordingly via PagerDuty, and automatically put something on the status page to say that we’ve identified a problem and are working on it,” explains Lamb.
“The online student experience is exceptionally important to the university... New Relic takes away the stress while you’re focusing on trying to fix things."
"With New Relic, our team can immediately see errors as well as throughput. Instead of three hours of bug-fixing, an error can be instantly alerted, and a deployment rolled back"
Continuous learning
UniSA recently migrated its main Moodle environments, Student Portal, and Teaching Dashboard—which provides academics with key program, course, and student details—from on-premise type authentication to Microsoft Azure AD. Thanks to New Relic, the team immediately identified errors occurring on the Student Portal.
Lamb says New Relic was quick to counter a potential problem that could have turned ugly.
"It was Sunday morning at 8 o’clock. The help desk was open, but no calls were coming through, even though we could see errors coming in through New Relic. Since we could see what the issue was, we quickly rolled back to the previous authentication before the problem escalated," Lamb says.
The team now uses New Relic for all deployments, using markers so they can immediately see the before and after, and automatically get insight into any poor performing queries or bad code. Lamb says it has enabled a much more proactive, rather than reactive, approach.
"We had an issue just recently where, on the Student Portal, two of the web nodes suddenly went to 700 percent CPU use after a deployment. With New Relic, we could immediately see that and drop the two nodes out that were causing the problem.
“The error rate diminished to zero. We could proactively fix the issue before it became a mass problem for students," he says.
