News

Exam Result Access Issues – Report on the events of Thursday February 4th 2016

5 Feb 2016

Semester One exam results were released yesterday around 12 noon. Earlier in the day an email was set to approximately 12, 000 students informing them of this fact.

The email was delivered to mailboxes between 11:49 and 12:17. This coincided with a spike in access to http://sit.ucc.ie, the webpage used by students to access MyStudentAdmin.  

Exam Result Access Issues – Report on the events of Thursday February 4th 2016

 

Approximately 2,500 hits a minute were coming in to the site, this looked to the server like a denial of service attack. Users began to experience time out errors when attempting to access SIT. Those for whom the SIT page did load, proceeded to click in to MyStudentAdmin.

Around this time IT Services noticed the performance issue on the main UCC webserver. The server was up and running normally, however page loads were slow. The server configuration was altered and the server rebooted. This stabilised the main UCC website and SIT. However, there were further issues to be discovered.

 

The server that authenticates user logins was unable to handle the rate of requests. BB5 (an authentication server, that I now really wished we’d called BB8) connects to the ITS database. ITS is the main student records database.  The configuration on this database only allowed 100 or so concurrent logins. This was an oversite, the server was recently updated and default setting were not changed. At 12:29 we had over 2,500 students attempting to access this database. As the database did not, at the time, allow this, students were denied access.

It took IT Services some time to link together the string of settings and dependencies that resulted in the access problems students experienced.

 

At approximately 3pm the systems were all working correctly together and students could access their results.

IT Services strives to deliver the best possible experience of IT to UCC’s students. Yesterday’s events are regrettable, and perhaps we could have anticipated them.  The chart below compares 2015 exam results day, with yesterday. Yesterday is the blue line.

 

As you can see attempts to access results began before they were released, with a massive spike around noon. The number of hits was well over double last year’s maximum, this combined with the limit on concurrent access to ITS was the root cause of yesterday’s login issues.

IT Services will learn from yesterday’s events, processes will be reviewed, causes and solutions will be documented to ensure such issues will be avoided in future.

 

 

Student IT Services

Boole Basement, University College Cork, College Rd.

Top