Baruch Computing and Technology Center (BCTC)

Information from CUNY Regarding Blackboard Service

From: CUNY-CMS Admin sublist [mailto:CMS-ADMIN@LISTSERV.CUNY.EDU] On Behalf Of Brian Cohen
Sent: Monday, April 27, 2009 5:08 PM
To: CMS-ADMIN@LISTSERV.CUNY.EDU
Subject: Blackboard 8.0 Status

Dear Colleagues:

I would like to take this time to update you on the current status of the Blackboard 8.0 outages today.

At about 8am today (April 27, 2009) CIS, while monitoring system performance observed a high use of the system resources that caused the BB8 system to become unstable and unresponsive. We initially waited a short time to see if this problem would resolve itself and, when it did not, we brought the system down. At that time we also notified our vendor community (Blackboard, Sun Microsystems, and Oracle) and collected the necessary log files and data they would need to help trouble shoot the problem and provide guidance and direction toward a resolution.

In order to restore the system it was agreed that the best approach would be to reset the system and all of its components. By noon the system was restored and made publicly available to the greater University community.

At about 1pm, the problem we observed in the morning was occurring again and as a result we immediately disabled the system pending further assistance from our vendors. Since this latest down time We have been applying several vendor provided patches and bringing up each of the Blackboard 8.0 components. We hope the guidance we have received from Sun and Blackboard will allow us to restore the system to a normal operating state.

Additionally, We are also working with Blackboard to complete the necessary planning to restore the system at the Blackboard hosted DR site should this outage continue. We will be consulting with many of our CUNY Blackboard advisors/faculty users as how best to activate the DR site when needed. While DR will soon be a possibility for the Blackboard application, the potential for data loss and down time as we move to the DR site and then return back to CIS is greatly increased and should be avoided if possible.

Over the past several weeks there has been little change to any parts of the system other then continued efforts to fine tune the application or to increase the frequency of our backups to insure the least amount of potential data loss in the event of a catastrophic failure. These fine tuning activities are reasonable and often necessary in a complex environment especially when new computer systems are put into production.

As I am sure you already know, the CIS team is committed to restoring the system as quickly as possible. We have all key staff working on this problem and have all our vendors either on the phone and for some, soon to be on site to assist and conduct a root cause analysis. I will keep you informed you of the progress achieved in this area.

Thank you
Brian (via Blackberry)