Baruch Computing and Technology Center (BCTC)
BB 8.0 Update
From: CUNY-CMS Admin sublist [mailto:CMS-ADMIN@LISTSERV.CUNY.EDU] On Behalf Of Brian Cohen
Sent: Tuesday, April 28, 2009 11:49 PM
Subject: BB 8.0 Update
For reasons I am sure you understand, this has been a very frustrating day for the team at CIS. Just prior to the sending of this update the CIS team was able to restore the Blackboard system.
Beginning around 10am today, Blackboard 8.0 began to show the same slow response time and high system resource needs as it did for the last week.
In spite of the efforts we undertook yesterday and under the advice and guidance of our vendor advisors, the high demand for CPU resources caused the system to once again become unresponsive. While CIS was able to restore the system quickly, the system failed again at 12:39pm with all the same traits as the earlier problems in the day. After several attempts to restart the system to acceptable levels of service, we were unable to achieve a level of service or response time needed to support our user community.
Throughout the day today, the CIS team has been working side-by-side with senior and highly specialized engineers from Blackboard, Oracle and Sun Microsystems. This team witnessed the erratic behavior and captured additional diagnostic information that will hopefully assist in identifying the root cause(s) of these latest BB8.0 problems. At this time the CIS team continues to work to get the system restored to a normal level of service and our vendor team is reviewing the data captured from today and will be meeting again on Wednesday to work through their findings.
While CIS strives to provide highly reliable Blackboard services, we have clearly been challenged by the problems that even some of the most experienced engineers and technicians cannot solve so easily. As discussed in previous reports, we have been working with Blackboard to formalize a contractual relationship for Disaster Recovery (DR) services. Today, Michael Chasen, CEO of Blackboard and I both agreed that best course of action at this time is to move the CUNY Blackboard 8.0 production environment over to a Blackboard hosted service. While in a hosted state, the CIS team will along with resources from Blackboard, Oracle, and Sun continue to focus on analyzing the data collected to identify the root
cause(s) of these ongoing problems. This hosted approach will allow our user community to continue to use the Blackboard system to meet their class requirements and for CIS to continue with the needed diagnostic work. I assure you, CIS will not move the system back to our data center until we are confident that either a root cause has been identified or sufficient proof is given that the Blackboard system can operate in a reliable and steady state at 57th Street.
To carry out this hosted system objective, I have identified a team of CIS staff responsible for achieving this transition to the Blackboard data center. Some of the tasks that need to be completed before we can cut over to a hosted environment include: moving the most current copy of the CUNY Blackboard 8.0 data/content to the Blackboard data center, establishing a technical and secure process that will allow our users to authenticate and login to the system at Blackboard, and modify the current network architecture to allow for the increased network traffic over the internet for this Blackboard usage. Teams from both Blackboard and CUNY have already taken steps in each of these areas. Data transfers have been tested and the actual transfer of data has started. I have been assured by the Blackboard CEO that his team will work quickly to accomplish this cut over and he will be able to provide a better estimate of the time needed to complete the cut over once the data transfer has been completed.
As always, I will continue to keep you informed of progress made in each of these areas.
Brian (via Blackberry)
If you wish to unsubscribe from the CMS-ADMIN List, please send an E-mail to:
"firstname.lastname@example.org". Within the body of the text, only write the following:"SIGNOFF CMS-ADMIN".