This story traces how the Australian arm of one of the world’s leading engineering companies, ABB, dealt with a sub-standard backup environment that was causing significant risk and compliance issues.
In 2005, ABB made a global decision to outsource all its IS Infrastructure services, including helpdesk, hosting, end user support, Network (LAN/WAN) and Voice. In Australia, the outsourcer took charge of 80-90 servers (predominantly Wintel with a few UNIX boxes); 1400 end users across 6 major campus locations and several smaller remote project sites.
Although ABB expected and planned for some issues with such a large project, some significant problems with backup started to emerge in 2006 and endured over a number of years despite the introduction of several improvement plans.
ABB experienced issues:
– Reporting was inconsistent
– Insufficient tape media provided to sites
– Data volume was exceeding backup capacity. This was coupled with long delays in upgrading capacity
– Mismanagement of backup media and off-site retention
– Agreed processes were breaking down
– SLAs for data restore were consistently missed
– Audits revealed that key applications were not being consistently backed up
– Data retention policies were not being followed.
The implications for the company were significant. Over and above the obvious risk of a catastrophic loss of data, came the inefficiencies associated with the extended retrieval times for lost data. Retrieval of lost data took anything from 1 week to 3 months. But in addition to this, ABB have high compliance hurdles to meet, both internal and external, including Sarbanes Oxley. Any breach of these compliance conditions exposed both individuals and the company to censure.
The scope of the challenge was significant. ABB had:
– More than 960Tb of data stored on tape media
– Over 10,000 tapes in use or storage
– Many media types (LTO1, LTO2, LTO3, LTO4, 4mm, 8mm, DAT, DDS, DLT3, DLT4 etc)
– A variety of Backup technologies: (Tivoli Storage Manager, Backup Exec, ARCserve)
– Hardware that was reaching or past end-of-life. (Most remote physical servers over 5-6 years old; A variety of tape auto-loaders of different makes/models)
ABB had already de-scoped its voice and network services and had started working with Logicalis in 2008/2009 so Logicalis had an existing understanding of ABB’s business and the technical environment. The two firms worked closely together to design the BaaS solution.
To meet ABB’s compliance requirements, EMC’s Avamar infrastructure was hosted on its premises and under their control.
– Immediately take control of all backups at remote sites
– Data from remote servers was copied onto portable media
– In parallel Avamar grid was installed by Logicalis in ABB locations
– Avamar agent remotely pushed to servers
– Data from portable media “seeded” onto the Avamar grid
– Backup policy was implemented and the data synchronized between remote servers and Avamar grid
ABB had some concerns with regard to the performance impact on the WAN. But because Logicalis was already running the network they figured out a backup timeframe that minimized any impact.
Subsequent backups have been running on average 30min-3hours over WAN outside of business hours.
“Killing two birds with one stone”
Whilst the new BaaS solution was being implemented it became clear that ABB could take the opportunity to upgrade some its legacy hardware. Some of its remote servers were 5-6 years old and its tape auto loaders were of a number of different makes and models.
ABB deployed new servers at a central location and were able to migrate the data from the legacy servers with minimal downtime in less than 2 weeks, before shipping the new server to the site for commissioning.
Currently all servers in Australia are on EMC’s Avamar technology.
– Lotus Domino Mail & Application servers
– File Servers
– Infrastructure servers (i.e. Domain controllers etc.)
– Database and application servers (i.e. MS SQL)
Future plans are being developed for an offsite archival solution using the same technology and ABB are also looking at expanding the solution to back up data from end user’s PCs.
The lessons learned
– All organisations must have a clear definition of its backup and data retention policy.
– Backup is not an archiving system, its there to recover data if you have a failure. If your organization needs an archiving system, that is a different solution. Do not confuse the two.
– Whilst tape backup is cheap, in the end the effort to manage it is high and younger staff are unfamiliar with technology.
– You must find a partner who really understand your business and prepared to go the extra mile.
ABB is one of the worlds leading engineering companies, with 145,000 employees in 100 countries with 2011 revenues of $38 Billion. The company has a mixed Swiss and Swedish heritage, but is now based in Switzerland.
ABB is heavily involved in power products and systems, discrete automation and motion technologies, low voltage products and process automation. It has been involved in a number of nation building projects. In Australia they were heavily involved in building the Murray link, the longest underground power link in the world. It has locations all round Australia, with campuses in all major cities. It runs a distributed computing environment.