����TV

In July 2017, ����TV was a mid-tier Certificate Authority and web security company with about 250 employees. About 40 of those employees were in Software Development, and they all sat within shouting distance of each other on a single floor of an office building in Lehi, Utah. By December of 2017, ����TV had exploded to more than 1,300 employees, with over 200 Software Developers, and scrum teams scattered across six global regions.

This rapid change occurred when ����TV acquired Symantec’s Website Security business on October 31, 2017. The purchase was triggered by a decision from major web browsers to distrust most TLS certificates, used to secure website communications, issued on Symantec’s infrastructure.

The distrust happened incrementally between April and October of 2018. In order to keep former Symantec customers, ����TV needed to restore the trust of the browser companies by reissuing millions of certificates within a few months, while at the same time providing customers with the new functionality to continue securing their websites and systems. Harnessing the newly expanded software development group to meet these challenges required massive and rapid changes to internal process, and agility was the only way to go.

Challenges

The newly expanded ����TV faced a daunting set of challenges, including a mountain of product offerings from the two companies, increased compliance scrutiny from industry groups, and the need to move rapidly to address the inherited waves of distrusts.

Product Catalog – ����TV needed to be able to respond quickly to customer needs in the product line to maintain the loyalty of their newly increased customer base. Also, compliance requirements for TLS certificates, as established by the Certificate Authority and Browser Forum (CABF), were evolving rapidly, and ����TV had to respond promptly to maintain trust with the browsers.

Both ����TV and Symantec came into the merger with numerous legacy products still being supported—many new products had been released without replacing legacy products. The new company needed to concentrate development efforts on a single, consolidated platform, and the product catalog had to be reduced to concentrate resources on solving new challenges.

Processes and Communications – The expanded size of the company mandated increased speed in decision making processes and communication channels. The new, larger ����TV needed to be able to respond rapidly to changing and emerging customer demands and regulatory requirements. It needed to become a learning organization that could “turn on a dime, for a dime” as new needs came to light.

Solutions

����TV decided on Scrum as the basis for their software development process. Both companies had used Scrum to some extent before the acquisition, but to establish a common baseline of attitudes and practices, Scrum training was immediately offered across the entire development organization. A three-hour course, based on , was developed and delivered by an internal Agile coach to over 140 team members and product owners during the first three months following the acquisition, either on-location or via web conference for remote locations. The training concentrated on the basics of Scrum – roles, ceremonies and artifacts (backlogs).

Members of the Product organization, most of whom had previously held the title of Product Manager, became Scrum Product Owners, and began building out product backlogs to detail features for the future products. Developers, QA engineers, writers, and UX designers stepped up to the challenge, took on scrum roles, and quickly organized into Scrum teams. This hands-on training inspired team members to be a scrum leader within their teams and was a catalyst for fast change.

Product Catalog – In accordance with Agile principles, ����TV worked to make data-driven decisions about product direction. User Experience Researchers engaged with customers of all types and sizes to gather input on customer needs and challenges. Feature comparisons between various existing products also helped to establish a baseline. All this data was used to decide on the direction and feature set for the consolidated future platform and products.

The go-forward features were concentrated in two areas: customer-facing functions would be built into ����TV CertCentral, our customer portal for ordering and managing certificates, and enhancements were also planned for the internal validation system, which is used by ����TV employees to verify customer information before issuing certificates. CertCentral and validation scrum teams worked to refactor their code bases and ramped up to act as mentors for new teams who would move over to help with consolidation efforts. Legacy products were not immediately pulled out of the channel, but development work was quickly wound down so the developers could move on to the new line.

Processes and Communications – Communications and training were critical. Technology-based communication channels were implemented to supplement face-to-face interactions (shouting across the office) that had been the primary method of communication before the acquisition. This included group chat systems, video and audio conferencing, and centralized, cloud-based repositories.

At the team level, those with experience in the go-forward product worked with new teams to share product experience and technical knowledge through joint sprint planning, backlog refinement, sprint demos, design review, and code review meetings. Daily standups were also temporarily combined across teams when warranted to facilitate teamwork and communication.

Sprint demos were also a primary means of sharing product knowledge across the whole company, with meetings recorded for sharing across time zones.

Area product owners were assigned to the two main product areas, CertCentral and Validation, and they work with local team product owners to coordinate backlogs.

Continuous process improvement is the goal. At the end of each sprint, teams hold retrospectives to review their work, process, and to talk about improvements they can make for the next sprint. An internal agile coach works with each scrum team on a rotating basis to share best practices and encourage adaptation. This internal coach, coupled with the scrum teams’ ability to embrace and persevere, has been our true success.

Results

Less than a year into the newly expanded ����TV, we’re still in the thick of growing, adapting, and changing. But results are already show. New, customer-focused features have helped validation wait times drop to near pre-acquisition levels despite significantly higher volumes. Average customer wait times to receive a certificate have dropped from several days immediately following the acquisition to a couple of hours. Teams have replaced nearly 5 million distrusted Symantec certificates. At a recent Customer Advisory Board meeting, a representative of a large customer said this: “While I was preparing to attend this meeting, I looked over my previous list of ‘still needs to be done’ items from ����TV. Amazingly, all but one of those things has been completed.”

In April 2018, just a few weeks into our Scrum adoption, we sent out a survey to development and product team members to gauge how they felt about the new processes. In October 2018, we reran the same survey to see what had changed. Some highlights of the changes that occurred over that six-month interval include:

• A nearly 20% increase in the number of team members that were regularly participating in Scrum ceremonies.
• Increase from 70% to over 90% of teams who understand how their work fits into the overall business direction.
• Drop from 55% to less than 25% of sprints are interrupted or abandoned.

Typical employee comments about Scrum adoption from the second survey include:

• “It's working well for the most part and is doing better at changing/adapting as needed.”
• “I think it is working super well in coordinating within the team.”
• “Keep improving the process!”

We are leaning into the challenges of our newly expanded company head on. While there is still plenty of room for growth and improvement, we feel that we are on a path of continuous positive change. We’re excited to see where this path will take us. And, thank you to all the customers who’ve taken time to give us feedback as we worked to learn what areas we can improve on.