Business continuity/disaster recovery for on-premise

This article is a subset of the Jira Align documentation of our Standard Operating Procedures for maintaining High Availability (HA) in our SaaS environment. This document should help on-premise customers plan and deliver their own High Availability solutions.

Major goals of the disaster recovery plan

• To minimize interruptions to normal operations.
• To limit the extent of disruption and damage.
• To minimize the economic impact of the interruption.
• To establish alternative means of operation in advance.
• To train personnel with emergency procedures.
• To provide for smooth and rapid restoration of service.

Application profile

The following sections detail the software components required to run the SaaS applications.

Website uptime Markov model

This diagram captures a high-level Markov model and implications for the solution site. This visualization is the documented uptime of the AWS components.

• ELB is a native component
• Web/app, connectors are on EC2 with EBS backing store
• SQL RDS is on RDS mirrored.

The calculated uptime for solution independent of software failure is:

Website & Connectors : Uptime[ELB] * Uptime[Web] * Uptime[SQL RDS] * Uptime[EBS] => 99.929%

Fault chain analysis – full user

This diagram captures the fault profile of the major elements that affect the use case/persona. The blue ovals are internal components and the green elements are external to AgileCraft (Jira Align) SaaS.

Fault chain analysis – Enterprise Insights user

This diagram captures the fault profile of the major elements that affect the use case/persona.

Green elements are external to AgileCraft (Jira Align) SaaS.

Fault chain analysis – connectors

This diagram captures the fault profile of the major elements that affect the use case/persona.

Green elements are external to AgileCraft (Jira Align) SaaS.

Logical architecture diagram

The diagram is intended to represent a Highly Available deployment of Jira Align that can be used on-premise.

There are three major points of design:

1. The load balancer for the web tier must be sticky and NOT round-robin. The user session state is maintained on the web server.
2. The connector for Jira and/or Azure DevOps is a Windows service. This service performs the synchronization queries. The system is fault tolerant to any form of failure at the service level because it performs the changes within SQL transactions locally. The service will retry based on the last completed work. This service is most easily deployed on a single server instead of a cluster. However, it is possible to configure using standard practices of Microsoft Clustered Services if so desired.
3. The SQL server should be deployed in a Microsoft ODBC compliance cluster. Note that the SQL server is primarily OLTP usage, so the fault-tolerant pair should be laid out for minimal impact to write operations. In our production SaaS, we use AWS storage which has 3000 IO/sec with an average latency of less than 1ms. We highly recommend a storage solution on SSD to enable this.

Information services backup procedures

The following section captures backup procedures. The Jira Align customer solution has a single system of record, where all customer data is stored for the application. This database is the only component required to regenerate all customer data.

AWS RDS provides the design and overall solution for database management as a service. Thus, the process for backup and restore of the actual database is a service described at https://aws.amazon.com/rds/details/backup/. After the restore, depending upon the use case, the operations team can do any of several next steps:

• Restore database from point in time
  • Perform RDS restore in AWS UI
  • SSMS connect to new RDS instance
  • If Single DB then:
    • AWS backup into S3
    • Restore backup
• Move customer production database into their UAT environment:
  • Create local SQL backup to S3 in SSMS
  • Run AWS restore and rename on top of the existing test system

The Jira Align backup retention policy is 35 days for multi-tenant and the same default for the dedicated instances. You can customize the retention policy in a dedicated instance.

As of September 2020, no data are 'trimmed' for hygiene purposes due to the nature and size of the data within the application.

Disaster recovery procedures

The following sections capture the steps to recover the solution in the event of different failure classes. This table maps to the website uptime model above. The table below maps failure elements that are expected or can be validated.

Automated recovery EC2

The settings below show how the elements are monitored:

This shows how the monitor recovers the server:

Emergency response procedures

This section documents the appropriate emergency response to a fire, natural disaster, or any other activity to protect lives and limit damage.

• Create HOT via PIR
  • Gather update from vendor or location about expected downtimes
  • Determine which use cases are at risk

Backup operations procedures

Backup operations procedures ensure that essential data processing operational tasks can be conducted after the disruption.

We recommend keeping track of quarterly backup testing internally.

Zone fault

We recommend keeping track of quarterly backup testing internally.

Disaster action list

This list provides possible initial actions that you might take following a disaster.

• Gather team
• Communicate to customers from Product Support communication list
• Create change log
• Execute recovery startup procedures

Fallback plan

Failback steps are only required for a horizontally scaled solution. The HA configuration of the solution can run with new primary without any changes.

Testing the disaster recovery plan

We recommend keeping track of quarterly backup testing internally.

Schedule

The process outlined here is executed quarterly so that we can ensure the plan's ongoing effectiveness in addition to familiarity with the process for the engineers involved.