Jira Align operations

Topics in this article:

• Backup and recovery validation
• Recovery process
• Capacity planning
• Adding capacity
• Certificate installation and rotation
• End-user management

Backup and recovery validation

The most important part of the solution and the single system of record is the site database hosted in SQL Server. All the other elements can be rebuilt with little impact on the overall system. In our implementation, we rebuild the network, OS instances, applications, and even move locations with automated scripts. The database hosts all the tangible elements of all the configuration.

Recovery process

Restoring databases

At the high level, the process is to create a new RDS instance of the entire database and then update the application connection strings to the new instance. With all of the fault tolerance built into the implementation, there has not been a case where we have had to recover a database for production. This process is used, however, for creating test replicas in UAT environments. This allows us to quickly update a customer environment for their own UAT testing to match their recent production changes to the actual data.

Each RDS instance is configured for an automatic daily snapshot.

To restore a database:

1. Open the AWS RDS UI.
2. Select the relevant database server.
3. Select Action > Restore to point in time.
   
4. The next page has additional configuration which includes:
   • Time (to the second)
   • DB engine
   • License model
   • Instance class
   • Mirroring config
   • Storage class
   • Tags
   • Parameter groups
   • VPC
   • Subnet
   • Accessibility
   • Availability zone
   • Security groups
   • Ports
   • Option groups
   • Active directory
   • Log config
   • Backup config
5. After selecting the correct elements, the new RDS is created along with the new connection string.
6. Atlassian then updates the web.config files for the sites, connectors, and mail server. We also validate by checking the site front end, Splunk logs, and deployment validation monitors.

Restoring a web server or connector

Web servers and connectors are deployed into EC2 with auto-recovery groups.

To restore a failed VM host (manually from the UI):

1. We select the instance from EC2 services and click Stop.
2. We then select Start.
   
   

However, our instances are configured to leverage AWS Status check, which sends an alert to OpsGenie and takes action to recover the instance.

If these servers fail for a reason that this does not fix it (this has never occurred), then we would create a new instance with our cloud formation template and run the installation process again, leveraging the existing database. The web servers and connectors only require the web.config files to be updated.

Capacity planning

In the Monitoring section, if the systems and counters are set up correctly, you will be proactively and be made aware in real-time of how software usage will require upgrades in system resources. If usage is too low, you can use the same data in trending to see if you can reduce resources.  

Adding capacity

When adding more resources to a cluster, there are a few factors to take into account:

Service type: BI ETL (SQL task that connects to the primary database)

Note: This service executes on a task scheduled basis. The workload is dependent upon the size of the primary database. The configuration of the BI ETL is a special case implementation that is not currently defined in the installation documents, but these notes are included for completeness.

• Storage capacity for space and Disk IO throughput max rate & IO latency
  Note: The transaction logging for the SSIS database can grow quickly. In our SaaS implementation, we minimize the impact by:
  • Setting transaction logging to SIMPLE
  • Setting maximum transaction log file to a reasonable minimum given the storage allocation of your server
  • Setting the SSIS database mode to Performance
  • Network utilization against max bandwidth and latency
  • Memory: Available Mbytes in balance with cache effectiveness
•  CPU:
  • Average Processor utilization vs. target of ETL process windows
  • Opportunity for server class migration (e.g. m5 to m5a)
• Pooled resources:
  • Scheduled task execution windows
  • Additional server instances in the multi-tenant environments

Service type: BI Data Processing (Azure Data warehouse and analysis services)

• Storage capacity - for space and Disk IO throughput max rate & IO latency
• Network utilization - against max bandwidth and latency
• ECU: Key Performance Indicator
  • Usage against peak
  • Additional server instances in the multi-tenant environments

Service type: Web tier

• Storage capacity - for space and Disk IO throughput max rate & IO latency
• Network utilization - against max bandwidth and latency
• Memory
  • Available memory is the Key Performance Indicator
  • Available Mbytes in balance with cache effectiveness
• CPU
  • Average Processor utilization vs. max sustained peak rate defined by Alert targets
  • Opportunity for server-class migration (e.g. m5 to m5a)
• Pooled resources
  • Moving clients into different load balancing groups
  • Moving a client into a new AWS region to match their user base

Service type: API

• Storage capacity - for space and Disk IO throughput max rate & IO latency
• Network utilization - against max bandwidth and latency
• Memory: Available Mbytes in balance with cache effectiveness
• CPU:
  • Average Processor utilization vs. latency targets
  • Opportunity for server-class migration (e.g. m5 to m5a)
• Pooled resources
  • Optimizing load balancer configuration
  • Additional server instances in the cluster
• Throttling
  • Optimizing configuration of API gateways to match usage

Service type: Connector

• Storage capacity - for space and Disk IO throughput max rate & IO latency
• Network utilization - against max bandwidth and latency
• Memory: Available Mbytes in balance with cache effectiveness
• CPU: Key Performance Indicator
  • Average Processor utilization
  • Opportunity for server-class migration (e.g. m5 to m5a)
• Pooled resources
  • Additional server instances in the multi-tenant environments

Service type: SQL RDS

• Storage capacity - for space and Disk IO throughput max rate & IO latency
• Network utilization - against max bandwidth and latency
• Memory: Cache effectiveness
• CPU:
  • Average Processor utilization vs. target defined in alerts
  • Opportunity for server-class migration (e.g. r4 to r5)
• Pooled resources
  • Moving databases to different RDS instances in the multi-tenant environments
  • Examine database instance count in RDS vs. maximum (100 is the new maximum as of May 2019)

Certificate installation and rotation

When the system is properly configured, these certificates are required for proper network communications:

Self-managed (digicert)

• Website SSL certificate: for each website or leveraging a wildcard domain certificate
• Intermediates: Certificate Authority certification if using self-signed for any of the above
• Connector - Mutual TLS Client certificate: to communicate with Jira Server if it is not a secure network

Atlassian has a partnership with Digicert to create and manage certificates that are part of Atlassian control. The SRE team has unique access to our agilecraft.com production certificates which provides the controls and audit logs to make this secure and compliant. If you use a self-signed certificate, which means generating an SSL certificate that is not from a current Internet-wide Certificate Authority such as DigiCert, then all of your users must have your custom CA installed in their local trust chain. This is probably manageable during a test phase, but is not practical for enterprise-wide deployment.

The website and Connector certificates are deployed via Octopus, which has a certificate manager built into the tool. The expiration for the certs are alerted via DigiCert, Octopus, and also by the monitoring tools at each site.

The website certificates are hosted on the IIS Server, Load balancers, and the CDN. Each of these solutions allows for programmatic updates of the binding, which enables SRE to make this change with a very low risk of downtime.

The connectors make outbound calls only so the control of the network activity is owned by the administrator.

To rotate the mutual TLS certificate used by the connector:

1. Add the fresh certificate bundle to Windows Server in cert:/localmachine/my.
2. Generate the public certificate as a DER in CER format.
3. Rename the certificate to clientcertificate.cer.
4. Place the certificate in C:\Instances\<Customer>\Jira1.
5. Open Windows Services.
6. Restart Agilecraft_Jira_<Customer>_1. This process will copy the public certificate that was generated into the SQL database for the site, and log that the certificate was changed.

Self-managed (AWS)

AWS manages the certificates that control access to the SQL server. These are rotated on a regular schedule and AWS provides the utilities to do this rotation, alerting, and validation.

For more information, see Microsoft's documentation to enable encrypted connections to the database engine. 

• SQL Server certificate: one for the SQL Server to communicate over TLS 1.2/FIPS as required 

Remotely shared

In some scenarios, we receive a certificate from the remote connection. This is the same process as above, except we have to manage the inbound private key in our certificate management solution instead of DigiCert.

• Connector - Mutual TLS Client certificate: same as above, but in some scenarios, you may have provided a unique certificate from your trusted CA.

End-user management

• SSO integration: The product documentation covers these steps in-depth. 
• Authentication: The product documentation covers these steps in-depth.
• User replication: The product documentation covers these steps in-depth.