Monitor Jira Align

Recurring HTTPS fetch of login page

Our default is one minute from three external locations. Select multiple site locations from which your customers primarily reside to both help debug issues per location, as well as to narrow down issue to network configuration vs. server/DB issues.

Our implementation uses Rackspace Intelligence to make HTTPS calls. The rule management engine pushes the alerts to OpsGenie.

Detection Value

• Issues between web tier and DB as well since the page will return a 500 error if the ODBC connection string fails.
• Calculate overall service availability based upon history of successful requests, a per-minute check will enable fine-grained resolution.

Recurring fetch of Certificate Validation

Our default is the same as HTTPS above, but that is overkill. Certificates should probably be only validated on a daily basis.

Detection Value

• Stale certificates
• Certificate minimum security rules

Static Elements

The web server hosts static elements that are marked for caching. In our cloud-based environments, we use a CDN, CloudFlare, which we monitor for cache effectiveness and locality of delivery. If the customer environment uses a CDN or reverse proxy cache this should be included in your analyses.

HTTP/S Requests

The rate of requests has very little effect on overall system performance.

The efficiency of the HTTPS handshake is important. However, the Time to Live for the handshake might force the end-user browser to reconnect too frequently. The defaults for the IIS configuration for this are a great place to start and finish.

Primary Resource Constraint

The server hosting IIS is primarily constrained hosting the memory for the ASP session state for the concurrent users. The best way to prevent faults in the application stack or reduced performance is to look at the W3WP process that hosts the application pool and ensure that it has enough memory. 

There are many practices to alert on memory constraints, but we look at the following:

Perfmon Object: Memory

Available Mbytes: when this counter is less than 1000 for 15 minutes, we send an alert to the team. This might indicate a need to add a server or memory to the existing server. However, we also check to make sure that some other process is not exceeding its normal usage. 

The application stack, minus the API, includes:

• Active Server Pages engine
• .NET Framework
• .NET Data Provider for SQL. 

From a reliability perspective, the application pool will restart based upon Application Pool Settings in IIS. The application pool must be restarted periodically to clear stale memory. This will show up in the Windows Event Log as Event ID: 5186 with accompanying description:

A worker process with process id of '2180' serving application pool 'test' was shutdown due to inactivity. Application Pool timeout configuration was set to 20 minutes. A new worker process will be started when needed.

Perfmon Object: Active Server Pages

ASP Requests rejected rate should be 0

When calculating availability, we do not solely consider the downtime in wall clock measurement of the servers during maintenance. We calculate the work that was actually missed. We capture a history of workload during the downtime to actually measure the transactions which were unfulfilled. The ASP requests/sec history is a good place to capture that or HTTP requests from the IIS log. 

Web Application Logging

The web server outputs errors and exceptions to a daily JSON formatted log file via the Serilog library.

The Windows server ASP engine has native performance counters. These are the most interesting

Perfmon Object: Active Server Pages

• Request Execution Time: Average request execution time should be less than 1 second
• Requests/sec
• Script Engines Cached: observe this count to help tune the cache hit rate
• Sessions Current: this is the count of active sessions that ASP keeps in memory per server. This correlates to your active users in the system. Assuming the system is not under memory constraints and is configured correctly.
• Templates Cached: observe this count to help tune the cache hit rate
• Template Cache Hit Rate: This should be greater than 90%, except directly after a restart.
• In Memory Templates Cached: observe this count to help tune the cache hit rate
• In Memory Template Cache Hit Rate: This should be greater than 90%, except directly after a restart.
• Script Engine Cache Hit Rate: This should be greater than 90%, except directly after a restart.
• Requests Executing: at peak, this should be greater than 1.0 which indicates the server workload is doing proper parallel work. The math is a bit tricky, but if requests executing times requests execution time exceeds 1.0, then the concurrent requests execution should be greater than 1.0.
• Requests Queued: this should not be above 0 for any extended period of time

The authentication engine and session management is controlled by a DLL that is executed within the Windows server Dot Net Framework. The performance counters of interest are

Perfmon Object: .NET CLR Memory

• % Time in GC (global): This should be less than 1%. If this exceeds 5% or more for 2 minutes, then your system is not behaving efficiently. There might be a configuration error or some other task running on the server which is impacting the execution of Jira Align. A good next step would be to look at this same counter broken down by process 

The application communicates to the SQL server via .NET data provider. This DLL has performance counters too:

Perfmon Object: .NET Data Provider for SqlServer

• NumberOfPooledConnections (for the web instance): The application pool interacts with the data component to stabilize the connections. If the connection count is 1 while under load, then there is a broken configuration somewhere. Having the correct concurrent connections enables parallel requests between SQL and Application Pool. 

There are two versions of the API: v1 and v2. The v1 workload runs within the same application pool as the .NET framework as the webserver. V2 is a .NET core application. This is a separate process which should be monitored for faults that might affect reliability. 

NOTE: API v2 is still in early Beta at this time.

Each connector is a Windows service running within its own process. During installation, the services are named. this name can be used to narrow the focus of the monitoring for the specific process.

Process Level WMI Monitoring

We then leverage WMI calls within Splunk by checking the following:

[WMI:Services]
interval = 60
disabled = 0
index = default
wql = select Name, ProcessId, Caption, DisplayName, State, Status, StartName, SystemName, StartMode, DelayedAutoStart, Started, PathName from Win32_Service where Name like 'AgileCraft_%'

We have checks to ensure that the processes are executing. We also watch trends for leaks. If the service exceeds a threshold, we restart it. Windows Services also have restart operations in the event of a service executable crash. These restarts are recorded in the event log. If the service restarts frequently then that would be a good defect to report back to Atlassian. This is managed code, so some restarts are essentially benign.

Sample log info for Services

20200121130315.625605
Caption=AgileCraft_Jira_Customer
DelayedAutoStart=True
DisplayName=AgileCraft_Jira_Customer
Name=AgileCraft_Jira_Customer
PathName="C:\Instances\Customer\Jira\AgileCraft_Jira.exe"
ProcessId=4440
StartMode=Auto
StartName=LocalSystem
Started=True
State=Running
Status=OK
SystemName=Customer
wmi_type=Services

Note: The connectors maintain transactional integrity in the database, so service restarts will not affect data integrity. Network errors are actually fairly common in our implementation over the public network. We also test many types of faults to insure they log the issues and the data are still safe.

Connector Application Logging

The connector outputs errors and exceptions to the application log in JSON format via the Serilog library.

The Backoffice service is a Windows service running within its own process. The same feedback applies as above.
The Backoffice service also logs information about sending email. 
{
    "Timestamp": "2020-05-15T13:22:18.6475545-05:00",
    "Level": "Information",
    "MessageTemplate": "Email sent",
    "Properties": {
        "ThreadId": 31,
        "MachineName": "Server1",
        "ServiceName": "AgileCraft_BackOffice_Server1",
        "Build": "10.73.0.5461",
        "Product": "AgileCraft.Backbone",
        "EmailId": 73388,
        "CorrelationId": 672329761,
        "Instance": "_datbaseName_"
    }
},
{
    "Timestamp": "2020-05-15T13:22:18.6475545-05:00",
    "Level": "Information",
    "MessageTemplate": "Complete",
    "Properties": {
        "SendEmailFailure": 0,
        "SendEmailSuccess": 1,
        "ThreadId": 31,
        "MachineName": "Server1",
        "ServiceName": "AgileCraft_BackOffice_Server1",
        "Build": "10.73.0.5461",
        "Product": "AgileCraft.Backbone",
        "CorrelationId": 672329761,
        "Instance": "_datbaseName_"
    }
}

Primary Resource Constraint

The database server in our implementation is constrained by Memory and CPU. The memory constraint is best viewed by looking at the cache hit rate.

The CPU targets for SQL server are also a subject of great discussions. Most DBA teams within enterprises have a default setting for alerting. We trigger at 60% for 15 minutes as a medium and 80% for the same time as a high. Since our solution is in AWS we can either change machine type or reorganize the workload (in the multi-tenant environment)

Public Network

The Web site page fetch also checks to see if the public-facing load balancer is up. The return codes will help us understand which part of the connection might be broken.

Load Balancer Health is checked via AWS monitoring.

It's set to alert to our Opsgenie integration for:

• unhealthy host counts in cluster
• excessive HTTP 500's in a 10 minute window
• Large response times in a 10 minute window

The Web site page fetch, also checks to see if the public-facing load balancer is up. The return codes will help us understand which part of the connection might be broken.