Diagnose ActivID AS

When you need to troubleshoot the system, ActivID AS can create log files on the ActivID AS server which can be used to troubleshoot the issue. You can generate a package of these logs files for analysis by technical support.

The ActivID AS diagnostic package contains the following:

• ActivID AS application configuration files (every obfuscated password is deleted)
• ActivID RADIUS Front End configuration information (such as dictionary, sites etc.)
• System activity information (if available)
• ActivID AS native libraries linker information
• File list under ACTIVID_HOME directories
• Memory usage
• Active TCP and UDP connections
• Routing table
• List of the network interfaces
• List of the running processes with their CPU usage, resident memory and virtual memory

About the Anonymize Substitution File

During troubleshooting, in rare cases, HID Global personnel might need to access the original PII (for example, an issue might occur due to the content of the LDAP username). To facilitate this, ActivID AS also generates a substitution file that can be used to retrieve the exact original value.

For example, HID Global personnel need to alert the customer that they are seeing some specific issues regarding user Name1, but only the customer knows the identity of user Name1. Similarly, a customer might ask HID Global personnel to pay special attention to user Name2. HID Global does not need to know who that is.

• The substitution file that the customer keeps, helps translate from the anonymized information to the PII and vice versa. If an actual PII needs to be provided to HID Global personnel, only that specific information is shared, and the other PII can be kept private by the customer.
• The substitution file contains the list of PII with all the corresponding replacement values separated by a colon (:).
• If there is no replacement value, then the PII is marked as <PII tag>_REMOVED_<PII tag>.
• The substitution file is the same for all files processed.

Problem Diagnosis Summary

1. Check the following potential deployment-specific causes of failure:
   • Make sure custom interface code is error-free
   • Check for invalid or incomplete configuration
   • Make sure the user has correct privileges
   • Check for invalid or incomplete entries in property files
   • Check for invalid or incomplete set up of the HSM(s)
2. Check the following potential infrastructure causes of failure:
   • Check for failure or incorrect configuration of the load balancing layer
   • Check network connectivity from the channel system to the ActivID AS server
   • Check connectivity from the application server(s) to the database
   • Check for HSM failure (where the HSM is shared across multiple application servers)
   • Check for database server failure

Problem Diagnosis − Troubleshooting Workflow

In the event that one of the components (database, HSM, application server, or application server) has failed, and the problem can be corrected by restarting that component, it is important to establish why the failure occurred. Review the information to help diagnose problems and make decisions.

Generate an Application Diagnostic Package

To generate a diagnostic package of the ActivID AS applications installed locally, as ftadmin , run the following script:

<ACTIVID_HOME>/ActivID_AS/bin/createDiagnosticPackage.sh

The anonymized diagnostic package (and corresponding substitution file) is generated in the /tmp directory.

The file name of the anonymized diagnostic package is Diag_<hostname>_<date>-<time>_UTC.tar.gz (where the time is in UTC).

The substitution file name matches that of the diagnostic package with the suffix _Substitution_Table (for example, Diag_<hostname>_<date>-<time>_UTC_Substitution_Table.txt).

The log files are generated with the configured logging level.

Generate an ActivID AS RADIUS FE Diagnostic Package

To generate a diagnostic package for the RADIUS Front End, as root, run the following script:

<ACTIVID_HOME>/ActivID_AS/bin/createDiagnosticPackage.sh

In addition to the FreeRADIUS log files, the RADIUS\raddb directory in the package also contains information about the RFE configuration (such as dictionary, sites etc.).

Level-One Diagnosis

With a level-one diagnosis you are trying to validate that:

• The system is fully installed. There might be issues with a system installation that do not show up in a level-one diagnosis.
• All components are up and running. In the event of a component failure:
  • Which component failed?
  • Does the system resume correct operation once that component has been restarted or fixed?

Failure Analysis Chart

Level One: ActivID Management Console Logon Test

This is a simple connectivity test using the ActivID Management Console to validate that all components are up and running.

1. Log on to the ActivID Management Console as a user with basic user-management privileges.
2. Search for users.
   • When you search for a user, the ActivID Management Console searches pages and makes requests (for example, searchUsers and getUser).
   • ActivID AS processes these requests and searches the database to retrieve a list of users based on your search.
   • The HSM verifies the user password and validates the signatures on records returned from the database.

The test confirms that there is at least one application server up and running, successfully connecting to an HSM, and that the database also is up and running.

• When the ActivID Management Console logon page is displayed, this verifies that the application is up and running.
• Logon success verifies the Application Server is up and running.
• Logon success verifies the database is up and running.
• Logon success verifies the HSM is connected and running.
• Even a response of “incorrect password” proves that the database and HSM are operating correctly. Failure of either of these components will result in an error page.

If the logon test fails, proceed to Level One: Application Server Check.

Level One: Application Server Check

If the ActivID Management Console Logon test fails, check that the application server is running and that the ActivID AS applications are correctly deployed and started.

For example, log on to the application server administration console to view applications that have been deployed. However, provided that the load balancing layer is working correctly, only one application server should be required to be operational in order to enable an ActivID Management Console logon.

1. Log on to the administration console for your application server.
2. Confirm that the following applications are deployed:
   • ActivID AS Authentication Services
   • ActivID Self-Service Portal
   • ActivID Management Console

If the problem is not with the applications deployed, proceed to Level One: Database Connectivity Test.

Level One: Database Connectivity Test

To test database connectivity, use the administration console of the application server. You should be able to test a JDBC connection from the console. This checks connectivity to the database directly from the application server (that is, without using ActivID AS). It validates that the database is up and running, and also that the data source configured in the application server can connect to the database using the configured user name and password.

If you have multiple security domains deployed, there will be a separate data source for each one. Test each of the data sources to ensure that each one is up and running. If one data source connection to the database is down, it should impact only the domain that use that data source. Other domains should be unaffected by the outage.

There are a number of potential causes for an unsuccessful connectivity test from a database:

• The database server is not running.
• There is a loss of network connectivity between application server and database server.
• You entered an invalid user name, password, or both for the configured data source.

The database connectivity test does not validate that the ActivID AS schema has been successfully created and populated. If the connectivity test is successful, but you have never successfully logged into this domain, it is worth confirming that the schema for this domain has been successfully created. A good way to check this is to run the SQL query – select * from ftresslogin;– under the user associated with the domain schema. This query should not fail and return a few records.

Level One: Confirm the HSM is Operational

For specific procedures on how to check if your HSM is working using the diagnostic utilities, refer to your HSM documentation.

Level-Two Diagnosis

Without needing to escalate to level two, there are a number of scenarios for which first-line (level one) support should be able to resolve a problem that seems to have reached level-two diagnosis.

Implementing level two diagnosis – calling HID Global Technical Support – should be necessary only when you have investigated the following scenarios first:

• Incorrect installation – the application was not installed correctly. Resolve this by modifying the properties file or create a missing key.
• Incorrect configuration – the application was not configured correctly. Resolve this by changing configuration using the ActivID Management Console.
• Is someone trying to use the system incorrectly?

If the problem does not fall into one of the above categories, the issue might relate to one of the following areas:

• Interface code (deployment-specific issues).
• Core ActivID AS functional logic, including problems with the ActivID Management Console or storage of data in the database.
• Issues with the SOAP interface layer.
• Technical issues with the code, such as ActivID AS not correctly managing connections to HSM, or memory leakage.

If your problem falls into one of these categories, then it requires escalation, and you should call HID Global Technical Support. First review Information to Clarify Before Level Two Diagnosis.

Information to Clarify Before Level Two Diagnosis

When does the problem occur?

• Calling ActivID AS from a remote system through the public API
• Using the ActivID Management Console

Is it a functional problem?

• Invalid authentication results
• User/credential or other administration issues
• Incorrect entries in the audit log

Is it an integration issue?

• Call to the API fails for unknown reason − always
• Call to the API fails for unknown reason – periodically

Is it a technical issue?

• Complete loss of service (unable to complete the ActivID Management Console logon test)
• ActivID Management Console logon test successful, but public API is unavailable
• Service is irregular or uncharacteristically slow