Troubleshooting

Connection timeout, Error 404 Gateway Unreachable

Troubleshooting using "nc" command
Troubleshooting Steps
  1. Login to the appviewx pods CLI to check communication
    1. On Prem : Login in to the appviewx CLI, go to the vendor pods.
    2. Saas : Login to the Cloud Connector, go to the platform pod.
  2. After logging in, run the commands below in the terminal.
    nc -zv <Ip or hostname> <8999 or configured port>
    where,
    • <Ip or hostname> is the hostname or IP of the windows agent machine.
    • 8999 is the default port of Windows Agent (provide the configured port in case of any custom port).
Expected Output:
Connection to 192.168.142.124 8999 port [tcp/*] succeeded!
Troubleshooting using "curl" command
Troubleshooting Steps
  1. Login to the appviewx pods CLI to check communication
    1. On Prem : Login in to the appviewx CLI, go to the vendor pods.
    2. Saas : Login to the Cloud Connector, go to the platform pod.
  2. After logging in, run the commands below in the terminal.
    curl -kv https://<hostname or ip>:8999/appviewx/rest/Ping
    where,
    • <Ip or hostname> is the hostname or IP of the windows agent machine.
    • 8999 is the default port of Windows Agent (provide the configured port in case of any custom port).
Expected Output:
Trying 192.168.142.124:8999...
* TCP_NODELAY set
* Connected to 192.168.142.124 (192.168.142.124) port 8999 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
* CAfile: /etc/ssl/certs/ca-certificates.crt
  CApath: /etc/ssl/certs
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
* TLSv1.2 (IN), TLS handshake, Request CERT (13):
* TLSv1.2 (IN), TLS handshake, Server finished (14):
* TLSv1.2 (OUT), TLS handshake, Certificate (11):
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
* TLSv1.2 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.2 (OUT), TLS handshake, Finished (20):
* TLSv1.2 (IN), TLS handshake, Finished (20):
* SSL connection using TLSv1.2 / ECDHE-RSA-AES256-GCM-SHA384
* ALPN, server accepted to use h2
* Server certificate:
*  subject: CN=windows-gateway.appviewx.com
*  start date: May 10 00:00:00 2024 GMT
*  expire date: May  9 23:59:59 2029 GMT
*  issuer: CN=AppViewX Intermediate CA; O=AppViewX Inc; L=Seattle; ST=Washington; C=US
*  SSL certificate verify result: unable to get local issuer certificate (20), continuing anyway.
* Using HTTP2, server supports multi-use
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* Using Stream ID: 1 (easy handle 0x563e972e1650)
> GET /appviewx/rest/Ping HTTP/2
> Host: 192.168.142.124:8999
> user-agent: curl/7.68.0
> accept: */*
> 
* Connection state changed (MAX_CONCURRENT_STREAMS == 100)!
< HTTP/2 403 
< content-length: 0
< date: Thu, 15 Aug 2024 07:13:44 GMT
< 
* Connection #0 to host 192.168.142.124 left intact

Access to local host was denied

To resolve the Access Denied error during installation verification,
  • When choosing the client certificate for authentication, and attempting to reach the https://localhost:8999/appviewx/rest/help URL in the browser, import the client certificate in the browser's personal store and then try again.

Empty response received from windows gateway

Potential Causes The above error messages can occur when the certificate uploaded while adding the Windows Gateway is different from the Client Certificate in the Windows Gateway Installation Package or if the certificate has expired.
Fix Ensure the following:
  • The same certificate used during installation has been added while configuring the Windows Gateway.
  • The latest version of the Client Certificate issued from AppViewX is used with AppViewX Windows Gateway.
Steps To update certificate in AVX UI:
  1. Go to general settings under CLM.
  2. Select edit option for the corresponding gateway agent and update the latest certificate under windows gateway details.
  3. Click Add and then Save.

Server returned HTTP response code:403

Potential Causes The above error messages can occur when the certificate uploaded while adding the Windows Gateway is different from the Client Certificate in the Windows Gateway Installation Package or if the certificate has expired.
Fix Ensure the following:
  • The same certificate used during installation has been added while configuring the Windows Gateway.
  • The latest version of the Client Certificate issued from AppViewX is used with AppViewX Windows Gateway.
Steps To update certificate in AVX UI:
  1. Go to general settings under CLM.
  2. Select edit option for the corresponding gateway agent and update the latest certificate under windows gateway details.
  3. Click Add and then Save.

Network Path not Found” during Certificate Push/Bind

Potential Causes This error can occur while attempting to perform a Push or Bind operation to a target server.

Port 445 Blocked: One of the most common scenarios under which this scenario occurs is when the port 445 is blocked in the target machine. This port is used for transferring any file from the Windows Gateway machine to the target server using SMB Protocol.

Troubleshooting Steps To update certificate in AVX UI:
  1. Check the Firewall Settings in the Windows Gateway machine and the target machine and check if any firewall rule is blocking the port 445.
  2. Allow or open the TCP port 445.
  3. After allowing/opening the port, check the communication from the windows gateway machine to the target machine using Test-NetConnection command.
    Test-NetConnection -ComputerName <hostname> -Port <port>
    Sample expected result:
    ComputerName : computername
    RemoteAddress : 93.184.216.34
    RemotePort : 80
    InterfaceAlias : Ethernet
    SourceAddress : 192.168.1.10
    PingSucceeded : True
    TcpTestSucceeded : True
  4. Retry the push or bind operation from the AppViewX GUI or Gateway.

RPC Server is unavailable

Potential Causes This error can occur while attempting to onboard a CA or initiate a CA Discovery. Some of the most common cases under which this error can occur are as follows:

CA Unreachable from Windows Gateway Machine: If Certificate Authority (CA) discovery fails, it typically indicates that the agent or system cannot locate or communicate with the CA.

Verification To verify if the CA is reachable:
  1. Execute the below command on the windows agent machine. It lists all CAs in the domain.
    certutil -config - -ping
    Expected Output:
  2. Select the required CA to check the connectivity.
    Expected Output if CA is reachable from the Gateway Machine
    Expected Output if CA is not reachable from the Gateway Machine
Troubleshooting Steps Checking the Certificate Authority Service: If the RPC Server Unavailable error persists, try the steps below.
  1. Navigate to the machine where the CA is configured.
  2. Open Run (Win + R) and type certsrv.msc, then hit the Enter key.
  3. In the Certificate Authority window, right-click on the CA name.
  4. Select All Tasks and then click Start Service.
  5. Return to the agent machine and rerun the certutil command to ensure the CA is now reachable.
    certutil -config - -ping

Denied by policy module

Potential Causes This error can occur during Certificate Enrollment, if the template used during enrolment is not provided the required permissions for the requesting user or service account.
Troubleshooting Steps
  1. Access Certificate Templates: Go to the Certificate Authority (CA), right-click on the Certificate Templates folder and select Manage.
  2. Locate the Template: From the list of available templates, identify the specific template associated with the failed enrollment. Now, right-click on the template and select Properties.
  3. Verify Security Settings: Go to the Security tab, ensure that the user or group has the appropriate privileges for enrollment.
  4. Restart the CA Service: First stop the service and then restart the CA service to apply the changes.
  5. Retry Enrollment: After permissions have been updated, retry the certificate enrollment process.

Curl test from port 8999 Fails

Description The installation completed successfully, and the service is running; however, the cURL test on port 8999 is still failing, and the AppViewX Help site remains unreachable.
Potential Causes This problem occurs when certificate binding fails during installation.
Troubleshooting Steps To verify if the certificate installation was successful, please review the installation log file located in the directory where the installer executable file was extracted.

Denied by Policy Module error during Certificate Enroll

Description When enrolling a certificate, the Denied By Policy Module error occurs if the Certificate Authority (CA) rejects the request due to policy validation failures.
Potential Causes If the Build from Active Directory option is selected in the template, the CA attempts to verify the request's source. When using a service account, Active Directory (AD) tries to validate the associated properties, but certain required attributes may be missing. As a result, MSCA denies the request.
Troubleshooting Steps To resolve this issue:
  • It is recommended to select Subject Name = Supply in CSR instead of Build from Active Directory when configuring the certificate template.

    If the error occurs/persists despite selecting Supply in CSR, the certificate enrollment error response will include a Request Disposition Message and Request Status Code. Execute resolution steps according to the disposition message and status code displayed.

    Alternatively, you can check the disposition message and status code manually. To do this:
    1. In the Certification Authority Desktop app, for the the required CA instance, go to Failed Requests.
    2. For more details on what triggered the error, for the corresponding Request ID, read the Request Status Code and the Request Disposition Message.

      You can sort the Request ID column/view the Request Submission Date to identify the correct request ID.

    3. Execute resolution steps according to the status code and disposition message.
  • Ensure that the CSR parameters populated for certificate enrollment are in accordance with the template configuration requirement defined by Microsoft CA.
    To view the template configuration:
    1. In the Certification Authority Desktop app, for the the required CA instance, go to Failed Requests.
    2. For the corresponding Request ID, see the Certificate Template mentioned.
    3. For the the CA instance, right click Certificate Templates.
    4. From the menu displayed, select Manage.

      A list of certificate templates is displayed.

    5. From the list of templates, right click the certificate template used for this enrollment request and select Properties.
    6. In the properties dialog box, go through all the template tabs to view the requirement configured for each parameter of the template.

Network Name Not Found error during certificate push/bind

Description This error typically occurs while attempting to perform a push or bind operation to a target server when the Windows Gateway machine is unable to access the specified network path.
Potential Causes
  • Incorrect Network Path or Hostname: The provided hostname or shared path may be invalid or incorrectly formatted.
  • SMB Share Not Available: The required shared folder may not exist or is not accessible on the target server.
  • Permission Issues: Insufficient access rights to the shared resource
  • Network or Firewall Restrictions: Network policies may be preventing access to the target system or shared path.
Troubleshooting Steps
  1. Validate network path accessibility
    1. Log in to the Windows Gateway machine.
    2. Open the Run dialog box (Win + R).
    3. Enter the network path:
      \\<target-server>\<share-name>
    4. Check if the same error (Network Name Not Found) is reproduced.

      If yes, the issue is at the network/share level and not specific to AppViewX.

  2. Try using FQDN.
    Access the share using the Fully Qualified Domain Name:
    \\<server-name>.<domain-name>\<share-name>
  3. Verify the shared resources on the target server.
    1. On the target server, run:
      net share
    2. Ensure the required share exists and is correctly configured.
  4. Validate permissions.
    Ensure the user account used for the operation has:
    • Access to the shared folder
    • Required share and NTFS permissions
  5. Check connectivity from the gateway.
    1. Try accessing the share directly through the File Explorer:
      \\<target-server>\<share-name>
    2. Provide credentials if prompted and verify access.

Error 0x80090322 when using the PowerShell mode

Potential Causes This error typically occurs due to a Kerberos authentication failure, most commonly caused by an SPN (Service Principal Name) conflict where the default HTTP/<FQDN> SPN is registered to an incorrect account.
Troubleshooting Steps
  1. Verify the error.
    1. Run the following command from the Windows Gateway machine:
      Enter-PSSession -ComputerName <FQDN>
    2. Confirm the error: WinRM cannot process the request (0x80090322)
  2. Check and fix SPN configuration.
    1. Run:
      setspn -q http/<FQDN>
    2. Ensure the SPN is associated with the target server’s computer account and not a different service account.
    3. If incorrect, update or remove the conflicting SPN entry.
Reference

Authentication failure during PowerShell remoting (WinRM)

Description This issue can occur during PowerShell-based push or bind operations when the target system is unable to correctly resolve or authenticate the provided user credentials.
Potential Causes
  • Username Format Issue: Using username or domain\username may not always be resolved correctly in certain environments.
  • Authentication Context Mismatch: The domain context of the user is not properly identified during authentication.
  • Logon Permission Restrictions: The account may not have the required access on the target machine.
Troubleshooting Steps
  1. Verify the issue.
    Attempt a remote session from the Windows Gateway:
    Enter-PSSession -ComputerName <FQDN> -Credential <username>
    The connection may fail due to authentication or logon-related errors.
  2. Use UPN format for authentication.
    • Provide the service account in the User Principal Name (UPN) format:
      username@domain
    • In some environments:
      • username → may fail
      • domain\username → may fail
      • username@domain → succeeds
    • Using UPN format ensures that the correct domain context is applied during authentication and helps avoid ambiguity in user resolution.

Authentication failure while using custom certificates

Description This issue can occur when using custom certificates for authentication, typically due to problems in certificate validation or chain building.
Potential Causes
  • Missing Authority Key Identifier (AKI): The leaf certificate may not include the AKI field.
  • Incomplete Chain Information: The gateway relies on both Authority Key Identifier (AKI) and Subject Key Identifier (SKI) to build and validate the certificate chain.
  • Certificate Chain Validation Failure: The issuing CA cannot be correctly identified.
  • Expired Certificate: The certificate used for authentication is no longer valid.
  • Invalid EKU (Enhanced Key Usage): Required usages such as Server Authentication and/or Client Authentication are not present.
Troubleshooting Steps
  1. Verify the certificate.
    Inspect the certificate and ensure:
    • Authority Key Identifier (AKI) is present
    • Subject Key Identifier (SKI) is present
    • Certificate is not expired
    • EKU includes:
      • Server Authentication, or
      • Client Authentication, or
      • both (based on use case)
  2. Request updated certificate.
    Check with the customer/CA team to re-issue the certificate with:
    • Both AKI and SKI extensions
    • Valid expiry period
    • Required EKU values
  3. (Workaround) Enable thumbprint-only validation
    To unblock the operation, configure the Windows Gateway to validate only the certificate thumbprint:
    1. Stop the AppViewX CLM service.
    2. Open Notepad as Administrator and open AppViewX.CLM.WindowsService.exe.config.

0x8009030e – Kerberos double-hop authentication failure during PowerShell remoting in Exchange Server

Description This issue occurs when performing PowerShell remoting operations that involve multiple authentication hops using Kerberos.
Potential Causes
  • Kerberos Double-Hop Limitation: By design, Kerberos does not allow credentials to be forwarded from one remote system to another (second hop).
  • Delegation Not Configured: The intermediate server is not trusted to delegate user credentials to downstream systems.
  • Missing or Incorrect SPNs: Required Service Principal Names (SPNs) for the target services are not configured.
  • WinRM Not Properly Configured: PowerShell remoting may not be enabled or properly configured for Kerberos authentication.
Troubleshooting Steps
  1. Verify the issue.
    1. From the Gateway server (ServerA), initiate a remote session:
      Enter-PSSession -ComputerName ServerB -Authentication Kerberos -Credential Domain\Username
    2. Execute operations that require access to downstream services (for example, Exchange/Active Directory).

      The operation may fail with errors such as: 0x8009030e (A specified logon session does not exist).

      This indicates a Kerberos double-hop limitation.
  2. Enable WinRM for PowerShell Remoting.
    1. On both Gateway (ServerA) and Exchange server (ServerB), run:
      Enable-PSRemoting -Force
      Ensure:
      • WinRM service is running.
      • Firewall rules allow WinRM traffic.
  3. Validate and configure SPNs.
    1. On the Domain Controller (ServerC), verify the SPNs:
      setspn -L ServerC
    2. Ensure the following SPNs exist:
      • HOST/ServerC
      • WSMAN/ServerC
    3. If the HOST/ServerC SPN is missing, execute the following command:
      setspn -S HOST/ServerC ServerC
      If the WSMAN/ServerC SPN is missing, execute the following command:
      setspn -S WSMAN/ServerC ServerC
  4. Configure Kerberos constrained delegation.
    1. On the Domain Controller (or AD management system):
      1. Open Active Directory Users and Computers.
      2. Navigate to the ServerBPropertiesDelegation tab.
      3. Select:
        • Trust this computer for delegation to specified services only
        • Use Kerberos only
      4. Click Add and select ServerC.
      5. Add the following services:
        • HOST/ServerC
        • WSMAN/ServerC
    2. Optionally, verify the delegation using PowerShell:
      Set-ADComputer "ServerB" -TrustedForDelegation $true
      (Get-ADComputer "ServerB" -Properties TrustedForDelegation).TrustedForDelegation
      
  5. Clear the Kerberos tickets.
    On the Gateway server (ServerA), run:
    klist purge
    Alternatively, restart the involved servers to refresh the Kerberos tickets.
  6. Test the Kerberos Double-Hop.
    1. From the gateway server, execute the following command:
      Enter-PSSession -ComputerName ServerB -Authentication Kerberos -Credential Domain\Username
    2. From within the session, run the following command:
      Add-PSSnapin Microsoft.Exchange.Management.PowerShell.SnapIn
      Get-ExchangeServer | ForEach-Object { $_.Name + ' -> ' + $_.AdminDisplayVersion }
      
      Successful execution of the command confirms that the double-hop authentication is working.