The UAG That Wouldn't Stay Diagnosed | When Vendors Don't Have the Answer
The UAG That Wouldn't Stay Diagnosed
A power cycle fixed it. That's not the same thing as knowing why it broke — and the gap between those two facts is the actual lesson.
Vendor documentation is good at describing the happy path. It's much thinner on what happens when every component in a Horizon access chain gets rebooted in the same maintenance window and one of them comes back slightly wrong — wrong enough to fail intermittently, not wrong enough to fail obviously. This is one of those cases.
The environment
The normal login sequence is straightforward:
The prior week's maintenance window included a full reboot of every server in this chain — both UAGs, both RADIUS servers, both Connection Servers. Nothing was reconfigured; it was a routine patch-and-reboot cycle.
The symptom
After the reboot, users began reporting a specific failure pattern: authentication would complete successfully — banner acknowledged, OTP accepted, AD credentials accepted — and then, at the point where the desktop session should load, the client would drop back to the disclaimer screen instead of an error message. Retrying sometimes reproduced the same loop several times before succeeding, and sometimes worked cleanly on the first attempt. No component was down. Nothing in a surface-level health check looked wrong.
An early working theory was that one specific UAG node was responsible — the failure seemed to correlate with sessions landing on it.
That correlation was observed through the load-balanced VIP, not by connecting to each UAG directly. That distinction matters more than it looks like it should. The Horizon web login sequence is a multi-step transaction — disclaimer, OTP, AD auth, session broker call — and that transaction depends on session continuity to one UAG instance throughout. If load balancer persistence (sticky sessions, whether IP-based or cookie-based) isn't holding correctly, a client can get bounced between UAG nodes mid-sequence. The session context breaks, and the client falls back to the disclaimer screen — which looks exactly like "this one UAG is broken," when the actual defect is upstream, in how the LB is distributing that specific client's requests.
In other words: before treating "it's this UAG" as established, the right test is to bypass the load balancer entirely and hit the suspect UAG's public address directly, repeatedly, and see if the failure rate holds. Skipping that step means a fix later on can't be distinguished from a coincidence.
What else was on the list
Given the failure point — right before desktop session establishment, right after a full-stack reboot — three other candidates ranked above "the UAG is just faulty":
- NTP drift. A freshly rebooted appliance that hasn't fully re-synced time can fail token or session validation intermittently, in a way that looks random and resolves itself once sync catches up.
- Stale certificate thumbprint. If a Connection Server's certificate was regenerated or renewed as part of the maintenance, and a UAG's cached thumbprint for that CS wasn't updated, backend handshakes through that specific UAG–CS pairing can fail while the same request through a different UAG (paired differently, or with a valid thumbprint) succeeds.
- Blast/PCoIP External URL misconfiguration. If the tunnel endpoint a UAG advertises to the client is unreachable, some client versions don't surface a clean error — they silently re-render the portal's starting screen, which is indistinguishable from "the session just reset."
None of these are things a health-check dashboard catches, because the UAG and the Connection Server are both individually reporting healthy — the defect lives in the relationship between them, or in the load balancer's handling of a multi-step session, not in any single component's uptime.
On packet capture
The instinct to reach for tcpdump on the UAG appliance is reasonable, but worth qualifying. UAG is a locked-down Photon OS appliance — SSH has to be explicitly enabled through the admin console first, and re-disabled afterward. More importantly: the failure described here happens before the display-protocol tunnel is established, so filtering for the protocol port often catches nothing. The traffic worth capturing is the control-plane traffic on 443, between client and UAG and between UAG and Connection Server. Even then, that traffic is TLS-encrypted — a capture shows connection resets, timeouts, and retransmits, not which authentication step failed or why. It's a tool for confirming a network-layer symptom exists, not for identifying root cause on its own. It has to be read alongside UAG log bundles and the Connection Server's event log, correlated by timestamp.
How it actually ended
The suspect UAG was powered off and back on. The symptom stopped.
That resolves the incident. It does not resolve the diagnosis. A power cycle clears in-memory session state, cached connections, and any transient handshake condition — which means it's consistent with nearly every candidate on the list above, and confirms none of them specifically. If the actual cause was a certificate thumbprint that only gets refreshed on service restart, this will recur at the next full-stack maintenance. If it was NTP drift during a slow post-boot sync, it may never recur under the same conditions. There's no way to tell which, after the fact, without the log correlation that wasn't captured before the fix was applied.
This is a fair trade under production pressure — a working VDI access path for a full user base outweighs a clean root-cause writeup in the moment. But it's worth naming plainly rather than writing it up as a solved case, because the next engineer who hits this same symptom shouldn't inherit false confidence that "reboot the UAG" is a diagnosis rather than a mitigation.
What to check first, next time
- Confirm the load balancer's session persistence configuration before attributing a multi-step auth failure to a specific backend node.
- Test suspect UAGs directly, bypassing the LB VIP, before trusting an observed correlation.
- After any full-stack reboot, check NTP sync status on every UAG and Connection Server before the first user login, not after a ticket comes in.
- Verify UAG-to-Connection-Server certificate thumbprints whenever a Connection Server certificate is touched during maintenance — even if the maintenance ticket didn't mention certificates.
- Pull UAG log bundles and Connection Server event logs, aligned by timestamp, before applying a fix that clears state — the fix will destroy the evidence.
Comments
Post a Comment