Category Archives: Event ID

Skype for Business Trusted Root Certificate Authorities strike again

15 Monday Jul 2019

Posted by in Event ID, MCU, RGS, Skype for Business

≈ 1 Comment

Tags

, , , ,

Issue

This issue was flagged as a Contact Center that’s unable to receive calls. Shortly after it was noted that users we where unable to join or create meetings using the “Meet Now” option within the SFB client and that the RGS service was failing to start.

Environment is Skype for Business 2015, Standard Edition.

Troubleshooting

All services were running with the exception of the Response group service. A quick dive into the event logs on the Front End Pool showed a number of errors as follows:-

Event ID 31147

Event ID 31147 LS Response Group Service – Cannot update active Match Making Server because SQL Server does not respond. Standard error for issues with connectivity to SQL.

That explains why the Response Group Service wont start. What else we got here?

Event ID 32178

Event ID 32178 LS User Service – Failed to Sync Data for Routing Group from backup store.

Cause indicates an issue with connectivity to the back-end database. Another SQL connectivity issue by the looks of it. Digging further, I also found a few other MCU errors like:

Event ID 61037

Event ID 61037 LS MCU Infrastructure – The Audio-Video Conferencing Server failed to send health notifications to the MCU factory.

And also:Event ID 61043

Event ID 61043 LS MCU Infrastructure – The IM Conferencing Server failed to send health notifications to the MCU factory.

Almost missed this sneaky Information message (should be Error right..?).

Event ID 61029

Event ID 61029 LS MCU Infrastructure – stating that the certificate applied to the front end pool was somehow invalid.

OK so here is what I have learnt so far:

  • Response Group Service has an issue connecting the SQL back-end (even though the back-end is local in Standard Edition)
  • MCU cant connect to the back-end either and its complaining about an invalid certificate

Resolution

My money is on certificates even though technically speaking the event log entry referring to this was informational. A quick look at the certificates proves that they all look fine and no expiring certs hanging around.

Double checked the trusted root CA and that too looks fine.

However..

When checking the Trusted Root Certificates Authorities store I spotted something that wasn’t right. Usually the certificates in this store have identical Issued To and Issued By entries.

Anything that deviates from that, as far as I have experienced, generally prevented the RTCSRV service from starting. Not the case this time around.

Could this be the reason why I saw the Invalid Certificate warning before? Asked the on-site engineer what the deal was with these certs. Turns out they had been added by an engineer for some firewall trusts he was mucking around with.

Removed these dodgy certs, gave the RGS Service another start and boom, running! A round of tests to confirm and we are cooking with gas.

Folks! Seriously, there is no need to drop certs from the SAME CA as your server on the local store. They are from the same CA..

Lesson for today, if you don’t know what you are doing with certificates be warned. You could accidentally take out a Contact Center.

 

 

Advertisement

Proxy side ICE connectivity check failed

22 Saturday Dec 2018

Posted by in Edge Server, Event ID 14402, Skype for Business, Troubleshooting

≈ 2 Comments

Tags

, ,

Issue

Initial complaint was that all users homed to a specific pool could not reach voicemail, neither could any callers leave messages.

Environment

Lync 2010 with an Enterprise Pool (co-located Mediation servers), multiple SBA’s and SBC’s as well as an Edge Pool. Voicemail in O365. To make things interesting, each pool has its own UM Policy.

Replicated the issue by dialing the subscriber access number in O365. Basically, the call is established, I hear silence for 10 seconds and the call is dropped.

Troubleshooting

Investigating I found that any users homed to the Enterprise Pool were experiencing the issue. Looking at the monitoring reports, diagnostic report shows the diagnostic ID 22 with a reason “Call failed to establish due to a media connectivity failure when both endpoints are internal”. Now that sounds familiar..

Diagnostic ID 22In the past I have seen these sorts of issues and found that they usually occur when there is a routing issue involving the Edge Pools and thus a breakdown in the candidate negotiation process. However, this is the first time I have seen this sort of issue when dialing a PSTN number.

Move the user to another pool and the issue disappears, right so pool specific.

Next stop, Edge server to see what its thoughts are on this call failure. Looking at the Event logs I find the error Event ID 14402 which corelates to the attempts to O365 Voicemail.

Event ID 14402The non-internal servers that are named in the error is the on-prem UM server. “Add it to the list of internal servers on the Access Edge Server”, well I don’t think I have seen that list since the OCS days. There must be a list of sorts but no bells ringing just yet.

Running snooper trace I got the same Diagnostic ID and reason as we saw earlier in the diagnostic report. But wait, there is also another clue. The exception thrown is “Proxy side ICE connectivity check failed”, and the component is “mediation server”.

both endpoints internalLooking at the Lync Workloads Protocal poster, this ICE traffic is on the internal interface. The Lync Workloads Protocal poster also confirms that the components affected are Mediation, UM and Enterprise Pool. So in summary, a specific Pool is unable to complete an ICE check with the Edge Pool. Testing from these servers I can confirm that this traffic is allowed (simply 443 TCP and 3478 UDP).

Wonder if anyone else can shed light on this, off to google. Not a single hit with the same diagnostic error or call scenario. Not entirely surprised, also I suspect that this is not the only symptom of the underlying issue. My google search did however find a possibility that the Mediation service and Edge service are not associated correctley (thanks Mattias)

And there it is..or in the case of the EdgeServer entry for the pool in question, there it is not.

Mediation Association

Solution

Add the association between the Mediation Server and the Edge Server, of course. Test and sorted. Found the list 😉

Skype Federation Search Stops Working – Event ID 62044

31 Monday Jul 2017

Posted by in Edge Server, Event ID, Event ID 62044, Federation, Skype Search

≈ 2 Comments

Tags

, ,

Issue

The Skype Federation search stops working after a DMZ move of the Edge server. IP addresses (both internal and external) have changed due to moving to new DMZ & internet connectivity. The changes were published to the Topology and everything else appears to be working fine.

1

Investigating

The Front End shows the following Event ID 62044 error. Of course this Event ID eludes to the underlying cause, network or configuration. Both have already been double checked, moving on to collecting more details.

2

No evidence on the Edge Servers logs at all.

Resolution

Right, so to recap, the Front End believes its network or configuration, I have confirmed that its neither. Mm..When in doubt, reboot.. right? Well in this case it was turn off- turn on.

Disable Skype federation on edge server in topology, publish. Run Setup on edge server.

Re-enable Skype federation on edge server in topology, publish. Run Setup on edge server.

Edge server logs this:

3

A quick peek at the Front End now shows that the Skype Proxy configuration is us and running

4

and confirmation the issue has now been resolved5

Skype search back in business!

Didn’t work for you?

Another common issue is that the Edge Server is unable to reach the Skype Consumer Directory service. This service is on skypegraph.skype.com port 443. If you cant reach this service on 443 you may have to have a chat to the network team.