Category Archives: Uncategorized

Edge Server: RTCMEDIARELAY Active Sessions high

23 Thursday Feb 2017

Posted by in Uncategorized

≈ Leave a comment

Tags

Issue

The issue manifested itself to the agents in the RGS queues. Agents were reporting that the calls were taking long to answer, and as time went by, this issue degraded to the point that they were unable to answer calls.

Later on I also noticed:

  • degradation of meeting audio and video
  • If nothing is done the Edge Server will eventually blue screen

Environment was Lync 2013, Enterprise Front End Pool – 3 Front Ends. Single Edge Server.Logs not reporting any unusual behavior.

Troubleshooting

Restarting Response Group Service made no difference. A peek at the Edge Server revealed that the RTCMEDIARELAY  Service was reporting a high number of Active Sessions.

rtcmediarelay

Monitoring this service, I noticed that the session count continued to rise over time. What was even more puzzling is that the count rose even though the user count was very low comparatively.

Using netstat I took a peek at active sessions but this was normal, no sign of the sessions reported by the RTCMEDIARELAY Service or of sessions not being closed.

Perhaps a quick restart of the services would free up the sessions I thought, but I couldn’t stop the services either (stop-CsWindowsService). Probably because of the restart service wanting to do this gracefully. Right, Edge server reboot it is.

The Edge server returned after a reboot and all appeared well, until a few days later when the RTCMEDIARELAY session count started to grow again.

Since I was tasked with a new pool as a side-by-side migration to Skype for Business, I decided that its not worth spending any more time chasing this issue as it will be decommissioned soon.

To my surprise, the new Edge Pool with 2 x Edge Servers on Skype for Business displayed the very same behavior. Only have about 10test users on the SFB pool.

Since this is now recurring on a brand new Topology, new Server OS and different product version (too much of a coincidence for my liking), I asked “So what’s the common factor?”

The virtual layer.

A quick call to the infrastructure engineer confirmed that VMWare was the underlying technology and “hasn’t changed since Lync 2013..”

I had since noticed that if the Edge Server with the hogh session count was not assisted then the machine would eventually blue screen. Its the dump data that revealed a “known” issue in VMWare.

The related article titled “Windows virtual machines using the vShield Endpoint TDI Manager or NSX Network Introspection Driver (vnetflt.sys) driver fails with a blue diagnostic screen (2121307) “can be found HERE.

Solution

By simply uninstalling the “NSX File Introspection Driver” in the VMWare Tools Setup on the Edge Servers, I was able to remove the faulting culprit.

vmware

Of course, one could disable the vShield Endpoint TDI Manager by means of the regedit modification mentioned in the article as well.

Rebooted the server and tested, all good. Monitored over a week to confirm.

Sorted.

 

Advertisement

Error returned while installing OcsMcu.msi, code 1603

19 Saturday Nov 2016

Posted by in Error Codes, Uncategorized

≈ 9 Comments

Tags

Just recently we have started seeing the Skype for Business installer option to “Connect to internet to check for update” actually find updates. Yay!

Hold on now, as this is where the issue begins..

On a new install (Enterprise Edition Pool) this week I though I might give it a bash and see how we go.

Seems we have found some updates, unfortunately there is no indication at this point as to which updates these are.

downloading-updates

Next I ran through STEP 1 and 2

Step 2 Fails.

ocsmcu-msi-error-1603

OK, so the log file alludes to the Apply Patch and it mentions OcsMcu.msi. I am beginning to suspect pre-install update wizard already.

Checking the log file we find the following errors:

Error 1

Product: Skype for Business Server 2015, Conferencing Server – Update ‘Skype for Business Server 2015, Conferencing Server (KB3149233)’ could not be installed. Error code 1603.

Error 2

Windows Installer installed an update. Product Name: Skype for Business Server 2015, Conferencing Server. Product Version: 6.0.9319.259. Product Language: 1033. Manufacturer: Microsoft Corporation. Update Name: Skype for Business Server 2015, Conferencing Server (KB3149233).

Installation success or error status: 1603.

So from these errors we can see that the Conferencing Server (OcsMcu.msi) Installed Version is expected to be from KB3149233. The second error calls out the version of OcsMcu.msi as 6.0.9319.259

Lets do a quick check to see what is actually installed. There are a few ways to do this. Great post by Tim Harrington at http://howdouc.blogspot.co.nz/2012/04/three-ways-to-determine-what-version-of.html on how to do this.

Since (at the time of writing) the latest KB (November 2016) is out and is newer than whats being reported here, I will use SkypeServerUpdateInstaller.exe as it seems I will need to run updates.

Here’s what’s been found:

skypeserverupdateinstaller

From this quick view it appears that some components have been updated BUT as you can see, the conferencing server update (OcsMcu) is on a much older version. The error log was expecting to find 6.0.9319.259 (July 2016)  but what we have is even older than that. That’s actually the original version on the ISO so it is safe to say that the Conferencing Server was not patched at all.

As a quick fix, thought I might try a sneaky update and hope that the aged OcsMcu would get patched.

Unfortunately the expected version is now mismatched and so the update attempt also throws an error:

ERROR 1603: OCSMCU.msp had errors installing.

Solution

In this case, since we are in the install process, we opted for uninstall and re-install. This time however we did NOT select the option to “Check for Updates” before running the install.

Note

 

How to federate Onprem with sub-domain\other domain hosted online

17 Monday Oct 2016

Posted by in Uncategorized

≈ Leave a comment

Office 365 Tenants with multiple Domains have become fairly common place. Be it the international organization with a primary domain and a number of child domains, or the university with a faculty and student domain. Perhaps it’s the merger of two or more organizations, where some domains are Hybrid and others will remain in the cloud (for now).

scenario

This is easily addressed with a Hybrid deployment; I hear you say. True BUT..

In some cases, the requirement is to have some of these domains as cloud ONLY.

What reasons might an organization have to keep one domain on premises and another in the cloud only?

Two I have come across are: –

User Accounts

In one instance I saw an environment where university students (60 0000 +) were on line only, while faculty were Onprem. The university was not keen to add 60 0000 + users account to Active Directory so that we could enable hybrid mode (especially since the students were never going to be moved to Onprem)

In a Hybrid scenario, all the users (regardless of whether they will be homed in the Cloud or Onprem) need to exist in Active Directory. When you have two organizations merging then this can be a big limiting factor. Sure you can setup trusts and use resource accounts etc., limiting and intensive non the less.

DNS

When enabling Hybrid mode its required to point the DNS records to the On premise infrastructure. Consider a deployment where the Tennant is in the USA and the On premise deployment is in New Zealand. Potentially the on Premise deployment may be expanded to other continents, BUT I find it rather challenging to explain to a customer that I need to point the DNS records to an NZ deployment that’s probably not considered “Enterprise” in terms of High Availability.

 

For the sake of understanding let’s talk about SIP Domains.

We will consider three SIP Domain configurations: –

Onprem SIP Domain – All users homed on premise, domain does NOT exist in the O365 Tenant

Hybrid SIP Domain – All user accounts in on premise AD, Skype users homed to either On premise or online

Online SIP Domain – Users DO NOT exist in On premise AD. SIP Domain is created in O365 Tenant and there is no requirement to have these users on premise

So to be clear, what we will be looking at is “Federation” between an Onprem or Hybrid domain with an Online Domain.

Here’s what you need to do:-

Step 1

Configure Hybrid mode for the On premise SIP Domain (if it’s not already done). Details of configuring Hybrid can be found HERE

NOTE

You do not need to add the child\additional domains to the on premise topology (there is no harm in doing so either)

Step 2

Configure the DNS

A rule of thumb is to point the On premise\Hybrid SIP domain DNS records to the On premise deployment, and the Cloud SIP Domain DNS records to the O365 cloud.

TIP

Be sure to add the “Cloud SIP Domain” to the internal DNS Zones as any users considered internal will need to resolve the SRV records to the Online servers.

SIP Domain Record Type DNS Record Port Target
Onprem\Hybrid SRV _sipfederationtls._tcp.ucsorted.com 5061 access.ucsorted.com
SRV _sip._tls.ucsorted.com 443 access.ucsorted.com
A access.ucsorted.com Onprem Access Edge Public IP
A\CNAME lyncdiscover.ucsorted.com Onprem Reverse Proxy Public IP
A meet.ucsorted.com Onprem Reverse Proxy Public IP
A dialin.ucsorted.com Onprem Reverse Proxy Public IP
A sfbweb.ucsorted.com Onprem Reverse Proxy Public IP
Cloud SRV _sipfederationtls._tcp.uc-heroes.co.nz 5061 Sipfed.online.lync.com
SRV _sip._tls.uc-heroes.co.nz 443 sipdir.online.lync.com
CNAME sip.uc-heroes.co.nz sipdir.online.lync.com
CNAME lyncdiscover.uc-heroes.co.nz Webdir.online.lync.com

 

If you have the DNS for the Online domain pointing to Onprem you will get a user not found error when trying to message an Online user.

Conversely, if you have the DNS for the Onprem domain pointing to Online you will get a user not found error when trying to message an Onprem user.

error

Step 3

Certificates

You will need to add a sip.domain.com SAN to the Edge server certificate for each online SIP domain that needs to be included in the “federation”.

In my case I had to add sip.uc-heroes.co.nz as a SAN to the existing UC SAN Certificate already deployed.

 

Under the hood

O365 will actually handle the on premise SIP domain as if it’s in a fully deployed hybrid deployment.

The on premise deployment will handle the Online SIP domain as if it’s a federated partner. Thus the usual behavior for federation will be followed.

 

Limitation

The primary limitation is with regard to Federation behavior between these On premise\Hybrid SIP Domains and the Online SIP Domains.

Typically, you won’t see rich presence, Presence Notes, detailed contact card etc for Federated contacts. These same limitations will apply in this setup.

The inverse in not true though, the Online SIP Domain users will see the Onprem\Hybrid users as if they were fully Hybrid.

Also, if you are “federating” between Onprem\Hybrid and Online as I have just described, then the Onprem deployment is authoritive for the Onprem\hybrid users and the online deployment is authoritive for the online users.

Note

This is not an officially supported configuration.