O365 EOP Phishing Rule

I had a client last week that was constantly complaining about receiving a lot of phishing emails. I had done everything I could think of in order to try and mitigate them in Exchange Online Protection (EOP) and finally broke down and opened up a support case. Upon submitting the numerous examples to the support engineer he guided me through implementing an undocumented change to the phishing threshold via a header manipulation.

We created a mail flow rule that stated if the sender is outside of the organization and the recipient is the client's domain, then add a message header "MS-Exchange-Organization-PhishThresholdLevel" and set the value to 2 (the default is 4):

Ever since implementing this the client has said they are no longer receiving any phishing emails.

WannaCrypt - Missing KB's?

After this weeks WannaCrypt fiasco I found a couple posted scripts for scanning a domain to check if servers had the necessary patches installed to mitigate against the vulnerability. After patching a couple of systems that the scripts said were vulnerable I found that they kept reporting that the patches were missing. Upon reviewing Get-Hotfix and doing a stare and compare at what was showing as having been installed via the "View Update History" I noticed that there were missing KB's.

The output from Get-Hotfix, you can see there are only 6 KB’s that start with KB40…

If I look at the Update History on the server I see there are some missing ones:

After a bit of research I found that this post on TechNet from Mervyn Zhang:

Windows Update and Office update are separated in two catalogs. Get-hotfix which leverage Win32_QuickFixEngineering only lists Windows updates. Starting with Windows Vista, Win32_QuickFixEngineering returns only the updates supplied by Component Based Servicing (CBS). These updates are not listed in the registry. Updates supplied by Microsoft Windows Installer (MSI) or the Windows update site (http://update.microsoft.com) are not returned by Win32_QuickFixEngineering. For your information:  Understanding Component-Based Servicinghttp://blogs.technet.com/askperf/archive/2008/04/23/understanding-component-based-servicing.aspx

 I then devised the following script that searched the Windows Update Agent COM Object for all KB's installed on a system including rollups and CU's. To download the script grab it from TechNet:

https://gallery.technet.microsoft.com/Search-Domain-For-Specific-4b6388e9

Exchange Hybrid Mailbox Move Fail - 401 Unauthorized

This summary is not available. Please click here to view the post.

Skype for Business Failover/Failback Issue

I just wrapped up a week long exercise with a client who had a complete failure of their VMware stack at their primary data center resulting in the need to perform an emergency failover to their DR site. This client had a recent deployment and luckily was following all of the best practices and had current backups of everything resulting in a fairly painless failover. The issue that we ran into was with the failback. When we attempted to fail the CMS back replication stopped and the file transfer service would not start. We saw the following in the event Log:

Log Name:      Lync Server
Source:        LS Master Replicator Agent Service
Date:          2/23/2017 8:16:07 PM
Event ID:      2035
Task Category: (2122)
Level:         Error
Keywords:      Classic
User:          N/A
Computer:      DenFE01.contoso.com
Description:
Skype for Business Server 2015, Master Replicator Agent is trying to connect to a backend that whose state does not match with the service sate.

Service State: 
Backup Backend State: 
Active Backend Connection String 
densql01.contoso.com
Cause: Possible issues with back-end database.
Resolution:
Fix the topology so that it matches with the backend and publish.



Log Name:      Lync Server
Source:        LS File Transfer Agent Service
Date:          2/24/2017 1:49:30 AM
Event ID:      1040
Task Category: (1121)
Level:         Warning
Keywords:      Classic
User:          N/A
Computer:      DenFE01.contoso.com
Description:
Skype for Business Server 2015, File Transfer Agent service is stopping.

Reason: The service is trying to start as Active service but the backend it is trying to connect is in Backup state. Backend connection string: Data Source=densql01.contoso.com;
                Initial Catalog=xds;
                Integrated Security=True;
                Application Name=File Transfer Agent;Failover Partner=densql02.contoso.com;



Log Name:      Lync Server
Source:        LS Backup Service
Date:          2/24/2017 2:32:56 AM
Event ID:      4080
Task Category: (4000)
Level:         Error
Keywords:      Classic
User:          N/A
Computer:      DalFE01.contoso.com
Description:
Skype for Business Server 2015, Backup Service central management backup module failed to complete export operation.

Configurations:
Backup Module Identity:CentralMgmt.CMSMaster
Working Directory path:\\dalcfile01.contoso.com\lyncshare\2-BackupService-6\BackupStore\Temp
Local File Store Unc path:\\dalcfile01.contoso.com\lyncshare\2-BackupService-6\BackupStore
Remote File Store Unc path:\\dencfile01.contoso.com\lyncshare\1-BackupService-6\BackupStore

Additional Message:
 Exception: Microsoft.Rtc.BackupService.ExportOperationException: Export operation (to zip archive \\dalcfile01.contoso.com\lyncshare\2-BackupService-6\BackupStore\Temp\z-CentralMgmt-f908fa8f-db02-4ab3-8338-17c30cf59a97.zip) is failed due to: Failed to execute stored procedure XdsQueryChangesForBackupReplica2. Native Error: 50000, Exception: ###50023:XdsQueryChangesForBackupReplica2:The central management store being accessed is not the active store. No data can be read or any changes can be made to this store.. Retriable: False. Cookie: <repl:Status xmlns:repl="urn:schema:Microsoft.Rtc.Management.Xds.ReplLayer.2008" FromMachine="CDCB9834-6AAC-43ab-8310-0D4D105EA23A" Supports="v1" ProductVersion="6.0.9319.0" />. ---> System.Data.SqlClient.SqlException: ###50023:XdsQueryChangesForBackupReplica2:The central management store being accessed is not the active store. No data can be read or any changes can be made to this store.
   at System.Data.SqlClient.SqlConnection.OnError(SqlException exception, Boolean breakConnection, Action`1 wrapCloseInAction)
   at System.Data.SqlClient.TdsParser.ThrowExceptionAndWarning(TdsParserStateObject stateObj, Boolean callerHasConnectionLock, Boolean asyncClose)
   at System.Data.SqlClient.TdsParser.TryRun(RunBehavior runBehavior, SqlCommand cmdHandler, SqlDataReader dataStream, BulkCopySimpleResultSet bulkCopyHandler, TdsParserStateObject stateObj, Boolean& dataReady)
   at System.Data.SqlClient.SqlDataReader.TryConsumeMetaData()
   at System.Data.SqlClient.SqlDataReader.get_MetaData()
   at System.Data.SqlClient.SqlCommand.FinishExecuteReader(SqlDataReader ds, RunBehavior runBehavior, String resetOptionsString)
   at System.Data.SqlClient.SqlCommand.RunExecuteReaderTds(CommandBehavior cmdBehavior, RunBehavior runBehavior, Boolean returnStream, Boolean async, Int32 timeout, Task& task, Boolean asyncWrite, SqlDataReader ds, Boolean describeParameterEncryptionRequest)
   at System.Data.SqlClient.SqlCommand.RunExecuteReader(CommandBehavior cmdBehavior, RunBehavior runBehavior, Boolean returnStream, String method, TaskCompletionSource`1 completion, Int32 timeout, Task& task, Boolean asyncWrite)
   at System.Data.SqlClient.SqlCommand.RunExecuteReader(CommandBehavior cmdBehavior, RunBehavior runBehavior, Boolean returnStream, String method)
  at System.Data.SqlClient.SqlCommand.ExecuteReader(CommandBehavior behavior, String method)
   at System.Data.SqlClient.SqlCommand.ExecuteReader()
   at Microsoft.Rtc.Common.Data.DBCore.Execute(SprocContext sprocContext, SqlConnection sqlConnection, SqlTransaction sqlTransaction)
   --- End of inner exception stack trace ---
   at Microsoft.Rtc.BackupService.BackupModules.XdsBackupModuleBase.QueryChanges(Zipper zipper, String oldCookie, String& newCookie, Boolean& isFullSync, ExportedDataStats& overallExportStats, Dictionary`2& queueExportStatsMap)
   at Microsoft.Rtc.BackupService.BackupModules.XdsBackupModuleBase.GetChanges(Zipper zipper, String oldCookie, String& newCookie, Boolean& isFullSync, ExportedDataStats& overallExportStats, Dictionary`2& queueExportStatsMap)
   at Microsoft.Rtc.BackupService.BackupModules.CentralMgmtBackupModule.GetChanges(Zipper zipper, String oldCookie, String& newCookie, Boolean& steadyState, Int32& numOfNewChanges, Nullable`1& numOfNewChangesFromTheOtherPool, Nullable`1& hasChangesSince, Boolean& forceSetErrorState, ChangesContext& context)
   at Microsoft.Rtc.BackupService.BackupModuleHandler.SendBackupDataTask.GetChanges(Boolean& steadyState, Int32& numOfNewChanges, Nullable`1& numOfNewChangesFromTheOtherPool, Nullable`1& hasChangesSince, Boolean& forceSetErrorState, ChangesContext& changesContext)
   at Microsoft.Rtc.BackupService.BackupModuleHandler.SendBackupDataTask.InternalExecute()
   at Microsoft.Rtc.Common.TaskManager`1.ExecuteTask(Object state)

Cause: Either network or permission issues. Please look through the exception details for more information.


So I verified that the SCP value was pointing to the primary pool, I logged into the SQL DB's in both sites however and the XDS DBConfigInt table both reported:

On the primary pool the dbo.configint value:
Name Value
CurrentState 3
DbVersionSchema 10
DbVersionSproc 15
DbVersionUpgrade 4
IsXdsReadOnly 0

On the seconday pool the dbo.configint value:
Name Value
CurrentState 3
DbVersionSchema 10
DbVersionSproc 15
DbVersionUpgrade 4
IsXdsReadOnly 0

CurrentState 3 means that they are in a "backup" state and not primary. So we then modified the SCP to point back to the secondary pool:

msRTCSIP-BackEndServer: changed to dalsql01.contoso.com
msRTCSIP-BackEndServermirror: changed to dalsql02.contoso.com

Then we modified the CMS database on the secondary pool's primary SQL server by using the following command:

Update [xds].[dbo].[DbConfigInt] Set Value=0 Where Name='CurrentState'

We then published topology, and replication was working with CMS on the secondary pool. We then re-failedback the CMS to the primary pool and this time it was successful. So at this point CMS was healthy, replication was working, users were able to sign in and make/receive calls, however users could not create new meetings. So I started analyzing the FE's event logs and ran across the following event:

Log Name:      Lync Server
Source:        LS User Store Sync Agent
Date:          2/24/2017 12:30:42 AM
Event ID:      57005
Task Category: (1061)
Level:         Error
Keywords:      Classic
User:          N/A
Computer:      DenFE01.contoso.com
Description:
Error encountered pushing data to RtcXds Blob Store

#CTX#{ctx:{traceId:1336022626, activityId:"199e5a7e-6a3c-4cde-82cb-3cf3694b01c2"}}#CTX#
Push cycle identifier: [DenFE01.contoso.com.2fd688f5-0f3a-407f-bab5-3fa5c3757443]
ItemCount: [20]
Error Message: [PushController: XdsPublishItems failed: System.Data.SqlClient.SqlException (0x80131904): ###50015:XdsPublishItems:Local write is not supported in system publications.
   at System.Data.SqlClient.SqlConnection.OnError(SqlException exception, Boolean breakConnection, Action`1 wrapCloseInAction)
   at System.Data.SqlClient.TdsParser.ThrowExceptionAndWarning(TdsParserStateObject stateObj, Boolean callerHasConnectionLock, Boolean asyncClose)
   at System.Data.SqlClient.TdsParser.TryRun(RunBehavior runBehavior, SqlCommand cmdHandler, SqlDataReader dataStream, BulkCopySimpleResultSet bulkCopyHandler, TdsParserStateObject stateObj, Boolean& dataReady)
   at System.Data.SqlClient.SqlDataReader.TryConsumeMetaData()
   at System.Data.SqlClient.SqlDataReader.get_MetaData()
   at System.Data.SqlClient.SqlCommand.FinishExecuteReader(SqlDataReader ds, RunBehavior runBehavior, String resetOptionsString)
   at System.Data.SqlClient.SqlCommand.RunExecuteReaderTds(CommandBehavior cmdBehavior, RunBehavior runBehavior, Boolean returnStream, Boolean async, Int32 timeout, Task& task, Boolean asyncWrite, SqlDataReader ds, Boolean describeParameterEncryptionRequest)
   at System.Data.SqlClient.SqlCommand.RunExecuteReader(CommandBehavior cmdBehavior, RunBehavior runBehavior, Boolean returnStream, String method, TaskCompletionSource`1 completion, Int32 timeout, Task& task, Boolean asyncWrite)
   at System.Data.SqlClient.SqlCommand.RunExecuteReader(CommandBehavior cmdBehavior, RunBehavior runBehavior, Boolean returnStream, String method)
   at System.Data.SqlClient.SqlCommand.ExecuteReader(CommandBehavior behavior, String method)
   at System.Data.SqlClient.SqlCommand.ExecuteReader()
   at Microsoft.Rtc.Common.Data.DBCore.Execute(SprocContext sprocContext, SqlConnection sqlConnection, SqlTransaction sqlTransaction)
ClientConnectionId:4f6d9a2e-01d4-4ca8-b449-2a194446cf67
Error Number:50000,State:1,Class:11]
Cause: Possible issues with back-end database.
Resolution:
Ensure the back-end is functioning correctly.


Log Name:      Lync Server
Source:        LS User Store Sync Agent
Date:          2/24/2017 12:30:42 AM
Event ID:      57006
Task Category: (1061)
Level:         Warning
Keywords:      Classic
User:          N/A
Computer:      DenFE01.contoso.com
Description:
RtcDb Sync Agent sproc failed

#CTX#{ctx:{traceId:1336022626, activityId:"199e5a7e-6a3c-4cde-82cb-3cf3694b01c2"}}#CTX#
Sproc: [XdsPublishItems]
Exception: [System.Data.SqlClient.SqlException (0x80131904): ###50015:XdsPublishItems:Local write is not supported in system publications.
   at System.Data.SqlClient.SqlConnection.OnError(SqlException exception, Boolean breakConnection, Action`1 wrapCloseInAction)
   at System.Data.SqlClient.TdsParser.ThrowExceptionAndWarning(TdsParserStateObject stateObj, Boolean callerHasConnectionLock, Boolean asyncClose)
   at System.Data.SqlClient.TdsParser.TryRun(RunBehavior runBehavior, SqlCommand cmdHandler, SqlDataReader dataStream, BulkCopySimpleResultSet bulkCopyHandler, TdsParserStateObject stateObj, Boolean& dataReady)
   at System.Data.SqlClient.SqlDataReader.TryConsumeMetaData()
   at System.Data.SqlClient.SqlDataReader.get_MetaData()
   at System.Data.SqlClient.SqlCommand.FinishExecuteReader(SqlDataReader ds, RunBehavior runBehavior, String resetOptionsString)
   at System.Data.SqlClient.SqlCommand.RunExecuteReaderTds(CommandBehavior cmdBehavior, RunBehavior runBehavior, Boolean returnStream, Boolean async, Int32 timeout, Task& task, Boolean asyncWrite, SqlDataReader ds, Boolean describeParameterEncryptionRequest)
   at System.Data.SqlClient.SqlCommand.RunExecuteReader(CommandBehavior cmdBehavior, RunBehavior runBehavior, Boolean returnStream, String method, TaskCompletionSource`1 completion, Int32 timeout, Task& task, Boolean asyncWrite)
   at System.Data.SqlClient.SqlCommand.RunExecuteReader(CommandBehavior cmdBehavior, RunBehavior runBehavior, Boolean returnStream, String method)
   at System.Data.SqlClient.SqlCommand.ExecuteReader(CommandBehavior behavior, String method)
   at System.Data.SqlClient.SqlCommand.ExecuteReader()
   at Microsoft.Rtc.Common.Data.DBCore.Execute(SprocContext sprocContext, SqlConnection sqlConnection, SqlTransaction sqlTransaction)
ClientConnectionId:4f6d9a2e-01d4-4ca8-b449-2a194446cf67
Error Number:50000,State:1,Class:11]

We then decided to drain services from one FE at time, and re-run Step 1 and Step 2 from the deployment wizard to reset the local SQL instance on each FE followed up by a reboot. After this process each FE came back up without issue and all functionally was restored.

Microsoft has confirmed that this is a bug and I will try to update this post once Microsoft releases a fix for this bug. 

Conferencing Modalities No Longer Function - MS16-065

Microsoft recently published a KB article related to a security bulletin MS16-065:

Symptoms

After you install the Microsoft .NET Framework Security Update MS16-065 on a Front End or Standard Edition server for Lync Server 2010, Lync Server 2013, or Skype for Business Server 2015, several conferencing modalities no longer function for internal users.

For a complete list of the .NET Framework updates that can cause this problem, see the Microsoft Security Bulletin MS16-065 - Important topic on the Microsoft TechNet website.

The following are known modalities affected by this issue:

·         Whiteboards

·         Uploading PowerPoint Presentations

·         Sharing Notes

·         Polls

·         Q&A

The error messages that users may receive when this problem occurs include the following:

·         We can’t connect to the server for presenting right now.

·         Network issues are keeping you from sharing notes and presenting whiteboards, polls and uploaded Pow…

·         An error occurred during the Skype Meeting.

I ran into this but it was not immediately apparent to which KB's this correlates to so I thought I would list out the ones that will cause this behavior and save someone else some time hunting:

Server 2008 R2:
KB3142024
KB3142033
KB3142037

Server 2012:
KB3142025
KB3142032
KB3142035

Server 2012 R2:
KB3142026
KB3142030
KB3142036

Update:

This also breaks the Lync Web App in Lync Server 2010 which is not documented in the KB at the time of this post. You will also need to add the following reg keys in [HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\.NETFramework\v2.0.50727\System.Net.ServicePointManager.SchSendAuxRecord]:

DWORD Name: C:\Program Files\Microsoft Lync Server 2010\OCSMCU\Web Meeting Conferencing\MeetingMCUSvc.exe
DWORD Value: 0

DWORD Name: C:\Windows\System32\inetsrv\w3wp.exe
DWORD Value: 0

Then restart the RTCMEETINGMCU service as well as perform an IISReset.

Mobility - Topology Doesn't Have Required SIP Listening Port Setting

I had a client with a Lync 2010 deployment recently phone me and tell me that some users were unable to login to their mobile client when outside of the organization. The first thing I did was check the MCX service on the pool that the users were homed to. I was immediately greeted with Server Error in MCX application:

I attempted to recycle the external MCX application pool which typically fixed this issue, however it did not resolve it. I then started digging through the event logs on the server and noticed the following error Application Event ID 1309:

To which the following portion of the error specifically caught my eye:

Exception message: Topology doesn't have required sip listening port setting

I then ran:

Get-CsService -Identity WebServer:FQDNOFFEPOOL.com


Which showed that the value for McXSipExternalListeningPort was $null:

Then I ran the following to set the MCX External Listening and Primary Listening ports to the correct values:

Set-CsWebServer -McxSipExternalListeningPort 5087 -McxSipPrimaryListeningPort 5086 -Identity WebServer:FRONENDSERVERFQDN.com


Then running the get-csservice command again showed that the values were correct:

At which point I invoked CMS replication, verified it had completed replicating and then performed and IISReset and the mobility service was working once again:

How to Disable Interfaces on AudioCodes Mediant 1000

One of our clients recently rolled out AudioCodes Element Management System (EMS) and noticed that they were receiving a lot of alarms about interfaces being down. You might also see these alarms show up on the gateway management page:

I wasn't able to find much online in the way of how to administratively down or disable the alarms on each gateway so I opened a support ticket figured it out and thought I should post this in the event that anyone else out there also needs to do this. 

First login to your gateway and determine which interface you want to turn disable the alarm on. The interfaces are read on the top row beginning as GB_0_1 on the left and then going two, three, four, etc, if you have another row of interfaces then it would be GB_X_1 with X being 1-9

Once you have written down which interface you want to remove, expand VoIP -> Network -> and select Ethernet Groups Table:

Select Index 0 (or whichever index has the interface under the member column) and then click edit:

In the edit record window click the drop down of the member you want to remove, and change it to none:

Click submit, and your changes should show the Index as no longer having that interface listed:

You will then need to restart the gateway for the changes to take effect