Office 365 – Analyze the security settings for your tenant

UPDATE 24/11/2015 – The script has been already updated has some bugs have been found. So please ensure you are always using the latest version by downloading it from the website

Microsoft is currently testing (early beta stage) an Office 365 security analyzer to help administrators to understand their current security configuration and potential risks they may have missed.

As said this is an early beta, so this will be slightly changed when released.

So, to have an early view, you need to connect to the following site http://aka.ms/o365securescore

As first action, you need to download a bunch of PowerShell scripts to gather the data; you will have to sign in with your Office 365 administrator account to download the files

To help you prepare the client on which you will run these scripts, you need to have:

full connectivity to Office 365 services – which includes Azure AD, Azure RMS, Skype for Business and SharePoint online
installed Microsoft Online Sign-In Assistant (http://go.microsoft.com/fwlink/?LinkID=286152), Azure AD PowerShell (http://go.microsoft.com/fwlink/p/?linkid=236297), Azure PowerShell (http://go.microsoft.com/fwlink/p/?linkid=320376&clcid=0x409), Azure RMS PowerShell (https://go.microsoft.com/fwlink/?LinkId=257721), Skype for Business PowerShell (http://go.microsoft.com/fwlink/?LinkId=294688) and SharePoint Online PowerShell (http://www.microsoft.com/en-us/download/details.aspx?id=35588)

Once you are ready, using a command prompt – as usual always use the run as administrator – run the securityscore.bat file you have extracted from the ZIP file downloaded earlier

You may also have a multiple security prompts to run the PowerShell

Then you will be asked to authenticate to your Office 365 tenant – if your admin account uses MFA, the script will fail.

Then the script is looking if you meet the prerequisites (PowerShell modules)

If you missed some the script is going to connect automatically to the related download page and stops

If everything is looking good, the script connects to Azure AD

You will be asked to confirm your participation as the script will gather data about how your end-users

If you have issue to connect to your tenant (ERROR: FAILED To connect to MSO Service) – as per below, try to edit the CollectAndScoreSecurityData.ps1 and the SecurityScorer.psm1 files to:

Cut the following lines from the SecurityScorer file

Write-Log "Connecting to Azure Active Directory"
$connectMso = Connect-MsolService -Credential $userCredential

Remove the following lines from the SecurityScorer file (just after the previous lines you have cut)

if ($connectMso -eq $null) {
Write-Error " FAILED To connect to MSO Service"
Throw "Failed to Connect to MSO Service" }

Past the cut lines to the CollectAndScoreSecurityData file after the lines, and replace the $userCredential variable with $AdminCredential

Write-Log " Set up the Service Sessions" 2
Get-ServiceSessions $AdminCredential

Then the script is connecting to both Exchange Online, Skype for Business and SharePoint Online.

NOTE 1 it is recommended to assign all the license to the account used for the scoring

NOTE 2 you may got some errors for some accounts during the Exchange Online analysis as some of your accounts may not have an Exchange Online license

Then the script analyze the password policy for each users found