UPDATE 24/11/2015 – The script has been already updated has some bugs have been found. So please ensure you are always using the latest version by downloading it from the website
Microsoft is currently testing (early beta stage) an Office 365 security analyzer to help administrators to understand their current security configuration and potential risks they may have missed.
As said this is an early beta, so this will be slightly changed when released.
So, to have an early view, you need to connect to the following site http://aka.ms/o365securescore
As first action, you need to download a bunch of PowerShell scripts to gather the data; you will have to sign in with your Office 365 administrator account to download the files
To help you prepare the client on which you will run these scripts, you need to have:
- full connectivity to Office 365 services – which includes Azure AD, Azure RMS, Skype for Business and SharePoint online
- installed Microsoft Online Sign-In Assistant (http://go.microsoft.com/fwlink/?LinkID=286152), Azure AD PowerShell (http://go.microsoft.com/fwlink/p/?linkid=236297), Azure PowerShell (http://go.microsoft.com/fwlink/p/?linkid=320376&clcid=0x409), Azure RMS PowerShell (https://go.microsoft.com/fwlink/?LinkId=257721), Skype for Business PowerShell (http://go.microsoft.com/fwlink/?LinkId=294688) and SharePoint Online PowerShell (http://www.microsoft.com/en-us/download/details.aspx?id=35588)
Once you are ready, using a command prompt – as usual always use the run as administrator – run the securityscore.bat file you have extracted from the ZIP file downloaded earlier
You may also have a multiple security prompts to run the PowerShell
Then you will be asked to authenticate to your Office 365 tenant – if your admin account uses MFA, the script will fail.
Then the script is looking if you meet the prerequisites (PowerShell modules)
If you missed some the script is going to connect automatically to the related download page and stops
If everything is looking good, the script connects to Azure AD
You will be asked to confirm your participation as the script will gather data about how your end-users
If you have issue to connect to your tenant (ERROR: FAILED To connect to MSO Service) – as per below, try to edit the CollectAndScoreSecurityData.ps1 and the SecurityScorer.psm1 files to:
- Cut the following lines from the SecurityScorer file
Write-Log "Connecting to Azure Active Directory"
$connectMso = Connect-MsolService -Credential $userCredential
- Remove the following lines from the SecurityScorer file (just after the previous lines you have cut)
if ($connectMso -eq $null) {
Write-Error " FAILED To connect to MSO Service"
Throw "Failed to Connect to MSO Service" }
- Past the cut lines to the CollectAndScoreSecurityData file after the lines, and replace the $userCredential variable with $AdminCredential
Write-Log " Set up the Service Sessions" 2
Get-ServiceSessions $AdminCredential
Then the script is connecting to both Exchange Online, Skype for Business and SharePoint Online.
NOTE 1 it is recommended to assign all the license to the account used for the scoring
NOTE 2 you may got some errors for some accounts during the Exchange Online analysis as some of your accounts may not have an Exchange Online license
Then the script analyze the password policy for each users found
Continue with DLP and Admin roles
Checks for illicit logon against Azure AD, suspicious malware or suspicious activities on delegated mailbox
You may be asked to validate there is no suspicious activities from time to time.
Please note that some activity reports displayed in the console may be quite complicated to read 
At the end you will get a security scoring and upload the results onto the web site
