Automating system administration tasks – Part1

In 2013 I wrote the article The morning of a system administrator where I had formulated the main questions to which – in my humble opinion – a system administrator should have the answers at the very beginning of his/her work day:

1) Were there any unexpected server reboots?

2) Is there any server that lacks of a disk space?

3) What are the sizes of my MS SQL Server databases?

4) Were there any connectivity problems with our Internet connection?

5) Were there any Active Directory modifications?

6) Were there any password resets?

As the years past, I added more and more questions to the list above and one day I realized it would be pertinent to divide the questions into several categories: one for each of the “computer field” administrators usually deal with:

I Windows Server

II Audit

III SQL Server

IV Exchange Server

Considering I want to have these answers by the time I enter the office, the main means of getting the corresponding information must be the scripts that run according to to their schedule in the Task Scheduler and send me reports via an email.  Alerts in the Performance Monitor can send notifications  when a certain condition is met. In this four-part series I’d like to offer the  methodology for the overall monitoring of the whole server infrastracture so that any administrator may get all necessary information as fast as possible – he/she would only need to read the reports sent as the email attachments by various scripts.

 

I Windows Server

The questions:

  1. Have nightly backup jobs completed successfully?
  2. Were there any connectivity problems with our Internet connection?
  3. Were there any unexpected server reboots?
  4. Is there any server that lacks of a disk space?  – the question  from the old article now should read as follows:
    Have any defined server alerts fired since yesterday?
  5. How well does the OS perform today (or did yesterday)?

 

Throughout this four-part series I will post lots of cmd, ps1 and vbs scripts saved as docx documents -wordpress.com would not allow to upload txt/ps1/vbs files – so please rename the docx files accordingly before use.
Some one-line cmd scripts are not published as the docx files because their content is fully visible as the screenshots.

 

The answers:

1) To get the email reports upon completion of the backup jobs I have attached the task Windows Backup Succeeded to the event 4 from the Microsoft-Windows-Backup-Operational log:

The task:

The scripts (I use .cmd script on the Actions tab of the task to run the corresponding same-titled .ps1 script):

Mail-DC.ps1Mail-DC-ps1

$login = “sysadmin@testenterprise@com”
$password = “123456” | Convertto-SecureString -AsPlainText -Force
$creds = New-Object System.Management.Automation.Pscredential -Argumentlist $login,$password
Send-MailMessage -From “sysadmin@testenterprise@com” -To “michael_firsov@testenterprise.com” -Subject ”  Windows Backup OK on DC” -Body “Windows Backup succeeded on DC” -SmtpServer mail.testenterprise.com -Port 25 -Credential $creds

The resulting email notification (each server will require its ownthe  task/script, of course):

 

2) Internet availability check

Internet.vbs script checks whether the internet connection is available by continuously pinging the ip of our ISP’s gateway (it, of course, may be any other host you choose to monitor): in case one of the pings is lost it makes up to two additional checks and if all three consecutive ping attempts have been unsuccessful it will log the corresponding record to the Internet.txt file and send it to the email address defined in the script. This Internet.txt file will also serve as the internet availability log since any interruptions in the internet availability will be recorded in this singe file and an administrator may easily know later when the internet channel was not functioning.

Internet.vbsInternet-vbs

Please set the StrTarget field in the downloaded script to the ip of some host on Internet you’d like to serve as a monitor of the availability of your internet channel (instead of a.b.c.d) and the strDirectory field to the folder you’d like the Internet.txt file to be placed in. This script will be pinging this host each 120000 milliseconds by default (=120 seconds or 2 minutes) – the following parameter lets you configure it as you like:

WScript.Sleep 120000 ‘Frequency of polling.

Once one of the ping attempts fails (no reply is received) the script tries to ping the monitored host for the second time after the 30 seconds delay by default, and  after another 30 seconds it makes the third attempt – this total 60 seconds worth pause should help avoid the intermittent network failures. If all the three ping attempts fail then the script writes the line into INTERNET.txt file and sends it to the configured mail recepient (please configure all e-mail settings in the script according to your needs!).

The script files:

 

You can run any .vbs scripts from scheduled tasks directly – without a .cmd file with the cscript command-line program I’m currently using  – I just prefer to start the vbs scripts that run continiously by means of the cscript command: in this case in order to stop the script in the Task Manager I should stop the cscript or Windows Console Host process:

The report file:

 

This e-mail message gets sent to my mailbox in case internet channel 1 is not available:

In fact the Internet.vbs script serves the two purposes:

  1. the email message in an administrator’s inbox informs him/her about the missing internet connection at the current time and
  2. the file attached will let an administrator know how long the connection was unavailable and how often such interuptions occur

To be able to get the internet channel availability report any time during a day I create the corresponding Internet Channel 1 scheduled task:

 

3) ServerReboots – ServerReboots-vbs

Please type in the names of your servers in the Aservers filed (Aservers = Array(“DC“,”TI“) by default).

The script file:

The task ServerReboots:

As ServerReboots.vbs is NOT going to be run continiously I start this script directly (withouth the corresponding .cmd file).

 

4. Alerts

The scripts:

CPU-Alert-MAIL-DC-cmd
CPU-Alert-MAIL-DC-ps1
FreeSPACE-Alert-MAIL-DC-cmd
RAM-Alert-MAIL-DC-cmd
FreeSPACE-Alert-MAIL-DC-ps1
RAM-Alert-MAIL-DC-ps1

 

For example, CPU-Alert-MAIL-DC.cmd runs CPU-Alert-MAIL-DC.ps1

$login = “sysadmin@testenterprise.com”
$password = “123456” | Convertto-SecureString -AsPlainText -Force
$creds = New-Object System.Management.Automation.Pscredential -Argumentlist $login,$password
Send-MailMessage -From “sysadmin@testenterprise.com” -To “michael_firsov@testenterprise.cpom” -Subject “CPU Alert on DC !” -Body “CPU Alert on DC! – CPU > 70%” -SmtpServer mail.testenterprise.cpom -Port 25 -Credential $creds

The other scipts pairs differ only in the -Subject and -Body fields: you can type there any warning you like to inform yourself what exactly alert have fired:

FreeSPACE–Alert-MAIL-DC.ps1
$login = “sysadmin@testenterprise.com”
$password = “123456” | Convertto-SecureString -AsPlainText -Force
$creds = New-Object System.Management.Automation.Pscredential -Argumentlist $login,$password
Send-MailMessage -From “sysadmin@testenterprise.com” -To “michael_firsov@testenterprise.cpom” -Subject “CPU Alert on DC !” -Body “CPU Alert on DC! – CPU > 70%” -SmtpServer mail.testenterprise.cpom -Port 25 -Credential $creds
RAM-Alert-MAIL-DC.ps1
$login = “sysadmin@testenterprise.com”
$password = “123456” | Convertto-SecureString -AsPlainText -Force
$creds = New-Object System.Management.Automation.Pscredential -Argumentlist $login,$password
Send-MailMessage -From “sysadmin@testenterprise.com” -To “michael_firsov@testenterprise.cpom” -Subject “RAM Alert on DC” -Body “RAM Alert on DC: RAM: Available bytes < 6 GB.”  -SmtpServer mail.testenterprise.cpom -Port 25 -Credential $creds

These scripts (.cmd scripts (!), which in turn will run the .ps1 scripts of the same name) will be run by the corresponding scheduled task.

Please note that the Triggers tag of the task is empty – to make the alert run after server reboot I must configure the Triggers tab of another FreeSpace-Alert task which gets created automatically at Microsoft\Windows\PLA section of the Task Scheduler (I’ll show it a bit later here).

Here are the tasks that make alerts persist over server reboots: each task that have been created at the Task Scheduler\(Library) level has a corresponding same-titled task at Microsoft\PLA level:

It’s also worth noting that alerts will run automatically upon a server restart only if they were running before this server restart!

So the last step in configuring alerts is to configure the Triggers tab of all alert tasks in the PLA section, for example:

You shouldn’t change anything on the Actions tab here – I just wanted to show how this tab look like at the PLA level.

In case (for example) Processor_Time happens to exceed 70%, the following message – after the Sample interval has passed! –  should arrive at michael_firsov@testenterprise.com (for this test I’ve change the Sample Interval from 15 min to 1) and Performance Monitor will keep sending this message utlill Proceesor_Time falls under 70%.

 

 

5. Performance monitoring

While creating alerts may help administrator know which OS parameters are in critical conditions, they can’t be used for establishing the overall system health – for this purpose we should create certain data collector sets that will collect the current performance counters values (daily/weekly/monthly) so that we could analyze that data later – that’s why it’s very important to establish your system monitoring policy right after OS (or any other program, such as SQL Server or Exchange Server) installation – to understand the trends in system performance an administrator should have an “image” of a clean system, otherwise observing the current values of various performance counters may not be of much interest.

For example, on my domain controller I’ve created the DC data collector  – DC-xml – which starts every day at 11:00AM and collects information for 5 minutes:

 

This data collector produces the following folders/files:

 

After importing the resulting .tsv files into – for example – MS Excel an administrator can later analyze data and correlate current values with any changes/procedures on the monitored server.

Part2

Advertisements
%d bloggers like this: