System monitors are tools that report information on the resources and performance of a given computer system. Data points about the CPU, memory, and hard disk are common to system monitors. But what use are these variables? In a production system, measurements of these variables provide critical insight into how your code is running and how well the machine is handling it. Reports can lead to software diagnoses of memory leaks, need for faster hardware, and cause behind failed I/O operations.
With the psutil module in Python, we gain a simple cross-platform interface between Python and the system for accessing a wide array of system information. By reporting these values to the Ubidots API at regular intervals, we will have a method of visualizing and analyzing these statistics on the cloud.
In this post, we will go through the process of creating a script to monitor CPU usage, memory usage, and hard disk usage in the cloud with Ubidots. The script we are building can be downloaded in whole if you prefer not to read as a tutorial.
Writing the script
Let us begin. Our goal is to create a program that uses the data source “<hostname> Monitor” to report values of three variables: cpu_percent (percent of CPU in use), mem_percent (percent of RAM in use), and disk_percent (percent of hard disk in use). To give the program some extra flavor, we will search for these variables and data source by name, and create them only if necessary.
The first part of any Python program is the shebang line and a description of the program in a docstring. Providing a good description is healthy practice.
#!/usr/bin/python
"""
Monitoring Statistics via Ubidots
This script sends CPU, Memory, and Disk Usage statistics to
Ubidots for visualization and insights. This script is cross
platform and will work on Windows, Linux, and OS X.
"""
We import four modules. We take just the gethostname() function from the socket module, and just the argv variable from the sys module.
from socket import gethostname
from sys import argv
import psutil
import ubidots
Let’s now define the main method. The program will take one command line argument, and it will be the Ubidots API key. The last line below instantiates an ApiClient instance that we will use for the rest of the program.
def main():
"""Main routine for the script."""
if len(argv) != 2:
print "Usage: %s API_KEY" % argv[0]
return
api = ubidots.ApiClient(argv[1])
Now we will write some code to search for the data source matching the desired name of our data source: “<hostname> Monitor.” If no such data source exists, we take the time to create it.
ds_name = gethostname() + " Monitor"
ds = None
for cur_ds in api.get_datasources():
if cur_ds.name == ds_name:
ds = cur_ds
break
if ds is None:
ds = api.create_datasource({"name": ds_name})
With our ds variable created, we will now follow a similar pattern of searching and creating if not found for each of the variables. Outside your main function, define this helper function:
def get_var_by_name(var_name, ds):
"""Search for a variable in a data source. If found,
returns the variable. If not found, returns None."""
for var in ds.get_variables():
if var.name == var_name:
return var
return None
Back in the main function:
`var_cpu = get_var_by_name("cpu_percent", ds)
var_mem = get_var_by_name("mem_percent", ds)
var_disk = get_var_by_name("disk_percent", ds)
if var_cpu is None:
var_cpu = ds.create_variable({"name": "cpu_percent",
"unit": "%"})
if var_mem is None:
var_mem = ds.create_variable({"name": "mem_percent",
"unit": "%"})
if var_disk is None:
var_disk = ds.create_variable({"name": "disk_percent",
"unit": "%"})
Now at the end of our main function, we will make appropriate calls to functions in the psutil module to save the value. The code below looks at all the physical partitions available, and chooses the first to report on.
# Utilize the psutil module to send values to Ubidots.
first_mnt = psutil.disk_partitions(all=False)[0].mountpoint
var_cpu.save_value({"value":
psutil.cpu_percent(interval=1)})
var_mem.save_value({"value":
psutil.virtual_memory().percent})
var_disk.save_value({"value":
psutil.disk_usage(first_mnt).percent})
Scheduling: cron, and without cron
Our desire is to have this code run at regular, timely intervals. Under Linux and Mac OS X, tasks can be automated to be run at N <minutes | hours | days> intervals using the cron service. If we are to schedule our script using cron, we should proceed as follows:
if __name__ == "__main__":
main()
Now to schedule it with cron, we should save and close the file, and type crontab -e into a new terminal. A sample cron line to add is:
* * * * * python /home/daniel/ubidots-sysmon.py 74ccf3e7957be38eh382cgfd107d70870edbb463
If not scheduling during cron, our option is to have the script loop over calls to the main() method, pausing for a certain amount of time to send results at regular intervals. The disadvantage to this is that the program must be kept running continuously to keep reporting statistics. However, the code to append to the python file would look like this:
if __name__ == ‘__main__’:
import time
while True:
main()
time.sleep(10) # pause for 10 seconds
Conclusion
There we have it, in an insignificant number of lines you can monitor your Windows, Linux, or OS X system from the cloud.
Where to go from here? To customize the script or change the variables it reports on, check out the psutil documentation for a complete list of measurements at your fingertips. Just as importantly, remember that Ubidots can notify you of events in your data, and provide insights.
Events can help you catch issues before they become a problem. If disk I/O is a major component of a service running on your system, consider setting up an event to notify you when your disk is nearly full. Consider setting up another event to notify you when memory usage exceeds 90% -- and catch the system before paging occurs.
Insights can also provide you more information about your data. To see how much memory a service on a device is using, we can determine the average percent of memory usage for the previous day. The figure below shows that on average 49.56% of the memory was in use yesterday.
To wrap up, I hope to have shown that while sensors measuring the outside world are the easiest to picture being paired with Ubidots, Ubidots is just as capable of being paired with sensors measuring the inside of a machine.
Have another project idea? Create a Ubidots account and get started!
Originally Published in Ubidots Blog October 24, 2013