Monitoring Oozie Coordinators Using Web API

If you work with oozie for ETL workflow executions chances are that you ended up overwhelmed when you need to monitor several workflows all at once and you found the web console of Oozie being too basic. Fear no more my friend, this time I want to share with you a little script I made for dealing with such situations.

Bear in mind before starting with the script:

• The bellow code is available on my github repo under creative commons license if you want to check it out for yourself before looking at the explanation.
• Oozie web API was used as the base of the script, if you’re not familiar with it, you should take a look on the reference page.
• The script is python-web based so you should have CGI module enabled on your Apache web server before proceding (here’s a guide for install apache on windows without additional packages that are unnecessary for the purpose of this post, in GNU/Linux this should be a piece of cake).

There are two main parts of the tool:

1. Filters form

Here you find parameters for getting the results for the workflows of your choice:

Select the status: choose between oozie status for coordinators (RUNNING, SUCCEEDED, KILLED, DONEWITHERROR)

Max jobs to search in: limit the number of jobs in which to search in for the proper workflows (this is for pagination on oozie API)

Select the workflows to analyse: This is really useful if you need to monitor a bunch of workflows everyday. All you need to do is edit oozieMonitoring.html file from line 60 to add your workflows names (you should use names here as the appName is defined without any date portion i.e: cristianpark-workflow and not cristianpark-workflow_Daily_20180220)

Custom workflow name:  Enter one workflow name (only one workflow currently supported) that want to monitor (sometimes you want to monitor a workflow outside your regular list of workflows).

 

2. Results script

For this script to work you should have CGI enabled and identify the path for python on your system.

Some quick notes on this:

• You should replace first line on coordinatorsMonitoring.py file #!C:\PATH\FOR\PYTHON\python.exe for your actual python (.exe on windows) executable path.
  
• The final URL generated within the script looks like http://OOZIE-LOCATION/oozie/v2/jobs?filter=status%3DSTATUS_FILTER&offset=1&len=JOBS_NUMBER_FILTER&timezone=GMT, feel free to explore it from your browser to see how the results look like.

If there are results for your workflows in the status filtered, you should see a screen like this (grouped by workflow name)

Where you can see all the instances a workflow has in execution (or in another state according to your selection) and pretty basic info about each instance.

If there are no results, you end up with something like this:

I hope this can help someone for his/her task of monitoring several workflows. This tool is in an alpha stage so you’re more than welcome to add your two cents to it.

 

Don’t Worry…….Be Linux!!!