Monitoring Oozie Coordinators Using Web API


If you work with oozie for ETL workflow executions chances are that you ended up overwhelmed when you need to monitor several workflows all at once and you found the web console of Oozie being too basic. Fear no more my friend, this time I want to share with you a little script I made for dealing with such situations.

Bear in mind before starting with the script:

  • The bellow code is available on my github repo under creative commons license if you want to check it out for yourself before looking at the explanation.
  • Oozie web API was used as the base of the script, if you’re not familiar with it, you should take a look on the reference page.
  • The script is python-web based so you should have CGI module enabled on your Apache web server before proceding (here’s a guide for install apache on windows without additional packages that are unnecessary for the purpose of this post, in GNU/Linux this should be a piece of cake).

There are two main parts of the tool:

1. Filters form

Here you find parameters for getting the results for the workflows of your choice:

Select the status: choose between oozie status for coordinators (RUNNING, SUCCEEDED, KILLED, DONEWITHERROR)

Max jobs to search in: limit the number of jobs in which to search in for the proper workflows (this is for pagination on oozie API)

Select the workflows to analyse: This is really useful if you need to monitor a bunch of workflows everyday. All you need to do is edit oozieMonitoring.html file from line 60 to add your workflows names (you should use names here as the appName is defined without any date portion i.e: cristianpark-workflow and not cristianpark-workflow_Daily_20180220)

Custom workflow name:  Enter one workflow name (only one workflow currently supported) that want to monitor (sometimes you want to monitor a workflow outside your regular list of workflows).

 

2. Results script

For this script to work you should have CGI enabled and identify the path for python on your system.

Some quick notes on this:

If there are results for your workflows in the status filtered, you should see a screen like this (grouped by workflow name)

Where you can see all the instances a workflow has in execution (or in another state according to your selection) and pretty basic info about each instance.

If there are no results, you end up with something like this:

I hope this can help someone for his/her task of monitoring several workflows. This tool is in an alpha stage so you’re more than welcome to add your two cents to it.

 

Don’t Worry…….Be Linux!!!

 

 

Anuncios

Etiquetas: , , , , , , , , ,

Responder

Introduce tus datos o haz clic en un icono para iniciar sesión:

Logo de WordPress.com

Estás comentando usando tu cuenta de WordPress.com. Cerrar sesión /  Cambiar )

Google+ photo

Estás comentando usando tu cuenta de Google+. Cerrar sesión /  Cambiar )

Imagen de Twitter

Estás comentando usando tu cuenta de Twitter. Cerrar sesión /  Cambiar )

Foto de Facebook

Estás comentando usando tu cuenta de Facebook. Cerrar sesión /  Cambiar )

w

Conectando a %s

A %d blogueros les gusta esto: