Error-handling, or event-handling, is an important requirement for any data integration project. Determining the events and “handling the encountered events” is a key design aspect in a robust design. Precise, detailed and meaningful reports enable efficient analysis of the captured events by the business and IT teams.

In this first of two installments, we describe best practices for handling unexpected and expected events within Jitterbit. Included are logging, validation and notification methods depending upon severity.

In a later second installment, we discuss, along with On Failure Operations, techniques for threshold checks and restart-ability practices that will ease maintenance tasks.

Jitterbit Logging

As you may know, informational field values helpful to analysis and troubleshooting are logged per Operation via the scripting WriteToOperationLog() function. See a more extensive use of WriteToOperationLog() near the end of Integration Pattern for Alerting, Logging and Error Handling in the Jitterbit documentation.

Event Types and Severities

Jitterbit events might be classified with expectations as follows:

 

Type Description Severity
Unexpected Whether deliberate, caught and/or data-driven; operation usually terminated Emergency, Alert, Critical or Error
Unexpected Jitterbit warnings, desired warnings and any Jitterbit error not considered severe Warning or Notice
Expected Other events requiring a log entry or notification Notice, Informational or Debug
Expected Data validations Warning, Notice, Informational or Debug

Data validations are unique processes in data integrations. We examine data validation within Jitterbit further below.

Unexpected Events

Most unexpected errors are trapped as Jitterbit Operation Errors. Operation errors, by default, will immediately terminate an operation chain. If understood as severe and catastrophic, these errors are best handled by On Failure Operations. Another method to trap and handle errors is within a Script using:

If( !RunOperation(“…”), … );

This way, if needed, one can assess the error seen in the $jitterbit.operation.last_error variable and use the RaiseError() function to terminate the operation (or even bubble-up to a parent or calling Script). Note, with RaiseError(), you can deliberately cause a Jitterbit error at any time within a Script. The term “Raise Error” is used to evoke a Jitterbit error and a common event that needs handling.

Expected Events

Expected errors are typically identified and logged during a data validation/quality processing phase just after extraction. Unlike most unexpected errors, when conducting data validations, you may not want the Operation chain to halt when a record fails. The validations should continue with the incoming data. If multiple validations are done on a record, all the validations should be done to ensure all the errors are captured.

Data Validation

When data is available in the row-column format, Jitterbit provides a rich and powerful data validation process. The File Format component includes options to validate each row or field.

The row/record is typically skipped but, depending upon target requirements, a default value can be applied or the field could be ignored. One can skip the row or abort the Operation (Raise Error) if a minimum number of columns are not available. In addition, per field, one can use a number of general validation rules and more specific content validation rules, including applying defaults or Scripts for handling events.

If an Operation’s Transformation uses a File Format as a Source, the Operation’s Options provides for emailing or storing valid or invalid records automatically. Once enabled, you can record invalid records in a dedicated Target file including errors in a final/last column that can be easily converted to multiple columns with a spreadsheet editor. Depending upon your data validation requirements, it may be worth transforming XML data into a flat file for validation (and further processing).

Tracking an error count and/or acting upon an event or error threshold check can be another requirement. Below, we discuss acting upon an event and, within the next installment, we’ll discuss threshold checks.

Event Notification Methods

A good practice is to centralize the event/error handling and logging design where appropriate.  Desirable features include:

  • Configurable details available when needed using a global log-level variable value

  • Event severity configuration

  • Optional immediate email of event

A formal scripted condition [ If(!RunOperation(),…RunScript(),…) ] or an On Failure Operation can call a parametrized script that processes the above event handling requirements. The script below (“Handle Event”) is easily modifiable and this example provides extensive event severity values and arguments. 

  1. $LOG.severity = {{“Debug”,“Informational”,“Notice”,“Warning”,“Error”,“Critical”,“Alert”,“Emergency”}, //code 0
  2. {“Debug”,“Informational”,“Notice”,“Warning”,“Error”,“Critical”,“Alert”}, //code 1
  3. {“Debug”,“Informational”,“Notice”,Warning,“Error”,“Critical”}, //code 2
  4. {“Debug”,“Informational”,“Notice”,“Warning”,“Error”}, //code 3
  5. {“Debug”,“Informational”,“Notice”,“Warning”}, //code 4
  6. {“Debug”,“Informational”,“Notice”}, //code 5
  7. {“Debug”,“Informational”}, //code 6
  8. {“Debug”} //code 7
  9. };
  10. // Allows extensive logging but visibility control using $LOG.level; For example, set $LOG.level=”Debug” to log events with code 0-7
  11. $LOG.level = IfEmpty($LOG.level, WriteToOperationLog(“Using $LOG.level=Emergency”);“Emergency”);
  12. _event_code = Int(_1); _event_type = _2;
  13. _event_msg = _3; //optional if called from On Failure or after a failed RunOperation()
  14. _send_event = Bool(IfEmpty(_4, false)); //optional
  15. _event_category = _5; //optional
  16. helperArr = Array(); i=0While(i<Length($LOG.severity[_event_code]), Set(helperArr,i,-1); i++);
  17. $jitterbit.operation.name=IfEmpty($jitterbit.operation.name,“NO OPERATION NAME AVAILABLE”);
  18. $jitterbit.operation.previous.error=IfEmpty($jitterbit.operation.previous.error,
  19. “NO EVENT_MSG OR PREVIOUS ERROR AVAILABLE”);
  20. If(!IsNull(lvlFnd=FindValue($LOG.level, $LOG.severity[_event_code], helperArr)),
  21.   type = msg = $jitterbit.operation.name+:+ _event_type ” :: “; // Error type
  22.   msg += IfEmpty(_event_msg,$jitterbit.operation.previous.error) “\n”; // Operation error, if needed
  23.   $EVENT.msg=msg; outcome=“”;
  24.   If(_send_event && (outcome=SendEmailMessage(…)) != “”), //always test email outcome
  25.     outcome=“Email Notification of EVENT/ERROR (” type “) failed: ” outcome “\nEmail Message Body:n”
  26.   );
  27.   WriteToOperationLog(outcome+$EVENT.msg);
  28.   outcome;
  29. );

It’s important to note that all logging could use the above script; one can script all event logging needs and control their visibility (and actions like Email) with the $LOG.level variable.

If the above script is titled “Handle Event”, we might use this within other Scripts like:

  1. If(!RunOperation(…),
  2.   result=RunScript(“…/Handle Event”, 0, “Emergency Type”, GetLastError(), false“FATAL”);
  3.   RaiseError(result $EVENT.msg) //catastrophic, abort (or catch) the operation chain
  4. );

Summary

The above practices with Jitterbit empower an integration design that is flexible and robust.  Best practices include:

  • Design error-handling capabilities for both the expected and unexpected errors for ill-behaving or corrupt records.

  • Centralize the event/error handling.

Try the above Handle Event Script, with invocations like the following. After execution, view the Operation Log.

  1. $LOG.level=“Debug”; WriteToOperationLog(“\nLOG.LEVEL:”+$LOG.level); // see results using different $LOG.level values
  2. RunScript(“…/Handle Event”,0,“Emergency type logged using $LOG.level=”
  3. +$LOG.level,“System is unusable.”,false,“FATAL”);
  4. RunScript(“…/Handle Event”,1,“Alert type logged using $LOG.level=”
  5. +$LOG.level,“Should be corrected immediately.”,false,“ALERT”);
  6. RunScript(“…/Handle Event”,2,“Critical type logged using $LOG.level=”
  7. +$LOG.level,“Critical conditions.”,false,“CRITICAL”);
  8. RunScript(“…/Handle Event”,3,“Error type logged using $LOG.level=”
  9. +$LOG.level,“Error conditions.”,false,“ERROR”);
  10. RunScript(“…/Handle Event”,4,“Warning type logged using $LOG.level=”
  11. +$LOG.level,“May indicate that an error will occur if action is not taken.”,false,“WARN”);
  12. RunScript(“…/Handle Event”,5,“Notice type logged using $LOG.level=”
  13. +$LOG.level,“Events that are unusual, but not error conditions.”,false,“NOTICE”);
  14. RunScript(“…/Handle Event”,6,“Informational type logged using $LOG.level=”
  15. +$LOG.level,“Normal operational messages that require no action.”,false,“INFO”);
  16. RunScript(“…/Handle Event”,7,“Debug type logged using $LOG.level=”
  17. +$LOG.level,“Information useful to developers for debugging the application.”,false,“DEBUG/ALL”);

Leave a Reply

Please log in using one of these methods to post your comment:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s