In this second of two installments, we continue with the notification of events & errors and the Handle Event Script used in the previous installment. We also propose techniques for (counter) threshold checks and restart-ability practices that will ease maintenance tasks.

It is a best practice to differentiate generic event notification from those specific to the business process.  The (earlier) Handle Event Script provides parameters for these details.  Below is a specific business process data validation script for a field name “rating” where only certain values are allowed. Its purpose is to log the invalid value and the fact that the default (“Warm”) was assigned.

  1. _fieldValue = _1; _fieldType = _2; _fieldFormat = _3; // _3 if available in field’s File Format (more common for Date types)
  2. fieldName = “rating”; fieldDefault = “Warm”;
  3. If(!Validate(“One Of”, ‘Cold,Warm,Hot’),
  4.   RunScript(“…/Handle Event”, 5, “Notice Type”, “Default: “ + Quote(fieldDefault) +” used for “ + Quote(_fieldValue),
  5.     false“NOTICE”);
  6.   false //return false to use File Format’s Default value
  7. , true //else return true; all is well
  8. );

As more events are logged within data validations, one might be interested in counting these events. We provide a counter handler below when we discuss threshold checks but first, we discuss a best practice for Operations in general and On Failure Operations for handling more important events.

As you will learn in the Restart-ability topic below, initializing variables correctly is an important scripting task. For example, an Operation’s $LOG.level value used by the Handle Event Script is best initialized in each Operation’s setup Script. It’s good practice to begin each Operation with a setup script to initialize this and other variables.

It’s also a good practice to summarize an Operation’s outcome in a final, wrap up Script in each Operation or perhaps a summary Email at the end of an Operation chain. This will typically include statistics of good/processed versus bad/unprocessed records (data validation) allowing for variance measurements over numerous executions. In an Email Target, one can attach the data validation’s failed records file as well.

On Failure Operations for Errors

Chained On Failure Operations get executed when a Jitterbit Operation Error or a Raise Error event occurs. The Raise Error event gets triggered within a Script with the RaiseError() function or with a field or row’s data validation.

Like On Success Operations, they are added using the Operation’s context menu. On Failure Operations are unique in that they get executed via the Raise Error event or scripted after a failed RunOperation(). Within, they can do anything any other operation can do but most are simply Scripts that call a Notification Method shown in our previous installment. They can also have other operations chained to them via the Operation’s On Failure/Success context menu items.

Counters and Error Threshold Checks

Along with potential Operation log entries or termination with On Failure Operations, one might want the same during data validation if a threshold amount of errors is reached. The key here is using counter variables.

A dedicated counter handler script –that might call Handle Event– could be used. The script might notify about the threshold once-only, rather than repeating with future checks. Below is an example of this dedicated script.

  1. _counter_name = _1;
  2. _counter_threshold = IfEmpty(_2, “n/a”); // optional
  3. _always_notify = Bool(IfEmpty(_3, true)); // optional
  4. _increment = Int(IfEmpty(_4, 1)); // optional
  5. _threshold_msg = IfEmpty(_5, “$”+counter_name + ” reached threshold of “ counter_threshold); // optional
  6. Set(_counter_name “__WARNED@ge” + _counter_threshold,
  7. Bool(IfEmpty(Get(_counter_name “__WARNED@ge” _counter_threshold)false)));
  8. Set(_counter_name, Get(_counter_name)+_increment);
  9. If(IsInteger(_counter_threshold) && Get(_counter_name) >= Int(_counter_threshold) && (_always_notify==true
  10.   || !Get(_counter_name “__WARNED@ge” _counter_threshold)),
  11.   result=RunScript(“…/Handle Event”, 4, “Threshold Check”, _threshold_msg, false, “WARNING”);
  12.   Set(_counter_name “__WARNED@ge” _counter_threshold, true);
  13. );
  14. Get(_counter_name); // return new value, if needed

The above script takes advantage of the Set() and Get() functions to dynamically define and reference customized global variables by string (rather than using the $ notation). In particular, the counter name and threshold value is used to define the global variable name and set this variable to indicate whether a warning was issued.

By default, Jitterbit will thread Operations for performance depending upon available server CPU’s. (Threads are a way for an Operation to divide itself into two or more simultaneously running tasks.) One possible side effect is miscounting counter values. To correct for threading, use the InitCounter() function in an Operation’s setup script.

As an example of using the Handle Counter Script, occasionally you might run into an ill-mannered endpoint that may require multiple connection attempts. Use the Handle Count script within an On Failure Operation to repeat an Operation until some threshold is reached. For example, an On Failure Script Operation might use the following to attempt an Operation three times.

  1. Count = RunScript(“…/Handle Counter”,“myCounter”, 3, false);
  2. If($myCounter__WARNED@ge3, // take care with this dynamic variable name
  3.   RaiseError(“Operation Attempted 3 times, each resulting in failure”)
  4. ,
  5.   RunOperation(…)
  6. );

Restart-ability

Now that you’ve implemented the event and –more importantly– error capturing, you may want your design easily restartable from any point-of-failure during execution. After correcting the design or data error, the Operations can be optimally restarted if you’ve designed your Operation chain so that it can be restarted from any point in the chain.

You’ll notice the use of IfEmpty() in the above Scripts and, more importantly, the mentioning of Operation setup scripts. Setup scripts are the best method to enable restarting your operation chain(s) at various points in the data pipeline chain. Source parameters, counters, and other variables can be (re)initialized with desired changes in IfEmpty() constructs if these variables were defined and assigned values upstream in the Operation chain. That is, changes to the IfEmtpy() constructs won’t impact the normal, complete Operation chain!

If Operation results are stored in, for example, initial, clean and load-ready staging directories (and archived as needed), one can rely on and continue an Operation chain. You can start from the next Operation’s setup script rather than starting the execution from its’ beginning. Directories in Jitterbit’s Temporary Storage make for good locations as these are purged automatically (by default, every 24 hours).

Summary

The above and previous practices with Jitterbit empowers an integration design that is flexible and robust.  Best practices include:

  • Differentiate generic from specific event notifications.

  • Use setup (initialize) and wrap up (summarize) Scripts.

  • Use InitCounter() for counters within threaded operations

  • Design for restart-ability

Try the above Handle Count Script with invocations like the following.  After execution, view the Operation Log.

Increment a counter called “myCounter” by 1 (default) with a threshold check at a value of 2.

  1. RunScript(“…/Handle Counter”,“myCounter”,2,false);
  2. RunScript(“…/Handle Counter”,“myCounter”,2,false); //warn once
  3. RunScript(“…/Handle Counter”,“myCounter”,2,false);

Leave a Reply

Please log in using one of these methods to post your comment:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s