When chaining Operations to execute consecutively, you might use Targets to “stage” data for the Source of the next Operation (or for reading or appending multiple times during various chained Operations). In many cases, these ‘staged” files are only needed during the Operation execution (or temporarily). This is the purpose of the Jitterbit Source or Target Temporary Storage component.  In this short post, I share some best practices for and nuances of the Temporary Storage component.

Within each default Jitterbit Operation, you can execute one primary activity – Transformation, Web Service Call, etc. You may call other Operations from within Scripts or Transformation maps (RunOperation) and this can be useful at times.  It’s not a good idea to use RunOperation() within a Transformation map if many record instances are transformed for performance reasons. With this best practice design approach, Operations “chained” (On-Success/Failure) to downstream Operations can be designed for unit or regression testing.

By default, “temp Storage” files (or temp files) exist for 24 hours on a Jitterbit Agent in a jitterbit/TemporaryStorage folder typically within a Server’s temporary storage location (/tmp or C:\Windows\temp).  (The lifetime and Server’s location of these files can be configured per Agent)  When using Agent Groups (or the Cloud Agent), these files will only exist on the Agent that executes the Operation chain.

When sourcing a previous Operation’s Target, you have the option of using the chained Operation’s Source context menu “Use Source from Previous Operation” (perhaps best read as “Use Previous Operation Target as Source”).  This might simplify a final design but I like to use temp files explicitly so I can easily list, log and view content during development efforts.

You can create mirrored Source and Target components using the Target or Source component context menu item “Copy to New Source” and “Copy to New Target” within the Project Tree.  You will want to be sure to change “both” components when you modify one component’s definition.  (One exception might be a utility Source temp file — see below)

With the understanding of where and how long temp files exist, you can use them as you would computer Random Access Memory (RAM). That is, use as often as needed (on demand) and give each a unique name so they can be easily referenced after an Operation is completed.  (If you configure their lifetime accordingly, this can be useful when troubleshooting production issues.)

I use a Script to generate a unique name for my temp files.

// NameGen: generates a unique name containing timestamp
purpose = IfEmpty(_1, "UNKNOWN_PURPOSE"); env=GetEnvironmentName();
// expect env+purpose+yyyy-mm-ddTHHMMSS_zzz or 
// env+UKNOWN_PURPOSE+yyyy-mm-ddTHHMMSS_zzz if purpose is undefined
// where timestamp is the Jitterbit Agent's Time Zone
isoTime = RegexReplace(Now_(), //similar to ISO time: yyyy-mm-ddTHHMMSS_zzz (zzz is microseconds)
  "(\\d{4}-\\d{2}-\\d{2}) (\\d{2}):(\\d{2}):(\\d{2})\.(\\d{3})","\\1T\\2\\3\\4_\\5");
name = env+purpose+isoTime; // decorate and return name

I’ve seen the use of the Guid() function to make temp file names unique but this defeats their real value – quick recall & access to the content as needed.  In the above naming convention, the use of a timestamp that includes microseconds can (effectively) guarantee uniqueness but also makes the files sort-able (and easily identifiable in a list –by the 3-digit microsecond).

I use something like the following in an Operation’s Setup Script (a leading Script of an Operation – a best-practice) so each temp file name is logged.

$%fileBasename = RunScript(“Scripts/Handlers/NameGen”,”QueryPassThru”);
RunScript(“Scripts/Handlers/Event”,6,”Target filename”,
  $%fileBasename+”.xml (”+GetServerName()+")");
//See my Capturing Events & Errors for Effective Analysis (Part 1) post 
//for the Event Script

Note the use of GetServerName() above so we know which Agent is hosting the temp file.

Within the Target definition, I’ll define the Target’s Filename form field value as [%fileBasename].xml. I might use the Path form field to park temp files in unique directories if advantageous.

Within Jitterbit Studio, you can display file lists and their content in a few ways. One way –and an advantage of mirroring a Target— is to define the Source’s default filename using {*}. (For example, [%fileBasename{*}].xml)  Now I can use the Test Connection button on the Source component and see all the temp file names sorted by time.  If I’m using this Source exclusively for listing or displaying temp file content (and not within an Operation), I might rename the Source to indicate this – QueryPassThru (utility) where the corresponding Target is called QueryPassThru and/or put this utility Source in its own Project Tree folder.

I also like to use Jitterbit’s Script Pad (shortcut: F4 key) for the following.

  1. To list files:
    FileList(“Sources/QueryPassThru (utility));
    or
    FileList(“Sources/QueryPassThru (utility)”,””,”*.xml”)
  2. To display file content:
    ReadFile(“Sources/QueryPassThru (utility))”,
    ”DEVQueryPassThru2018-01-02T030405_678.xml”)

Note, filename value is found in the logs if suggested use of RunScript(..Event..) is in your Operations Setup Script.  In all cases, I use Ctrl-C and Ctrl-V for quick Cut & Paste.

There are a couple caveats to using Test Connection, FileList() and/or ReadFile() above in Script Pad. These caveats only apply to these file use cases and do not apply when Operations are executing.

  1. If using an Agent Group with multiple Agents or Jitterbit’s Cloud Agent, the necessary TemporaryStorage directory on the Server may be on another server so you could likely receive an empty array for #1 (or Test Connection) and/or a “File not found” error for #2 above. Re-testing may switch to another Agent.
  2. When using Script Pad, you are actually using the Agent in a “test mode” so the Script output is limited to some 50K bytes. If you use ReadFile() on a file with a size greater than 50K bytes, ReadFile() will fail with an error indicating this.

If one or both of these cases prevent effective use, one alternative might be to define and use an Email Message and a Target of type Email to email the temp file to you when executing the Operation. See Creating an Email Message and Creating an Email Target for details.  Here, you would have to change your Operation by replacing your temp file Target with the Email Target as the alternative to using Script Pad.

Another approach might be to WriteToOperationLog(ReadFile(..)) in an Operation’s wrapup script (final Script/component in an Operation — another best practice) but the log will truncate after the same 50K byte limit… but you will be able to see the first 50K bytes!.  You will want to execute FlushFile() prior to the WriteToOperationLog(ReadFile()) above so the file is available for reading within the same Operation.

Swapping Targets, Sources, and even Transformations are procedures used in unit testing and regression testing Operations and Operation chains.  This is a topic for a later post.

Leave a Reply

Please log in using one of these methods to post your comment:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s