Functions
A Function node is a procedure that can filter or transform data. It has one or more inputs that can be attached to connectors, to datasets, or to other function nodes. It also provides one or more outputs that can be attached to the inputs of other function nodes or can be used to generate the data stored by the dataset. There are different types of functions, each of which has inputs and outputs of specific types (JSON, CSV, and so on). A function node can be configured to specify the precise details of the data transformation that is to be carried out.
Functions are used to construct the data flows for Datasets and Adapters. To add a function to the data flow then either drag a specific function from the palette on the left of the studio canvas, or click the Add Node icon in the Dataset / Adapter title-bar to insert a generic Script function.

After positioning a function node on the studio canvas, the following operations are available by clicking on the corresponding icon at the top of the node.
JavaScript
Handful of the functions - and in particular the generic Script function node - allow you to enter raw JavaScript. The inputs to the node are accessible via the inputs object, and the function outputs are provided by means of an output object. For example, if the node has inputs lights and status then these can be accessed in the script as inputs.lights
and inputs.status
.
Regarding translation, the following languages are supported and accordingly the desired translation should be provided within the following JSON object keys:
English: en
Italian - it
Hindi - hi
Dutch - de
Arabic - ar
Russian - ru
Japanese - jp
Slovensky - sk
For example,
outputs.content = { "sk": { continent: "kontinent", country: "krajina", state: "stav", district: "okres", city: "mesto", colony: "kolónia", area: "oblasť", lane: "pruh" } };
This should be provided as input to the translations input in the Localization function.
Some more examples of JavaScript are as follows:
Sort function
Sorts the array of properties in ascending or descending order.
input.sort(function(a, b) { return new Date(b.Date) - new Date(a.Date); });
Underscore to camelCase format
Changes the object keys from underscore to camelCase format.
const renameKeys = obj => Object .entries(obj) .reduce((acc, [key, val]) => { const modifiedKey = key.replace(/_([a-z])/g, g => g[1].toUpperCase()); return ({ ...acc, ...{ [modifiedKey]: val }, }); }, {});
Unique Properties Filter
Removes the duplicate values thus filtering them out so that only the unique values are present in the array.
let fill = Array.from(new Set(data.map((res => res.type))));
In addition to the native JavaScript scripting language, the Quantela platform supports the following extensions:
console - function scripts can use
console.log
,console.info
andconsole.warn
. For example:console.log("Input Object", {a: 1, b: 2})
R - custom functions can take advantage of the whole RamdaJS functional library. For example:
Rename keys
Renames the keys to a user-friendly format.
const renameKeys = R.curry((keysMap, obj) => R.reduce((acc, key) => R.assoc(keysMap[key] || key, obj[key], acc), {}, R.keys(obj)) ); R.map(renameKeys({ "": 'Status', 'Ward/Mandal': 'Ward', 'Hospital District': `hospital`, 'Zone/District': `Zone` }), input);
Group by property and add results on the specified property
Groups by a property and then adds the results based on the specified property.
const sumBy = prop => vals => reduce( (current, val) => evolve({ [prop]: add(val[prop]) }, current), head(vals), tail(vals) ) const groupSumBy = curry((groupOn, sumOn, vals) => values(map(sumBy(sumOn))(groupBy(prop(groupOn), vals))) )
Merge multiple arrays into one array
Merges multiple arrays into one single array.
fn = R.pipe( R.map(R.indexBy(R.prop('property'))), R.reduce(R.mergeWith(R.merge), {}), R.values ), newArray = fn([arrayOne, arrayTwo, arrayThree])
Merge arrays based on the index property
Merges arrays based on the specified property.
R.map(x => { data = R.values(R.mergeWith( R.mergeLeft, R.indexBy(R.prop('property'), input), R.indexBy(R.prop('property'), x) )); }
Filter by a property and find the length of the result using 'compose' function
Filters based on a property and then finds the length based on the result using the compose function.
R.compose(R.length, R.filter(R.pathEq(['property'], 'value')))(input);
Group by any property
Groups by the specified property.
R.groupBy(R.prop(property));
Sums data based on properties
Adds the data based on the specified property.
R.sum(R.map(item=> item.property,data));
Reject or omit few records
Rejects or omits the records which satisfies the condition.
R.reject(x => x.property == 'value', data);
Data modification using the 'map' function
Data transformations using the 'map' function.
let output = R.map((device) => { For ex: // Converting Unix time to ISO string. device.last_updated = new Date(device.payload.d.t * 1000).toISOString(); return device }, deviceDetails)
Unique by property
Filters the unique values based on the specified property.
R.uniqBy(R.prop("property"), input);
Multilevel Group by
Groups by using multiple 'group bys'.
const groupByLetterNumberBool = pipe( groupBy(prop("letter")), map(groupBy(prop("number"))), map(map(groupBy(prop("bool")))) );
Pluck by given property
Plucks the data by the specified property.
R.sum(R.pluck('property')(data));
Key-Value Pair Converter
Converts two separate keys to a single key-value pair.
{ "name": "", "value": "" } == > { name: value } R.pipe(R.indexBy(R.prop('name')), R.map(R.prop('value')))(input);
Papa Parse - the papaparse JavaScript parser can be used to parse CSV strings.
moment - the moment.js library can be used to parse, validate, manipulate, and display dates and times. For example:
Time conversion to specific time zones
Converts the timestamp from UTC to the specified time zone.
moment.utc(timestamp).tz('America/New_York').format('YYYY-MM-DD HH:mm:ss z');
JSONata - jsonata expressions can be used to query and transform data. For example:
Group by property and sorting
Groups by the specified property and then sorts it.
$${$string(`property`):{"Zone":count}} $${$string(`property`): $.{ "status": status, "timestamp": timestamp }^(> timestamp)}
JSONata Filter function
Filters the property.
$filter($$, function($v, $i, $a){($v.property!=null)}
Filter based on count and variables
Filters the data based on the count and the specified variables.
($my: = $$ { `Patient Name`: { "a": `Patient Name` } }; $patientName: = $my.*[$count(a) > 1].a[0]; $$[`Patient Name` in $patientName])
JSONata expression execution
Dynamic execution of an expression.
let expr = jsonata("$$.*"); expr.evaluate(data) or jsonata("$$.*").evaluate(data);
Group and sum of a column
Groups and adds the results based on the specified property.
$$ { key: $sum(property) }~ > $each(function($v, $k) { { "key": $k, "key": $v } })
Merge all objects in an array
Merges all the objects in an array.
$zip( * .$keys(), *.*) { $[0]: $[1] }
UTILS.datasets - we can use UTILS to invoke datasets using the syntax
UTILS.datasets(datasetName, payload)
. For example, if there is a dataset called 'Get Light by Id' and it takes vendor_name and light_id as inputs then we can invoke this as follows:let light = await UTILS.datasets("Get Light by Id", {vendor_name: "Bajaj", light_id: 12345})
Error Handling if adapter/dataset returns an error or an empty response
let data = await UTILS.datasets("data-set-name", { "key": value }) if (data.error) { outputs.content = [] } else { outputs.content = data; }
UTILS.connectors - we can invoke connectors using the syntax
UTILS.connectors(connectorName, input)
. For example:let my_connector = await UTILS.connectors( "Connector Name", {"payload": "your input payload"} )
UTILS.functions - we can also invoke any built-in functions within our scripts using the syntax
UTILS.functions(functionName, input, args)
. For example:let light = await UTILS.functions( "JSON To CSV", [ {a: 1, b: 2},{a: 3, b: 4},{a: 5, b: 6},{a: 7, b: 8} ], [",", "\r\n"] )
Note that all functions, connectors and datasets methods are asynchronous, so you will need to either catch Promises or use await while invoking.
UTILS.redis - we can store or retrieve data from Redis using the following:
UTILS.redis.setCacheValue - we can store data in Redis using the syntax
await UTILS.redis.setCacheValue(keyName,value,expiry,keyPattern)
where 'expiry' and 'keyPattern' values are optional. The 'expiry' value will be in seconds so for the given number of seconds it will be persisted in Redis and after that will get flushed.However, if you don’t want to set any expiry you can use the syntax
await UTILS.redis.setCacheValue(keyName,value)
.UTILS.redis.getCacheValue - we can retrieve data from Redis using the syntax
let data= await UTILS.redis.getCacheValue(keyName)
. If you have used 'keyPattern' in UTILS.redis.setCacheValue then use the syntaxlet data= await UTILS.redis.getCacheValue(keyName, keyPattern)
.
UTILS.currentUser - we can get the logged in user's info and roles using the syntax
let data= await UTILS.currentUser()
.UTILS.flowContext - we can get the current flow context using the syntax
let data= UTILS.flowContext
.mock - functions can use Mock.js to generate simulation data.
turf - function scripts can use Turf.js for geospatial analysis.
through2 - through2 can be used as a pipe for modifying or transforming streams.
X2JS - the X2JS library is a convenient tool for converting between XML and JSON objects.
logger - used to log messages from the custom functions.
cheerio - cheerio can be used to parse HTML. For example:
const $ = cheerio.load('<h2 class="title">Hello world</h2>') $('h2.title').text('Hello there!') $('h2').addClass('welcome') $.html()
crypto - crypto is an encryption utility for crypto standards. The syntax is
crypto.HmacSHA1("Message", "Key")
.google - Google's official node API feature. Google Spreadsheets is one of the subset API. You can refer to it here. For example, to ask for permissions from a user to retrieve an access token, you redirect them to a consent page. To create a consent page URL:
const oauth2Client = new google.auth.OAuth2( YOUR_CLIENT_ID, YOUR_CLIENT_SECRET, YOUR_REDIRECT_URL ); // generate a url that asks permissions for Blogger and Google Calendar scopes const scopes = [ 'https://www.googleapis.com/auth/blogger', 'https://www.googleapis.com/auth/calendar' ]; const url = oauth2Client.generateAuthUrl({ // 'online' (default) or 'offline' (gets refresh_token) access_type: 'offline', // If you only need one scope you can pass it as a string scope: scopes });
FeedParser - FeedParser is used to parse RSS feeds and can be initialized using the following syntax:
//initialise FeedParser var feedparser = new FeedParser([options]);
Preview a function node
To preview the data output by this function node, click on the Preview icon:
If there are any unresolved inputs to the node then you would be asked to enter a value for these inputs. Depending on the format of the data, it can be previewed in different ways:
Select the JSON option to view it as a structured sequence of attribute-value pairs. You can click on any field in the structure to expand it and see the lower level data, and click again to collapse it. Alternatively click the Expand All Fields and Collapse All Fields buttons which apply to the whole JSON structure.
Select the Text option to view the data as unstructured data.
Select the Map option to view the data on a geographical map. This option is only applicable to GeoJSON data. Each location would be displayed on a world map so that you can verify that the format and content of the data returned by the provider is correct.
In the Input option, the data which was entered in the Default Data field of Node Inputs in the Configuration would be pre-populated here without needing to enter the data again.
Also, you can click on the Full Screen icon to maximize the panel in the preview so that there is more real-estate to work on changes.
Configure a function node
To view or edit the configuration or properties of the function node, click on the Config icon:
The configuration options in the Configuration tab depend on the type of function and are described in detail in the corresponding topic.
Click on the Properties tab to review and update the following properties:
Property | Description |
---|---|
Node name | A descriptive name for the function. |
Tags (optional) | One or more descriptive tags for the function. |
Version (optional) | Version information. |
Icon (optional) | An icon to be displayed for this function in the data flow studio. |
Press the Proceed button to save any changes.
You can click on the Full Screen icon to maximize the side panel in the node configuration so that there is more real-estate to work on scripts.
Also, you can click on the Help icon to assist in using the different connector types, functions and providers.
Click Submit and Generate Schema to configure the function node with schema or Submit to configure the function node without schema.
Copy a function node
Click the Copy icon on the provider node to easily copy and paste a particular node or a set of connected nodes and it can be reused in the same or different dataset or adapter.
Delete a function node
To delete a function node from the canvas, click on the Delete icon:
Note: there is no way to undo this action.