Blog

Quick Tips and Tricks : Low Code Data Retrieval API using Function as a Service (FAAS)

Posted on by
Quick Tips and Tricks

A user requires a production grade api that can be used to query data stored in an enterprise database quickly. In most fast paced environments, this scenario calls for implementing a serverless function using OpenFAAS running on a bare metal , datacenter or cloud based Kubernetes clusters.

Using the OpenFAAS command line interface (cli) a function is created using a python3-debian template. Using python libraries the function queries the database as detailed in a separate DevStory. The function, deployed to a Kubernetes cluster, is exposed as an endpoint behind an NGINX reverse proxy within a datacenter cluster or API Gateway on AWS.

Clients manually or programmatically call the function to retrieve data from the enterprise database. This architectural pattern has been used frequently on a trade floor to provide a secure, IT policy compliant, auditable and production scale api at high velocity and low code. It allows developers to be extremely responsive to user needs in fast paced environments.

Factoryzetm Functions as a Service provides an implementation of OpenFAAS deployable to a bare metal K8s cluster, Tanzu and Open Shift clusters in the data center and EKS, AKS and GKE in the cloud.

Energiewerks 3D – Discover, Design, Deliver : Part 1

Posted on by

Emphasis of the Energiewerks 3D process, on delivering small incremental value adding functionality, from concept to delivery, with continuous testing by subject matter experts, product owners and implementors at every stage of the delivery process, ensures no large scale failures occur. A “fail fast and often while working at pace” approach, allows any inevitable failure to meet persona’s needs to be caught early, and corrected quickly, with limited rework and cost overruns.

Sanjay Gupta – Product Manager

What is Energiewerks 3D ?

Discover, Design and Deliver system functionality repeatably, failing fast in some cases, yet delivering small increments of value to personas continuously, is the hallmark of Energiewerks way of building complex systems.

Discovery processes involves extensive user interviews to define personas and their needs, Making, testing and fine tuning prototypes using opinionated patterns is the design phase and implementation of prototypes using opinionated idioms and architectures delivered on various platforms ( mobile, web, desktop, cloud and devices) coupled with operations and support is third D.

Energiewerks 3D borrows from Shape Up, Design Thinking, SAFe, patterns and idioms, all time boxed and repeatable resulting in successful delivery of incremental system functionality.

Energiewerks teams are located across continents. Subject matter experts in the US, Europe, UAE and India. Product managers and designers in the US and India. Developers in several locations in the US, Europe and India. Traditional “in the same room” techniques such as Design Thinking, Shape Up, Agile development / Scrum did not work for us. COVID-19 and work from home mandates, required online collaborative tooling. This prompted the development of the Energiewerks 3D methodology.

Energiewerks 3D incorporates tooling such as BaseCamp, Miro, Design Thinking templates, TimeTimers, Zoom rooms, Bitbucket, Pipelines, Kubernetes, Cloud environments, Lambda/Kappa architectures, Mobile/Web/Desktop frameworks, multi-model databases and micro-services in an opinionated solution to deliver products, projects and services at a rapid pace at low cost and high velocity.

Subsequent posts will describe our successful delivery approach using 3D – Discover, Design and Deliver

Quick Tips and Tricks: Production-grade Trade/Deal Capture in a day using functions as a service.

Posted on by
Quick Tips and Tricks

Scenario :

Complex deal type, contributing large positions to the commodity trading firm’s proprietary portfolio needed to be captured. ETRM system (CXL) support would take months. The opportunity was now.

Quick Tips and Tricks :

To support the business opportunity and capture deals, we used Factoryzetm Functions As a service deployed in a secure, datacenter deployed, bare metal, Kubernetes cluster to provide an enterprise-scale production-grade solution in three hours ( discovery, solution design to delivery into production).

The deal payload was packaged as a JSON string conforming to a JSON schema stored in an Apache Schema Registry. This JSON deal is sent to a function deployed using horizontal pod autoscaling to write to the Commodity Logictm database.

The function comprises a handler written in Python ( support for GO, Typescript, and several other languages) that validates and parses JSON payload. Then writes deal to MongoDB collection in the Commodity Logictm document database collection associated with deal type. Subsequently, a message is sent to Apache KAFKA topics, where Commodity Logictm consumers process the deal for risk, compliance, surveillance, positions, near real-time P&L, and other needs.

Quick Tips and Tricks : Send json with spaces using jo and curl on Mac

Posted on by
Quick Tips and Tricks

Scenario :

A developer wants to send json containing spaces in the values to a FAAS function. The payload for the function is a json with three name-value pairs : commodity, role_name and subsidiary_name. The values may contain spaces. This trick is one way to send to the function using curl from the command line. Prevents the curl function from failing to understand

Quick Tips and Tricks :

brew install jo
c="NG"
r="Settlements"
s="ABC, LLC"
apiurl="http://localhost/function/somefunction"
curldata=$(jo commodity="$c" role_name="$r" subsidiary_name="$s")
curl -d "$curldata" "$apiurl"

Weekly Ensight : Clean up on WordPress aisle 6

Posted on by

An analyst, using Energiewerks Ensights integrated with the WordPress blogging platform, to disseminate fundamental analysis reports and studies over the last couple of years, wanted to delete the stale posts and associated media attached to the post to free up space on the blog.

In total there were 18k+ posts and about 92k+ media items ( png, pdf, jpg, xls) files that needed to be deleted from the blog.

Using the Energiewerks Ensights toolkit, this was accomplished using the following python code snippet. 

from utils.WordpressUtilities import WordPressUtilities
from datetime import datetime,  timedelta

if __name__ == '__main__':
    wp = WordPressUtilities()
    wp.configure()
    wp.deleteBlogPostsOlderThanDate(datetime.now() - timedelta(days=30))
    wp.deleteMediaItemsOlderThanDate(datetime.now() - timedelta(days=30))

Under the covers, Energiewerks Ensights uses the python wordpress xmlrpc library to

  1. instantiate a WordPress client using the blog URL, username, and password configured within Energiewerks Ensights
  2. then uses the client rpc call GetPosts providing a dictionary of parameters that contains two parameters – increment and offset, to retrieve a set of blog posts.
  3. iterates through the set of blog posts and delete blog posts that meet the date criteria specified ( in this case anything of 30 days or more vintage )
  4. Energiewerks Ensights then iteratively gets 1000 media items using the rpc call GetMediaLibrary on the media object.
  5. Iterates through the 1000 media items returned at a time, selects media items that meet the date criterion and subsequently the category of items from the composite name of the item, and delete the media item from the blog.

Energiewerks Ensights provides simple yet powerful tools to simplify day to day activities of fundamental analysts on the energy, commodity, equity, currency, and equity trading desks of some of the largest trading firms globally.

Posted in Uncategorized
%d bloggers like this: