Red Erik

Milestones

Milestones Five - m5

Erlang-Red Message Tracing

Using the introspection node package, implemented message tracing of Erlang-Red messages.

Note: that there are two types of messages in Erlang-Red - the Erlang type of messages and the Erlang-Red types of messages. A disadvantage of using a message passing based programming language to implement a message passing programming paradigm like flow-based programming! Either way, this feature traces Erlang-Red messages (the kind passed between nodes) and not Erlang messages (the kind passed between processes).

A second point - related to the feature - message tracing can be activated at any time and does not require deploying the flow to the server, i.e., message of a live running server are being traced.

A third point is that this feature was originally implemented for Node-RED and all I did was port the backend to Erlang. Now I have the same insights that I have in Node-RED but in Erlang-Red.

One more point: this feature can definitely flood the flow editor into submission. Use with care. The server should be fine since all it is doing is sending messages out on a web-socket via an exclusive process for the web-socket connection.

A fifth point: in the message tracing, the status of the node is abused to have a “msg received” status appear. That message (only in Erlang-Red) now contains the process id of the underlying Erlang process for the node.
Nodes for gen_event & gen_statem behaviours

Erlang-Red has become more Erlang-like by not only implementing the supervisor behaviour but also the gen_event and gen_statem behaviours. Both of these make use of the module node. The module node can be used to implement complete Erlang modules in Erlang-Red. These modules can then be used to define the behaviour of the gen_event and gen_statem nodes.

The event handler is completely dynamic with modules being added as handlers at runtime, the state machine is static with a module being added at design time. A state machine handler can however dynamically replace the handler if so desired.

The event handler will need some more TLC to make it static and dynamic. The overhead of configuring the event handler at runtime is strictly speaking not necessary. The corresponding test cases: eventhandler and state machine.

When I started out on this project (all of 2.5 months ago!), I intended to implement Node-RED features and nodes in Erlang. Now having created the Erlang-only nodes which implement these behaviours, I’m thinking that wouldn’t it be nice to create Node-RED equivalent nodes. Half the work is done: these nodes are all packaged into an Node-RED node package and can be installed into Node-RED, only there they do nothing. However having a supervisor to ensure that things get restarted in NodeJS can be, perhaps, just maybe, also useful.

The assert nodes that I created actually do work with Node-RED and Erlang-Red, so these are the first ever nodes that are cross platform.

The deeper philosophical point I’m trying to make here is that a visual flow-based programming approach can lead to cross pollination of programming ideas. It can also lead to interesting learnings around programming paradigms. But for those sceptical about visual programming and the benefits of it - especially how can a “serious” application be developed visually in a browser - go back to using punch cards. We came from giant mechanical calculators, went to punch cards then to keyboards and now we’re stuck at Artificial Intelligence and typing in prompts to get code we might or might quite understand. Amazing, awesome, I’m so excited - truly. As programmers, we haven’t started to use the true benefits of visually constructing our programs and applications, instead we claim that the keyboard is the only instrument for creating instructions for a piece of metal with many wires and many CPUs. Somehow it does remind me of knocking two stones together to make fire and the amazement when someone comes along with an Artificial Stone in the form of lighter.
Improved documentation using the Flow2UML node

I dislike writing technical documentation. I consider code enough documentation - make the code read like a good murder mystery and presto it’s a clear whodunnit. But Erlang code is damn compact and even I don’t understand why and what I did two months ago. Not good. The second thing I dislike is creating flow diagrams - either visually or textually. But there is no way around it. If I want to be able to understand how things work in six months time, I have to create documentation.

Mermaid seems to be the tool of choice for such things so back in the good days of Node-RED, I created a node package for converting flows to Mermaid-UML-like diagrams, i.e. a flowchart. That flowchart, being defined in Mermaid syntax, can be embedded and displayed in Node-RED flow documentation - neat! But because I wanted extra features, I created the mermaid-flowchart node for creating more complex flowcharts.

Some examples are here, here and here. And inside Erlang-Red:

But creation is one thing, usage is another - how I do use these diagrams? Well Node-RED supports embedded mermaid diagrams and by extension, so does Erlang-Red:

That’s the nice thing, the documentation of a flow is embedded in the flow and goes wherever the flow goes. I created red-erik.org to host that documentation in a webpage. So for example, the gif above is online at red-erik.org.

For any Node-RED readers: the original flow2UML package also contains the new mermaid flowchart node. So it’s possible to create these complex flowcharts in Node-RED also.
Timeout support for function and link-call nodes

This might seem minor, but both required a complete re-think of the Erlang architecture. The function node now creates up to two Erlang processes per Erlang-Red message received. The Erlang architecture for the function node is documented and describes why this needs to be.

Timeout for the link-call node was much simpler since it can be implemented using the Erlang-Red message object. Basically an erlang:start_timer is set when the Erlang-Red message is passed onto the link-in node. If the message comes back before the timer is triggered, the timer is cancelled. If on the other hand, the message isn’t received, the timer is triggered and a timeout error is raised. In creating the unit test for this feature, I realised that the message - once it arrived - is still passed on! That is Node-RED behaviour, so Erlang-RED copied that. IMHO that doesn’t make sense since the message timed-out, however I can understand that if the message makes it back, then why lose the data - obviously took a long time to get it, so don’t waste it!
Better documentation describing the Erlang architecture

I am starting to be happy with the Erlang architecture so I’ve started to document it. Starting with the supervisor node, a technical description of how it interacts with the Erlang code base to implement Erlang-Red node supervision using Erlang process supervision. There is technical description of how the link call node works.

Once I’m through with documenting everything, I will probably end up modifying everything again - programming is truly a Sisyphusian activity.

Milestone Four - m4

Supervisor Node

One of the most important concepts of Erlang is the supervisor behaviour. It defines an elegant approach for monitoring, restarting and stopping processes that are unhealthy. It thus allows for complex yet stable architectures to be created in Erlang. Erlang-Red now has its very own supervisor node to provide the same behaviour but visually.

The supervisor node is fully feature compatible to the supervisor behaviour and the node provides a simple mechanism of selecting nodes to be supervised and also ordered. Supervisors have a specific ordering of processes and this is also supported by the supervisor node by being able to sort nodes, each node being a process.
Added Pencil to Flow-editor

Might seem silly but it is important for documentation. Test flows can now be viewed over at Red-Erik.org (hosted via flowhub.org), so that test flows can now be better documented via “pink link technology”, for example, flow one, flow two and flow three.

Pink links highlight nodes and groups in the flow diagram making it simpler to explain flows within the flow description. These pink links are a html stanzas, e.g.: <a class='ahl-group-only' data-ids='0bf41347dde34db0'>name</a> - which are a real pain to write by hand. Node ids had to be copied, the class name typed out, node ids found from the flow editor … blah blah :(

Also important to note is that flow documentations is found via the flow tab:

This means that flow documentation lives with the flow code and is displayed both in Erlang-Red (within the flow editor) and at flows.red-erik.org. (All acknowledgement to the Node-RED team for putting this in the editor, I am just piggy-backing their work and extending it to my needs.)

Back to the pencil: I now select a node or group from the info panel and click on the pencil and I have my HTML stanza:

The nice thing is that the info panel is usable while editing the documentation for a flow, so the pencil can always be used to create the HTML stanza for the pink links.
FlowHubPull node: dynamic flow code loading

FlowHub.org is my attempt at creating a visual code hosting platform. Part of that is defining interdependencies between flows. For me a flow is the flow tab (for others a “Flow” are all flow tabs). I like to think in flow tabs and creating functionality that fits into a single flow tab.

Flows can be linked via the link nodes and I do this very often. But what is missing is the loading of flows into Erlang-Red dynamically. So for example, this test flow first triggers the flow hub pull node to load another flow into Erlang-Red. It then triggers the test which contains a link to the other flow.

The FlowHub pull node is currently limited to the flows contained within the Erlang-Red test-suite but there isn’t any reason - other than laziness - why this cannot be extended to include other sources of flow code.
Remove catch around function node

With the introduction of a supervisor node, I removed the exception handling around the function node - I actually had to do this to get the function node to fail (was using a divide by zero error to test the supervisor node).

But it made me realise that Erlang is a lot different to NodeJS when it comes to exception handling. Erlang processes are lightweight and designed to fail, so let them fail becomes the intention. NodeJS on the other hand wants to handle all exceptions and not let the system get unstable, so there is much use of the try-catch-final pattern within NodeJS code.

Erlang heals systems by restarting processes if they fail. It is assumed the system is stable until something fails. Perhaps the complete inverse of what NodeJS does.

By removing the catch block on the function node, I’m moving away from the NodeJS way of doing things and moving towards a more Erlang-like approach.

Ironically the problem with catching errors in something as generic as the function node (which takes straight up Erlang code and executes the code) is that: how to deal with the exceptions? All I was doing was pushing it to the debug panel … but I could do that far more descriptively by using a catch node connected to a debug node.

Its only a minor change yet it represents a major step forward for Erlang-Red because it allows Erlang folks to think in terms of Erlang when using Erlang-Red.

Additionally Erlang-Red now has two nodes for error handling: catch node for catching exceptions and the supervisor node for restarting failing processes. As they say in German: twice holds better!
Erlang, Elixir, BEAM?

Tried but failed to get Erlang-Red compiling in an Elixir environment. It does so but then Erlang-Red won’t compile in an Erlang environment. I decided to focus on the Erlang path - after all its Erlang-Red not BEAM-Red. My experience is not up to getting Erlang-Red to compile for both Elixir and Erlang, perhaps someone else will make in-roads there.

Milestone Three - m3

Elixir code constructively includable.

Since Elixir also runs on the BEAM VM and there are some libraries that aren’t available in Erlang, I made an effort to integrate third party Elixir libraries - in a structured manner - into the codebase of Erlang-RED. Many thanks to @filmor for the tip of using their exerl plugin. Works like a charm!

This effort lead to two new nodes using Elixir codebases: the markdown node uses earmark and the csv node uses nimble_csv. Nimble might be removed again since there are pure Erlang CSV libraries - we’ll see.
Erlang JSONata has moved out

Moved the codebase for the JSONata parser out into its own repository. I wasn’t actively extending it and moving it out will allow others to extend and add to the JSONata functionality supported in Erlang-RED. It’s always sad when the children move out but its for the best!
Function node in Erlang!

One of the more important nodes in Node-RED is the function node because it allows for creating Javascript code, i.e., dropping down to high-code instead of low-code. This can be helpful for adding functionality to a flow that is not covered by any available node. What the function node offers is syntax highlighting and error highlighting in code. This makes it a mini editor inside of Node-RED.

An initial emulation of this this for Erlang has been made so that Erlang-RED flows can now also speak Erlang. The implementation is still rudimentary but it shows what needs to be done to get this happening. A function provides much power and the Erlang one is no different - i.e. it’s a window into the server hosting Erlang-RED. So a sandbox needs to be created and much thought into how to ensure that nothing breaks or gets maliciously damaged.

Use at own risk! is the conclusion.
Add FlowHub.org nodes for managing flow test cases

FlowHub.org is my attempt to make visual programming truly visual. One of the first things I did was to create a visualisation of Node-RED outside of Node-RED, i.e., flows displayed in webpages. From that initial effort, FlowHub.org was created to support visual version controlling of flow code - both inside of Node-RED/Erlang-RED and also externally via a website.

Adding FlowHub to Erlang-RED now allows me to maintain the test flow repository and easily import tests into either Node-RED or Erlang-RED - with the same consistent interface. As an aside: FlowHub is also coded in Node-RED, i.e., dog-fooding all the way down to the turtles.
Red-Erik.org utilises multiple flows

The initial release of Red-Erik was based on a single, now its a multiple flow monster! Why is this important? Because it ensures that the link nodes work across multiple flows - this is their main purposes. Also it made me think about how I could get Erlang-RED to execute multiple flows. In doing so, it became clear that executing and designing flows are very much different activities. When the flows get executed, they basically just become an Erlang architecture of processes, nothing remains of the original flows. Each process only knows where to send its messages to - when it receives a message. There is no overall structure of the flow in memory. This is great because there is no overhead to maintain such a structure.

Milestone Two - m2

JSONata parser and evaluator

JSONata is a transformation language for JSON objects that is heavily used in Node-RED. Within Node-RED it provides a vital function by offering functionality for manipulating the msg object (i.e., the data flow) without having to code Javascript code. Because of this and also wanting to be 100% compatible with NodeRED, I created a JSONata parser in Erlang and implemented basic functionality that JSONata provides. The parser is defined in yecc and can be easily extended in either the evaluator or in the yecc definition. Having JSONata support is a major step forward for the Erlang-RED!
External Connectivity: network connectivity via HTTP-in and MQTT nodes, command connectivity via the exec node

Having spent the initial part of the project developing the routing and flow control nodes, this milestone has three new nodes for accessing the external world.

The HTTP in is my personal favourite because with it, it is possible to do static http routing within Erlang-RED. Even better, combined with the dynamic linking of the link nodes, it is even possible to do dynamic web routing.

MQTT nodes provide message bus connectivity. The mqtt nodes are particularly useful for connectivity between Erlang-RED and NodeRED: it is now trivial to have instances of Erlang-RED communicate with NodeRED.

Exec node can be used to shell-out to a new process. Finally it is possible to read cat /etc/passwd using Erlang-RED - have hours of fun and games with family and friends by hacking each other installation of Erlang-RED ;)

With these three nodes done, it is possible to create applications for routing MQTT traffic or HTTP traffic to MQTT or vice versa.
100+ Visual test flows for ensuring compatibility

There are just over 100 visual flow tests to ensure that Erlang-RED nodes are compatible with the current Node-RED functionality. These “test” flows can be utilised by other projects (e.g. Py-RED) to also ensure compatibility or by NodeRED itself for regression testing.

Creating these flows has also given me insights into the specifics of existing nodes. It also lead to pull requests for the original NodeRED.

In the long term, I will push these tests to a separate repository so that they are independent of the project.
Internal architecture for utilising third party Erlang libraries

With the implementation of mqtt nodes using the emqtt library it became clear that each library will need a manager for handling communication between the node and the library.

This pattern was also implemented for the exec node that utilises the erlexec library and which has a manager to coordinate the message passing between erlexec and the exec node.

What I learnt was that libraries generate messages and if these are sent directly to the node, then the node behaviour would become chaos. I am torn between having generic messages and passing them off the individual nodes and being strict on the set of message that a node can receive.

But having a manager is actually the tie-breaker, being able to remove library specific code to the manger while the node concentrates on “node specific” functionality is optimal. Done right, it should be possible to replace third-party libraries without modification to the node code. What is node specific functionality? Something like ensure that its status is displayed correctly - in this case the messages waiting at a delay node.

Division of responsibility is simple when nodes worry about their status and managers about external resources.
https://red-erik.org the very first Erlang-RED application

Red-Erik.org represents an initial headless installation of Erlang-RED. The flow being executed consists of a http-in node listening to the ‘/‘ path, two template nodes the contain the contents of the page and a http-response node that sends back the content when requested.

It is oxymoronic to speak of a “flow executing” since what actually happens is that the flow represents a blueprint for an Erlang architecture. When the docker image spins up, the Erlang processes are created (four - one the http in, two for two template nodes and one for the http response node) and then messages are sent.

The processes aren’t wired together, all that happens is that the http-in node will send a message when a request from a browser is received. That message goes to the first template node that sends it on to the second template node after modifying the message to contain its payload. Using a mustache template, the second template inserts the payload into a html layout. That is then sent to the http respond node that is connected to the second template. The http respond node sends the response to the client.

So the flow specification is exactly that: a specification for a bunch Erlang processes that are completely independent of one another.

Milestone One - m1

Working integration of Erlang and Node-RED flow editor

Static flow editor frontend codebase communicates with Erlang via http endpoints. Flow editor codebase is retrieved from a running Node-RED instance using a well defined script. A clear path to updating the flow editor is thus assured: updates to the live instance are reflected in the static version in the project.
Erlang codebase

The basic shape of an Erlang project has been followed. Initially project codebase was unstructured but many at the Erlang forum helped to improve that structure and form it in to proper Erlang codebase. Using rebar3 for release and build management, eunit for testing and ered_ as project prefix.
Erlang Architecture stabilised - Supervisors and Processes all the way down.

Main gen_server components are placed under supervision to ensure stability. Nodes are gen_server processes but are not placed under supervision be will be in the long term. The ideas behind the architecture have been documented.
Initial collection of implemented nodes

An initial collection of nodes have been implemented, they are not feature complete but provide the basis for initial testing. The intention is to create as many nodes as possible to ensure that the underlying architecture supports these nodes. Feature completeness will hopefully not affect the architecture in the same way as initial creation. Using the testing panel within the flow editor, it is possible to check what works and what does not.
Flow Driven Development Process

A collection of approximately sixty flow tests have be created to support development. The idea is to develop these flows in conjunction with a working Node-RED installation and then transport these flows here. Erlang-RED Nodes are then developed until the functionality is implemented, that is, until the test passes. Test flows can also be marked as pending (using the flow environment so that future functionality can be staked out. This development process has also be documented.

Milestones are spots in the life of the codebase where everything comes together and there is stability. Milestones are not planned, they are discovered once reached. Milestones have no relation to versions.