Fourteen for, Fourteen against, why I love, hate and connect with Node-RED

A figure eight in Node-RED.

Clarification: This article is based on Node-RED version 3.0.2. Some new features are upcoming (for example deep linking described below) and thus will be available in future updates of Node-RED. There is a discussion on the Node-RED forum that points this out.

Fourteen pros and cons for and against Node-RED from the perspective of a programmer. I enjoy working with Node-RED, in-particular experimenting with its flexibility and approachability. For a good deal of my professional life I have been a card-carrying Emacs user, Node-RED is changing that. Node-RED introduces a accessible approach to programming, it is moving from a one-dimensional keyboard-based editor paradigm to a two-dimensional mouse-based visual drawing paradigm.

Node-REDs has been developed in Node.js and is built around a client-server architecture. All flows are designed in the visual web-based editor and deployed to the server where flows are then executed. Flows consist of executable nodes (nodes represent algorithmic code blocks) interconnected by data links that model the data flows between nodes. Nodes can have multiple inputs and outputs so that data is cloned and shared amongst many nodes. Flow execution is transparently concurrent as data flows are modelled and not algorithmic execution. Users model how the data flows between separate nodes that in turn are executed concurrently, when the data becomes available. This makes triggering background tasks trivial as it is a matter of diverting data to an additional set of nodes.

These pros and cons come from working with Node-RED over a six months period and having experimented with several use-cases that go beyond the intended purpose of Node-RED as an “IoT monitoring and dashboard”-tool. For example, I use Node-RED as a mind-map tool which might be counter-intuitive but has the advantage that my mind-maps are executable: I can generate text from all paths between two random nodes. Another plus is flexibility in connections: classic mind-maps tools tend to extend out from a main topic, Node-RED allows for cyclical references and ad-hoc connections between various ideas. An example mind map for this article is online.

A larger project was the development of a backend for the Bid’n’Buy platform. The codebase is available at GitHub. The use-case was to develop the API for a website and mobile app solely using Node-RED. It was a fun project with many learnings. The backend is based on the concepts underlying typical web-frameworks such as Ruby on Rails or Pythons Flask but holistically visual. My personal favourite flow is the visualisation of the router. The flow routes the API requests to other flows based on path and method introspection. It takes care of the authentication and end-point validation, returning a not-found or error response if necessary.

I then began experimenting with Node-REDs palette manager and wanted to create my own nodes. I ended up creating an Artificial Neural Networks (ANNs) builder. The purpose was to demonstrate the inner workings of Artificial Intelligent (AIs) models. These ANNs are extremely simplified however aim to transport the general concepts of AI creation, showing the interconnectedness of the various layers that make up an AI. It was also a great learning experience in understand how the frontend editor interacts with the Node-RED backend server. The codebase is available at GitHub and nodes are available for Node-RED.

Because I found Node-RED aesthetically pleasing, I wanted to have a tool for generating SVGs from designed flows. This led me to the idea of creating introspection nodes that can be used within the editor to analyse flows within the editor frontend, without needing deployment. The idea is to have a collection of nodes for doing a type of linting on flows, whatever linting would be in a visual programming language. The codebase is available at GitHub and nodes are available for Node-RED.

Out of these various use-cases, I have come up with my personal pros and cons on Node-RED.

Disclaimer

These pros/cons should be taken with a large pinch of salt since I did — explicitly — decide on use-cases that aren’t directly supported by Node-RED. That is to say, I exploited Node-REDs flexibility to experiment.

I write up these cons/pros with the idea of using Node-RED as a general development platform, that is to say, taking it out of the IoT world and thinking of Node-RED as a new approach to developing software in general. Why? Because — I think — Node-RED can provide a basis for better communication amongst various teams within companies, e.g. business people talking and understanding tech people and vice versa.

I write this up purely for egotistical reasons and unashamedly do not wish for anyone to take these ideas seriously nor out of context nor apply them to anything they might be doing. Having said that, I would also be grateful for any feedback and insights — many remarks of gratitude in advance!

Pros

1. Easy to approach with an gentle learning curve as one gets familiar with the various builtin nodes. Visual and flow-based programming requires an alternative mindset to traditional editor-based coding. Node-REDs user interface is approachable however the meaning of connections and nodes requires understanding of the underlying concepts, for example: what does a switch node do and how can I use it?
2. Once understood, code becomes more approachable for non-programmers. The flows are quickly understood and explained, flows can be quickly modified, extended or abstracted. Business logic is presented visually and can be discussed in visual terms. This allows for a closer interplay between programmers and business teams, with the common language becoming a visual interactive one.
3. Web-based, Node-RED fits into any modern browser. No need to install a coding environment. Node-RED has a server-client architecture with modifications being deployed from within the browser. Once the server has been setup, any user with a browser can deploy and make changes to the production server.
4. Extensions are managed using Node.js npm packages. These can be registered with the Node-RED palette manager so that packages appear in Node-RED. Extensions can define new nodes, sub-flows and flow examples, these can all be accessed directly via the palette management system.
5. Node-RED is as flexible as Node.js: anything that can be done with Node.js can be done in Node-RED visually and flow-based. Any npm package can be integrated into the Node-RED server. All flows are executed on the server side and designed using the web-based editor frontend.
6. Support for parallel modification to flows is provided via notification dialog. All users are immediately notified of deployed changes on the server side. Users can then merge those changes into the local version of the flows. Conflicts can occur but frequently changes can be trivially auto-merged.
7. Data conversion between formats is done automatically. Support for XML, JSON, CSV, YAML and HTML are provided out of the box. Additionally SQL results are automagically converted to javascript objects. This removes the necessity for much boiler-plate code.
8. Inbuilt templating support using Mustache syntax can be used for creating SQL queries or HTML content or any other formats supported by Node-RED. For example, this makes easy to create email templates within Node-RED. These email templates can then be modified by non-programmers.
9. Debugging is a matter of connecting a debug node to a nodes output and deploying to the server. All debug output is streamed to the editor frontend.
10. Adding extra business functionality into flows is trivial: connecting the output of one node to the input of another or linking flows via link nodes. Data flows are cloned automatically and can be diverted as required, there are no limits to connections between nodes.
11. Refactoring the code becomes a matter of redesigning flows by moving nodes around or creating sub-flows that can be used as nodes in other flows. Visualisation makes complexity stand out, complex flows are visually complex.
12. Multiple flows support is provide by tabs, one tab for each flow. Data flows across tabs (and thereby flows) using link nodes. Link nodes have two ends: in and out, and provide a great mechanism for refactoring complex flows.
13. Sub-flows provide a mechanism for code encapsulation to avoid code repetition. Sub-flows are trivally created by highlighting a group of nodes and clicking on create sub-flow menu point. Sub-flows can be updated and edited, code changes will be applied to all usages of the sub-flow.
14. Parallelisation of code is automagical. Designed flows describe the flow of data, data is automatically copied to all nodes into which the data flows. These nodes are independent from one another and run concurrently.
15. No need for background jobs since, for example, sending emails is done by diverting the data flow from a request-response flow to the necessary email sending nodes. Request and response cycles are executed in parallel to tasks such as sending emails or updates to databases.
16. Inject nodes allow code to be triggered in regular intervals making cron-jobs redundant. Node-RED allows the modelling of complete backend system and becomes a type-of UML diagram.
17. Automatic version control allows breaking changes to be rolled back. What business breaks, business can fix!
18. Easy to get started using the docker image. Unfortunately there are limited playgrounds for experimenting with Node-RED online.
19. Sharing and copying flows is easy because of the well thought UI which allows flows to be copied to the clipboard. These can then be pasted into an import dialog. Flows are exported in json form making them human readable.

Cons

I see Node-RED as general development environment, so these cons do not apply to many users. Many cons are out-of-scope for the further development of Node-RED — this I acknowledge and respect.

Why do I see Node-RED as general development tool? Because I belief that code should not be secret sauce hidden from business teams and stakeholders. Cooperation, communication and many-eyes lead to better coding and business decisions. My personally experience is that when business and tech work together, the best results are achieved.

I spent many years working in tech as an Emacs coder, deploying many systems and maintaining those. Coding was always visual for me: maximum 80 character lines and syntax highlight. But I realised that explaining code to non-tech people is simply not possible using Emacs. That’s why things such UML were invented. I see Node-RED as the link between UML diagrams and Emacs, making business logic clear for all to see and extend.

1. Editor is non-mobile compatible which could be argued should not be necessary since who would program/code on a mobile phone? Unfortunately Node-RED is approachable by many people, in particular managers who might be tempted to make quick changes between business meetings. One step I took was to hide the sidebars using a IsMobile node - definitely a hacky solution.
2. No linting or other introspective tooling. The only checking of correctness is syntactical but on the other hand linting makes no sense in a visual flow based programming language. But other tools would make sense, for example path discovery between two points in the flow, i.e. how many flows send a specific email?.
3. Flows can get complex and it is had to lose track what does what. Therefore a certain amount of maintenance of flows should (or must?) be done. This becomes the refactoring of Node-RED: making data flow lines straight and creating sub-flows for repeating node flows. It requires discipline to avoid code duplication and complexity.
4. Deployment is limited to a single server. There is no clear way to support a testing-staging-production deployment/development strategy. In its defence, Node-RED was never designed for general application to all possible use-cases, it is a IoT dashboard tool that is amazingly useful for other applications and use-cases — but due to its flexibility not by design.
5. No scripted deployment strategy. A tool such as capistrano or puppet for scripted deployment is not included but that goes back to the original goals of Node-RED. As the Node-RED API is documented, it would be trivial task to create such a tool.
6. Node-RED has no specific support for testing or the development of tests for the flows created. Of course one can create ones own testing process but there is no support out-of-the-box and with that and hence no best practices for implementing testing within Node-RED.
7. Difficult to work in parallel on the same flows. There is support for merging and notification if the server flows change but the shown differences are difficult to judge whether they are relevant or not. For example, a diff between two flows (i.e. your local flow and the changed flow on the server) will show you differences in positions of nodes — this isn’t particular relevant to the business logic. So diffs become very noisy if there has been a rearrangement of nodes.
8. No prevention of code duplication and it is tempting to copy & paste existing nodes, causing code duplication — in-particular of function nodes.
9. Need to name nodes sensibly else it becomes very complex very quickly. There is automatic naming of nodes which works well for some nodes but not for all, e.g. function nodes.
10. Comment node is very small and can’t be resized to make it useful. Also it can’t be attached to nodes to make it clear which nodes the comment might be referring to.
11. It is difficult to change the default settings on nodes and some nodes have misunderstand-able defaults — switch for example checks all conditions by default, instead of a switch statement that stops after the first match — which for me would be more common. This behaviour is a reflection of a C-programming language convention since C case statements go through all cases unless a break is used - the switch node is emulating that behaviour. For me that was unexpected.
12. Difficult to judge which data is required for which part of the flow. That sometimes leads to attributes missing on the data object, the attributes were not added by the previous flow parts but are assumed for the current flow. The contents of the msg object isn't always obvious - making it difficult to jump into a flow from somewhere else. (msg being the javascript object that is passed from node to node and can be extended by each node as required.)
13. You can have strange bugs if the query attribute is defined on msg since the Postgres node will always take the query attribute before the query defined in the node itself. This makes for strange bugs if the query attribute is defined by one node and not deleted somewhere before calling the Postgres node. This also applies to the template nodes and other nodes which take values from the msg object that override default node-defined values.
14. Procrastination trap: in aligning the various flows, and generally making them appear nicer, one can convince oneself that one is working even though one isn’t doing anything useful (although this has to be taken with a smaller pinch of salt since cleaning and simplifying a flow is important but sometimes one ends up splitting hairs). On the other hand, it is easy to get back into the “codebase” (i.e. the flows) by doing some straightening of lines and making flows look nicer (e.g. renaming nodes).
15. Easily extendable but the internal Node-RED APIs are not completely documented. Sometimes one is left in the dark concerning API usage. It is also important to remember where the code is being run: on the client or the server, the node-red API differs but the javascript object is called RED on both sides.
16. No fine-grained user permissions. Users can either read or write all flows, it isn’t possible to restrict users to specific flows. Authenticated users can deploy all flows which does represent a risk in multi-team environments.
17. Deep linking to specific nodes is not supported. I created myself a hack workaround by creating the Navigator node that supports node specific links with /n/XXXXX.

Done.

Much gratitude for getting this far!

Corrections, Updates and Clarifications

From the Node-RED forum, some updates:

1. @Totallyinformation points out that sub-flows do create code duplication on the server side, to avoid that, using `link-call` nodes might be a better option. Link-call nodes link to `link-in` nodes and return when a corresponding `link-out` node is reached. In this sense link-call nodes represent a type of synchronisation of flows: link-call nodes return when the reference other flow has completed. Additionally, to avoid hanging code, link-call nodes can be given a timeout.
2. @HaroldPetersInskipp points out that my comments on linting (con #2) are completely and utterly invalid! There is in fact a linting tool for Node-RED available on GitHub.
3. Clarification of what I meant with “deep linking” (con #17), this has nothing to do with the various link nodes in Node-RED rather it references to (sorry for the confusion):

In the context of the World Wide Web, deep linking is the use of a hyperlink that links to a specific, generally searchable or indexed, piece of web content on a website (e.g. “https://example.com/path/page"), rather than the website’s home page (e.g., “https://example.com"). The URL contains all the information needed to point to a particular item. Deep linking is different from mobile deep linking, which refers to directly linking to in-app content using a non-HTTP URI.

Source: Wikipedia