Out of Scope: Projects beyond Node-REDs intended use-case.
Node-RED is a web-based visual flow-programming environment aimed at the IoT market. This is one interpretation of what Node-RED is. But Node-RED is a lot more, Node-RED is flexible, very flexible. Based on NodeJS, it encourages the use of existing NodeJS packages. This turns Node-RED into a visualisation of NodeJS code creating an environment where both programmers and non-programmers can gain an understanding of the problem domain and its solution.
It is this latter feature of Node-RED that I find fascinating, the ability to visually code to making programming become a cooperative activity with non-programmers. Stakeholders gain an understanding of technical solutions and developers gain an insight into the business requirements.
Taking that a step further, Node-RED makes coding more approachable for everyone. Stakeholders can design and implement their own algorithm, business flows and ideas. Programmers are no longer the glorified single point of technical knowledge — it is time to rein-in the silo thinking of tech-based companies where programmers are held up as technological gurus, bring wisdom down from the mount.
This might sound somewhat extreme and challenging to the current status-quo however as I write these words, developers are already beginning to lose their mystic. Artificial Intelligence (AI) allows the many to begin entering the world of programming, many online programming courses, user groups and hackathons have been pulling away as the mysticism of programming for years. Node-RED is a further tool for opening an industry that has been shrouded in mystic symbolism for far too long.
Thankful developers are a inclusive lot and enjoy sharing their ideas. One needs only look at the success of the Open Source movement to know that programmers enjoy sharing and demonstrating their skills to their peers. Coding is one of the few industries where sharing is something that is glorified and honoured. It is difficult to come up with parallels at scale, even the academic world has become a silo of knowledge with large publishing houses hoarding and gatekeeping knowledge through expensive subscription fees and alike.
Node-RED is a natural extension of this open-source sharing mentality. Initially developed at IBM, Node-RED was open-sourced under the guise of the Javascript Foundation, being a founding project of the foundation. A community has grown around Node-RED, bring it to the forefront of visual flow-based programming. Node-RED is beginning to emerge from its IoT essence into a more generalised programming paradigm. By doing so, Node-RED is help to shift programming into the browser.
This article will present five different, non IoT projects that have been developed using Node-RED. Each represents an itch that needed scratching and exploration of what is possible with Node-RED. To provide context I will briefly describe the Node-RED architecture before presenting the five ideas.
Architecture
Node-REDs has been developed in NodeJS and is built around a client-server architecture. All flows are designed in the visual web-based editor and deployed to the server where flows are then executed. Flows consist of executable nodes (nodes represent algorithmic code blocks) interconnected by data links that model the data flows between nodes. Nodes can have multiple inputs and outputs so that data is cloned and shared amongst many nodes. Flow execution is transparently concurrent as data flows are modelled and not algorithmic execution. Users model how the data flows between separate nodes that in turn are executed concurrently, as the data becomes available.
The distinction between editor and server is important to keep in mind since execution of nodes is done in the context of the server and flow manipulation in the context of the editor. There are nuanced difference between the two environments that are important to understand when developing new nodes. For me the editor is a standalone tool that can be useful in analysing flows before they are deployed to the server and even manipulating the editor directly to provide functionality that I might be missing.
Node-RED has inbuilt web and socket servers, these can be easily configured using the corresponding nodes. This makes communicating with flows as simple as dragging a node onto a flow.
My demo installation of Node-RED is available at demo.openmindmap.org and each of the following examples are wholly or partly hosted there. Feel free to play around, any changes made are read-only and cannot be deployed. Flows can nevertheless be exported: the right to use, share and apply for absolutely only non-morally-evil purposes is hereby given.
If you are interested in hosting your own installation of Node-RED, then I would suggest using Heroku, there is a GitHub repo with a basic installation and my modified installation for getting Node-RED running with user authentication for the dashboard.
Project 1: RSS aggregation
Back when Yahoo Pipes was a thing, the first thing I did was build an RSS aggregator using pipes. That was for me the main use-case for Pipes since it demonstrated how one could combine existing sources, filter them to ones needs and generate a personalised RSS feed.
Doing the same with Node-RED, one notices that Node-RED is a step beyond Pipes. Instead of getting an RSS feed out of Node-RED, I can create a dashboard which is updated as sources are retrieved. Node-RED makes data reusing and sharing very simple, so I also created a random link page using the same data as my RSS dashboard.
The flow gives the impression of complexity but as any good score for a musical piece, initial complexity is replaced by beauty once the piece is heard and understood. The common sub-flow abstracts away some of the complexity and provides common functionality used for all RSS source feeds. Sub-flows are a basis for drying flows to avoid repetition. Once a sub-flow has been defined, a further step would be to create custom nodes that solidify that abstraction.
Project 2: AI is a complex algorithm
Humans are complex in their way and Artificial Intelligence (AI) is non-human understandable algorithmic complexity. Intelligence, in this case, is complexity that we don’t understand and therefore assume to be intelligence. This assumption is unfortunately not true: all AIs are simply complex algorithms. This project aims to make that clear, providing some relativism to an topic that has caused so much hysteria. Think of AI as AC — Artificial Complexity.
I once studied Artificial Intelligence when it was used for Data Mining and the world consisted of grey-screen monitors connected to Sun Sparc Stations. Artificial Neural Networks (ANNs) were the inner structure of those AIs. Turns out that in the approximately 25 years that have passed, AIs — at their core — still have layers of nodes connected to each other. Now layers might be two dimensional and far more complex but the principles are the same. Layers of nodes that are interconnected with the layer below. Each node has an activation function and each connection has a weight. The connection weights are adjusted during a training phase. This is a very broad-strokes explanation but at its heart, ANNs are algorithmic complexity. (Aside: I once printed out the underlying algorithm for an ANN that I had created by evolutionary processes: the printout went over many many pages of mathematical formulas all finely formatted in LaTex!)
This I wanted to demonstrate and visually present the architecture of AIs. To do this, I created nodes the can communicate backwards. Nodes in Node-RED are directional in that they receive input and pass output out to their output connections. Normally nodes are not aware of their connections, being triggered when input is available. Nodes generate output that is passed to all their output connections. Node-RED transparently manages all this, nodes have a trigger function that receives an msg object and they call a send(..) method with their modified msg object. Nodes are not aware of their input connections nor of their output connections.
Nodes within an ANN are aware of their connections, adjusting weights on those connections during training phases. ANNs are trained using back-propagation whereby differences in expected results and generated results are propagated backwards through the network causing nodes to adjust weightings on their input connections — proportionally to the influence that connection had on the nodes output. This meant that the Node-RED node used for the ANN had to be aware of it’s input connections and also maintain a corresponding collection of weights, one for each input connection.
All of this came together in a Node-RED package node-red-contrib-neuralnet-builder and made for (my I think) nice looking flows. The example flow is not executable as the demo server is read-only but all the basic features of ANNs do work including the back-propagation training.
Project 3: Doodling with Node-RED
One of the down sides of Node-RED is that I found myself sometimes just aligning nodes and calling it work — Node-RED has a high procrastination potential! I got around it by calling it visual refactoring: if I can make a flow more understandable then it must be a form of refactoring.
I started to do more and more “refactoring” and began to realise that Node-RED can be used as a drawing tool. This is very nerdy exploration of Node-RED. I wondered whether I could have an SVG export of my flows so that I could rework those as diagrams or images. I found a reasonable plugin for Firefox but it had a number of drawbacks, especially since Node-RED uses CSS classes to hide things within SVG.
In the end, I created the Screenshot node. It can be triggered to generate a SVG that can be reworked within Node-RED. The content management flow is a demonstration. Used in combination with the SVG graphics node, screenshots can be displayed in the dashboard (or a default image if the editor is not currently active).
I began to think about the comparison between humans and AIs and I wanted to show that visually. I came up with this flow that pretends to show a human brain (it could be mistaken for a cauliflower) and its AI counterpart. Going one step further, armed with a plotter, I plotted this flow as a vector graphic:
For me this image and how I got to it represents the strength of Node-RED. That image is a working executable flow, i.e., working code however completely pointless yet somehow also indicative of the world we live in.
Project 4: AI can better understand me now that I have a mind-map
Mind-maps are visual representation of the algorithm that drives our thought processes. One thought leads to another, leads to another and so on. When generalised and abstracted, the mind-map provides insights into our brains’ creation of ideas and development of concepts. This is wonderful for us as we better understand ourselves.
Mind-maps are also a wonderful possibility to visualise ones filter bubble or better said information bubbles: where do we get our information from and what else is there that we do not know about. My initial idea for creating my mind-map was to have a two-dimensional representation of my thoughts, only later did I come to realise that what I had created was a two-dimensional visualisation of my information bubble, that bubble that comes from the algorithmic filters applied throughout the internet.
My next thought was that my mind map is not a street map. The difference between a mind map and a street map is that street maps do not have blanks. My mind map has plenty of blanks, white areas where no nodes dare to tread but there must be something there. I imagined my mind map being filled in by other peoples mind maps allowing exploration of the thought space that is human existence. Overlap between mind-maps could be done by matching terms and terminology.
I decided to call the idea openmindmap.org as a homage to openstreetmap.org with the intention one day to have a global mind map for better navigating the world of thoughts around us. I provide a demonstration but that has become more of a playground for my Node-RED flows.
Later I realised what potential a mind-map has for the training of AIs, as mind-maps also represent a digital representation of our thought processes. Whatever is digital is fair-game for AIs, so I am not that keen on providing AI with an even more simplistic understanding of us. Each of us is a complex creature with many sides. Providing an AI with a simple, watered-down mind map and pretending that it is the complete representation of a person is simply foolish, however it does make for great marketing: create a mind-map and have an AI think like you and then think for you.
I maintain my mind-map locally and occasionally I execute the flows to find all pathways between two points in my mind — makes for interesting reading. My mind-map is an executable visual program and whatever comes out seems to the result of the connections between nodes.
Project 5: Redback: Web-app API backend in Node-RED
This was my first dive into Node-RED. Since I do not have any IoT devices, I decided that I wanted to design and implement a backend for an application, both web and mobile. This meant that Node-RED would provide an API endpoint and both applications would rely on this endpoint to provide the user experience.
I modelled the flows on the concepts that all major web frameworks (e.g., Flask, Rails, Django) have — routers, controllers, views, and so on. This gave me many flows and much to work on. In the end, it all worked really well and the site is up and running at Bid’n’Buy.club.
The flows can be viewed on my Node-RED playground:
- Router
- Email Templates
- API GET flows and API POST flows
- User verification flows
- Customer support dashboard
For me, that provide the first glimpse one how useful Node-RED can be. The entire codebase of an application can be viewed, understood and manipulated by non-programmers. Codebase is available on GitHub, for the backend and the frontend.
Where to from here?
As first examples, web frameworks tend to begin by building a blog site as a first example of using the framework — this too could be done with Node-RED. It comes close to building a CMS (Content Management System) but one step at a time. My aim would be to use the dashboard component of Node-RED and have it become centre stage. For this, I want one domain to map to the blog (i.e. the dashboard) while a second domain (or sub-domain) points to the flow editor. This is possible, the next step then is to create the blog entries.
I continue to search for a good representation of text blocks that can be combined to create longer texts. My attempt of using a mind-map is semi working, one example is the text I wrote for open mind map org, that text is generated by this flow. The final version is over here. So it became difficult to polish the text using Node-RED within the context of a mind-map and I took the draft and reworked. This was not too hard since I am using Markdown.
The End of the Beginning
That was my initial adventure with Node-RED, perhaps one day I will actually use it for what it was designed!
If you wish to find out more about Node-RED and what it can do, checkout the Node-RED conference 2022, all talks are online and provide a good selection of what other people are doing with Node-RED. The keynote presentation provides a good background and history of Node-RED.
Join the Node-RED forum, there are many good people to help you get started, its a great resource when learning Node-RED.
If you want to try out Node-RED, then I would suggest installing Docker and using the Node-RED docker image, more details available at Node-RED.org.
If you also have an out-of-box around-the-corner Node-RED project or have ideas or questions or whatever else related to Node-RED, drop me a line at node-red <at> spreads-the <dot> love — thanks for getting this far!
