The Case For Salesforce-Postgres Synchronisation

Salesforce provides everything a company needs to completely describe their business flow. It is a walled garden for customer data and customers sacrifice their business flexibility to the customisation possibilities of Salesforces’ software: what Salesforce can’t do, their customers can’t either.

What I am going to talk about is a project I was involved in to get out of the walled data garden and providing more flexibility integrating Salesforce into a companies business flow. Goal is not to leave Salesforce behind, instead to provide integrational flexibility.

The Idea

My experience of Salesforce stems from a contract job for a Salesforce customer. I would say they were a small to medium size Salesforce customer. Their intentions are to continue with Salesforce and my project aims were to push more data into Salesforce.

The product I was hired to build was an application that would be integrated at the very front of the business flow, to support teams to doing customer acquisition. In Salesforce terms, we were dealing with Opportunities and before they got converted into Leads. When we discussed how to get this data into Salesforce, the obvious solution would have been to use the Salesforce API directly. However, besides having to do many specialised API calls, there would also have to be a certain amount of back propagation from Salesforce. Modifications in Salesforce would also have to be synchronised back to the application.

Back synchronisation from Salesforce had not previously happened at this company, so there was no reusable approach.

The forecasted complexity of the application prevented it from being integrated into Salesforce. Also as quite a few employees would end up working with the application, the extra costs of providing each with a Salesforce seat was not negligible. On top of that, all external services that the application required, would also have to be integrated into Salesforce.

So we decided on an external application using a Postgres database as storage. Instead of using the Salesforce API directly, a yet unknown Salesforce-Postgres bi-directional synchronisation tool was imagined.

At this point, we stumbled upon Heroku Connect which provides exactly the service we needed: a bi-directional, realtime synchronisation between Salesforce and Postgres. Using Heroku Connect, we could easily create our application on top of an Postgres instance and have all modifications automagically pushed to Salesforce, without having to deal directly with the Salesforce API.

Cutting a long story short, after a lot of back and forth with Heroku sales-reps and based on the amount of data we were forecasting on pushing into Salesforce, we discovered that Heroku Connect would have been prohibitively expensive for our needs. So expensive in fact, that we decided to roll our own Heroku Connect.

After six person months, both applications were in production. As required, the sales support application solely interacts with Postgres, it knows nothing of Salesforce. The Heroku Connect clone application only synchronises data between Postgres and Salesforce, it does not make any semantical assumptions of the data and it does not have any dependencies on the applications interacting with Postgres.

I shan’t go into any technical details, I will only say there was no rocket science involved in getting these applications up and running. We discovered many edge-cases of the Salesforce API and quirks generally of how Salesforce works, but nothing that represented a deal breaker.

With this experience in mind, I began to wonder whether there a case to be made for becoming a Heroku Connect competitor?

Heroku Connect

What makes Heroku Connect so prohibitively expensive that a company can employ developers to replicate it? Even including maintenance and infrastructure costs. Is pricing related to business or technological costs?

Heroku was purchased by Salesforce in December 2010 and Heroku Connect was introduced four years later, in May 2014. Since then, Heroku has published seven blog posts related to Heroku Connect, the last in June 2018. That does not seem to make Heroku Connect core technological for Heroku.

Whether Heroku was working on Heroku Connect before being acquired by Salesforce is speculation. Certainly four years to bring a product to market would not be unheard of in a corporate environment. If they did spend four years creating Heroku Connect, then the development costs would definitely need to be recouped via end pricing.

Also Salesforce has little interest in allowing customers to leave their walled data garden. Yes, Salesforce does offer APIs to retrieve data in bulk, update data or be informed of realtime changes using the Streaming API. But all these APIs have request limits, limits which can be extended by paying more. Not surprisingly, these limits do not apply to Heroku Connect.

Another aspect to the pricing politics of Heroku Connect could be that the potential-customers will make the assumption that expense equates to complex. My experience is that Heroku Connect is not a product of insurmountable complexity.

Two other sources of income for Salesforce are seats and customisations. Customers can save on both of these by using Heroku Connect. So pushing Heroku Connect might well be counter to Salesforce business model.

Salesforce Customisation

Salesforce is a complex product. There is no two ways about it, it isn’t simply a glorified database. Not everyone can make changes to it and comprehend the ramifications of those changes. So customers end up hiring Salesforce consultants to customise their Salesforce instances.

Compare the skill levels of a developer that creates an application to write into a Postgres database with those of a Salesforce consultant. Now imagine the salary differences between these two occupations.

I would argue that the former is cheaper than the latter. Not to mention the talent pool size of each.

So here would be another potential cost saving. If Postgres and Salesforce are magically synchronised, companies won’t need to customise their Salesforce installation. Instead they could hire a developer to build applications that interacted with Postgres.

Salesforce Seats

Each user of Salesforce requires a seat. This can become prohibitively expensive for certain businesses. One only has to imagine a call centre and consider the cost of having each call agent having access to Salesforce. If one then considers the high turnover amongst call agents, it additionally becomes a security nightmare.

There is a sweet spot between seat count and the point where Salesforce customers start to investigate alternative solutions to pushing data into their Salesforce instance.

So the irony is that the cost per seat, in itself, is an incentive for specific customers to search for alternatives to using Salesforce directly. Whether these alternatives are based on utilising the API or more abstract solutions such as bi-directional synchronisation is depended on the specific use case involved.

In addition to financial savings, there are a couple of technological advantages of synchronisation between Salesforce and Postgres.

Data Insights

Salesforces’ query interface (SOQL) it notoriously deficient. It misses quite a few of the more useful and complex SQL operators that can provide real insights. Having the ability to use all the possibilities of SQL to query Salesforce data is a real advantage. Postgres provides this.

Data Warehousing a Postgres database is far simpler (and cheaper) than integrating Salesforce into a Data Warehouse solution.

Centralising API Usage

Having a centralised synchronisation service fetching changes and pushing updates to Salesforce implies that this service becomes a single point of: API updates, data errors and system failures.

API Updates implies that if the Salesforce API needs updating, there is a single point where that needs doing. Instead of various applications that use the Salesforce API directly. If all applications update Postgres and a single application updates Salesforce from Postgres, then any changes to the Salesforce API only need to be applied to one application.

Data Errors meaning that if data isn’t being synchronised correctly into Salesforce, it can only come from the tool synchronising data to Salesforce. This is not strictly correct since semantic data errors made by the applications and stored into Postgres are blindly synchronised to Salesforce.

Finally, system failure is the effect that if the synchronisation goes down, all updates to Salesforce stop. However the other applications, since they interact with Postgres, can actually continue. Their changes aren’t being synchronised to Salesforce in realtime and when the synchronisation applications comes back up, care needs to be taken sending updates to Salesforce.

It depends on the specific use case whether these characteristics of the architecture are overall good or bad, personally I find a centralised synchronisation service encapsulating the Salesforce API a good thing.

Conclusion

From a technical perspective there are several good reasons for having a bi-directional Salesforce-Postgres synchronisation service. Such a solution can provide flexibility and cost savings. However, there is simply no business case to be made for becoming a Heroku Connect competitor. Why?

I would argue this is related to customer focus: companies that use Salesforce are not focused on saving money using Salesforce, they are focused on Salesforce assisting them to make money selling their products.

In turn, these companies do not apply their technical resources to solutions that might produce cost sayings, rather they prefer technical solutions that make them money. For Salesforce customers, Heroku Connect does not make money, instead it saves money.

So it makes more sense to spend money on customising Salesforce to make money in the now, than spending money on a product that potential saves money in the future.

Having said that, I hope I could at least raise awareness of an alternative approach to using and integrating Salesforce. Thanks for reading!