It’s not easy to reach for the cloud
The adoption of cloud solutions by businesses has gone mainstream. Companies often choose several different solutions to address various business problems. At the conception stage, such projects often seem straightforward and unlikely to present major challenges, since many of the traditional activities like hardware and software management are now integrated into the cloud solution. However, this is a misguided perception.
What is not being considered are the integration issues that arise when an organization adopts a cloud strategy. Every cloud application provider assures us that integration will be easy and straightforward and should not have a major impact on the implementation project. However, this assumption is not true. Integrating a cloud application raises challenges similar to those of integrating on-premise applications. Additional challenges are encountered when dealing with security concerns and integration capabilities of the SaaS application.
The simplest form of integration is the export and import of CSV files and the delivery of these files via SFTP. But even this can present unexpected challenges, because exports from cloud applications are either a full dump or a delta extraction; in some cases, they do not consistently deliver the same CSV structure with the same headings, since a column might not contain data. These inconsistencies need to be addressed in the data transformation. When transmitting XML files, an issue that may arise is that escape characters are embedded in a name and address string, which could make the transformation fail on the receiver side. File transfers require a significant amount of file management that can to a certain degree be managed via scripts; but the process is always vulnerable to failures. In many cases, files require additional encryption, which adds another layer of complexity.
The next level of complexity would introduce web service calls using either SOAP or REST. The challenge here is to decide whether a bi-directional integration is desired. Bi-directional integrations always raise security issues, since the cloud application will make inbound calls to the on-premise application. A simple approach is often used, wherein a pub/sub scenario is applied and the on-premise application picks up the data from the cloud. Because of business process requirements, this approach is often not feasible. Another problem with web service integration is that cloud applications will encapsulate the CSV file structure in the payload of the web service. This is not a web service integration but rather an alternative way to deliver the CSV file. Such an approach causes additional complexity, since now the payload must be parsed and reconstructed on the receiver end to be able to access the different columns.
From a transactional data perspective, issues arise when the receiving system is not processing transactions because of various data validation issues or other business-related constraints. Since in many cases, error recovery features on the sending system are absent, these problems can quickly lead to data contamination or missed transactions.
Frequently, the cloud instances do not map to the on-premise instances. In other words, the cloud environment consists only of a test environment and a production environment, whereas the on-premise application may contain three or more environments. This causes additional challenges, since in many cases, certain complex integration scenarios should first be developed using a proof of concept. There is no capability in developing these scenarios with just a test instance; additional instances need to be acquired from the cloud provider to conduct proper integration development, which often adds significant cost to the project.
A sound middleware platform will help in improving the integration and provide in many cases additional robustness from a monitoring, orchestration, and error recovery perspective. But fundamental design issues such as timing of invocation, parsing of data, and synchronization and recovery will still need to be addressed.
There is no simple solution to all these issues. However, there are some steps that can be taken to mitigate and ease these implementations.
Foresight is everything. It is important that from a project management and architecture perspective, a vertical as well as a horizontal planning and design process be conducted. The vertical approach looks at the overall business process and technological implications of an integration. This is helpful in that the business process design is not disconnected from the underlying technologies. With integration scenarios, the business process will always be constrained by the integration capabilities of the sending and receiving application. From a horizontal perspective, one needs to assess on the business process level the ergonomic impact of the integration to the user. The following questions should be considered:
- Is the integration useable by the user and does it simplify the process or add more steps?
- How will user error be handled?
- Is multiple sign-on necessary?
- What reconciliation processes are necessary?
- What audit trails are required?
On the technical level these questions might be helpful:
- What transport mechanism is the best fit?
- Are there integration patterns we can apply?
- What are the security and data integrity issues that need to be addressed?
- What is the data loading and synchronization processes between cloud and on-premise?
- In case of data contamination in the cloud do we have a backup and how do we recover?
Any integration design should consider these questions, and the answers should be reflected in the final architecture. When designing integration solutions, one should consider that automation does not always equal productivity. Sometimes it makes sense to use a more simplistic solution to achieve the same (or greater) efficiencies.
In many cases, a good proof of concept will help to identify uncertainties and problems early so that corrective actions can be taken to mitigate any associated risks. The proof of concept does not have to consider all aspects of the integration, but should address the most critical factor: the general connectivity between the cloud and the on-premise application.
Cloud integrations are not trivial and require a lot of foresight, planning, and design discussion. It is therefore important that the project manager, solution architect, and developer work closely with the business so that a sound and robust solution can be developed and delivered.