Common Issues in Addon Solution Architecture

Hi Everyone,

During past time, I used to review several ISV solutions and have collected some common issues that may significantly complicate development and support.

Unsplittable Code

Problem – Some ISV Solutions code and Acumatica code are very heavily connected and looks like “yin yang”.

• That means that there is no clear boarder between Acumatica and ISV solution that gives hard time to understand where exactly is the problem – Acumatica or Customization.
• And this leads to even bigger problem – unsplittable code makes impossible for ISV team to investigate bugs and issues – are they related to Acumatica or Custom code. In the end it makes impossible to escalate issues to the Acumatica team as there is no clear reproduction steps.
• It also makes upgrade problem complicated, long and painful.

Solution – design proper architecture with clear border between Acumatica modules and Custom functions.

• Review the Architecture of the ISV solution and try to rebuild part that are heavily connected to Acumatica thought code. Especially pay attention to processing functions and places where Acumatica code is replaced. One of the options is to involve Acumatica ISV team and review solution architecture with them. Acumatica team can be very useful in avoiding many common issue. Just simple share your design document and Acumatica team can review it and give recommendations.
• Replace code calls to Acumatica screens from direct programming API to Web Services API calls (from Graphs to REST API). This approach gives big flexibility as if issue happens after API call you can clear escalation pattern to Acumatica development team, as Web Services API is standard function and very easy to reproduce.

Technical Dept

Problem – Some ISV solution with history have a lot of old functions that were done in the past and may not be needed anymore.

• These functions slow down a performance, make upgrades more complicated and potentially can break essential calculations like stock quantity. As example I saw the old code that was designed as a workaround for a bug, but it was in the product for more that 4 versions. In the end this code slowed down system performance for 30 seconds every release operation.
• Technical dept becomes a bigger and bigger problem though time. Something that is easy to change now, will be impossible to change after few years, as other parts will depend on our technical dept
  • Putting off refactoring defers the cost of doing the upgrade (Debt)
  • Adds on additional costs of dependency (Interest on Debt)
  • Over time, the (Debt + Interest) grows exponentially with every deferral (Compounded Interest)
  • Large Technology Debt = can’t afford to refactoring and it becomes easier o rebuild software than update (Bankruptcy)

Solution – Keep eye on architecture while you developing product, invest time on fixing technical debt while it is not a problem.

• Create list of customization/integrations with reasons that are done for Acumatica.
• Review list by your team and work on removing technical dept.
• Share rest list with Acumatica ISV team and review one by one if that still makes sense or possible to remove.

Regression Testing

Problem – when solutions are small we intent to avoid testing in favor of saving time and money.

• However during development, upgrade and further customization you don’t know how much of the functionality is working now and how much will work on the next version.
• You don’t know the test coverage and how stable is your solution. You also don’t know how risky is to update on the new version/update.
• Finally you spend a lot of time for QA or Consultants to do manual tests of the same functionality over an over again, instead of focusing on reproduction steps of customer issues.

Solution – use a regression test framework from the beginning

• Use Acumatica Test Framework to create automated black-box tests that can cover the main part of the solution. Test will can help to validate that existing functionality won’t be broken with minor or major updates and will save months of work for client and development team.

Issues Investigation

Problem – some issues happen in combination of Acumatica and ISV solution may be random or don’t have a clear patter to reproduce.

• This makes issue impossible fix sometimes. As a result customer suffer a lot and have to wait for significant about of time.

Solution – in case of random issues few activities needs to be done from Development and QA teams:

• Ask user if they know what they did and can reproduce it.
• Use database audit fields (like Create/LastModified-DateTime) and references to analyse problem. Our target should be to guess what potentially could be a reason of a problem?
• Enable Request Profilers and Audit Logs. As soon as we got a case of error, we should review time and use logs to understand what potentially could happen? For example you can find that one entity may be updated twice by different users.
• Involve QA team that can check user flow and try to simulate the same approach.
• Analyse the code and its behavior in the problematic areas.
• Emulate users behavior on the testing stand. I heard from different companies that it is impossible to emulate 1000 users behavior, but in my experience I had a few cases where I was able to use Web Services API and console app with multiple treads to emulate heavy load and reproduce an issue. This actually may be a last chance to find an issue when other ways do not work.
• Beside all other points, there should be a person assigned on the issue who definitely know what should he do now and what will be his next step in case current approach does not work. Acumatica ISV team also can help with advises and issues investigations.

Database Knowledge

Problem – I notices that some teams put insufficient of focus on database knowledge, data architecture and query optimization.

• Acumatica uses relational database inside and if queries are not optimal and slow, the whole system may be slow in the end.
• Database queries optimization knowledge may be essential for medium/large customers. In-optimal query that gets data for 60 seconds may lead to whole server slowness. Just for example if you run query for 4 times (because you were tied to wait) it will fully occupy 4 cores on the sever and Acumatica may become inaccessible. So your team members need to know how to use Request Profiller and SQL Server Management Studio to analyse and optimize queries. A few tips you can find here.
• Sometimes adding one index may save 90% of the time for query execution. This happens then SQL Server does not have an index for data search and it starts to use loops. Iteration thought data is always slower than index.
• Database maintenance need to be like an everyday hygiene: Databases should have recurring maintenance procedures – rebuilding of indexes, updating statistics, files defragmentation.

Solution – build a database knowledge

• You may need to have at least one database expert that can analyse performance problems and assist others.
• In case queries with slow performance comes from the Acumatica functions itself, you need to have an instruction on how to escalate the problem to the Acumatica team – get a backup of tables involved in the query, add an SQL query to reproduce on tables included, add steps on how to trigger the query from the interface. As soon as this is done, our team will take optimization and will deliver the fix.

Version Upgrades

Problem – delay in the Acumatica version upgrades with 1-2 years leads to keeping customers on almost outdated version

• Sometimes from partners I hear that R2 is stabler than R1 or that you can use new version only after 3 months after the release data due to the stability. I admit that it might be a case long time ago when Acumatica was more a startup.
  • However now Acumatica significantly changes the release process and ensure that each version is tested and adopted before release it.
  • R2 is not a bugfix version of R1. There is no difference in reliability between R1 and R2 – both are brand new releases that contains new function and developed in the same way.
  • Due to high number of Automatic regression and unit tests designed by Acumatica recently, we maintain the same quality between releases.
• Acumatica does significantly reduce number of bug-fixes in the version when new version comes. This means that when 2019R2 will be released, Acumatica will issue less updates to 2019R1 and will mostly focus on the new version. After 2 years release becomes deprecated and is not in the support anymore
• This may lead to longer periods of time between bug report an fix in the not-current version.
• Also some issues cannot be fixed in the older version due to breaking changes are not allowed in the updates of Acumatica released versions.
• You have to ask customers to install a version you support that may lead to the problems with other solutions.

Solution – keep your solution in compatibility with the latest version of Acumatica.

• I recommend your to have a road-map on how you can use early adopter program from Acumatica to keep your solution up to date.
• Update is the process and cannot be done in a day, that mean that if you start to plan your solution update at the day when new version is issued, you will be able to go life with it only within 3-6 months.

I need to highlight that all ISV team related activities will require ISV contract to be signed. However it will give you much more than a support: marketplace, unlimited development cases, connection to ISV team, techical account manager and free training passes.

Hope it helps! Have a fast development!