It looked as a good idea: let the web application also do reporting until we have a performance problem, sooner than expected we had a performance problem, the web application was doing more than expected. Using more hardware didn't scale (and was costly on time and budget wise), the Plan B need attention: let's do a service for the data processing.. how difficult can it be?
Spoiler alert: our worst fears became true, we got near real time processing with few effort, on few weeks with excellent performance.
I will discuss all the journey our engineering team had designing our real time data processor, from the idea, the Architecture Review Board process to implementation.
As usual, we stand on many giant's shoulders: Ruby, AWS SDK, PostgreSQL and Dry-rb gems. This project was implemented on Amazon SQS was also planned to be easily moves to any other vendor, avoiding provider lock-in.
Creating a data processor is one of the many things Ruby can do as good as many other languages, having and carefully plan to move all the logic from a web application to a service should never be complicated, and should be documented not only in companies wiki, but also in the community.
I have been working with data processing on many technologies and languages, using closed and open source libraries the last 15 years.
A Senior Software Engineer now applying Ruby as a daily basis, having a full conversation with datasets, collections and queries all day long. Reviewing code and learning how to debloat the unbloatable. I have been programming on many languages, like ancient xBase (Visual FoxPro), C# and Python; on every change I’ve learn not only the technology behind the language also the culture, I’ve learn how to implement what I have been using on C# and Python into Ruby.