I have a scenario in which multiple users of the a...
# springc
chavanshashank
03/22/2021, 12:32 PMI have a scenario in which multiple users of the application will be simultaneously uploading excel sheet(small to huge) to the spring boot backend.
After uploading that excel sheet, I have to process it for validations using POI for that which is fine and then store the whole excel sheet data in a database.
What libraries or architecture should I choose? Is Spring data flow is the correct choice?!
Or Uploading those files to Amazon S3 bucket and after uploading Queuing them in Amazon SQS and processing them in Spring MVC is sufficient for concurrent file uploads, processing and storing them in a DB without table inconsistency.
😶 2
K 1
p
Peter
03/22/2021, 2:39 PMFor excel sheet -> DB, I think spring batch could be an option.
👍 1
c
chavanshashank
03/22/2021, 2:42 PMYep. What about processing multiple files simultaneously and storing them in a DB?
Do I need to store them if I use Spring Batch?
Or Spring Batch can handle multiple files simultaneously?
And I think it will need more CPU and Memory resources right?
a
ashmelev
03/22/2021, 5:14 PMI have tried both approaches and have currently settled on the second (i.e. Spring Multipart File -> S3 -> message (with S3 URL) to SQS -> multiple replica's of SQS readers to process from SQS). The system scales very well as the resources are fairly distributed (between upload and processing) and S3 can store the raw original (well, it goes to Glacier eventually) and the "cooked" (parsed) product.
👍 1
c
chavanshashank
03/22/2021, 6:09 PMNice! And what about DB table data consistency if you are storing concurrently?
a
ashmelev
03/22/2021, 6:39 PMIn our use case it's never the "same" file - i.e. each user uploads sequentially. There are sometime hundreds uploading concurrently End of Month processing), but we do not have a "concurrency" issue like that. In any case the SQS would like the documents up "single-file" anyway - that should help with the concurrency/collision issue (you could coordinate multiple reader/processors using a FencedLock in HZ if absolutely necessary).
👍 1
t
Tristan Blakers
03/22/2021, 10:58 PMhard to answer this definitively without more context about the application, but if you want a cluster-aware job scheduler that plays nicely with spring, check out https://github.com/kagkarlsson/db-scheduler
👍 1
Tristan Blakers
03/22/2021, 10:59 PMone advantage of keeping it in-process is that it's much easier to integration test than the usual rube-goldberg machine of SQS/Lambda triggers
👍 2
Tristan Blakers
03/22/2021, 11:01 PMotoh, uploading huge files to your application server may not be great from a CPU/RAM/disk perspective, and offloading that compute to a separate microservice may be simpler overall
👍 1
12 Views