Secondary Sales Extraction (SSE)
- orderstack
- No Comments
The Secondary Sales Extraction is an ETL tool made by OrderStack, which helps our clients capture the secondary sales data from their stockists and store it for further analysis and generating reports.
Overview:
- The stockists send their secondary sales data files every month.
- The files are of PDF, Excel, Text, HTML or CSV format.
- The SSE system collects these emails and extracts the data.
- After extracting the data, the data is stored in a database for further analysis and generating reports.
- The full system is automated.
Project Lifecycle:
Requirement Gathering
Our initial setup includes mapping all the stockists in our system for the respective client. This includes recieving valid email ids of the stockists, their addresses and some basic information about the stockists.
Analysis
After the stockists mapping is complete, we move on the setting up the email CRONs for automating the email flow wherin the files that are sent by each stockist are collected and thereupon extraction is done on the respective files.
Development
Based on our plan, we led the development in the following manner.
- We created a NodeJs server where email CRONs were implemented.
- This NodeJs server interacted with a UI and had RESTful APIs created on it to let the system be controlled by the UI itself.
- The UI is written on the VueJs framework.
- The UI is dynamic charts and visualizations where the data is fetched from aggregated MongoDb Queries.
- The full system is automated wherein around 12000 files per month are extracted per client.
- The files which are accepted into the system are of the following formats: PDF, Excel, Text, HTML and CSV.
Testing:
For the testing of the system, we first host the project on a server and run a MongoDB instance on it (or a different server, depending on the client requirement). After that, the stockists mapping is done.
Deployment:
The deployment of the system takes place when the testing is complete and all the stockists are mapped. After the stockists are mapped, the email CRONs are run and the files start flowing through the system to get extracted and stored in the database for further analysis.
