Ordering and purchasing an electronic document may seem trivial to the average user. In reality however, electronic document delivery is a complex task that involves several sophisticated systems, such as this by Altamira.
Some systems are self-sufficient, some are in the form of plug-ins that can be used for any e-shop, and others are third-party systems. The symbiosis of these systems is what allows the final product to be delivered to the customer.
In a cooperative project between the Austrian music publisher Universal Edition and sofasession GmbH, the project participants approached us to supply parts of such a system. Universal Edition (UE) is literally our oldest client, as they have been operating on the market since 1901. UE has a large database of sheet music in the form of electronic documents. This sheet music or documents are their products, which can be ordered through the e-shop and which the client manages semi-manually. Our solution will allow them to easily manage documents and will allow their customers to order.
Requirements
The complexity consists of several system features, such as the size of the document database, their safe storage, administration, ordering via any e-shop, or delivery of these documents to a third-party app.
Up to five systems are involved in the ordering process. They communicate with each other to achieve a common goal, which is the successful delivery of an electronic document to the customer.
- Hydra – Document storage and management system
- Universaledition.com – Document ordering system (e-shop)
- Getnote e-shop plugin – E-shop plug-in for sending orders to Getnote
- Getnote – Document delivery system
- Newzik – Document viewing system (third party)
Our client’s requirements were specific. The client has terabytes of PDF documents, i.e. products that must be securely stored and managed. Under managing, we can imagine, for example, their creation, editing, marking, trimming, splitting, merging etc. These documents are stored in the client’s infrastructure for security reasons.
In addition to our client, another provider can supply the documents. Therefore, the system must allow the instance that will supply the documents of this other provider to the system to run. The provider will then be able to import the products from the CSV file and assign them to the files themselves representing the documents.
Electronic documents can be sold by any e-shop (in our case we are talking about sheet music). Therefore, this system must be able to deliver documents from any e-shop to a third-party app that allows them to be viewed. As this is a third party, the system may become unavailable. There are mechanisms in place to treat this.
The documents must be stored securely and are only available if the customer successfully passes the ordering process and pays.
Technical solution
Our Altamira team worked with the client on systems for storing, managing and delivering documents. In addition, we created an e-shop plug-in that allows to send an order to an electronic document delivery system. We also worked on the integration of third-party systems, i.e. a system for viewing documents.
We built the systems on Vue.js, Laravel, MySQL, LAMP and Docker technologies. Vue.js is a fast and modern frontend framework. Laravel is a framework that allows good architectural principles to be followed by default. MySQL is a database that has been tested for years, which allows to process a larger number of records.
LAMP and Docker technologies have been advantageous because they allow us to run systems at multiple document providers very quickly or deploy systems to a server as needed.
How do the systems work?
Smooth communication between systems is what enables successful delivery of documents to the client. The system supports multiple document providers that have documents stored securely. Providers upload their documents to Hydra – a system for storing and managing documents. Subsequently, Getnote – the document deliverer downloads their metadata. If the customer successfully completes the order on the universaledition.com e-shop, the deliverer downloads the documents from the provider and sends them to Newzik – a browser where the client can view them.
Challenges
Communication with third parties
The plug-in that connects the customer’s account in the document browser to the document deliverer must be secure to prevent theft of documents. It must also support integration with any system.
Several mechanisms have been used to secure this, such as encryption, temporary stamping, obfuscation (obfuscation of the program’s source code), compilation of JavaScript code into a byte code, limitation to one specified domain, or back-authentication with the server. Interestingly, the resulting code cannot be read by a human. Only the machine can read and run it.
Integrating the plug-in with existing e-shops is easy. The system generates a set of codes in various programming languages, which can be copied and pasted into any e-shop.
Delivering documents to a third party is not that simple. The system must create a virtual contract between the provider and the customer that contains the metadata about the documents, including which pages are in the document. Therefore, the system must be able to read the PDF document.
In addition, the document itself may be large (e.g. 2 GB) or there may be a large amount of them (e.g. 2,000). Several systems need to work together to make all this possible. The core of the system – the web server and the client making the requests – must have increased limits on file size and connection length. This is so that, for example, the system can keep the connection for a long time if a large file is uploaded.
Therefore, the component that ensures the delivery of the documents to the third party must include corrective mechanisms in case of an error. For example, the file may not load correctly or completely.
Like what you’re reading?
Importing and mapping metadata to documents
Providers have their documents and their metadata stored in various databases and formats, but almost everyone allows them to be exported to an XLS file. Therefore, our system also allows to import metadata from large XLS files (100+ MB).
The XLS file does not have to be in a specific format, but our system provides a user interface where each column of the XLS file can be mapped by the user to our database. For example, if the XLS file has an artist name in the third column, the user can create this mapping and the system will create metadata where the artist’s name will be in the third column.
After importing the metadata, you need to map this metadata to documents. However, this is not an easy task, because there is no identifier in the metadata by which we could find the documents. Files are often named differently and finding a document by number is difficult.
For example, if there is a document code file in the folder, say Mahler-AB12345-test.txt, but no other code is there, then all documents in this folder are assigned under the number AB12345.
Editing PDF documents
The documents need to be edited because they are not the same size, are split, have a separate header from the content or are too large.
The document storage and management system can edit documents – for example, create, edit, tag, split and merge using drag-and-drop technology, split pages in half, crop marks for printing and compress. These features are included in the web interface, so working with documents is very convenient.
Some of the functions are automated. In addition to editing, the system allows to detect double-sided pages on one side of a document or crop marks etc.
In conclusion
We have developed several systems for the client that allow them to manage and order a large number of electronic documents. Customer requirements increased the complexity of the solution, so we had to deal with several challenges, such as integration with third parties or importing XML documents.
Our systems help the client manage their products and make their shopping easier for their customers. Are you interested in a similar solution? Do not hesitate to contact us and arrange a non-binding meeting.
