In order to fix the performance problems that we where having when showing large libraries in TM (like the SI library with 3000+ articles ) I had to literately go back to basics and change the way the data was consumed by the browser.
In the past this is what used to happen:
- The first major thing to load was the left-hand-side treeview which was sent as a JSON object in the format expected by the jstree jQuery component
- With that information, the tree was created and special events were set for when one of the nodes was selected.
- After selection (on a library, a folder or a view), on the first time there would be a JSON request made to the server which would fetch a large Table/Array (in a format compatible with the DataTable jQuery plug-in). This data was cached using jQuery data objects, so we only needed to fetch this data once (per library, folder or view).
- here we have one of the major bandwidth bottlenecks. Although there would be only one fetch per library/folder/view, there was no reuse of data between them. Which is absolutely not optimal since ‘an library contains all items of its subfolders’ and each ‘subfolder contains all items of its views’
- another problem was performance, specially in IE, which really struggled when handling large data-sets and creating/displaying them as html tables
There was also a lot of processing happening when calculating the ‘applied filters’ mappings (i.e. which checkboxes to select) which would also really put a strain on the browsers (IE specially)
Finally, the amount of repeated data/strings sent to the browser was also enormous (thing about how many times the word ‘Design’ was sent?)In addition to all of this, I also didn’t like how:
- a lot of UnitTests were not working (they were created before the move to the XML database)
- the existing UnitTests didn’t handle very well TM’s AJAX/WebServices driven world (with lots of gaps in the type of tests that were possible)
- how many inter-dependencies existed on TM client code (with lots of inter-connected scripts),
- how long small changes where taking (mainly due to the interdependencies and lack of solid unit tests)
- how hard it was for the new TM developers to get their heads around TM architecture and start quickly writing UnitTests and PoCs.
- as the main developer I stopped doing TDD (Test Driven Development) and where not creating Unit Tests for new features of bug fixes
And the last point is the most important one. There are a number of bugs currently opened in GitHub, but very few (in any) have UnitTests associated with them. And this is where we are going wrong. We really need to have a one-to-one mapping between bug/issues at GitHub and UnitTests. This way, when coding the fix, one of the first milestones is to make those tests pass or fail
So, this is what I did (with the latest code pushed to GitHub):
- There is a new way data is sent to the client (this is based on a number of core JSON objects fetched on TM launch, for example: an indexed unique list of strings , the guidance items mappings as simple int arrays (each int is a xrefs to the unique strings), the entire tree structure including the guidanceItems for each view.
- There is also a client-side data-processing step, which only happens once, is quite fast and it is responsible for creating the data objects that will be required by the multiple viewers
An non-CSS version of TM which is as close to pure html that one could get (using jQuery to allow easy navigation and visualization)