Major update to TeamMentor client-architecture, UnitTests back in action and real-time IE browsing

In order to fix the performance problems that we where having when showing large libraries in TM (like the SI library with 3000+ articles ) I had to literately go back to basics and change the way the data was consumed by the browser.

In the past this is what used to happen:

  • On the first hit of the main page, all Javascript files would be loaded and the main GUI components would be rendered
  • The first major thing to load was the left-hand-side treeview which was sent as a JSON object in the format expected by the jstree jQuery component
  • With that information, the tree was created and special events were set for when one of the nodes was selected.
  • After selection (on a library, a folder or a view), on the first time there would be a JSON request made to the server which would fetch a large Table/Array (in a format compatible with the DataTable jQuery plug-in). This data was cached using jQuery data objects, so we only needed to fetch this data once (per library, folder or view).
  • here we have one of the major bandwidth bottlenecks. Although there would be only one fetch per library/folder/view, there was no reuse of data between them. Which is absolutely not optimal since ‘an library contains all items of its subfolders’ and each ‘subfolder contains all items of its views’
  • another problem was performance, specially in IE, which really struggled when handling large data-sets and creating/displaying them as html tables

There was also a lot of processing happening when calculating the ‘applied filters’ mappings (i.e. which checkboxes to select) which would also really put a strain on the browsers (IE specially)

Finally, the amount of repeated data/strings sent to the browser was also enormous (thing about how many times the word ‘Design’ was sent?)In addition to all of this, I also didn’t like how:

  • a lot of UnitTests were not working (they were created before the move to the XML database)
  • the existing UnitTests didn’t handle very well TM’s AJAX/WebServices driven world (with lots of gaps in the type of tests that were possible)
  • how many inter-dependencies existed on TM client code (with lots of inter-connected scripts),
  • how long small changes where taking (mainly due to the interdependencies and lack of solid unit tests)
  • how hard it was for the new TM developers to get their heads around TM architecture and start quickly writing UnitTests and PoCs.
  • as the main developer I stopped doing TDD (Test Driven Development) and where not creating Unit Tests for new features of bug fixes

And the last point is the most important one. There are a number of bugs currently opened in GitHub, but very few (in any) have UnitTests associated with them. And this is where we are going wrong. We really need to have a one-to-one mapping between bug/issues at GitHub and UnitTests. This way, when coding the fix, one of the first milestones is to make those tests pass or fail

So, this is what I did (with the latest code pushed to GitHub):

  • There is now a fully working set of Unit Test (covering both server-side code and browser/javascript code)
  • There is a new way data is sent to the client (this is based on a number of core JSON objects fetched on TM launch, for example: an indexed unique list of strings , the guidance items mappings as simple int arrays (each int is a xrefs to the unique strings), the entire tree structure including the guidanceItems for each view.
  • There is also a client-side data-processing step, which only happens once, is quite fast and it is responsible for creating the data objects that will be required by the multiple viewers

An non-CSS version of TM which is as close to pure html that one could get (using jQuery to allow easy navigation and visualization)

A number of simple html files (used by the UnitTest) which show how the difference data sources can be consumed There was also a major re-factoring of some of the Javascript and server side code (designed to make TM server and client code easier to test). Basically what I did was to go back-to-basics in TM development and really understand how data can/should be sent + processed + viewer by the different browsers (It is pretty interesting to see what types of activities are hyper fast in jQuery and what are really slow)


About Dinis Cruz

Dinis Cruz is the main developer of the OWASP O2 Platform and TeamMentor
This entry was posted in Architecture, JQuery. Bookmark the permalink.

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s