Postmortem: CMS Content
CMS Content was the name of a codebase I put a lot of effort into during my tenure at USD. In ways I'm proud of it, it was also a huge learning experience. The purpose of the portlet was to serve content in the student portal at the university. We had a piece of software that we were paying 10k$/y and wanted to migrate off of it. The original hope had been to migrate to a different Content Management System (CMS). This influenced several of the architectural choices of the portlet, most notably its modularity. I had intended from the beginning to allow for multiple data sources for documents, in a pluggable manner that would allow you to simply plug new datasources into the portlet and go. This would, and did, allow us to transition relatively seamlessly from our existing CMS onto something else.
CMS Content Original Interface
The CMS content portlet was a long educational experience for me. It was my first real adventure in making an interactive interface on the web that actually mattered (someone other than me was going to use it). I make no claims to be a designer, and it shows. It was a basic interface and it did work. But ever button press resulted in a full page refresh. It is certainly one thing I lament about my time at USD. I learned and antiquated development practices because the uPortal project is really quite old and so are its methods, but USD's was even older.
A little while after this I rewrote the API and the UI at the same time. This second UI is pretty good. It served us quite well and if I hadn't gone through a web components renaissance it probably would still be the interface that is used to day. You could add new documents on the right and it would update the portlet as you made changes, with no page refreshes between said changes which grealy improved the speed at which one could work on a portlet. On the left you could select a layout and it would give you some basic information about it.
Finally there was this version of the UI. This is the UI I made using Vue. Again, not a designer, it has a certain ugliness to it. But because it was made with web components it was really quite easy to do live previews of the portlet as you made changes. I was pretty proud of this UI on that front. I wanted to include this particular picture though cause it showcases a little woopsie on my part. The document, in this case the content about opening your USD inbox, is supposed to be below the entire editor area. Because I did the floating and flowing wrong it has actually pushed up into the empty space to left by the fact that there is only one document subscribed. Something I probably could have fixed if I had stayed around longer.
What Went Right
A lot went right in this project. I managed to migrate USD off of the old CMS I think in 2018 and so they saved 30k$ between my finishing the initial work and my departure from USD, and will continue to save in my absence.
Modularity
The system was quite modular and it worked rather well. I wrote a total of 5 document sources. CommonSpot, DNN, Internal, InteralForms, and InternalReports. Though only the last three remained by the time I left. I had hopes to include a document source for the university's knowledge base. I also wanted to allow the portal to pull in documents from other locations, but there was never interest for such. One of my favorite pieces of modularity was what I referred to as "responders". These were modular bits of code that allowed for a variety of responses to form requests. You could send the result of the form to various people, you could send a confirmation email, things would also get saved into the database to be used to create reports. I had hopes for installing all sorts of other results but again, there wasn't the interest for such.
Migration
Migrating off of the old CMS went rather well. Because I unified the document sources into a single interface it was easy enough to just request all the documents from the old CMS with the adapter writen for it and the write all of those to the internal document source. Had we gotten around to setting up a new external CMS (I was eyeing up openEquella) copying everything over to it would have been just as easy.
REST APIs
This was one of my first forays into REST development. My initial attempts writing an API were pretty bad but mostly because I didn't understand the underlying technology of JSR286 portlets. My second attempt I didn't yet understand the concepts of nouns and verbs in REST APIs so it was again pretty bad, in addition to not really understanding HTTP methods. My third attempt, the final attempt, I finally understood these things and ended up with an API for document, form, and report that all appropriate provided the correct information through the various HTTP methods. It was also in this third and final version that I introduced OIDC and was able to provide security on the documents as well. Not that there was usually anything sensative in the documents themselves, but the same security system also secured writing documents which was important.
Forms/Reports
The forms and reports part of the portlet worked really quite well. It had a few hiccups but it was a fairly expressive system by the time I was finished with it and could be edited by people who weren't programmers. Reports had no editor as it was, it instead was the results of the forms. I wanted to provide multiple options for reports, but there wasn't anyone asking for it. I did however have to do some creative work to get the security on the portlet working. Before I was able to use OIDC with the REST APIs I had to upgrade the underlying portal from version 4 to 5. So I implemented my own form of a token based security to secure sensative content on the portal while still making it a lazy loadable report (which was important as some reports had thousands of responses). Once we upgraded the portal to verion 5 I was able to use OIDC and so stripped out my custom code.
Web Components
This was where I really cut my teeth on Web Components as well. I wrote the front end in Vue as that was what the portal community at large had decided on as the web component front end on the portal. I hadn't really done a lot of web development before I started working on the portal for USD. So I must admit as an outsider that web components seem like a real "well duh" idea and I'm surprised some one didn't come up with them sooner. I started with the report display so that I could create a lazy loaded table, but once I had mastered it I slowly moved the entire user facing interface over to Vue. The portlet as a result was almost entirely just a API that the Vue component queried to display content.
What Went Wrong
Despite all the good, there were a number of things that I was not happy with or went wrong. Some were in my control, some were not.
File Explorer
Obviously I needed to provide a file tree in order to see all the documents neatly grouped together in logical ways. I did create such, but I introduced some errors into said tree while I was working on it. At one point every time you saved a document it would save a level higher than where it currently was. Which meant you had to select the folder you wanted to save in every time. Needless to say no body did, and we rapidly wound up with all the documents on the root of the file tree, effectivly elimintating it as a useful construct. Which leads me to the next problem.
Lack of Interest Outside of Me
A recurring theme you may have noticed is "I wanted to do x but there wasn't the interest". The file tree issue should have been a big deal, no one ever mentioned it. The portal got some 30k visits a day and if it had the slightest outage people got upset super fast. But so long as the husk was there nobody cared about the portal. There were so many cool and interesting things that could have been done with the portal if it people who actually cared were in charge of it. It was one of the notes I made during my departure from USD was to move the portal from marketing to res-live or some one of that nature. During the time I was there I kept begging for a portal redesign but I always got "We are working on a public website redesign right now, we'll get to it after that." for 5 years I got that line and they only started the new website in earneast the summer I left. Nobody cared about the portal, such a wasted opportunity.
No Decission on External CMS
Similar story to the above mentioned lack of interest. The internal editor for the portal's content wasn't great. But it wasn't supposed to be, we were supposed to get a new CMS to manage the content externally. I find it to be proof positive about the portal's usage by those responsible for its content that they never complained about the subpar interface most likely because they never used it.
Attachments
I decided to save myself the effort of creating a system for storing attachments in CMS Content by using the portal's existing attachments portlet. This turned out to be a terrible idea. The portal's attachment portlet was not designed for the demands of this type of system. For one there was no manager for the attachments. You were expected to create an attachment when you needed it for something like the annoncemence portlet, and just forget about it after that. So there was no way to see what files had already been uploaded. Additionally because of that there was no way to update a file once it had been uploaded. My inital attachments manager portlet worked until we got 2.3k attachments, at which point the portlet choked because the sql query to get all the document names included the binary data for the attachments, which mean every time the portal queried what attachments there were, it got some 4 gb of data back. This was not a good approach. If I had to do it again, I would make a custom attachments subsystem for CMS Content.
Version Control
There was no version control on the documents. Not an oversight, it was something I wanted to do. There was occationally interest, but when they said "ah! I deleted all the text! Can you restore it?" I would say no and they would ask about back ups. I would say I could implement such a system but you'll have to submit a ticket and ask for it. Appearently the interest in it wasn't even enough to submit a ticket.
Multiview portlets
One of the things I did early on was provide multiple layouts for a single portlet. The portlets had a "Normal" and "Maximized" concept that one could use to display different content at different times. I saw it was there so I decided to leverage it. While this may have been a neat possibilitiy it created more problems than it solved. I eventually removed the feature because maintaining it was way more trouble than it is worth. This taught me to not implement a feature unless your client was asking for it.
Conclusions
Looking back one of the things that I know would be different if I knew where it would have ended when I started would be I wouldn't have even taken on this project. Because the original plans had been to continue hosting externally the approach I took made sense. But if I had known we would end up hosting everything internally I might have simply improved the existing CMS portlet in the portal. The solution I made was more work and a bit over engineered for the final result, but I didn't know that was going to be the result until a year or more after it was done.
There was a lot I learned from this project over the years. It really was a major growth time for me. You could track my development as a web developer basically using just this portlet's code base. Some 750+ commits over 5 years of work. I'm proud of the work I've done and so I'm not in a "burn it with fire" mentality as I watch it go. But I am happy to not be working on it any more. The portal is old, the project should really be rebooted. Java is a horrible language. All of these things make me happy to not have to work on it again. Good by CMS Content, you were like a father to me, I loved you like a son.
Dreams that never were
A couple notes about things that never happened but I had dreamed of doing:
Document of Documents
In order to get the forms to work correctly I had setup a system of rendering the documents into an HTML string and then returning that. Which means I could render any document, including layouts if I wanted. I had imagined setting up a document source which would allow you to create new synthetic documents that would allow you to join documents together as you saw fit. There wasn't much use for such a thing so it never came to fruition. It might have had application though in the below.
Document Responder
The confirmation responder would respond to the person filling out the form with a preset string. I wanted to make it so you could select a document to be sent in the response.
Specified Reports
Take the previous two and add the ability to use a report with only a specific user selected and you could create a email that includes a default thank you message but also the results in the form of a report with just the specified result of what you just did.
Form response screen and duplicate entry protection
Some forms you would want a user to fill out only once. I wanted the ability to set a response screen so once a student filled out the form it would show the result screen and then be locked on that if the form maker so specified.