In the previous two installments of this series we discussed the target audience for Joomla! 4 and beyond and the vision for the end user. In this third installment we'll see things from the developers' perspective, defining a vision for the PHP code's architecture and design goals.

Accessibility

I'm told that Joomla! needs accessibility improvements. This is apparently required for using Joomla! in the public sector in many countries. This is not my area so I guess this requires input from accessibility experts.

A modern CSS framework

Joomla! 3 is stuck with Bootstrap 2 for ages. Even worse, it's not even Bootstrap 2, it's out own botched down version of it. Maybe that made sense in 2011, but in 2015 it's like trying to use smoke signals when everyone else is texting. We need a modern, up to date CSS framework.

There are two school of though on this. One school of thought says that we should develop our own CSS framework abstraction or even our own CSS framework, period. I disagree with this approach because it requires rare talent we barely have and it's not guaranteed that we will continue to have in the future. In simple words, it's no longer a great idea if you can't guarantee the longevity of such a project.

The other school of thought is effectively Joomla! growing a pair. Bootstrap is a great framework and we should stick to it. As in: pledge to always use the latest version of it, at most 6 months after its release to cater for Joomla!'s development cycle. We should NOT be afraid to break b/c with regards to updating to the latest version of Bootstrap. I know I was against this four years ago, but I've seen where staying rooted in an obsolete CSS framework got us.

JLayout for the win

JLayout is a great way to abstract some of the complexities of using different CSS frameworks. Please keep in mind that it only addresses static content, not Javascript interactions which require knowing a. the DOM model of the framework in use and b. the availability of helper Javascript functions. Essentially, JLayout is an HTML abstraction layer.

What if we could use JLayout to also provide a Javascript abstraction? I know that some JLayout templates already load their own Javascripts but these only address the Joomla! core needs. If I'm a 3PD who wants to display buttons dynamically or change the label color on the fly, though Javascript, my only option currently is to make AJAX calls which produce a new HTML page and replace the contents of the page the user is currently viewing. This is wrong and impractical for so many reasons!

Let's develop a common Javascript library, proxying to Bootstrap. If a developer decides to use CSS Framework XYZ they are obliged to provide an override for this common Javascript library. This allows 3PDs to support different templates without the ugly approach of force loading a namespaced copy of Bootstrap (yeah, I do that shit myself...) or inventing their own CSS framework which doesn't look "right" in a third party template.

Supercharge view templates

One of the innovations of Joomla! 1.5, back in '06, was the introduction of view templates which separated the code which does stuff from the code which renders stuff. That was great... nine years ago. Have you taken a look at our view templates? There's an awful lot of PHP code in there. Front-end developer need to have much more than a superficial understanding of PHP to style things.

Meanwhile in the PHP world Laravel happened. And it has these stupidly simple to understand Blade templates. Oh my, he nailed it! It's powerful as heck and easy for front-end developer to understand and customise without going into PHP specifics. And yes, it can be ported to Joomla!, I have already done that.

The Blade syntax allows us to simplify the lives of developers even more by adding support for constructs like @jlayout('com_example.foo.bar', $this->items, $somethingElse) Compare this to the typical JLayout invocation in core code and you see where this is going. In the words of Darth Sidius: POWER! UNLIMITED POWER!

PHP 5.4 and later

PHP 5.3 is dead since August 2014. PHP 5.4 will be dead in August 2015 but at least we know that all shared hosts support it – except those which shouldn't be allowed to be connected to the Internet! – at least as an option.

PHP 5.4 allows us to write leaner code with less duplication using Traits. This is the practical reason behind this suggestion. We could easily shave off 15% of our core components' code and reduce the amounts of bugs we're dealing with. Double win.

MySQL (and compatible variants) only

I know that this will upset the 10 people who are using Joomla! with Microsoft SQL Server and PostgreSQL. There are several practical reasons for doing this, though:

  • Query optimization. Our queries are suboptimal and lead to bad performance, especially on larger sites. If we know that the core is only supposed to work with MySQL we can use the expertise of MySQL DBAs in the Joomla! community with a long standing offer to help to make Joomla! fast. That's one hell of enterprise feature and major marketing point.
  • Reduce the barrier to entry and errors in deployment. If someone delivers a new feature with schema changes they're supposed to deliver these changes in MySQL, PostgreSQL and MS SQL Server format. In two different locations each. This puts off developers who are very experienced with PHP and MySQL but can't afford to learn and test against PostgreSQL and MS SQL Server. Even worse, the PostgreSQL and MS SQL Server schema changes usually get merged untested and errors are only discovered long after the release.
  • We don't have the people to support 3+ database server technologies. Proof of that is that the PostgreSQL integration was broken for the longest time (at least 1 year) and nobody noticed. Even when we noticed, it took several months to get it fixed. I still have no idea if the MS SQL Server driver really works.

So let's just drop support for additional database types. We can leave the drivers for them in the core (Joomla! Framework?) for 3PDs and enterprise users to use them. The core will only support running on MySQL.

As for MySQL itself, since we can't raise the minimum requirement to MySQL 5.6 yet we'll have to put up with core tables in need of full text search support to stick to MyISAM (and chop the head off any moron who suggests converting them to InnoDB).

Get rid of UCM

UCM was half baked, is half baked and will forever remain half baked, so let's just pull it out. It is NOT required for Tags and Content Versioning. We just need tags assignment and content assignment tables with one extra field denoting the content type. Then we can use something like Laravel's "morphable" relations to make them work. If you read Laravel's documentation you'll see that this is the use case for morphable relations, after all.

Bonus points: this greatly reduces the content duplication and clutter of core tables, simplifies queries and speeds things up.

Remove a lot of core extensions

Per the existing plan. One thing we currently suck at is advertising to our users the core supported extensions. I think that the Install from Web landing page should have two areas: core supported extensions and a curated list of commonly sought after JED categories. This allows us to tell users to go there to quickly add the features they don't find in the core. This approach works perfectly for WordPress: the landing page of the Add Plugin page is essentially a list of Automattic owned or sponsored plugins.

Namespaces everywhere

Since this is a major version I believe we can finally break b/c and provide fully namespaced code. This will also reduce our code since it will be finally possible to extend a front-end MVC class from a back-end MVC class. Previously that was just not possible because they both had the same name.

Moreover we can completely do away with JLoader. Instead, use Composer's PSR-4 autoloader which is already shipped with Joomla!. This will do away with bugs in 3PD code caused by core classes moving from libraries/joomla to libraries/cms, or classes which do not follow the current autoloader's scheme.

Backwards compatibility layer, at least for 4.x

A new version of the CMS is useless without 3PD extensions. It's our job as developers to ease the transition of code written for Joomla! 3 to Joomla! 3. This can be done by providing a compatibility layer which adds class aliases and backports of obsolete code for the most commonly used core APIs. This will allow developers to work their way towards Joomla! 4 compatibility without feeling they have to start from scratch.

It would pay great dividends if we also document the conversion process for existing code. If anyone reading this complains about the feasibility of such a plan I have the page which proves that I put my documentation where my mouth is.

Better Composer integration

Move composer.json and composer.lock into the libraries folder, not libraries/vendor. This allows adding requirements without essentially hacking core.

Allow 3PDs to add requirements to Joomla!'s Composer installation during the extension installation / update. We need to see if this is possible on shared hosts without CLI access – for context see this Drupal 8 discussion. This needs to be tied to an overall com_installer refactoring. The feasibility study should concentrate on whether it's possible to make the dependency discovery and individual dependency download and installation separate steps which can each complete inside a single page load.

Finally, drop all "invented here" core libraries in favour of Composer dependencies (especially things like JImage, JGithub etc).

DI Container

Ah, my favorite pet peeve. Right now we have this magic global black box called JFactory. Get rid of JFactory, replace it with a DI container (something like Pimple?) which is then fed to the application object. Anything you want can be fetched in a manner similar to Joomla\Application::getInstance()->getContainer()->user.

I also suggest taking it a step further. Each extension type gets its own container type which gives access to the main DI container and local dependencies that make sense for the specific component type. You should be able to spawn the Container of an extension from anywhere, e.g. Joomla\Container\Component::getInstance('com_something'). This allows for painless HMVC. Been there, done that – in FOF 3.

Overhaul MVC

I know that the Joomla! Framework has a "new MVC" which I call "no MVC" because this is what it feels like. I disagree with its approach for mass distributed software like Joomla! and its extensions. I do understand why you may need this for decentralized applications. I suggest keeping it in the core for these use cases but not using it as the default development paradigm for Joomla!. Remember what our target audience is? Exactly.

Instead, I propose following Laravel's lead, without the façades – a major reason why some serious PHP developers hate Laravel. Let's do away with the Model vs Table separation. Create a Model for non-data-aware stuff and a DataModel for data-aware stuff. using a vocabulary and feature set similar to Laravel's Model + Eloquent is a good idea. I've already done the work in FOF 3 and made my development a breeze. Less code, faster execution, less and easier to fix bugs are the immediate benefits.

Also, as I said above, add Blade template language support to our Views. I am half tempted to propose allowing views to be defined with XML forms –which lends itself to RAD features such as scaffolding– but unless you're willing to essentially make a RAD framework the core default it's unadvisable.

Well, OK, what the hell. Let's just put FOF 3 into the core. I already know that the biggest drawback will be for me. It will however benefit everyone else by having a RAD framework in the core. Right now we bleed developers to Laravel, ZF etc because it's easier to develop there, even though you have to do everything from scratch. If we remove that "coding for Joomla! sucks" feeling we can regain a lot of that audience. That audience does indirectly or directly drive the choice of a platform for big business' new sites so there's a plan to increase our market share.

JSON API with virtually no code

If we could allow access to the CMS data (and data management) though a JSON API we would let advanced uses of the CMS. For example using Angular.js to render pages, remotely managing the content, systems interconnection, easy mobile application development etc.

The groundwork for easy JSON views with HAL support is already done in FOF. No matter if it becomes part of the core or not the HAL support is something we have to put in. I know we already have com_ajax but it's only useful to do AJAX like it's 2008 all over again. HAL allows API consumers to discover content more easily, without having to know the exact inner workings of the server.

Improve the schema installation and updates

At this point each core database change requires editing two files, one for updates in the com_admin component and one for installation in the installation application. This is error prone. This is a problem already solved with FOF's FOF30\Database\Installer using XML files which allow the installer code to create and/or update the schema automatically. It would also simplify the Fix Database feature: you can use the same code as the schema installer/updater.

Since this was already discussed for inclusion in Joomla! 3 I think it's a given?

Improve the extensions installer

Last year I've written a whitepaper on it. We can and should add support for package signing as well. Some specifications work is already done by David Jardin. I'm sure there's a revised version somewhere butI can't find it.

I know, you think I'm done with this big subject in 30 words. Well, the two documents are about twice as big as this entire blog post :)

Decouple menus and routing

The current menu system defining routing leads to duplicated content and some impossible to solve problems. If a leaf node (e.g. article) can be routed through two or more menus there's no way to know for sure which URL you should follow unless you also know which Itemid you should use too. If no Itemid is provided chances are the "wrong" one will be picked, leading to the "wrong" modules being loaded.

Fixing that MAY cause some problems with our module system. If every leaf node article has a canonical URL you might end up again with the "wrong" modules being displayed in some situations. For example if I have a menu item for an entire category and another menu item for one of its articles (with different module assignment) then clicking on that odd article will yield an entirely different page layout than the other articles of the category. This is not necessarily bad, but it does change user expectations compared to Joomla! 3 and earlier.

Finally, com_redirect actually becomes important since it'll be the easiest way to add custom URLs. In Joomla! 3 and earlier the typical method was creating items in hidden (no module for them published) menus instead.

I do need feedback from people who have a better grasp on routing.

Testing and QA

Upgrade PHPCS rules to PHPCS 2 format. This allows contributors to auto-fix their code style (the bane of our existence for large contributions).

Unit and functional tests should be written for our common use cases first. The short term goal is not 80+% test coverage, it's making sure that regressions don't happen.

Joomla! Framework

I think that we can now all admit that trying to create a generic PHP framework to compete with the likes of Symfony, Zend Framework etc was a waste of valuable resources. So what we should do with Joomla! Framework?

One idea is to completely fold it back to the CMS core, much like Joomla! Platform was before we called it Joomla! Platform. I think that's the wrong approach because it ties CMS development with framework development. This is not always a good idea. The framework is basically an R&D lab. You just don't put R&D under production.

A better way to approach this is as a semi-detached project. The goal of JF is to provide a stable platform for Joomla! (the CMS) to be built on. This requires a major shift in JF's development since it has to keep in mind and cater for the complexities of mass distributed code. For example, the Uri package just can't ignore the need for a forced $live_site instead of consulting the $_SERVER superglobal. But having it separate from the CMS allows it to both do R&D at its own pace without affecting CMS development and lets anyone interested use it outside the CMS.

As for Joomla!, it can pull in JF packages through Composer. This should be the one and only way to do it, not the current situation of having a human copy .php files from a gazillion Git repositories into the libraries/joomla folder.

I'd like to discuss these ideas further with Andrew and Michael. Perhaps we could do a joint post on the future of the Framework. I think that if we frame the framework right (pun intended) it can be a successful child project of the Joomla! organisation. Basically, we need to (re)define its target audience and discuss how it can finally move forward.

47 thoughts on “Joomla! 4 and Beyond: architecture and design”

  1. Friday, 06 October 2017 03:47
    I love all of these suggestions for Joomla, however not sure what the UCM was in Joomla. I agree as well I don't see an independent Joomla framework getting traction, I think pulling non-essentials out of core to create a very light core when people want to build out custom applications with it makes more sense.
    1. Friday, 06 October 2017 06:29
      UCM (Unified Content Management) was a half-baked approach to creating unified content with arbitrary (and programmatically defined) content types. It is what is used to power tags and content history in a clumsy manner which ends up duplicating the content in your database. It needs to be removed and replaced with a true custom content types solution. In the grand scheme of things I think that Joomla! 4 can survive just fine without this particular change :)
  2. Saturday, 23 May 2015 19:16
    Hi Nicholas,

    I don't know whether it is the right place or not, but I need to suggest adding more plugin events for database queries, to be able to customize the query parts, does it make sense?

    Thanks
    1. Saturday, 23 May 2015 19:20
      No, this has a massive performance impact and security implications. It makes more sense to have plugin events in Controllers and Models. This is the approach we have been using in FOF. It does do what you have in mind.
      1. Monday, 25 May 2015 16:00
        Thanks!
  3. Saturday, 23 May 2015 22:08
    CSS Framework + JLayout - With some tweaks to our inner APIs we can be at a point where there isn’t a hard core dependency on a single framework. Why is this important? Designers aren’t always using Bootstrap, and quite frankly I think our override system (extensions, media, and JHtml) is strong enough to allow this. Yes, those folks using another library need to be prepared to do a bit of heavy lifting to add the support, but the possibility of eliminating vendor lock on one framework vendor or version is real.

    Enhanced templates - Huge +1 here. In a sense, JHtml kind of provides these helpers, but they aren’t a native templating thing. Actually, there isn’t anything resembling a proper templating engine in the CMS today. I think it’d be pretty awesome to get something like Blade into the CMS, and the sooner the better.

    PHP bump - So I’m willing to go so far as to suggest that new features (that are opt-in) be written toward 5.4+. The first thing that would focus on for me would be the MVC layer, as that is an entirely opt-in thing. Maybe that new MVC layer can be built with Blade templating (or whatever engine) built right into it.

    Database Platform - The CMS was built on MySQL and the support for the other drivers is a major hack at best. As an application, I think we’re reaching the point where we need to pull the plug on support for them. Keeping the drivers is a good idea though, in the end our database API should be platform agnostic but this shouldn’t force extensions to do the same (and hurt performance or development; you can use Doctrine all day long and still optimize on MySQL).

    Namespaces - I say legacy MVC don’t even consider it, the new MVC layer should force extensions to be namespaced. I say we also need to settle on a namespace structure for the CMS (whether that be using the top level Joomla, using Joomla\CMS, JoomlaCMS, or whatever else) and begin migrating more code into the namespaced environment.

    DI - We’ve had a DI container since 2012 (it shipped with 3.2). Extensions are getting on board with it, core still loves JFactory. Not saying our DI container is perfect, but it’s there and we need to put it to use ASAP.

    MVC - So this one’s a bit interesting. At a Framework level where the components are distributable beyond the CMS, I think the “No MVC” layer works OK; it leaves it to the application to build their CRUD implementations. In the CMS though, it definitely isn’t an option to ship without a CRUD layer. I see no issue building that layer on the "No MVC” code as meeting the interface requirements is dead simple anyway.

    QA - The Framework’s actually doing somewhat better than the CMS right now. A majority of the packages have adequate coverage (http://framework.joomla.org/status), I think our issue right now is what the quality of those tests actually is and whether they can catch potential regressions or issues. Tests in the CMS are a lot more tricky to write (especially once you involve JFactory) and in some cases the classes themselves are just borderline untestable. So this part needs multiple steps to implement, including identifying poorly covered APIs (most) and identifying code structures that need refactoring to improve testing (and the structure overall). On the PHPCS 2 thing, that’s already a work in progress in the main coding standards repo and hopefully something that can be finished soon.

    Framework - No, we can’t compete with full stack frameworks. At the same time, I think the top level code is versatile enough that it can be an option for smaller projects where those frameworks could be overkill. So I wouldn’t necessarily go in a direction where the Framework HAS to carry support for all the CMS features. In 3.4, the Framework’s URI package was merged in and JUri was adjusted to extend this keeping support for all the CMS features (like $live_site) while enabling the parent code to keep its agnostic approach.

    The Framework API has taken a modular approach. I kind of like it this way. If we keep it where the Framework should support the CMS’ needs (but doesn’t necessarily carry the full feature set, like my thoughts above on the URI code), it lets us focus on improving the structure of those components for the CMS with the side effect of making them usable beyond the CMS (like our issue tracker).

    As for what exactly the Framework offers, it needs a diet. I’d suggest keeping our GitHub package (it's used in production in several of our own tools, they don't offer a SDK like the majority of our other packages have, and quite frankly compared to KnpLabs I think our package has a more natural feel of the GitHub API, plus a lot of the packages seem to be platform specific), but the rest of our API adapters at this point fall under the “well intended but unmaintained” category and should be allowed to rest peacefully. Our String package has over 12K downloads on Packagist in ~2 years, no idea what the source is though. Considering it has our UTF-8 compat code in it, and UTF-8 is still tricky in PHP, I think this is a sign of success.
  4. Sunday, 24 May 2015 12:39
    CSS Frameworks: I agree. We also need a common Javascript library, otherwise using Bootstrap makes no sense to begin with (Carousels, tooltips etc require Javascript initialisation).

    Enhanced templates: Adding Blade support only requires modifying Views to through a view template engine. I've already written the code.

    PHP bump: Go all in or not at all. The MVC layer does NOT require PHP 5.4. I've already done the work in FOF 3, that's why I'm so sure.

    Database Platform: We need to pull the plug. Keep the drivers in but don't require the CMS support any db platform except MySQL. You can NOT fully optimise queries in a db-agnostic manner. At best you'll end up with the crap we currently have.

    Namespaces: Fully agree. I've done that in FOF 3 and saved me a lot of headaches.

    DI: I know. It's not used in core and nobody is using it. I have no idea if it's maintained or not, that's why I put Pimple with a question mark :)

    MVC: Two reasons not to do that. First, you're tying the CMS MVC to the Framework MVC. If the latter changes you have to reimplement the former. Moreover, it's a different development paradigm (e.g. one class per task) which is NOT apt for mass distributed applications like J! components. The benefits of the Framework MVC are lost in the context of this kind of software. Frankly, people do prefer using Laravel of other PHP frameworks. What does that tell you about what the bulk of developers want? Surely, the "no-MVC" paradigm is better for large scale apps but this is not our primary or even secondary target audience.

    QA: That's why I put the Container requirement in there. JFactory is a pain in the posterior during testing. Ask Davide about all the funky crap we were doing in FOF 2 tests to work around it. The Container solves that problem once and for all as you do know :) About everything else: awesome!

    Framework: Not all features, just the ones which would otherwise cause a lot of additional code to be built on top. The Uri package is a prime use case. Implementing JUri on top of Uri required lot of code. Hacking Uri to add support for live_site resulted in much less code being written. I've done that in AWF, that's why I used this package as an example. It is a fine balance but we can do it. Besides, adding support for something doesn't invalidate Framework's utility outside the CMS. Adding a null by default static $live_site variable to the Uri class doesn't make it unsuitable for the Issue Tracker.

    Framework diet: I understand that the GitHub package may have been a wrong example. I'm not perfectly sure what can be removed or not, it does require some discussion. I think you and Andrew are the most qualified people to lead that discussion. The Composer libraries I do regularly use are very specific and have no overlap with JFW classes anyway :)
    1. Sunday, 24 May 2015 18:48
      My first contenders for the Framework's diet are basically every non-GitHub API wrapper. Download numbers on Packagist will tell you why I say this. Unused by most, unmaintained by us. Let it go.

      With the URI package, at the Framework level we still have some structural issues with actually being able to support that live_site thing. The CMS implementation uses the data out of JConfig, the Framework has no concept of a configuration store (our DI code doesn't have a parameter store, something Symfony's does, and our application package supports an injected config but not everything has a dependency to that package). Maybe just having a static var works fine and it's left to the developer to set it. Another small issue is the method where live_site is actually calculated (JUri::base()) is tied to the CMS' two-app approach. Maybe the Framework's UriHelper class can absorb the helper methods in JUri (base, root, current) in some form, but it does need a little bit of thinking if we want to make it work.

      In a bit of irony, JControllerLegacy could meet the No MVC interface requirement, even if the legacy controller model doesn't match the intention of the interface. Even my last big project took that style approach (turning the execute method into a task dispatcher of sorts). Not saying it's right by any means, but I'm far past single task controllers being viable in a large project (and this comes in part from how many controllers we have in the issue tracker).

      In truth, I wonder how many devs are actually building on an MVC style approach. Symfony itself isn't MVC by any measure and I don't have the experience with Laravel to judge it.
      1. Sunday, 24 May 2015 19:15
        I agree on the Uri package approach.

        <blockquote>
        I'm far past single task controllers being viable in a large project (and this comes in part from how many controllers we have in the issue tracker).
        </blockquote>

        Thank you. That's exactly my case against the Framework MVC in mass distributed code. It makes sense when you are building web services, not so much when your code is 80% handling the same formulaic views with minor changes to handle specific business rules. I think that describes the majority of Joomla! components – definitely all the core ones.

        <blockquote>
        In truth, I wonder how many devs are actually building on an MVC style approach. Symfony itself isn't MVC by any measure and I don't have the experience with Laravel to judge it.
        </blockquote>

        That's a good question. I guess it depends on what kind of sites are you talking about. MVC –as we mean it– is not as scalable as a services based approach. Likely they just have a front application consuming web pages and churning the result in HTML.
  5. Sunday, 24 May 2015 17:19
    In general there is a lot of things to agree on - and i think there is by most developers and people involved around the core of the community a consensus about most of the things you write at this point - so ill only add a few questions for debate.

    Why Blade and not something like Twig?

    I think for frontenders Twig is more mature and easier to onboard.

    As for FOF in core - i think the real question might be Conventions over Configuration or the other way around - which Again leads to the usecases of what it should forfil - so is going conventions over configuration too limiting or will it enable enough for it to make sense - or perhaps even more important how do we ensure overridability to a point where it is not a limitation.

    We spent quite a lot of time considering FOF and ended up making redCORE instead because FOF was to much convention based for the complex sololutions.

    In redCORE we also have JSON/HAL (and SOAP/WDSL for that matter) with a super simple XML manifest configuration layer (and oauth2) - any of the 9000 joomla extensions can be supported by simple configuration in a XML manifest file - so while it does require someone writes a simple manifest file its a lot more powerfull than a convention based one if you are building more complex systems. ( https://github.com/redCOMPONENT-COM/redCORE )

    Again the idea is configuration over conventions so instead of adding a generic method for exposing content to JSON we get more complexity by being able to configure it.

    Beyond this i would also add ACL to the table - not just in terms of end users but also in terms of an ACL that actually holds its own from an arcitechtural point of view - if you have a J3.4 running with say 500 ACL Groups in it your modules, menus, content etc. is pretty much almost useless with a 30+ second wait time on all loads - ACL has to be completely rethought into something thats scalable and multidimensional - and perhaps while doing so also consider the workflow possibilities that lies herein.

    Finally on framework - id fully support using as many standard libraries as possible and focusing our own efforts into the Unique libraries that gives special or additional power to the CMS - so in part the J! Framework could be in most parts made up of existing libraries but those we maintain and develop should be those that ads something Unique to the table an existing one could not - and as i understand Michael that is in many ways his vision too.
    1. Sunday, 24 May 2015 17:43
      Hello Ronni,

      Blade over Twig because Blade is a syntax convention, Twig is a full stack third party templating framework. Blade has minimal performance penalty, Twig has a massive one. Blade is internally translated to PHP which is readable by a developer –in a location which is printed out in case of an error– whereas Twig doesn't.

      As for FOF and CoC, you haven't seen FOF 3, have you? Please take a look especially at https://github.com/akeeba/fof/wiki/The-Factory As you can see the default is to use the BasicFactory which is "dumb": if you don't create a class it won't do anything for you.

      As for redCORE it is NOT a RAD framework, despite your wishful thinking. It's similar to what I had done back in 2009, two years before FOF 1.0, i.e. a set of common methods on top of the legacy Joomla! MVC. It's what you do 1-3 years before you realize you need a more thorough solution. Basically you followed the wrong direction. Instead of making CoC optional and controllable you did away with it in most places. A hardcoded preference is a hardcoded preference no matter how you dice it. With FOF 3 I removed hardcoded preferences. There are overridable defaults.

      FYI FOF 1.x and 2.x where not good solutions. They were stopgaps. With FOF 1.x I tried to replace Nooku Framework in Akeeba Subscriptions with something that has a strong b/c policy. FOF 2.x was forced out of the door too early since I was asked for FOF to be included in Joomla!. It didn't happen at first, I merely added some features its users were requesting and got myself into planning / architecture mode. This resulted in two different solutions. AWF is the first try at something that's standalone. I wanted it to replace FOF but given the complexities of Joomla! integration it couldn't happen. So FOF 3 was born, doing away with b/c and focusing on proper architecture. Try it for a small project if you don't believe me. You'll see why I built it :)

      <blockquote>
      Again the idea is configuration over conventions so instead of adding a generic method for exposing content to JSON we get more complexity by being able to configure it.
      </blockquote>

      You are describing FOF 3. The convention is there if you want to get a prototype quickly. You can of course customise, either by selecting which fields to show or by overriding the entire JSON output format. It's up to you.

      Oh, BTW, having used XML forms and XML configuration files for any kind of views long before you do I can give you a fair warning. Do not try to make an infinitely configurable system through XML files. In the end of the day you'll be trying to create a programming language in XML which is a waste of time and reinvention of the wheel. You can to figure out the crossover level of complexity between XML and PHP and offer customization through XML only up to that point and not any further.

      <blockquote>
      Beyond this i would also add ACL to the table
      </blockquote>

      I agree, but I would leave that for 5.0 (if I included my ideas for 5.0 the articles would be larger than an encyclopedia). The big two "problems" with the current ACL system are a. explicit Deny trumps explicit Allow which is the opposite of the intuitive grasp people have on access control and b. the privileges are calculated on the fly and stored semi-temporarily in the session causing delays and requiring people to log out and back in everytime the admin changes something.

      One way around the second "problem" is caching permissions in the database instead of the session and updating them every time an admin changes something. This transposes the delays from the public site to the admin experience. Or you could have that with a CRON job or something like that. It needs a very long discussion. I don't see a right or wrong way to do that.

      Framework: I think we're all on the same page, even if we use different words for it. I just read this very interesting article which further enhances this position http://blog.dominodatalab.com/reflections-on-buy-vs-build/?hn=1 Basically, it says that it doesn't make sense to build something unless it contributes directly to your core business and you can gain advantage by rolling your own instead of using off-the-shelf components.
  6. Sunday, 24 May 2015 17:56
    Fully agreed redCORE is not a RAD - it might have started out with that intention in part but quickly became a Foundation for all of our extensions to allow for quicker output for client jobs by having to spend less time on the same basic tasks time and Again.

    However we also use it as a playground where Joomla itself is not quite there yet and as with elements of our Multilanguage, Webservices etc. we try and lead a way for something that can be adopted back to Joomla in some or another form as it happened with the sublayouts.

    So redCORE will remain for our specific needs (newest addition is payment handling across all of our extensions) and will not have the goal if being a RAD - but our RAD :)

    I did not check FOF 3.0 out yet - but i see that the exact reasons for why we hestitated to go with FOF 2.0 is out in 3.0 so that is interesting :)

    As for Twig vs. Blade - i understand the points and its important for backenders (and ressource useage etc) yet for Frontenders and Siteimplementors i fear the knowledge level in terms of php might be too high.

    Ideally in the solutions we do today and in the future you can relatively quickly deploy Joomla as your base - install a few 3rd parties for core functions of the client (say CCK or Ecommerce or both) and then setup webservices to built the value chain for your client so its tied into PIM, Masterdata, DAM, ERP, CRM etc. as needed so the solution is not a standalone but part of a vibrant living value chain (which could ofcause include any IoT devices).

    To do so we need a very scalable and flexible Foundation where you can then relatively quickly map data in or out and via some form of templating map those data to be used on the site with as Little Work as possible - stored locally to make it independant of live webservices but attained through webservices from many sources and utilized on the site.

    I really think the solutions of tomorrow will be solutions of chained systems and we need to adopt towards that - weather it be more simple or in complex and large organizations or cooperations the right design and arcitecture will assure that its possible.
  7. Sunday, 24 May 2015 18:25
    OK, makes sense now. Last time I'd checked –arguably a good year ago– you were saying it was a RAD framework. I'm glad we agree on what it is and it isn't :)

    <blockquote>
    I did not check FOF 3.0 out yet - but i see that the exact reasons for why we hestitated to go with FOF 2.0 is out in 3.0 so that is interesting :)
    </blockquote>

    To be honest, FOF 1.x/2.x wasn't suitable for Akeeba Backup and Admin Tools either. I had to shoehorn these components into FOF 1.x/2.x because I needed the Joomla! abstraction. After the successful AWF experiment I decided to take the experience I had gained to Joomla!. Now I just have to convert every single component. Six weeks since the first FOF 3 beta we are two down, one in progress, four to go :D

    <blockquote>
    As for Twig vs. Blade - i understand the points and its important for backenders (and ressource useage etc) yet for Frontenders and Siteimplementors i fear the knowledge level in terms of php might be too high.
    </blockquote>

    Have you used Blade? There's hardly any PHP. If your developers are good and methodical they can separate the templates into small bits forming a template hierarchy that's easy for front-end developers and designers to handle.

    <blockquote>
    Ideally in the solutions we do today and in the future you can relatively quickly deploy Joomla as your base
    </blockquote>

    That's the idea. A lean core which allows you to install off-the-shelf software, configure it rapidly and integrate it without much fuss. The power of deep integration is the differentiating factor against WordPress and the ease of it is the differentiating factor against Drupal.

    <blockquote>
    stored locally to make it independant of live webservices but attained through webservices from many sources and utilized on the site.
    </blockquote>

    That's a laudable goal but not the main focus. Go back to the first post of this series to see what is the target audience. I understand that what you express here is your ideal mode of operation for your company</em> but you're not the typical Joomla! user any more than I am. We use Joomla! differently than our target audience. We must make a rock solid, good product for them. If we can add the features which will help <em>us</em> too, more power to us. I believe we <em>can do it, just not as our major focus. Otherwise we'll end up building a CMS based around web services and a Javascript-powered display/templating system which is alien to our target audience.

    Reality check. We're racing for the second place with a chaotic difference to the guy in the first place. If we shoot for the moon we'll end up crashing to the wall. I say let's get a foothold on the second place and try to regain some market share to strengthen our position. Anyone who wants to shoot for the moon is welcome to fork off and try their lack. They don't have to take everyone else down with them.

    <blockquote>
    I really think the solutions of tomorrow will be solutions of chained systems and we need to adopt towards that - weather it be more simple or in complex and large organizations or cooperations the right design and arcitecture will assure that its possible.
    </blockquote>

    If you honestly believe that you want to diss the existing user base and try to build a product for a crowded market, five years too late, then please do fork. Unless this is not what you mean.

    All the "IT future" predictions have a long tradition of falling far off reality. 640Kb of RAM should be enough. Apple will be sold to IBM. RAMBUS is the future of RAM. HTML5 applications will supplant native mobile apps. Have you ever heard of them? They sound ridiculous now but when they were uttered they sounded more than certain.

    I do</em> see that web services will continue to play a great role in the mid-term, but not for the kind of small to medium sites our target audience is interested in building. I also do see that native mobile applications will continue to dominate in the short term and possibly beyond (barring a massive paradigm shift nobody has predicted yet). It is therefore to our best mid-term interest to produce a product which creates glorified HTML sites but <em>does allow data export AND interaction through web services to power the mobile applications of the future.

    Of course your company can use these web services features to create a richer web experience. Nothing stops you. But YOU are not the core target audience of Joomla!. You are the exception proving the rule. We can somewhat cater for you but if you really believe that the future lies exclusively in web services you should not be basing your business exclusively on Joomla!. It sounds like what you really want is a collection of web service providers built on Symfony and a Joomla! or custom site (probably Joomla! to reduce costs) running custom code consuming these services.
    1. Sunday, 24 May 2015 18:44
      I agree we are not the core audience from the bottom - but the goal should be that we can take Joomla and use our own redCORE + Extensions and other 3rd Parties to do what we need - while the core of Joomla is not something that prohibits that.

      Then we can remai open for all as we are truely scalable and extending.

      I dont think IoT is a matter of time - its already here - integration of JSON to and from mobile apps has happened for years (we run many native apps from Joomla sites and we use redCOREs webservices for it) but there is so much more to it than just smartphone apps, which in it self would truely be a goal that pointed backwards and not forward.

      I thouch base with many clients every year from the small ones to the big ones and across all of these there are some requirements and needs that stay in focus.

      A website is no longer just a Html site - its a place where our business (weather it be ecommerce, consultancy of craftmanship etc.) is open when our real business is closed - i.e. the opening hours of a business might be from 8 to 16 - but the website should extend that through information, selfservices etc. 24/7 and help to digitize our businesses and make it relevant even out of normal business hours - the days of the business Card in html is dead.

      To do so - even small Companies need their website to be part of something bigger - a value chain - in their business - for the last many decades that value chain was more or less built around their ERP and often most other systems where in part manual or driven by manual processes.

      The integrations are many and in multiple directions, but at the base of it, it reaches beyond just smartphone apps it reaches out to all sorts of apps and systems and to truely remain relevant we need to make the best web application thats good at the core of it - while allowing it to be extended and Integrated so it can help form value chains - where those chains create value for the end users (businesses and organizations).

      I think adding business logics and value chains to businesses and organizations is beyond any specific technology or technological claim - its a Foundation of the future - and where we add the most value to the end users businesses and organizations.

      Then we need to make sure that the part of value chain that Joomla has is as optimal as possible and as best as possible and a relevant part of the full value chain.

      There are many examples of this and what we see today is most of these a emerging and do custom integrations 1 by 1 in a static manner to specific other systems - but having an infrastructure that allows even a site implementor to tie in the Joomla website into a value chain empowers the people who build websites to be relevant also in the future.

      I dont think in any way that what i am experiences on a daily basis is excluding from anything what so ever written in your posts sofar - i just look at Little ahead as why we need webservices and why we need a scalable core based on the 100's of businesses and organizations i go out to every year and talk about their needs and how we can help transform their website from something thats more or less stand a lone to something thats a living part of their organization.
      1. Sunday, 24 May 2015 18:57
        I agree we are not the core audience from the bottom - but the goal should be that we can take Joomla and use our own redCORE + Extensions and other 3rd Parties to do what we need - while the core of Joomla is not something that prohibits that.
        I agree and I did make the point that we should facilitate you as long as it doesn't get into the way of the rest. Obviously, as long as you have redCORE and Joomla! supports the format=raw you lose no functionality. Win for everyone :)
        I dont think IoT is a matter of time - its already here
        I can hardly imagine an IoT platform built on Joomla!. You need scalability that Joomla! cannot offer. The kind of scalability you need for this use case is beyond what Joomla! should be focusing on when 99% of its users will run it on a single –often shared!– server.
        I thouch base with many clients every year from the small ones to the big ones and across all of these there are some requirements and needs that stay in focus.
        Completely different needs for web services catering for a limited need and web services of IoT scale.
        The integrations are many and in multiple directions, but at the base of it, it reaches beyond just smartphone apps
        Which is basically what I'm saying in the post about web services in different words. Again, the thing is NOT about making web services possible, it's about scale. Adding a RESTful JSON API (or even a SOAP API if you're a raging masochist) is not the hard part. It's dead simple. Scaling it is hard. Joomla! is not apt for that kind of scaling. Sure you can work around that with a custom database driver which uses different servers (or server pools) for read and write, tricks to move static content to a CDN and put multiple instances behind a load balancer, but this is not the core business of Joomla!.
        Then we need to make sure that the part of value chain that Joomla has is as optimal as possible and as best as possible and a relevant part of the full value chain.
        Achoo! I'm allergic to big corp newspeak. To put it abundantly clear: in the post I DID say that web services can and MUST be made available to open up all those possibilities that took you 500 words of big corp newspeak to not describe. What is it that you do not understand? I have to repeat myself yet again and tell you that just because web services will be possible it DOES NOT make Joomla! a great choice for Internet of Things scale platforms. IoT platforms are supposed to manage the data flow of thousands to millions of connected devices. Joomla! is simply not built for scale. YOU can work around that and all the power to you. But this doesn't mean we should add these scaling features to the core because they will affect the performance and complexity for the 99% and, frankly, it's something that the 1% can provide. I'm pretty sure your company can write a redSCALING solution with the necessary plugins and database drivers to make Joomla! scalable. I know that this has been possible since Joomla! 1.5. Joomla! 2.5 and beyond only made it easier...
        1. Sunday, 24 May 2015 19:17
          I think your logical reasoning is fine if you believe that IoT is only for big cooperations and big data.

          I dont.

          IoT will be something everyone will be using in all Company sizes and even privates - Internet of Things just Means that we will see a Whole new generation of devices that will be a part of our solution chains.

          Today we do responsiveness based on viewports but in the future responsiveness also Means that if a client (even a Little Company) are utilizing some IoT devices they will also be a part of the value chain for that Company.

          So its not just about scalability - yes the BIG FISH will scale up in masses and they need something entirely different to handle BIG DATA - but even Little fish will require handling of Little data.

          So IoT and webservices as a whole is a lot more complex and need an even higher abstraction level and ways to allow for configuration, adoptation, integration etc. onwards.

          So id suggest you reconsider what IoT is and its relevance even to small companies and organizations - and consider how much a part of the future value chains of those companies and organizations Joomla will be and what is needed to do so - in reality i dont think its a lot more than is on the table but its important our scope for webservices is focused on that and not so much on smartphone apps - there isnt much market disruption in getting years late to the party.
          1. Sunday, 24 May 2015 19:34
            I don't think that IoT means what you think it means. Can we all agree that it means what the 121 references in WikiPedia define it to be? http://en.wikipedia.org/wiki/Internet_of_Things

            Yes, IoT will pay a very important role. By consuming data produced by IoT enabled "things". Which you can already do. Having a small</em> company build its own IoT platform is a bit idiotic unless it offers them a unique advantage. How would that apply to a <em>small</em> company is beyond me. I hardly think that a <em>small</em> company would benefit, for example, by implementing its in-house fleet management instead of using a third party solution and <em>consuming data into its own site, e.g. to let clients know when their order will be arriving with greater precision than "sometime between 10:00 to 18:00". This is something you can already do.

            <blockquote>
            So id suggest you reconsider what IoT is and its relevance even to small companies and organizations
            </blockquote>

            I already have thought of that. It's 80% consuming data and 20% making data available for interconnectivity. In my example above an e-commerce application on your site would be able to export pick lists to your WMS (which would also notify the site if something is low or out of stock). Once the picking is complete the WMS would link back to the e-commerce which changes the order status to "ready for dispatch" and exports the delivery orders and address information to your fleet management which generates the delivery schedule for your trucks/vans/drones/motorcycles/whatever and pushes them to the tablets/PDAs of the delivery people. As they move along their route their location and schedule fulfillment is pushed to the fleet management which sends this information back to the e-commerce site where the client can get an accurate prediction of the delivery time (probably down to a 10' window) and even get sent alerts by SMS, e-mail or push notification.

            How does Joomla! play a part in this? It's running the e-commerce application and facilitates 1. the export of data (pushed to a web service or pulled from its web services) and 2. the import of data (again push or pull). Pushing data out and pulling data in is already possible. Having external services push data in and pull data out is the missing block, hence the very relevant point in my post about web services.

            Where Joomla! doesn't fit this model? It's mostly unsuitable for use as the fleet management system. In order to serve the 99% we need to manage a session for every request, not to mention bringing up a rather thick layer of internal objects and services. This means bad performance and increased server load.

            Moreover the fleet management requires some data crunching that is simply unsuitable for PHP and much more suitable for a functional programming language like F#. Essentially the big data or little data crunching begs for a different programming language and a different database technology, putting this outside the domain of Joomla!.

            As for the WMS, it can be based on Joomla! for small scale operations. It may be cheaper to do that too. It just won't cut it if you're talking about massive operations and I'm not talking about behemoths like Amazon but even the few thousand items per day volumes of many medium sized businesses. I have one particular case in mind, one that I worked with for the better part of 6 months (pharmaceuticals) optimizing their operations.

            Does this example explain better what I have in mind when I say that Joomla! doesn't scale as an IoT platform but should still provide interconnection?
  8. Monday, 25 May 2015 01:22
    Nic, I think you needed to break this post up. It's too big to comment in one huge gelatinous mass :)
    1. Monday, 25 May 2015 10:48
      Yeah, I know, it's a longer than I expected. You can try posting multiple comments. WordPress has threading in the comments :)
  9. Monday, 25 May 2015 15:04
    Ok, here goes.

    UI/Behavioural standards

    This is THE issue in my opinion that is going to plague the CMS template industry for many years (regardless of which one). And even though I agree a common JS library (aka Interface) is probably a sensible direction, it's still compounded by the fact that it may need to be woven into data-binding solutions like Angular or Ember and friends.

    This is a really hard problem to solve. I do know that it's not going to be solved without a lot of stakeholders at a table, and I think the project needs to be responsible for organising it.

    Layouts

    My preference would be to see true markup-only layouts and move away from any type of PHP based system that allows a novice to "hack it because they can". I personally love Handlebars and partials. In this case, the less power the better.

    PHP

    It's time to stop listening to the one or two prophets of doom that keep saying Joomla will collapse if we increase the minimum PHP requirements. The policy should be that the minimum required is the lowest supported version of PHP that PHP has released, plus some buffer (so today that would be 5.4.a-few-less-than-current).

    MySQL

    My wishlist includes changing the ORM to be able to support a file based data storage system, or Mongo, or even memory. Node does it so PHP can do it.

    UCM

    UCM is exceedingly powerful (ala Elgg) but I agree it's either an all in or all out thing. I do think the basic Joomla "content" unit should have a core way to extend itself and let extension developers duke it out over who can provide the easiest UI to use.

    Core Extensions

    I've written passionately about this and the possibilities distributions would open up.

    I do think we need to remodel the whole extension installation thing though. The way WordPress's presents to the user and functions is much better in my opinion.

    Namespaces

    Yep. Get rid of
    JLoader
    as fast as we can.

    Joomla 4

    There are two ways to do this. Either bring in new code now with namespacing that begins to form a natural b/c layer within the 3.x source. Or, build 4 with a legacy layer ala Joomla 1.0 to 1.5 style.

    Pros and cons either way but I think the important point is to avoid another 1.5 to 1.6 transition.

    Composer

    No brainer.

    DI Container

    Here we get into some trouble because DIC's have been abused heavily. In my opinion the container or service locator is fine as long as it goes no further than the Controller level.

    MVC

    I really, really like Paul Jones' ADR paradigm and I think it fits what most extensions are trying to do.

    There needs to be a much better separation between "request" and "response" and the "bit" in between (be that MVC, ADR, pick your poison) should not care. This is something the Node people are doing much better. And while I'm at it, I'd love to explore the concept of "middleware" to replace most of the abuse that suffers at the hand of the current plugin system. The Command Pattern would be a far more powerful replacement.

    As for FOF3 (I can't resist), no - I've already got two versions of FOF installed in my sites, I don't want a third :P (joking!).

    JSON

    Completely agree.

    Schema

    Yeah that's fine, as long as the code is built with interfaces that allow other file and schema formats in mind.

    Extension Installation

    This is one area where I do need to harp that the code needs to be SOLID.

    Further, I think we need to completely rethink the concept of the extension. I think WordPress is doing it right with just two (plus language packs). A "plugin" should be able to register what it can do with the system. A "theme" (or template) is worried about display. Simple.

    Menus and Routing

    Shooting from the hip:

    Every extension must define it's unique resource routes as a minimum, and these are only allowed to return an object that can be converted to JSON (bingo, you have your REST API).

    Navigation marries a custom route to a page layout (not necessarily a component like it does now). Perhaps page layouts are merely a "structure" to display a number of "modules" arranged in a particular pattern.

    Perhaps custom navigation routes are always fully pre-computed (we have menu router generators) so that "menu" routing is just a simple db lookup.

    Maybe for pages that are truly static, Joomla is able to fully generate those pages and serve them statically (ala Github pages style).

    Testing and QA

    So here's where we might differ a lot. You can't check for regressions unless you have code coverage over the "behaviours" of the code. But what I was taught about coverage and how to test was completely wrong and in the last year or so I've rediscovered how to do testing better (check out anything Uncle Bob Martin has written or recorded).

    It's a long conversation but you actually want to be aiming to get close to 100% code coverage when you have tested all of the "behaviours" or "use cases" that are supposed to be achieved. The difference to the way I was taught is that coverage is the sum of all the different types of testing - in fact, it could be achieved with no unit tests and only a modest suite of integration tests.

    UI testing is hard and should be done sparingly.

    I cannot impress enough that we should be aiming to achieve true test-driven (red/green/refactor) development practices. It's not easy to learn, hard to master, but don't we all want to be craftsmen when we grow up? It is achievable and there are significant productivity benefits in going that way.

    Yes, I know, I know. You all think I'm crazy. I would have agreed with you once too.

    Joomla Framework

    So here's the thing. Regardless of where the Joomla Framework has been, it's what Joomla needs now that is important. Nothing else matters. To that end I think there is a real need for "a" team to be responsible for the engine-room level direction and decision making, particularly when we are thinking about probably including more and more code with Composer (in fact I'll argue most of the code in the CMS should live in
    /vendors/
    ). Someone needs to make sure they work together and simply keep the stupid things updated.

    Call that "Framework" or whatever you want, I see there is a need for a team to be responsible for that. Above that, you have the team that takes that code and makes what we call the Joomla CMS and the extensions. And there are obviously other specialties but I'm currently leaning towards the idea that the library level code is a specialty in its own right.

    Whatever it's called, everything needs to be designed to be modular with the lightest coupling possible.

    Anyway, that's a very long conversation that so few words can't possible give justice to. I'd love to chat to you Nic about how we can actually develop a plan to move Joomla forward (sounds like a good name for a campaign).

    I think what's lacking in the leadership at the moment is a solid plan of action, but we've already been over this territory. Personally I'd like to see an immediate API freeze as of 3.4 (yes, I know that will affect some plans, but the idea that we are going to try to "fix" that router again scares the living daylights out of me) so that we don't start adding more crap code that we just need to throw out in a few months.