Joomla! 4 and Beyond: architecture and design

In the previous two installments of this series we discussed the target audience for Joomla! 4 and beyond and the vision for the end user. In this third installment we’ll see things from the developers’ perspective, defining a vision for the PHP code’s architecture and design goals.

Accessibility

I’m told that Joomla! needs accessibility improvements. This is apparently required for using Joomla! in the public sector in many countries. This is not my area so I guess this requires input from accessibility experts.

A modern CSS framework

Joomla! 3 is stuck with Bootstrap 2 for ages. Even worse, it’s not even Bootstrap 2, it’s out own botched down version of it. Maybe that made sense in 2011, but in 2015 it’s like trying to use smoke signals when everyone else is texting. We need a modern, up to date CSS framework.

There are two school of though on this. One school of thought says that we should develop our own CSS framework abstraction or even our own CSS framework, period. I disagree with this approach because it requires rare talent we barely have and it’s not guaranteed that we will continue to have in the future. In simple words, it’s no longer a great idea if you can’t guarantee the longevity of such a project.

The other school of thought is effectively Joomla! growing a pair. Bootstrap is a great framework and we should stick to it. As in: pledge to always use the latest version of it, at most 6 months after its release to cater for Joomla!’s development cycle. We should NOT be afraid to break b/c with regards to updating to the latest version of Bootstrap. I know I was against this four years ago, but I’ve seen where staying rooted in an obsolete CSS framework got us.

JLayout for the win

JLayout is a great way to abstract some of the complexities of using different CSS frameworks. Please keep in mind that it only addresses static content, not Javascript interactions which require knowing a. the DOM model of the framework in use and b. the availability of helper Javascript functions. Essentially, JLayout is an HTML abstraction layer.

What if we could use JLayout to also provide a Javascript abstraction? I know that some JLayout templates already load their own Javascripts but these only address the Joomla! core needs. If I’m a 3PD who wants to display buttons dynamically or change the label color on the fly, though Javascript, my only option currently is to make AJAX calls which produce a new HTML page and replace the contents of the page the user is currently viewing. This is wrong and impractical for so many reasons!

Let’s develope a common Javascript library, proxying to Bootstrap. If a developer decides to use CSS Framework XYZ they are obliged to provide an override for this common Javascript library. This allows 3PDs to support different templates without the ugly approach of force loading a namespaced copy of Bootstrap (yeah, I do that shit myself…) or inventing their own CSS framework which doesn’t look “right” in a third party template.

Supercharge view templates

One of the innovations of Joomla! 1.5, back in ’06, was the introduction of view templates which separated the code which does stuff from the code which renders stuff. That was great… nine years ago. Have you taken a look at our view templates? There’s an awful lot of PHP code in there. Front-end developer need to have much more than a superficial understanding of PHP to style things.

Meanwhile in the PHP world Laravel happened. And it has these stupidly simple to understand Blade templates. Oh my, he nailed it! It’s powerful as heck and easy for front-end developer to understand and customise without going into PHP specifics. And yes, it can be ported to Joomla!, I have already done that.

The Blade syntax allows us to simplify the lives of developers even more by adding support for constructs like @jlayout(‘com_example.foo.bar’, $this->items, $somethingElse) Compare this to the typical JLayout invocation in core code and you see where this is going. In the words of Darth Sidius: POWER! UNLIMITED POWER!

PHP 5.4 and later

PHP 5.3 is dead since August 2014. PHP 5.4 will be dead in August 2015 but at least we know that all shared hosts support it – except those which shouldn’t be allowed to be connected to the Internet! – at least as an option.

PHP 5.4 allows us to write leaner code with less duplication using Traits. This is the practical reason behind this suggestion. We could easily shave off 15% of our core components’ code and reduce the amounts of bugs we’re dealing with. Double win.

MySQL (and compatible variants) only

I know that this will upset the 10 people who are using Joomla! with Microsoft SQL Server and PostgreSQL. There are several practical reasons for doing this, though:

  • Query optimization. Our queries are suboptimal and lead to bad performance, especially on larger sites. If we know that the core is only supposed to work with MySQL we can use the expertise of MySQL DBAs in the Joomla! community with a long standing offer to help to make Joomla! fast. That’s one hell of enterprise feature and major marketing point.
  • Reduce the barrier to entry and errors in deployment. If someone delivers a new feature with schema changes they’re supposed to deliver these changes in MySQL, PostgreSQL and MS SQL Server format. In two different locations each. This puts off developers who are very experienced with PHP and MySQL but can’t afford to learn and test against PostgreSQL and MS SQL Server. Even worse, the PostgreSQL and MS SQL Server schema changes usually get merged untested and errors are only discovered long after the release.
  • We don’t have the people to support 3+ database server technologies. Proof of that is that the PostgreSQL integration was broken for the longest time (at least 1 year) and nobody noticed. Even when we noticed, it took several months to get it fixed. I still have no idea if the MS SQL Server driver really works.

So let’s just drop support for additional database types. We can leave the drivers for them in the core (Joomla! Framework?) for 3PDs and enterprise users to use them. The core will only support running on MySQL.

As for MySQL itself, since we can’t raise the minimum requirement to MySQL 5.6 yet we’ll have to put up with core tables in need of full text search support to stick to MyISAM (and chop the head off any moron who suggests converting them to InnoDB).

Get rid of UCM

UCM was half baked, is half baked and will forever remain half baked, so let’s just pull it out. It is NOT required for Tags and Content Versioning. We just need tags assignment and content assignment tables with one extra field denoting the content type. Then we can use something like Laravel’s “morphable” relations to make them work. If you read Laravel’s documentation you’ll see that this is the use case for morphable relations, after all.

Bonus points: this greatly reduces the content duplication and clutter of core tables, simplifies queries and speeds things up.

Remove a lot of core extensions

Per the existing plan. One thing we currently suck at is advertising to our users the core supported extensions. I think that the Install from Web landing page should have two areas: core supported extensions and a curated list of commonly sought after JED categories. This allows us to tell users to go there to quickly add the features they don’t find in the core. This approach works perfectly for WordPress: the landing page of the Add Plugin page is essentially a list of Automattic owned or sponsored plugins.

Namespaces everywhere

Since this is a major version I believe we can finally break b/c and provide fully namespaced code. This will also reduce our code since it will be finally possible to extend a front-end MVC class from a back-end MVC class. Previously that was just not possible because they both had the same name.

Moreover we can completely do away with JLoader. Instead, use Composer’s PSR-4 autoloader which is already shipped with Joomla!. This will do away with bugs in 3PD code caused by core classes moving from libraries/joomla to libraries/cms, or classes which do not follow the current autoloader’s scheme.

Backwards compatibility layer, at least for 4.x

A new version of the CMS is useless without 3PD extensions. It’s our job as developers to ease the transition of code written for Joomla! 3 to Joomla! 3. This can be done by providing a compatibility layer which adds class aliases and backports of obsolete code for the most commonly used core APIs. This will allow developers to work their way towards Joomla! 4 compatibility without feeling they have to start from scratch.

It would pay great dividends if we also document the conversion process for existing code. If anyone reading this complains about the feasibility of such a plan I have the page which proves that I put my documentation where my mouth is.

Better Composer integration

Move composer.json and composer.lock into the libraries folder, not libraries/vendor. This allows adding requirements without essentially hacking core.

Allow 3PDs to add requirements to Joomla!’s Composer installation during the extension installation / update. We need to see if this is possible on shared hosts without CLI access – for context see this Drupal 8 discussion. This needs to be tied to an overall com_installer refactoring. The feasibility study should concentrate on whether it’s possible to make the dependency discovery and individual dependency download and installation separate steps which can each complete inside a single page load.

Finally, drop all “invented here” core libraries in favour of Composer dependencies (especially things like JImage, JGithub etc).

DI Container

Ah, my favorite pet peeve. Right now we have this magic global black box called JFactory. Get rid of JFactory, replace it with a DI container (something like Pimple?) which is then fed to the application object. Anything you want can be fetched in a manner similar to Joomla\Application::getInstance()->getContainer()->user.

I also suggest taking it a step further. Each extension type gets its own container type which gives access to the main DI container and local dependencies that make sense for the specific component type. You should be able to spawn the Container of an extension from anywhere, e.g. Joomla\Container\Component::getInstance(‘com_something’). This allows for painless HMVC. Been there, done that – in FOF 3.

Overhaul MVC

I know that the Joomla! Framework has a “new MVC” which I call “no MVC” because this is what it feels like. I disagree with its approach for mass distributed software like Joomla! and its extensions. I do understand why you may need this for decentralized applications. I suggest keeping it in the core for these use cases but not using it as the default development paradigm for Joomla!. Remember what our target audience is? Exactly.

Instead, I propose following Laravel’s lead, without the façades – a major reason why some serious PHP developers hate Laravel. Let’s do away with the Model vs Table separation. Create a Model for non-data-aware stuff and a DataModel for data-aware stuff. using a vocabulary and feature set similar to Laravel’s Model + Eloquent is a good idea. I’ve already done the work in FOF 3 and made my development a breeze. Less code, faster execution, less and easier to fix bugs are the immediate benefits.

Also, as I said above, add Blade template language support to our Views. I am half tempted to propose allowing views to be defined with XML forms –which lends itself to RAD features such as scaffolding– but unless you’re willing to essentially make a RAD framework the core default it’s unadvisable.

Well, OK, what the hell. Let’s just put FOF 3 into the core. I already know that the biggest drawback will be for me. It will however benefit everyone else by having a RAD framework in the core. Right now we bleed developers to Laravel, ZF etc because it’s easier to develop there, even though you have to do everything from scratch. If we remove that “coding for Joomla! sucks” feeling we can regain a lot of that audience. That audience does indirectly or directly drive the choice of a platform for big business’ new sites so there’s a plan to increase our market share.

JSON API with virtually no code

If we could allow access to the CMS data (and data management) though a JSON API we would let advanced uses of the CMS. For example using Angular.js to render pages, remotely managing the content, systems interconnection, easy mobile application development etc.

The groundwork for easy JSON views with HAL support is already done in FOF. No matter if it becomes part of the core or not the HAL support is something we have to put in. I know we already have com_ajax but it’s only useful to do AJAX like it’s 2008 all over again. HAL allows API consumers to discover content more easily, without having to know the exact inner workings of the server.

Improve the schema installation and updates

At this point each core database change requires editing two files, one for updates in the com_admin component and one for installation in the installation application. This is error prone. This is a problem already solved with FOF’s FOF30\Database\Installer using XML files which allow the installer code to create and/or update the schema automatically. It would also simplify the Fix Database feature: you can use the same code as the schema installer/updater.

Since this was already discussed for inclusion in Joomla! 3 I think it’s a given?

Improve the extensions installer

Last year I’ve written a whitepaper on it. We can and should add support for package signing as well. Some specifications work is already done by David Jardin. I’m sure there’s a revised version somewhere butI can’t find it.

I know, you think I’m done with this big subject in 30 words. Well, the two documents are about twice as big as this entire blog post 🙂

Decouple menus and routing

The current menu system defining routing leads to duplicated content and some impossible to solve problems. If a leaf node (e.g. article) can be routed through two or more menus there’s no way to know for sure which URL you should follow unless you also know which Itemid you should use too. If no Itemid is provided chances are the “wrong” one will be picked, leading to the “wrong” modules being loaded.

Fixing that MAY cause some problems with our module system. If every leaf node article has a canonical URL you might end up again with the “wrong” modules being displayed in some situations. For example if I have a menu item for an entire category and another menu item for one of its articles (with different module assignment) then clicking on that odd article will yield an entirely different page layout than the other articles of the category. This is not necessarily bad, but it does change user expectations compared to Joomla! 3 and earlier.

Finally, com_redirect actually becomes important since it’ll be the easiest way to add custom URLs. In Joomla! 3 and earlier the typical method was creating items in hidden (no module for them published) menus instead.

I do need feedback from people who have a better grasp on routing.

Testing and QA

Upgrade PHPCS rules to PHPCS 2 format. This allows contributors to auto-fix their code style (the bane of our existence for large contributions).

Unit and functional tests should be written for our common use cases first. The short term goal is not 80+% test coverage, it’s making sure that regressions don’t happen.

Joomla! Framework

I think that we can now all admit that trying to create a generic PHP framework to compete with the likes of Symfony, Zend Framework etc was a waste of valuable resources. So what we should do with Joomla! Framework?

One idea is to completely fold it back to the CMS core, much like Joomla! Platform was before we called it Joomla! Platform. I think that’s the wrong approach because it ties CMS development with framework development. This is not always a good idea. The framework is basically an R&D lab. You just don’t put R&D under production.

A better way to approach this is as a semi-detached project. The goal of JF is to provide a stable platform for Joomla! (the CMS) to be built on. This requires a major shift in JF’s development since it has to keep in mind and cater for the complexities of mass distributed code. For example, the Uri package just can’t ignore the need for a forced $live_site instead of consulting the $_SERVER superglobal. But having it separate from the CMS allows it to both do R&D at its own pace without affecting CMS development and lets anyone interested use it outside the CMS.

As for Joomla!, it can pull in JF packages through Composer. This should be the one and only way to do it, not the current situation of having a human copy .php files from a gazillion Git repositories into the libraries/joomla folder.

I’d like to discuss these ideas further with Andrew and Michael. Perhaps we could do a joint post on the future of the Framework. I think that if we frame the framework right (pun intended) it can be a successful child project of the Joomla! organisation. Basically, we need to (re)define its target audience and discuss how it can finally move forward.

47 Replies to “Joomla! 4 and Beyond: architecture and design”

  1. Hi Nicholas,

    I don’t know whether it is the right place or not, but I need to suggest adding more plugin events for database queries, to be able to customize the query parts, does it make sense?

    Thanks

    1. No, this has a massive performance impact and security implications. It makes more sense to have plugin events in Controllers and Models. This is the approach we have been using in FOF. It does do what you have in mind.

  2. CSS Framework + JLayout – With some tweaks to our inner APIs we can be at a point where there isn’t a hard core dependency on a single framework. Why is this important? Designers aren’t always using Bootstrap, and quite frankly I think our override system (extensions, media, and JHtml) is strong enough to allow this. Yes, those folks using another library need to be prepared to do a bit of heavy lifting to add the support, but the possibility of eliminating vendor lock on one framework vendor or version is real.

    Enhanced templates – Huge +1 here. In a sense, JHtml kind of provides these helpers, but they aren’t a native templating thing. Actually, there isn’t anything resembling a proper templating engine in the CMS today. I think it’d be pretty awesome to get something like Blade into the CMS, and the sooner the better.

    PHP bump – So I’m willing to go so far as to suggest that new features (that are opt-in) be written toward 5.4+. The first thing that would focus on for me would be the MVC layer, as that is an entirely opt-in thing. Maybe that new MVC layer can be built with Blade templating (or whatever engine) built right into it.

    Database Platform – The CMS was built on MySQL and the support for the other drivers is a major hack at best. As an application, I think we’re reaching the point where we need to pull the plug on support for them. Keeping the drivers is a good idea though, in the end our database API should be platform agnostic but this shouldn’t force extensions to do the same (and hurt performance or development; you can use Doctrine all day long and still optimize on MySQL).

    Namespaces – I say legacy MVC don’t even consider it, the new MVC layer should force extensions to be namespaced. I say we also need to settle on a namespace structure for the CMS (whether that be using the top level Joomla, using Joomla\CMS, JoomlaCMS, or whatever else) and begin migrating more code into the namespaced environment.

    DI – We’ve had a DI container since 2012 (it shipped with 3.2). Extensions are getting on board with it, core still loves JFactory. Not saying our DI container is perfect, but it’s there and we need to put it to use ASAP.

    MVC – So this one’s a bit interesting. At a Framework level where the components are distributable beyond the CMS, I think the “No MVC” layer works OK; it leaves it to the application to build their CRUD implementations. In the CMS though, it definitely isn’t an option to ship without a CRUD layer. I see no issue building that layer on the “No MVC” code as meeting the interface requirements is dead simple anyway.

    QA – The Framework’s actually doing somewhat better than the CMS right now. A majority of the packages have adequate coverage (http://framework.joomla.org/status), I think our issue right now is what the quality of those tests actually is and whether they can catch potential regressions or issues. Tests in the CMS are a lot more tricky to write (especially once you involve JFactory) and in some cases the classes themselves are just borderline untestable. So this part needs multiple steps to implement, including identifying poorly covered APIs (most) and identifying code structures that need refactoring to improve testing (and the structure overall). On the PHPCS 2 thing, that’s already a work in progress in the main coding standards repo and hopefully something that can be finished soon.

    Framework – No, we can’t compete with full stack frameworks. At the same time, I think the top level code is versatile enough that it can be an option for smaller projects where those frameworks could be overkill. So I wouldn’t necessarily go in a direction where the Framework HAS to carry support for all the CMS features. In 3.4, the Framework’s URI package was merged in and JUri was adjusted to extend this keeping support for all the CMS features (like $live_site) while enabling the parent code to keep its agnostic approach.

    The Framework API has taken a modular approach. I kind of like it this way. If we keep it where the Framework should support the CMS’ needs (but doesn’t necessarily carry the full feature set, like my thoughts above on the URI code), it lets us focus on improving the structure of those components for the CMS with the side effect of making them usable beyond the CMS (like our issue tracker).

    As for what exactly the Framework offers, it needs a diet. I’d suggest keeping our GitHub package (it’s used in production in several of our own tools, they don’t offer a SDK like the majority of our other packages have, and quite frankly compared to KnpLabs I think our package has a more natural feel of the GitHub API, plus a lot of the packages seem to be platform specific), but the rest of our API adapters at this point fall under the “well intended but unmaintained” category and should be allowed to rest peacefully. Our String package has over 12K downloads on Packagist in ~2 years, no idea what the source is though. Considering it has our UTF-8 compat code in it, and UTF-8 is still tricky in PHP, I think this is a sign of success.

  3. CSS Frameworks: I agree. We also need a common Javascript library, otherwise using Bootstrap makes no sense to begin with (Carousels, tooltips etc require Javascript initialisation).

    Enhanced templates: Adding Blade support only requires modifying Views to through a view template engine. I’ve already written the code.

    PHP bump: Go all in or not at all. The MVC layer does NOT require PHP 5.4. I’ve already done the work in FOF 3, that’s why I’m so sure.

    Database Platform: We need to pull the plug. Keep the drivers in but don’t require the CMS support any db platform except MySQL. You can NOT fully optimise queries in a db-agnostic manner. At best you’ll end up with the crap we currently have.

    Namespaces: Fully agree. I’ve done that in FOF 3 and saved me a lot of headaches.

    DI: I know. It’s not used in core and nobody is using it. I have no idea if it’s maintained or not, that’s why I put Pimple with a question mark 🙂

    MVC: Two reasons not to do that. First, you’re tying the CMS MVC to the Framework MVC. If the latter changes you have to reimplement the former. Moreover, it’s a different development paradigm (e.g. one class per task) which is NOT apt for mass distributed applications like J! components. The benefits of the Framework MVC are lost in the context of this kind of software. Frankly, people do prefer using Laravel of other PHP frameworks. What does that tell you about what the bulk of developers want? Surely, the “no-MVC” paradigm is better for large scale apps but this is not our primary or even secondary target audience.

    QA: That’s why I put the Container requirement in there. JFactory is a pain in the posterior during testing. Ask Davide about all the funky crap we were doing in FOF 2 tests to work around it. The Container solves that problem once and for all as you do know 🙂 About everything else: awesome!

    Framework: Not all features, just the ones which would otherwise cause a lot of additional code to be built on top. The Uri package is a prime use case. Implementing JUri on top of Uri required lot of code. Hacking Uri to add support for live_site resulted in much less code being written. I’ve done that in AWF, that’s why I used this package as an example. It is a fine balance but we can do it. Besides, adding support for something doesn’t invalidate Framework’s utility outside the CMS. Adding a null by default static $live_site variable to the Uri class doesn’t make it unsuitable for the Issue Tracker.

    Framework diet: I understand that the GitHub package may have been a wrong example. I’m not perfectly sure what can be removed or not, it does require some discussion. I think you and Andrew are the most qualified people to lead that discussion. The Composer libraries I do regularly use are very specific and have no overlap with JFW classes anyway 🙂

    1. My first contenders for the Framework’s diet are basically every non-GitHub API wrapper. Download numbers on Packagist will tell you why I say this. Unused by most, unmaintained by us. Let it go.

      With the URI package, at the Framework level we still have some structural issues with actually being able to support that live_site thing. The CMS implementation uses the data out of JConfig, the Framework has no concept of a configuration store (our DI code doesn’t have a parameter store, something Symfony’s does, and our application package supports an injected config but not everything has a dependency to that package). Maybe just having a static var works fine and it’s left to the developer to set it. Another small issue is the method where live_site is actually calculated (JUri::base()) is tied to the CMS’ two-app approach. Maybe the Framework’s UriHelper class can absorb the helper methods in JUri (base, root, current) in some form, but it does need a little bit of thinking if we want to make it work.

      In a bit of irony, JControllerLegacy could meet the No MVC interface requirement, even if the legacy controller model doesn’t match the intention of the interface. Even my last big project took that style approach (turning the execute method into a task dispatcher of sorts). Not saying it’s right by any means, but I’m far past single task controllers being viable in a large project (and this comes in part from how many controllers we have in the issue tracker).

      In truth, I wonder how many devs are actually building on an MVC style approach. Symfony itself isn’t MVC by any measure and I don’t have the experience with Laravel to judge it.

      1. I agree on the Uri package approach.

        I’m far past single task controllers being viable in a large project (and this comes in part from how many controllers we have in the issue tracker).

        Thank you. That’s exactly my case against the Framework MVC in mass distributed code. It makes sense when you are building web services, not so much when your code is 80% handling the same formulaic views with minor changes to handle specific business rules. I think that describes the majority of Joomla! components – definitely all the core ones.

        In truth, I wonder how many devs are actually building on an MVC style approach. Symfony itself isn’t MVC by any measure and I don’t have the experience with Laravel to judge it.

        That’s a good question. I guess it depends on what kind of sites are you talking about. MVC –as we mean it– is not as scalable as a services based approach. Likely they just have a front application consuming web pages and churning the result in HTML.

  4. In general there is a lot of things to agree on – and i think there is by most developers and people involved around the core of the community a consensus about most of the things you write at this point – so ill only add a few questions for debate.

    Why Blade and not something like Twig?

    I think for frontenders Twig is more mature and easier to onboard.

    As for FOF in core – i think the real question might be Conventions over Configuration or the other way around – which Again leads to the usecases of what it should forfil – so is going conventions over configuration too limiting or will it enable enough for it to make sense – or perhaps even more important how do we ensure overridability to a point where it is not a limitation.

    We spent quite a lot of time considering FOF and ended up making redCORE instead because FOF was to much convention based for the complex sololutions.

    In redCORE we also have JSON/HAL (and SOAP/WDSL for that matter) with a super simple XML manifest configuration layer (and oauth2) – any of the 9000 joomla extensions can be supported by simple configuration in a XML manifest file – so while it does require someone writes a simple manifest file its a lot more powerfull than a convention based one if you are building more complex systems. ( https://github.com/redCOMPONENT-COM/redCORE )

    Again the idea is configuration over conventions so instead of adding a generic method for exposing content to JSON we get more complexity by being able to configure it.

    Beyond this i would also add ACL to the table – not just in terms of end users but also in terms of an ACL that actually holds its own from an arcitechtural point of view – if you have a J3.4 running with say 500 ACL Groups in it your modules, menus, content etc. is pretty much almost useless with a 30+ second wait time on all loads – ACL has to be completely rethought into something thats scalable and multidimensional – and perhaps while doing so also consider the workflow possibilities that lies herein.

    Finally on framework – id fully support using as many standard libraries as possible and focusing our own efforts into the Unique libraries that gives special or additional power to the CMS – so in part the J! Framework could be in most parts made up of existing libraries but those we maintain and develop should be those that ads something Unique to the table an existing one could not – and as i understand Michael that is in many ways his vision too.

    1. Hello Ronni,

      Blade over Twig because Blade is a syntax convention, Twig is a full stack third party templating framework. Blade has minimal performance penalty, Twig has a massive one. Blade is internally translated to PHP which is readable by a developer –in a location which is printed out in case of an error– whereas Twig doesn’t.

      As for FOF and CoC, you haven’t seen FOF 3, have you? Please take a look especially at https://github.com/akeeba/fof/wiki/The-Factory As you can see the default is to use the BasicFactory which is “dumb”: if you don’t create a class it won’t do anything for you.

      As for redCORE it is NOT a RAD framework, despite your wishful thinking. It’s similar to what I had done back in 2009, two years before FOF 1.0, i.e. a set of common methods on top of the legacy Joomla! MVC. It’s what you do 1-3 years before you realize you need a more thorough solution. Basically you followed the wrong direction. Instead of making CoC optional and controllable you did away with it in most places. A hardcoded preference is a hardcoded preference no matter how you dice it. With FOF 3 I removed hardcoded preferences. There are overridable defaults.

      FYI FOF 1.x and 2.x where not good solutions. They were stopgaps. With FOF 1.x I tried to replace Nooku Framework in Akeeba Subscriptions with something that has a strong b/c policy. FOF 2.x was forced out of the door too early since I was asked for FOF to be included in Joomla!. It didn’t happen at first, I merely added some features its users were requesting and got myself into planning / architecture mode. This resulted in two different solutions. AWF is the first try at something that’s standalone. I wanted it to replace FOF but given the complexities of Joomla! integration it couldn’t happen. So FOF 3 was born, doing away with b/c and focusing on proper architecture. Try it for a small project if you don’t believe me. You’ll see why I built it 🙂

      Again the idea is configuration over conventions so instead of adding a generic method for exposing content to JSON we get more complexity by being able to configure it.

      You are describing FOF 3. The convention is there if you want to get a prototype quickly. You can of course customise, either by selecting which fields to show or by overriding the entire JSON output format. It’s up to you.

      Oh, BTW, having used XML forms and XML configuration files for any kind of views long before you do I can give you a fair warning. Do not try to make an infinitely configurable system through XML files. In the end of the day you’ll be trying to create a programming language in XML which is a waste of time and reinvention of the wheel. You can to figure out the crossover level of complexity between XML and PHP and offer customization through XML only up to that point and not any further.

      Beyond this i would also add ACL to the table

      I agree, but I would leave that for 5.0 (if I included my ideas for 5.0 the articles would be larger than an encyclopedia). The big two “problems” with the current ACL system are a. explicit Deny trumps explicit Allow which is the opposite of the intuitive grasp people have on access control and b. the privileges are calculated on the fly and stored semi-temporarily in the session causing delays and requiring people to log out and back in everytime the admin changes something.

      One way around the second “problem” is caching permissions in the database instead of the session and updating them every time an admin changes something. This transposes the delays from the public site to the admin experience. Or you could have that with a CRON job or something like that. It needs a very long discussion. I don’t see a right or wrong way to do that.

      Framework: I think we’re all on the same page, even if we use different words for it. I just read this very interesting article which further enhances this position http://blog.dominodatalab.com/reflections-on-buy-vs-build/?hn=1 Basically, it says that it doesn’t make sense to build something unless it contributes directly to your core business and you can gain advantage by rolling your own instead of using off-the-shelf components.

  5. Fully agreed redCORE is not a RAD – it might have started out with that intention in part but quickly became a Foundation for all of our extensions to allow for quicker output for client jobs by having to spend less time on the same basic tasks time and Again.

    However we also use it as a playground where Joomla itself is not quite there yet and as with elements of our Multilanguage, Webservices etc. we try and lead a way for something that can be adopted back to Joomla in some or another form as it happened with the sublayouts.

    So redCORE will remain for our specific needs (newest addition is payment handling across all of our extensions) and will not have the goal if being a RAD – but our RAD 🙂

    I did not check FOF 3.0 out yet – but i see that the exact reasons for why we hestitated to go with FOF 2.0 is out in 3.0 so that is interesting 🙂

    As for Twig vs. Blade – i understand the points and its important for backenders (and ressource useage etc) yet for Frontenders and Siteimplementors i fear the knowledge level in terms of php might be too high.

    Ideally in the solutions we do today and in the future you can relatively quickly deploy Joomla as your base – install a few 3rd parties for core functions of the client (say CCK or Ecommerce or both) and then setup webservices to built the value chain for your client so its tied into PIM, Masterdata, DAM, ERP, CRM etc. as needed so the solution is not a standalone but part of a vibrant living value chain (which could ofcause include any IoT devices).

    To do so we need a very scalable and flexible Foundation where you can then relatively quickly map data in or out and via some form of templating map those data to be used on the site with as Little Work as possible – stored locally to make it independant of live webservices but attained through webservices from many sources and utilized on the site.

    I really think the solutions of tomorrow will be solutions of chained systems and we need to adopt towards that – weather it be more simple or in complex and large organizations or cooperations the right design and arcitecture will assure that its possible.

  6. OK, makes sense now. Last time I’d checked –arguably a good year ago– you were saying it was a RAD framework. I’m glad we agree on what it is and it isn’t 🙂

    I did not check FOF 3.0 out yet – but i see that the exact reasons for why we hestitated to go with FOF 2.0 is out in 3.0 so that is interesting 🙂

    To be honest, FOF 1.x/2.x wasn’t suitable for Akeeba Backup and Admin Tools either. I had to shoehorn these components into FOF 1.x/2.x because I needed the Joomla! abstraction. After the successful AWF experiment I decided to take the experience I had gained to Joomla!. Now I just have to convert every single component. Six weeks since the first FOF 3 beta we are two down, one in progress, four to go 😀

    As for Twig vs. Blade – i understand the points and its important for backenders (and ressource useage etc) yet for Frontenders and Siteimplementors i fear the knowledge level in terms of php might be too high.

    Have you used Blade? There’s hardly any PHP. If your developers are good and methodical they can separate the templates into small bits forming a template hierarchy that’s easy for front-end developers and designers to handle.

    Ideally in the solutions we do today and in the future you can relatively quickly deploy Joomla as your base

    That’s the idea. A lean core which allows you to install off-the-shelf software, configure it rapidly and integrate it without much fuss. The power of deep integration is the differentiating factor against WordPress and the ease of it is the differentiating factor against Drupal.

    stored locally to make it independant of live webservices but attained through webservices from many sources and utilized on the site.

    That’s a laudable goal but not the main focus. Go back to the first post of this series to see what is the target audience. I understand that what you express here is your ideal mode of operation for your company but you’re not the typical Joomla! user any more than I am. We use Joomla! differently than our target audience. We must make a rock solid, good product for them. If we can add the features which will help us too, more power to us. I believe we can do it, just not as our major focus. Otherwise we’ll end up building a CMS based around web services and a Javascript-powered display/templating system which is alien to our target audience.

    Reality check. We’re racing for the second place with a chaotic difference to the guy in the first place. If we shoot for the moon we’ll end up crashing to the wall. I say let’s get a foothold on the second place and try to regain some market share to strengthen our position. Anyone who wants to shoot for the moon is welcome to fork off and try their lack. They don’t have to take everyone else down with them.

    I really think the solutions of tomorrow will be solutions of chained systems and we need to adopt towards that – weather it be more simple or in complex and large organizations or cooperations the right design and arcitecture will assure that its possible.

    If you honestly believe that you want to diss the existing user base and try to build a product for a crowded market, five years too late, then please do fork. Unless this is not what you mean.

    All the “IT future” predictions have a long tradition of falling far off reality. 640Kb of RAM should be enough. Apple will be sold to IBM. RAMBUS is the future of RAM. HTML5 applications will supplant native mobile apps. Have you ever heard of them? They sound ridiculous now but when they were uttered they sounded more than certain.

    I do see that web services will continue to play a great role in the mid-term, but not for the kind of small to medium sites our target audience is interested in building. I also do see that native mobile applications will continue to dominate in the short term and possibly beyond (barring a massive paradigm shift nobody has predicted yet). It is therefore to our best mid-term interest to produce a product which creates glorified HTML sites but does allow data export AND interaction through web services to power the mobile applications of the future.

    Of course your company can use these web services features to create a richer web experience. Nothing stops you. But YOU are not the core target audience of Joomla!. You are the exception proving the rule. We can somewhat cater for you but if you really believe that the future lies exclusively in web services you should not be basing your business exclusively on Joomla!. It sounds like what you really want is a collection of web service providers built on Symfony and a Joomla! or custom site (probably Joomla! to reduce costs) running custom code consuming these services.

    1. I agree we are not the core audience from the bottom – but the goal should be that we can take Joomla and use our own redCORE + Extensions and other 3rd Parties to do what we need – while the core of Joomla is not something that prohibits that.

      Then we can remai open for all as we are truely scalable and extending.

      I dont think IoT is a matter of time – its already here – integration of JSON to and from mobile apps has happened for years (we run many native apps from Joomla sites and we use redCOREs webservices for it) but there is so much more to it than just smartphone apps, which in it self would truely be a goal that pointed backwards and not forward.

      I thouch base with many clients every year from the small ones to the big ones and across all of these there are some requirements and needs that stay in focus.

      A website is no longer just a Html site – its a place where our business (weather it be ecommerce, consultancy of craftmanship etc.) is open when our real business is closed – i.e. the opening hours of a business might be from 8 to 16 – but the website should extend that through information, selfservices etc. 24/7 and help to digitize our businesses and make it relevant even out of normal business hours – the days of the business Card in html is dead.

      To do so – even small Companies need their website to be part of something bigger – a value chain – in their business – for the last many decades that value chain was more or less built around their ERP and often most other systems where in part manual or driven by manual processes.

      The integrations are many and in multiple directions, but at the base of it, it reaches beyond just smartphone apps it reaches out to all sorts of apps and systems and to truely remain relevant we need to make the best web application thats good at the core of it – while allowing it to be extended and Integrated so it can help form value chains – where those chains create value for the end users (businesses and organizations).

      I think adding business logics and value chains to businesses and organizations is beyond any specific technology or technological claim – its a Foundation of the future – and where we add the most value to the end users businesses and organizations.

      Then we need to make sure that the part of value chain that Joomla has is as optimal as possible and as best as possible and a relevant part of the full value chain.

      There are many examples of this and what we see today is most of these a emerging and do custom integrations 1 by 1 in a static manner to specific other systems – but having an infrastructure that allows even a site implementor to tie in the Joomla website into a value chain empowers the people who build websites to be relevant also in the future.

      I dont think in any way that what i am experiences on a daily basis is excluding from anything what so ever written in your posts sofar – i just look at Little ahead as why we need webservices and why we need a scalable core based on the 100’s of businesses and organizations i go out to every year and talk about their needs and how we can help transform their website from something thats more or less stand a lone to something thats a living part of their organization.

      1. I agree we are not the core audience from the bottom – but the goal should be that we can take Joomla and use our own redCORE + Extensions and other 3rd Parties to do what we need – while the core of Joomla is not something that prohibits that.

        I agree and I did make the point that we should facilitate you as long as it doesn’t get into the way of the rest. Obviously, as long as you have redCORE and Joomla! supports the format=raw you lose no functionality. Win for everyone 🙂

        I dont think IoT is a matter of time – its already here

        I can hardly imagine an IoT platform built on Joomla!. You need scalability that Joomla! cannot offer. The kind of scalability you need for this use case is beyond what Joomla! should be focusing on when 99% of its users will run it on a single –often shared!– server.

        I thouch base with many clients every year from the small ones to the big ones and across all of these there are some requirements and needs that stay in focus.

        Completely different needs for web services catering for a limited need and web services of IoT scale.

        The integrations are many and in multiple directions, but at the base of it, it reaches beyond just smartphone apps

        Which is basically what I’m saying in the post about web services in different words. Again, the thing is NOT about making web services possible, it’s about scale. Adding a RESTful JSON API (or even a SOAP API if you’re a raging masochist) is not the hard part. It’s dead simple. Scaling it is hard. Joomla! is not apt for that kind of scaling. Sure you can work around that with a custom database driver which uses different servers (or server pools) for read and write, tricks to move static content to a CDN and put multiple instances behind a load balancer, but this is not the core business of Joomla!.

        Then we need to make sure that the part of value chain that Joomla has is as optimal as possible and as best as possible and a relevant part of the full value chain.

        Achoo! I’m allergic to big corp newspeak. To put it abundantly clear: in the post I DID say that web services can and MUST be made available to open up all those possibilities that took you 500 words of big corp newspeak to not describe. What is it that you do not understand?

        I have to repeat myself yet again and tell you that just because web services will be possible it DOES NOT make Joomla! a great choice for Internet of Things scale platforms. IoT platforms are supposed to manage the data flow of thousands to millions of connected devices. Joomla! is simply not built for scale. YOU can work around that and all the power to you.

        But this doesn’t mean we should add these scaling features to the core because they will affect the performance and complexity for the 99% and, frankly, it’s something that the 1% can provide. I’m pretty sure your company can write a redSCALING solution with the necessary plugins and database drivers to make Joomla! scalable. I know that this has been possible since Joomla! 1.5. Joomla! 2.5 and beyond only made it easier…

        1. I think your logical reasoning is fine if you believe that IoT is only for big cooperations and big data.

          I dont.

          IoT will be something everyone will be using in all Company sizes and even privates – Internet of Things just Means that we will see a Whole new generation of devices that will be a part of our solution chains.

          Today we do responsiveness based on viewports but in the future responsiveness also Means that if a client (even a Little Company) are utilizing some IoT devices they will also be a part of the value chain for that Company.

          So its not just about scalability – yes the BIG FISH will scale up in masses and they need something entirely different to handle BIG DATA – but even Little fish will require handling of Little data.

          So IoT and webservices as a whole is a lot more complex and need an even higher abstraction level and ways to allow for configuration, adoptation, integration etc. onwards.

          So id suggest you reconsider what IoT is and its relevance even to small companies and organizations – and consider how much a part of the future value chains of those companies and organizations Joomla will be and what is needed to do so – in reality i dont think its a lot more than is on the table but its important our scope for webservices is focused on that and not so much on smartphone apps – there isnt much market disruption in getting years late to the party.

          1. I don’t think that IoT means what you think it means. Can we all agree that it means what the 121 references in WikiPedia define it to be? http://en.wikipedia.org/wiki/Internet_of_Things

            Yes, IoT will pay a very important role. By consuming data produced by IoT enabled “things”. Which you can already do. Having a small company build its own IoT platform is a bit idiotic unless it offers them a unique advantage. How would that apply to a small company is beyond me. I hardly think that a small company would benefit, for example, by implementing its in-house fleet management instead of using a third party solution and consuming data into its own site, e.g. to let clients know when their order will be arriving with greater precision than “sometime between 10:00 to 18:00”. This is something you can already do.

            So id suggest you reconsider what IoT is and its relevance even to small companies and organizations

            I already have thought of that. It’s 80% consuming data and 20% making data available for interconnectivity. In my example above an e-commerce application on your site would be able to export pick lists to your WMS (which would also notify the site if something is low or out of stock). Once the picking is complete the WMS would link back to the e-commerce which changes the order status to “ready for dispatch” and exports the delivery orders and address information to your fleet management which generates the delivery schedule for your trucks/vans/drones/motorcycles/whatever and pushes them to the tablets/PDAs of the delivery people. As they move along their route their location and schedule fulfillment is pushed to the fleet management which sends this information back to the e-commerce site where the client can get an accurate prediction of the delivery time (probably down to a 10′ window) and even get sent alerts by SMS, e-mail or push notification.

            How does Joomla! play a part in this? It’s running the e-commerce application and facilitates 1. the export of data (pushed to a web service or pulled from its web services) and 2. the import of data (again push or pull). Pushing data out and pulling data in is already possible. Having external services push data in and pull data out is the missing block, hence the very relevant point in my post about web services.

            Where Joomla! doesn’t fit this model? It’s mostly unsuitable for use as the fleet management system. In order to serve the 99% we need to manage a session for every request, not to mention bringing up a rather thick layer of internal objects and services. This means bad performance and increased server load.

            Moreover the fleet management requires some data crunching that is simply unsuitable for PHP and much more suitable for a functional programming language like F#. Essentially the big data or little data crunching begs for a different programming language and a different database technology, putting this outside the domain of Joomla!.

            As for the WMS, it can be based on Joomla! for small scale operations. It may be cheaper to do that too. It just won’t cut it if you’re talking about massive operations and I’m not talking about behemoths like Amazon but even the few thousand items per day volumes of many medium sized businesses. I have one particular case in mind, one that I worked with for the better part of 6 months (pharmaceuticals) optimizing their operations.

            Does this example explain better what I have in mind when I say that Joomla! doesn’t scale as an IoT platform but should still provide interconnection?

  7. Nic, I think you needed to break this post up. It’s too big to comment in one huge gelatinous mass 🙂

    1. Yeah, I know, it’s a longer than I expected. You can try posting multiple comments. WordPress has threading in the comments 🙂

  8. Ok, here goes.

    UI/Behavioural standards

    This is THE issue in my opinion that is going to plague the CMS template industry for many years (regardless of which one). And even though I agree a common JS library (aka Interface) is probably a sensible direction, it’s still compounded by the fact that it may need to be woven into data-binding solutions like Angular or Ember and friends.

    This is a really hard problem to solve. I do know that it’s not going to be solved without a lot of stakeholders at a table, and I think the project needs to be responsible for organising it.

    Layouts

    My preference would be to see true markup-only layouts and move away from any type of PHP based system that allows a novice to “hack it because they can”. I personally love Handlebars and partials. In this case, the less power the better.

    PHP

    It’s time to stop listening to the one or two prophets of doom that keep saying Joomla will collapse if we increase the minimum PHP requirements. The policy should be that the minimum required is the lowest supported version of PHP that PHP has released, plus some buffer (so today that would be 5.4.a-few-less-than-current).

    MySQL

    My wishlist includes changing the ORM to be able to support a file based data storage system, or Mongo, or even memory. Node does it so PHP can do it.

    UCM

    UCM is exceedingly powerful (ala Elgg) but I agree it’s either an all in or all out thing. I do think the basic Joomla “content” unit should have a core way to extend itself and let extension developers duke it out over who can provide the easiest UI to use.

    Core Extensions

    I’ve written passionately about this and the possibilities distributions would open up.

    I do think we need to remodel the whole extension installation thing though. The way WordPress’s presents to the user and functions is much better in my opinion.

    Namespaces

    Yep. Get rid of JLoader as fast as we can.

    Joomla 4

    There are two ways to do this. Either bring in new code now with namespacing that begins to form a natural b/c layer within the 3.x source. Or, build 4 with a legacy layer ala Joomla 1.0 to 1.5 style.

    Pros and cons either way but I think the important point is to avoid another 1.5 to 1.6 transition.

    Composer

    No brainer.

    DI Container

    Here we get into some trouble because DIC’s have been abused heavily. In my opinion the container or service locator is fine as long as it goes no further than the Controller level.

    MVC

    I really, really like Paul Jones’ ADR paradigm and I think it fits what most extensions are trying to do.

    There needs to be a much better separation between “request” and “response” and the “bit” in between (be that MVC, ADR, pick your poison) should not care. This is something the Node people are doing much better. And while I’m at it, I’d love to explore the concept of “middleware” to replace most of the abuse that suffers at the hand of the current plugin system. The Command Pattern would be a far more powerful replacement.

    As for FOF3 (I can’t resist), no – I’ve already got two versions of FOF installed in my sites, I don’t want a third 😛 (joking!).

    JSON

    Completely agree.

    Schema

    Yeah that’s fine, as long as the code is built with interfaces that allow other file and schema formats in mind.

    Extension Installation

    This is one area where I do need to harp that the code needs to be SOLID.

    Further, I think we need to completely rethink the concept of the extension. I think WordPress is doing it right with just two (plus language packs). A “plugin” should be able to register what it can do with the system. A “theme” (or template) is worried about display. Simple.

    Menus and Routing

    Shooting from the hip:

    Every extension must define it’s unique resource routes as a minimum, and these are only allowed to return an object that can be converted to JSON (bingo, you have your REST API).

    Navigation marries a custom route to a page layout (not necessarily a component like it does now). Perhaps page layouts are merely a “structure” to display a number of “modules” arranged in a particular pattern.

    Perhaps custom navigation routes are always fully pre-computed (we have menu router generators) so that “menu” routing is just a simple db lookup.

    Maybe for pages that are truly static, Joomla is able to fully generate those pages and serve them statically (ala Github pages style).

    Testing and QA

    So here’s where we might differ a lot. You can’t check for regressions unless you have code coverage over the “behaviours” of the code. But what I was taught about coverage and how to test was completely wrong and in the last year or so I’ve rediscovered how to do testing better (check out anything Uncle Bob Martin has written or recorded).

    It’s a long conversation but you actually want to be aiming to get close to 100% code coverage when you have tested all of the “behaviours” or “use cases” that are supposed to be achieved. The difference to the way I was taught is that coverage is the sum of all the different types of testing – in fact, it could be achieved with no unit tests and only a modest suite of integration tests.

    UI testing is hard and should be done sparingly.

    I cannot impress enough that we should be aiming to achieve true test-driven (red/green/refactor) development practices. It’s not easy to learn, hard to master, but don’t we all want to be craftsmen when we grow up? It is achievable and there are significant productivity benefits in going that way.

    Yes, I know, I know. You all think I’m crazy. I would have agreed with you once too.

    Joomla Framework

    So here’s the thing. Regardless of where the Joomla Framework has been, it’s what Joomla needs now that is important. Nothing else matters. To that end I think there is a real need for “a” team to be responsible for the engine-room level direction and decision making, particularly when we are thinking about probably including more and more code with Composer (in fact I’ll argue most of the code in the CMS should live in /vendors/). Someone needs to make sure they work together and simply keep the stupid things updated.

    Call that “Framework” or whatever you want, I see there is a need for a team to be responsible for that. Above that, you have the team that takes that code and makes what we call the Joomla CMS and the extensions. And there are obviously other specialties but I’m currently leaning towards the idea that the library level code is a specialty in its own right.

    Whatever it’s called, everything needs to be designed to be modular with the lightest coupling possible.

    Anyway, that’s a very long conversation that so few words can’t possible give justice to. I’d love to chat to you Nic about how we can actually develop a plan to move Joomla forward (sounds like a good name for a campaign).

    I think what’s lacking in the leadership at the moment is a solid plan of action, but we’ve already been over this territory. Personally I’d like to see an immediate API freeze as of 3.4 (yes, I know that will affect some plans, but the idea that we are going to try to “fix” that router again scares the living daylights out of me) so that we don’t start adding more crap code that we just need to throw out in a few months.

    1. Your comment and my reply are longer than the post, but it’s an interesting conversation to finally have 🙂

      UI/Behavioural standards

      I agree that it’s hard. Moving to Angular or similar JS-based technologies is beyond the product’s scope. I can see how an enterprise distribution (hint!) could do that, especially if we finally ship core components with proper web services support, per yesterday’s discussion with Ronni.

      As for Joomla! the CMS we all know, we can add the intermediate JS library to minimise the impact of 3PD template developers implementing a different CSS framework than Joomla!’s preferred (latest Bootstrap). It’s not a perfect solution but I’ve yet to see a “perfect” solution for markup abstraction that’s easy for site integrators and end users to use.

      Layouts

      I disagree. This would lead to all control structures being hidden in a non-overridable View or Controller class, depending on the development paradigm. The power of Joomla! for end users and integrators lies in our awkward template system. Remember that integrators demand being able to override Views (bad idea). If we add more restrictions to what they can do they will move away. Blade is a best compromise solution. Strong emphasis on the word “compromise”.

      PHP

      Full ACK. I can tell you right upfront that 10%-20% of our users will have short-term upgrade problems. Half of them will be left behind forever. Not unlike moving from Joomla! 2.5 to 3.x, I might add. This is according to the analytics I collect from the sites my software is installed. Usual disclaimers about accuracy, lies and statistics are in effect.

      MySQL

      Even though you can, it doesn’t mean you should. The current database driver system allows us to support multiple db servers –even awkwardly– but it CANNOT produce optimised queries. If you want query optmisation you have to look at the entire product, from the way you define indices to WHERE clause precedence. Sticking with one database system will let us make Joomla! fast. Moreover it allows us to use the full power of the database server technology instead of aiming for the highest common denominator of supported features across all supported database servers, something which we currently do to the detriment of performance.

      That approach is not incompatible with having an ORM. At least to the extent where a glorified Active Record based solution like Laravel’s Eloquent can be called “an ORM”.

      UCM

      I’d say that a CMS based on UCM is great, but it’s not Joomla!. It’s more like Drupal. UCM is extremely powerful but also very complex.

      I do think the basic Joomla “content” unit should have a core way to extend itself and let extension developers duke it out over who can provide the easiest UI to use.

      Isn’t this pretty much what our half-baked UCM tried to do? Developers found it very limiting that it’s only supporting text-based content types and either ignore it or hate it with a passion. If you want to fix it you need to allow complex types, Drupal-style. Unless you have something else in mind and I didn’t catch your drift.

      Core Extensions

      Full ACK. The Install from Web was a necessary, bold step forward. It needs to be the primary (but not only, unlike WordPress) method of installing extensions. And it must promote the core-supported extensions by default, just like WordPress does.

      Joomla 4

      I agree that a 1.5 to 1.6 style transition would be disastrous. It already took 3+ years for people to trust our ability to provide one click updates. As for the best method to follow, it depends on available manpower. The best approach from a technical POV is namespace everything and add a b/c layer to ease trasition. The practical approach is import legacy code and namespace progrssively, with a b/c layer that grows over time to ease transition. The latter runs the risk of holding us back. There’s nothing more permanent than a temporary solution or process…

      DI Container

      The application being a master controller guarantees a DI Container. I’m also not sure what you mean exactly with “Controller level”. The Controller needs to pass its DI Container to the Model and View since these do not (should not!) have direct knowledge of the controller calling them. If you mean the container shouldn’t spawn models and views, all right. IMHO the container should just hold a per-extension Factory service spawning MVC objects, so I guess I’m saying to stop the Container at the extension level – which is practically means at the main Controller level. Are we saying the same thing?

      MVC

      From a developer‘s POV the Command Pattern is more powerful. From a site integrator‘s POV the plugin system is more powerful. With the plugin system you trade some performance and code abstraction for easy customisation of the code without having to change the code itself (flexibility). Considering our target audience –and even though I know better as a developer– I prefer the plugin system.

      Schema

      Sure thing. I just proposed what I’ve written and tested out there. There’s no reason to not allow other schema formats, no matter if it’s another encapsulation of the same payload (JSON, YAML, …) or an entirely different format altogether.

      Extension Installation

      I completely disagree with WordPress’ approach because it does the exact opposite of what you say it does 🙂 A WordPress theme defines when half of the hooks are executed. It’s possible to lose core functionality with a theme. The WordPress plugins are a convoluted mess exactly because they are not supposed to do anything besides mild interaction with the content. This has led to complex plugins like WooCommerce requiring you to either create a gazillion pages with specific “short codes” (like Joomla!’s content plugin codes), or require you to install a custom theme providing the right hooks. WordPress has it wrong. Joomla! has it right.

      I agree that we have too many extension types. Component and module should be one: as long as you have HMVC it’s possible. Library and file should be one type, too. Plugin, template, language as they are. As for packages they are not extension types per se, they are aggregations of extensions.

      Menus and Routing

      I disagree with the web-services-only approach not because I think it’s bad but because this is not what our target audience is interested in. While that’s an interesting design, I would ship it as a different product, not Joomla!. What you describe is a very nice enterprise product, not an end-user / site integrator product.

      Testing and QA

      I do not disagree on principle, only on the practical aspect. We just don’t have the manpower. Stalling development to testing would lose more manpower and effectively kill the project. But we can’t live with permanently fixing regressions. That’s why I said that the short term goal should be testing the critical paths of the code. Simply put, I don’t care about how back-end filters work. It’s not a major issue if something is semi-broken and fixed without a test to prove it. I do care if an unauthorised user can view privileged content or submit content they are not allowed to.

      Joomla Framework

      I agree with you. The whatever-we-call it is the engine room / R&D of the project and it necessarily needs to be a team separate from the CMS maintenance for practical reasons.

      As for an API freeze, yes, it’s necessary. It won’t necessarily mean a feature freeze because, let’s face it, there’s still room for minor improvements and we can’t afford to lose any more users who think that Joomla! is dead in the water (nevermind that it practically is, it’s the perception that matters). There are some features from the list of part 2 which can be implemented in Joomla! 3 and ported with minimal to no effort in Joomla! 4. But the one thing we DO need a.s.a.p. is starting working on Joomla! 4. As in now, not next week or next month. We’ve already lost two years.

    2. Some further points on MVC and routing that sprang to mind.

      Joomla!’s success with developers can be attributed to it being easy to quickly develop on while not prohibiting you to write more advanced code. A novice developer can start writing pretty much formulaic applications spitting out static HTML. An advanced developer can write their component as a collection of semi-RESTful web services and one or more views serving the template to render the data acquired from the services.

      Compared to WordPress we have a definite advantage. In WordPress you can provide AJAX data if you go through its AJAX endpoint which is like a poor man’s version of com_ajax (yeah, that limited). By necessity a developer wishing to write something more complex than static content to the output might go through that route, essentially reinventing the wheel, the chassis and the whole darn car in the process.

      It’s in our best interest to keep this low barrier to entry. It is our duty to improve the capabilities of the system for the advanced developer. I see two ways we can do that

      1. Web services. We could agree that there’s a reserved route leading to web services provided by components. I use the term “components” to describe both components and modules which, IMHO, should be the one and same type. Each components defines its route and yes, as you said, there you have a RESTful API out of the box.
      2. Pages as collections of component-views. Each menu item would essentially define a page. A page consists of one or more component views arranged by the template according to the selected module positions. Note: there is no “component of the page/menu item” any more. All displayed components are first class citizens, much like Drupal does.

      From an architectural perspective this requires some obvious changes. You need HMVC and I’m using this term in the loose, web application definition. Essentially the Container of each component allows it to display on any Page without assuming anything about the page layout or using ugly constants (no more JPATH_COMPONENT). If you want to POST something to the component you can either do this through Javascript, directly to the predictable route of its web service, or to a component view, always pushing the return URL back to the page you currently are.

      The obvious problem is how you tell each component on the page to display a specific set of data, e.g. if I have a page showing the forum categories in the main module area how do I tell it to now display a specific post. One way could be per-page routing (with the obvious question of which of the components on the page should handle the routing) or using data in the URL fragment (which makes the URLs ugly). This is the problem I have no solution for.

      Alternatively, you could do away with per-page routing altogether. But this kinda “breaks the web”, doesn’t it? I’d need a menu item for every forum post to have a direct URL to it. If I don’t I have a lot of inaccessible content to the search engines. And that’s the other side of the problem I have no solution for.

      I welcome any suggestions and constructive criticism.

      1. Nic, I think we are on the same page enough not to worry about it, at I’m not worried about the few edge cases where there will always a difference of opinion.

        If I may, I’d strongly encourage you (if this isn’t already the plan) to find a room at JAB for 4 hours and sit down with a “doc” with these headings and start writing a “Joomla Forward” manifesto with other developers that are interested. After that, we find a group of people that are committed to meeting for an hour a week to polish it and then we work out how it can be executed.

        That’s my suggestion anyway.

        1. Thank you, Andrew. That’s pretty much my plan. I hope I can enough people willing to move Joomla! forward. Well, we’ll know in less than a week 🙂

  9. 1) Media Manager (like in WordPress)
    2) Using the latest version of Bootstrap (with the ability to use Bootstrap shortcodes in articles… )
    3) Better SEO (for example, it needs an easier way to have URLs like WordPress… )
    4) Comments and, in general, make it more Blog/Magazine friendly

    I would really like stay with Joomla as my main CMS, but as I use WordPress more and more – I realize how much better for many things… the only thing that, for now, keep me away from WordPress is this: https://www.akeebabackup.com/support/admin-tools/22582-admin-tools-pro-for-wordpress.html

    1. I hear you! What you’ve written is exactly the same thing I hear from pretty much every Joomla! user in every JoomlaDay I’ve attended the last 4 years. So, let’s see where we’re standing:

      1. Already in progress! However don’t expect WordPress-style URLs. WordPress is essentially just com_content and nothing else can work with it. If you’ve ever tried adding an e-shop into it you’ve either had to mess around with custom pages and short codes (blerch!) or special themes which dictate custom URL structures, therefore you’re no longer using WordPress URLs. At least Joomla! is trying to provide a good compromise solution for all. Not easy, but worth the hassle.
      2. The consensus does linger towards it.
      3. That’s the routing stuff I described. So yes, in progress.
      4. Comments will be a core-supported extension. Like weblinks, you will be able to install it when installing Joomla! or from the Extensions Manager. I’m not 100% sure it will make it in Joomla! 4.0 because… time!

      Regarding comments, core comments are going to be very basic, pretty much like WordPress’. More realistically you should use one of the excellent comment plugins for Joomla! such as CComments for self-hosted comments or Disqus Comments for Disqus integration. They are free and can do far more than what WordPress does out of the box.

      1. When I started to use (to play with) WordPress, I was 99% sure that there is no need to keep using Joomla! (at least for the needs of the sites that I want to create). Simply, if you want to create a Magazine (news portal) website for someone – WordPress is much better for that.

        When it comes to Bootstrap 3 – if you are using a theme created with it – you can use that Bootstrap 3 (all of its functionality), and also there are good Bootstrap-shortcodes extensions which make easy-to-use Bootstrap regardless whether the theme is using it or not…
        I suppose that you can do the same in Joomla if you are using a template that is created with Bootstrap 3? Or, if not, that you can find free Bootstrap3-shortcodes extension? To be honest, I have not tested Bootstrap in Joomla, I just did a quick search on Google for free shortcodes (Bootstrap 3) extension, but I have not found it…

        Also, when it is about e-commerce – WooCommerce (at least for me) is much better than any Joomla! e-commerce extension (I don’t just mean VirtueMart).
        I know that this blog post isn’t about third party extensions/plugins, but availability of good and functional extensions is one of the most important (right?) things for end-CMS users (dummies) such as myself.

        And there are many other things where (at least for me) WP wins, but I do not want to be boring and to spam here…

        Since I am paranoid when it comes to security, I needed to ask you about Admin Tools Pro for WordPress (I saw that there is Akeeba Backup for WP so I was like: “Yes! Probably, there will be soon ATPro for WP, I just need to ask when… and I am going to use WordPress from now“), but your answer was like a slap in the face :>

        Finally, the conclusion is – YOU WILL NEVER FIND AN ALMOST PERFECT CMS :> Let’s face it…

        Sorry for my bad English

    1. Well, putting four completely unrelated features into one idea is unlikely to attract any attention.

  10. We need to have caution. Big changes at once, can have a negative impact on the community. It’s hard having to redo a lot of work every time new ideas arise.

    ABOUT BOOTSTRAP
    The big problem leaving the Bootstrap is to have one more thing to worry about. I think that the interaction with Bootstrap is the best thing. But we need to end the dependence of the customized version of Joomla!

    SIMPLE SOLUTION:
    A transition style sheet kept for a certain period prescribed. enough time for developers to adapt their extensions!!
    We will use the current version of Bootstrap without breaking the previous version!

    I already have it !! It’s simple and i believe that many also have done it!

    1. You don’t need a transition or backwards compatibility layer. It’s trivial to include BOTH Boostrap 2 AND Bootstrap 3 markup in your extension at the same time, in the same file, without any problems. I don’t speculate. I’ve done that, 18 months ago, in Akeeba Ticket System. It takes some planning but it’s doable.

      On top of that, it won’t even be necessary for Joomla! 4. The idea is that the JLayout system can be used to abstract the CSS framework for developers. They will have to replace hard-coded HTML with calls to JLayout and the rest is handled by Joomla! and the template.

      And yeah, no more customised Bootstrap versions. We all concluded that it was a silly thing to do.

    1. I don’t see a reason to add a hard dependency on yet another thick library we have no use for in the Joomla! core. You can always load create.js in your template or extension if you need to add animation, sound, media preloading etc. I personally find these features distracting and distasteful, as much as I found them distracting and distasteful back in 1998 – remember what we called DHTML? It’s MUCH worse for people with visual impairments. So if I had any say in this my opinion would be a big “NO BLOODY WAY” on integrating with a framework for silly dancing shit on the page. If you want to use it on your extension be my guest. It’s a free market.

  11. So whats the follow up from these blogs and discussions?

    What will be the next step forward for Joomla to focus on and prioritize?

    The competition from everywhere will just be harder and harder and for example Wix.com these 60 Million web pages c o u l d have been built with Joomla also i f Joomla also had a different strategy with cloud thinking in mind and focus on u s e r s needs.

    With competitors like WordPress, Wix and similar there is smaller and smaller segments for Joomla web system dont you think so?

    So how can we also use and integrate cloud based system and reusing Joomla for more needs where also Joomla could be player. I am sure that Joomla.org could be smarter used, marketing against these groups too and leverage what Joomla already have as a web cms system..

    http://www.wix.com/about/features

    1. We have already talked about this during the J and Beyond 2015 conference. There is an architecture code sprint planned in ten days from now and code will start getting written after that. Supposedly there should be a statement issued from the PLT by now but yeah, it’s still not published. Sorry, I don’t have any official capacity in the Joomla! project so I can’t tell you why it’s not published yet or when will it be published.

      As for “cloud” it doesn’t mean anything to me in particular. “Cloud” is a fancy term for saying “on someone else’s computer”. Joomla! is a Free and Open Source Software project producing a PHP-based CMS which can be installed on any server, everyone can have access to its source code and write extensions or customise it to their liking. Wix and other so-called “cloud based” systems are FENCED GARDENS. You do not have access to the source code, you cannot customise them freely, you cannot choose your hosting and you cannot extend them any way you please. If I was called to contribute free of charge (or even for a fee) to such a system I wouldn’t even bother. If you think that Joomla! should betray the ideals of Free and Open Source Software by becoming a closed source product milking people from money you’ll get no sympathy from me.

      So this comes back to my first blog post about target audience. I doubt there is much overlap between the target audience of Wix and Joomla!. Maybe a VERY small overlap, namely the people who are not sure if they can handle a CMS on their own. I’m afraid the way to win this crowd has little to do with Joomla! itself –definitely not with its architecture for developers which is the topic of the post you are commenting on– but mostly with site integrators who have to put these clients’ minds at ease. The only thing Joomla! can do and is doing well in that respect already is security. The other major thing is ease of use which again has nothing to do with architecture and everything to do with the second blog post in the series.

      But do bear in mind that while you can win a portion of the market segment of Wix, WordPress and other simple platforms you can’t outperform them (see first blog post regarding target audience). If you try to provide a solution which is equally good for all market segments, from the small “let me cobble together some content and magically make it look nice” user to the big corporate user the only possible outcome is coming up with something that sucks equally for all. And again I have to give you the same conclusions as my first blog post. So please stop making me having to repeat the 6000+ words of this trilogy and just read it again 🙂

  12. Hi Nicholas – thank you for all your time/thought/efforts for Joomla, even now as you and your fellow countrymen face a major financial crisis. I would be interested to hear your thoughts and impressions about the Joomla 4 architecture sprint last weekend. Maybe that’s a subject to consider for an upcoming blog?

    1. There are a few formalities left before Marco can publish the report of the sprint. Then I can consider writing blog posts depending on my available time.

    2. Apparently neither me nor anyone else is allowed to discuss anything regarding the architecture in public for various reasons. Everything has to go through the management and marketing. So please don’t expect me to write any blog post whatsoever or answer any question (direct or indirect) about what I am working on or the overall architecture of Joomla! 4. Please direct all your enquiries for information to Marco Dings of the Joomla! Production Leadership Team. Thank you for no longer asking me about what I am working on / for / with.

  13. Understood – I am more interested in your thoughts and impressions than what is hopefully coming soon from official channels, but I accept what you shared about the process, and I respect your wishes. Thanks again. 🙂

  14. Joomla 4 should have development portal for local client/server environment. Once website/update is final on local computer. User should be able install or update on one click (Security is must). And backup option for update previous data before applied

  15. I agree and disagree with “MySQL (and compatible variants) only”

    I agree that it’s currently broken, but where I disagree some is the solution. I would like to see something like DoctrineDBAL replacing all of the existing Joomla solutions. If something like DoctrineDBAL isn’t implemented, then I agree there should only be “MySQL (and compatible variants) only”

  16. I love all of these suggestions for Joomla, however not sure what the UCM was in Joomla. I agree as well I don’t see an independent Joomla framework getting traction, I think pulling non-essentials out of core to create a very light core when people want to build out custom applications with it makes more sense.

    1. UCM (Unified Content Management) was a half-baked approach to creating unified content with arbitrary (and programmatically defined) content types. It is what is used to power tags and content history in a clumsy manner which ends up duplicating the content in your database. It needs to be removed and replaced with a true custom content types solution. In the grand scheme of things I think that Joomla! 4 can survive just fine without this particular change 🙂

Leave a Reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.