IRC Discussion on the current state of the web2py plugin system, what we want out of a plugin system, where it should go, different implementations, and developing a spec for developing plugins. - Tuesday March 08 2010, 22:02 - mdipierro: Just put son to bed. Hope he stays there mdipierro: Fine otherwise mengu: cool mrfreeze: little massimo? did you teach him python yet? mdipierro: LOL. Only loops mdipierro: He can mess up windows better than anybody I know. mdipierro: and that is a skill. mrfreeze: I usually does that itself mrfreeze: it mrfreeze: everyone excited about plugins? mdipierro: yes. mr freeze tell us all about it mdipierro: I would separate two things if possible - 22:02 - mrfreeze: go for it mdipierro: 1) what can we improve without changing web2py source (except perhaps admin) so that it remains agnostic mdipierro: 2) what can we do if we change web2py source mdipierro: now you go on mrfreeze: I think one of the main debates is a folder based plugin system vs. putting things in the application folder, yes? Turicas: mdipierro, hi mdipierro: Hi Alvaro mdipierro: I think this falls under 2). Anything under 1)? mrfreeze: I'm not sure. Does it? What would need to be changed? thadeusb: What do you define as "changes to web2py source"... does this exclude addition of new classes such as PluginManager ? mdipierro: If you put plugin models controllers into a plugin folder, web2py would have to look into multiple places before it finds a controller. This requires changing lots of stuff, including admin, the way apps are bytecode compiled and it may be slower because of the extra lookup. That is why I thought we should first discuss any other issue and leave this as the last one. Assuming there is any other issue mdipierro: No. A pluginManager would not costitute changng source. - 22:09 - thadeusb: ... mrfreeze: A plugin manager could control this I think. It could also solve the problem of how to disable a plugin without uninstalling it mdipierro: tell us all about PluginManager then Turicas: I think we MUSTN'T think in implementation or if it needs to change a lot of code Turicas: we need to think in our customer Turicas: and our customer is a web developer that uses web2py Turicas: what will be better for him? mdipierro: I am not being pros or against. I am just saying let's first discuss what can we do without changing source because we will have easier agreement. Then we tackle the more complex problem thadeusb: I have a really big question. Are plugins for the "end user" or the person using web2py (such as a person uses wordpress) or are plugins for developers (those making the web2py apps)? Turicas: thadeusb, very nice question thadeusb: It seems that this has overlapped in discussions, and I would like to make it clearer. Like Turicas says. mdipierro: I think both types of plugins can coexist. Can we serve both? Turicas: mdipierro, if we split things in 'without coding' and 'coding' we are thinking as web2py developers. we need to think as web2py customers mdipierro: Yes but the issue of where plugins are located in a filesystem is an implementation issue not a functional one. LoganLK: i think plugins should be bundled with the application as opposed to being put in a plugin directory or in site-packages. One of the pains of django's reusable app model is that the app never does quite what you want it to. You always need to change it. - 22:14 - LoganLK: That leads to you having to fork the reusable app, or remove the altered app from site-packages and putting it locally. Or having to extend every base class the app provides. mdipierro: I agree. Is anybody proposing a change in plugin system so that plugins do not belong to the app? Turicas: i think we will have more modular and organized files if each plugin has its own directory mdipierro: again this is an implementation issue. Let's discuss functionality first. What do you want to achieve? thadeusb: on both types: I don't think so. It depends, what is your end goals for web2py, are you trying to create a system so non-programmers can "drag and drop" functionality with only a little bit of glue code, or are you trying to create a framework that lets real programmers get their work done faster the way they want? From the way I see it, web2py is kind of in the middle of both of these, where its not enough for real programmers, yet almost too much for no thadeusb: n-programmers. Turicas: if you want to install the plugin, just copy and paste its directory in web2py/applications/your-app/plugins/ Turicas: (like we do with applications!) LoganLK: Mayby web2y could borrow from Kohana PHP's HMVC structure thadeusb: My biggest gap in understanding is not knowing where Massimo is wanting to take web2py, because the end goal and desire for web2py will have a massive impact on the way plugins will be designed. mrfreeze: i like the functionality of the current system, just not the organization. I need a way to control which resources are exposed to plugins. mrfreeze: i need to keep my business objects protected from plugins Turicas: mrfreeze, cool thadeusb: Agr - 22:19 - mdipierro: Thedeus this is a good point. My main job is to make sure it remains consistent and keep backward compatibility. Where it goes you tell me. MrFreeze, tell us more thadeusb: That might be your "job" but you obviously have a vision of where you would like to see web2py go from here. mdipierro: For me the goal is always make it easier and easier. mrfreeze: i may be mixing issues here, but can a current plugin expose more than one view? Turicas: i think web2py plugin system should focus on web developers, not in app users' Turicas: if some app want to have its own plugin system, so web developer should create it (and really use some code of web2py's plugin system) Tobarja: does easier lead to hand-holding to this is how it's done (take it or leave it)? mrfreeze: meaning, can you have a link in a plugin that takes you to another view in the same plugin and stay in the parent page? mdipierro: I see two level of plugins. The low level I am mostly safisfied on (althought open for discussion) and I envision a new higher level that should work at the application level. Some for the user to drag and drop and use as in {{=plugin}} without programming. These plugins would have their own controller that provides (optionally) a configuration interface. mdipierro: Current system allows for more than one view mdipierro: Yes you need a little of extra JS. - 22:24 - mdipierro: Let me clarify a couple of things LoganLK: Current web2py 'plugins' aren't really plugins (in a Wordpress sense). They are 'first class' controllers, models, views and static files. Because the plugin is bundled with your application (not in site-packages) you should edit/integrate/merge the plugin into your app's code mdipierro: Logan in right mdipierro: The current system is a means to built on.... mdipierro: the current system provide two things: mdipierro: 1) a way to package and unpack ANY subset of an application mdipierro: 2) the LOAD function that allows to embed the action exposed by a plugin into any page (we call these components, not plugins) mdipierro: The current plugin mechanism does not provide: Turicas: web2py should auto discover plugins and load them when/where it specifies to be loaded mdipierro: a) a convention for what should be a plugin LoganLK: I am a fan of this implementation for plugins: http://v3.kohanaphp.com/guide/about.filesystem mdipierro: b) a convention for naming plugin variables so that they do not conlfict with each other mdipierro: What Logan suggests is excatly what we have now. Assuming I understand the charts. Turicas: c) an way to restrict access of other app info (db, session etc.) LoganLK: Yes, I support that model thadeusb: yes, the name determines the order in which executed, with letters such as xyz overriding anything defined in abc. - 22:29 - mdipierro: The reason the current system does not do a) and b) is not because that is important but because a) and b) can be implemented of top of the current system but the vice versa would not be true mdipierro: ell us more about c) mdipierro: tell us more about c) mrfreeze: it would be trivial to 'initialize' the plugin system with the objects that the developer chooses to hand to the plugin system. Of course, this would be in name only. A nasty plugin could still access all global objects. thadeusb: I would think developers smart enough to read the source code of an app before executing. mdipierro: I agree but I may want a plugin that, for example, goes over my db and deletes all records with name starting with 'A' and then removes itself. Why not? thadeusb: *plugin* thadeusb: "why" would you want a plugin that does this!? I don't see any real world use for this. LoganLK: mdipierro: every open source application has that issue. At the end of the day it's the developers responsibility to check the code. LoganLK: I could make a Django reusable app that drops everything, i could package a shell script as a .deb that deletes ~, I could paste pipe bombs on ubuntuforums when someone is trying to figure out how to start x. I don't think it is sane for a framework to check that. - 22:34 - mdipierro: That is my point. We cannot make plugin safe because a) it is not possible; b) because a plugin should be able to do anything (including editing other files in the app itself, for example change layout of css file) Turicas: i agree with b). but i think it is possible mdipierro: So I think we all agree that safety is not the goal. The goal is to avoid accidental conflict by having "certified" plugins that follow some naming convention and have a registration process to "advertise" some plugin central about their presence. mdipierro: Still some plugins only contain static files, some only controllers, some only models, and combinations thereof. mdipierro: ? mrfreeze: i agree. a naming convention for plugin objects would be a good start. mdipierro: Yes this is the main goal I think. Going back to the folder structure for a second. - 22:40 - mrfreeze: how would URLs work with a folder based system? mdipierro: we could create a plugins.py controller (or perhaps have appadmin do it) that delegates all request to stuff in a plugins/ folder. But it would make everything much more complex thadeusb: Should plugins have their own "views" that get exposed .... should /init/plugin_comments/latest actually be a view that can be rendered like any other controller? mdipierro: "compile app" would become a nightmare LoganLK: applications/XXX/plugins/XXX/ (dir. structure of applications/XXX/ replicated) ? thadeusb: What if there was an "installed_plugins = ['a', 'b']" this way web2py will only look for plugins specified? This solves the issue of having to "autodiscover" Turicas: mdipierro, i prefer 'plugin.py' mdipierro: Right now thay can have their own views thadeusb: *should* they have their own views? thadeusb: or *should* all plugins be exposed manually, by LOAD, or otherwise? mrfreeze: i forget, where do static files installed by a plugin live? in static/plugin_name? mdipierro: It depends on what it does. A layout plugin would just have a layout.html view. Plugins that expose components should also have view, but the views should not extend a layout. mdipierro: yes... mdipierro: currently: mdipierro: applications/models/plugin_.py mdipierro: application/controllers/plugin_.py mdipierro: application/views/plugin_/* mdipierro: applications/static/plugin_/* mdipierro: but this again a convention only used in admin thadeusb: applications/models/plugin_/*.py mdipierro: ANY file in the app can be packaged into a plugin - 22:45 - mdipierro: tar zcvf web2py.plugin.mylayout.html views/layout.html static/* Turicas: do we need a plugin system? mdipierro: Right now there is no convention mdipierro: I think we need a convention Turicas: what if we have a lot of apps? mdipierro: Apps do not talk well among each other. That is Django's problem Turicas: so we need to create a plugin system or create a way to apps talk well to each other? mdipierro: The fact is that I would prefer to have a very strict convention but leave web2py convention agnostic LoganLK: how about a plugin directory which mirrors the main application directory structure, LoganLK: applications/blog/plugins//controllers/ LoganLK: applications/blog/plugins//models/ LoganLK: applications/blog/plugins//views/ Turicas: LoganLK, it is where I like to come Turicas: plugins are little apps Turicas: that can 'talk' well with the main app Turicas: so I think plugins should have its own path mdipierro: I am not saying no. I am saying this is an implementation issue. what does it buy us? mrfreeze: modularity mdipierro: So in your "definition" a layout.hml file in its own is not a plugin. thadeusb: no. mdipierro: OK. So what you are proposing is a subset of what we have. You can already make a plugin.py file that defers all requests to /app/plugin/whatever to code in app/plugins Turicas: servername/appname/plugin/pluginname/static/blah.jpg Turicas: this URL is so boring - 22:50 - mdipierro: ok. So all you need is to write a plugin.py mdipierro: yet this will be very limiting mrfreeze: framework plugin vs. application plugin? mdipierro: The plugin will not be able, for example, to define a function to be exposed and available to your other controllers and views that do not belong to plugins. thadeusb: good question Nathan. mdipierro: Good point. LoganLK: Say a request goes to site.com/post/foo . Web2py would first check the application directory for a controller named post, if it didn't exist then it would iterate over the plugin dirs for the controller. thadeusb: That is kinda what I've been trying to get at. Are plugins like wordpress that you install and they work (meaning you go to /init/plugin_/function) thadeusb: Or are plugins ment to define a standard codebase that is reused across multiple applications (like comments, ratings) mdipierro: First this is a mess. The plugin would not work if there is a controller with the same name. Second this does not address what happens to modules. third, still this would not allow me to create plugins that define new helpers for example since they would be called ONLY when invoked by the user. thadeusb: They only define the things I need such as models, functions, forms, etc.. everything that makes up this subset, but it is still up to me as the developer to integrate this functionality into MY site. mdipierro: /init/plugin_/function is what we have now. LoganLK: mdipierro: If you had a controller called comments, its likely you had already implemented your own comment system = why would you need to use the plugin? + It's python. In the dal, bootstrap/whatnot file , just import the helper form the plugin thadeusb: there is no way a plugin can know how to integrate into MY website correctly and exactly the way I want it to. - 22:55 - LoganLK: which is why they should be applications specific.. because your editing the hell out of it LoganLK: brb thadeusb: I either have to A) edit the plugin, or hope the plugin was designed to be configurable and flexible mdipierro: Logan is right we could go a long way by writing string conventions to make sure they do not mess up each other and assumptions mengu: what a plugin is consistent of? mdipierro: That is the point. Different people expect different things. mdipierro: The current system allows to build all those different things. mrfreeze: Side question: why is the environment copied in LOAD? mengu: we cannot adapt wordpress' plugin system. we don't need hooks in our applications. Turicas: plugins are little apps :} mdipierro: Because if you do LOAD(ajax=True) a new environment is created and if you do LOAD(ajax=False) you want it to behave the same without ajax (for example LOAD(vars=dict(...)) you do not want it to override the request.vars of the parent page/ mrfreeze: ah, thanks. mdipierro: Let's pause for a minute and ... Turicas: drink a coffee? mdipierro: anybody should define what they... - 23:01 - mdipierro: expect the work "plugin" to mean and what basic features it should have. mdipierro: I start. hehehe rppowell: Greetings. mdipierro: I define two types of plugins: low level plugins: "any subset of an app, what we have today"; high level plugin: "a special subset of an app that is authonomous, does not conflict with other plugins, but can still change the behaviour of the parent app". In the second case I have in mind a CSM-like app, althought extensible mdipierro: your turn - 23:06 - mrfreeze: High level plugins sounds closer to what I want. mengu: it should extend the application. not become the application itself. mdipierro: explain mengu mengu: think about a simple application. like a blog. what you need with it? for example pagination. thadeusb: comments thadeusb: ratings mrfreeze: From a plugin developer standpoint I want a way to package modules, components, scripts, etc. into re-usable chunks that can be consume by developers more easily. From a plugin user standpoint, I want to control which resources plugins use and explicitly control how they are exposed. mengu: yes, such general things that will use models. it can also create new models too. mengu: would i sound like an idiot if i suggest that instead of file system or keeping the plugins in the application, what if the plugins are re-usable python apps? so we can go ahead and import them and use in our app. - 23:11 - mdipierro: how is what you propose different than these: http://www.web2py.com/plugins/default/sortable mrfreeze: who? mengu: how do i install it? download and import via the app manager? mengu: if so, it's not what i mean. mdipierro: yes or just unzip over the app. mdipierro: what do you mean mdipierro: ? mrfreeze: who? mdipierro: mengu said "it's not what I mean" mengu: i'll write something like "python web2py.py install-plugin sortable" in my terminal. and the plugin will be installed to my system, not to my application. i will import it in my app when i need it. thadeusb: I view plugins more like the higher level plugin, but not too high level. I view plugins as subsets of commonly used code (authentication, emailing, comments, tagging, ratings, sorting, tablefying). I don't wantt he plugin to BE my application, I want it to provide functionality so that I don't have to re-invent the wheel. mdipierro: I thnk you Mr Freeze and I agree. We need a convention. I am trying to figure out if pleople want something else mengu: that also requires gluon to be in PYTHONPATH - 23:17 - mrfreeze: I would definitely prefer the sub-app approach if possible. It feels more like a python package. mrfreeze: but i'll take what i can get mdipierro: @menu. for now let's define "menu-plugins" as thing that go somewhere, not in the app, but that apps can choose to use. Turicas: i agree with mengu on plugins can be 'global' Turicas: really, i think a plugin can be either global or only from an app Turicas: for example: routes.py is global - but i think we need routes.py by each app too. so i think plugins should be this way too mdipierro: mengu-plugins can be created easily by creating a plugin manager app (could admin itself) and expose functionality via REST for example. mdipierro: I agree about routes but has nothing to do with plugins Turicas: no? Turicas: plugins are here to be reusable Turicas: if I have 5 apps and want to use 'comments' plugin i'll need to install it 5 times Turicas: if a new version of this plugin comes i need to upgrade 5 plugins Turicas: where is 'reusable' there? mdipierro: So Turcas and menu both want "menu-plugins". Turicas: mengu* mdipierro: What I am trying to get at is that different people want different this and they do not exclude each other. They can all be implemented on top of the current system. mengu: mdipierro: it's mengu - 23:22 - mdipierro: "mengu-turicas-plugins" can be implemented easily by making a plugin manager application. But here is the problem... mrfreeze: I want to be able to distribute modules though a plugin. Would this be possible with a folder-based system? Turicas: i want that a plugin have the same directory-tree of an app mengu: mdipierro: global web2py plugin as python app is just a suggestion. we don't have to do it. mdipierro: if your app uses a mengu-turicas-plugin, how to you package the plugin with it? You app is no longer authonmous you have to introduce the concept of dependences and I do not know of any system that manages dependencies properly. Two apps many need different version of the same plugins for example mdipierro: @mrfreeze, right now you have that mrfreeze: I know, just wandering if a folder-based system excludes it. Turicas: mrfreeze, why excludes? mrfreeze: if a folder-based system has downsides, i want to know mdipierro: It just makes it more difficult as everything else. The plugins folder cannot be added to the sys.path beacuse since we host multiple apps and multiple plugins the result would be dependent on the order plugins have been instaled - 23:27 - mdipierro: Are we taling about a folder within the app or a folder at the web2py level? mrfreeze: folder in app Turicas: i think: Turicas: web2py/plugins/pluginX Turicas: AND Turicas: web2py/applications/my-app/plugins/pluginX mdipierro: each of them has downsides: mdipierro: web2py/plugins/pluginX Turicas: the concept i think is: plugins are apps (little) and apps can have other apps inside it (as plugins) mrfreeze: i prototyped a sub-app based plugin system. it seemed to work but the URLs got tricky. mdipierro: plugins would not longer be packages and distributed with the app AND If two apps require different versions of the some plugin you are screwed. mdipierro: web2py/applications/my-app/plugins/pluginX mrfreeze: web2py/applications/my-app/plugins/pluginX is how I think of it LoganLK: +1 for web2py/applications/app/plugins/pluginX mdipierro: What would this do that you do not have today. It is just a different implementation + you would no longer be able to have plugins that define helpers for example or alter the db or provide a new layout. mrfreeze: run_models_in wouldn't work? LoganLK: you would be able to define helpers. It's python after all. Just import plugins.foobar.helper - 23:33 - mdipierro: not so simple mdipierro: say you have mdipierro: models/db.py mdipierro: plugins/a/models/d.py mdipierro: plugins/d/models/a.py mdipierro: in which order should they be executed? LoganLK: your main app model(s) first. Then plugins are iterated through mrfreeze: The plugin system is instantiated in the application model and executes the plugin models then Turicas: mdipierro, but i want to provide a new layout, define helpers etc. LoganLK: why would you want to define a whole new layout in a plugin? Turicas: i want web2py to run applications/app-name/models/*.py AND applications/app-name/plugins/*/models/*.py Turicas: mdipierro, in the order you wrote Turicas: first: app's model Turicas: second: plugins' model starting buy plugin_name (ascendent) Tobarja: announcement: new plugin: 000000000000000000000000000000_me_first mdipierro: This /app-name/plugins/*/models/*.py break backward compatibility. What if the app has a plugin.py? This can only be done IF you create a controller/plugin.py Turicas: hahaha mdipierro: lol Turicas: backward compability sucks. anyone here have plugin.py controller? no? ok, so we can use it mdipierro: lol - 23:38 - thadeusb: I actually am starting to agree with Turicas from earlier... "Do we need a plugin system?" I think we are trying to stick a box into a circle. Tobarja: on a serious note, using an order(like alphabetical) could mean you're tweaking module names to get the desired effect mrfreeze: plugin would become a reserved word mdipierro: LOL thadeusb: What do we "really" need? thadeusb: 1) A way to register plugins (author, version) Turicas: a way that apps can talk well each other? thadeusb: 2) A way to automatically update plugins (auto download from PluginCentral) thadeusb: Turicas: Why do apps need to talk to each other? mdipierro: Why are we discussin an implementation issue? Whether a plugin is in app/plugins/a/modules/a.py or in app/modules/plugin_a.py is just an implementation issue. What does it buy you? Tobarja: shouldn't apps talk to each other via another method? json/xml/rpc Turicas: thadeusb, if they talk nice with each other we probably don't need a plugin system mdipierro: we are not talking about app-app communication. mdipierro: I think this discussio has sidetracked. thadeusb: I can see two apps needing to talk to each other. Say you have a "blog" app, and a "photos" app, if you wanted to aggregate the two together in a "latests" app it needs access to both databases.... but again, this is about plugins not app communication. Turicas: ok Turicas: i was thinking in little apps... Turicas: comment app, rating app... mrfreeze: I think of auth and crud as plugins. You instantiate them with the db you want and expose them through a url . mdipierro: As usual we end up discussing where files go and that is irrelevant to me. It would require lots of changes in web2py source. thadeusb: if you have a comment plugin and a rating plugin, its up to you to write the glue code to make them "talk" to each other. assuming the plugins were designed in such a way to allow for this LoganLK: Existing system is nice. I downloaded the comment plugin, edited it to fit my needs and away I went. The plugin is bundled with my application, not some global web2py plugin dir or site-packages. Do we agree that it works asis now? - 23:43 - mdipierro: But nobody has made a case why location of files is important. What I would be interested in is conventions. They are an issue whether we relocate the files or not. thadeusb: I don't see anything a new system would solve that we do not currently have in the current implementation. mdipierro: What should a plugin rely on (db, auth, service)? mdipierro: what should we do so that it does not pollute the namespace? Turicas: thadeusb, but what is a plugin? can it be an app? mrfreeze: Storage object for plugin resources? thadeusb: What we are lacking is a naming convention, and a "design" convention, such as plugins *should* be designed *this* way not *that* way. thadeusb: I currently do this in my plugins mdipierro: A plugin can be an entire app since an app is a subset of itself. thadeusb: Please take a look at this http://code.google.com/p/blogitizor/source/browse/src/models/plugin_comments.py thadeusb: Just the first couple of lines, mdipierro: I like it thadeusb: I create a meta plugin_comments Storage() object... all settings live inside this object. All functions are plugin__ mdipierro: That is what I do too. thadeusb: Unfortunately this makes for very long variables/function names... but there should be no pollution unless you have two comment plugins mdipierro: I think that is what should formalize LoganLK: +1 thadeusb: Also view the same comment implemented (Auth/Crud) like thadeusb: http://code.google.com/p/blogitizor/source/browse/src/modules/plugin_comments.py thadeusb: It is a class that resides in Modules, and you have to tell the plugin to initialize the database, and you pass a DAL object to it. thadeusb: This keeps the plugin from being able to access your "entire" database. - 23:48 - mdipierro: Do we need to make a choice? I like them both. They are both consistent with current mechanism. Why not have a convention that allows both? Turicas: if not 'db' in globals() or not 'auth' in globals(): Turicas: raise HTTP(500, 'plugin_comments requires "db" and "auth"') thadeusb: As Massimo has been saying, it is all up to the plugin developer on how he wants to expose it... if you don't like plugins that auto use your db globals() then don't use that plugin =/ Turicas: what to having something like get_all_databases() ? Turicas: i really DONT like the ideia of having the name 'db' fixed for plugins' use mrfreeze: what if you have 2 db connections? thadeusb: So we need a convention to determine what database a plugin uses. Turicas: mrfreeze, so get_all_databases() will return a list with 2 DAL objects thadeusb: Here is a question Turicas: mydb = DAL('...', export_for_plugins=False) thadeusb: Does the database need to be different between plugins? Or can all plugins use the same database although different from your main database. mdipierro: right now the scaffoling app has db.py which defined db, auth, service, mail Turicas: my_other_db = DAL('...', export_for_plugins=True) mdipierro: and crud Turicas: other_db = DAL('...', export_for_plugins=True) Turicas: so get_all_databases should return [my_other_db, other_db] mdipierro: if anybody has more than once db connection that would have to edit that and create another crud for the other connection rppowell: When you use a plug in, don't you have to import/init it? Is there a way to pass parameters to the plugin, like 'db' or database defs? thadeusb: IWoudl it be ok to have a `plugin_db = DAL('...')` and then all plugins should assume this. For normal uses you can have `plugin_db = db` LoganLK: In some bootstrap/dal.py/plugin.py file, why not import the plugin and pass db connections as args? mdipierro: no Alvaro. DAL does not need to know about plugins Turicas: ooooor we can define what databases plugin will use Turicas: plugin_X.databases = [a, b, c] Turicas: (even if I have a, b, c, d, e etc. in my models) mrfreeze: plugins = Plugins(globals(),db) Turicas: mrfreeze, what about a multi-database app? rppowell: Or, the plugin can have some base 'defaults', but can be defined in some plugin config setup? mdipierro: in the new DAL mdipierro: databases = [x for x in globals() if isinstance(globals()[x],DAL)] thadeusb: So should plugin models always look at a PluginManager() class to get the `db`, `auth`, `session` from ? Could a convention be that plugins should never reference the "global" objects? And if so, then it is not an "official" web2py plugin? Turicas: PEP8 Massimo, please! mdipierro: I think plugins = Plugins(globals(),db) is a reasonable option - 23:53 - LoganLK: why not pass the object (dal, session, whatever) instances into the plugins? Turicas: mdipierro, what about a multi-database app? Turicas: hey thadeusb: So for my first comments example, it will go... thadeusb: plugins.db.define_table('plugin_comments', ....) Turicas: we are talking about getting the right database for plugin to use Turicas: and not about permission, right? Turicas: plugin will have all permissions mrfreeze: turicas - db1=DAL(...) db2 = DAL(...) plugins=Plugins(globals(),db2) Turicas: so i think we don't need to "don't use globals" LoganLK: comments = Comments(settingsTuple=commentSettings, db=dal(instance)) mdipierro: the plugin already sees everything. Why limit what the plugin can read. I am more concerner adout what the plugin can write. Turicas: SO I think that this code will help us: mdipierro: I agree lvaro. Turicas: databases = [database for database in globals() if isinstance(globals()[database], DAL)] Turicas: i can get the defined databases with this line in my plugin mrfreeze: yes, it's just a convention, not a security measure mdipierro: mind only in new DAL in old dal: Turicas: and if my app wants that plugin X write in database1 and not in database2? we can use meta information on plugin to do it! Turicas: plugin_XXX.settings.database = db2 mdipierro: databases = [database for database in globals() if isinstance(globals()[database], (SQLDB,GQLD] and you would have to import SQLDB and GQLDB thadeusb: WE will need a check thadeusb: if plugin_XXX.settings: thadeusb: plugin_xxx.settings = Storage() LoganLK: Why are people against instantiating and passing in arguments for the plugin? thadeusb: else: thadeusb: use the existing object ? mrfreeze: cycling through globals is slow mdipierro: If you have one database we can assume it is "db". If you have multiple daabases why should it specified outside the plugin? Each plugin may want to use a different one. thadeusb: LoganLK: Because plugins can be models as well (meaning they are executed by web2py like all other models). Turicas: should DATABASES be global? Turicas: (each time DAL is instantiated web2py should append the new instance to DATABASES) mdipierro: I do not see why we need to access globals. A plugin never needs all databases. It needs one. Somewhere in the plugin we need to specify which one. mdipierro: We can just refer to it by name. no? - 23:58 - Turicas: mdipierro, but I don't think we need to restrict access to all databases Turicas: and don't restrict the number of databases a plugin can access thadeusb: Would each plugin ever need its "own" db instance? Turicas: thadeusb, i think not Turicas: what about an css-only plugin? thadeusb: Meaning, would you have one instance for YOUR models, and another for one plugin and another for another plugin? - Wednesday March 10 2010, 00:00 - Turicas: but i think a plugin can create its own database thadeusb: I don't want a plugin making databases for me... what If I am on a shared server with only a limited amount of databases available ? LoganLK: you wouldn't install the plugin? Turicas: thadeusb, so f*** you what are you using this plugin? thadeusb: Also, I don't want my plugin using SQLITE either, thats a nightmare on a production system. Turicas: *why mdipierro: I am not suggesting we restrict it. I am just saying the plugin can discover them (and we agree on that) but most plugins just want one and we need to specify which one. MrFreeze suggested plugin=Plugin(db) or plugin_db=db. Both solutions assume that you specify it outside the plugin itself. I think this should be specified in the plugin itself plugin_me.settings.db=db so that different plugins can act on different dbs if mdipierro: LOL thadeusb: Turicas: I probably wouldn't be =) LoganLK: in some plugin.py (or dal.py, or a bootstrap.py equiv) why not have a load_plugins = ['comments', 'ratings'] thadeusb: I think we agree though that plugins should not need to act on different databases? LoganLK: plugin_args = ['db1': dal (instance of dal), 'settings': settingsClassForAll] Turicas: so we'll work as in django: hey, use just one database! LoganLK: and then web2py will automatically import those plugins and pass the args automatically mdipierro: I think we agree that most plugins will not but we should not limit this mrfreeze: thadeus, i disagree. I don't want plugins touching my business data. Turicas: mrfreeze, so don't use plugins Turicas: or use just your plugins...or trusted plugins mdipierro: What about we require that each plugin starts with thadeusb: mrfreeze: you misunderstand me... I am saying there is no reason for [db = business, db1 = comment_plugin, db2 = rating_plugin] thadeusb: We just need: PLUGIN_DB mdipierro: plugin_me=Storage(dict(db=db)) - 00:05 - mrfreeze: gotcha thadeusb: so both comment_plugin and rating_plugin will just use PLUGIN_DB. Which does not contain business data. mdipierro: and if you do not want to use db (because it should not touch your data) do mdipierro: plugin_me=Storage(dict(db=other_db)) Turicas: hey Turicas: we are talking about plugins accessing all data but mrfreeze don't wants this mdipierro: no PLUGIN_DB because then all plugin will be forced to use the same PLUGIN_DB and that is not walways what you want. mrfreeze: as long as I can separate it, i'm good Turicas: do we really need to restrict this? LoganLK: It would be ignorant of any developer to install a plugin if they have not looked at the source already. therefore we shouldn't be worried about putting framework level safeguards/sandboxes in. thadeusb: I don't want plugins touching my business database either... but at the same time I don't think we need a database for each plugin, I think plugins can share a dedicated plugin database. Turicas: create some environemnt, as in views? LoganLK: "oh no. im worried firefox will steal my user info from my computer so i wont install it" = not a valid reason mdipierro: comments/rating/etc need auth and auth uses db. mdipierro: I see very little use for a plugin that does not uses auth. Turicas: mdipierro, it could uses just session Turicas: or even just cookies mdipierro: yes. then this is not a concern - 00:10 - mrfreeze: what about: plugin = Plugin(globals()) plugin.register('myplugin',db1) plugin plugin.register('another',db2) mdipierro: IF the plugin needs a db than we the plugin needs to be configured mdipierro: we all agree witht his? thadeusb: Ok so if we really want thadeusb: plugins = PluginManager() thadeusb: plugins['comment'].db = DAL('sqlite:comments') thadeusb: plugins['ratings'].db = DAL('firebird:ratings') thadeusb: plugins['default'].db = db thadeusb: # in models/plugin_comments.py thadeusb: if not plugins['comment']: thadeusb: use default thadeusb: else: thadeusb: use existing mdipierro: not quite mrfreeze: massimo - i agree, we need per plugin config mdipierro: if we do this than, as I said, we assume all plugins use one db and all use the same db. mdipierro: I think each plugin needs mdipierro: plugin_.settings.db=db thadeusb: .... mdipierro: or mrfreeze: perhaps a model in the app dedicated to plugin config mdipierro: plugins_.settings.dbs=[db,db1,db2,db3] mdipierro: if it wants more than one thadeusb: why would aplugin ever want more than one db! and even then, how would itk now what to do with each one!? thadeusb: Please give me a real world example of where this would be applied... there is no since in needing functionality that will never be used and just get in the way later. mdipierro: we could have a plugin_meta that allows to configure plugin by editing them for us. That is another story I think. We still need more specs first LoganLK: for the billionth time, why not pass the objects - config, session, dal, form, request as arguments into the plugin? Turicas: mdipierro, plugin_.settings.db is not a good choice Turicas: i think we need some type of sandbox Turicas: if we need to use 'plugin_.settings.db' it'll be VERY slow mdipierro: A plugin that exposes a function that merges data from dba into dbb. This could be useful for apps that share data across network. dba could be temporary-in-memory db. Turicas: each time to access some database thing, python will search in plugin_. after in settings mdipierro: Why slow? mdipierro: I see what you are saying. Turicas: sooo Turicas: to be easier to me I'll do: - 00:15 - mdipierro: plugin__db=db Turicas: to be easier to me I'll do in my plugin: thadeusb: plugins[''] is just a Storage() object containing the plugins settings. Could this not also be used to store the db objects? Turicas: db = plugin_.settings.db thadeusb: With this the plugin could "merge" the settings with its defaults. Turicas: so I can use db.table.insert, db().select() etc. Turicas: mdipierro, plugin__db is a long name Turicas: it'll be so boring to do: mdipierro: You can do this inctrollers but not in plugin models since they are executed all in the same namescape before controllres and you do should not redefine db. Turicas: plugin__db(plugin__db.table1.user_id == plugin__db.table2.id).select(plugin__db.table1.something, plugin__db.table2.blabla) mdipierro: You can do mdipierro: _db = plugin_.settings.db thadeusb: plugins = PluginManager() thadeusb: plugins['comment'].db = DAL('sqlite:comments') thadeusb: plugins['comment'].require_captcha = False thadeusb: # in models/plugin_comments.py thadeusb: if 'require_captcha' not in plugins['comment'].keys(): thadeusb: plugins['comment'].require_captcha = True nfreeze: weird, guess i'm nfreeze now. thadeus, imagine a charting plugin that reads data from one db and stores calculated chart data in another mdipierro: what if we just say that: thadeusb: This provides the ability to define plugin settings before the plugin is executed... while still allowing the plugin to fill in the defaults where needed. thadeusb: SOOO if you hacve a plugin that needs multiple databases thadeusb: plugins['merger'].databases = [db1, db2, db3] mdipierro: plugins can use any variable that start wth underscore but should not expect that variable to be passed to the controllers Turicas: we need more KISS mdipierro: Thadeus. The issue here is multiple databases. The issue is that Nathan and I do not want a global plugins. Since one plugin may act one db (for comments) and one may act on another one (for ratings) thadeusb: This is why plugins should be classes, that way they can be passed the variables they need, and then refer to the variables as `self.` thadeusb: Massimo. There is no issue. nfreeze: i'm still here! - 00:20 - thadeusb: Massimo. Maybe I am not making my point clear enough. Turicas: thadeusb, if you need plugins as classes you need modules, don't plugins thadeusb: plugins['comment'].db = db1 thadeusb: plugins['rating'].db = db2 mdipierro: I see what you suggest then thadeusb: what is wrong with this implementation? It provides everything that we are attempting to accomplish... AFAIK Turicas: hey mdipierro: so the configuration for all plugins would be in a single place thadeusb: Then when the model/plugin_comment.py gets executed, it will check the global plugins variable, to get its settings/database/everything from. thadeusb: Exactly! Turicas: i think TDD can help us in getting all questions that we need thadeusb: but the plugins variable is just a dictionary with the plugin name in it... nothing special. thadeusb: so its a dictionary of storage objects... basically, used for plugin settings and variables. mdipierro: we can make it a storage than and import it from gluon.tools. This may be a good solution. thadeusb: in the instance where a setting does not exist in this storage object, the plugin can set a default parameter, like how jQuery plugins can be passed options and they are merged together with the defaults. mdipierro: I would also require that if not plugins in globals(): raise HTTP(400,'plugin missing') mdipierro: Everybody agrees with this? nfreeze: i agree. mdipierro: with Thadeus proposal I mean? nfreeze: yes mdipierro: so in gluon.tools mdipierro: class PluginManager(Storage): pass mdipierro: and in db.py mdipierro: plugins = PluginManager() - 00:25 - Turicas: should a plugin access other plugins' data? mdipierro: one moment Alvaro mdipierro: it needs to be little smarter because we want plugins.name.db and not plugins.name['db']. right? thadeusb: Turicas: I would think no, because a plugin should be self contained. nfreeze: massimo - yes thadeusb: massimo - yes mdipierro: So class PluginManager must be more complex but I think I know how to do this. mdipierro: How do we avoid the problem of mdipierro: plugin.name.db.define_table(....) is this too long or ok? Turicas: thadeusb, what if myplugin try to access plugins.yourplugin.db? Turicas: mdipierro, _db = plugin.name.db Turicas: hey mdipierro: so specs should say that we allow local vars with _ Turicas: mdipierro, you could put plugin_x's models in some function Turicas: so all _db etc. will be local to that function thadeusb: Massimo, I don't see any other way and be able to keep the current functionality but have no namespace pollution. thadeusb: It might have to be something we need to "live" with for plugin development. mdipierro: by the name of the function would pollute the namespace unless we have a convention for that mdipierro: sould all plugins say mdipierro: def plugin__function(....): pass mdipierro: plugin__function(plugins) mdipierro: so you can have local variables inside? - 00:30 - thadeusb: If only there was a way to create a multi-line lambda function Turicas: mdipierro, i was just thinking in a way to web2py encapsulates plugins' in a 'sandbox' thadeusb: plugin.name.functionX = lambda... but this would be impossible for complex functions. mdipierro: I agree thadeusb: you agree with? mdipierro: you nfreeze: the sub-app approach avoids this, no? or is that off the table? thadeusb: the sub-app approach does avoid this, but also how would the sub-app expose functions to be used elsewhere in your app. mdipierro: it is not off the table but does not avoid this problem because models in sub-app still need to be executed in the same namespace. It is not on the table either since there is no advantage mdipierro: why do you say it avoids this? thadeusb: I think for functions a naming convention plugin__functionX, this will only be fore publicly exposed functions such as plugin_comments_get_latest() thadeusb: I do not see any problem with this, because if I want to call a public function of a plugin, it better be in the globals "somewhere" mdipierro: it would avoid it ONLY if the models in sub-add where to be executed in a different context and they would have very limited functionality mdipierro: how about nfreeze: i thought they were executed in a copied environment mdipierro: than they could not expose any function to controllers. - 00:36 - nfreeze: gotcha thadeusb: It is kind of a "web2py idiom" to have lots of things in the global namespace. thadeusb: The only problem with it is that it is frowned upon in the academia of software development. mdipierro: that is what define models (something that gets exposed everywhere) if you do not want that put it in modules. mdipierro: I am not saying everything should be exposed. I am saying things that you want exposed go there. thadeusb: What if the plugins functions were actually in a module. mdipierro: The less you put there the better thadeusb: And then thadeusb: plugins..functions = local_import('plugin_') thadeusb: plugins.name.functions.get_latest() thadeusb: **It doesnt have to be functions, just an example** mdipierro: no because the module would not see request, response, session, cache, T. You would have to pass everything. thadeusb: true. thadeusb: that would be a pita. thadeusb: unless it was exec_file instead of local_import? mdipierro: then you either execfile( ) in {..session, request, etc etc...} mdipierro: or just execfile. In the latter case you still expose everything - 00:41 - mdipierro: what about mdipierro: def _(plugin): ..... mdipierro: _(plugins.name) Turicas: guys, i need to go. good night for you. please make decisions made with common sense (and think about KISS . []s nfreeze: thanks turicas. good night. mdipierro: I will have to go too in few minutes. We can do this again. Thanks Alvaro nfreeze: yes, i am too sleepy to think. Turicas: mdipierro, we need to do this all weeks and need PEP8 and TDD in web2py code too ;D mdipierro: all weeks can be hard but we'll try. Turicas: cya thadeusb: Nite thadeusb: I am staying for a bit logner thadeusb: longer* mdipierro: I can stay another 15mins nfreeze: me too thadeusb: is def _(plugin): valid syntax? mdipierro: yes nfreeze: so the issue is : how can plugins expose functions to applications? nfreeze: without polluting the global namespace? mdipierro: everything in def _(..) will be local mdipierro: everything outside gets exposed mdipierro: def _(plugin): plugin.db.define_table(....) mdipierro: _(plugin.this) mdipierro: def plugin_this_function(): pass # is exposed - 00:47 - thadeusb: What is the point of having a function named "_" that gets overridden in the next plugin? mdipierro: that is the point. It only lives there. It never needs to be called outside mdipierro: we can call it _run if you prefer mdipierro: or __init__ thadeusb: Are db definitions the only thing that will go in this function? thadeusb: I'm just confused, why have a function named _ that only defines the tables when those don't need to be in a function to declare them mdipierro: no. everything that you pass to plugins.name.xxx mdipierro: btw. the plugin should also do mdipierro: if not plugins.name.db: plugins.name.db=db to set a default. not? thadeusb: OOOOK I get you thadeusb: _(p): thadeusb: p.db.define_table() thadeusb: p.X = Y thadeusb: _(plugin.name) thadeusb: This is to shorten so that you don't have to do plugins.name.db, you can just do p.db thadeusb: ? - 00:53 - mdipierro: yes thadeusb: I also agree that assume db a default if not defined. thadeusb: Ok. I like this. SO then functions that are private can be declared within the scope of _ mdipierro: same for auth if needed thadeusb: yes nfreeze: yes mdipierro: what about crud and services? nfreeze: plugins should have a README with requirements thadeusb: same thing, they should be defined in the settings, or the plugin can assume the defaults... or raise an exception if it does not exist. mdipierro: yes mdipierro: where? I can think of tw places thadeusb: Where would the plugins README file be stored? thadeusb: poke poke you owe me a coke! mdipierro: app/plugin_name.readme thadeusb: +1 nfreeze: sounds good mdipierro: or mdipierro: app/plugins/plugin_name.readme thadeusb: app/plugin_name.readme mdipierro: ok that requires a change in admin but a minor one thadeusb: Ok so I have one last concern. mdipierro: I think we made some progress. What do you think? thadeusb: So a plugin will have version and author information. thadeusb: What happens when we want to upgrade a plugin (automatically like through plugincentral) nfreeze: i think so. i would still like the sub-app approach but understand if it's not possible. mdipierro: and license. yes. It could be read automatically on top of the admin/plugin page - 00:58 - thadeusb: I assume admin will be used for this. How will admin discover this information? mdipierro: It already discovers plugins to that would not be a problem. mdipierro: One more thing to sleep over. mdipierro: What about even higher level plugins that may want to: mdipierro: 1) register a new menu item mdipierro: 2) register events to respond to thadeusb: 1 - We need a better menu system mdipierro: I think we may want more conventions. thadeusb: 2 - web2py needs a proper event signal/slot system anyways mdipierro: Any volunteer to draft what we said? nfreeze: i nominate thadeus mdipierro: +1 to both comments mdipierro: I second thadeusb: Lol ! nfreeze: haha mdipierro: all in in favor? thadeusb: Ok, I can do it, but it might be until the weekend. nfreeze: aye thadeusb: For now I have the IRC logs and will post those so everyone can read them thadeusb: So we need to schedule another IRC chat, and we can set a list of topics for that chat. thadeusb: 1) Refine plugin conventions thadeusb: 2) Discuss a proper menu system nfreeze: 3) explore or kill the sub-app issue forever thadeusb: 4) Signal/slots for plugins and web2py mdipierro: yes. 1) and 2) should be easy. 4) is another big can of worms but we need to open it. thadeusb: ok so our next IRC discussion will be on topics 1, 2, and 3, and maybe begin on 4.