doc/developers-guide/producers.xml

   1 <chapter id="introduction">
   2 <title>The producer framework</title>
   3 Please read the presentation of the <glossterm linkend="producers">producers</glossterm> concept
   4 for an introduction.
   5
   6 <section><title>How to use the <filename>producers.xml</filename> file</title>
   7   [FIXME: this should be in the admin's guide, not here]
   8
   9 <para>
  10 Please check the standard <filename>producers.xml</filename> file
  11 for two fully commented, real-world examples :
  12 the <code>&lt;nodedefinition name="Language"&gt;</code> node
  13 and the <code>&lt;producer name="articles"&gt;</code> node.
  14 </para>
  15
  16   <section><title>Introduction</title>
  17
  18   <para>
  19   Mir allows admins to fully configure the producer tasks and set up
  20   arbitrary producers through the <filename>producers.xml</filename> file.
  21   </para>
  22
  23   <para>
  24   Producers consist of "nodes". Every node has a specific function.
  25   For example, it is possible to use a node to generate a file out of a template.
  26   Or it is possible to use a node to enumerate over a collection of articles.
  27   </para>
  28
  29
  30
  31   <para>
  32   A producer is defined using a Producer tag:
  33   <programlisting>
  34     &lt;producer name="content"/&gt;
  35   </programlisting>
  36   This would define a producer named <code>content</code>.
  37
  38   </para><para>
  39   In a producer, <emphasis>verbs</emphasis> must be defined.
  40   Verbs are sub-tasks of a producer.
  41
  42   <programlisting>
  43
  44     &lt;producer name="content"&gt;
  45       &lt;verbs&gt;
  46         &lt;verb name="new"&gt;
  47         &lt;/verb&gt;
  48         &lt;verb name="all"&gt;
  49         &lt;verb&gt;
  50       &lt;/verbs&gt;
  51     &lt;producer name="content"/&gt;
  52   </programlisting>
  53
  54   This would define a producer with verbs named <code>all</code> and <code>new</code>.
  55
  56   </para><para>
  57   And also the specific nodes and their relationship should be specified:
  58
  59   <programlisting>
  60
  61     &lt;producer name="content"&gt;
  62       &lt;verbs&gt;
  63         &lt;verb name="new"&gt;
  64         &lt;/verb&gt;
  65         &lt;verb name="all"&gt;
  66         &lt;verb&gt;
  67       &lt;/verbs&gt;
  68       &lt;body&gt;
  69         &lt;Generate
  70             generator="/producer/startpage.template"
  71             destination="${config.storageRoot}/index.shtml"/&gt;
  72       &lt;/body&gt;
  73     &lt;/producer&gt;
  74   </programlisting>
  75
  76   we will later learn that this producer generates a single file.
  77
  78   </para><para>
  79   Producers can be made to do different things for different verbs:
  80
  81   <programlisting>
  82
  83     &lt;producer name="content"&gt;
  84       &lt;verbs&gt;
  85         &lt;verb name="new"&gt;
  86           &lt;Set key="count" value="3"/&gt;
  87         &lt;/verb&gt;
  88         &lt;verb name="all"&gt;
  89           &lt;Set key="count" value="5"/&gt;
  90         &lt;verb&gt;
  91       &lt;/verbs&gt;
  92       &lt;body&gt;
  93         &lt;Generate
  94             generator="/producer/startpage.template"
  95             destination="${config.storageRoot}/index.shtml"/&gt;
  96       &lt;/body&gt;
  97     &lt;/producer&gt;
  98   </programlisting>
  99
 100   if a producer is called with a specific verb, first the nodes of that verb
 101   are processed, and only thereafter the body: in our case, if the producer
 102   <code>content</code> is called with verb <code>new</code>, first the variable <code>count</code>
 103
 104   is set to <code>3</code>, and after that, a file is generated.
 105
 106 </para>
 107   </section>
 108 <section><title>Node arguments</title>
 109   <para>
 110   Nodes can have arguments.
 111   Arguments that amount to integer values can be direct expressions.
 112   Arguments that amount to text values can be enriched with expressions between ${}.
 113
 114   </para>
 115   <para>
 116   Some examples:
 117   <programlisting>
 118     &lt;Log message="This article has the following title: ${content.title}"/&gt;
 119
 120   </programlisting>
 121   <code>Log</code> has 1 mandatory argument, <code>message</code>. This argument should be text,
 122         but can be enriched with expressions enclosed by ${ and }. In this example,
 123         the title of an article is logged.
 124
 125   <programlisting>
 126     &lt;Set key="age" value="34+22*(3+2)"/&gt;
 127   </programlisting>
 128   <code>Set</code> has 2 mandatory argument: <code>key</code> and <code>value</code>.
 129         The key arugment is a fixed text,
 130         the value argument is a direct expression, in this case of arithmetic nature.
 131
 132   </para>
 133
 134   </section>
 135 <section><title>Expressions</title>
 136
 137
 138   <para>
 139   Expressions, either direct expressions, or expressions between ${}, can contain the following
 140   constructions:
 141     </para>
 142
 143   <para>
 144   <table border="1">
 145     <tr><td>a string literal</td>       <td><code>'hello'</code></td></tr>
 146     <tr><td>a numeric literal</td>      <td><code>2138</code></td></tr>
 147
 148     <tr><td>a variable reference</td>   <td><code>content.title</code></td></tr>
 149     <tr><td>arithmetic operators</td>   <td><code> 3 + 4 *(2+5-2)</code></td></tr>
 150     <tr><td>string operators</td>   <td><code> 'hello' ++ ' ' ++ 'Mir'</code></td></tr>
 151
 152     <tr><td>boolean operators</td>      <td><code>(content.id==3) or (content.id in (5,7,2,8) and (content.title!='hello')</code></td></tr>
 153   </table>
 154
 155   </para>
 156   </section>
 157 <section><title>Node types and statements</title>
 158     <para>
 159     Here's  a list of different node types that can be used inside
 160   producer definitions.
 161 Currently, there exists only one statement (&lt;nodedefinition&gt;), that
 162 is declared outside producer definitions.
 163
 164
 165     Here an overview:
 166
 167     </para>
 168     <table border="1">
 169
 170       <tr><td>Name</td>                       <td bgcolor="#eea8a8"><code>Set</code></td></tr>
 171       <tr><td>Purpose</td>                    <td>Alter a variable's using a free expression</td></tr>
 172       <tr><td colspan="2">
 173               Arguments</td></tr>
 174       <tr><td>key</td>                        <td>The variable</td></tr>
 175
 176       <tr><td>value</td>                      <td>The expression to set the variable to</td></tr>
 177       <tr>
 178         <td>Example</td>
 179         <td>
 180           <code>
 181             &lt;Set key="data.result" value="3 + 5 * (5-2)"/&gt;
 182
 183           </code>
 184         </td>
 185       </tr>
 186
 187
 188       <tr><td>Name</td>                       <td bgcolor="#eea8a8"><code>Define</code></td></tr>
 189       <tr><td>Purpose</td>                    <td>Alter a variable's using a string</td></tr>
 190       <tr><td colspan="2">
 191
 192               Arguments</td></tr>
 193       <tr><td>key</td>                        <td>The variable</td></tr>
 194       <tr><td>value</td>                      <td>The string to set the variable to</td></tr>
 195       <tr>
 196         <td>Example</td>
 197
 198         <td>
 199           <code>
 200             &lt;Define key="filename" value="/var/www/${content.id}.shtml"/&gt;
 201           </code>
 202         </td>
 203       </tr>
 204
 205
 206
 207       <tr><td>Name</td>                       <td bgcolor="#eea8a8"><code>If</code></td></tr>
 208
 209       <tr><td>Purpose</td>                    <td>Create a conditional part of a producer</td></tr>
 210       <tr><td colspan="2">
 211               Arguments</td></tr>
 212       <tr><td>condition</td>                  <td>The expression to test</td></tr>
 213       <tr><td colspan="2">
 214               Sub tags</td></tr>
 215
 216       <tr><td>then</td>                       <td>The part to process if the expression evaluates to true</td></tr>
 217       <tr><td>else</td>                      <td>The part to process if the expression evaluates to false</td></tr>
 218
 219
 220       <tr><td>Name</td>                       <td bgcolor="#eea8a8"><code>nodedefinition</code>&nbsp;&nbsp;(statement)</td></tr>
 221
 222       <tr><td>Purpose</td>                    <td>Acts as a "function" (or "macro") that can be "called" elsewhere in a producer.
 223 More precisely, it is a way to define a new producer node type
 224 inside the  <filename>producers.xml</filename> file.
 225 (check the <code>Language</code> node in that file for an example).
 226 </td></tr>
 227       <tr><td colspan="2">
 228               Arguments</td></tr>
 229       <tr><td>name</td>                  <td>The name of the newly created node ("function")</td></tr>
 230       <tr><td colspan="2">
 231               Sub tags</td></tr>
 232
 233       <tr><td>parameters</td>                 <td>a list describing the arguments the "function" must be given </td></tr>
 234       <tr><td>definition</td>                 <td>the actual body of the function, containing the code that should be executed. Note that this may contain a <code>&lt;sub/&gt;</code> tag that is replaced by the child nodes of the calling node. </td></tr>
 235
 236
 237
 238
 239       <tr><td>Name</td>                       <td bgcolor="#eea8a8"><code>Log</code></td></tr>
 240
 241       <tr><td>Purpose</td>                    <td>Log a message in the producer log</td></tr>
 242       <tr><td colspan="2">
 243               Arguments</td></tr>
 244       <tr><td>message</td>                    <td>The message to log</td></tr>
 245
 246
 247
 248       <tr><td>Name</td>                       <td bgcolor="#eea8a8"><code>Enumerate</code></td></tr>
 249       <tr><td>Purpose</td>                    <td>Enumerate over the results of a query</td></tr>
 250       <tr><td colspan="2">
 251               Arguments</td></tr>
 252       <tr><td>key</td>                        <td>The variable name that receives the enumerated record</td></tr>
 253
 254       <tr><td>table</td>                      <td>The table that is used to enumerate over</td></tr>
 255       <tr><td>selection (optional)</td>       <td>The condition (where clause) of the query.</td></tr>
 256       <tr><td>order (optional)</td>           <td>The order in which the results of the query are enumerated.</td></tr>
 257       <tr><td>skip (optional)</td>            <td>The number of records to skip</td></tr>
 258
 259       <tr><td>limit (optional)</td>           <td>The maximum number of records to enumerate</td></tr>
 260       <tr><td colspan="2">
 261               Sub tags</td></tr>
 262       <tr><td colspan="2">
 263               This node can have subnodes that will be processed for every enumerated record</td></tr>
 264 <!--
 265       <tr>
 266         <td>Remarks</td>
 267         <td>
 268         </td>
 269       </tr>
 270       <tr>
 271         <td>Example</td>
 272         <td>
 273           <code>
 274             &lt;Generate generator="/producer/content.template" destination="/var/www/${content.date.formatted.yyyy}/${content.date.formatted.MM}/${content.id}.shtml"/&gt;
 275           </code>
 276         </td>
 277       </tr>
 278  -->
 279
 280
 281
 282
 283       <tr><td>Name</td>                       <td bgcolor="#eea8a8"><code>List</code></td></tr>
 284
 285       <tr><td>Purpose</td>                    <td>Store the results of a query into a variable</td></tr>
 286       <tr><td colspan="2">
 287               Arguments</td></tr>
 288       <tr><td>key</td>                        <td>The variable name that receives the result list</td></tr>
 289       <tr><td>table</td>                      <td>The table that is used to select from</td></tr>
 290
 291       <tr><td>selection (optional)</td>       <td>The condition (where clause) of the query.</td></tr>
 292       <tr><td>order (optional)</td>           <td>The order in which the results of the query are put into the list.</td></tr>
 293       <tr><td>skip (optional)</td>            <td>The number of records to skip</td></tr>
 294       <tr><td>limit (optional)</td>           <td>The maximum size of the list</td></tr>
 295
 296
 297
 298       <tr><td>Name</td>                       <td bgcolor="#eea8a8"><code>Batch</code></td></tr>
 299       <tr><td>Purpose</td>                    <td>Divide the results of a query into batches</td></tr>
 300       <tr><td colspan="2">
 301               Arguments</td></tr>
 302
 303       <tr><td>key</td>                        <td>The variable name that receives the batch</td></tr>
 304       <tr><td>infokey</td>                    <td>The variable name that receives meta information on the batches</td></tr>
 305       <tr><td>table</td>                      <td>The table that is used to select from</td></tr>
 306       <tr><td>batchsize</td>                  <td>The size of a batch (the first batch however varies in size)</td></tr>
 307
 308       <tr><td>selection (optional)</td>       <td>The condition (where clause) of the query.</td></tr>
 309       <tr><td>order (optional)</td>           <td>The order in which the results of the query are put into the list.</td></tr>
 310       <tr><td>skip (optional)</td>            <td>The number of records to skip</td></tr>
 311       <tr><td>process (optional)</td>         <td>The maximum number of batches to process</td></tr>
 312
 313       <tr><td>minbatchsize (optional)</td>    <td>The minimal size of the first batch</td></tr>
 314       <tr><td colspan="2">
 315               Sub tags</td></tr>
 316       <tr><td>batches</td>                    <td>The part to process for every batch</td></tr>
 317       <tr><td>batchlist</td>                  <td>The part to process once with the meta info</td></tr>
 318
 319
 320
 321
 322
 323
 324       <tr><td>Name</td>                       <td bgcolor="#eea8a8"><code>Generate</code></td></tr>
 325       <tr><td>Purpose</td>                    <td>Generate a page using a generator (i.e. an abstraction of a template)</td></tr>
 326       <tr><td colspan="2">
 327
 328               Arguments</td></tr>
 329
 330       <tr><td>generator</td>                  <td>the generator to use</td></tr>
 331       <tr><td>destination</td>                <td>the specification of the destination.</td></tr>
 332       <tr><td>parameters</td>                 <td>Additional configuration info for the generator (for
 333                                                   freemarker this now only contains the wanted encoding,
 334                                                   empty for the default).</td></tr>
 335
 336       <tr>
 337         <td>Remarks</td>
 338         <td>
 339            This node is used to have an actual page generated.
 340            The generator parameter usually is the name of a template.
 341            The destination is the file to be generated.
 342            Variable references are possible in all arguments, and, especially for the
 343                  destination attribute, widely used.
 344         </td>
 345       </tr>
 346       <tr>
 347         <td>Example</td>
 348
 349         <td>
 350           <code>
 351             &lt;Generate generator="/producer/content.template" destination="/var/www/${content.date.formatted.yyyy}/${content.date.formatted.MM}/${content.id}.shtml"/&gt;
 352           </code>
 353         </td>
 354       </tr>
 355
 356
 357
 358       <tr><td>Name</td>                       <td bgcolor="#eea8a8"><code>GenerateMedia</code></td></tr>
 359       <tr><td>Purpose</td>                    <td>
 360 The generateMedia node instructs the media handler associated with the media to
 361 "reproduce" the media. In practice this can mean generate an icon from an image,
 362 writing an image from the database to the web root, or create an m3u file or so.
 363 Media handling is limited at this moment, but  a serious
 364 redesign is planned right after 1.1
 365 </td></tr>
 366       <tr><td colspan="2">
 367
 368               Arguments</td></tr>
 369
 370       <tr><td>key</td>                  <td>FIXME???</td></tr>
 371
 372
 373
 374       <tr><td>Name</td>                       <td bgcolor="#eea8a8"><code>DeleteFile</code></td></tr>
 375       <tr><td>Purpose</td>                    <td>Delete a file</td></tr>
 376       <tr><td colspan="2">
 377               Arguments</td></tr>
 378       <tr><td>filename</td>                   <td>The filename of the file to delete</td></tr>
 379
 380
 381
 382
 383
 384       <tr><td>Name</td>                       <td bgcolor="#eea8a8"><code>SetFileDate</code></td></tr>
 385       <tr><td>Purpose</td>                    <td>Set a file's date</td></tr>
 386       <tr><td colspan="2">
 387               Arguments</td></tr>
 388
 389       <tr><td>filename</td>                   <td>The filename</td></tr>
 390       <tr><td>date</td>                       <td>The date to use</td></tr>
 391
 392
 393
 394
 395       <tr><td>Name</td>                       <td bgcolor="#eea8a8"><code>Resource</code></td></tr>
 396       <tr><td>Purpose</td>                    <td>Make a message resource bundle available. In other words, this defines a function that can be used in expressions to reference a bundle. </td></tr>
 397       <tr><td colspan="2">
 398
 399               Arguments</td></tr>
 400       <tr><td>key</td>                        <td>The name of the function.</td></tr>
 401       <tr><td>bundle</td>                     <td>The bundle to use</td></tr>
 402       <tr><td>language (optional)</td>        <td>The specific language to use</td></tr>
 403       <tr>
 404         <td>Example</td>
 405         <td>
 406           <code>
 407             &lt;Resource bundle="bundles.producer" key="lang" language="en"/&gt;
 408           </code>
 409           This makes the english language producer bundle available
 410           through the "lang()" function. One
 411           can then  use expressions like  ${lang("page.title")} to
 412           refer to a that value.
 413         </td>
 414       </tr>
 415
 416
 417
 418
 419
 420       <tr><td>Name</td>                       <td bgcolor="#eea8a8"><code>Execute</code></td></tr>
 421       <tr><td>Purpose</td>                    <td>Execute a script</td></tr>
 422       <tr><td colspan="2">
 423               Arguments</td></tr>
 424
 425       <tr><td>command</td>                    <td>The command to execute</td></tr>
 426
 427
 428
 429
 430
 431
 432       <tr><td>Name</td>                       <td bgcolor="#eea8a8"><code>ModifyContent</code></td></tr>
 433       <tr><td>Purpose</td>                    <td>Modify a field of an article</td></tr>
 434       <tr><td colspan="2">
 435
 436               Arguments</td></tr>
 437       <tr><td>key</td>                        <td>The variable containing the article</td></tr>
 438       <tr><td>field</td>                      <td>The field to modify</td></tr>
 439       <tr><td>value</td>                      <td>The value to set the field to</td></tr>
 440
 441
 442
 443       <tr><td>Name</td>                       <td bgcolor="#eea8a8"><code>MarkContent</code></td></tr>
 444       <tr><td>Purpose</td>                    <td>Mark an article as produced</td></tr>
 445       <tr><td colspan="2">
 446               Arguments</td></tr>
 447
 448       <tr><td>key</td>                        <td>The variable containing the article</td></tr>
 449
 450       <tr><td>Name</td>                       <td bgcolor="#eea8a8"><code>IndexContent</code></td></tr>
 451       <tr><td>Purpose</td>                    <td>adds/updates content to the search index (search engine)</td></tr>
 452       <tr><td colspan="2">
 453               Arguments</td></tr>
 454
 455       <tr><td>key</td>                        <td>The variable containing the article (FIXME????)</td></tr>
 456       <tr><td>pathToIndex</td>                        <td>(FIXME????)</td></tr>
 457
 458       <tr><td>Name</td>                       <td bgcolor="#eea8a8"><code>UnIndexContent</code></td></tr>
 459       <tr><td>Purpose</td>                    <td>removes content from the search index (search engine)</td></tr>
 460       <tr><td colspan="2">
 461               Arguments</td></tr>
 462
 463       <tr><td>key</td>                        <td>The variable containing the article (FIXME????)</td></tr>
 464       <tr><td>pathToIndex</td>                        <td>(FIXME????)</td></tr>
 465
 466     </table>
 467 </section>
 468 </section>
 469
 470 <section><title>How the producer framework is implemented</title>
 471
 472 <!-- <para>A Producer is a set of tasks, scripted in xml. Producers allow -->
 473 <!-- mir installations to have their own actions that can be called for -->
 474 <!-- instance when a new article is posted.  Originally producers were -->
 475 <!-- mostly used to generate pages, but they are used for a lot of -->
 476 <!-- other tasks such as pulling rss feeds for the global wire on -->
 477 <!-- indymedia.org. Producers are added and configured through the -->
 478 <!-- <filename>producers.xml</filename> file.</para> -->
 479
 480
 481 <para>The xml nodes contained within a <code>&lt;producer&gt;</code>
 482 tag  in the <filename>producers.xml</filename> file define
 483 a small  program.
 484 This program  (or script) may  contain constructs
 485 such as  <code>if</code> clauses, loops  and variables... The  program is parsed  into a
 486 tree  of <classname>ProducerNode</classname>s (figure).   The root  of this  tree is  defined  in a
 487 <classname>NodedProducer</classname> (which  is the  only class that  currently implements
 488 the  <classname>Producer</classname>  interface).   When  the producer  is  executed,  the
 489 <function>produce()</function> methods of each  node are recursively called, effectively
 490 executing the program as it was scripted. </para>
 491
 492 <figure><title>A producer in the <filename>producer.xml</filename> file is parsed into a tree of <classname>ProducerNodes</classname></title>
 493 <mediaobject>
 494 <imageobject>
 495 <imagedata fileref="figures/producer-node-tree-example.eps" format="EPS"></imagedata></imageobject>
 496 <imageobject>
 497 <imagedata fileref="figures/producer-node-tree-example.png" format="PNG"></imagedata></imageobject>
 498 <textobject>
 499 </textobject>
 500 </mediaobject>
 501 </figure>
 502
 503 <para>
 504 </para>
 505 </section>
 506 </chapter>