How does talend handle modifications of job at run time?

Question

As I know when we create a project in talend, it creates three files namely project_name.item, project_name.properties and project_name.java. When we add a component to the project, the parameters for that component is taken from the component_java.xml file. When we specify the parameter values for that component and save the job, it will be automatically added in the item file. For example a typical _java.xml file for tMap component looks like this.

<COMPONENT>
   <HEADER PLATEFORM="ALL"
    SERIAL="" 
    VERSION="2.1" 
    STATUS="ALPHA" 
    COMPATIBILITY="WIN32" 
    AUTHOR="Talend" 
    RELEASE_DATE="20070703A" 
    STARTABLE="false" 
    EXTENSION="org.talend.designer.mapper" 
    SCHEMA_AUTO_PROPAGATE="false" 
    DATA_AUTO_PROPAGATE="false" 
    HAS_CONDITIONAL_OUTPUTS="true"
    PARTITIONING="AUTO" 
    >
      <SIGNATURE></SIGNATURE>
   </HEADER>

  <FAMILIES>
    <FAMILY>Processing</FAMILY>
  </FAMILIES>
   <DOCUMENTATION>
      <URL/>
   </DOCUMENTATION>
    <CONNECTORS>
        <CONNECTOR BUILTIN="true" CTYPE="FLOW" MIN_INPUT="1" MIN_OUTPUT="1"/>
        <CONNECTOR CTYPE="LOOKUP" COMPONENT="tAdvancedHash"/>
        <CONNECTOR CTYPE="ITERATE" MAX_OUTPUT="0" MAX_INPUT="0"/>
        <CONNECTOR CTYPE="SUBJOB_OK" MAX_INPUT="0"/>
        <CONNECTOR CTYPE="SUBJOB_ERROR" MAX_INPUT="0"/>
        <CONNECTOR CTYPE="COMPONENT_OK" MAX_OUTPUT="0"/>
        <CONNECTOR CTYPE="COMPONENT_ERROR" MAX_OUTPUT="0"/>
        <CONNECTOR CTYPE="RUN_IF" MAX_OUTPUT="0"/>
    </CONNECTORS>
   <PARAMETERS>
      <PARAMETER NAME="MAP" FIELD="EXTERNAL" NUM_ROW="1">
         <DEFAULT/>  
      </PARAMETER>
      <PARAMETER
        NAME="LINK_STYLE"
        FIELD="CLOSED_LIST"
        NUM_ROW="1"
      >
        <ITEMS DEFAULT="">
          <ITEM NAME="AUTO"   VALUE="AUTO" />
          <ITEM NAME="BEZIER_CURVE"   VALUE="BEZIER_CURVE" />
          <ITEM NAME="LINE"   VALUE="LINE" />
        </ITEMS>
      </PARAMETER>

      <PARAMETER NAME="TEMPORARY_DATA_DIRECTORY" FIELD="DIRECTORY"
            NUM_ROW="2" GROUP="STORE_ON_DISK"
      >
            <!--<DEFAULT>"__COMP_DEFAULT_FILE_DIR__/temp"</DEFAULT>
      --></PARAMETER>

      <PARAMETER NAME="PREVIEW" FIELD="IMAGE" SHOW="true" NUM_ROW="4">
         <DEFAULT/>
      </PARAMETER>

       <PARAMETER
      NAME="DIE_ON_ERROR"
      FIELD="CHECK"
      NUM_ROW="100"
      SHOW="false"
         >
      <DEFAULT>true</DEFAULT>
    </PARAMETER>
    <PARAMETER
      NAME="LKUP_PARALLELIZE"
      FIELD="CHECK"
      NUM_ROW="100"
      SHOW="false"
         >
      <DEFAULT>false</DEFAULT>
    </PARAMETER>
   </PARAMETERS>

    <ADVANCED_PARAMETERS>
      <PARAMETER NAME="ROWS_BUFFER_SIZE" FIELD="TEXT"
            REQUIRED="true" NUM_ROW="1" SHOW="true" GROUP="STORE_ON_DISK"
      >
            <DEFAULT>2000000</DEFAULT>
      </PARAMETER>

      <PARAMETER
        NAME="CHANGE_HASH_AND_EQUALS_FOR_BIGDECIMAL"
        FIELD="CHECK"
        NUM_ROW="10"
      >
        <DEFAULT>false</DEFAULT>
      </PARAMETER>
    </ADVANCED_PARAMETERS>

   <CODEGENERATION>
    <TEMPLATES INPUT="TMAP_OUT" OUTPUT="TMAP_IN">

      <TEMPLATE NAME="TMAP_OUT" COMPONENT="tMapOut">
        <LINK_TO NAME="TMAP_IN" CTYPE="ROWS_END"/>
      </TEMPLATE>

      <TEMPLATE NAME="TMAP_IN" COMPONENT="tMapIn"/>

    </TEMPLATES>
  </CODEGENERATION>
  <RETURNS/>

   <PLUGINDEPENDENCIES>
    <PLUGINDEPENDENCY ID="org.talend.designer.abstractmap" />
   </PLUGINDEPENDENCIES>

</COMPONENT>

And the project_name.item file looks like this (only component part I am showing)

<node componentName="tMap" componentVersion="2.1" offsetLabelX="0" offsetLabelY="0" posX="352" posY="64">
    <elementParameter field="TEXT" name="UNIQUE_NAME" value="tMap_1"/>
    <elementParameter field="EXTERNAL" name="MAP" value=""/>
    <elementParameter field="CLOSED_LIST" name="LINK_STYLE" value="AUTO"/>
    <elementParameter field="DIRECTORY" name="TEMPORARY_DATA_DIRECTORY" value=""/>
    <elementParameter field="IMAGE" name="PREVIEW" value="_xpwnoHOoEeSkDJ1bbi1FTg-tMap_1-PREVIEW.bmp"/>
    <elementParameter field="CHECK" name="DIE_ON_ERROR" value="true"/>
    <elementParameter field="CHECK" name="LKUP_PARALLELIZE" value="false"/>
    <elementParameter field="TEXT" name="ROWS_BUFFER_SIZE" value="2000000"/>
    <elementParameter field="CHECK" name="CHANGE_HASH_AND_EQUALS_FOR_BIGDECIMAL" value="false"/>
    <elementParameter field="TEXT" name="CONNECTION_FORMAT" value="row"/>
    <elementParameter field="CHECK" name="INFORMATION" value="false"/>
    <elementParameter field="MEMO" name="COMMENT" value=""/>
    <elementParameter field="CHECK" name="VALIDATION_RULES" value="false"/>
    <metadata connector="FLOW" label="out1" name="out1">
      <column comment="" key="false" name="newColumn_1" nullable="true" originalDbColumnName="newColumn" pattern="" sourceType="" type="id_String" usefulColumn="true"/>
      <column comment="" key="false" name="newColumn" nullable="true" pattern="" sourceType="" type="id_String" usefulColumn="true"/>
    </metadata>
    <nodeData xsi:type="MapperData">
      <uiProperties shellMaximized="true"/>
      <varTables sizeState="INTERMEDIATE" name="Var"/>
      <outputTables sizeState="INTERMEDIATE" name="out1">
        <mapperTableEntries name="newColumn_1" expression="row1.newColumn " type="id_String" nullable="true"/>
        <mapperTableEntries name="newColumn" expression="row1.newColumn1 " type="id_String" nullable="true"/>
      </outputTables>
      <inputTables sizeState="INTERMEDIATE" name="row1" matchingMode="UNIQUE_MATCH" lookupMode="LOAD_ONCE">
        <mapperTableEntries name="newColumn" type="id_String" nullable="true"/>
        <mapperTableEntries name="newColumn1" type="id_String" nullable="true"/>
      </inputTables>
    </nodeData>
  </node>

If we clearly see all the parameters in .item file are taken from tMap_java.xml file except metadata, nodedata (hence inputTables, outputTables etc..) which are generated at run time. All I need to know is how does talend generates this part and populates the same in .item file at run time? In fact I need to know how does metadata is generated in .item file for any component. My requirement is I want to generate a .item file from using _java.xml files of the component. I am not able to generate metadata part in .item file just by looking at the _java.xml file of the component.

Answer 1

TOS projects (repository, metadata, items...) are used only for designing the jobs (ie the graphical gui and wizards), not for executing them.

When a job is executed or exported, a bounce of parametrized javajet files are executed. Javajet code outputs regular .java files where component parameters are already resolved. This .java code is in turn compiled and packaged in a .jar file which is in turn executed.

In other words, everything sitting on the javajet side cannot be set at runtime, but only at compile time.

In addition, you cannot use parameter *_java.xml for low-level setting project properties, as this is the metadata schema for the codegenerator to feed parameters into javajet. It's a property file for the component, not the job itself.