List of Figures

1.1. CloverETL Designer
1.2. CloverETL Server
1.3. CloverETL Cluster
1.4. CloverETL Community Edition
7.1. CloverETL Designer Splash Screen
7.2. Workspace Selection Dialog
7.3. CloverETL Designer Introductory Screen
7.4. CloverETL Help
8.1. Choose licensing
8.2. Dialog for specifying license
8.3. License agreement
8.4. Select Activate online radio button, enter your license number and password and click Next.
8.5. Confirm you accept the license agreement and click Finish button.
9.1. Choosing Install New Software in Help Menu
9.2. Available Software (in Eclipse 4.3)
9.3. Adding a Repository to the Eclipse
9.4. Installing Eclipse Plugin
9.5. Overview of CloverETL Plugins to be Installed
9.6. Accepting License Agreements
9.7. License Manager
10.1. License Manager showing installed licenses.
10.2. CloverETL License dialog
13.1. SmartScreen warning
13.2. Launching installer from SmartScreen
13.3. User Account Control Preventing the Intallation
13.4. Windows 10 Firewall
14.1. ETL Server Integration
14.2. Show component icon: enabled
14.3. Show component icon: disabled
14.4. Show component background: enabled
14.5. Show component background: disabled
14.6. Show component description: enabled
14.7. Show component description: disabled
14.8. Show rich tooltips: enbled
14.9. Show rich tooltips: disabled
14.10. Open context menu for newly created edge: enabled
15.1. CloverETL Runtime
15.2. Accessing CloverETL Runtime menu
15.3. Restarting CloverETL Runtime
15.4. Adding library to classpath using VM parameters
15.5. CloverETL Runtime - Logging
15.6. Setting the Master password
15.7. CloverETL Runtime - User Classpath
15.8. User Classpath - Advanced Options
16.1. ETL Server Integration
16.2. ETL Server Integration
17.1. Execution Monitoring
18.1. Setting The Java Runtime Environment
18.2. Preferences Wizard
18.3. Installed JREs Wizard
20.1. Refresh Operation
21.1. CloverETL Perspective
21.2. Selecting Components
21.3. Components in Palette
21.4. Removing Components from the Palette
21.5. Graph Editor with an Opened Palette of Components
21.6. Closing the Graphs
21.7. Rulers in the Graph Editor
21.8. Grid in the Graph Editor
21.9. A Graph before Selecting Auto-Layout.
21.10. A Graph after Selecting Auto-Layout.
21.11. Six New Buttons in the Tool Bar Appear Highlighted (Align Middle is shown)
21.12. Alignments from the Context Menu
21.13. Navigator Pane
21.14. Outline Pane
21.15. Outline Pane - Subgraphs
21.16. Outline Pane with Minimap
21.17. Show Element ID Enabled
21.18. Opening the ETL Graph Cleanup
21.19. ETL Graph Cleanup
21.20. Locking an Element
21.21. Locking an Element - Message dialog
21.22. Accessing a locked graph element - you can add any text you like to describe the lock.
21.23. Properties Tab
21.24. Console Tab
21.25. Problems Tab
21.26. Clover - Regex Tester Tab
21.27. Execution tab of a graph running on CloverETL Designer
21.28. Execution tab of a graph running on CloverETL Server
21.29. Execution tab of a clustered graph running on CloverETL Cluster
22.1. Placeholder File
22.2. Naming a CloverETL Project
22.3. CloverETL Project subdirectories
22.4. CloverETL Server Project Wizard - Server Connection
22.5. CloverETL Server Project Wizard - Sandbox Selection
22.6. CloverETL Server Project Wizard - Clustered Sandbox Creation
22.7. Giving a Name to the New CloverETL Server Project
22.8. CloverETL Examples Project Wizard
22.9. Renaming CloverETL Examples Projects
22.10. Context Menu with Convert to Server Project option
22.11. Convert local project to CloverETL Server project wizard
22.12. Convert local project to CloverETL Server project wizard II
22.13. Convert to Local Project
22.14. Project Folder Structure inside Navigator Pane
22.15. Opening the Workspace.prm File
22.16. Workspace.prm File
22.17. Basic Eclipse Perspective
22.18. Selecting CloverETL Perspective
22.19. CloverETL Perspective
22.20. Connection failed
22.21. Connection reestablished
22.22. CloverETL Connection
22.23. CloverETL Connection
23.1. Creating a New Graph
23.2. Naming a New CloverETL Graph
23.3. Graph Editor with a New Graph and the Palette of Components
23.4. Components Selected from the Palette
23.5. Components are Connected by Edges
23.6. Creating an Input File
23.7. Creating the Contents of the Input File
23.8. Metadata Editor with Default Names of the Fields
23.9. Metadata Editor with New Names of the Fields
23.10. Edge Has Been Assigned Metadata
23.11. Opening the Attribute Row
23.12. Selecting the Input File
23.13. Input File URL Attribute Has Been Set
23.14. Output File URL without a File
23.15. Output File URL with a File
23.16. Defining a Sort Key
23.17. Sort Key Has Been Defined
23.18. Running the Graph
23.19. Result of Successful Run of the Graph
23.20. Contents of the Output File
24.1. Running a Graph from the Main Menu
24.2. Running a Graph from the Context Menu
24.3. Running a Graph from the Upper Tool Bar
24.4. Successful Graph Execution
24.5. Console Tab with an Overview of the Graph Execution
24.6. Counting Parsed Data
24.7. Run Configuration - Main Tab
24.8. Run Configuration - Parameters Tab
24.9. Run Configuration - Refresh tab
24.10. Connect to Job action position in the main toolbar
24.11. Connect to Job action position in the Execution view
24.12. Connect to Job dialog
25.1. URL File Dialog
25.2. URL File Dialog - Local files
25.3. URL File Dialog - Workspace view
25.4. URL File Dialog - Clover Server
25.5. URL File Dialog - Hadoop HDFS
25.6. Edit URL Dialog
25.7. Edit URL Dialog
25.8. Example of Generated OpenSSH Private Key
25.9. Edit URL Dialog - Amazon S3
25.10. URL File Dialog - Remote files
25.11. URL File Dialog - Input Port
25.12. URL File Dialog - Output Port
25.13. URL File Dialog - Dictionary
25.14. Configured URL File Dialog
25.15. Edit Value Dialog
25.16. Find Wizard
25.17. Go to Line Wizard
25.18. Open Type Dialog
26.1. Import (Main Menu)
26.2. Import (Context Menu)
26.3. Import Options
26.4. Import Projects
26.5. Import from CloverETL Server Sandbox Wizard (Connect to CloverETL Server)
26.6. Import from CloverETL Server Sandbox Wizard (List of Files)
26.7. Import Graphs
26.8. Import Metadata from XSD
26.9. Import Metadata from XSD - Review
26.10. Import Metadata from DDL
27.1. Export Options
27.2. Converting Graph to Jobflow
27.3. Converting Graph to Jobflow II
27.4. Converting Jobflow to Graph
27.5. Converting Jobflow to Graph
27.6. Converting Subgraph to Graph
27.7. Converting Subgraph to Graph II.
27.8. Export Graphs to HTML
27.9. Export to CloverETL Server Sandbox
27.10. Export Image
28.1. Edge tracking example
28.2. An example of a medium level of tracking information
28.3. An example of a high level tracking information
28.4. Enlarging the Font of Numbers
28.5. Setting the Font Size
29.1. CloverETL Search Tab
29.2. Search Results
30.1. CloverETL Server Project Displayed after Opening CloverETL Designer
30.2. Prompt to Open CloverETL Server Project
30.3. Opening CloverETL Server Project
30.4. Network connections window
31.1. Add Components dialog - finding a sorter.
31.2. Find Components dialog - the searched text is highlighted both in component names and description.
31.3. Edit Component Dialog (Properties Tab)
31.4. Running a Graph with Disabled Component
31.5. With Default PassThrough
31.6. With PassThrough to second Output Port
31.7. Simple Renaming Components
31.8. Running a Graph with Various Phases
31.9. Setting the Phases for More Components
31.10. Component allocation dialog
31.11. Allocation cardinality decorator
31.12. Defining Group Key
31.13. Defining Sort Key and Sort Order
31.14. Creating Metadata from a Template
32.1. Selecting the Edge Type
32.2. Creating Metadata on an empty Edge
32.3. Assigning Metadata to an Edge
32.4. Metadata in the Tooltip
32.5. Debugging options
32.6. Properties of an Edge
32.7. Filter Editor
32.8. Debug Properties Dialog
32.9. Choosing Inspect Data from Context Menu
32.10. Data Inspector
32.11. Search Data
32.12. Search Options
32.13. Export Debug Data to CSV
32.14. Data Inspector Preferences
32.15. Debug mode in the Properties tab
32.16. Debugging option in the Debug Configuration window
33.1. Externalizing and/or Exporting Internal Metadata
33.2. Selecting a Location for a New Externalized and/or Exported Internal Metadata
33.3. Creating External (Shared) Metadata in the Main Menu and/or in the Navigator Pane
33.4. Internalizing External (Shared) Metadata
33.5. Metadata propagation: metadata is propagated from the first edge on the left side to all connected edges.
33.6. Changing auto-propagated metadata to user-defined.
33.7. Changing user-defined metadata to auto-propagated.
33.8. Different priorities of metadata propagation
33.9. Metadata propagated from the component
33.10. Metadata propagated from the component II.
33.11. Metadata propagated from the component, metadata template is defined within the component.
33.12. Metadata propagated from the another edge
33.13. Metadata propagated from a distant edge
33.14. Advanced metadata propagation - DataIntersection
33.15. Overview of directions of metadata propagation
33.16. Metadata propagated from an unconnected distant edge
33.17. Metadata propagated from the another edge
33.18. Extracting Metadata from Delimited Flat File
33.19. Extracting Metadata from Fixed Length Flat File
33.20. Setting Up Delimited Metadata
33.21. Setting Up Fixed Length Metadata
33.22. Extract Metadata from Excel Spreadsheet Wizard
33.23. Format Extracted from Spreadsheet Cell
33.24. Extracting Internal Metadata from a Database
33.25. Database Connection Properties Dialog
33.26. Selecting Columns for Metadata
33.27. Generating a Query
33.28. DBF Metadata Editor
33.29. Extract metadata from Salesforce - specify connection
33.30. Extract metadata from Salesforce - enter SOQL query
33.31. Extract metadata from Salesforce - edit created metadata
33.32. Specifying Lotus Notes connection for metadata extraction
33.33. Lotus Notes metadata extraction wizard, page 2
33.34. Merging two metadata - conflicts can be resolved in one of the three ways (notice the radio buttons at the bottom).
33.35. Creating Database Table from Metadata and Database Connection
33.36. Metadata Editor for a Delimited File
33.37. Metadata Editor for a Fixed Length File
33.38. Trackable Fields Selection in Metadata Editor
34.1. Creating Internal Database Connection
34.2. Externalizing Internal Database Connection
34.3. Internalizing External (Shared) Database Connection
34.4. Database Connection Properties Dialog
34.5. Adding a New JDBC Driver into the List of Available Drivers
34.6. Advanced tab of the Database connection dialog
34.7. JNDI resource - Basic tab
34.8. JNDI resource - Basic tab
34.9. Using password from secure graph parameter
34.10. Connecting to MS SQL with Windows authentication.
34.11. Adding path to the native dll to VM parameters.
34.12. Externalizing JMS Connection from Context menu
34.13. Edit JMS Connection Wizard
34.14. QuickBase Connection Dialog
34.15. Lotus Notes Connection Dialog
34.16. Hadoop Connection Dialog
34.17. MongoDB Connection Dialog
34.18. MongoDB Connection Dialog - Advanced Tab
34.19. Creating Salesforce Connection
34.20. Salesforce Connection Dialog
34.21. Salesforce Connection Dialog II
34.22. Salesforce Connection Dialog III
35.1. Creating Internal Lookup Table
35.2. Externalizing Wizard
35.3. Selecting Lookup Table Item
35.4. Lookup Table Internalization Wizard
35.5. Lookup Table Wizard
35.6. Simple Lookup Table Wizard
35.7. Edit Key Wizard
35.8. Simple Lookup Table Wizard with File URL
35.9. Simple Lookup Table Wizard with Data
35.10. Changing Data
35.11. Database Lookup Table Wizard
35.12. Appropriate Data for Range Lookup Table
35.13. Range Lookup Table Wizard
35.14. Persistent Lookup Table Wizard
35.15. Aspell Lookup Table Wizard
36.1. Creating a Sequence
36.2. Editing a Sequence
36.3. A New Run of the Graph with the Previous Start Value of the Sequence
37.1. Externalizing Internal Parameters
37.2. Internalizing External (Shared) Parameter
37.3. Example of a Graph parameters editor
37.4. Graph Parameters Type Editor
37.5. Select Editor Type Dialog
37.6. Edit Parameter Value
37.7. Edit Parameter Value
37.8. Multiline string parameter - configuration
37.9. Multiline string parameter - validating XML
37.10. File URL Dialog - Configuration
37.11. Select Types Dialog - Choosing file extension(s)
37.12. Properties - Usage
37.13. Single Field - Configuration
37.14. Single Field - Choosing the Field
37.15. Multiple Fields - Configuration
37.16. Multiple Fields - Choosing the Field
37.17. Field Mapping - Configuration
37.18. Field Mapping - Choosing the Field
37.19. Join Key - Configuration
37.20. Join Key - Configuration
37.21. Enumeration - Configuration
37.22. Character set
37.23. Time Zone - Configuration
37.24. Time Zone - Usage
37.25. Field Type
37.26. Locale
37.27. Graph parameters editor with secure attribute
37.28. Filter Component Configured by Graph Parameter
39.1. Dictionary Dialog with Defined Entries
40.1. Pasting a Note to the Graph Editor Pane
40.2. Enlarging the Note
40.3. Highlighted Margins of the Note Have Disappeared
40.4. Changing the Note Label
40.5. Writing a New Description in the Note
40.6. A New Note with a New Description
40.7. Toolbar for Format Editing
40.8. Note with formatted text and markup
40.9. Toolbar for Format Editing
40.10. A Folded Note
40.11. Properties of a Note
41.1. Define Error Actions Dialog
41.2. Transformations Tab of the Transform Editor
41.3. Copying the Input Field to the Output
41.4. Mapping of Inputs to Outputs (Connecting Lines)
41.5. Editor with Fields and Functions
41.6. Input Record Mapped to Output Record Using Wildcards
41.7. Transformation Definition in CTL (Source Tab)
41.8. Java Transform Wizard Dialog
41.9. Info after Java Transform Wizard Dialog
41.10. Confirmation Message
41.11. Transformation Definition in CTL (Transform Tab of the Graph Editor)
41.12. Content Assist (Record and Field Names)
41.13. Content Assist (List of CTL Functions)
41.14. Error in Transformation
41.15. Transformation Definition in Java
42.1. Parallel Run
42.2. Parallel Run
42.3. Parallel Run with Cluster Components
42.4. Component Allocation
42.5. Component Allocation
43.1. Subgraph Layout
43.2. Example of subgraph with multiple output ports
44.1. Subgraph Component
44.2. Example of User-defined Component
45.1. Original graph without subgraphs
45.2. Wrapping components into a subgraph
45.3. Wrapping Subgraph Wizard
45.4. CloverETL Graph with the Subgraph Component
45.5. A new Subgraph
45.6. Export as subgraph parameter button
45.7. Public parameter appeared as a subgraph component attribute
45.8. Use parameter as value button
45.9. Setting up an Optional Port
45.10. Setting up an Optional Port in Graph Editor
45.11. Setting up Color of Subgraph
45.12. Setting up Icon of Subgraph
45.13. Dialog for Filling Required Parameters
45.14. Subgraph providing metadata
45.15. Metadata propagated from Subgraph component
45.16. Subgraph explicitly defines input metadata for customers
45.17. Using subgraph with matching metadata
45.18. Generic subgraph not defining explicit metadata in its body
45.19. Metadata propagate through the Subgraph component
46.1. Subgraph - Reader
46.2. Subgraph - Writer
46.3. Subgraph - Transformer
46.4. Subgraph - Executor
51.1. Main .rjob editor
51.2. Run Configuration of Data Services
51.3. Data Service test result in console
52.1. Reading body content from the port of Input component
52.2. Convert Graph to Data Service
52.3. Export to Data Service REST job - I.
52.4. Export to Data Service REST job - II.
52.5. Export to Data Service REST job - III.
52.6. Publishing multiple Data Services at once
52.7. Unpublishing multiple Data Services at once
55.1. Viewing Data in Components
55.2. XML Features Dialog
55.3. Configuring prefix selector in ComplexDataReader. Rules are defined in the Selector properties pane. Notice the two extra attributes for regular expressions.
55.4. Sequences Dialog
55.5. A Sequence Assigned
55.6. Edit Key Dialog
55.7. Generated Query with Question Marks
55.8. Generated Query with Output Fields
55.9. Reading records from database
55.10. Reading query from input port
55.11. Incremental reading - first read
55.12. Incremental reading - second read
55.13. Mapping to Clover fields in EmailReader
55.14. Incremental reading - first read
55.15. Incremental reading - second read
55.16. JSONExtract - mapping the list
55.17. Example mapping of nested arrays - the result.
55.18. SpreadsheetDataReader Mapping Editor
55.19. Basic Mapping – notice leading cells and dashed borders marking the area data will be taken from
55.20. The difference between global data offsets set to 1 (default) and 3. In the right hand figure, reading would start at row 4 (ignoring data in rows 2 and 3).
55.21. Global data offset is set to 1 to all columns. In the third column, it is locally changed to 3.
55.22. Rows per record is set to 4. This makes SpreadsheetDataReader take 4 Excel rows and create one record out of their cells. Cells actually becoming fields of a record are marked by a dashed border; therefore, the record is not populated by all data. Which cells populate a record is also determined by the data offsets setting, see the following bullet point.
55.23. Rows per record is set to 3. The first and third columns contribute to the record by their first row (because of the global data offset being 1). The second and fourth columns have (local) data offsets 2 and 4, respectively. Thus the first record will be formed by 'zig-zagged' cells (the yellow ones – follow them to make sure you understand this concept clearly).
55.24. Retrieving format from a date field. Format Field was set to the "Special" field as target.
55.25. Reading mixed data using two leading cells per column. Rows per record is 2, Data offset needed to be raised to 2 – looking at the first leading cell which has to start reading on the third row.
55.26. The Mapping Dialog for XMLExtract
55.27. Parent Elements
55.28. Editing Namespace Bindings in XMLExtract
55.29. Selecting subtype in XMLExtract
56.1. Viewing Data on Components
56.2. Generated Query with Question Marks
56.3. Generated Query with Input Fields
56.4. Generated Query with Returned Fields
56.5. EmailSender Message Wizard
56.6. Edit Attachments Wizard
56.7. Attachment Wizard
56.8. Defining the Bean structure - click the Select combo box to start.
56.9. Mapping editor in JavaBeanWriter after first open.
56.10. Example mapping in JavaBeanWriter
56.11. Mapping editor in JavaMapWriter after first open.
56.12. Example mapping in JavaMapWriter
56.13. Mapping arrays in JavaMapWriter - notice the array contains a dummy element 'State' which you bind the input field to.
56.14. Mapping editor in JSONWriter after first open. Metadata on the input edge(s) are displayed on the left hand side. The right hand pane is where you design the desired JSON tree. Mapping is then performed by dragging metadata from left to right (and performing additional tasks described below).
56.15. Example mapping in JSONWriter - employees are joined with projects they work on. Fields in bold (their content) will be printed to the output file - see below.
56.16. JSONWriter mapping
56.17. Spreadsheet Mapping Editor
56.18. Explicit mapping of the whole record
56.19. The difference between global data offsets set to 1 (default) and 3. In the right hand figure, writing would start at row 4 with no data written to rows 2 and 3.
56.20. Global data offsets is set to 1. In the last column, it is locally changed to 4. In the output file, the initial rows of this column would be blank, data would start at D5.
56.21. With Rows per record set to 2 in leading cells Name and Address, the component always writes one data row, skips one and then writes again. This way, various data does not get mixed (overwritten by the other one). For a successful output, make sure Data offsets is set to 2.
56.22. Rows per record is set to 3. Data in the first and third column will start in their first row (because of their data offsets being 1). The second and fourth columns have data offsets 2 and 4, respectively. The output will, thus, be formed by 'zig-zagged' cells (the dashed ones – follow them to make sure you understand this concept clearly).
56.23. Writing into a template. Its original content will not be affected, your data will be written into Name, Surname and Age fields.
56.24. Partitioning by one data field
56.25. Mapping summary
56.26. Create Mask Dialog
56.27. Tableau Table Structure
56.28. Mapping Editor
56.29. Adding Child to Root Element
56.30. Wildcard attribute and its properties
56.31. Attribute and its properties
56.32. Element and its properties
56.33. Mapping editor toolbar
56.34. Binding of Port and Element
56.35. Generating XML from XSD root element
56.36. Source tab in Mapping editor
56.37. Content Assist inside element
56.38. Content Assist for ports and fields
56.39. Writing non-standard xml
57.1. Denormalizer code workflow
57.2. Merge
57.3. Normalizer code workflow
57.4. Rollup code workflow
57.5. SimpleGather
57.6. XSLT Mapping
57.7. An Example of Mapping
58.1. Source Tab of the Transform Editor in Joiners
58.2. Hash Join Key Dialog
58.3. Join Key Wizard (Master Key Tab)
58.4. Join Key Wizard (Slave Key Tab)
58.5. LookupJoin - how it works
58.6. Edit Key Wizard
58.7. LookupJoin with Range Lookup Table
58.8. An Example of the Join Key Attribute in the RelationalJoin Component
58.9. Join Key Wizard (Master Key Tab)
58.10. Join Key Wizard (Slave Key Tab)
59.1. Example of typical usage of Barrier component
59.2. Example of mapping for the Fail component
59.3. Example of Loop component usage
61.1. Usage example of ParallelRepartition component
61.2. Example of actual working of ParallelRepartition component in runtime
62.1. DataBase Configuration
62.2. AddressDoctor Parameters
62.3. Input mapping wizard
62.4. Input mapping wizard
62.5. Output mapping
62.6. Transform Editor in ProfilerProbe
62.7. Import/Externalize metrics buttons
62.8. Validator rules editor
62.9. Validator - Active rules
62.10. Validator - Error output mapping
62.11. Validator - If - then - else without else branch
63.1. Foreign Key Definition Wizard (Foreign Key Tab)
63.2. Foreign Key Definition Wizard (Primary Key Tab)
63.3. Foreign Key Definition Wizard (Foreign and Primary Keys Assigned)
63.4. Transform Editor in HTTPConnector
63.5. Multipart entities in input mapping
63.6. Transform Editor in HTTPConnector
63.7. Choosing WS operation name in WebServiceClient.
64.1. Join Key Wizard (Master Key Tab)
64.2. Join Key Wizard (Slave Key Tab)
64.3. An Example of the Join Key Attribute in ApproximativeJoin Component
64.4. Matching Key Wizard (Master Key Tab)
64.5. Matching Key Wizard (Slave Key Tab)
67.1. Inspect Action Pop-up Dialog
67.2. Expressions View