Programmatically fetch all data from multiple cloud apps like salesforce, sap and oracle using soap based web services

Question

I need to programmatically fetch almost all data from multiple applications like salesforce, sap and oracle. Most of these applications provide soap based APIs via web services. I wrote a bit of client code using Apache Axis (wsdl2java) to fetch records from one entity (table) such as contacts. There are tens or hundreds of entities in most applications. I figure I would've to write lot of code to get all the data. Is there a better way to go about this?

Answer 1

If you want to extract data on a regular basis, you can use Talend Open Studio (Opensource) to extract all data; we are doing that using Talend in several SF projects. It generates java code that you can deploy on any server running java and use cron as a scheduler.

We have also written a java based app that uses metadata to dynamically fetch all SF custom objects in order to automatically extract the daily changes from each object for daily (incremental) backup.

Klaus

Answer 2

Ok. This is an attempt to answer my own question. I just got an idea: I am thinking I might be able to achieve this by figuring out mapping between all the objects and their corresponding wrapper classes that Axis or jax-ws generates and then creating the meta-data (object_type, class_name, method_name, data_types, etc.,) and then having a meta-data driven code to fetch all the records. Do you think that would be a good approach?

Really appreciate if I could get more ideas and comments from the community. Thanks very much.

Answer 3

I realize you said you need to programmatically access the data, but have a look at salesforce's data export feature.

For programatic export using the SOAP API, if you're wanting to stay loosely typed and not hard code the entity list you support, you'll probably want to use the Partner API. You'll do a describeGlobal() to get the list of supported entities, aka Objects. Then you can do a describeSObjects() to get the list of fields for an Object. At that point you can query from that Object and retrieve the records.

You might also want to look at the Bulk API, which is designed for dealing with large data sets. The payload is CSV, which may be easier for you to deal with.