Tuesday, June 17, 2014

Migrating from Classic FDM to FDMEE - Episode 0

Migrating or not migrating, that's the question :-)

I have been discussing with many customers about FDMEE migration and I would like to share my feedback with you.
First thing we need to bear in mind is that we are migrating to a "new" product based on different platform and providing several new functionality that will help us to get a better performance in our data load process.
Typical questions will be:

  • Do we have a migration tool (next > next > next ...)?
  • Can I reuse all metadata defined in my Classic FDM application (locations, import formats, etc.)?
  • What about my mappings? can I migrate them?
  • My application has lots of customization using VBScript, can I migrate them to FDMEE?
  • I have built some custom reports in FDM, can I reuse them in FDMEE?
  • What about Batch Loader?
  • So how can I migrate then?
  • and thousands more...
For me, there are two migration steps:
  1. Technical migration
  2. Functional migration
Let me clarify this. 
During the technical migration we will install FDMEE in our new architecture. We will register our source and target systems, configure credentials for ERP systems in ODI if needed, etc...
Why do I scope the rest of the migration as functional? Because I consider this migration a kind of once-in-a-lifetime opportunity to review your application design and get the best performance. Why would my application need to be reviewed? Here you have some reasons:
  • Classic FDM has some limitations that are now covered by FDMEE. For example, when multi-dimensional mappings were needed, most of the people used mapping scripts which don't perform so well. In FDMEE, we have a new mapping type called Multi-Dimensional mapping. This new mapping type will probably replace all your mapping scripts. In addition, you can use look-up dimensions in FDMEE to support your multi-dimensional mappings.
  • The more locations you have, the more you will have to maintain. With new data load rules you may simplify your location structure. Two locations for two different file types may be replaced by one location and two data load rules having each one import format.
  • Import process is almost always the bottle neck in Classic FDM. Mapping process is performed during the import process and 90% of the customers having performance issues during import follow this rule: Wrong mapping logic design -> poor performance during import (I have seen source files with 1000 rows taking more than 30 minutes...) It's a perfect chance to review your mapping logic in order to get the best performance. Make use of new mapping types!!! multi-dimensional, #SQL, and #FORMAT mask mappings.
  • You will have to re-write your scripts (import, mapping, event, and custom). Even if you keep VB for event and custom scripts, you will have to do some changes as VB Script is replaced by VB .NET. On the other hand, if you decide to use event and custom scripts in Jython...you will have to completely re-write them. Import scripts (dump) support Jython only. Integration scripts are not supported anymore as import scripts so they need to be implemented in BefImport event script (Jython or VB .NET). And lastly, VB mapping scripts are not supported anymore so you will have to re-write them as Jython or SQL mapping scripts (or replace by multi-dimensional mappings). Being said that, you will have a great opportunity to review your code which sometimes is not as efficient as you think. Also, if you forgot commenting your code, don't forget to do it now :-)
Some answers...

  • Do we have a migration tool (next > next > next ...)? Not in It will be probably be included in but we don't know what will be available for migration yet. The tool will be mainly for FDM metadata (locations, import formats, mappings...)
  • Can I reuse all metadata defined in my Classic FDM application (locations, import formats, etc.)? Yes you can. Some of them have been slightly re-designed but the core functionality is in FDMEE as well. 
  • What about my mappings? can I migrate them? Explicit, In, Between, Like mappings also exist in FDMEE. You will have to re-write your mapping scripts.
  • My application has lots of customization using VBScript, can I migrate them to FDMEE? Already answered above:-)
  • I have built some custom reports in FDM, can I reuse them in FDMEE? Crystal reports have been replaced by RTF templates created in BI Publisher. You can reuse the SQL queries but will have to rebuild the report in RTF format. I don't think migration utility will cover reports.
  • What about Batch Loader? Open batches are now available as a type of batch definition. Same functionality so you will not have many problems migrating them. 
  • So how can I migrate then? There is no unique answer for this question. It all depends on your application(s). In some cases it will be better to start from scratch in FDMEE so you can improve your design. You can also build custom SQL process to move objects such as import formats or locations between FDM and FDMEE databases.

Some advice...
  • Please, take your time for the migration
  • Review your design. Poor design in FDM will be a poor design in FDMEE so don't expect a big performance improvement if you keep it
  • Make use of new FDMEE functionality. It may take some time to rebuild your application but you will be happy with results
  • Find an experienced resource for application assessment. I would suggest someone not involved in the design of your current application. If I built the application I will hardly find any design issue
  • Work from high level to detailed level. That will help you to have a better understanding of your integration flow
  • Do you need 10 days? I would not be worry if you need some time to assess/review/migrate. Performing well during these stage will make you save lot of time/money :-)
Which is your feedback? Please comment!


  1. Nice post. Many clients are going through these same issues hoping that an migration tool will fix it but you can't change bad design with a technical migration.

    1. That's the key point. Customers don't realize they have a bad design until they get performance issues...

  2. I had many simple import scripts in FDM Classic that I was able to do away with completely due to the new functionality offered in FDMEE. I strongly agree with the advice given here. This should NOT be considered a migration, but rather it should be considered a ground up build in order to take advantage of the full functionality. You should keep this in mind when considering time & cost.

  3. Brilliant post Francisco, as usual!

  4. Good post! What should be mentioned that is no longer possible to alter the Ui anymore. There are customers which have special buttons with custom Scripts and so one. These guys have big worries regarding the Migration to fdm ee.

    1. Hi Tim, which kind of UI modifications you refer? Any customization regarding custom script execution should be replaced by new custom script functionality like having input parameters. Web and Custom scripts are now merged in Custom Scripts definition.
      It's true that you could easily customize classic FDM by defining custom options in the database. That's bit more complex now due to the way ADF works.
      What is also true is that any customer should know that customization may be impacted by upgrades/migrations flows...
      Let me know which customization they are worried about so I can better advice.

  5. Francisco - I have enjoyed your posts they have been informative. Here's a note for you though:

    Regarding Batch Processing - Normal Batch Processing does work. However I have tried Multi-Load with a text file and have not had success. Oracle support indicated that text files are not supported for Batch Multi-Load "Delopment has confirmed that loading a TEXT is not supported." and that I could request an enhancement for text files. Bit of a step backwards. Version is

    Anybody else experience this? Any solutions?



Thanks for feedback!