Why you should restart AIMS/MapGuide when testing ....

I noticed by chance that a few layers - which are all based on one Oracle view - were significantly slower in our production system but no performance issue occurred in our test environment. Whilst investigating the issue I made a few mistakes which made the whole process of finding and fixing the issue more time consuming than necessary. Here are the details:

After noticing the issue I asked my colleagues if they had any idea why this is and whether anything had changed recently. None of them had an explanation so I spent some time checking / testing:

- running performance reports in MapGuide/AIMS noticing that time spent on "Layer" is 20times slower in production
- comparing view definitions but only cursorily (looking at WHERE clause only) : mistake #1
- re-creating spatial index for base table : no change
- running 1-click maintenance in Map-Administrator : no change
- copy a layer definition from production to test environment : no change
- trying to load view into other applications to see whether the issue is related to application or to view but GeoRaptor couldn't add layer to map due to rowid issue, QGIS rejected view complaining that view didn't have a proper spatial index and TatukGIS Viewer basically saying something similar (first the message about the spatial index issue made me re-re-create the spatial index but testing further views in QGIS/TatukGISViewer I got the same messages for all other views as well).
- running a few spatial SQL queries in SQLDeveloper but there wasn't any significant difference between test and production : mistake # 2 (this result caused quite some confusion as I did not have an explanation, it did not fit and even contradicted other findings!)

After spending some time on running these tests a colleague came back to me and told me, that the view had changed recently but when he applied the change he didn't notice any performance issue whilst generating graphics in Map. The change was made by adding a subquery to the SELECT clause. He asked if the change should be removed but I replied that I will do some tests now taking the new information into account : mistake # 3.

As the change to the view definition had only been applied in the production system I applied the change in the test system as well - here mistake # 4 occurred. I then ran the MapGuide/AIMS performance report again but the performance had not decreased. Performance was good as before. Then I changed the data source of the Oracle user I was testing with and pointed it to our production database. When I ran the test again performance was slow. I stopped any further testing for the day and did something else.

At this point my theory was that the same query has different execution plans in Oracle (production and test) therefore different execution times due to different Oracle settings. What didn't really fit was that when I ran the spatial queries in SQLDeveloper I didn't see any significant difference in performance. But I didn't have anything else to test so finally I decided to replace the view definition in production with the previous view definition - basically removing the change which had been applied recently. I re-ran the performance report in MapGuide/AIMS and suddenly I got an error message:

Failed to stylize layer: DP_xxx
An exception occurred in FDO component.
ORA-00904: "G"."OWNER_OBJTYP": invalid identifier
...

Now I realised one of my mistakes. MapGuide/AIMS caches certain settings/definitions. I don't know how it works in detail but probably roughly like that:
- when AIMS/MapGuide starts it reads the Oracle schema and caches some information
- (SELECT) statements are being build based on the cached information and used against Oracle

For a view MapGuide/AIMS seems to read the view definition and then creates a SELECT statement which includes all columns of the view. 
If the view definition in Oracle changes and new columns are added to the view MapGuide can still run the SELECT statement against the new view definition - basically just ignoring all newly added columns.  But when columns have been removed from the view the SELECT statement still includes those columns and Oracle will response with ora-00904.

Mistake 4 - when I applied the change to view definition in the test system I did not restart MapGuide/AIMS. MapGuide was still using the SELECT statement which did not include the newly added column in the view. But exactly that new column had a negative impact on performance.
If I had restarted MapGuide/AIMS at this point in time I could have avoided further testing and head-scratching.

Mistake 3 - if I had listened to my colleague who offered to remove the change in production I might have noticed the cause of the problem earlier (because even without restarting MapGuide/AIMS we would have seen the Ora-00904 message in the log file).

Mistake 2 - my spatial query was simplified, the select clause included only "fid, geom". As it did not include the column which caused the performance issue there is really no surprise that performance in both production and test were nearly the same. Conclusion: always use (if possible) exactly the same (SELECT) statement as the application when you do testing!

Mistake 1 - I should have noticed the difference in view definitions myself just by copying the view definitions from production and test into an editor and then noticing the different numbers of lines....

So at the end a trivial issue but it took longer then necessary to get it fixed.

Why you should restart AIMS/MapGuide when testing ....

I noticed by chance that a few layers - which are all based on one Oracle view - were significantly slower in our production system but no performance issue occurred in our test environment. Whilst investigating the issue I made a few mistakes which made the whole process of finding and fixing the issue more time consuming than necessary. Here are the details:

After noticing the issue I asked my colleagues if they had any idea why this is and whether anything had changed recently. None of them had an explanation so I spent some time checking / testing:

- running performance reports in MapGuide/AIMS noticing that time spent on "Layer" is 20times slower in production
- comparing view definitions but only cursorily (looking at WHERE clause only) : mistake #1
- re-creating spatial index for base table : no change
- running 1-click maintenance in Map-Administrator : no change
- copy a layer definition from production to test environment : no change
- trying to load view into other applications to see whether the issue is related to application or to view but GeoRaptor couldn't add layer to map due to rowid issue, QGIS rejected view complaining that view didn't have a proper spatial index and TatukGIS Viewer basically saying something similar (first the message about the spatial index issue made me re-re-create the spatial index but testing further views in QGIS/TatukGISViewer I got the same messages for all other views as well).
- running a few spatial SQL queries in SQLDeveloper but there wasn't any significant difference between test and production : mistake # 2

After spending some time on running these tests a colleague came back to me and told me, that the view had changed recently but when he applied the change he didn't notice any performance issue whilst generating graphics in Map. The change was made by adding a subquery to the SELECT clause. He asked if the change should be removed but I replied that I will do some tests now taking the new information into account : mistake # 3.

As the change to the view definition had only been applied in the production system I applied the change in the test system as well - here mistake # 4 occurred. I then ran the MapGuide/AIMS performance report again but the performance had not decreased. Performance was good as before. Then I changed the data source of the Oracle user I was testing with and pointed it to our production database. When I ran the test again performance was slow. I stopped any further testing for the day and did something else.

At this point my theory was that the same query has different execution plans in Oracle (production and test) therefore different execution times due to different Oracle settings. What didn't really fit was that when I ran the spatial queries in SQLDeveloper I didn't see any significant difference in performance. But I didn't have anything else to test so finally I decided to replace the view definition in production with the previous view definition - basically removing the change which had been applied recently. I re-ran the performance report in MapGuide/AIMS and suddenly I got an error message:

Failed to stylize layer: DP_xxx
An exception occurred in FDO component.
ORA-00904: "G"."OWNER_OBJTYP": invalid identifier
...

Now I realised one of my mistakes. MapGuide/AIMS caches certain settings/definitions. I don't know how it works in detail but probably roughly like that:
- when AIMS/MapGuide starts it reads the Oracle schema and caches some information
- (SELECT) statements are being build based on the cached information and used against Oracle

For a view MapGuide/AIMS seems to read the view definition and then creates a SELECT statement which includes all columns of the view. 
If the view definition in Oracle changes and new columns are added to the view MapGuide can still run the SELECT statement against the new view definition - basically just ignoring all newly added columns.  But when columns have been removed from the view the SELECT statement still includes those columns and Oracle will response with ora-00904.

Mistake 4 - when I applied the change to view definition in the test system I did not restart MapGuide/AIMS. MapGuide was still using the SELECT statement which did not include the newly added column in the view. But exactly that new column had a negative impact on performance.
If I had restarted MapGuide/AIMS at this point in time I could have avoided further testing and head-scratching.

Mistake 3 - if I had listened to my colleague who offered to remove the change in production I might have noticed the cause of the problem earlier (because even without restarting MapGuide/AIMS we would have seen the Ora-00904 message in the log file).

Mistake 2 - my spatial query was simplified, the select clause included only "fid, geom". As it did not include the column which caused the performance issue there is really no surprise that performance in both production and test were nearly the same. Conclusion: always use (if possible) exactly the same (SELECT) statement as the application when you do testing!

Mistake 1 - I should have noticed the difference in view definitions myself just by copying the view definitions from production and test into an editor and then noticing the different numbers of lines....

So at the end a trivial issue but it took longer then necessary to get it fixed.

AutoCAD Map 2017, job enabled IMs and Oracle 12.1.0.2.0

Map 2017 ReadMe states:

• Oracle 12cR1 (12.1.0.2.0) is not recommended for Industry Models with jobs. You should use other versions of Oracle like 11.2.0.4.0 or 12.1.0.1.0 to work with jobs.

This issue with Jobs and 12.1.0.2.0 has been fixed now by Autodesk.  Autodesk  has published a TS regarding the solution.

Publish to MapGuide/AIMS - Spatial Filter II

As described in a previous posting  when you publish data from Map to MapGuide/AIMS the published layer files get a spatial filter assigned. The filter will decrease performance and needs to be removed. So usually we remove the spatial filters after publishing from Map.

But today we had user reporting to us that some layers don't appear on the map although they used to appear in the past.

As it turned out spatial filters had been assigned to those layers but the filters covered our area of interest and therefore no one noticed it. But a few weeks ago we switched to a new coordinate system and had to transform our spatial data into the new system. The spatial filters defined in some layers had not been transformed as we had not been aware of them and assuming all had been removed. After removing the spatial filters the layers re-appeared on our maps.

The issue with regards to layers getting a spatial filter assigned when being published has been reported to Autodesk some time back. Maybe it has been fixed by now in newer releases.

Here is the core bit of the PHP based script for removing spatial filters - as usual a quick and dirty solution:

 if ($_POST['hiddenparam'] == '') die();       
      if ($_POST['projektpfad'] == '') die();       
      $MGAdminPassword = $_POST['password'];  
      try  
      {                      
           include './config.schema.list.php';  
           include '../vaw.lib/initializeMapGuide.php';  
           $Verzeichnis = $_POST['projektpfad'];             
           echo "<html><body>";  
           echo "<p>MapGuide Pfad: $Verzeichnis </p>";  
           $VerzeichnisResourceId = new MgResourceIdentifier($Verzeichnis);  
           $LayerReader = $resourceService->EnumerateResources($VerzeichnisResourceId, -1, "");  
           $Xml = $LayerReader->ToString();  
           $LDomDoc = DOMDocument::loadXML($Xml);  
           $pfad = new DOMXPath($LDomDoc);  
           $frage = "//ResourceDocument/ResourceId";  
           $ebenen = $pfad->query($frage);  
           $anzlayer = 0;  
      foreach($ebenen as $knoten)  
      {  
           $Layer = $knoten->nodeValue;  
           $LayerResourceId = new MgResourceIdentifier($Layer);  
           $layerName = $LayerResourceId->GetName();  
           $layerType = $LayerResourceId->GetResourceType();  
           if($layerType == 'LayerDefinition')  
           {  
                echo "<b>Layername: $layerName </b><br>";  
                $tmpReader = $resourceService->GetResourceContent($LayerResourceId);  
                $layerXML = $tmpReader->ToString();                 
                $doc = DOMDocument::loadXML($layerXML);                            
                $FilterNode = $doc->getElementsByTagName('Filter');  
                foreach($FilterNode as $node)   
                {  
                     if(!stristr($node->nodeValue, "GeomFromText"))  
                     {                      
                          //kein Spatial Filter  
                          //echo " <b>Kein Spatial Filter.</b><br>";  
                     }  
                     else  
                     {  
                          $node->nodeValue = '';  
                          echo "<b>Spatial Filter - vorhanden und entfernt.</b> <br><br>";  
                          echo "<small>vorhandene Filter: {$node->nodeValue} .</small>";  
                     }  
                }                           
                //echo $doc->ToString();  
                $modifiedLayerDefinition = $doc->saveXML();  
                $byteSource = new MgByteSource($modifiedLayerDefinition, strlen($modifiedLayerDefinition));  
                $byteSource->SetMimeType(MgMimeType::Xml);  
                //$resourceId = new MgResourceIdentifier($LayerResourceId);  
                $resourceService->SetResource($LayerResourceId, $byteSource->GetReader(), null);  
           }  
      }  
 }  
 catch ( MgException $e )  
   {  
     $errorMsg = $errAuthenticationFailed;  
   }  
   catch ( Exception $e )  
   {  
     $errorMsg = $e->getMessage();  
   }  

Map 2013, SP2

Map API - MgFeatureReader - improve speed

The developer example for reading features with MgFeatureReader looks like that:

 while (featureReader.ReadNext())  
    {  
      featureCount++;  
      propertyCount = featureReader.GetPropertyCount();  
      for (int i = 0; i < propertyCount; i++)  
      {  
        propertyName = featureReader.GetPropertyName(i);  
        propertyType = featureReader.GetPropertyType(propertyName);  
        PrintPropertyValueFromReader(featureReader, propertyType, propertyName, ref sb);  
      }  
    }  

(form MapGuide Api Reference).

At least with Map this will be slow. Performance can be improved significantly if  the GetPropertyName and GetPropertyType calls are removed from within the FOR loop.

Example: a small application in order to displays feature attributes in a DataGrid. 

DataGrid

The sample SDF file contains 110'000 features and has 11 columns (text and numbers).

Using code similar to the one given above it takes 19 sec to populate and display the datagrid.

By removing the two calls the process takes less then 1 second and is even slightly faster then opening  Map's data table (mapdatatable):


Befehl: MAPDATATABLE2
 Finished in 0 seconds.
Befehl: MAPDATATABLE3

 Finished in 19 seconds.

All it takes is to get property name and property type for each column only once and store this information in an array (colProps) before iterating over the FeatureReader:

 ...  
  while (featureReader.ReadNext())  
       {  
         counter++;  
         newRow = dt.NewRow();          
         for (int i = 0; i < colProps.Length; i++)  
         {  
           propertyName = colProps[i].PropertyName;  
           propertyType = colProps[i].PropertyType;  
                          ....  

Map 2013, SP2

Revisited: Jobs in AutoCAD Map - job-disable a feature class

Just to let you know - the "solution" for having job-disabled feature classes in a job enabled document as described here works well for us. We have been using the solution for nearly a year in production and haven't had any issues.

Map 2013, SP2

Map IM - generating graphics and DXF files

Recently we encountered two issues regarding DXF files and generating graphics. Both only happen if you enable "Reuse Active Drawing" in "Generate Graphics" application options.

1) Display Model is loaded into wrong drawing

If you have opened one or more drawings (DWG) and one DXF file and you generate graphics over the DXF file: the display model layers wont load into the DXF file (as expected) but in one of the other currently opened drawings.
Solution: close all other drawings or save DXF as Drawing

2) error message when generating graphics

An error message is displayed and the Display Model doesn't load when generating graphics over a DXF file. Error message shown:

Drawing of Display Model in Map failed. Adding layer failed: layername. 
Failed to retrieve message for "MgLayerNotFoundException".
The resource was not found.

This happens only if the DXF has been saved with a coordinate system. 
Solution: save DXF as Drawing file.

Both issues have been confirmed by Autodesk for Map 2017.