Showing posts with label Performance. Show all posts
Showing posts with label Performance. Show all posts

Friday, 16 February 2018

ArcMap slow?

ArcMap was extremly slow on my PC. Opening attribute tables, the Add Data dialog box, a model oder even a toolbox took ages (such as opening a toolbox > 30secs).
Now, I'm used to applying "Windows Classic" design in order to speed up things.
Turns out - since switching back to standard Windows 7 design ArcMap performance has improved significantly on my machine! Opening a larger model now only takes 1-2 secs (if at all)!

ArcMap 10.3, Windows 7

Monday, 12 September 2016

Oracle ANSI JOIN syntax improves peformance

Using ANSI JOIN syntax instead of Oracle Join snytax is recommended - see here:

http://www.orafaq.com/node/2618

We found out the hard way:

- a third party application issues simple but nested SQL statements against Map IM views
- for some views we noticed that the query ran for a long time and generated a time out 
- just by changing the JOIN syntax to ANSI join the issue was resolved

Map 2017

Saturday, 30 July 2016

Why you should restart AIMS/MapGuide when testing ....

I noticed by chance that a few layers - which are all based on one Oracle view - were significantly slower in our production system but no performance issue occurred in our test environment. Whilst investigating the issue I made a few mistakes which made the whole process of finding and fixing the issue more time consuming than necessary. Here are the details:

After noticing the issue I asked my colleagues if they had any idea why this is and whether anything had changed recently. None of them had an explanation so I spent some time checking / testing:

- running performance reports in MapGuide/AIMS noticing that time spent on "Layer" is 20times slower in production
- comparing view definitions but only cursorily (looking at WHERE clause only) : mistake #1
- re-creating spatial index for base table : no change
- running 1-click maintenance in Map-Administrator : no change
- copy a layer definition from production to test environment : no change
- trying to load view into other applications to see whether the issue is related to application or to view but GeoRaptor couldn't add layer to map due to rowid issue, QGIS rejected view complaining that view didn't have a proper spatial index and TatukGIS Viewer basically saying something similar (first the message about the spatial index issue made me re-re-create the spatial index but testing further views in QGIS/TatukGISViewer I got the same messages for all other views as well).
- running a few spatial SQL queries in SQLDeveloper but there wasn't any significant difference between test and production : mistake # 2 (this result caused quite some confusion as I did not have an explanation, it did not fit and even contradicted other findings!)

After spending some time on running these tests a colleague came back to me and told me, that the view had changed recently but when he applied the change he didn't notice any performance issue whilst generating graphics in Map. The change was made by adding a subquery to the SELECT clause. He asked if the change should be removed but I replied that I will do some tests now taking the new information into account : mistake # 3.

As the change to the view definition had only been applied in the production system I applied the change in the test system as well - here mistake # 4 occurred. I then ran the MapGuide/AIMS performance report again but the performance had not decreased. Performance was good as before. Then I changed the data source of the Oracle user I was testing with and pointed it to our production database. When I ran the test again performance was slow. I stopped any further testing for the day and did something else.

At this point my theory was that the same query has different execution plans in Oracle (production and test) therefore different execution times due to different Oracle settings. What didn't really fit was that when I ran the spatial queries in SQLDeveloper I didn't see any significant difference in performance. But I didn't have anything else to test so finally I decided to replace the view definition in production with the previous view definition - basically removing the change which had been applied recently. I re-ran the performance report in MapGuide/AIMS and suddenly I got an error message:

Failed to stylize layer: DP_xxx
An exception occurred in FDO component.
ORA-00904: "G"."OWNER_OBJTYP": invalid identifier
...

Now I realised one of my mistakes. MapGuide/AIMS caches certain settings/definitions. I don't know how it works in detail but probably roughly like that:
- when AIMS/MapGuide starts it reads the Oracle schema and caches some information
- (SELECT) statements are being build based on the cached information and used against Oracle

For a view MapGuide/AIMS seems to read the view definition and then creates a SELECT statement which includes all columns of the view. 
If the view definition in Oracle changes and new columns are added to the view MapGuide can still run the SELECT statement against the new view definition - basically just ignoring all newly added columns.  But when columns have been removed from the view the SELECT statement still includes those columns and Oracle will response with ora-00904.

Mistake 4 - when I applied the change to view definition in the test system I did not restart MapGuide/AIMS. MapGuide was still using the SELECT statement which did not include the newly added column in the view. But exactly that new column had a negative impact on performance.
If I had restarted MapGuide/AIMS at this point in time I could have avoided further testing and head-scratching.

Mistake 3 - if I had listened to my colleague who offered to remove the change in production I might have noticed the cause of the problem earlier (because even without restarting MapGuide/AIMS we would have seen the Ora-00904 message in the log file).

Mistake 2 - my spatial query was simplified, the select clause included only "fid, geom". As it did not include the column which caused the performance issue there is really no surprise that performance in both production and test were nearly the same. Conclusion: always use (if possible) exactly the same (SELECT) statement as the application when you do testing!

Mistake 1 - I should have noticed the difference in view definitions myself just by copying the view definitions from production and test into an editor and then noticing the different numbers of lines....

So at the end a trivial issue but it took longer then necessary to get it fixed.

Why you should restart AIMS/MapGuide when testing ....

I noticed by chance that a few layers - which are all based on one Oracle view - were significantly slower in our production system but no performance issue occurred in our test environment. Whilst investigating the issue I made a few mistakes which made the whole process of finding and fixing the issue more time consuming than necessary. Here are the details:

After noticing the issue I asked my colleagues if they had any idea why this is and whether anything had changed recently. None of them had an explanation so I spent some time checking / testing:

- running performance reports in MapGuide/AIMS noticing that time spent on "Layer" is 20times slower in production
- comparing view definitions but only cursorily (looking at WHERE clause only) : mistake #1
- re-creating spatial index for base table : no change
- running 1-click maintenance in Map-Administrator : no change
- copy a layer definition from production to test environment : no change
- trying to load view into other applications to see whether the issue is related to application or to view but GeoRaptor couldn't add layer to map due to rowid issue, QGIS rejected view complaining that view didn't have a proper spatial index and TatukGIS Viewer basically saying something similar (first the message about the spatial index issue made me re-re-create the spatial index but testing further views in QGIS/TatukGISViewer I got the same messages for all other views as well).
- running a few spatial SQL queries in SQLDeveloper but there wasn't any significant difference between test and production : mistake # 2

After spending some time on running these tests a colleague came back to me and told me, that the view had changed recently but when he applied the change he didn't notice any performance issue whilst generating graphics in Map. The change was made by adding a subquery to the SELECT clause. He asked if the change should be removed but I replied that I will do some tests now taking the new information into account : mistake # 3.

As the change to the view definition had only been applied in the production system I applied the change in the test system as well - here mistake # 4 occurred. I then ran the MapGuide/AIMS performance report again but the performance had not decreased. Performance was good as before. Then I changed the data source of the Oracle user I was testing with and pointed it to our production database. When I ran the test again performance was slow. I stopped any further testing for the day and did something else.

At this point my theory was that the same query has different execution plans in Oracle (production and test) therefore different execution times due to different Oracle settings. What didn't really fit was that when I ran the spatial queries in SQLDeveloper I didn't see any significant difference in performance. But I didn't have anything else to test so finally I decided to replace the view definition in production with the previous view definition - basically removing the change which had been applied recently. I re-ran the performance report in MapGuide/AIMS and suddenly I got an error message:

Failed to stylize layer: DP_xxx
An exception occurred in FDO component.
ORA-00904: "G"."OWNER_OBJTYP": invalid identifier
...

Now I realised one of my mistakes. MapGuide/AIMS caches certain settings/definitions. I don't know how it works in detail but probably roughly like that:
- when AIMS/MapGuide starts it reads the Oracle schema and caches some information
- (SELECT) statements are being build based on the cached information and used against Oracle

For a view MapGuide/AIMS seems to read the view definition and then creates a SELECT statement which includes all columns of the view. 
If the view definition in Oracle changes and new columns are added to the view MapGuide can still run the SELECT statement against the new view definition - basically just ignoring all newly added columns.  But when columns have been removed from the view the SELECT statement still includes those columns and Oracle will response with ora-00904.

Mistake 4 - when I applied the change to view definition in the test system I did not restart MapGuide/AIMS. MapGuide was still using the SELECT statement which did not include the newly added column in the view. But exactly that new column had a negative impact on performance.
If I had restarted MapGuide/AIMS at this point in time I could have avoided further testing and head-scratching.

Mistake 3 - if I had listened to my colleague who offered to remove the change in production I might have noticed the cause of the problem earlier (because even without restarting MapGuide/AIMS we would have seen the Ora-00904 message in the log file).

Mistake 2 - my spatial query was simplified, the select clause included only "fid, geom". As it did not include the column which caused the performance issue there is really no surprise that performance in both production and test were nearly the same. Conclusion: always use (if possible) exactly the same (SELECT) statement as the application when you do testing!

Mistake 1 - I should have noticed the difference in view definitions myself just by copying the view definitions from production and test into an editor and then noticing the different numbers of lines....

So at the end a trivial issue but it took longer then necessary to get it fixed.

Thursday, 7 July 2016

Map API - MgFeatureReader - improve speed

The developer example for reading features with MgFeatureReader looks like that:


 while (featureReader.ReadNext())  
    {  
      featureCount++;  
      propertyCount = featureReader.GetPropertyCount();  
      for (int i = 0; i < propertyCount; i++)  
      {  
        propertyName = featureReader.GetPropertyName(i);  
        propertyType = featureReader.GetPropertyType(propertyName);  
        PrintPropertyValueFromReader(featureReader, propertyType, propertyName, ref sb);  
      }  
    }  

(form MapGuide Api Reference).

At least with Map this will be slow. Performance can be improved significantly if  the GetPropertyName and GetPropertyType calls are removed from within the FOR loop.

Example: a small application in order to displays feature attributes in a DataGrid. 

DataGrid

The sample SDF file contains 110'000 features and has 11 columns (text and numbers).

Using code similar to the one given above it takes 19 sec to populate and display the datagrid.

By removing the two calls the process takes less then 1 second and is even slightly faster then opening  Map's data table (mapdatatable):


Befehl: MAPDATATABLE2
 Finished in 0 seconds.
Befehl: MAPDATATABLE3

 Finished in 19 seconds.


All it takes is to get property name and property type for each column only once and store this information in an array (colProps) before iterating over the FeatureReader:

 ...  
  while (featureReader.ReadNext())  
       {  
         counter++;  
         newRow = dt.NewRow();          
         for (int i = 0; i < colProps.Length; i++)  
         {  
           propertyName = colProps[i].PropertyName;  
           propertyType = colProps[i].PropertyType;  
                          ....  


Map 2013, SP2


Tuesday, 19 April 2016

MsMpEng.exe Slowing down MapGuide

The blog title is not entirely correct - MapGuide/AIMS is not directly affected, but - here are the details:

Suddenly (Monday morning) users started to complain that our AIMS based WebGIS is particularly slow. As it turned out it was not AIMS server itself but the application around it which was slow. Opening a form  or performing a search suddenly took seconds,  but map creation was still fast as usual. We do not use AIMS's Industry Model but TBVIEW1  as an better performing alternative. 

We have three servers in use - the issue occurred only on one of them. TaskManager showed some unusual activity on the machine affected - it appeared as process MsMpEng.Exe was running nearly constantly consuming 25% of CPU power. Further investigation showed, that MsMpEng got active as soon as the WebGIS application interacted with Oracle, for instance when opening a form and remained active for some time. No other processes were using CPU power at the same time. For testing purpose our IT disabled MsMpEng.exe temporarily and the performance returned back to normal. They are now investigating why this happens and why it happens only on one of three machines (W2008R2). 

AIMS 2013, SP2

Friday, 29 January 2016

VBA enabler slows down AutoCAD

Here is one example for the known issue that VBA enabler slows down AutoCAD.

If you insert a block using _insert it will be quick (in our case 3-4 secs) for a larger drawing inserted. When inserted via command line without dialog box (for example: _-insert "C:/temp/ow.dwg") ist takes more then 30sec.

It happens with Map 2013 SP2, 64bit. Apparently this performance issue has been fixed and doesn't occur anymore in newer releases. Shame that it took me quite a long time to figure out, that VBA was the culprit.

Map 2013, SP2

Tuesday, 30 June 2015

MapGuide / AIMS - performance, file access

We run AIMS on virtualized W2008R2 machines with 6 GB RAM. Most data displayed as maps comes directly from Oracle Spatial. Only for one job enabled Map industry model  we decided to export 40+ feature classes and views to an SDF file. File size is roughly 200MB.

In order to speed things up I created a RAM Disk on the server and did some performance tests:
- hard disk access (read/write) was already much better then on my physical machine
- depending on RAM disk driver and test performed read access was faster by up to factor 6 compared to hard disk access

Surprisingly moving the SDF file to the RAM disk did not help improving performance in AIMS/MapGuide. 

Conclusion seems to be that W2008R2 has already cached the SDF file and no further improvement is possible here. As the SDF file is used in all web mapping projects and layers based on file are visible by default I assume that the file will be cached all the time. 


Here are a few screenshots related to the testing:

- hard disk performance, local machine:



- hard disk performance, virtualized 2008r2 server




- ram disk performance, local machine (imdisk)




- ram disk performance, w 2008r2 (softperfect)





- AIMS/MapGuide performance test with sdf based map - sdf file on disk on virtualized w2008r2 server




- AIMS/MapGuide performance test with sdf based map - sdf file on ram disk on virtualized w2008r2 server



AIMS 2013

Wednesday, 19 November 2014

MapGuide / AIMS performance

Most of the data we publish to our intranet web gis (based on AIMS 2013) comes from Oracle via FDO Topobase provider. For performance reasons we have created an SDF file for the most often used base data - containing 30+ tables and 700'000 features. To my surprise the SDF file is not as fast as I expected. By changing the default value "DataConnectionPoolExcludedProviders" performance increased noticable.

By default SDF files are excldued from ConnectionPooling:
 
DataConnectionPoolExcludedProviders = OSGeo.SDF,OSGeo.SHP

By uncommenting the line and restarting AIMS things got better:

#DataConnectionPoolExcludedProviders = OSGeo.SDF,OSGeo.SHP

This has been mentioned before - but somehow I didn't know it. Here is an older reference:

http://zacster.blogspot.ch/2008/10/mapguide-performance-tips.html


In the next few days I will look into SQLite files as an alternative to SDF. A first test seems to be promising - SQLite file was much faster than SDF.

Map 2013, SP2

Thursday, 13 March 2014

Creating Security Role takes a very long time

We ran into the following issue: it was not possible to create a security role as the process took a long time and eventually we had to kill SecurityAdminitrator before it finished. Whilst the process is running SecurityAdministrator doesn't respond anymore and it looks like the application has crashed. But SecurityAdmin is just waiting that the following SQL statement finishes in Oracle:

insert into mdsys.sdo_geom_metadata_table(sdo_OWNER, sdo_table_name, sdo_column_name, sdo_DIMINFO, sdo_SRID) select 'WT_EW_05_INFO', s.synonym_name, m.column_name, m.DIMINFO, m.SRID from all_sdo_geom_metadata m, all_synonyms s where m.owner=s.table_owner and m.TABLE_NAME = s.TABLE_NAME and s.owner='WT_EW_05_INFO'

We had that issue with Oracle 11.2.0.3 on AIX.

Autodesk Support suggested using SQL Tuning Advisor to find a better execution plan. So we did and it resolved the issue. The security role is now created within seconds. Drawback is - you need an extra license for Oracle's Tuning Package (luckily we have one).  You need to use Enterprise Manager to run the advisor.


AutoCAD Map 2013, SP2