Uploading MapGuide package - Exception
If you modify a MapGuide package make sure that you zip the correct files and folders afterwards.
I unzipped a MapGuide package, changed some values and zipped the folder but received the following error message when trying to upload the file:
Maestro error message:
value cannot be nul
Parametername: entry
System.ArgumentNullException: value cannot be nul
Parametername: entry
bei ICSharpCode.SharpZipLib.Zip.ZipFile.GetInputStream(ZipEntry entry)
bei Maestro.Packaging.PackageBuilder.UploadPackageNonTransactional(String sourceFile, UploadPackageResult result) in C:\working\JenkinsCI\home\slave_win\jobs\Maestro trunk\workspace\Maestro.Packaging\PackageBuilder.cs:Zeile 217.
MapGuide error message:
ERROR_MESSAGE: An exception occurred in DWF component. File not found in archive
STACK_TRACE: - MgLibraryRepositoryManager.LoadResourcePackage() line 183 file c:\working\build_area\mapguide\2.5.2\x64\mgdev\server\src\services\resource\LibraryRepositoryManager.cpp - MgResourcePackageLoader.Start() line 150 file c:\working\build_area\mapguide\2.5.2\x64\mgdev\server\src\services\resource\ResourcePackageLoader.cpp - MgResourcePackageLoader.CreateByteReader() line 108 file c:\working\build_area\mapguide\2.5.2\x64\mgdev\server\src\services\resource\ResourcePackageLoader.cpp - MgZipFileReader.ExtractArchive() line 61 file c:\working\build_area\mapguide\2.5.2\x64\mgdev\server\src\services\resource\ZipFileReader.cpp
My mistake was to zip the top level folder - which I got when I unzipped the package. But you need to zip the content of the top level folder, not the top level folder itself.
AIMS 2017
About things that happened to me at work.
Showing posts with label Autodesk Infrastructure MapServer. Show all posts
Showing posts with label Autodesk Infrastructure MapServer. Show all posts
Friday, 4 August 2017
Thursday, 2 March 2017
MapGuide log file
It is good practice to monitor log files. Unfortunately MapGuide log files are not easy to read:
- StackTrace details take up more space then the message itself, it is difficult to see when an error message starts and ends
- certain messages might appear very often although there are not really indicating an issue (such as "Session expired" message).
Here is Python3 script which reads MapGuide logs files and gets rid of StackTrace details and also filters out uninteresting messages. It adds a line number so that you can find the message in the original log file quickly. It also creates a simple summary for messages and their frequency. As we have two MapGuide servers the script is configured to read two different log file folders.
If you want to use it you need to configure:
- the path for the MapGuide log files
- the path for saving the shortened log file
- error messages you want to exclude
- the number of most recent log files you want to process (such as the 3 most recent ones)
I haven't done much in Python yet - the script doesn't do much error handling and has other shortcomings as well.
- StackTrace details take up more space then the message itself, it is difficult to see when an error message starts and ends
- certain messages might appear very often although there are not really indicating an issue (such as "Session expired" message).
Here is Python3 script which reads MapGuide logs files and gets rid of StackTrace details and also filters out uninteresting messages. It adds a line number so that you can find the message in the original log file quickly. It also creates a simple summary for messages and their frequency. As we have two MapGuide servers the script is configured to read two different log file folders.
If you want to use it you need to configure:
- the path for the MapGuide log files
- the path for saving the shortened log file
- error messages you want to exclude
- the number of most recent log files you want to process (such as the 3 most recent ones)
I haven't done much in Python yet - the script doesn't do much error handling and has other shortcomings as well.
Thats how the result might look like:
****************
**************** Logfile: //wsstadt529/logfiles/MapGuide\Error.log
****************
<start>
3 DATE_TIME 2017-03-02 / 01:50:19
4 Success: Server stopped.
<end>
<start>
DATE_TIME 2017-03-02 / 01:51:23
6 Success: Server started.
<end>
<start>
DATE_TIME 2017-03-02 / 07:27:12
444 Error: An exception occurred in FDO component.
445 Error occurred in Feature Source (Library://FS_BEAR/WT_PO_BW_B/Data/TopobaseDefault.FeatureSource): Zeichenfolge ist kein gültiger Filter. (Cause: Zeichenfolge ist nicht korrekt formatiert. , Root Cause: Zeichenfolge ist nicht korrekt formatiert. )
446
<end>
...
<start>
DATE_TIME 2017-03-02 / 13:35:34
953 Error: An exception occurred in FDO component.
954 Error occurred in Feature Source (Library://FS_BEAR/WT_PO_BW_B/Data/TopobaseDefault.FeatureSource): Zeichenfolge ist kein gültiger Filter. (Cause: Zeichenfolge ist nicht korrekt formatiert. , Root Cause: Zeichenfolge ist nicht korrekt formatiert. )
955
<end>
*********************************************
************* Summary ***********************
*********************************************
# 3 :: Error: Failed to stylize layer: ZIM_Anlageobjekte_DynaS_GeoRest_MBR
# 36 :: Error: Failed to stylize layer: ZH_Orthofoto2015
# 1 :: Success: Server started.
# 1 :: Success: Server stopped.
# 27 :: Error: An exception occurred in FDO component.
************* File(s) ***********************
File processed: //wsstadt529/logfiles/MapGuide\Error.log
# of messages:68
# of messages excluded: 51
Here is the script:
import os
import sys
from collections import Counter
""" extract date and time from logfile line """
def extractdatetime(a_text_line):
#DateTime Format in MapGuide-LogFile:
#<2016-12-15T11:00:34> <2015-07-05T12:39:55>
date = a_text_line[1:11]
time = a_text_line[12:20]
return (date, time)
""" checks whether a certain error message should be ignored / excluded from further processing """
def isErrorMessagesToExclude(a_message):
for text_to_find in errormessage_to_exclude:
if text_to_find in a_message:
return True
return False
""" removes the StackTrace details from an error message """
def removeStackTrace(a_message):
pos = a_message.find('StackTrace:')
if pos > -1:
return a_message[0:pos]
else:
return a_message
""" saves file """
def saveFile(text, filename, mode):
f = open(filename, mode)
f.write(text)
f.close()
""" returns a sorted list of files for a given directory , sorted by date
see: http://stackoverflow.com/questions/4500564/directory-listing-based-on-time
"""
def sorted_ls(path):
mtime = lambda f: os.stat(os.path.join(path, f)).st_mtime
result = list(sorted(os.listdir(path), key=mtime))
return result
""" keeps only MapGuide Logfiles and filters out all other files """
def filterLogfiles(files_in_dir):
filtered_files = []
for filename in files_in_dir:
if filename.endswith(".log") and filename.startswith("Error"):
filtered_files.append(filename)
return filtered_files
""" processes a single MG log file and simplifies content """
def processLogFile(logfile):
# counter for line number in log file
line_number = 0
# counter for number of messages processed
counter_messages = 0
# counter for messages we exclude from processing
counter_messages_ingnored = 0
# internal counter
count_opening_tag = 0
message = ''
message_part = ''
# encoding utf-8, ascii : returns error message while reading a certain character/bytes
with open(logfile, encoding='“latin-1') as a_file:
for a_line in a_file:
line_number += 1
# all messages start with '<' and first line also contains date&time
if '<' in a_line:
count_opening_tag +=1
#get date and time
str_date, str_time = extractdatetime(a_line)
a_line = 'DATE_TIME ' + str_date + ' / ' + str_time + '\n'
# we processing the first line of the current message
if count_opening_tag == 1:
# line number and date/time information in one new line
message_part += str(line_number) + ' ' + a_line
# we have reached the first line of the following message - now we need tp process the previous message
if count_opening_tag == 2:
#first we check whether the message can be ignored
if isErrorMessagesToExclude(message_part) is False:
# we remove the StackTrace details
message_part = removeStackTrace(message_part)
counter_messages += 1
# we wrap the message text in <start><end> tags
message += '\n<start>\n'
# we add the processed message to the result
message += message_part
message += '\n<end>'
else:
counter_messages_ingnored += 1
#as this is the first line of the "next" message already a_line contains the DATE_TIME for it
message_part = a_line
# reset counter - the current line is the first line of the next message
count_opening_tag = 1
# last line reached - last message block is not yet fully processed
# code from above is repeated here to close the processing of last message in logfile
if isErrorMessagesToExclude(message_part) is False:
message_part = removeStackTrace(message_part)
message += '\n<start>\n'
message += message_part
message += '\n<end>'
counter_messages += 1
else:
counter_messages_ingnored += 1
temp = ["File processed: "+logfile, "# of messages:"+str(counter_messages), "# of messages excluded:\t "+str(counter_messages_ingnored) ]
summary_files.append(temp)
print("File processed: "+logfile)
print("# of messages:"+str(counter_messages))
print("# of messages excluded:\t "+str(counter_messages_ingnored))
return message
""" converts the newly created log file(s) summary into a list """
def convertToList(processedlogfile):
with open(processedlogfile, encoding='“latin-1') as a_file:
error_message = False
list_final = []
date_temp =''
time_temp = ''
error1_temp = ''
error2_temp = ''
line_counter = 0
for a_line in a_file:
line_counter += 1
if '<start>' in a_line:
line_counter = 1
error_message = True
if '<end>' in a_line:
error_message = False
line_counter = 0
list_temp = [date_temp, time_temp, error1_temp, error2_temp]
list_final.append(list_temp)
if error_message:
if line_counter == 2:
date_temp = a_line[10:20]
time_temp = a_line[23:]
if line_counter == 3:
error1_temp = a_line[5:].strip()
if line_counter == 4:
error2_temp = a_line[5:].strip()
return list_final
""" iterates over all relevant files and creates summary"""
def processLogfiles(logfile_dir, saveLogFileName, number_of_most_recent_files):
# create a new file for the results
saveFile('', saveLogFileName, 'w')
# get all files from directory with MapGuide logs
files_in_directory = sorted_ls(logfile_dir)
# filter out all non MapGuide log files
files_in_directory = filterLogfiles(files_in_directory)
# only process the most recent files
if(number_of_most_recent_files > 0):
index = ((number_of_most_recent_files ) * -1)
files_in_directory = files_in_directory[index:]
# interate over relevant log files
for filename in files_in_directory:
fn = os.path.join(logfile_dir, filename)
# process single log file
log_file_short = processLogFile(fn)
# create header
header = "\n\n****************"
header += "\n**************** Logfile: " + fn
header += "\n****************\n\n"
# write to file
saveFile(header, saveLogFileName, 'a')
saveFile(log_file_short, saveLogFileName, 'a')
# all files have been processed and relevant information has been written into one file
# now we want to get a summary of logged issues
resultList = convertToList(saveLogFileName)
'''
resultList is list of lists, lists have 4 items each
items 3 and 4 are equal to line 1 and 2 of a message
now we just count item 3 and get a Dictionary where frequency of message is key and message itself is value
'''
res = (Counter(mysublist[2] for mysublist in resultList))
text = "\n\n*********************************************"
text += "\n************* Summary ***********************"
text += "\n*********************************************\n\n"
for message, number in res.items():
text += "# " + str(number)+ "\t :: " + message +'\n'
# append summary
saveFile(text, saveLogFileName, 'a')
summ = "\n************* File(s) ***********************\n"
for alist in summary_files:
summ += "\n".join(alist)+"\n"
# append summary
saveFile(summ, saveLogFileName, 'a')
if __name__ == ("__main__"):
""" add any message you want to ignore when processing the log files"""
errormessage_to_exclude = [
'Session has expired',
'Resource was not found: Session:',
'Die Sitzung (',
'Error: Authentication failed'
]
# number of most recent log files to process
# 0 - for all files to be processed
# can be overwritten when python script is called with parameter
number_most_recent_files = 3
#if argument is provided we assume its a number
if len (sys.argv) == 2 :
number_most_recent_files = int(sys.argv[1])
# path to MapGuide error log directory
logfile_dir1 = '//wsstadt529/logfiles/MapGuide'
logfile_dir2 = '//wsstadt516/logfiles/MapGuide'
# file name for result - simplified log file:
saveLogFileName1 = 'c:/temp/aims_log_processed_529.txt'
saveLogFileName2 = 'c:/temp/aims_log_processed_516.txt'
# start processing
summary_files = []
processLogfiles(logfile_dir1, saveLogFileName1, number_most_recent_files)
summary_files = []
processLogfiles(logfile_dir2, saveLogFileName2, number_most_recent_files)
# open file(s) in Editor
os.startfile(saveLogFileName1)
os.startfile(saveLogFileName2)
Tuesday, 14 February 2017
MapGuide Logfile - two warnings and what they mean
MapGuide log file contains the following warnings:
<2017-02-14T08:54:36> 4712 Ajax Viewer 10.99.66.2 Administrator
Warning: An exception occurred in FDO component.
Beim Abrufen eines Eigenschaftenwerts stimmte der Eigenschaftentyp nicht überein.
StackTrace:
- MgStylizationUtil.ExceptionTrap() line 813 file StylizationEngine.cpp
the error message translates roughly as
"property value and property typ do not match"
The message was caused by the following filter definition:
W_ID_TYPE = 1 AND ( W_ID_TYPE_IP NULL OR NOT W_ID_TYPE_IP IN ( '1') )
W_ID_TYPE_IP is numeric but the value is given as string.
The second one:
<2017-02-14T12:19:44> 3444 10.99.66.65 Anonymous
Warning: An exception occurred in FDO component.
Die Kennung 'w_id_type' wurde nicht erkannt.
StackTrace:
- MgStylizationUtil.ExceptionTrap() line 813 file StylizationEngine.cpp
the error message translates roughly as
"property 'w_id_type' not found."
The message was caused by the following filter definition:
w_id_type IN ( 11) AND NOT ID_SITE_TYPE IN (2007)
Property names as case-sensitive. The feature class does not contain 'w_id_type' but 'W_ID_TYPE'.
Unfortunately for both warnings no further information is given to where exactly the issues occur. I stumbled across the cause of the first warning by looking for the cause of the second warning. The second warning tells you exactly what the problem is - unfortunately I only realized that after I had spent several hours looking for it. 'w_id_type' is an attribute we use in many feature classes and 71 layers reference the attribute in filter expressions. It was somehow obvious that there is a problem in one of the layers but not in all of them as the warning did not appear all the time. Took me quite some time to figure out what was going on - despite the warning being clear about it.
AIMS 2017
Saturday, 30 July 2016
Why you should restart AIMS/MapGuide when testing ....
I noticed by chance that a few layers - which are all based on one Oracle view - were significantly slower in our production system but no performance issue occurred in our test environment. Whilst investigating the issue I made a few mistakes which made the whole process of finding and fixing the issue more time consuming than necessary. Here are the details:
After noticing the issue I asked my colleagues if they had any idea why this is and whether anything had changed recently. None of them had an explanation so I spent some time checking / testing:
- running performance reports in MapGuide/AIMS noticing that time spent on "Layer" is 20times slower in production
- comparing view definitions but only cursorily (looking at WHERE clause only) : mistake #1
- re-creating spatial index for base table : no change
- running 1-click maintenance in Map-Administrator : no change
- copy a layer definition from production to test environment : no change
- trying to load view into other applications to see whether the issue is related to application or to view but GeoRaptor couldn't add layer to map due to rowid issue, QGIS rejected view complaining that view didn't have a proper spatial index and TatukGIS Viewer basically saying something similar (first the message about the spatial index issue made me re-re-create the spatial index but testing further views in QGIS/TatukGISViewer I got the same messages for all other views as well).
- running a few spatial SQL queries in SQLDeveloper but there wasn't any significant difference between test and production : mistake # 2 (this result caused quite some confusion as I did not have an explanation, it did not fit and even contradicted other findings!)
After spending some time on running these tests a colleague came back to me and told me, that the view had changed recently but when he applied the change he didn't notice any performance issue whilst generating graphics in Map. The change was made by adding a subquery to the SELECT clause. He asked if the change should be removed but I replied that I will do some tests now taking the new information into account : mistake # 3.
As the change to the view definition had only been applied in the production system I applied the change in the test system as well - here mistake # 4 occurred. I then ran the MapGuide/AIMS performance report again but the performance had not decreased. Performance was good as before. Then I changed the data source of the Oracle user I was testing with and pointed it to our production database. When I ran the test again performance was slow. I stopped any further testing for the day and did something else.
At this point my theory was that the same query has different execution plans in Oracle (production and test) therefore different execution times due to different Oracle settings. What didn't really fit was that when I ran the spatial queries in SQLDeveloper I didn't see any significant difference in performance. But I didn't have anything else to test so finally I decided to replace the view definition in production with the previous view definition - basically removing the change which had been applied recently. I re-ran the performance report in MapGuide/AIMS and suddenly I got an error message:
Failed to stylize layer: DP_xxx
An exception occurred in FDO component.
ORA-00904: "G"."OWNER_OBJTYP": invalid identifier
...
Now I realised one of my mistakes. MapGuide/AIMS caches certain settings/definitions. I don't know how it works in detail but probably roughly like that:
- when AIMS/MapGuide starts it reads the Oracle schema and caches some information
- (SELECT) statements are being build based on the cached information and used against Oracle
For a view MapGuide/AIMS seems to read the view definition and then creates a SELECT statement which includes all columns of the view.
If the view definition in Oracle changes and new columns are added to the view MapGuide can still run the SELECT statement against the new view definition - basically just ignoring all newly added columns. But when columns have been removed from the view the SELECT statement still includes those columns and Oracle will response with ora-00904.
Mistake 4 - when I applied the change to view definition in the test system I did not restart MapGuide/AIMS. MapGuide was still using the SELECT statement which did not include the newly added column in the view. But exactly that new column had a negative impact on performance.
If I had restarted MapGuide/AIMS at this point in time I could have avoided further testing and head-scratching.
Mistake 3 - if I had listened to my colleague who offered to remove the change in production I might have noticed the cause of the problem earlier (because even without restarting MapGuide/AIMS we would have seen the Ora-00904 message in the log file).
Mistake 2 - my spatial query was simplified, the select clause included only "fid, geom". As it did not include the column which caused the performance issue there is really no surprise that performance in both production and test were nearly the same. Conclusion: always use (if possible) exactly the same (SELECT) statement as the application when you do testing!
Mistake 1 - I should have noticed the difference in view definitions myself just by copying the view definitions from production and test into an editor and then noticing the different numbers of lines....
So at the end a trivial issue but it took longer then necessary to get it fixed.
After noticing the issue I asked my colleagues if they had any idea why this is and whether anything had changed recently. None of them had an explanation so I spent some time checking / testing:
- running performance reports in MapGuide/AIMS noticing that time spent on "Layer" is 20times slower in production
- comparing view definitions but only cursorily (looking at WHERE clause only) : mistake #1
- re-creating spatial index for base table : no change
- running 1-click maintenance in Map-Administrator : no change
- copy a layer definition from production to test environment : no change
- trying to load view into other applications to see whether the issue is related to application or to view but GeoRaptor couldn't add layer to map due to rowid issue, QGIS rejected view complaining that view didn't have a proper spatial index and TatukGIS Viewer basically saying something similar (first the message about the spatial index issue made me re-re-create the spatial index but testing further views in QGIS/TatukGISViewer I got the same messages for all other views as well).
- running a few spatial SQL queries in SQLDeveloper but there wasn't any significant difference between test and production : mistake # 2 (this result caused quite some confusion as I did not have an explanation, it did not fit and even contradicted other findings!)
After spending some time on running these tests a colleague came back to me and told me, that the view had changed recently but when he applied the change he didn't notice any performance issue whilst generating graphics in Map. The change was made by adding a subquery to the SELECT clause. He asked if the change should be removed but I replied that I will do some tests now taking the new information into account : mistake # 3.
As the change to the view definition had only been applied in the production system I applied the change in the test system as well - here mistake # 4 occurred. I then ran the MapGuide/AIMS performance report again but the performance had not decreased. Performance was good as before. Then I changed the data source of the Oracle user I was testing with and pointed it to our production database. When I ran the test again performance was slow. I stopped any further testing for the day and did something else.
At this point my theory was that the same query has different execution plans in Oracle (production and test) therefore different execution times due to different Oracle settings. What didn't really fit was that when I ran the spatial queries in SQLDeveloper I didn't see any significant difference in performance. But I didn't have anything else to test so finally I decided to replace the view definition in production with the previous view definition - basically removing the change which had been applied recently. I re-ran the performance report in MapGuide/AIMS and suddenly I got an error message:
Failed to stylize layer: DP_xxx
An exception occurred in FDO component.
ORA-00904: "G"."OWNER_OBJTYP": invalid identifier
...
Now I realised one of my mistakes. MapGuide/AIMS caches certain settings/definitions. I don't know how it works in detail but probably roughly like that:
- when AIMS/MapGuide starts it reads the Oracle schema and caches some information
- (SELECT) statements are being build based on the cached information and used against Oracle
For a view MapGuide/AIMS seems to read the view definition and then creates a SELECT statement which includes all columns of the view.
If the view definition in Oracle changes and new columns are added to the view MapGuide can still run the SELECT statement against the new view definition - basically just ignoring all newly added columns. But when columns have been removed from the view the SELECT statement still includes those columns and Oracle will response with ora-00904.
Mistake 4 - when I applied the change to view definition in the test system I did not restart MapGuide/AIMS. MapGuide was still using the SELECT statement which did not include the newly added column in the view. But exactly that new column had a negative impact on performance.
If I had restarted MapGuide/AIMS at this point in time I could have avoided further testing and head-scratching.
Mistake 3 - if I had listened to my colleague who offered to remove the change in production I might have noticed the cause of the problem earlier (because even without restarting MapGuide/AIMS we would have seen the Ora-00904 message in the log file).
Mistake 2 - my spatial query was simplified, the select clause included only "fid, geom". As it did not include the column which caused the performance issue there is really no surprise that performance in both production and test were nearly the same. Conclusion: always use (if possible) exactly the same (SELECT) statement as the application when you do testing!
Mistake 1 - I should have noticed the difference in view definitions myself just by copying the view definitions from production and test into an editor and then noticing the different numbers of lines....
So at the end a trivial issue but it took longer then necessary to get it fixed.
Why you should restart AIMS/MapGuide when testing ....
I noticed by chance that a few layers - which are all based on one Oracle view - were significantly slower in our production system but no performance issue occurred in our test environment. Whilst investigating the issue I made a few mistakes which made the whole process of finding and fixing the issue more time consuming than necessary. Here are the details:
After noticing the issue I asked my colleagues if they had any idea why this is and whether anything had changed recently. None of them had an explanation so I spent some time checking / testing:
- running performance reports in MapGuide/AIMS noticing that time spent on "Layer" is 20times slower in production
- comparing view definitions but only cursorily (looking at WHERE clause only) : mistake #1
- re-creating spatial index for base table : no change
- running 1-click maintenance in Map-Administrator : no change
- copy a layer definition from production to test environment : no change
- trying to load view into other applications to see whether the issue is related to application or to view but GeoRaptor couldn't add layer to map due to rowid issue, QGIS rejected view complaining that view didn't have a proper spatial index and TatukGIS Viewer basically saying something similar (first the message about the spatial index issue made me re-re-create the spatial index but testing further views in QGIS/TatukGISViewer I got the same messages for all other views as well).
- running a few spatial SQL queries in SQLDeveloper but there wasn't any significant difference between test and production : mistake # 2
After spending some time on running these tests a colleague came back to me and told me, that the view had changed recently but when he applied the change he didn't notice any performance issue whilst generating graphics in Map. The change was made by adding a subquery to the SELECT clause. He asked if the change should be removed but I replied that I will do some tests now taking the new information into account : mistake # 3.
As the change to the view definition had only been applied in the production system I applied the change in the test system as well - here mistake # 4 occurred. I then ran the MapGuide/AIMS performance report again but the performance had not decreased. Performance was good as before. Then I changed the data source of the Oracle user I was testing with and pointed it to our production database. When I ran the test again performance was slow. I stopped any further testing for the day and did something else.
At this point my theory was that the same query has different execution plans in Oracle (production and test) therefore different execution times due to different Oracle settings. What didn't really fit was that when I ran the spatial queries in SQLDeveloper I didn't see any significant difference in performance. But I didn't have anything else to test so finally I decided to replace the view definition in production with the previous view definition - basically removing the change which had been applied recently. I re-ran the performance report in MapGuide/AIMS and suddenly I got an error message:
Failed to stylize layer: DP_xxx
An exception occurred in FDO component.
ORA-00904: "G"."OWNER_OBJTYP": invalid identifier
...
Now I realised one of my mistakes. MapGuide/AIMS caches certain settings/definitions. I don't know how it works in detail but probably roughly like that:
- when AIMS/MapGuide starts it reads the Oracle schema and caches some information
- (SELECT) statements are being build based on the cached information and used against Oracle
For a view MapGuide/AIMS seems to read the view definition and then creates a SELECT statement which includes all columns of the view.
If the view definition in Oracle changes and new columns are added to the view MapGuide can still run the SELECT statement against the new view definition - basically just ignoring all newly added columns. But when columns have been removed from the view the SELECT statement still includes those columns and Oracle will response with ora-00904.
Mistake 4 - when I applied the change to view definition in the test system I did not restart MapGuide/AIMS. MapGuide was still using the SELECT statement which did not include the newly added column in the view. But exactly that new column had a negative impact on performance.
If I had restarted MapGuide/AIMS at this point in time I could have avoided further testing and head-scratching.
Mistake 3 - if I had listened to my colleague who offered to remove the change in production I might have noticed the cause of the problem earlier (because even without restarting MapGuide/AIMS we would have seen the Ora-00904 message in the log file).
Mistake 2 - my spatial query was simplified, the select clause included only "fid, geom". As it did not include the column which caused the performance issue there is really no surprise that performance in both production and test were nearly the same. Conclusion: always use (if possible) exactly the same (SELECT) statement as the application when you do testing!
Mistake 1 - I should have noticed the difference in view definitions myself just by copying the view definitions from production and test into an editor and then noticing the different numbers of lines....
So at the end a trivial issue but it took longer then necessary to get it fixed.
After noticing the issue I asked my colleagues if they had any idea why this is and whether anything had changed recently. None of them had an explanation so I spent some time checking / testing:
- running performance reports in MapGuide/AIMS noticing that time spent on "Layer" is 20times slower in production
- comparing view definitions but only cursorily (looking at WHERE clause only) : mistake #1
- re-creating spatial index for base table : no change
- running 1-click maintenance in Map-Administrator : no change
- copy a layer definition from production to test environment : no change
- trying to load view into other applications to see whether the issue is related to application or to view but GeoRaptor couldn't add layer to map due to rowid issue, QGIS rejected view complaining that view didn't have a proper spatial index and TatukGIS Viewer basically saying something similar (first the message about the spatial index issue made me re-re-create the spatial index but testing further views in QGIS/TatukGISViewer I got the same messages for all other views as well).
- running a few spatial SQL queries in SQLDeveloper but there wasn't any significant difference between test and production : mistake # 2
After spending some time on running these tests a colleague came back to me and told me, that the view had changed recently but when he applied the change he didn't notice any performance issue whilst generating graphics in Map. The change was made by adding a subquery to the SELECT clause. He asked if the change should be removed but I replied that I will do some tests now taking the new information into account : mistake # 3.
As the change to the view definition had only been applied in the production system I applied the change in the test system as well - here mistake # 4 occurred. I then ran the MapGuide/AIMS performance report again but the performance had not decreased. Performance was good as before. Then I changed the data source of the Oracle user I was testing with and pointed it to our production database. When I ran the test again performance was slow. I stopped any further testing for the day and did something else.
At this point my theory was that the same query has different execution plans in Oracle (production and test) therefore different execution times due to different Oracle settings. What didn't really fit was that when I ran the spatial queries in SQLDeveloper I didn't see any significant difference in performance. But I didn't have anything else to test so finally I decided to replace the view definition in production with the previous view definition - basically removing the change which had been applied recently. I re-ran the performance report in MapGuide/AIMS and suddenly I got an error message:
Failed to stylize layer: DP_xxx
An exception occurred in FDO component.
ORA-00904: "G"."OWNER_OBJTYP": invalid identifier
...
Now I realised one of my mistakes. MapGuide/AIMS caches certain settings/definitions. I don't know how it works in detail but probably roughly like that:
- when AIMS/MapGuide starts it reads the Oracle schema and caches some information
- (SELECT) statements are being build based on the cached information and used against Oracle
For a view MapGuide/AIMS seems to read the view definition and then creates a SELECT statement which includes all columns of the view.
If the view definition in Oracle changes and new columns are added to the view MapGuide can still run the SELECT statement against the new view definition - basically just ignoring all newly added columns. But when columns have been removed from the view the SELECT statement still includes those columns and Oracle will response with ora-00904.
Mistake 4 - when I applied the change to view definition in the test system I did not restart MapGuide/AIMS. MapGuide was still using the SELECT statement which did not include the newly added column in the view. But exactly that new column had a negative impact on performance.
If I had restarted MapGuide/AIMS at this point in time I could have avoided further testing and head-scratching.
Mistake 3 - if I had listened to my colleague who offered to remove the change in production I might have noticed the cause of the problem earlier (because even without restarting MapGuide/AIMS we would have seen the Ora-00904 message in the log file).
Mistake 2 - my spatial query was simplified, the select clause included only "fid, geom". As it did not include the column which caused the performance issue there is really no surprise that performance in both production and test were nearly the same. Conclusion: always use (if possible) exactly the same (SELECT) statement as the application when you do testing!
Mistake 1 - I should have noticed the difference in view definitions myself just by copying the view definitions from production and test into an editor and then noticing the different numbers of lines....
So at the end a trivial issue but it took longer then necessary to get it fixed.
Friday, 10 July 2015
MapGuide / AIMS - show number of sessions
As shown in Jackie's blog one can easily count the number of current sessions. I put such a counter on our internal web site where we keep links to different web interfaces and scripts related to our AIMS/MapGuide installations:
The script is written in PHP:
The vaw_settings.php file contains only the path to MapGuide session repository ($pathSessionRepository).
The script for counting current sessions is called from within another PHP script which runs on one of the AIMS servers. It also calls the slightly modified version of the script on the remote server:
Depending on whether the script runs locally or remotely it needs to be modified slightly (see comments in scripts) and the way the script is called varies as well (relative path vs. complete URL, include vs. file_get_contents).
AIMS/MapGuide 2013
The script is written in PHP:
<?php
// for remote server
//
// (1) INLCUDE ../vaw_settings.php;
//
// for local server
//
// (1) INLCUDE Verweis mit ./vaw_settings.php;
//
include './vaw_settings.php';
$fi = new FilesystemIterator($pathSessionRepository, FilesystemIterator::SKIP_DOTS);
//
$number = (iterator_count($fi)-1)/2 ;
// for remote server
//
// (2) echo $number;
//
// for local server
//
// (2) return $number;
//
return $number;
?>
The vaw_settings.php file contains only the path to MapGuide session repository ($pathSessionRepository).
The script for counting current sessions is called from within another PHP script which runs on one of the AIMS servers. It also calls the slightly modified version of the script on the remote server:
...
$script_number_users = 'misc/count_logged_users.php';
...
//local server
//only relative path to script
$request_url = $script_number_users;
echo "# of users logged in ($label_server1):____".include($request_url).".";
//remote server
//complete url
$request_url = $basis_url_server3.$url_vaw_scripte.$script_number_users;
echo "# of users logged in ($label_server3):____".file_get_contents($request_url).".";
Depending on whether the script runs locally or remotely it needs to be modified slightly (see comments in scripts) and the way the script is called varies as well (relative path vs. complete URL, include vs. file_get_contents).
AIMS/MapGuide 2013
Tuesday, 30 June 2015
MapGuide / AIMS - performance, file access
We run AIMS on virtualized W2008R2 machines with 6 GB RAM. Most data displayed as maps comes directly from Oracle Spatial. Only for one job enabled Map industry model we decided to export 40+ feature classes and views to an SDF file. File size is roughly 200MB.
In order to speed things up I created a RAM Disk on the server and did some performance tests:
- hard disk access (read/write) was already much better then on my physical machine
- depending on RAM disk driver and test performed read access was faster by up to factor 6 compared to hard disk access
Surprisingly moving the SDF file to the RAM disk did not help improving performance in AIMS/MapGuide.
Conclusion seems to be that W2008R2 has already cached the SDF file and no further improvement is possible here. As the SDF file is used in all web mapping projects and layers based on file are visible by default I assume that the file will be cached all the time.
Here are a few screenshots related to the testing:
- hard disk performance, local machine:
- hard disk performance, virtualized 2008r2 server
- ram disk performance, local machine (imdisk)
- ram disk performance, w 2008r2 (softperfect)
- AIMS/MapGuide performance test with sdf based map - sdf file on disk on virtualized w2008r2 server
- AIMS/MapGuide performance test with sdf based map - sdf file on ram disk on virtualized w2008r2 server
AIMS 2013
In order to speed things up I created a RAM Disk on the server and did some performance tests:
- hard disk access (read/write) was already much better then on my physical machine
- depending on RAM disk driver and test performed read access was faster by up to factor 6 compared to hard disk access
Surprisingly moving the SDF file to the RAM disk did not help improving performance in AIMS/MapGuide.
Conclusion seems to be that W2008R2 has already cached the SDF file and no further improvement is possible here. As the SDF file is used in all web mapping projects and layers based on file are visible by default I assume that the file will be cached all the time.
Here are a few screenshots related to the testing:
- hard disk performance, local machine:
- hard disk performance, virtualized 2008r2 server
- ram disk performance, local machine (imdisk)
- ram disk performance, w 2008r2 (softperfect)
- AIMS/MapGuide performance test with sdf based map - sdf file on disk on virtualized w2008r2 server
- AIMS/MapGuide performance test with sdf based map - sdf file on ram disk on virtualized w2008r2 server
AIMS 2013
Subscribe to:
Comments (Atom)