Thursday, 21 December 2017

ArcPy - find layers with joins

Here is a brute force script for finding layers with JOINs. As there doesn't seem to be a "hasJoin" property I just use arcpy.RemoveJoin_management which throws an exception if there is no join. This is script is only usefull for once off checks.

ArcMap 10.3

 # find (all) joins in MXD by removing them!  
 # usage: copy MXD and run script or run script and don't save MXD as all JOINS will be removed  
 # result will be a list of all layers where JOIN has been removed succesfully  
 import arcpy  
 mxd = arcpy.mapping.MapDocument("CURRENT")  
 layers = arcpy.mapping.ListLayers(mxd)  
 allJoins = ''  
 for layer in layers:  
     arcpy.RemoveJoin_management (  
     print("join removed: "  
     allJoins += '\n'  
   # no join throws exception  
   except Exception as err:  

Wednesday, 20 December 2017

ArcPy - MoveLayer fails with "assert refe_c is not None, 'Did not find reference layer.'"

Just wanted to copy and paste all layers into new MXD file. Unfortunately order of layers in TOC is not retained. So I googled for script and found one - but MoveLayer failed with error message "assert refe_c is not None, 'Did not find reference layer.'". The post did not mention anything like that.

Solution in my case : for the reference layer a fresh reference is required. 

old: refLayer=moveLayer
new: refLayer = arcpy.mapping.ListLayers(mxdT,, dfT)[0]  

ArcMap 10.3.1

Here is the script:

 # -*- coding: utf-8 -*-  
 # source code initially taken from:  
 # assumption : no layer groups  
 # my use case : copy and paste all layers into new MXD, layer order is not retained - run script to restore layer order  
 import arcpy  
 mxd_source = r"C:\temp\source1.mxd"  
 mxd_target = r"C:\temp\target.mxd"  
 mxdT = arcpy.mapping.MapDocument(mxd_target)  
 # assumption: only one DF  
 dfT = arcpy.mapping.ListDataFrames(mxdT, "")[0]  
 mxdS = arcpy.mapping.MapDocument(mxd_source)  
 # assumption: only one DF  
 dfS = arcpy.mapping.ListDataFrames(mxdS, "")[0]  
 #initialize refLayer with layer  
 refLayer = arcpy.mapping.ListLayers(mxdT,"",dfT)[0]  
 for layerSource in arcpy.mapping.ListLayers(mxdS,"",dfS):    
   print("processing layer in source: '" + sourceLayer+"'.")    
   for layerTarget in arcpy.mapping.ListLayers(mxdT,"",dfT):  
    moveLayer= a+b     
    if sourceLayer==moveLayer:  
      print("moving layer:'"+moveLayer+"', reference layer: '""'.")  
      arcpy.mapping.MoveLayer(dfT,refLayer,moveLayer, "After")       
      #get fresh layer reference - otherwise MoveLayer will fail next time:  
      refLayer = arcpy.mapping.ListLayers(mxdT,, dfT)[0]             

Thursday, 19 October 2017

Job available

We are currently looking for a GIS specialist. The job description can be found here:

Please apply if you are interested.

Tuesday, 12 September 2017

Warning: An exception occurred in FDO component.

I got the following warning in MapGuide log file:

Warning: An exception occurred in FDO component.
        Zeichenfolge ist kein gültiger Ausdruck. 
  - MgStylizationUtil.ExceptionTrap() line 231 file SE_ExpressionBase.cpp

The message means "string is invalid expression".

In my case it seems the issue is related to text label style. The text label was originally published using MTEXT as style. Later on - due to a bug in MapGuide - I changed the text style to TEXT. But since I changed TEXT back to MTEXT the message doesn't appear in the log files anymore. 

Autodesk Infrastructure MapServer 2017

Thursday, 17 August 2017

File based Industry Model - how to modify views

When you use a file based Map Industry Model (IM) you might want to extend the data model and add attributes to one of the tables. If you are not familiar with databases, SQL and the concept of "table" and "view" this blog post will explain some basic concepts behind Map IMs and also show how you can add attributes to both tables and views. 

Tables and views 

AutoCAD Map comes with ready-to-use Industry Models. These IMs contain many tables, views, forms, labels and so on. Tables actually store data, views only provide a "view" on one or more tables and do not contain any data. Look here for more details: 

Nothing prevents you from storing all your data in just one big table. But there are disadvantages to it and therefore data is usually split across multiple tables. Here is more information on why: 

When you have data across multiple tables you need to combine data in order to get the information you need - "views" are one way to achieve that. 

In Autodesk Infrastructure Administrator (AIA) tables and views appear the same at first glance. But if the Data Model was set up properly a view will have a display name ("caption") containing the word "view". Whenever you create a new table or attribute in AIA you need to enter a name but you can also enter a "caption" - a more descriptive name:

Feature class name and caption 
The caption will be used in AIA and Map, not the name of the object: 

List of all tables/views in "Point" - column "Feature Class" shows captions

Whenever you need to work directly in the database you first need to find out the name of the table/view. The names are shown in AIA as well - just look below the tree view where the caption and the name of the table/view (in brackets) are displayed: 

For a selected item the name and caption is shown in the status bar below the Data model tree view.

You will also notice another convention - in a databases a view name often contains "_v_" to indicate that it is a view and not a table. 

Example: in WA Industry Model there is a table called: "Fitting" (see screenshot above):

"Fitting" - display name for table "WA_FITTING" shown in AIA and Map 
"WA_FITTING" - name of the table in the database (WA indicating that the table belongs to the WA Industry Model) 
"Fitting View" - display name for view "WA_V_FITTING" shown in AIA and Map 
"WA_V_FITTING" - name of the view in the database  

In AIA you can easily add attributes to a table. If you add an attribute you usually want to have the attribute shown on your form as well in order to be able to enter data. So you need to open the form for the table in AIA Form Designer and add the new attribute there as well. In case the new attribute is also needed to style features in your drawing you need to incorporate it in your layer definition. 

But layers in Display Models often are based on views and not on tables. In this case adding an attribute to a table is not sufficient - the attribute also needs to be added to the view in question. Now, here is a limitation of AIA - you cannot create or modify an existing view the same way you can create or modify a table. For a view the context menu function "Add attribute" is not available: 

For a view "Add Attribute" is not available

In order to do that you need to use a tool called "SQL Sheet" which comes with AIA - we will come to that in a second.

How do you know whether a layer in Display Model is based on a table or on view? 
After generating graphics using the Display Model in question you can check either by hovering the mouse on top of the layer name in Display Manager - a flyout will show the layer's data source (table/view name, not the caption) - 

Layer data source name is shown in fly out for layer in Display Manager

or open the StyleEditor for the layer and the table/view name is displayed there as well on the top: 
Layer data source name is shown in Style Editor

 If the name contains "_V_" you can be quite certain its a view.  

By the way - the layer name as shown in Display Manager is often different from the table/view names the layer is based on. You have three different names by now: 
- the actual table/view name as used in the database 
- the caption for a table/view as shown in AIA and Map 
- the layer name in DisplayManager  
All point to the same thing - a certain table or view - but might have (slightly) different names. 

Modify an existing view 

Example - add a new attribute to the Fitting table and also use the attribute when styling the layer in Map. As the layer in Map is based on a view (see screenshot above) we need to modify it as well. 

1) add new attribute to table 

2) in Form Designer drag and drop attribute to the form 

3) save drawing (important!) 

4) open SQL Sheet (File menu, choose "SQL Sheet"),  

5) choose "SqLite"  and your drawing file (containing the file based IM, drawing should not be opened in Map at the same time) 

4) in top right corner choose "Views" to get a list of all views in your current IM 

5) select the view to modify - WA_V_FITTING -  on the right hand side below the list you will see the attributes of the view 

6) right-click on view name and choose "Modify View" - in the left side panel the view definition will be shown 

where g.FID_ATTR = a.FID 

As you can see the view takes data from two tables - WA_POINT and WA_FITTING. 
WA_POINT stores the geometry of a feature, WA_FITING stores all attributes which are useful for fittings. We added a new attribute to WA_FITTING earlier. This attribute needs to be added to the view as well. 

You can also notice a letter after each table name (g and a) - they are called "alias" in order to shorten then whole expression. Without alias (which could be a word not just a single letter) the statement would read like this: 


The alias or table name is given to indicate from which table a certain attribute comes from (required in cases where the same attribute name is used in more than one table). 

Now you only need to add your attribute name (including the alias) - don't forget to add an additional comma between the last attribute and the new attribute: 

, a.MA_SIZE 
where g.FID_ATTR = a.FID 

Attribute name needs to be upper cases. 

Click on the green triangle to execute the statement - you will get the following error message: 

SQL execution error number 0, table WA_V_FITTING already exists 

We cannot overwrite a view - we have to delete the old one first and re-create it afterwards. Add the following line before the CREATE VIEW statement: 

, a.MA_SIZE 
where g.FID_ATTR = a.FID 

Execute the query again, result should be:

Command executed (0). 

, a.MA_SIZE 
where g.FID_ATTR = a.FID 
Command executed (0). 

If you receive an error message like this: 

SQL execution error number 0, no such column: a.MA_SIZE 
, a.MA_SIZE 

Did you save your drawing after adding the attribute to your table?  If you haven't saved the drawing SQLSheet doesn't seem to "see" the new attribute. 

7. close SQL Sheet 

8. Close AIA 

9. open AIA and reload your drawing, check the view in Data Model tree view - the attribute is now available in the view as well: 

Before modifying your data base with SQL Sheet - back up your drawing file. 

Friday, 4 August 2017

Uploading MapGuide package - Exception

Uploading MapGuide package - Exception

If you modify a MapGuide package make sure that you zip the correct files and folders afterwards.
I unzipped a MapGuide package, changed some values and zipped the folder but received the following error message when trying to upload the file:

Maestro error message:

value cannot be nul
Parametername: entry

System.ArgumentNullException:  value cannot be nul
Parametername: entry
   bei ICSharpCode.SharpZipLib.Zip.ZipFile.GetInputStream(ZipEntry entry)
   bei Maestro.Packaging.PackageBuilder.UploadPackageNonTransactional(String sourceFile, UploadPackageResult result) in C:\working\JenkinsCI\home\slave_win\jobs\Maestro trunk\workspace\Maestro.Packaging\PackageBuilder.cs:Zeile 217.
MapGuide error message:

   ERROR_MESSAGE:    An exception occurred in DWF component. File not found in archive
STACK_TRACE:    - MgLibraryRepositoryManager.LoadResourcePackage() line 183 file c:\working\build_area\mapguide\2.5.2\x64\mgdev\server\src\services\resource\LibraryRepositoryManager.cpp - MgResourcePackageLoader.Start() line 150 file c:\working\build_area\mapguide\2.5.2\x64\mgdev\server\src\services\resource\ResourcePackageLoader.cpp - MgResourcePackageLoader.CreateByteReader() line 108 file c:\working\build_area\mapguide\2.5.2\x64\mgdev\server\src\services\resource\ResourcePackageLoader.cpp - MgZipFileReader.ExtractArchive() line 61 file c:\working\build_area\mapguide\2.5.2\x64\mgdev\server\src\services\resource\ZipFileReader.cpp

My mistake was to zip the top level folder - which I got when I unzipped the package. But you need to zip the content of the top level folder, not the top level folder itself. 

AIMS 2017

Friday, 9 June 2017

Map-Plot extension - iusse with FDO hatches in a rotated viewport

As mentioned already - back then for Map 2013 - there is an issue with hatches in a rotated viewport when using Map (FDO) layers. For details see here. The issue has not been resolved yet (Map 2017 SP1).

To get around it we basically replaced the problematic Map layer hatch with an AutoCAD hatch. We created 3 drawings (one for each of the print scales: 1:500, 1:1000 and 1:2000) containing only the hatches which cause issues, these hatches are plain AutoCAD hatches which print fine in a rotated viewport. These drawings were created by Map-DWG export.

As we use our own batch plot extension (as mentioned here) I just needed to extend it in order to attach one of the three drawings as XREF automatically before the layout gets printed off.

Map 2017, SP1

 IPlot plot = myLibrary.FindPlot(FIDPlot);  
 PltPlotRenderer r = PltPlotRendererFactory.CreateRenderer(RendererConfiguration.Plot, plot, RenderingCompatibility.CurrentVersion);  
 r.Render(); // renders the plot as AutoCAD layout  
 // xrefs   
 string xref_pfad = "S:\\GIS\\Va\\Map3D2017\\Startbilder\\";  
 string xref_500 = "Wald_500.dwg";  
 string xref_1000 = "Wald_1000.dwg";  
 string xref_2000 = "Wald_2000.dwg";  
 string plotname = plot.Name;          
 string xref = xref_500;  
 if (plotname.Contains("500"))  
      xref = xref_500;  
 else if (plotname.Contains("1000"))  
      xref = xref_1000;  
      xref = xref_2000;  
 //attach matching xref            
 attachDrawing2(xref_pfad, xref);  
 //send xref to bottom  
 // regenerate all viewports  
           //copy and paste from AutoCAD forum ? - didn't keep the link,   
           public static void RegenAll()  
       Document doc = Autodesk.AutoCAD.ApplicationServices.Application.DocumentManager.MdiActiveDocument;  
       Editor ed = doc.Editor;  
       Database db = doc.Database;  
       Viewport vp;  
       using (DocumentLock docLock = Autodesk.AutoCAD.ApplicationServices.Application.DocumentManager.MdiActiveDocument.LockDocument())  
         using (Transaction tr = db.TransactionManager.StartTransaction())  
           LayoutManager lm = LayoutManager.Current;  
           if (lm.CurrentLayout.ToLower() == "model") db.TileMode = false;  
           lm.CurrentLayout = lm.CurrentLayout;  
           Layout layout = tr.GetObject(lm.GetLayoutId(lm.CurrentLayout), OpenMode.ForRead) as Layout;  
           ObjectIdCollection vpIds = layout.GetViewports();  
           if (vpIds.Count < 2) { ed.Regen(); return; }  
           for (int i = 1; i < vpIds.Count; i++)  
             vp = tr.GetObject(vpIds[i], OpenMode.ForWrite) as Viewport;  
             Autodesk.AutoCAD.ApplicationServices.Application.SetSystemVariable("Cvport", vp.Number);  
     //copy and paste from :_  
     public void attachDrawing2(string filepath, string filename)  
       Database acCurDb;  
       acCurDb = Autodesk.AutoCAD.ApplicationServices.Application.DocumentManager.MdiActiveDocument.Database;  
       string f = filepath + filename;  
       if (!File.Exists(f))  
         return ;       
       string  name = Path.GetFileNameWithoutExtension(f);  
         using (var tr = acCurDb.TransactionManager.StartOpenCloseTransaction())  
           DocumentLock docLock = Autodesk.AutoCAD.ApplicationServices.Application.DocumentManager.MdiActiveDocument.LockDocument();  
           using (docLock)  
             var xId = acCurDb.AttachXref(f, name);  
             if (xId.IsValid)  
               BlockTable acBlkTbl;  
               acBlkTbl = tr.GetObject(acCurDb.BlockTableId, OpenMode.ForRead) as BlockTable;  
               var btr = tr.GetObject(acBlkTbl[BlockTableRecord.ModelSpace], OpenMode.ForWrite) as BlockTableRecord;  
               Point3d insPt = new Point3d(0, 0, 0);  
               var br = new BlockReference(insPt, xId);  
               tr.AddNewlyCreatedDBObject(br, true);  
       catch (Autodesk.AutoCAD.Runtime.Exception e)  
           //send XREF to bottom of draw order, copy and paste - didnt keep link  
     public void toBack()  
       Database acCurDb;  
       acCurDb = Autodesk.AutoCAD.ApplicationServices.Application.DocumentManager.MdiActiveDocument.Database;  
         using (var tr = acCurDb.TransactionManager.StartTransaction())  
           DocumentLock docLock = Autodesk.AutoCAD.ApplicationServices.Application.DocumentManager.MdiActiveDocument.LockDocument();  
           using (docLock)  
             //var bt = (BlockTable)tr.GetObject(acCurDb.BlockTableId, OpenMode.ForRead);  
             //var ms = (BlockTableRecord)tr.GetObject(bt[BlockTableRecord.ModelSpace], OpenMode.ForRead);  
             var bt = tr.GetObject(acCurDb.BlockTableId, OpenMode.ForRead) as BlockTable;  
             var ms = tr.GetObject(bt[BlockTableRecord.ModelSpace], OpenMode.ForWrite) as BlockTableRecord;  
             DrawOrderTable drawOrderTab = tr.GetObject(ms.DrawOrderTableId, OpenMode.ForWrite) as DrawOrderTable;  
             ObjectIdCollection ids = new ObjectIdCollection();  
             // Loop through the contents of the modelspace  
             foreach (var id in ms)  
               // We only care about BlockReferences  
               var br = tr.GetObject(id, OpenMode.ForRead) as BlockReference;  
               if (br != null)  
                 // Check whether the associated BlockTableRecord is  
                 // an external reference  
                 var bd = (BlockTableRecord)tr.GetObject(br.BlockTableRecord, OpenMode.ForRead);  
                 if (bd.IsFromExternalReference)  
             if(ids.Count> 0)  

Wednesday, 24 May 2017

Map Industry Model Plot Extension

If you use Map Industry Model Plot Extension make sure that the layers in your main display model do not contain a spatial filter. A spatial filter is automatically created when you save a display model. In order to remove the spatial filter you need to open the .layer file in an editor and remove the filter setting:

<Filter>GEOM ENVELOPEINTERSECTS GeomFromText('POLYGON XYZ ((698785 258855 0, 700411 258855 0, 700411 260393 0, 698785 260393 0, 698785 258855 0))')</Filter>



It seems that Map applies the filter from the layer file when generating a Map plot. If the spatial filter does not match with the area you want to create a plot for the layer content will not be shown.

Map 2017, Sp1

Thursday, 4 May 2017

simple web site for converting drawing file into 3d pdf using FME

Once in while our CIVIL users want to convert a drawing file into a 3d PDF file. As they are not familiar with FME and dont have access to it anyway I set up a web page where a drawing file can be uploaded and gets converted to a 3d PDF. FME 32bit is installed on the server just for this purpose.

The website consists of two frames. The first frame allows to upload the drawing file, the second frame just lists the contents of the folder on the server where the PDF file gets stored.

Here is the PHP code for the upload. After uploading it triggers the execution of a BATCH file (see below) which starts the FME workbench. The workbench is a simple ACAD drawing reader connected to an 3d PDF writer. Everything is kept very simple.

 //refresh download frame in order to list newly created PDF file  
 <h1>DWG zu 3D PDF</h1>  
 <form action="<?php echo $_SERVER["PHP_SELF"]; ?>" method="post" enctype="multipart/form-data">  
   DWG ausw&auml;hlen:  
   <input type="file" name="uploaded" id="fileToUpload"><br><br>  
      DWG hochladen:  
   <input type="submit" value="DWG hochladen" name="submit">  
 Nach dem Hochladen kann es einige Zeit dauern, bis die DWG konvertiert wurde. Bitte warten, bis eine Meldung erscheint.<br><br>  
 Im unteren Fenster sollte nach erfolgreicher Erstellung der PDF diese zum Download aufgelistet sein - falls nicht die gesamte Webseite aktualisieren.  
 <br><b>PDF bitte herunterladen (rechte Maustaste >> "Link speichern unter") - im Webbrowser werden keine 3d PDFs angezeigt.</b>  
 // PHP.INI  
 // upload_max_filesize = 20M  
 // --> if files are too big variables such as $_FILES['uploaded']['name'] and others are empty  
 // max_execution_time = 90  
 // FastCGI PHP in IIS execution time set to 90 secs  
 // Upload folder : set permissions  
 if ($_POST['submit'] == '') die();       
 $target = "E:/www_fme_3dpdf_batch/uploads/";   
 $target = $target . basename( $_FILES['uploaded']['name']) ;   
 if(move_uploaded_file($_FILES['uploaded']['tmp_name'], $target))  
      //sending message to browser - doesnt work with IIS it seems :-(  
      echo "<br><br>Datei <b>'". basename( $_FILES['uploaded']['name']). "'</b> hochgeladen.";   
      echo "<br><br>Dateikonvertierung gestartet....<br><br>";        
      echo "<small><pre>";  
      system("cmd /c E:/www_fme_3dpdf_batch/run_fme_acad2pdf.bat 2>&1", $output);  
      echo "</small></pre>";  
      echo "Problem aufgetreten. Datei nicht hochgeladen.";   

The batch file content:

 SET SOURCE="E:\www_fme_3dpdf_batch\uploads"  
 SET OUTPUT="E:\www_fme_3dpdf_batch\downloads"  
 DEL /Q %OUTPUT%\*.*  
 FOR %%F IN ("%SOURCE%\*.dwg") DO (    
     C:\apps\FME32\fme.exe acad2pdf.fmw --SourceDataset_ACAD "%%F"^  
                --DestDataset_PDF "%OUTPUT%\%%~nF.pdf"^  
                --LOG_FILE "%OUTPUT%\%%~nF_load.log"  
 DEL /Q %SOURCE%\*.*  

Wednesday, 15 March 2017

Map Crash with error message "Method not found: '!!0[] System.Array.Empty()'"

I had my PC freshly installed and afterwards neither AutoCAD Map 2017 nor 3ds Max 2017 ran. Both crashed with error messages which looked like an .Net issue to me. Our IT department took my PC back and tried to reinstall - but to no avail. Apparently they had to change the harddisk to get both applications to run properly. 

Anyway - Map and Map-Administrator run now but only if I avoid loading a third party plugin. As soon as the plugin loads both programs crash - which doesn't happen on any other PC I tried. The CER report details show the following message:

<InnerException type="System.MissingMethodException"><Message>Methode nicht gefunden: "!!0[] System.Array.Empty()".</Message><StackTrace><Method>

It seems that the plugin was compiled against .net 4.6 but the .net runtime is lower than that. You can find a more detailed explanation here (reply by Alexandru).

That is a bit odd as Map 2017 itself requires .net 4.6 and therefore should not start up or run without 4.6 - but it does. When I checked the version of .net framework under "Installed Programs" it showed 4.6 but also 4.5:

The problem seems to be with language pack and / or registry settings.  Here are the two keys where the values don't match:

HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\NET Framework Setup\NDP\v4\Full\1031\Version = 4.6.00081
HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\NET Framework Setup\NDP\v4\Full\1033\Version = 4.5.51209

On second PC both keys have the same value and under  "Installed Programs" it shows .net 4.6 twice:

Map 2017, SP1

Thursday, 2 March 2017

MapGuide log file

It is good practice to monitor log files. Unfortunately MapGuide log files are not easy to read:
- StackTrace details take up more space then the message itself, it is difficult to see when an error message starts and ends
- certain messages might appear very often although there are not really indicating an issue (such as "Session expired" message).

Here is Python3 script which reads MapGuide logs files and gets rid of StackTrace details and also filters out uninteresting messages. It adds a line number so that you can find the message in the original log file quickly. It also creates a simple summary for messages and their frequency. As we have two MapGuide servers the script is configured to read two different log file folders.  

If you want to use it you need to configure:
- the path for the MapGuide log files
- the path for saving the shortened log file 
- error messages you want to exclude
- the number of most recent log files you want to process (such as the 3 most recent ones)

I haven't done much in Python yet - the script doesn't do much error handling and has other shortcomings as well.

Thats how the result might look like:

**************** Logfile: //wsstadt529/logfiles/MapGuide\Error.log

3   DATE_TIME 2017-03-02 / 01:50:19
4    Success: Server stopped.

DATE_TIME 2017-03-02 / 01:51:23
6    Success: Server started.

DATE_TIME 2017-03-02 / 07:27:12
444    Error: An exception occurred in FDO component.
445           Error occurred in Feature Source (Library://FS_BEAR/WT_PO_BW_B/Data/TopobaseDefault.FeatureSource): Zeichenfolge ist kein gültiger Filter.  (Cause: Zeichenfolge ist nicht korrekt formatiert. , Root Cause: Zeichenfolge ist nicht korrekt formatiert. )


DATE_TIME 2017-03-02 / 13:35:34
953    Error: An exception occurred in FDO component.
954           Error occurred in Feature Source (Library://FS_BEAR/WT_PO_BW_B/Data/TopobaseDefault.FeatureSource): Zeichenfolge ist kein gültiger Filter.  (Cause: Zeichenfolge ist nicht korrekt formatiert. , Root Cause: Zeichenfolge ist nicht korrekt formatiert. )

************* Summary ***********************

# 3 :: Error: Failed to stylize layer: ZIM_Anlageobjekte_DynaS_GeoRest_MBR
# 36 :: Error: Failed to stylize layer: ZH_Orthofoto2015
# 1 :: Success: Server started.
# 1 :: Success: Server stopped.
# 27 :: Error: An exception occurred in FDO component.

************* File(s) ***********************
File processed: //wsstadt529/logfiles/MapGuide\Error.log
# of messages:68
# of messages excluded: 51

Here is the script:

 import os  
 import sys  
 from collections import Counter  
 """ extract date and time from logfile line """  
 def extractdatetime(a_text_line):  
   #DateTime Format in MapGuide-LogFile:  
   #<2016-12-15T11:00:34> <2015-07-05T12:39:55>  
   date = a_text_line[1:11]  
   time = a_text_line[12:20]  
   return (date, time)  
 """ checks whether a certain error message should be ignored / excluded from further processing """  
 def isErrorMessagesToExclude(a_message):    
   for text_to_find in errormessage_to_exclude:  
     if text_to_find in a_message:  
       return True  
   return False  
 """ removes the StackTrace details from an error message """  
 def removeStackTrace(a_message):    
   pos = a_message.find('StackTrace:')  
   if pos > -1:  
     return a_message[0:pos]  
     return a_message  
 """ saves file """  
 def saveFile(text, filename, mode):  
   f = open(filename, mode)  
 """ returns a sorted list of files for a given directory , sorted by date  
 def sorted_ls(path):  
   mtime = lambda f: os.stat(os.path.join(path, f)).st_mtime  
   result = list(sorted(os.listdir(path), key=mtime))    
   return result  
 """ keeps only MapGuide Logfiles and filters out all other files """  
 def filterLogfiles(files_in_dir):  
   filtered_files = []  
   for filename in files_in_dir:  
     if filename.endswith(".log") and filename.startswith("Error"):   
   return filtered_files  
 """ processes a single MG log file and simplifies content """  
 def processLogFile(logfile):    
   # counter for line number in log file  
   line_number = 0  
   # counter for number of messages processed  
   counter_messages = 0  
   # counter for messages we exclude from processing  
   counter_messages_ingnored = 0  
   # internal counter  
   count_opening_tag = 0    
   message = ''  
   message_part = ''  
   # encoding utf-8, ascii : returns error message while reading a certain character/bytes  
   with open(logfile, encoding='“latin-1') as a_file:  
     for a_line in a_file:                    
       line_number += 1        
       # all messages start with '<' and first line also contains date&time  
       if '<' in a_line:  
         count_opening_tag +=1  
         #get date and time  
         str_date, str_time = extractdatetime(a_line)  
         a_line = 'DATE_TIME ' + str_date + ' / ' + str_time + '\n'                  
       # we processing the first line of the current message  
       if count_opening_tag == 1:  
         # line number and date/time information in one new line  
         message_part += str(line_number) + '  ' + a_line          
       # we have reached the first line of the following message - now we need tp process the previous message       
       if count_opening_tag == 2:          
         #first we check whether the message can be ignored  
         if isErrorMessagesToExclude(message_part) is False:  
           # we remove the StackTrace details  
           message_part = removeStackTrace(message_part)            
           counter_messages += 1  
           # we wrap the message text in <start><end> tags  
           message += '\n<start>\n'  
           # we add the processed message to the result  
           message += message_part  
           message += '\n<end>'  
           counter_messages_ingnored += 1  
         #as this is the first line of the "next" message already a_line contains the DATE_TIME for it  
         message_part = a_line  
         # reset counter - the current line is the first line of the next message  
         count_opening_tag = 1  
     # last line reached - last message block is not yet fully processed  
     # code from above is repeated here to close the processing of last message in logfile  
     if isErrorMessagesToExclude(message_part) is False:  
       message_part = removeStackTrace(message_part)                  
       message += '\n<start>\n'  
       message += message_part  
       message += '\n<end>'    
       counter_messages += 1  
       counter_messages_ingnored += 1  
     temp = ["File processed: "+logfile, "# of messages:"+str(counter_messages), "# of messages excluded:\t "+str(counter_messages_ingnored) ]  
     print("File processed: "+logfile)  
     print("# of messages:"+str(counter_messages))    
     print("# of messages excluded:\t "+str(counter_messages_ingnored))      
     return message  
 """ converts the newly created log file(s) summary into a list """  
 def convertToList(processedlogfile):  
   with open(processedlogfile, encoding='“latin-1') as a_file:  
     error_message = False  
     list_final = []  
     date_temp =''  
     time_temp = ''  
     error1_temp = ''  
     error2_temp = ''  
     line_counter = 0  
     for a_line in a_file:        
       line_counter += 1  
       if '<start>' in a_line:          
         line_counter = 1   
         error_message = True  
       if '<end>' in a_line:  
         error_message = False  
         line_counter = 0  
         list_temp = [date_temp, time_temp, error1_temp, error2_temp]  
       if error_message:           
         if line_counter == 2:  
           date_temp = a_line[10:20]  
           time_temp = a_line[23:]  
         if line_counter == 3:  
           error1_temp = a_line[5:].strip()  
         if line_counter == 4:  
           error2_temp = a_line[5:].strip()  
   return list_final  
 """ iterates over all relevant files and creates summary"""  
 def processLogfiles(logfile_dir, saveLogFileName, number_of_most_recent_files):     
   # create a new file for the results  
   saveFile('', saveLogFileName, 'w')    
   # get all files from directory with MapGuide logs  
   files_in_directory = sorted_ls(logfile_dir)    
   # filter out all non MapGuide log files  
   files_in_directory = filterLogfiles(files_in_directory)  
   # only process the most recent files   
   if(number_of_most_recent_files > 0):    
     index = ((number_of_most_recent_files ) * -1)  
     files_in_directory = files_in_directory[index:]    
   # interate over relevant log files  
   for filename in files_in_directory:      
       fn = os.path.join(logfile_dir, filename)  
       # process single log file  
       log_file_short = processLogFile(fn)  
       # create header      
       header = "\n\n****************"  
       header += "\n**************** Logfile: " + fn  
       header += "\n****************\n\n"        
       # write to file  
       saveFile(header, saveLogFileName, 'a')  
       saveFile(log_file_short, saveLogFileName, 'a')  
   # all files have been processed and relevant information has been written into one file          
   # now we want to get a summary of logged issues  
   resultList = convertToList(saveLogFileName)      
   resultList is list of lists, lists have 4 items each   
   items 3 and 4 are equal to line 1 and 2 of a message    
   now we just count item 3 and get a Dictionary where frequency of message is key and message itself is value    
   res = (Counter(mysublist[2] for mysublist in resultList))  
   text = "\n\n*********************************************"  
   text += "\n************* Summary ***********************"  
   text += "\n*********************************************\n\n"  
   for message, number in res.items():  
     text += "# " + str(number)+ "\t :: " + message +'\n'    
   # append summary    
   saveFile(text, saveLogFileName, 'a')    
   summ = "\n************* File(s) ***********************\n"  
   for alist in summary_files:  
     summ += "\n".join(alist)+"\n"  
   # append summary    
   saveFile(summ, saveLogFileName, 'a')    
 if __name__ == ("__main__"):  
   """ add any message you want to ignore when processing the log files"""  
   errormessage_to_exclude = [  
                 'Session has expired',   
                 'Resource was not found: Session:',   
                 'Die Sitzung (',   
                 'Error: Authentication failed'                  
   # number of most recent log files to process  
   # 0 - for all files to be processed  
   # can be overwritten when python script is called with parameter  
   number_most_recent_files = 3  
   #if argument is provided we assume its a number  
   if len (sys.argv) == 2 :  
     number_most_recent_files = int(sys.argv[1])  
   # path to MapGuide error log directory       
   logfile_dir1 = '//wsstadt529/logfiles/MapGuide'    
   logfile_dir2 = '//wsstadt516/logfiles/MapGuide'    
   # file name for result - simplified log file:  
   saveLogFileName1 = 'c:/temp/aims_log_processed_529.txt'  
   saveLogFileName2 = 'c:/temp/aims_log_processed_516.txt'  
   # start processing  
   summary_files = []  
   processLogfiles(logfile_dir1, saveLogFileName1, number_most_recent_files)        
   summary_files = []  
   processLogfiles(logfile_dir2, saveLogFileName2, number_most_recent_files)      
   # open file(s) in Editor  

Friday, 17 February 2017

AutoCAD Civil installation causes problem with existing Map installation

If you install AutoCAD Map 2017 and afterwards Civil 2017 exporting a Map drawing (FDO layers) as CAD drawing (command: _MAPTOACAD) might fail.

In particular if a template is used when exporting. Map will perform the export but will not finish the process - the dialog box won't disappear and the process cannot be cancelled. Map still uses CPU resources and needs to be closed forcefully. The drawing will be created and can be opened - but I'm not sure if all elements have been exported or not.
I had to reinstall Map to solve the issue.

Map 2017, SP1

Thursday, 16 February 2017

Map-Plot and plotting in background

Again it took me some time to figure out that Map-Plot and AutoCAD background plotting don't go well together. With Map 2013 we did use Map Plot for plotting maps as PDFs and we did not have an issue with regards to background-plotting although I do not know whether we just had not enabled background-plotting or the behavoir has changed in Map 2017.

Here are a few observations of Map 2017 and Map-Plots:

- Map Plot batch plotting does not work, Map will crash after the second plot and no PDF will be created
- if a single Map-plot is plotted manually (via context menu) an AutoCAD layout is being created and the map correctly rendered but it wont plot to the device (at least as long as a PDF plotter is being used), one needs to plot the AutoCAD layout manually
- when the AutoCAD layout has been created and the layout is plotted the resulting PDF file is sometimes empty, if the layout is plotted a second time the PDF file will be fine (if background-plotting is enabled)
- when the layout has been created and the layout is plotted error messages might appear:
FDO-Verbindung konnte nicht geöffnet werden. (cannot open FDO connection)
ORA-01005:  kein Kennwort angegeben; Anmeldung verweigert. (null password given; logon denied)
WARNUNG: Beim Anwenden von Stilen auf das Element PLT_V_GRID_EXTLINE_TBL sind folgende Probleme aufgetreten:
Objektquelle ist nicht verbunden. (whilst applying styles to element PLT_V_GRID_EXTLINE_TBL the following issue occured: feature source not connected)
and the PDF is created correctly (if background-plotting is enabled)

If you use Map-plots and you run into issues check whether background plotting is enabled or not. If you need batch plotting and you get crashes as well - have a look here.

I also noticed that disabling background-plotting seems to be ignored until Map is restarted,.

Map 2017, SP1

Tuesday, 14 February 2017

MapGuide Logfile - two warnings and what they mean

MapGuide log file contains the following warnings:

<2017-02-14T08:54:36> 4712 Ajax Viewer Administrator
 Warning: An exception occurred in FDO component.
        Beim Abrufen eines Eigenschaftenwerts stimmte der Eigenschaftentyp nicht überein. 
  - MgStylizationUtil.ExceptionTrap() line 813 file StylizationEngine.cpp
the error message translates roughly as 
  "property value and property typ do not match"
The message was caused by the following filter definition:  

W_ID_TYPE  = 1 AND  (   W_ID_TYPE_IP  NULL    OR   NOT  W_ID_TYPE_IP  IN ( '1') )

W_ID_TYPE_IP is numeric but the value is given as string.  
 The second one:
<2017-02-14T12:19:44> 3444 Anonymous
 Warning: An exception occurred in FDO component.
        Die Kennung 'w_id_type' wurde nicht erkannt. 
  - MgStylizationUtil.ExceptionTrap() line 813 file StylizationEngine.cpp  
the error message translates roughly as 
  "property 'w_id_type' not found."

The message was caused by the following filter definition:  

w_id_type  IN ( 11)   AND NOT  ID_SITE_TYPE  IN (2007)

Property names as case-sensitive. The feature class does not contain 'w_id_type' but 'W_ID_TYPE'.

Unfortunately for both warnings no further information is given to where exactly the issues occur. I stumbled across the cause of the first warning by looking for the cause of the second warning. The second warning tells you exactly what the problem is - unfortunately I only realized that after I had spent several hours looking for it. 'w_id_type' is an attribute we use in many feature classes and 71 layers reference the attribute in filter expressions. It was somehow obvious that there is a problem in one of the layers but not in all of them as the warning did not appear all the time. Took me quite some time to figure out what was going on - despite the warning being clear about it.

AIMS 2017


Friday, 3 February 2017

MapGuide - renderMap - transparency

The RenderMap functions takes a MgColor object in order to set the background color of the map.
I wanted to have a transparent background - according to the API docs you can create a MgColor object by providing values for RGB and alpha:

MgColor(int red, int green, int blue, int alpha);

Problem is, the map background wont show up transparent. In order to have the background transparent MgColor needs to be called by passing hexvalues instead:


AIMS 2017