[mapguide-users] MapGuide (actually FDO) 64-bit Linux blockers

[Jackie Ng]

Hi All,

In my MapGuide roadmap announcement I expressed my desire to hopefully see
the "preview" label for MapGuide on 64-bit Linux removed for the next major
release (3.3).

The "preview" label has stuck for the longest time due to some major
blockers in certain FDO providers. In the hopes of being able to pool in
community effort and/or knowledge to solving this problem faster, I'm going
to detail the blockers that currently lie in the following providers:

- SHP Provider
- PostgreSQL Provider
- King Oracle

This is going to be super technical. Be warned.

For the SHP Provider, the blocker is in the SHP spatial indexing code. It
makes heavy use of the C++ "long" and "unsigned long" data types which is
not portable across platform. sizeof(long) == 8 for 64-bit Linux, while
sizeof(long) == 4 for every other platform we target. The incorrect size
assumptions result in corrupted spatial queries which manifest as features
from a SHP source "randomly" disappearing when queried. Another unrelated
problem is that the SHP test suite currently has test failures on both
Windows and Linux meaning we don't have an objective baseline for "it
works". We should get the test suite back into a passing state first before
being able to tackle this problem with confidence.

For the PostgreSQL provider, the blocker is also due to use of "long" and
"unsigned long" in the PostGISDriver code. I've partially fixed this
(https://trac.osgeo.org/fdo/changeset/7621), so that the unit test suite at
least runs to completion with test failures on 64-bit, instead of its
previous behavior of segfaulting out due to incorrect size assumptions in
the PostGISDriver. Now that the unit test suite runs, the majority of the
failures are various overflow errors reported by libpq which once again I
suspect is due to incorrect data type size assumptions. The "blocker" status
can be removed from this provider when the unit test suite is all passing.

For the King Oracle provider, I think it is (and has been) a complete
non-starter on Linux. The provider code has built without errors against the
Linux Oracle Instant Client SDK for the longest time, but admittedly I never
actually tested this provider on Linux, so it's been a "use at your own
risk" type deal for this provider (even on 32-bit linux). Has this provider
ever worked for anyone on Linux? The key blocker I'm seeing here is mainly
this *8 year old* issue (https://trac.osgeo.org/fdo/ticket/562). I'm not
well versed with this particular provider (nor the OCI library), so my
ability to tackle this problem head-on is limited unlike the other FDO
providers. The lack of an easy to setup and run unit test suite on this
provider also hampers development somewhat.

So that is what I think are the blockers that prevent me from considering
MapGuide on 64-bit Linux to be production-ready.

Which is a shame, because a production-ready MapGuide on 64-bit Linux is a
gateway to better docker containers (nobody does and should run 32-bit
applications inside a docker container!), and the reason why the docker
angle is of major interest to me is that we can start to explore some
highly-scalable and load-balanced architectures that are possible with a
reliable MapGuide docker container. But if half the FDO providers don't work
properly, well ... there's isn't really much of a point.

So there's my thoughts on the state of 64-bit Linux support for MapGuide.
Thoughts? Insights? Do you even care about MapGuide working on 64-bit Linux?
Sound off right here on this thread.

- Jackie



--
Sent from: http://osgeo-org.1560.x6.nabble.com/MapGuide-Users-f4182607.html

[Gabriele Monfardini]

Dear Jackie,

we do care about MapGuide on 64-bit Linux.
We have some of them, even in production.

But we just use FDO OGR provider, with gdal compiled with PostgreSQL
support.
We're not using PostgreSQL provider since it need a connection per schema
instead of a connection per db.
We would need many more database connections and to create a lot of
datasources, which is not at all convenient for us.

Anyway we support your effort on this, and thank you for your work!

Best regards,

Gabriele Monfardini

Post by Jackie Ng
Hi All,
In my MapGuide roadmap announcement I expressed my desire to hopefully see
the "preview" label for MapGuide on 64-bit Linux removed for the next major
release (3.3).
The "preview" label has stuck for the longest time due to some major
blockers in certain FDO providers. In the hopes of being able to pool in
community effort and/or knowledge to solving this problem faster, I'm going
- SHP Provider
- PostgreSQL Provider
- King Oracle
This is going to be super technical. Be warned.
For the SHP Provider, the blocker is in the SHP spatial indexing code. It
makes heavy use of the C++ "long" and "unsigned long" data types which is
not portable across platform. sizeof(long) == 8 for 64-bit Linux, while
sizeof(long) == 4 for every other platform we target. The incorrect size
assumptions result in corrupted spatial queries which manifest as features
from a SHP source "randomly" disappearing when queried. Another unrelated
problem is that the SHP test suite currently has test failures on both
Windows and Linux meaning we don't have an objective baseline for "it
works". We should get the test suite back into a passing state first before
being able to tackle this problem with confidence.
For the PostgreSQL provider, the blocker is also due to use of "long" and
"unsigned long" in the PostGISDriver code. I've partially fixed this
(https://trac.osgeo.org/fdo/changeset/7621), so that the unit test suite at
least runs to completion with test failures on 64-bit, instead of its
previous behavior of segfaulting out due to incorrect size assumptions in
the PostGISDriver. Now that the unit test suite runs, the majority of the
failures are various overflow errors reported by libpq which once again I
suspect is due to incorrect data type size assumptions. The "blocker" status
can be removed from this provider when the unit test suite is all passing.
For the King Oracle provider, I think it is (and has been) a complete
non-starter on Linux. The provider code has built without errors against the
Linux Oracle Instant Client SDK for the longest time, but admittedly I never
actually tested this provider on Linux, so it's been a "use at your own
risk" type deal for this provider (even on 32-bit linux). Has this provider
ever worked for anyone on Linux? The key blocker I'm seeing here is mainly
this *8 year old* issue (https://trac.osgeo.org/fdo/ticket/562). I'm not
well versed with this particular provider (nor the OCI library), so my
ability to tackle this problem head-on is limited unlike the other FDO
providers. The lack of an easy to setup and run unit test suite on this
provider also hampers development somewhat.
So that is what I think are the blockers that prevent me from considering
MapGuide on 64-bit Linux to be production-ready.
Which is a shame, because a production-ready MapGuide on 64-bit Linux is a
gateway to better docker containers (nobody does and should run 32-bit
applications inside a docker container!), and the reason why the docker
angle is of major interest to me is that we can start to explore some
highly-scalable and load-balanced architectures that are possible with a
reliable MapGuide docker container. But if half the FDO providers don't work
properly, well ... there's isn't really much of a point.
So there's my thoughts on the state of 64-bit Linux support for MapGuide.
Thoughts? Insights? Do you even care about MapGuide working on 64-bit Linux?
Sound off right here on this thread.
- Jackie
--
Sent from: http://osgeo-org.1560.x6.nabble.com/MapGuide-Users-
f4182607.html
_______________________________________________
mapguide-users mailing list
https://lists.osgeo.org/mailman/listinfo/mapguide-users

[Jackie Ng]

Well that is a somewhat anti-climactic, but practical solution.

Do we just build the OGR provider with Oracle and PostgreSQL driver support
(not hard, we already need the relevant SDKs/libs to build their current
dedicated FDO providers) and just champion that as the preferred solution on
64-bit Linux?

- Jackie



--
Sent from: http://osgeo-org.1560.x6.nabble.com/MapGuide-Users-f4182607.html

[Hans Milling]

You can ditch the Shape provider also, as this is also supported by the OGR
provider.



--
Sent from: http://osgeo-org.1560.x6.nabble.com/MapGuide-Users-f4182607.html

[GordonL]

This would be a great topic to show as a wiki page - all the various OGR
connections (SHP, Oracle, etc) using MapGuide...





--
Sent from: http://osgeo-org.1560.x6.nabble.com/MapGuide-Users-f4182607.html

[Benoit Begin]

While OGR is a neat provider, we did experience certain issues with it that
makes me feel that for some data sources it might not be the best solution.
We would basically get unreliable results when doing queries against it,
which isn't really good when users are trying to run queries on their data
for decision-making purposes.

With OGR on MapInfo (TAB) we would get bad results when applying both a
spatial filter and attribute filter at the same time. On a
MgFeatureQueryOptions we would call up SetSpatialFilter and then SetFilter
and we would get invalid results. The same code that would do the equivalent
of the SetFilter on a C# DataTable would work just fine.

Although if we do start focusing more on making sure the OGR provider is
tested and validated under multiple spatial data types. I do wonder if using
an abstraction layer (OGR) on top of another abstraction layer (FDO), we
might have a harder time debugging issues and getting better performances. I
don't have any metrics at the moment, but I could likely work on some if
it's something that could be useful.



--
Sent from: http://osgeo-org.1560.x6.nabble.com/MapGuide-Users-f4182607.html

[Hans Milling]

Hi Beoit

True, it would be best to use the designated providers instead of the more
generic OGR. But if this is the only way (for a start) to make a 64 bit
Linux build, then I guess people can live with using the OGR provider for
Shape etc. until the other compiler issues has been fixed with the other
providers.
Most of our clients use the OGR for MapInfo TAB files, and we haven't
experiences any issues. Also the performance is must faster with TAB files
than an Oracle database (In general my experience is that these optimized
spatial files are faster than databases).

Best regards
Hans Milling...



--
Sent from: http://osgeo-org.1560.x6.nabble.com/MapGuide-Users-f4182607.html

[Jackie Ng]

The OGR provider for the longest time had no unit tests that can objectively
validate the provider is of some baseline level of quality and stability. We
have a test suite for that now (added when I implemented full FDO expression
support for the provider), albeit it is smaller compared to other FDO
providers. If there are issues with the provider we should have tests in
place to catch them, and now that we have a test suite in place for the
provider, there's no real excuses anymore.

I'm not too concerned about the abstraction-layer-over-abstraction-layer
aspect of the OGR provider. ODBC is an abstraction layer and there's an FDO
provider for that. GDAL provider is another one.

- Jackie



--
Sent from: http://osgeo-org.1560.x6.nabble.com/MapGuide-Users-f4182607.html

[Jackie Ng]

The OGR unit test I just added tells me that you should be able to mix
regular and spatial filters together on this provider without any problems.

https://trac.osgeo.org/fdo/browser/branches/4.1/Providers/OGR/Src/UnitTest/SelectTests.cpp#L367

- Jackie



--
Sent from: http://osgeo-org.1560.x6.nabble.com/MapGuide-Users-f4182607.html

[Hans Milling]

Gordon, there is an official list of vector formats supported by OGR:

http://www.gdal.org/ogr_formats.html

Best regards
Hans Milling...




--
Sent from: http://osgeo-org.1560.x6.nabble.com/MapGuide-Users-f4182607.html