Adding OpenGV as submodule (optional) and use Github actions for windows CI (#1656)

* Adding OpenGV as submodule (optional)

* Remove submodule to migrate to FetchContent

* using FetchContent

* remove empty .gitmodules

* Fixing OpenGV not able to find eigen on windows

* patching opengv to adjust -march=native based on PCL

* fixing patching on windows

* eigen fix

* PR cancel on-going CI builds if new commit is added to PR

* try another approach

* updating patch with some logs

* more debug mesage

* fixing EIGEN_INCLUDE_DIR

* enable rolling_builds on appveyor

* removed rolling_builds appveyor

* fixing eigen cache

* test

* more debug logs

* another try

* cleanup

* updated appveyor to work with fetchcontent

* removed mkdir build (appveyor)

* appveyor: trying ninja to increase CI speed

* removed mkdr

* appveyor: spitting opengv and rtabmap builds to be under 60 min per job

* appveyor: caching dependencies

* using global configuration

* using baked image

* fixing wget in ps

* fixing not support for

* pip error

* reverted pip install

* fixed realsense cache

* fixing multi step build

* removing baked image

* typo

* Windows: Converted appveyor to github actions

* updated boost version

* fixing boost

* udpated  boost config

* boost...

* platform_version

* install boost directly

* silent boost install

* very silent boost

* fixing ls

* added boost install dir

* showing boost install dir

* moved windows dependencies in external action file

* explicitly save boost cache to same time on iterations

* pip install gdown

* update

* caching more deps

* caching all depts

* removed explicit boost cache save

* mscv 14.0

* forcing building visual studio 14 2015

* installing v12 in 2022 instead

* setup cmd prompt

* init right toolset

* fixing system version for opengv compilation error

* fixing package and artifact

* CMAKE_VS_WINDOWS_TARGET_PLATFORM_VERSION

* removing cmd

* find manually psapi.lib

* resolve

* refactor psapi env

* forcing 10.0.19041.0

* using windows=2022 runner instead

* added vcpkg

* commenting windows-latest for now

* trying vcpkg instead

* search path

* updated path

* moved json

* removed deleted file for git

* adding a real version

* using version-string

* cleanup vcpkg

* building artifacts with vcpkg

* removed old windows build approach, added custom dependencies (gtsam, libpointmatcher) to vcpkg build

* udpated vcpkg version for issue https://github.com/microsoft/vcpkg/pull/49103

* try with minimal dep first

* disable opengv for now

* try vcpkg single compilation

* space

* updated link

* fixed qupote

* fixing cache name

* adding debug folder

* adjust path

* adjust path

* providing vcpkg binaries instead

* ident

* added triplet

* added BOOST_ROOT

* boost root

* boost root

* fixed path

* adding eigen headers

* try path

* cmake prefix path

* boost timer def

* libnabo fixes

* pointmatcher prefix

* disabling pointmathcer tests

* updated cmake parameters

* changed how file is downloaded

* trying curl instead

* puttoing backe InvokeWeb because it is a dropbox issue

* skipping optional deps for now

* ficing env variable

* triplet

* installing triplets

* not overriding default vcpkg env variables

* missing path

* manifet install off

* that was working locally

* missing protobuf path

* fixing tiff not found

* fixing vtk not found

* more vtk fixes

* another thy

* changing download url

* updated url

* try

* shoudl work now

* protobuf exe

* readding tiff

* explicit vcpkg installed folder

* missing commands

* try without tiff

* ficing vtk comple path

* -DPSAPI_LIBRARIES=Psapi.lib

* quoting

* fixing psapi required

* Set up MSVC Developer Command Prompt

* disabling pckaging for now

* openni.ini

* renabling packing

* Added stripped deps

* fix name

* updated binaries

* format

* updated opengv eigen path

* added tbb dep

* updatd archive name with vs version

* updated archive in action

* fixing patch error

* corrupted

* updated gtsam version / vcpkg

* removed appveyor. Set internal opengv build disabled by defaut (because build can be very long on some machines), will enable it inside the ros release branches instead.

* updated opengv patch

* gtsam mkl dep

* updated vcpkg binaries

* updated patches

* removed mkl dep

* missing eigen in gtsam dep

* disabling gtsam till we find a compatible version locally

* all working locally!

* removed ninja

* working python calls

* fixed hard symlink for python3.dll

* fixed qt missing png, fixed opengv not finding eigen with config, fixed python3.dll missing

* fixed flaoting dockwidget on start, removed cmd line window when launching bundled app

* use sub-packages cudnn

* Added cuda dev workflow

* fixed archive name

* Updated deps with pytorch cuda

* updating ci PATH

* fixing ci build without torch

* rename cuda artifacts

* cache cuda, add job to test internal opengv build

* Updated output artifacts zip name

* windows package: only zip on pull request

* Change USE_INTERNAL_OPENGV to BUILD_OPENGV

* use use-github-cache

* updated artifacts path

corelib/src/CMakeLists.txt

@@ -791,37 +791,14 @@ IF(CUVSLAM_FOUND)
 ENDIF(CUVSLAM_FOUND)
 
 IF(GTSAM_FOUND)
-	# Make sure GTSAM is built with system Eigen, not the included one in its package
-	IF(GTSAM_INCLUDE_DIR)
-		SET(INCLUDE_DIRS 
-			${INCLUDE_DIRS} 
-			${GTSAM_INCLUDE_DIR} 
-		)
-	ELSE()
-		SET(INCLUDE_DIRS 
-			${INCLUDE_DIRS} 
-			${GTSAM_INCLUDE_DIRS}
-		)
-	ENDIF()
 	SET(SRC_FILES
     	${SRC_FILES}
 		optimizer/gtsam/GravityFactor.cpp
 	)
-	IF(WIN32)
-		# GTSAM should be built in STATIC on Windows to avoid "error C2338: THIS_METHOD_IS_ONLY_FOR_1x1_EXPRESSIONS" when building GTSAM
-		add_definitions("-DGTSAM_IMPORT_STATIC") 
-	ENDIF(WIN32)
 	SET(LIBRARIES
 		${LIBRARIES}
-		gtsam # Windows: Place static libs at the end
+		gtsam
 	)
-	IF(WIN32)
-		#explicitly add metis target on windows (after gtsam target)
-		SET(LIBRARIES
-			${LIBRARIES}
-			metis
-		)
-	ENDIF(WIN32)
 ENDIF(GTSAM_FOUND)
 
 IF(WITH_MADGWICK)

corelib/src/python/PyMatcher.cpp

@@ -228,16 +228,32 @@ std::vector<cv::DMatch> PyMatcher::match(
 				int len2 = PyArray_SHAPE(np_ret)[1];
 				int type = PyArray_TYPE(np_ret);
 				UDEBUG("Matches array %dx%d (type=%d)", len1, len2, type);
-				UASSERT_MSG(type == NPY_LONG || type == NPY_INT, uFormat("Returned matches should type INT=5 or LONG=7, received type=%d", type).c_str());
-				if(type == NPY_LONG)
+				UASSERT_MSG(type == NPY_INT32 || type == NPY_UINT32 || type == NPY_INT64 || type == NPY_UINT64, uFormat("Returned matches should type INT32=%d UINT32=%d, INT64=%d or UINT64=%d, received type=%d", NPY_INT, NPY_UINT32, NPY_INT64, NPY_UINT64, type).c_str());
+				if(type == NPY_UINT64)
 				{
-					long* c_out = reinterpret_cast<long*>(PyArray_DATA(np_ret));
+					long long* c_out = reinterpret_cast<long long*>(PyArray_DATA(np_ret));
 					for (int i = 0; i < len1*len2; i+=2)
 					{
 						matches.push_back(cv::DMatch(c_out[i], c_out[i+1], 0));
 					}
 				}
-				else // INT
+				if(type == NPY_INT64)
+				{
+					unsigned long long* c_out = reinterpret_cast<unsigned long long*>(PyArray_DATA(np_ret));
+					for (int i = 0; i < len1*len2; i+=2)
+					{
+						matches.push_back(cv::DMatch(c_out[i], c_out[i+1], 0));
+					}
+				}
+				else if(type == NPY_UINT32)
+				{
+					unsigned int* c_out = reinterpret_cast<unsigned int*>(PyArray_DATA(np_ret));
+					for (int i = 0; i < len1*len2; i+=2)
+					{
+						matches.push_back(cv::DMatch(c_out[i], c_out[i+1], 0));
+					}
+				}
+				else // NPY_INT
 				{
 					int* c_out = reinterpret_cast<int*>(PyArray_DATA(np_ret));
 					for (int i = 0; i < len1*len2; i+=2)

corelib/src/python/PythonInterface.cpp

@@ -9,6 +9,7 @@
 #include <rtabmap/utilite/ULogger.h>
 #include <rtabmap/utilite/UThread.h>
 #include <pybind11/embed.h>
+#include <filesystem>
 
 namespace rtabmap {
 
@@ -16,6 +17,12 @@ PythonInterface::PythonInterface()
 {
 	UINFO("Initialize python interpreter");
 	guard_ = new pybind11::scoped_interpreter();
+
+	// Tell Python to look in this directory for DLLs
+	std::string exe_dir = std::filesystem::current_path().string();
+    pybind11::module_ os = pybind11::module_::import("os");
+    os.attr("add_dll_directory")(exe_dir);
+
 	pybind11::module::import("threading");
 	release_ = new pybind11::gil_scoped_release();
 }

corelib/src/python/rtabmap_superglue.py

@@ -38,7 +38,6 @@ def init(descriptorDim, matchThreshold, iterations, cuda, model):
     global superglue
     superglue = SuperGlue(config.get('superglue', {})).eval().to(device)
 
-
 def match(kptsFrom, kptsTo, scoresFrom, scoresTo, descriptorsFrom, descriptorsTo, imageWidth, imageHeight):
     #print("SuperGlue python match()")
     global device
@@ -77,6 +76,8 @@ def match(kptsFrom, kptsTo, scoresFrom, scoresTo, descriptorsFrom, descriptorsTo
        
     matchesArray = np.stack((matchesFrom, matchesTo), axis=1);
     
+    # rtabmap expects format:
+    #   matches: array Nx2 (type=9 or uint64)
     return matchesArray
 
 

corelib/src/python/rtabmap_superpoint.py

@@ -8,6 +8,7 @@
 import random
 import numpy as np
 import torch
+import os
 
 #import sys
 #import os
@@ -21,15 +22,19 @@ torch.set_grad_enabled(False)
 device = 'cpu'
 superpoint = []
 
+script_dir = os.path.dirname(os.path.abspath(__file__))
+
 def init(cuda):
     #print("SuperPoint python init()")
     
     global device
     device = 'cuda' if torch.cuda.is_available() and cuda else 'cpu'
     
+    weights_abs_path = os.path.join(script_dir, "superpoint_v1.pth")
+    
     # This class runs the SuperPoint network and processes its outputs.
     global superpoint
-    superpoint = SuperPointFrontend(weights_path="superpoint_v1.pth",
+    superpoint = SuperPointFrontend(weights_path=weights_abs_path,
                           nms_dist=4,
                           conf_thresh=0.015,
                           nn_thresh=1,
@@ -47,10 +52,14 @@ def detect(imageBuffer):
     # use copy to make sure memory is correctly re-ordered
     pts = np.float32(np.transpose(pts)).copy()
     desc = np.float32(np.transpose(desc)).copy()
+    
+    # rtabmap expects format:
+    #   pts:         array Nx3 (type=11 or float)
+    #   descriptors: array NxDIM 35x256 (type=11 or float)
     return pts, desc
 
 
 if __name__ == '__main__':
     #test
-    init(True)
+    init(False)
     detect(np.random.rand(640,480)*255)

corelib/src/python/rtabmap_trace_superpoint.py

@@ -0,0 +1,13 @@
+import os
+import sys
+from pathlib import Path
+
+import torch
+import torchvision
+from demo_superpoint  import SuperPointNet
+model = SuperPointNet()
+model.load_state_dict(torch.load("superpoint_v1.pth"))
+model.eval()
+example = torch.rand(1, 1, 640, 480)
+traced_script_module = torch.jit.trace(model, example, check_trace=False)
+traced_script_module.save("superpoint_v1.pt")