Exposing C++ Libraries with clbind

To expose overloaded functions, you have to cast the function pointer to the correct signature. Suppose the function from the previous example was overloaded. It would then need to be exposed as follows:

def("addThreeNumbers-double", (double(*)(double, double, double)) &addThreeNumbers),

It is important that every function have a unique Lisp name - similar to name mangling in C++. The convention we adopt in Clasp is to append type names to the original name.

clbind can handle member functions of derived classes correctly, provided that they are all exposed. To expose the inheritance structure of C++ classes, expose the base class, and then use the following definition format for the derived class:

class_<CppDerivedClassName, CppBaseClassName>("lisp-class-name")

If multiple inheritance brings in several base classes, use the following format:

class_<CppDerivedClassName, bases<CppBaseClassName1, CppBaseClassName2>>("lisp-class-name")

If a base class is a pure virtual class, i.e. it has only pure virtual functions or its constructor is private, make sure to use the no_default_constructor option when exposing the base class, or you will get a compilation error.

Add constructors to exposed classes with the function def_constructor:

template<typename... Types, typename... Policies>
class_& def_constructor(const string&         name,
                        constructor<Types...> sig,
                        Policies...           policies)

In this function

name

is the name of the constructor that will be visible in clasp. Again, it will be lispified.

sig

is the parameter signature of the C++ constructor. Use a comma-separated parameter-list list in the form constructor<parameter-list>() of all the types used as parameters in the constructor you want to expose.

policies

7.

Exposing member functions is similar to exposing free functions. Call the class_ member function def:

template<class F, class... Policies>
class_& def(char const*       name, 
            F                 fn, 
            Policies...       policies )

Thus exposing a member function is not different from exposing free functions, and the same arguments apply. The exception to this rule is the lambda-list (arguments), which always requires self as its first parameter, which becomes the this argument within the method.

namespace dv {
CL_EXPOSE
void double_vector_startup() {
  using namespace clbind;
  package_ s("DV");
  class_<DoubleVector>(s,"double-vector" )
    .   def_constructor("make-double-vector-with-size",constructor<int>())
    .   def_constructor("make-double-vector-with-values",constructor<const vector<double>&>())
    .   def("fill",&DoubleVector::fill)
    .   def("add",&DoubleVector::add)
    .   def("dot",&DoubleVector::dot)
    .   def("at",&DoubleVector::at)
    .   def("dump",&DoubleVector::dump);

}
};

As Common Lisp does not have the notion of static member functions, exposing them is similar to exposing free functions.

Exposing public member variables works similar to exposing member functions.

Some C++ libraries provide base classes that the library user is meant to subclass to add additional application specific functionality. For this situation Clasp, allows one to create classes in Common Lisp that derive from these C++ classes, and implement methods that may be called from both C++ and Common Lisp code.

An example of this is within Clasp itself - where Clasp exposes Clang's ASTMatchers library. Clasp exposes a facility of the Clang ASTMatcher library that evaluates callbacks on Clang's abstract syntax trees.

To make a class derivable, in place of class_, use derivable_class_, and provide two class template arguments. The first template argument is a class that needs to be provided to clbind and is shown below (in this case DerivableMatchCallback). The second template argument is the original library class that is to be subclassed (in this case clang::ast_matchers::MatchFinder::MatchCallback).

derivable_class_<DerivableMatchCallback, clang::ast_matchers::MatchFinder::MatchCallback> cl_bc(m,"MatchCallback",create_default_constructor);
cl_bc.def("run", &DerivableMatchCallback::default_run)
     .def("onStartOfTranslationUnit", &DerivableMatchCallback::default_onStartOfTranslationUnit)
     .def("onEndOfTranslationUnit", &DerivableMatchCallback::default_onEndOfTranslationUnit);

The DerivableMatchCallback must be defined before the derivable_class_ declaration above.

namespace asttooling {
  class DerivableMatchCallback;                        // (1)
};

template <>                                            // (2)
struct gctools::GCInfo<asttooling::DerivableMatchCallback> {
  static bool constexpr NeedsInitialization = false;
  static bool constexpr NeedsFinalization = false;
  static GCInfo_policy constexpr Policy = unmanaged;   // (3)
};

namespace asttooling {
class DerivableMatchCallback                
    : public clbind::Derivable<clang::ast_matchers::MatchFinder::MatchCallback> {  // (4)
  typedef clang::ast_matchers::MatchFinder::MatchCallback AlienBase;               // (5)
public:

  virtual void run(const clang::ast_matchers::MatchFinder::MatchResult &Result) {  // (6)
    const clang::ast_matchers::MatchFinderMatchResult conv(Result);
    core::T_sp val =  translate::to_object<const clang::ast_matchers::MatchFinderMatchResult &>::convert(conv);
    core::eval::funcall(asttooling::_sym_run, this->asSmartPtr(), val);
  }
  void default_run(const clang::ast_matchers::MatchFinderMatchResult &Result) {    // (7)
    SIMPLE_ERROR(BF("Subclass must implement"));
  };

  virtual void onStartOfTranslationUnit() {                                        // (8)
    printf("%s:%d entered onStartOfTranslationUnit funcalling\n", __FILE__, __LINE__);
    core::eval::funcall(_sym_onStartOfTranslationUnit, this->asSmartPtr());
  }
  void default_onStartOfTranslationUnit() {
    printf("%s:%d entered default_onStartOfTranslationUnit\n", __FILE__, __LINE__);
    this->AlienBase::onStartOfTranslationUnit();
  }

  void describe() {                                                               // (9)
    printf("%s:%d Entered DerivableMatchCallback::describe()\n", __FILE__, __LINE__);
    printf("this=%p  typeid(this)@%p  typeid(this).name=%s\n", this, &typeid(this), typeid(this).name());
    printf("dynamic_cast<void*>(this) = %p\n", dynamic_cast<void *>(this));
    printf("dynamic_cast<core::T_O*>(this) = %p\n", dynamic_cast<core::T_O *>(this));
    printf("typeid(dynamic_cast<core::T_O>*>(this))@%p  typeid.name=%s\n", &typeid(dynamic_cast<core::T_O *>(this)), typeid(dynamic_cast<core::T_O *>(this)).name());
    printf("dynamic_cast<Derivable<clang::ast_matchers::MatchFinder::MatchCallback>*>(this) = %p\n", dynamic_cast<Derivable<clang::ast_matchers::MatchFinder::MatchCallback> *>(this));
    printf("dynamic_cast<DerivableMatchCallback*>(this) = %p\n", dynamic_cast<DerivableMatchCallback *>(this));
    printf("alien pointer = %p\n", this->pointerToAlienWithin());
    printf("_Class: %s\n", _rep_(this->_Class).c_str());
    for (size_t i(0); i < this->numberOfSlots(); ++i) {
      printf("_Slots[%lu]: %s\n", i, _rep_(this->instanceRef(i)).c_str());
    }
  }
  virtual ~DerivableMatchCallback() {                                            // (10)
    // Non trivial dtor  
  }
};
};

1. A forward declaration of the DerivableMatchCallback class for the next piece, GCInfo.
2. A gctools::GCInfo template struct is used to tell the Clasp memory manager how to deal with this class. The NeedsInitialization field tell the memory manager that the DerivableMatchCallback::initialize() function must be called after the object is allocated. The NeedsFinalization field tells the memory manager that the destructor for this class needs to be registered with a finalizer. NeedsFinalization is used for resources like streams and anything that needs cleanup when it is collected. The Policy tell the memory manager how the memory for this object is managed.
   1. Policy = normal means the object is managed by the memory manager: it can be collected and it can be moved.
   2. Policy = collectable_immobile means the object can be collected by the memory manager but it cannot be moved.
   3. Policy = atomic means the object contains no internal pointers (such as strings or integer vectors) and so it can be placed in special memory that doesn't need to be scanned during garbage collection.
   4. Policy = unmanaged means the object will not be automatically collected and it cannot be moved. This is used in special cases like static vectors.
3. Instances of DerivableMatchCallback cannot be moved or automatically collected. They need to be managed manually and carefully so that they do not leak memory.
4. The DerivableMatchCallback inherits from a special template class, clbind::Derivable<clang::ast_matchers::MatchFinder::MatchCallback>. This makes it inherit from both the C++ class MatchCallback and the Clasp Instance_O class, which adds Common Lisp slots to the object.
5. The AlienBase type needs to be defined for derivable_class_ to function.
6. The virtual void run(...) {...} method is defined by clang::ast_matchers::MatchFinder::MatchCallback and we need to overload it. The body of this method translates the argument(s) into Common Lisp types and then invokes a Common Lisp function, core::eval::funcall(asttooling::_sym_run, this->asSmartPtr(), val), that the programmer will define in Common Lisp.
7. The void default_run(...) method is a non-virtual method that is exposed to Common Lisp. If a C++ base class defines the run method, then default_run should call it. If no C++ base class defines the run method, then an error should be signalled, and the programmer must provide a run method in Common Lisp.
8. In this example, the onStartOfTranslationUnit~/~default_onStartOfTranslationUnit are another pair of functions that allow the user to overload an on-start-of-translation-unit method from Common Lisp.
9. A describe method is provided to print internal information about a DerivableMatchCallback instance.
10. The DerivableMatchCallback class should have a destructor.

In the above example, the run~/~default_run pair of methods demonstrate what you need to do to overload the run C++ method from Common Lisp.

In Common Lisp, to create a derived class one would use

(defclass count-match-callback (ast-tooling:match-callback) ()        ;; (1)
  (:metaclass core:derivable-cxx-class))

(core:defvirtual ast-tooling:run ((self count-match-callback) match)  ;; (2)
  (let* ((nodes (ast-tooling:nodes match))
         (id-to-node-map (ast-tooling:idto-node-map nodes))
         (node (gethash :whole id-to-node-map)))
    (advance-match-counter)))

1. The derived class is defined using cl:defclass as usual. It is a subclass of the exposed class. It has the special metaclass :metaclass core:derivable-cxx-class.
2. The core:defvirtual macro is used to overload the asttooling:run method. The overloaded method takes two arguments. The first argument is the instance, self, and the second argument was passed from the C++ run virtual method.

Translating from C++ to Common Lisp objects is done by specializing the templated struct to_object, in the namespace translate, to the type of the C++ object. The static function member function convert of that struct takes an object of that type as a parameter, and returns the Common Lisp object. The translator must appear in the C++ source code before any functions/methods are exposed that need to use it. Translators are incorporated into the template code that clbind generates for each exposed function/method.

Here is an example defining a conversion from std::pair<int,int> into a cl:cons. Once this definition is in place, def can be used to expose C++ functions that take a std::pair<int,int> as a parameter.

namespace translate
{
  template <>
  struct to_object<std::pair<int,int>>
  {
    static core::T_sp convert(std::pair<int,int> arg)
    {
       core::Cons_sp cons = core::Cons_O::create(core::Integer_O::create(arg.first),
                                                 core::Integer_O::create(arg.second));
       return cons;
    }
  };
};

Translating from Common Lisp to C++ objects is done by specializing the templated struct from_object, in the namespace translate, to the C++ object type. A constructor must be provided that takes the Common Lisp object as a parameter, and writes the resulting C++ object into an object called _v. from_object::DeclareType must be defined to be the C++ type in question.

Here is the converse of the above example, converting a Lisp cons into a std::pair<int,int>.

namespace translate
{
  template <>
  struct from_object<std::pair<int,int>>
  {
    typedef std::pair<int,int> DeclareType;                             // (1)
    DeclareType _v;                                    
    from_object(core::T_sp obj)
    {
      if (obj.consp()) {
          this->_v = std::make_pair(core::clasp_to_int(CONS_CAR(obj)),  // (2)
                                    core::clasp_to_int(CONS_CDR(obj)));
      }
      TYPE_ERROR(obj,cl::_sym_Cons_O);                                  // (3)
    }
  };
};

1. DeclareType is used by clbind, and must be defined in to the from_object type.
2. We store the translated result into the _v field. This is so that if the argument is used as a return value from a Lisp function, the value can be recovered from here.
3. If the type of obj doesn't match what this translator handles, then signal a type error that tells the user what types are accepted.

Let's say you have a C++ function that uses reference parameters to output values, like this:

void addMul(int x, int y, int z, int& sum, int& product ) {
  sum = x + y + z;
  product = x * y * z;
}

Common Lisp doesn't have a concept of "pass-by-reference", but it does have multiple return values. The pureOutValue<N> policy tells clbind that a C++ parameter passed by reference should be translated into Lisp multiple return values.

using namespace clbind;
package_ pkg("HELLO-WORLD",{"HW"},{});
pkg.scope.def( "addMul", &addMul, pureOutValue<3>(), pureOutValue<4>() ); // 1

1. The pureOutValue<3>() and pureOutValue<4>() arguments tell clbind that the third and fourth arguments to the addMul function references that are written into but not read from. The argument counting starts at 0. pureOutValue<N>() further says that these values can be passed in uninitialized. When the function returns the values in sum and product, they should be returned as the first and second multiple-return values.

COMMON-LISP-USER> (hw:add-mul 2 3 4)

9
24

The function returns the two values 9 and 24.

In that example, the function had void return type, so the references provided all return values into Lisp. For other return types, clbind uses the returned value as the primary return value into Lisp, and out references provide subsequent return values.

int returnThreeValues(int& second, int& third)
{
  second = 2;
  third = 3;
  return 1;
}
//...
  s.def("returnThreeValues",&returnThreeValues,
        clbind::pureOutValue<0>(),
        clbind::pureOutValue<1>());

COMMON-LISP-USER> (multiple-value-list (hw:return-three-values))

(1 2 3)

In some cases, a reference parameter may be used both to pass values into a C++ function and out of it. For example, let's say you have a C++ function like this:

void addMulRunning(int x, int y, int z, int& sum, int& product ) {
  sum = x + y + z + sum;
  product = x * y * z * product;
}

In this case the outValue<N> policy tells clbind that values will be passed in to these arguments and multiple return values will be returned using these arguments.

using namespace clbind;
package_ pkg("TEACH",{},{});
pkg.scope.def( "addMulRunning", &addMulRunning, outValue<3>(), outValue<4>() );

The outValue<3>() and outValue<4>() arguments tell clbind that the third and fourth arguments to the addMulRunning function are references used as both arguments and return values. The argument counting starts at 0.

COMMON-LISP-USER> (multiple-value-list (hw:add-mul-running 2 3 4 5 6))
(14 144)

As with pureOutValue, for non-void functions, the C++ return value will serve as the primary Lisp return value, and the reference parameters will provide subsequent return values.

adopt<n> is used to instruct clbind that a pointer to an object that is returned by a function is to be managed by Clasp's memory manager. The template argument for adopt can be "result", as in adopt<result>, to indicate the function return value pointer is to be adopted. The template argument can also be an integer 0…N, as in adopt<0>, to indicate that the first argument is a pointer that should be adopted by the memory manager. adopt<i> when i is an integer must be combined with pureOutValue<i>.

A lambda list for the Lisp version of the function to be exposed can be provided as a policy. The most convenient way to do this is to use the _ll user literal. The lambda list is provided as a string, which clbind will parse as a Common Lisp lambda list. For example, "(x y z)"_ll means the Lisp lambda list (x y z).

A documentation string, accessible by cl:documentation, can be provided as a policy. Just pass "documentation string here"_docstring as an argument to def.

Camel case strings are converted to hyphenated names by inserting hyphens into the final name whenever there is a transition between a lower case character and an upper case character.

A few examples:

1. aCamelCaseName -> a-camel-case-name
2. ANameWithANumber42 -> aname-with-anumber42

Examples:

1. a_name_with_underscores -> a-name-with-underscores
2. a_nameWithUnderscores -> a-name-with-underscores

Underscores and camel case can be mixed.