How to build Ceylan 0.7 on Windows

Objective

The goal here is to explain to developers how to build the 0.7 version of the Ceylan library for Windows, on Windows (no cross-compilation).

If a user just needs the Ceylan DLL so that she can run an executable depending on the Ceylan library (in this case only Ceylan-0.7-release-multithread.dll is required), then she can download, if available, a Ceylan prebuilt user package from Ceylan File Releases on Sourceforge. If an up-to-date archive is not available, please ask us and we will add it quickly.

If a user just want to develop a program or a library depending on the Ceylan library (hence needing Ceylan installed headers, and preferably Ceylan-0.7-debug-multithread.dll and Ceylan-0.7-debug-multithread.lib) and does not wish to build Ceylan by herself, then she can download, if available, a Ceylan prebuilt developer package from Ceylan File Releases on Sourceforge. If an up-to-date archive is not available, please ask us and we will add it quickly.

If you are still reading, then it is that you really want to build the Ceylan library by yourself, or that you are setting up the tools to run afterwards.

Here the target platform is Windows XP, be it Home or Professional edition. Note that we will not have any link with the .NET framework.

Getting the recommended build toolchain

First of all, one should start from an up-to-date Windows version, regarding the system and Service Packs (use Windows Update) and general tools (ex: Firefox, antivirus/spyware/adware, etc.).

For increased safety, prior versions of build-related tools should be unistalled beforehand (ex: Visual Studio, Platform SDK, .NET framework, DirectX SDK, TortoiseSVN, etc.).

Then the computer should be rebooted and the free space on hard disk should be checked (a few gigabytes are needed on Windows, due to the size of the Microsoft tools).

C++ compiler and linker: Visual C++ 2008 Express Edition

On Windows, for the sake of simplicity, no MSYS/MinGW/Cygwin-based approach for the build toolchain is recommended by this Ceylan guide, instead of which the proposed method uses Visual Studio 2008, Express edition, and more precisely Visual C++ 2008 Express (9.0), which is the IDE and compiler toolchain that we will use here.

It can be downloaded free of charge. All these packages can be used freely, including for commercial use, and they are fully functional for our needs. One may install them with default settings (including default paths), for increased safety. This automated install process will take quite a lot of time.

Downloading the full MSDN documentation as well is not strictly necessary, but helps a lot whenever developing offline: just select the relevant check-box of the Visual installer, and be prepared to download a global package of approximately 340 megabytes. Selecting SQL Server or Silverlight is not necessary at all.

The Windows SDK combines now two formerly separate SDKs: the Platform SDK (PSDK) and the .NET Framework SDK. Moreover, one no longer has to manually integrate the Windows SDK content with VC++ Express: with the Visual Studio 2008 Express versions, one can also build Win32 applications right out of the box. So no more special actions are to be performed here. See this MSDN blog entry for further informations.

Note that you will have to register your Visual Studio 2008 copy, otherwise it will cease to work after 30 days (at least if you do not use the procedure to burn an ISO, not tested here). If you want to install these products on multiple computers, you will have to register them on each targeted computer. To register, simply run your Express Edition and click Help -> Register Product.

Source control: Subversion (SVN)

To retrieve Ceylan sources, one needs a SVN client, we recommend Tortoise SVN. One may use as well the SVN client available with Cygwin, although it cannot handle properly long path names. It is convenient for scripting though.

Note that when using the SVN client provided by Cygwin, a checkout may report a success whereas it actually failed, if it resulted into too long paths. One should use TortoiseSVN after this checkout and perform an update, just to be sure all files are here.

More generally, one should avoid trying to perform any install from a directory already too much nested in the filesystem tree (ex: in a deep directory below My Documents), as some Windows tools may fail in that case.

Basic shell and terminal: Cygwin (and MSYS)

It is not strictly necessary for Ceylan (as opposed to the OSDL case) but, among others, Cygwin can provide a basic UNIX-like environment, that can be used for example to automate the running of the Ceylan test suite. If being not too limited in disk space and having a lot of time to spend on it, one may click on the package setup window to replace All Default by All install to retrieve all Cygwin packages. Numerous packages can be needed, for example by Ceylan's playTests.sh or the LOANI scripts: md5sum, unzip, wget, ping, tar, gunzip, bzip2, svn, make, automake, ssh, etc.

Note that in no way the user application, by using Ceylan, will depend on Cygwin DLL (whose licence would not comply with open-source distribution), we are using Cygwin only for their shell and terminal. MSYS could be used as well.

Text editor

There are plenty of them that are well-designed and open source, for example one may use:

They have some drawbacks though: Syn has a few bugs, does not highlight Erlang syntax (same for Notepad++) and does not seem to be maintained anymore, and we did not managed to register jEdit to handle specific file extensions, due to its Java Jar nature.

LOANI users should stop here: they just needed to build a proper environment for LOANI to run. Among other installations, LOANI will take care of Ceylan by itself. Read below if you are in the process of building Ceylan by yourself.

Getting the Ceylan sources

To download the Ceylan sources once Tortoise SVN is installed, right-click in a window showing the content of the folder in which the sources should be transfered to, choose in the menu SVN Checkout, and enter the URL of Ceylan SVN repository: https://ceylan.svn.sourceforge.net/svnroot/ceylan.

Prefer performing your check-outs in an absolute path rather close to the root of the filesystem (avoid long pathnames), as various Windows tools are surprisingly limited when dealing with long paths.

Configuring the Ceylan library being built

To load the Ceylan project in Visual Express editor, just double-click on the latest Ceylan-0.7.sln file, for example, click on trunk/src/conf/build/visual-express/Ceylan-0.7.sln. Basic settings should be fined for most users, power-users can read the following advanced settings.

Ceylan Core

Each Ceylan version comes with a configuration file dedicated to Windows, trunk/src/code/CeylanConfigForWindows.h, which enables only the basic features provided by Ceylan, i.e. the core features (ex: file management), and the ones that are available under all Windows platforms (ex: networking). Most developers shouldn't have to modify it.

Ceylan Features

How to activate a ported feature

Some other features, such as multithreading, plugin support, etc. are not readily available on a vanilla Windows installation, they are therefore initially deactivated. Not all of them have been ported to Windows, hence some of them cannot be available on this platform for the moment.

To activate a ported feature (most developers will stick to the default settings and won't change anything here), one has first to declare it is requested, by specifying in trunk/src/code/CeylanConfigForWindows.h the relevant symbol (ex: #define CEYLAN_USES_THREADS 1).

Then the relevant additional packages, if any, have to be installed and declared to the IDE. For example, activating the multithreading support would require to have the Windows pthread emulation layer installed beforehand. This feature is currently not ported yet.

More generally, there is at the time of this writing no feature ported to Windows that is not activated by default on Windows builds.

Available ported features

So, beyond the Ceylan core, following features are readily available in vanilla Windows builds:

network support [CEYLAN_USES_NETWORK]: it requires that the Winsock2 headers are available, but they should already be obtained by installing the Windows PSDK.

Features that are currently not available

Other features are currently not ported to Windows, either:

because they do not apply to this platform:
- file descriptor support for non-socket streams [CEYLAN_USES_FILE_DESCRIPTORS]
- symbolic links support [CEYLAN_USES_SYMBOLIC_LINKS]
- advanced file attribute support [CEYLAN_USES_ADVANCED_FILE_ATTRIBUTES]
- file lock support[CEYLAN_USES_FILE_LOCKS]]
- advanced process management support [CEYLAN_USES_ADVANCED_PROCESS_MANAGEMENT]
- signal support [CEYLAN_USES_SIGNALS]
or because they could be ported, but have not been ported yet:
- regular expression support [CEYLAN_USES_REGEX]
- multithreading support [CEYLAN_USES_THREADS]
- plugin support [CEYLAN_USES_PLUGINS]

Generating the Ceylan library

One just has to right-click, in the Solution Explorer panel, on Ceylan-0.7-library, select preferably Regenerate and wait for the termination of the build process. That's it. The operation should notably produce, with no warning nor error, Ceylan-0.7-debug-multithread.dll and Ceylan-0.7-debug-multithread.lib, in trunk/src/code.

Generating the Ceylan tests

One just has to right-click, in the Solution Explorer panel, on the Ceylan-0.7 solution itself, select preferably Generate and wait for the termination of the build process. That's it. It should notably produce, with no warning nor error, the library, if not done already (both .dll and .lib), and the numerous tests, in trunk/test. They can be easily run in a test suite tracking automatically test successes and failures: from a Cygwin terminal, just run playTests.sh, from trunk/test.

That's it! You should have here everything needed to make good use of the Ceylan library. In case of problem, re-read everything carefully and send us a mail should the problem persists. For in-depth technical details about Windows-specific inner workings of Ceylan, one may continue the reading. Most people will stop here, and continue by using by themselves the library.

Solving other Windows-specific pitfalls

Template support with Visual Express

There is an issue with Windows DLL and templates, notably STL ones. See this remarkable article and the advice in the Microsoft knowledge base.

This led us to define pragmas around some template constructs in header files, ex:

/*
 * Takes care of the awful issue of Windows DLL with templates.
 *
 * @see http://ceylan.sourceforge.net/Ceylan-latest/BuildForWindows.html
 * to understand it, and to be aware of the associated risks.
 *
 */
#pragma warning( push )
#pragma warning( disable: 4251 )

	std::list _instances ;

#pragma warning( pop )

Exporting symbols from DLL

During the build on Windows of the Ceylan library, CEYLAN_DLL is expected to be equal to __declspec( dllexport ), ex: CEYLAN_DLL=__declspec( dllexport ).

During the build on Windows of code using the Ceylan library, CEYLAN_DLL is expected to be equal to __declspec( dllimport ), ex: CEYLAN_DLL=__declspec( dllimport )".

These CEYLAN_DLL symbols are defined accordingly on Windows in the C/C++ preprocessor defines panel (library properties), and on UNIX they are replaced by an empty value, thanks to -D command-line preprocessor defines set at configure-time.

When writing code (in the Ceylan library), CEYLAN_DLL is to be specified on:

exported class declarations, ex:

class CEYLAN_DLL WhiteNoiseGenerator: public RandomGenerator { ...

exported non-method functions, ex:

CEYLAN_DLL Ceylan::Float32 Sqrt( Ceylan::Float32 x ) ;

exported symbols, ex:

extern CEYLAN_DLL const SignalNumber BrokenPipe ;

Special management for the extern keyword, with for example:

CEYLAN_EXTERN_TEMPLATE template class CEYLAN_DLL std::list ;

does not seem necessary, according to our tests.

Understanding preprocessor symbols used to detect the Windows platform

There are two of them:

CEYLAN_RUNS_ON_WINDOWS: this symbol is to be used in Ceylan public headers only, both by the Ceylan library itself and by libraries and executables using it, so that compiler-specific and platform-specific issues can be solved (ex: Windows socket descriptors cannot be directly mapped to UNIX file descriptors). This is the only case where total encapsulation of specific datatypes was not enforced, for the sake of simplicity and efficiency. All other cases that would impact the UNIX build have been encapsulated, see Ceylan::System::Thread::SystemSpecificThreadIdentifier for instance. As CEYLAN_RUNS_ON_WINDOWS is not defined on UNIX, on these platforms installed headers are indeed free from configuration-specific informations
CEYLAN_ARCH_WINDOWS: this is a usual CEYLAN_ARCH_* symbols, only used for the Ceylan internal implementation (*.cc), user code does not need to know it

Setup of executables or libraries using the Ceylan library

Each of those Ceylan-dependent binaries must define:

in Configuration Properties -> C++ -> General -> Additional include directories, the path to a directory containing all the Ceylan headers must be specified. This is needed for compiling. Such a directory is provided in the case LOANI is used, otherwise one might copy all the headers in one directory by oneself, or specify them directly from the Ceylan build tree, with something like: "$(CeylanBaseDir)/src/code";"$(CeylanBaseDir)/src/code/generic";"$(CeylanBaseDir)/src/code/interfaces";"$(CeylanBaseDir)/src/code/logs";"$(CeylanBaseDir)/src/code/maths";"$(CeylanBaseDir)/src/code/middleware";"$(CeylanBaseDir)/src/code/modules";"$(CeylanBaseDir)/src/code/network";"$(CeylanBaseDir)/src/code/system
in Configuration Properties-> C++ -> Preprocessor Definitions: "CEYLAN_DLL=__declspec( dllimport )";"CEYLAN_RUNS_ON_WINDOWS=1". This is needed for compiling.
that it references the Ceylan library: in Project Dependencies, check Ceylan-0.7-library. This is needed for linking. Alternately, one can just specify, in the linker settings, an additional dependency, Ceylan-0.7-{debug|release}-mt.lib, and adjust the additional library directories accordingly.

In the linker settings, Generate Debug informations may be set to yes, according to the user needs. For all these settings, one may refer to the Ceylan property sheets, in trunk/src/conf/build/visual-express: CeylanProperties.vsprops for the library, CeylanPropertiesForTests.vsprops for code using it.

Manifests

We do not know for the moment whether manifests solve or are another form of DLL hell, so we keep them on with default settings, it seems to work correctly.

Some informations are available here:

How to add a new Ceylan unit test

An approach to be followed directly with the Visual Studio IDE is described below. However we finally found it preferable to edit manually the *.sln and *.vcproj files. The goal is to have lean and mean project files and to centralize as much as possible information into the shared property sheet. Use the compare-visual-studio-projects.sh script to check that all project files have the same structure.

If wanting to use directly the Visual Studio IDE:

open the Ceylan solution
add a new project to it
choose project type: Visual C++ -> General -> Empty Project
choose project name, ex: testCeylanBasicResourceManager for source in generic/testCeylanBasicResourceManager.cc. Note that the initial project name is not generic-testCeylanBasicResourceManager, so that the associated files and directory names remain short
choose project location in the relevant module, ex: trunk/test/generic for testCeylanBasicResourceManager, and select Solution: Add to solution
remove default directories: Header, Resource, Source Files
add in project an existing element, ex: testCeylanBasicResourceManager.cc, which should lie in the parent directory of the one of this test
the actions below will lead to a test*.vcproj file that will have nevertheless to be stripped-down (as the other files for test projects): the large majority of settings should be removed, as they should be set once for all in the shared test property sheet

set all the test settings. Now the method is utterly simple, it just involves adding to the test project an existing property sheet (use the Property Manager tab), namely trunk/src/conf/build/visual-express/CeylanPropertiesForTests.vsprops, then go to the project properties and:

add a reference to the Ceylan library to link to: in Common Properties -> References, click on Add New Reference and select Ceylan-x.y library
set the output directory: in Configuration Properties -> General -> Output Directory, replaces for output directory $(SolutionDir)$(ConfigurationName) by ..

Previous actions used to be (there are deprecated now, as not centralized in a shared property sheet and hardcoding debug/release settings for example):

In configuration properties,

set General -> Output Directory to ".."

add in C/C++ -> General -> Additional Include Directories:

"$(SolutionDir)code";"$(SolutionDir)code/generic";"$(SolutionDir)code/interfaces";"$(SolutionDir)code/logs";"$(SolutionDir)code/maths";"$(SolutionDir)code/middleware";"$(SolutionDir)code/modules";"$(SolutionDir)code/network";"$(SolutionDir)code/system"

add in C/C++ -> Preprocessor -> Preprocessor Definitions: "CEYLAN_DLL=__declspec( dllimport )"; "CEYLAN_RUNS_ON_WINDOWS=1"
choose in C/C++ -> Code Generation -> Runtime Library: "Multi-threaded Debug DLL (/MDd)". Selecting "Multi-threaded DLL (/MD)", "Multi-threaded Debug (/MTd)" or "Multi-thread (/MT)" leads to a direct crash.
choose in Linker -> Debugging: "Generate Debug Info: Yes (/DEBUG)"

rename in the IDE the project for this test, so that it is prefixed by its module. Ex: testCeylanBasicResourceManager becomes generic-testCeylanBasicResourceManager. It allows to sort the projects in the IDE panel without having the long names in the filesystem
build the project
from a cygwin terminal, go to C:\Documents and Settings\<MyUserName>\My Documents\Ceylan\trunk\src\code\ for example (a symbolic link is useful here), and run for example (check the logs and the return code): ../../test/generic/generic-testCeylanBasicResourceManager.exe --consolePlug; echo $?. Doing so allows to use the latest Ceylan DLL file (Ceylan-0.7-debug-multithread.dll), that is generated in src/code. Another solution to find the Ceylan library is to copy the test executable to the directory in which the DLL is generated (trunk/src/code)
add the project file to SVN repository: for example, right-click on the trunk/test/generic/testCeylanBasicResourceManager directory, and select only testCeylanBasicResourceManager.vcproj

The test*.vcproj files could be generated automatically, but it would not be far more convenient (generate a GUID, register to the solution, etc.).

The recommended way of retrieving the Ceylan library on Windows is to check-out a stable (tag-based) SVN tree: the source archive is a tar.bz2, which is not very convenient.

The recommended way of testing the Ceylan library is to check-out a stable (tag-based) SVN tree and to run the testing suite from a Cygwin terminal, since the script playTests.sh can be run on all supported platforms, including Cygwin-based Windows (note that Cygwin is used only for the shell and UNIX commands it provides; none of its libraries is used by Ceylan): cd trunk/test && ./playTests.sh.

Beware to your firewall, that should detect that the tests use the network and that they change at each new build: you must use your firewall (ex: ZoneAlarm) to authorize these tests, so that the test suite can run fine. Or you may modify playTests.sh so that, even when it detects network availability, it does not pass the --online option to the tests: doing so will prevent the tests from using the Internet, with DNS queries for example (however local client/server instances will still be created and be detected by your firewall).

Note also that if your firewall pops up a window asking the user whether a test executable can access a network interface, then most probably the test will fail because of a time-out: the time needed for the user to answer will be fatal, the test manager will believe that such long durations are abnormal. One thus has to stop and re-run the test suite, and acknowledge the related firewall requests until all needed executables have been granted the right to access the network, so that the test suite can be non-interactive once again.

The ldd tool is lacking on the Windows platform, one may use Dependency Walker instead.

Please react!

If you have information more detailed or more recent than those presented in this document, if you noticed errors, neglects or points insufficiently discussed, drop us a line!

[Top]

Last update: Monday, January 3, 2011