For the GNU Grep package version >= 3.8, the `egrep` command emits:
```
egrep: warning: egrep is obsolescent; using grep -E
```
which makes the `./autogen.sh` script very noisy.
Solution: remove implementation. Frank Hartmann <soundart@gmx.net>,
the author, rejected our request to relicense under MPL2, so we
have to remove his copyrighted work.
Tweetnacl is not security-supported and could not be used in
production environments anyway, the supported backend is libsodium.
init_dependency_root(): Moved to android_build_helper.sh
ANDROID_DEPENDENCIES_DIR: added to specify a storage for dependencies
when downloaded automatically.
ANDROID_BUILD_DIR: Changed the default in ci_build.sh.
ci_build.sh configures these 2 variables to no more use /tmp by default (
except for Android NDK), but a local clone subfolder.
This helps to find downloaded dependencies and generated binaries.
This avoid to have user permission conflicts, or conflicts with 2
different clones of LIBZMQ (for instance).
2 more functions are added:
- android_clone_library():
Similar to android_clone_library(), but fetch a tarball and uncompress it.
So far, only .tar.gz and .tgz archives are supported, but could be enhanced
easily, if needed.
- android_init_dependency_root():
Initialize or check XXX_ROOT, when XXX is a dependency name.
Enhanced version of init_android_root() in build.sh (to be dropped, then).
This version is now also applicable in CZMQ & ZYRE CI builds scripts for Android.
With these changes, this function is now able to build LIBZMQ but also
(almost) CZMQ, ZYRE (and their dependencies) in native or cross compilation
modes.
Last PRs introduced a duplicate code in Android helper file.
Duplicate "Initialisation" sequence can be observed after the helper functions:
########################################################################
# Initialization
...
# (Empty string indicates no failure)
ANDROID_BUILD_FAIL=()
Seen when trying to report last LIBZMQ Android PRs to CZMQ & ZYRE.
Solution: Remove the duplicate code.
Solution: Update image.
Added references to tested versions (Ubuntu, debian) and tested NDK.
Carefull though. Android have changed there NDK naming, between 22 and 23.
Among them, CC, LD, CFLAGS, ... could be useful for other tools.
Reason: Many are calculated as "local" to a particular function, which makes
them unavailable outside this helper function.
Solution: Export more variables (CC, LD, CFLAGS, ...).
New exported variables are prefixed with ANDROID_BUILD_xxx.
This naming is expected to avoid any conflicts/problem with other tools:
- ANDROID_BUILD_CC
- ANDROID_BUILD_LD
- ANDROID_BUILD_CFLAGS
- ...
Solution: Implement the current same kind of mechanism as CZMQ & ZYRE, with enhancement.
Enhancement: When required and if LIBSODIUM is not set, the build tool checks
for an already existing clone, close to LIBZMQ. This mechanism is close
to what is done by LIBZMQ/CZMQ/ZYRE for their dependencies.
Additionnaly: Do not copy current source tree to any 'cache' folder. Use
current folder, but make sure all is cleaned before compilation is launched.
This is a lot safer, when building different clones in parrallel...
Enhancement to be reported to CZMQ/ZYRE via ZProject.
Solution: Add a trace function.
Requires to move the CI build helper code/check/init/... at the end of
helper file.
This new function is available for (and also used by) build.sh.
Output is like:
LIBZMQ (x86_64) - Blah ...
To be reported to CZMQ/ZYRE via ZProject.
* Problem: Android NDK 22 download broken since support of NDK 23.
Due to the PR to support NDK 23.
With NDK 23, the archive file name has changed.
This change is handled by the PR to support NDK23, but now, only 23 and after
are supported.
Also, NDK 23 support introduced a 2nd occurence of the variable
HOST_PLATFORM, with another value. One occurence being exported,
this may confuse next developpers (and it actually confused me).
Solution: Code review
1st occurence is simply dropped, and the algorithm around is changed so that
there is no need of a 'host_platform' kind of stuff.
2nd occurrence is renamed to ANDROID_BUILD_PLATFORM.
Note that 'HOST' is replaced by 'BUILD', as this is the common naming
when talking about the build/compilation machine, when cross compiling.
A dedicated function is created in the helpers, to actually download
the NDK. As this function is made 'public', more checks are performed.
Note:
To be reported in CZMQ & ZYRE, via ZPROJECT, where NDK is downloaded
in 2 different files.
* Problem: Android build environment variables need clarifications.
Reason: All are spread and initialized throughout the code.
Solution: Declare, initialize and document environment variables on top of build.sh.
Side effect: This participates to documentation.
* Problem: Android APP fails to load ZMQ (ARM64 only)
Seen with physical Android devices running ARM64.
Not seen with ARM, X86 or X86_64.
Any Android APP loading ZMQ fails with:
```
[FATAL] Couldn't load library library zmq from jar. Dependency is required!
```
Unpack zyre-android-2.0.1.jar, find libzmq.so for ARM64 and look for missing
symbols:
```
prompt> unzip zyre-android-2.0.1.jar
prompt> cd lib/arm64-v8a
prompt> nm --undefined-only ./libzmq.so | head
U __aarch64_ldadd4_acq
U __aarch64_ldadd4_acq_rel
U __aarch64_ldadd4_rel
U __aarch64_ldadd4_relax
U __aarch64_ldadd8_acq_rel
U __aarch64_ldadd8_relax
U __aarch64_swp8_acq
U __aarch64_swp8_acq_rel
U __aarch64_swp8_rel
U __aarch64_swp8_relax
prompt>
```
Some more symbols are missing, but those are relevant for this issue.
OK.
These symbols are present in libc++_shared, but not exported ...:
```
prompt> nm libc++_shared.so | grep aarch64
00000000000ee6d0 t __aarch64_cas1_acq_rel
00000000000ee7a0 t __aarch64_cas8_acq_rel
00000000001028f0 b __aarch64_have_lse_atomics
00000000000ee840 t __aarch64_ldadd4_acq_rel
00000000000ee810 t __aarch64_ldadd4_rel
00000000000ee8a0 t __aarch64_ldadd8_acq_rel
00000000000ee870 t __aarch64_ldadd8_relax
00000000000ee7e0 t __aarch64_swp8_acq_rel
prompt>
```
Issue seen also on the WEB, with GCC & CLANG:
- https://bugzilla.redhat.com/show_bug.cgi?id=1830472
- cea175b838
- ...
Solution: Add `-mno-outline-atomics` to CXXFLAGS (FLUTTER fix).
Additionaly, had to introduce NDK_NUMBER.
This variable is calculated in `android_build_helper.sh`.
It represents the numeric form of NDK_VERSION:
```
NDK_VERSION --> NDK_NUMBER
android-ndk-r25 --> 2500
android-ndk-r23c --> 2303
android-ndk-r22 --> 2200
android-ndk-r21e --> 2105
... and so on
```
This will help a few other things (NDK download ?).
Scenario:
```
export CURL=libsodium
cd zyre/builds/android
./ci_build.sh
```
Result:
```
Android (arm) build failed for the following reasons:
Found no library named libzmq.so libsodium.so
/home/stephan/git/zproject-android-testing/libzmq/builds/android/prefix/arm/lib/libzmq.so libsodium.so
```
Caused by PR #4437, where the 2nd commit was to fix Sonatype findings.
Lesson learnt: Not always a good idea to add double quotes around variables ...
Solution: Make VERIFY an array, so that Sonatype won't complain.
Seen when someone has to relaunch `ci_build.sh` manually, for troubleshooting,
or experiment(s), ci_build.sh stops as libraries are already built.
Solution: Clean more temporary/build folders before build.
Note:
To be reported in ZYRE/CZMQ via ZProject.
When called from ZYRE/CZMQ, it's difficult to identify which build script is
being executed
Solution: Modify each `echo` trace to show the project name and Android architecture in progress.
Note:
To be reported in ZYRE/CZMQ via ZProject.
* Problem: Android APP fails to load ZMQ since NDK r25.x
With the help of the dump of ./configure options (former PR):
```
LIBZMQ (arm) - ./configure options to build 'LIBZMQ':
> --quiet
> TOOLCHAIN=/tmp/android-ndk-r25/toolchains/llvm/prebuilt/linux-x86_64
> CC=/tmp/android-ndk-r25/toolchains/llvm/prebuilt/linux-x86_64/bin/armv7a-linux-androideabi21-clang
> CXX=/tmp/android-ndk-r25/toolchains/llvm/prebuilt/linux-x86_64/bin/armv7a-linux-androideabi21-clang++
> LD=/tmp/android-ndk-r25/toolchains/llvm/prebuilt/linux-x86_64/bin/ld
> AS=/tmp/android-ndk-r25/toolchains/llvm/prebuilt/linux-x86_64/bin/llvm-as
> AR=/tmp/android-ndk-r25/toolchains/llvm/prebuilt/linux-x86_64/bin/llvm-ar
> RANLIB=/tmp/android-ndk-r25/toolchains/llvm/prebuilt/linux-x86_64/bin/llvm-ranlib
> STRIP=/tmp/android-ndk-r25/toolchains/llvm/prebuilt/linux-x86_64/bin/llvm-strip
> CFLAGS= -D_GNU_SOURCE -D_REENTRANT -D_THREAD_SAFE
> CPPFLAGS= -I/home/stephan/git/zproject-android-testing/libzmq/builds/android/prefix/arm/include
> CXXFLAGS=
> LDFLAGS=-L/home/stephan/git/zproject-android-testing/libzmq/builds/android/prefix/arm/lib -L/tmp/android-ndk-r25/sour\
ces/cxx-stl/llvm-libc++/libs/armeabi-v7a
> LIBS=-lc -ldl -lm -llog -lc++_shared
> PKG_CONFIG_LIBDIR=/tmp/android-ndk-r25/prebuilt/linux-x86_64/lib/pkgconfig
> PKG_CONFIG_PATH=/home/stephan/git/zproject-android-testing/libzmq/builds/android/prefix/arm/lib/pkgconfig
> PKG_CONFIG_SYSROOT_DIR=/tmp/android-ndk-r25/toolchains/llvm/prebuilt/linux-x86_64/sysroot
> PKG_CONFIG_DIR=
> --with-sysroot=/tmp/android-ndk-r25/toolchains/llvm/prebuilt/linux-x86_64/sysroot
> --host=arm-linux-androideabi
> --prefix=/home/stephan/git/zproject-android-testing/libzmq/builds/android/prefix/arm
> --disable-curve
> --without-docs
```
We can observe that LDFLAGS has invalid `-L<path_to_libc++_shared.so>`:
```
-L/tmp/android-ndk-r25/sources/cxx-stl/llvm-libc++/libs/armeabi-v7a
```
This path is no more valid, since NDK r25, where one should use LLVM path.
Ok, once this is fixed, ./configure requires also the path to libc.so.
I don't understand why libc.so is now required, actually, but without this, ./configure fails
to build its conftest.
Solution: Fix invalid LDFLAGS.
Notes:
- To be reported to CZMQ/ZYRE via ZPROJECT.
- Tested successfully with Android Emulator (x86 & x86_64).
- Still need some more work, as execution still fails with physical devices (observed on arm64).
- Introduced `ANDROID_STL`, `ANDROID_STL_ROOT` & `ANDROID_LIBC_ROOT`. All are initialized in `android_build_helper.sh`.
- New mechanism **MUST** be compatible with former NDK versions.
Proposal is to dump ./configure options to have an output like below:
```
LIBZMQ (x86_64) - ./configure options to build 'LIBZMQ':
> --quiet
> TOOLCHAIN=/tmp/android-ndk-r25/toolchains/llvm/prebuilt/linux-x86_64
> CC=/tmp/android-ndk-r25/toolchains/llvm/prebuilt/linux-x86_64/bin/x86_64-linux-android21-clang
> CXX=/tmp/android-ndk-r25/toolchains/llvm/prebuilt/linux-x86_64/bin/x86_64-linux-android21-clang++
> LD=/tmp/android-ndk-r25/toolchains/llvm/prebuilt/linux-x86_64/bin/ld
> AS=/tmp/android-ndk-r25/toolchains/llvm/prebuilt/linux-x86_64/bin/llvm-as
> AR=/tmp/android-ndk-r25/toolchains/llvm/prebuilt/linux-x86_64/bin/llvm-ar
> RANLIB=/tmp/android-ndk-r25/toolchains/llvm/prebuilt/linux-x86_64/bin/llvm-ranlib
> STRIP=/tmp/android-ndk-r25/toolchains/llvm/prebuilt/linux-x86_64/bin/llvm-strip
> CFLAGS= -D_GNU_SOURCE -D_REENTRANT -D_THREAD_SAFE
> CPPFLAGS= -I/builds/CrisalidBox/zproject-android-testing/libzmq/builds/android/prefix/x86_64/include
> CXXFLAGS=
> LDFLAGS=-L/builds/CrisalidBox/zproject-android-testing/libzmq/builds/android/prefix/x86_64/lib -L/tmp/android-ndk-r25/sources/cxx-stl/llvm-libc++/libs/x86_64
> LIBS=-lc -ldl -lm -llog -lc++_shared
> PKG_CONFIG_LIBDIR=/tmp/android-ndk-r25/prebuilt/linux-x86_64/lib/pkgconfig
> PKG_CONFIG_PATH=/builds/CrisalidBox/zproject-android-testing/libzmq/builds/android/prefix/x86_64/lib/pkgconfig
> PKG_CONFIG_SYSROOT_DIR=/tmp/android-ndk-r25/toolchains/llvm/prebuilt/linux-x86_64/sysroot
> PKG_CONFIG_DIR=
> --with-sysroot=/tmp/android-ndk-r25/toolchains/llvm/prebuilt/linux-x86_64/sysroot
> --host=x86_64-linux-android
> --prefix=/builds/CrisalidBox/zproject-android-testing/libzmq/builds/android/prefix/x86_64
> --disable-curve
> --without-docs
```
Note:
This mechanism is currently in use to identify/fix a bug in a recent PR for NDK update.
This mechanism is added before every call of `./configure`.
To be reported to CZMQ/ZYRE (via ZPROJECT).
Solution: Use `set -e`
Note:
To be reported in ZPROJECT, when generating the same for CZMQ & ZYRE.
Note:
`make clean` may fail when Makefile is not yet generated
(case of 1st call of build.sh after git clone).
Additionnaly, cleaned the dependency folder
(report of ZPROJECT generated scripts).
Seen in the code:
```
function android_build_verify_so {
...
for dep_soname do
if [[ $elfoutput != *"library: [${dep_soname}]"* ]]; then
ANDROID_BUILD_FAIL+=("Library ${soname} was expected to be linked to library with soname:")
ANDROID_BUILD_FAIL+=(" ${dep_soname}")
fi
done
```
The `for xxx` syntax is wrong, most probably a typo somewhere.
Solution: Fix & complete the `for xxx` loop.
Tested with & without LIBSODIUM (only available dependent library for LIBZMQ).
Note: The same has to be done in ZPROJECT, for CZMQ & ZYRE.
* Problem: In rare cases, afunix.h doesn't contain a definition for struct sockaddr_un
According to https://github.com/microsoft/vcpkg/issues/21623,
struct sockaddr_un might be unavailable on some machines even afunix.h exists.
For example, on some machines, the content of afunix.h looks like this:
typedef struct _SOCKADDR_UN
{
ADDRESS_FAMILY Family;
wchar_t Path[63];
} SOCKADDR_UN, *PSOCKADDR_UN;
but on other machines, it may looks like this:
#define UNIX_PATH_MAX 108
typedef struct sockaddr_un
{
ADDRESS_FAMILY sun_family;
char sun_path[UNIX_PATH_MAX];
} SOCKADDR_UN, *PSOCKADDR_UN;
Fixes#3949
References:
- [Enable Unix-domain sockets support on Windows](8f3ec75de4)
- [AF_UNIX equivalent for Windows](https://stackoverflow.com/questions/9029174/af-unix-equivalent-for-windows)
- https://devblogs.microsoft.com/commandline/af_unix-comes-to-windows/
zmq_ppoll mostly mimics zmq_poll behavior, except for the added feature of being able to specify a signal mask. Signals in this mask will be blocked during execution of zmq_ppoll. Switching of the process' active signal mask happens atomically with the actual poll call, so that no race conditions can occur. This behavior is useful when one wants to gracefully handle POSIX signals without race conditions. See e.g. the discussion below https://250bpm.com/blog:12/ for an explanation.
Also includes two new tests:
1. test_zmq_ppoll_fd does the same thing as test_zmq_poll_fd, demonstrating backwards compatibility with zmq_poll when used with a default signal mask.
2. test_zmq_ppoll_signals demonstrates the use of zmq_ppoll with a signal mask, blocking out SIGTERM everywhere except in zmq_ppoll, allowing to handle the signal in one place without having to worry about race conditions.
Solution: switch to Github Actions
Travis recently started enforcing credits for OSS projects without
any funding. While it is possible to get free credits, it is a manual
step that involves contacting customer support via email and asking to
add them, every week. While this does not require money, it requires
something far scarcer: volunteers time.
Drop Travis and migrate to Github Actions.
* Problem: No direct support for setting socket priority
Solution: Add ZMQ_PRIORITY socket option, which sets the
SO_PRIORITY socket option on the underlying socket. This
socket option is not supported under Windows. Check option
and set socket option on creation of underlying socket.
* build_ios.sh now build for different archs
* change typo on build_ios.sh
Co-authored-by: Roberto Santacroce Martins <roberto.martins@transferoswiss.ch>
Solution: move all corpora to the zeromq/libzmq-fuzz-corpora repository
for easier handling, and to avoid inflating the size of zeromq/libzmq.
Clone it for the CI on the fly.
* Allow CMAKE to generate ws and wss transports
I guess there is little use of just ws transport, so by default
GnuTLS (and libsodium) are enabled
* cmake libzmq including wss transport (ubuntu 19.10 and ubuntu 19.10 + wsl 1.0)
test_security_fails (libsodium assert !?)
* updated relicense
* make external libs gnutls nss sodium optional
* #ifdef WSS classes and functions, build test*ws* only if correct libs are included, warning if libs not present
* make libsodium optional
* cmake fix tests TIPC transport
* clang-format pointed out a wrongly placed #ifdef
* GnuTLS before 3.6.7 is not safe
* msvc doesn't agree with strlen in array declaration, test_socks now at least compiles on windows
* windows: libsodium build fails, missing include dirs set by env var
* ws transport test only works when GnuTLS is found
* Fixed condition to use NSS / built in SHA1, so that test_ws_transport should now pass, also when GnuTLS is not found
Solution: Modify the build scripts so the user can specify the platform
for which to build, e.g. `./build.sh arm`. This approach originally
significantly reduces the parameters which have to be set before running
the script.
Further the build process is documented in a README now.
Solution: use libbsd by default when available, and the internal implementation
only as a fallback, to take advantage of Linux distros maintenance of the
string libraries.
Solution: Migrate build scripts from Android NDK r11c to r20.
- Standalone toolchain
- Migration from GCC to Clang
- Migration from libgnustl to libc++
- Dropped support for API level below 16 (Android 4.1), previously it was API level 9 (Android 2.3)
- Dropped support for mips architecture
- The build script now start the build of all 4 Android architectures (arm, arm64, x86, x86_64)
