29 Commits

Author SHA1 Message Date
Billy O'Neal
f698b0e8be
Change VM size to Standard_D32ds_v4. For some reason they are cheaper than D16 right now??? (#18953)
Remove no longer used "unstable" scaffolding.
Use ephemeral OS disks for better latency.
Add ability to change whether temp disks are used or explicitly provisioned disks are used; explicitly provisioned disks turned out to be more expensive than expected before.
Disable Windows Updates in the scale set because the machines are very short lived; working on a mechanism of patching the images instead.
2021-07-15 13:47:58 -07:00
Billy O'Neal
957d2024b8
Update to Ubuntu 20.04. (#18892)
* Update Ubuntu to 20.04.

* [tfhe] Disable -Wall -Werror

* [sophus] Disable -Werror.

* Fix typo in deployment of CUDA.

* Add libxxf86vm-dev for opensubdiv.

* [opensubdiv] Fix OpenCL search on Ubuntu 20.04, and document libxxf86vm-dev dependency

* [msgpack11] Disable -Werror.

* [qtwayland, qt5wayland] Add system library notice.

* Add python-is-python3 to unbreak tensorflow, v8, and others

* [graphqlparser] Remove directive rejected by Bison 3.5.1

* [ogdf] Mark as failing in ci.baseline.txt. Issue filed https://github.com/microsoft/vcpkg/issues/18936

* [bde] Mark failing in ci.baseline.txt, issue filed https://github.com/microsoft/vcpkg/issues/18937

* [soem] Patch out -Werror

* Update pool.
2021-07-14 12:46:05 -07:00
Billy O'Neal
7d169b467c
Update pool to get print spooler CVE fix. (#18891) 2021-07-10 16:47:55 -07:00
Billy O'Neal
2ed44b4546
Update VMs and pick up VS2019 16.10 (#18233)
* Cherry-pick https://github.com/microsoft/vcpkg/pull/15598

* Hook deploy-inteloneapi into create-vmss.ps1.

* Add script to resolve https://github.com/microsoft/vcpkg/issues/17521

* Move tls settings deployment to the front and respond to script triggering a reboot.

* Go back to provisioning an extra disk to workaround https://github.com/microsoft/vcpkg/issues/18379

* Disallow public access to blob storage and require TLS 1.2.

* Update Pools.

* Update tool to 2021-06-19

* [simage] Skip simage on uwp platforms as it appears broken by 16.10.

* [tensorflow-cc] Skip because changes in our MacOS hardware broke the port.
2021-06-22 19:16:04 -07:00
nicole mazzuca
e6cabdece5
[ci] Update macOS to 11 (#17376)
* start 2021-04-16 process

Major new things:
* update to macOS Big Sur (11.*)
* switch from VirtualBox to Parallels due to ^ (and also ARM)

Minor new things:
* update from xcode CLT 12 to 12.4

This PR includes new stuff for creating the base box for parallels.
Still to do: using the new box type.

* update the vagrantfile stuff

* link the CI to the new version

* modify how macOS boxes are made

the Vagrantfile for the final VM will only download the azure pipeline stuff

this allows us to keep the versions of brew applications stable

* fix cpus and memory

* add vagrant plugins to installables

* Skip cppgraphqlgen ICE.

* [sdformat6] Remove unneeded include(FindBoost) which causes problems when the system cmake version doesn't match the one deployed by vcpkg.

* switch to dmg for installing osxfuse/sshfs

* Set cores to 11 (leaving 1 thread for host)

Co-authored-by: Billy Robert ONeal III <bion@microsoft.com>
2021-04-29 07:39:04 -07:00
Billy O'Neal
a9b27ed5df
Update VMs, CMake to 3.20.1, CUDA to 11.3, and pwsh to 7.1.3 (#17331)
* Update to CUDA 11.3 on Windows.

* Update PowerShell to 7.1.3.

* Update CUDA to 11.3 on Linux.

* "Explode" VM provisioning scripts for more consistent feedback during deploy. This resolves the deployment script often hanging with no feedback as to why.

* [libdatachannel] Fix use of deprecated result_type typedef.

Submitted upstream as https://github.com/paullouisageneau/libdatachannel/pull/413

* [libvpx] Get the libvpx outputs from the correct place.

(Perhaps VS2019 version 16.10 moved where these are placed? I've been defensive and left an attempt to find from the old location in place.)

* [chromaprint] Support implementations where lrintf is an intrinsic.

* Add provision-entire-image script.

* [cudnn] Disable download-on-the-fly due to licensing concerns.

* Add libnccl to Linux VMs.

* [wangle] Disable x64-windows due to an ICE.

* [cmake] Update cmake to 3.20.1 to avoid https://gitlab.kitware.com/cmake/cmake/-/issues/21571 race

* [libudis86] Fix passing a bogus working directory which fails on CMake 3.20.x

* [dartsim] Disable unit tests, examples, and tutorials, some of which have CMake authoring errors rejected by 3.20.1.

* Add thrust to the cuda installees.

* [tensorflow] Put .bzl in CURRENT_BUILDTREES_DIR to avoid running out of disk space in CI and to respect --clean-after-build.

* [dimcli] Skip port broken by changes in VS2019 project system.

* [upb] Disable an additional warning.

* [libhv] Fix typo DISABLE_PARALLEL => DISABLE_PARALLEL_CONFIGURE

* Update pools
2021-04-26 10:27:45 -07:00
Billy O'Neal
aa60b7efa5
[vcpkg] Download vcpkg.exe rather than building it in bootstrap on Windows. (#15474)
This reduces bootstrap cost for Windows customers, resolving the issue initially submitted as #12502 .

The `toolsrc` tree was extracted to https://github.com/microsoft/vcpkg-tool. `bootstrap.sh` was changed to download the right source tarball, extract, and build it. This was chosen over the previous attempt, a submodule, over concerns of accidentally destroying people's local modifications.
2021-02-04 10:15:44 -08:00
ankurverma85
aa83f4bd2a
[vcpkg] Add x64-windows-static-md to pipeline (#15411)
* Add x64-windows-static-md to pipeline

* Update azure-pipelines.yml

* Update test-modified-ports.ps1

* Replicate skip/fail from x64-windows-static

* Add failures to ci.baseline

* Update ci.baseline.txt

* Update ci.baseline.txt

* Update ci.baseline.txt

* Update ci.baseline.txt

Co-authored-by: ankurv <ankurv@microsoft.com>
2021-01-08 11:55:31 -08:00
Billy O'Neal
229f537c94
[opentracing] Update VMs 2020-12 (#15151)
* Add meson from https://github.com/microsoft/vcpkg/pull/12860/

* Add autoconf-archive from https://github.com/microsoft/vcpkg/pull/13081/

* Add kf5windowsystem libs from https://github.com/microsoft/vcpkg/pull/13467/

* Open the FTP and SFTP ports from https://github.com/microsoft/vcpkg/pull/14412/

* Add libxcb-util0-dev from https://github.com/microsoft/vcpkg/pull/14678/

* Add libasound2-dev from https://github.com/microsoft/vcpkg/pull/14774

* Remove no longer necessary apt-mark calls.

* Update nasm on Linux.

* Fix longstanding bug where Storage was publicly accessible and change to generate SAS token rather than File Share

* Delete no longer necessary azure storage firewall rules.

* Install the newer Windows SDK with the VS installer instead of manually.

* Install the VS2015 and VS2017 compilers.

* Update Powershell-Core to 7.1.0.

* Update source of WDK.

* Update pools.

* [opentracing] Repair arm64-windows failures caused by mojibake in `expected.hpp` and errors in opentracing-cpp's lint for arm64 where it thinks exceptions are disabled when they are enabled.

Fixes:
C:\Dev\vcpkg\buildtrees\opentracing\src\b67575dab0-0250653c81.clean\3rd_party\include\opentracing/expected/expected.hpp(1): warning C4828: The file contains a character starting at offset 0x4a77 that is illegal in the current source character set (codepage 65001).

Fixes:
D:\buildtrees\opentracing\src\b67575dab0-0250653c81.clean\include\opentracing/tracer.h:223:5: error: cannot use 'try' with exceptions disabled [clang-diagnostic-error]
    try {
    ^

* [mmloader] Patch out overrides of CMAKE_C_FLAGS and CMAKE_CXX_FLAGS that inserted /WX.

Note that this port sets /GS-, possibly because it may be for authoring shellcode.

* Bump storage API version to 2020-04-08.
2020-12-22 14:52:58 -08:00
nicole mazzuca
30767175d5
[vcpkg ci] upload diff from clang-format to artifacts (#15141)
* [vcpkg ci] upload diff from clang-format to artifacts

this allows people who do not have access to clang-format to format their code via diff
2020-12-16 13:14:35 -08:00
nicole mazzuca
6b117c9c7e
[vcpkg docs] Check for documentation generation in CI (#14614)
* [vcpkg docs] Change how documenting port functions works

Instead of using `##`, use comment blocks for documentation.

Also, add some minor docs and change RST -> MD
so we actually get docs generated.

* add CI stuff

* regenerate docs

* fix vcpkg_find_acquire_program to not use _execute_process
2020-12-01 13:37:26 -08:00
nicole mazzuca
397d3d7188
[vcpkg osx ci] update 2020-09-28 (#13799)
* [vcpkg osx ci] update to newer bits

these ones should actually work!

* update vagrantfile

* correct scripts, reformat, add docs

* Finish Get-InternalBaseBox, move archives

In order to avoid people maliciously deleting the boxes,
the archives share and file share are in different users now.

* baseline libqcow to be fixed later

* actually check for the mount point

* fix Install-Prerequisites again

* add +x to Get-InternalBaseBox.ps1

additionally, fix some errors
2020-09-29 16:49:49 -07:00
nicole mazzuca
4cbbcbddfd
[vcpkg macos ci] Switch to using our own base boxes, to fix bringing up mac machines (#13619)
* [vcpkg ci:osx] Remove brew install

* add instructions for creating a new vagrant box

* fix the vagrant scripts for the new box

* finish fixing the setup

* [mecab jxrlib] fix ports for CI

mecab needed to use an actual ref that wasn't master,
and jxrlib needed a patch for xcode 12 CLTs.

Additionally, this fixes the mecab version to be a date, the date of the last commit,
since `1.0` is not the correct version (mecab doesn't have released versions)

* [many ports] fix compile with Xcode 12 CLTs

This mostly means fixing errors on implicit-function-declaration,
and removing some Werrors
* alac-decoder
* apr
* argtable2
* arrow
* hyperscan
* mcpp
* minizip
* mosquitto
* stormlib

* [many ports] even more Xcode 12 CLT fixes

* [jxrlib darknet] fix the last ports! (hopefully)

* CRs, plus minor wip changes to osx scripts
2020-09-24 15:15:06 -07:00
Billy O'Neal
8054263f15
[skia, ompl, capnproto] Update VMs for 2020-08-12 (including VS 16.7) (#12884)
Co-authored-by: Nicole Mazzuca <mazzucan@outlook.com>
2020-08-27 12:27:39 -07:00
ras0219
481738beae
[vcpkg] Add vcpkg export to E2E tests. Enable E2E tests on all platforms. (#12198)
* [vcpkg] Add `vcpkg export` to E2E tests. Enable E2E tests on MacOS.

* [vcpkg] Fix export --raw --output-dir=/path/ by changing directory to new export root

Co-authored-by: Robert Schumacher <roschuma@microsoft.com>
Co-authored-by: Billy Robert O'Neal III <bion@microsoft.com>
2020-08-10 10:22:51 -07:00
nicole mazzuca
3871d73349
[vcpkg manifests] fix some issues (#12227)
Fixes #12190
Fixes #12208
Fixes #12234
Fixes #12286
Fixes #12315
Fixes #12186
Fixes #12331
Fixes googleapis/google-cloud-cpp#4487
2020-07-09 12:32:31 -07:00
nicole mazzuca
9606917c81
[vcpkg ci] Update formatting CI (#12314)
* stage checking formatting

this means that the port CI won't run if the formatting failed

* fix invalid names

* add formatting of manifests

* fix dependsOn name

* fix a thing

* CRs

* oops, typo
2020-07-07 14:09:33 -07:00
Billy O'Neal
532db1d8f7
[vcpkg] VM Update (#12176)
* [qt5] Add more dependent Linux packages for https://github.com/microsoft/vcpkg/pull/11596/

* [vcpkg] Add LLVM 10 to Windows CI machines for https://github.com/microsoft/vcpkg/pull/11655/

* [vcpkg] Add missing Param block to InstallLLVM function.

* [vcpkg] Add throw after all Write-Error calls for consistency with https://github.com/microsoft/vcpkg/pull/12172
2020-07-01 10:53:10 -07:00
Billy O'Neal
9f8cc9ee88
[vcpkg] Delete unnecessary tombstone deletion step now that CI rebuilds everything (#12120) 2020-06-25 22:19:17 -07:00
Billy O'Neal
7ced7d721e
[vcpkg] Do not fail just because cleaning tombstones failed in CI. (#12074) 2020-06-24 11:39:47 -07:00
Billy O'Neal
00e0dbdcb1
[vcpkg] Clean tombstones in every build except PR validations (#12000) 2020-06-17 21:31:30 -07:00
Billy O'Neal
be298744f7
[vcpkg] Merge unit test pass into x86-windows. (#11858) 2020-06-11 17:49:22 -07:00
Billy O'Neal
55d6bd1b7d
[vcpkg] [cudnn] [msmpi] [openmpi] Update VMSS (#11365)
Co-authored-by: @JackBoosY 
Co-authored-by: Stefano Sinigardi <stesinigardi@hotmail.com>
Co-authored-by: Voskrese <10104740+Voskrese@users.noreply.github.com>
2020-06-11 11:13:09 -07:00
Billy O'Neal
5504dfa7da
[vcpkg] Turn on tests and PREfast in CI, and fix tests to pass. (#11239)
Co-authored-by: nicole mazzuca <mazzucan@outlook.com>
2020-05-14 12:49:31 -07:00
Billy O'Neal
6920b49047
[vcpkg] [llvm] [mpir] Bump Linux VM memory size and do all operations on the temporary disk. (#11174)
* [vcpkg] Bump Linux VM memory size and do all operations on the temporary disk.

* [llvm] disable optional dependencies to libxml2 and zlib

* [vcpkg] Reduce quotas on the share to something reasonable.

* [mpir] Skip on Linux because it conflicts with libgmp.

Co-authored-by: yurybura <yurybura@gmail.com>
2020-05-10 15:17:27 -07:00
Billy Robert O'Neal III
b8755728ab [vcpkg] Onboard Linux to VMSS, open 'git' port, and switch back to Azure Spot
* Adds scripts to generate scale sets for testing Linux.
    * Note workaround for https://github.com/microsoft/azure-pipelines-agent/pull/2929
* Switches Windows validation to 'Spot' VMs.
* Opens the git port 9418.
* Removes provisioning of the no longer used 'logs' file share.
* Changes Azure region to 'westus2', which is cheaper.
* Adds +x to all the scripts in scripts/azure-pipelines.
* Use 'xml-results' for all platforms instead of 'raw xml results' on Windows.
2020-04-30 21:51:31 -07:00
Billy O'Neal
7341c295b6
[vcpkg] Copy macos pipelines into azure-pipelines.yml 2020-04-28 13:02:39 -07:00
Billy Robert O'Neal III
9256735ad5 WIP 2020-04-22 02:47:03 -07:00
Billy Robert O'Neal III
070a18974b Change supporting infrastructure to use Azure Virtual Machine Scale Sets for vcpkg's PR builds, which should both improve our PR build times and reduce Azure spending by shutting down machines when they aren't being used.
Included is a script that sets up all vcpkg's Azure infrastructure for Windows PR tests, and several updates to baselines. The baseline updates are generally caused by an updated copy of the MSVC++ compiler caused by updating the VMs, but some are caused by missed failures only detected now because this did a cleared out archives directory first.

Some of the build infrastructure isn't what I'd call 'pretty' (e.g. we're split into more scripts and such than I'd like) but this mirrors how our existing PR system works.

It is expected that the existing vcpkg Windows PR system will hate these baseline updates so we'll need to merge this, then remove that (duplicate) workflow immediately afterwards, then delete all the Windows VMs powering the old infrastructure.
2020-04-21 17:12:21 -07:00