Ports Overlay partial implementation (#6981)

* Ports Overlay feature spec

* Ports Overlay implementation

* [--overlay-ports] Refactor handling of additional paths

* Code cleanup

* [--overlay-ports] Add help

* [depend-info] Support --overlay-ports

* Add method to load all ports using PathsPortFileProvider

* Make PortFileProvider::load_all_control_files() const

* Remove unused code

* [vcpkg] Avoid double-load of source control file between Build::perform_and_exit and Build::perform_and_exit_ex

* [vcpkg] Clang format

* [vcpkg] Fixup build failure introduced in b069ceb2f231

* Report errors from Paragraphs::try_load_port()

docs/specifications/ports-overlay.md

@@ -0,0 +1,178 @@
+# Ports Overlay (Jun 19, 2019)
+
+
+## 1. Motivation
+
+### A. Allow users to override ports with alternate versions
+
+It's a common scenario for `vcpkg` users to keep specific versions of libraries to use in their own projects. The current recommendation for users is to fork `vcpkg`'s repository and create tags for commits containing the specific versions of the ports they want to use.
+
+This proposal adds an alternative to solve this problem. By allowing `vcpkg` users to specify additional locations in their file system containing ports for:
+
+  * older or newer versions of libraries,
+  * modified libraries, or
+  * libraries not available in `vcpkg`.
+
+These locations will be searched when resolving port names during package installation, and override ports in `<vcpkg-root>/ports`.
+
+### B. Allow users to keep unmodified upstream ports
+
+Users will be able to keep unmodified versions of the ports shipped with `vcpkg` and update them via `vcpkg update` and `vcpkg upgrade` without having to solve merge conflicts.
+
+
+## 2. Other design concerns
+
+* Allow a set of `vcpkg` commands to optionally accept additional paths to be used when searching for ports.
+* Additional paths must take precedence when resolving names of ports to install.
+* Allow users to specify multiple additional paths.
+* Provide a simple disambiguation mechanism to resolve ambiguous port names.
+* After resolving a port name, the installation process has to work the same as for ports shipped by `vcpkg`.
+* This **DOES NOT ENABLE MULTIPLE VERSIONS** of a same library to be **INSTALLED SIDE-BY-SIDE**.
+
+
+## 3. Proposed solution
+
+This document proposes allowing additional locations to search for ports during package installation that will override and complement the set of ports provided by `vcpkg` (ports under the `<vcpkg_root>/ports` directory).`
+
+A new option `--overlay-ports` will be added to the `vcpkg install`, `vcpkg update`, `vcpkg upgrade`, `vcpkg export`, and `vcpkg depend-info` commands to specify additional paths containing ports. 
+
+From a user experience perspective, a user expresses interest in adding additional lookup locations by passing the `--overlay-ports` option followed by a path to:
+
+* an individual port (directory containing a `CONTROL` file),
+  * `vcpkg install sqlite3 --overlay-ports="C:\custom-ports\sqlite3"`
+
+* a directory containing ports,
+  * `vcpkg install sqlite3 --overlay-ports=\\share\org\custom-ports`
+
+* a file listing paths to the former two.
+  * `vcpkg install sqlite3 --overlay-ports=..\port-repos.txt`
+
+    _port-repos.txt_
+    
+    ```
+    .\experimental-ports\sqlite3
+    C:\custom-ports
+    \\share\team\custom-ports
+    \\share\org\custom-ports
+    ```
+    *Relative paths inside this file are resolved relatively to the file's location. In this case a `experimental-ports` directory should exist at the same level as the `port-repos.txt` file.*
+
+_NOTE: It is not the goal of this document to discuss library versioning or project dependency management solutions, which require the ability to install multiple versions of a same library side-by-side._ 
+
+### Multiple additional paths 
+
+Users can provide multiple additional paths by repeating the `--overlay-ports` option.
+
+```
+vcpkg install sqlite3 
+    --overlay-ports="..\experimental-ports\sqlite3" 
+    --overlay-ports="C:\custom-ports" 
+    --overlay-ports="\\share\team\custom-ports
+```
+
+### Overlaying ports
+
+Port name resolution follows the order in which additional paths are specified, with the first match being selected for installation, and falling back to `<vcpkg-root>/ports` if the port is not found in any of the additional paths.
+
+No effort is made to compare version numbers inside the `CONTROL` files, or to determine which port contains newer or older files.
+
+### Examples
+
+Given the following directory structure:
+
+  ```
+  team-ports/
+  |-- sqlite3/
+  |---- CONTROL
+  |-- rapidjson/
+  |---- CONTROL
+  |-- curl/
+  |---- CONTROL
+
+  my-ports/
+  |-- sqlite3/
+  |---- CONTROL
+  |-- rapidjson/
+  |---- CONTROL
+
+  vcpkg
+  |-- ports/
+  |---- <upstream ports>
+  |-- vcpkg.exe
+  |-- preferred-ports.txt
+  ```
+* #### Example #1:
+
+  Running:
+
+  ```
+  vcpkg/vcpkg.exe install sqlite3 --overlay-ports=my-ports --overlay-ports=team-ports
+  ```
+
+  Results in `my-ports/sqlite3` getting installed as that location appears first in the command line arguments.
+
+* #### Example #2:
+  
+  A specific version of a port can be given priority by adding its path first in the list of arguments:
+
+  ```
+  vcpkg/vcpkg.exe install sqlite3 rapidjson curl 
+      --overlay-ports=my-ports/rapidjson 
+      --overlay-ports=vcpkg/ports/curl
+      --overlay-ports=team-ports
+  ```
+
+  Installs:
+    * `sqlite3` from `team-ports/sqlite3`
+    * `rapidjson` from `my-ports/rapidjson`
+    * `curl` from `vcpkg/ports/curl`
+
+* #### Example #3:
+
+  Given the content of `preferred-ports.txt` as:
+
+  ```
+  ./ports/curl
+  /my-ports/rapidjson
+  /team-ports
+  ```
+
+  A location can be appended or prepended to those included in `preferred-ports.txt` via the command line, like this:
+
+  ```
+  vcpkg/vcpkg.exe install sqlite3 curl --overlay-ports=my-ports --overlay-ports=vcpkg/preferred-ports.txt
+  ```
+
+  Which results in `my-ports/sqlite3` and `vcpkg/ports/curl` getting installed.
+
+
+## 4. Proposed User experience
+
+### i. User wants to preserve an older version of a port
+
+Developer Alice and her team use `vcpkg` to acquire **OpenCV** and some other packages. She has even contributed many patches to add features to the **OpenCV 3** port in `vcpkg`. But, one day, she notices that a PR to update **OpenCV** to the next major version has been merged. 
+
+Alice wants to update some packages available in `vcpkg`. Unfortunately, updating her project to use the latest **OpenCV** is not immediately possible. 
+
+Alice creates a private GitHub repository and checks in the set of ports that she wants to preserve. Then provides her teammates with the link to clone her private ports repository.
+
+```
+mkdir vcpkg-custom-ports
+cd vcpkg-custom-ports
+git init 
+cp -r %VCPKG_ROOT%/ports/opencv .
+git add .
+git commit -m "[opencv] Add OpenCV 3 port"
+git remote add origin https://github.com/<Alice's GitHub username>/vcpkg-custom-ports.git
+git push -u origin master
+```
+
+Now her team is able to use: 
+
+```
+git clone https://github.com/<Alice's GitHub username>/vcpkg-custom-ports.git
+vcpkg update --overlay-ports=./vcpkg-custom-ports
+vcpkg upgrade --no-dry-run --overlay-ports=./vcpkg-custom-ports
+``` 
+
+to upgrade their packages and preserve the old version of **OpenCV** they require.

toolsrc/include/vcpkg/build.h

@@ -20,7 +20,7 @@ namespace vcpkg::Build
     namespace Command
     {
         void perform_and_exit_ex(const FullPackageSpec& full_spec,
-                                 const fs::path& port_dir,
+                                 const SourceControlFileLocation& scfl,
                                  const ParsedArguments& options,
                                  const VcpkgPaths& paths);
 

toolsrc/include/vcpkg/dependencies.h

@@ -48,7 +48,7 @@ namespace vcpkg::Dependencies
                           const RequestType& request_type);
 
         InstallPlanAction(const PackageSpec& spec,
-                          const SourceControlFile& scf,
+                          const SourceControlFileLocation& scfl,
                           const std::set<std::string>& features,
                           const RequestType& request_type,
                           std::vector<PackageSpec>&& dependencies);
@@ -57,7 +57,7 @@ namespace vcpkg::Dependencies
 
         PackageSpec spec;
 
-        Optional<const SourceControlFile&> source_control_file;
+        Optional<const SourceControlFileLocation&> source_control_file_location;
         Optional<InstalledPackageView> installed_package;
 
         InstallPlanType plan_type;
@@ -129,26 +129,31 @@ namespace vcpkg::Dependencies
 
     struct PortFileProvider
     {
-        virtual Optional<const SourceControlFile&> get_control_file(const std::string& src_name) const = 0;
+        virtual Optional<const SourceControlFileLocation&> get_control_file(const std::string& src_name) const = 0;
+        virtual std::vector<const SourceControlFileLocation*> load_all_control_files() const = 0;
     };
 
     struct MapPortFileProvider : Util::ResourceBase, PortFileProvider
     {
-        explicit MapPortFileProvider(const std::unordered_map<std::string, SourceControlFile>& map);
-        Optional<const SourceControlFile&> get_control_file(const std::string& src_name) const override;
+        explicit MapPortFileProvider(const std::unordered_map<std::string, SourceControlFileLocation>& map);
+        Optional<const SourceControlFileLocation&> get_control_file(const std::string& src_name) const override;
+        std::vector<const SourceControlFileLocation*> load_all_control_files() const override;
 
     private:
-        const std::unordered_map<std::string, SourceControlFile>& ports;
+        const std::unordered_map<std::string, SourceControlFileLocation>& ports;
     };
 
     struct PathsPortFileProvider : Util::ResourceBase, PortFileProvider
     {
-        explicit PathsPortFileProvider(const VcpkgPaths& paths);
-        Optional<const SourceControlFile&> get_control_file(const std::string& src_name) const override;
+        explicit PathsPortFileProvider(const vcpkg::VcpkgPaths& paths, 
+                                       const std::vector<std::string>* ports_dirs_paths);
+        Optional<const SourceControlFileLocation&> get_control_file(const std::string& src_name) const override;
+        std::vector<const SourceControlFileLocation*> load_all_control_files() const override;
 
     private:
-        const VcpkgPaths& ports;
-        mutable std::unordered_map<std::string, SourceControlFile> cache;
+        Files::Filesystem& filesystem;
+        std::vector<fs::path> ports_dirs;
+        mutable std::unordered_map<std::string, SourceControlFileLocation> cache;
     };
 
     struct ClusterGraph;
@@ -181,7 +186,7 @@ namespace vcpkg::Dependencies
     std::vector<ExportPlanAction> create_export_plan(const std::vector<PackageSpec>& specs,
                                                      const StatusParagraphs& status_db);
 
-    std::vector<AnyAction> create_feature_install_plan(const std::unordered_map<std::string, SourceControlFile>& map,
+    std::vector<AnyAction> create_feature_install_plan(const std::unordered_map<std::string, SourceControlFileLocation>& map,
                                                        const std::vector<FeatureSpec>& specs,
                                                        const StatusParagraphs& status_db);
 

toolsrc/include/vcpkg/postbuildlint.h

@@ -9,5 +9,6 @@ namespace vcpkg::PostBuildLint
     size_t perform_all_checks(const PackageSpec& spec,
                               const VcpkgPaths& paths,
                               const Build::PreBuildInfo& pre_build_info,
-                              const Build::BuildInfo& build_info);
+                              const Build::BuildInfo& build_info,
+                              const fs::path& port_dir);
 }

toolsrc/include/vcpkg/sourceparagraph.h

@@ -68,6 +68,15 @@ namespace vcpkg
         Optional<const FeatureParagraph&> find_feature(const std::string& featurename) const;
     };
 
+    /// <summary>
+    /// Full metadata of a package: core and other features. As well as the location the SourceControlFile was loaded from.
+    /// </summary>
+    struct SourceControlFileLocation
+    {
+        std::unique_ptr<SourceControlFile> source_control_file;
+        fs::path source_location;
+    };
+
     void print_error_message(Span<const std::unique_ptr<Parse::ParseControlErrorInfo>> error_info_list);
     inline void print_error_message(const std::unique_ptr<Parse::ParseControlErrorInfo>& error_info_list)
     {

toolsrc/include/vcpkg/vcpkgcmdarguments.h

@@ -15,6 +15,7 @@ namespace vcpkg
     {
         std::unordered_set<std::string> switches;
         std::unordered_map<std::string, std::string> settings;
+        std::unordered_map<std::string, std::vector<std::string>> multisettings;
     };
 
     struct VcpkgPaths;
@@ -41,10 +42,22 @@ namespace vcpkg
         StringLiteral short_help_text;
     };
 
+    struct CommandMultiSetting
+    {
+        constexpr CommandMultiSetting(const StringLiteral& name, const StringLiteral& short_help_text)
+            : name(name), short_help_text(short_help_text)
+        {
+        }
+
+        StringLiteral name;
+        StringLiteral short_help_text;
+    };
+
     struct CommandOptionsStructure
     {
         Span<const CommandSwitch> switches;
         Span<const CommandSetting> settings;
+        Span<const CommandMultiSetting> multisettings;
     };
 
     struct CommandStructure
@@ -74,6 +87,7 @@ namespace vcpkg
 
         std::unique_ptr<std::string> vcpkg_root_dir;
         std::unique_ptr<std::string> triplet;
+        std::unique_ptr<std::vector<std::string>> overlay_ports;
         Optional<bool> debug = nullopt;
         Optional<bool> sendmetrics = nullopt;
         Optional<bool> printmetrics = nullopt;
@@ -88,6 +102,6 @@ namespace vcpkg
         ParsedArguments parse_arguments(const CommandStructure& command_structure) const;
 
     private:
-        std::unordered_map<std::string, Optional<std::string>> optional_command_arguments;
+        std::unordered_map<std::string, Optional<std::vector<std::string>>> optional_command_arguments;
     };
 }

toolsrc/include/vcpkg/vcpkgpaths.h

@@ -50,8 +50,6 @@ namespace vcpkg
         static Expected<VcpkgPaths> create(const fs::path& vcpkg_root_dir, const std::string& default_vs_path);
 
         fs::path package_dir(const PackageSpec& spec) const;
-        fs::path port_dir(const PackageSpec& spec) const;
-        fs::path port_dir(const std::string& name) const;
         fs::path build_info_file_path(const PackageSpec& spec) const;
         fs::path listfile_path(const BinaryParagraph& pgh) const;
 

toolsrc/src/tests.plan.cpp

@@ -47,7 +47,8 @@ namespace UnitTest1
 
         Assert::AreEqual(plan.spec.triplet().to_string().c_str(), triplet.to_string().c_str());
 
-        Assert::AreEqual(pkg_name.c_str(), plan.source_control_file.get()->core_paragraph->name.c_str());
+        auto* scfl = plan.source_control_file_location.get();
+        Assert::AreEqual(pkg_name.c_str(), scfl->source_control_file->core_paragraph->name.c_str());
         Assert::AreEqual(size_t(vec.size()), feature_list.size());
 
         for (auto&& feature_name : vec)
@@ -79,7 +80,7 @@ namespace UnitTest1
     /// </summary>
     struct PackageSpecMap
     {
-        std::unordered_map<std::string, SourceControlFile> map;
+        std::unordered_map<std::string, SourceControlFileLocation> map;
         Triplet triplet;
         PackageSpecMap(const Triplet& t = Triplet::X86_WINDOWS) noexcept { triplet = t; }
 
@@ -94,7 +95,8 @@ namespace UnitTest1
         {
             auto spec = PackageSpec::from_name_and_triplet(scf.core_paragraph->name, triplet);
             Assert::IsTrue(spec.has_value());
-            map.emplace(scf.core_paragraph->name, std::move(scf));
+            map.emplace(scf.core_paragraph->name,
+                        SourceControlFileLocation{std::unique_ptr<SourceControlFile>(std::move(&scf)), ""});
             return PackageSpec{*spec.get()};
         }
     };

toolsrc/src/tests.update.cpp

@@ -21,9 +21,9 @@ namespace UnitTest1
 
             StatusParagraphs status_db(std::move(status_paragraphs));
 
-            std::unordered_map<std::string, SourceControlFile> map;
+            std::unordered_map<std::string, SourceControlFileLocation> map;
             auto scf = unwrap(SourceControlFile::parse_control_file(Pgh{{{"Source", "a"}, {"Version", "0"}}}));
-            map.emplace("a", std::move(*scf));
+            map.emplace("a", SourceControlFileLocation { std::move(scf), "" });
             Dependencies::MapPortFileProvider provider(map);
 
             auto pkgs = SortedVector<OutdatedPackage>(Update::find_outdated_packages(provider, status_db),
@@ -45,9 +45,9 @@ namespace UnitTest1
 
             StatusParagraphs status_db(std::move(status_paragraphs));
 
-            std::unordered_map<std::string, SourceControlFile> map;
+            std::unordered_map<std::string, SourceControlFileLocation> map;
             auto scf = unwrap(SourceControlFile::parse_control_file(Pgh{{{"Source", "a"}, {"Version", "0"}}}));
-            map.emplace("a", std::move(*scf));
+            map.emplace("a", SourceControlFileLocation { std::move(scf), "" });
             Dependencies::MapPortFileProvider provider(map);
 
             auto pkgs = SortedVector<OutdatedPackage>(Update::find_outdated_packages(provider, status_db),
@@ -71,9 +71,9 @@ namespace UnitTest1
 
             StatusParagraphs status_db(std::move(status_paragraphs));
 
-            std::unordered_map<std::string, SourceControlFile> map;
+            std::unordered_map<std::string, SourceControlFileLocation> map;
             auto scf = unwrap(SourceControlFile::parse_control_file(Pgh{{{"Source", "a"}, {"Version", "0"}}}));
-            map.emplace("a", std::move(*scf));
+            map.emplace("a", SourceControlFileLocation{ std::move(scf), "" });
             Dependencies::MapPortFileProvider provider(map);
 
             auto pkgs = SortedVector<OutdatedPackage>(Update::find_outdated_packages(provider, status_db),
@@ -92,9 +92,9 @@ namespace UnitTest1
 
             StatusParagraphs status_db(std::move(status_paragraphs));
 
-            std::unordered_map<std::string, SourceControlFile> map;
+            std::unordered_map<std::string, SourceControlFileLocation> map;
             auto scf = unwrap(SourceControlFile::parse_control_file(Pgh{{{"Source", "a"}, {"Version", "2"}}}));
-            map.emplace("a", std::move(*scf));
+            map.emplace("a", SourceControlFileLocation{ std::move(scf), "" });
             Dependencies::MapPortFileProvider provider(map);
 
             auto pkgs = SortedVector<OutdatedPackage>(Update::find_outdated_packages(provider, status_db),

toolsrc/src/vcpkg/build.cpp

@@ -23,6 +23,7 @@
 #include <vcpkg/vcpkglib.h>
 
 using vcpkg::Build::BuildResult;
+using vcpkg::Dependencies::PathsPortFileProvider;
 using vcpkg::Parse::ParseControlErrorInfo;
 using vcpkg::Parse::ParseExpected;
 
@@ -34,34 +35,26 @@ namespace vcpkg::Build::Command
     static constexpr StringLiteral OPTION_CHECKS_ONLY = "--checks-only";
 
     void perform_and_exit_ex(const FullPackageSpec& full_spec,
-                             const fs::path& port_dir,
+                             const SourceControlFileLocation& scfl,
                              const ParsedArguments& options,
                              const VcpkgPaths& paths)
     {
         const PackageSpec& spec = full_spec.package_spec;
+        const auto& scf = *scfl.source_control_file;
         if (Util::Sets::contains(options.switches, OPTION_CHECKS_ONLY))
         {
             const auto pre_build_info = Build::PreBuildInfo::from_triplet_file(paths, spec.triplet());
             const auto build_info = Build::read_build_info(paths.get_filesystem(), paths.build_info_file_path(spec));
-            const size_t error_count = PostBuildLint::perform_all_checks(spec, paths, pre_build_info, build_info);
+            const size_t error_count =
+                PostBuildLint::perform_all_checks(spec, paths, pre_build_info, build_info, scfl.source_location);
             Checks::check_exit(VCPKG_LINE_INFO, error_count == 0);
             Checks::exit_success(VCPKG_LINE_INFO);
         }
 
-        const ParseExpected<SourceControlFile> source_control_file =
-            Paragraphs::try_load_port(paths.get_filesystem(), port_dir);
-
-        if (!source_control_file.has_value())
-        {
-            print_error_message(source_control_file.error());
-            Checks::exit_fail(VCPKG_LINE_INFO);
-        }
-
-        const auto& scf = source_control_file.value_or_exit(VCPKG_LINE_INFO);
         Checks::check_exit(VCPKG_LINE_INFO,
-                           spec.name() == scf->core_paragraph->name,
+                           spec.name() == scf.core_paragraph->name,
                            "The Source field inside the CONTROL file does not match the port directory: '%s' != '%s'",
-                           scf->core_paragraph->name,
+                           scf.core_paragraph->name,
                            spec.name());
 
         const StatusParagraphs status_db = database_load_check(paths);
@@ -80,7 +73,7 @@ namespace vcpkg::Build::Command
         features_as_set.emplace("core");
 
         const Build::BuildPackageConfig build_config{
-            *scf, spec.triplet(), fs::path{port_dir}, build_package_options, features_as_set};
+            scf, spec.triplet(), fs::path(scfl.source_location), build_package_options, features_as_set};
 
         const auto build_timer = Chrono::ElapsedTimer::create_started();
         const auto result = Build::build_package(paths, build_config, status_db);
@@ -128,10 +121,19 @@ namespace vcpkg::Build::Command
         // Build only takes a single package and all dependencies must already be installed
         const ParsedArguments options = args.parse_arguments(COMMAND_STRUCTURE);
         std::string first_arg = args.command_arguments.at(0);
+
         const FullPackageSpec spec = Input::check_and_get_full_package_spec(
             std::move(first_arg), default_triplet, COMMAND_STRUCTURE.example_text);
+
         Input::check_triplet(spec.package_spec.triplet(), paths);
-        perform_and_exit_ex(spec, paths.port_dir(spec.package_spec), options, paths);
+
+        PathsPortFileProvider provider(paths, args.overlay_ports.get());
+        const auto port_name = spec.package_spec.name();
+        const auto* scfl = provider.get_control_file(port_name).get();
+
+        Checks::check_exit(VCPKG_LINE_INFO, scfl != nullptr, "Error: Couldn't find port '%s'", port_name);
+
+        perform_and_exit_ex(spec, *scfl, options, paths);
     }
 }
 
@@ -360,6 +362,12 @@ namespace vcpkg::Build
         auto& fs = paths.get_filesystem();
         const Triplet& triplet = spec.triplet();
 
+        if (!Strings::starts_with(Strings::ascii_to_lowercase(config.port_dir.u8string()),
+                                  Strings::ascii_to_lowercase(paths.ports.u8string())))
+        {
+            System::printf("-- Installing port from location: %s\n", config.port_dir.u8string());
+        }
+
 #if !defined(_WIN32)
         // TODO: remove when vcpkg.exe is in charge for acquiring tools. Change introduced in vcpkg v0.0.107.
         // bootstrap should have already downloaded ninja, but making sure it is present in case it was deleted.
@@ -428,7 +436,8 @@ namespace vcpkg::Build
         }
 
         const BuildInfo build_info = read_build_info(fs, paths.build_info_file_path(spec));
-        const size_t error_count = PostBuildLint::perform_all_checks(spec, paths, pre_build_info, build_info);
+        const size_t error_count =
+            PostBuildLint::perform_all_checks(spec, paths, pre_build_info, build_info, config.port_dir);
 
         auto bcf = create_binary_control_file(*config.scf.core_paragraph, triplet, build_info, abi_tag);
 
@@ -705,7 +714,7 @@ namespace vcpkg::Build
                 }
             }
 
-            System::print2("Could not locate cached archive: ", archive_path.u8string(), "\n");
+            System::printf("Could not locate cached archive: %s\n", archive_path.u8string());
 
             ExtendedBuildResult result = do_build_package_and_clean_buildtrees(
                 paths, pre_build_info, spec, maybe_abi_tag_and_file.value_or(AbiTagAndFile{}).tag, config);

toolsrc/src/vcpkg/commands.autocomplete.cpp

@@ -90,7 +90,8 @@ namespace vcpkg::Commands::Autocomplete
             const auto port_name = match[2].str();
             const auto triplet_prefix = match[3].str();
 
-            auto maybe_port = Paragraphs::try_load_port(paths.get_filesystem(), paths.port_dir(port_name));
+            // TODO: Support autocomplete for ports in --overlay-ports
+            auto maybe_port = Paragraphs::try_load_port(paths.get_filesystem(), paths.ports / port_name);
             if (maybe_port.error())
             {
                 Checks::exit_success(VCPKG_LINE_INFO);

toolsrc/src/vcpkg/commands.buildexternal.cpp

@@ -23,7 +23,12 @@ namespace vcpkg::Commands::BuildExternal
             std::string(args.command_arguments.at(0)), default_triplet, COMMAND_STRUCTURE.example_text);
         Input::check_triplet(spec.package_spec.triplet(), paths);
 
-        const fs::path port_dir = args.command_arguments.at(1);
-        Build::Command::perform_and_exit_ex(spec, port_dir, options, paths);
+        auto overlays = args.overlay_ports ? *args.overlay_ports : std::vector<std::string>();
+        overlays.insert(overlays.begin(), args.command_arguments.at(1));
+        Dependencies::PathsPortFileProvider provider(paths, &overlays);
+        auto maybe_scfl = provider.get_control_file(spec.package_spec.name());
+        Checks::check_exit(
+            VCPKG_LINE_INFO, maybe_scfl.has_value(), "could not load control file for %s", spec.package_spec.name());
+        Build::Command::perform_and_exit_ex(spec, maybe_scfl.value_or_exit(VCPKG_LINE_INFO), options, paths);
     }
 }

toolsrc/src/vcpkg/commands.ci.cpp

@@ -232,12 +232,17 @@ namespace vcpkg::Commands::CI
             {
                 // determine abi tag
                 std::string abi;
-                if (auto scf = p->source_control_file.get())
+                if (auto scfl = p->source_control_file_location.get())
                 {
                     auto triplet = p->spec.triplet();
 
                     const Build::BuildPackageConfig build_config{
-                        *scf, triplet, paths.port_dir(p->spec), build_options, p->feature_list};
+                        *scfl->source_control_file, 
+                        triplet, 
+                        static_cast<fs::path>(scfl->source_location), 
+                        build_options, 
+                        p->feature_list
+                    };
 
                     auto dependency_abis =
                         Util::fmap(p->computed_dependencies, [&](const PackageSpec& spec) -> Build::AbiEntry {
@@ -351,7 +356,8 @@ namespace vcpkg::Commands::CI
         }
 
         StatusParagraphs status_db = database_load_check(paths);
-        const auto& paths_port_file = Dependencies::PathsPortFileProvider(paths);
+        
+        Dependencies::PathsPortFileProvider provider(paths, args.overlay_ports.get());
 
         const Build::BuildPackageOptions install_plan_options = {
             Build::UseHeadVersion::NO,
@@ -369,7 +375,10 @@ namespace vcpkg::Commands::CI
 
         XunitTestResults xunitTestResults;
 
-        std::vector<std::string> all_ports = Install::get_all_port_names(paths);
+        std::vector<std::string> all_ports =
+            Util::fmap(provider.load_all_control_files(), [](auto&& scfl) -> std::string {
+                return scfl->source_control_file.get()->core_paragraph->name;
+            });
         std::vector<TripletAndSummary> results;
         auto timer = Chrono::ElapsedTimer::create_started();
         for (const Triplet& triplet : triplets)
@@ -378,13 +387,13 @@ namespace vcpkg::Commands::CI
 
             xunitTestResults.push_collection(triplet.canonical_name());
 
-            Dependencies::PackageGraph pgraph(paths_port_file, status_db);
+            Dependencies::PackageGraph pgraph(provider, status_db);
 
             std::vector<PackageSpec> specs = PackageSpec::to_package_specs(all_ports, triplet);
             // Install the default features for every package
             auto all_feature_specs = Util::fmap(specs, [](auto& spec) { return FeatureSpec(spec, ""); });
             auto split_specs =
-                find_unknown_ports_for_ci(paths, exclusions_set, paths_port_file, all_feature_specs, purge_tombstones);
+                find_unknown_ports_for_ci(paths, exclusions_set, provider, all_feature_specs, purge_tombstones);
             auto feature_specs = FullPackageSpec::to_feature_specs(split_specs->unknown);
 
             for (auto&& fspec : feature_specs)

toolsrc/src/vcpkg/commands.dependinfo.cpp

@@ -6,6 +6,9 @@
 #include <vcpkg/commands.h>
 #include <vcpkg/help.h>
 #include <vcpkg/paragraphs.h>
+#include <vcpkg/dependencies.h>
+
+using vcpkg::Dependencies::PathsPortFileProvider;
 
 namespace vcpkg::Commands::DependInfo
 {
@@ -35,7 +38,7 @@ namespace vcpkg::Commands::DependInfo
         return output;
     }
 
-    std::string create_dot_as_string(const std::vector<std::unique_ptr<SourceControlFile>>& source_control_files)
+    std::string create_dot_as_string(const std::vector<const SourceControlFile*>& source_control_files)
     {
         int empty_node_count = 0;
 
@@ -64,7 +67,7 @@ namespace vcpkg::Commands::DependInfo
         return s;
     }
 
-    std::string create_dgml_as_string(const std::vector<std::unique_ptr<SourceControlFile>>& source_control_files)
+    std::string create_dgml_as_string(const std::vector<const SourceControlFile*>& source_control_files)
     {
         std::string s;
         s.append("<?xml version=\"1.0\" encoding=\"utf-8\"?>");
@@ -109,7 +112,7 @@ namespace vcpkg::Commands::DependInfo
     }
 
     std::string create_graph_as_string(const std::unordered_set<std::string>& switches,
-                                       const std::vector<std::unique_ptr<SourceControlFile>>& source_control_files)
+                                       const std::vector<const SourceControlFile*>& source_control_files)
     {
         if (Util::Sets::contains(switches, OPTION_DOT))
         {
@@ -124,7 +127,7 @@ namespace vcpkg::Commands::DependInfo
 
     void build_dependencies_list(std::set<std::string>& packages_to_keep,
                                  const std::string& requested_package,
-                                 const std::vector<std::unique_ptr<SourceControlFile>>& source_control_files,
+                                 const std::vector<const SourceControlFile*>& source_control_files,
                                  const std::unordered_set<std::string>& switches)
     {
         const auto source_control_file =
@@ -154,7 +157,11 @@ namespace vcpkg::Commands::DependInfo
     {
         const ParsedArguments options = args.parse_arguments(COMMAND_STRUCTURE);
 
-        auto source_control_files = Paragraphs::load_all_ports(paths.get_filesystem(), paths.ports);
+        // TODO: Optimize implementation, current implementation needs to load all ports from disk which is too slow.
+        PathsPortFileProvider provider(paths, args.overlay_ports.get());
+        auto source_control_files = Util::fmap(provider.load_all_control_files(), [](auto&& scfl) -> const SourceControlFile * {
+            return scfl->source_control_file.get();
+        });
 
         if (args.command_arguments.size() >= 1)
         {
@@ -178,7 +185,7 @@ namespace vcpkg::Commands::DependInfo
 
         for (auto&& source_control_file : source_control_files)
         {
-            const SourceParagraph& source_paragraph = *source_control_file->core_paragraph;
+            const SourceParagraph& source_paragraph = *source_control_file->core_paragraph.get();
             const auto s = Strings::join(", ", source_paragraph.depends, [](const Dependency& d) { return d.name(); });
             System::print2(source_paragraph.name, ": ", s, "\n");
         }

toolsrc/src/vcpkg/commands.edit.cpp

@@ -79,6 +79,7 @@ namespace vcpkg::Commands::Edit
             const auto& fs = paths.get_filesystem();
             auto packages = fs.get_files_non_recursive(paths.packages);
 
+            // TODO: Support edit for --overlay-ports
             return Util::fmap(ports, [&](const std::string& port_name) -> std::string {
                 const auto portpath = paths.ports / port_name;
                 const auto portfile = portpath / "portfile.cmake";

toolsrc/src/vcpkg/commands.search.cpp

@@ -7,6 +7,9 @@
 #include <vcpkg/paragraphs.h>
 #include <vcpkg/sourceparagraph.h>
 #include <vcpkg/vcpkglib.h>
+#include <vcpkg/dependencies.h>
+
+using vcpkg::Dependencies::PathsPortFileProvider;
 
 namespace vcpkg::Commands::Search
 {
@@ -63,7 +66,10 @@ namespace vcpkg::Commands::Search
         const ParsedArguments options = args.parse_arguments(COMMAND_STRUCTURE);
         const bool full_description = Util::Sets::contains(options.switches, OPTION_FULLDESC);
 
-        auto source_paragraphs = Paragraphs::load_all_ports(paths.get_filesystem(), paths.ports);
+        PathsPortFileProvider provider(paths, args.overlay_ports.get());
+        auto source_paragraphs = Util::fmap(provider.load_all_control_files(), [](auto&& port) -> const SourceControlFile * { 
+            return port->source_control_file.get(); 
+        });
 
         if (args.command_arguments.empty())
         {

toolsrc/src/vcpkg/commands.upgrade.cpp

@@ -43,7 +43,8 @@ namespace vcpkg::Commands::Upgrade
 
         StatusParagraphs status_db = database_load_check(paths);
 
-        Dependencies::PathsPortFileProvider provider(paths);
+        // Load ports from ports dirs
+        Dependencies::PathsPortFileProvider provider(paths, args.overlay_ports.get());
         Dependencies::PackageGraph graph(provider, status_db);
 
         // input sanitization
@@ -85,12 +86,12 @@ namespace vcpkg::Commands::Upgrade
                     not_installed.push_back(spec);
                 }
 
-                auto maybe_scf = provider.get_control_file(spec.name());
-                if (auto p_scf = maybe_scf.get())
+                auto maybe_scfl = provider.get_control_file(spec.name());
+                if (auto p_scfl = maybe_scfl.get())
                 {
                     if (it != status_db.end())
                     {
-                        if (p_scf->core_paragraph->version != (*it)->package.version)
+                        if (p_scfl->source_control_file->core_paragraph->version != (*it)->package.version)
                         {
                             to_upgrade.push_back(spec);
                         }

toolsrc/src/vcpkg/dependencies.cpp

@@ -22,7 +22,7 @@ namespace vcpkg::Dependencies
 
     struct ClusterSource
     {
-        const SourceControlFile* scf = nullptr;
+        const SourceControlFileLocation* scfl = nullptr;
         std::unordered_map<std::string, std::vector<FeatureSpec>> build_edges;
     };
 
@@ -92,12 +92,12 @@ namespace vcpkg::Dependencies
             if (it == m_graph.end())
             {
                 // Load on-demand from m_provider
-                auto maybe_scf = m_provider.get_control_file(spec.name());
+                auto maybe_scfl = m_provider.get_control_file(spec.name());
                 auto& clust = m_graph[spec];
                 clust.spec = spec;
-                if (auto p_scf = maybe_scf.get())
+                if (auto p_scfl = maybe_scfl.get())
                 {
-                    clust.source = cluster_from_scf(*p_scf, clust.spec.triplet());
+                    clust.source = cluster_from_scf(*p_scfl, clust.spec.triplet());
                 }
                 return clust;
             }
@@ -105,15 +105,17 @@ namespace vcpkg::Dependencies
         }
 
     private:
-        static ClusterSource cluster_from_scf(const SourceControlFile& scf, Triplet t)
+        static ClusterSource cluster_from_scf(const SourceControlFileLocation& scfl, Triplet t)
         {
             ClusterSource ret;
-            ret.build_edges.emplace("core", filter_dependencies_to_specs(scf.core_paragraph->depends, t));
+            ret.build_edges.emplace("core", 
+                filter_dependencies_to_specs(scfl.source_control_file->core_paragraph->depends, 
+                    t));
 
-            for (const auto& feature : scf.feature_paragraphs)
+            for (const auto& feature : scfl.source_control_file->feature_paragraphs)
                 ret.build_edges.emplace(feature->name, filter_dependencies_to_specs(feature->depends, t));
 
-            ret.scf = &scf;
+            ret.scfl = &scfl;
             return ret;
         }
 
@@ -151,12 +153,12 @@ namespace vcpkg::Dependencies
     }
 
     InstallPlanAction::InstallPlanAction(const PackageSpec& spec,
-                                         const SourceControlFile& scf,
+                                         const SourceControlFileLocation& scfl,
                                          const std::set<std::string>& features,
                                          const RequestType& request_type,
                                          std::vector<PackageSpec>&& dependencies)
         : spec(spec)
-        , source_control_file(scf)
+        , source_control_file_location(scfl)
         , plan_type(InstallPlanType::BUILD_AND_INSTALL)
         , request_type(request_type)
         , build_options{}
@@ -268,37 +270,145 @@ namespace vcpkg::Dependencies
         return left->spec.name() < right->spec.name();
     }
 
-    MapPortFileProvider::MapPortFileProvider(const std::unordered_map<std::string, SourceControlFile>& map) : ports(map)
-    {
-    }
+    MapPortFileProvider::MapPortFileProvider(const std::unordered_map<std::string, SourceControlFileLocation>& map)
+        : ports(map)
+    {}
 
-    Optional<const SourceControlFile&> MapPortFileProvider::get_control_file(const std::string& spec) const
+    Optional<const SourceControlFileLocation&> MapPortFileProvider::get_control_file(const std::string& spec) const
     {
         auto scf = ports.find(spec);
         if (scf == ports.end()) return nullopt;
         return scf->second;
     }
 
-    PathsPortFileProvider::PathsPortFileProvider(const VcpkgPaths& paths) : ports(paths) {}
+    std::vector<const SourceControlFileLocation*> MapPortFileProvider::load_all_control_files() const
+    {
+        return Util::fmap(ports, [](auto&& kvpair) -> const SourceControlFileLocation * { return &kvpair.second; });
+    }
 
-    Optional<const SourceControlFile&> PathsPortFileProvider::get_control_file(const std::string& spec) const
+    PathsPortFileProvider::PathsPortFileProvider(const vcpkg::VcpkgPaths& paths,
+                                                 const std::vector<std::string>* ports_dirs_paths) 
+        : filesystem(paths.get_filesystem())
+    {
+        if (ports_dirs_paths)
+        {
+            for (auto&& overlay_path : *ports_dirs_paths)
+            {
+                if (!overlay_path.empty())
+                {
+                    auto overlay = fs::stdfs::canonical(fs::u8path(overlay_path));
+
+                    Checks::check_exit(VCPKG_LINE_INFO,
+                                       filesystem.exists(overlay),
+                                       "Error: Path \"%s\" does not exist",
+                                       overlay.string());
+
+                    Checks::check_exit(VCPKG_LINE_INFO,
+                                       fs::stdfs::is_directory(overlay),
+                                       "Error: Path \"%s\" must be a directory",
+                                       overlay.string());
+
+                    ports_dirs.emplace_back(overlay);
+                }
+            }
+        }
+        ports_dirs.emplace_back(paths.ports);
+    }
+
+    Optional<const SourceControlFileLocation&> PathsPortFileProvider::get_control_file(const std::string& spec) const
     {
         auto cache_it = cache.find(spec);
         if (cache_it != cache.end())
         {
             return cache_it->second;
         }
-        Parse::ParseExpected<SourceControlFile> source_control_file =
-            Paragraphs::try_load_port(ports.get_filesystem(), ports.port_dir(spec));
 
-        if (auto scf = source_control_file.get())
+        for (auto&& ports_dir : ports_dirs)
         {
-            auto it = cache.emplace(spec, std::move(*scf->get()));
-            return it.first->second;
+            // Try loading individual port
+            if (filesystem.exists(ports_dir / "CONTROL"))
+            {
+                auto maybe_scf = Paragraphs::try_load_port(filesystem, ports_dir);
+                if (auto scf = maybe_scf.get())
+                {
+                    if (scf->get()->core_paragraph->name == spec)
+                    {
+                        SourceControlFileLocation scfl{ std::move(*scf), ports_dir };
+                        auto it = cache.emplace(spec, std::move(scfl));
+                        return it.first->second;
+                    }
+                }
+                else
+                {
+                    vcpkg::print_error_message(maybe_scf.error());
+                    Checks::exit_with_message(VCPKG_LINE_INFO,
+                                              "Error: Failed to load port from %s",
+                                              spec, ports_dir.u8string());
+                }
+            }
+
+            auto found_scf = Paragraphs::try_load_port(filesystem, ports_dir / spec);
+            if (auto scf = found_scf.get())
+            {
+                if (scf->get()->core_paragraph->name == spec)
+                {
+                    SourceControlFileLocation scfl{ std::move(*scf), ports_dir / spec };
+                    auto it = cache.emplace(spec, std::move(scfl));
+                    return it.first->second;
+                }
+            }
         }
+
         return nullopt;
     }
 
+    std::vector<const SourceControlFileLocation*> PathsPortFileProvider::load_all_control_files() const
+    {
+        // Reload cache with ports contained in all ports_dirs
+        cache.clear();
+        std::vector<const SourceControlFileLocation*> ret;
+        for (auto&& ports_dir : ports_dirs)
+        {
+            // Try loading individual port
+            if (filesystem.exists(ports_dir / "CONTROL"))
+            {
+                auto maybe_scf = Paragraphs::try_load_port(filesystem, ports_dir);
+                if (auto scf = maybe_scf.get())
+                {
+                    auto port_name = scf->get()->core_paragraph->name;
+                    if (cache.find(port_name) == cache.end())
+                    {
+                        SourceControlFileLocation scfl{ std::move(*scf), ports_dir };
+                        auto it = cache.emplace(port_name, std::move(scfl));
+                        ret.emplace_back(&it.first->second);
+                    }
+                }
+                else
+                {
+                    vcpkg::print_error_message(maybe_scf.error());
+                    Checks::exit_with_message(VCPKG_LINE_INFO,
+                                              "Error: Failed to load port from %s",
+                                              ports_dir.u8string());
+                }
+                continue;
+            }
+
+            // Try loading all ports inside ports_dir
+            auto found_scf = Paragraphs::load_all_ports(filesystem, ports_dir);
+            for (auto&& scf : found_scf)
+            {
+                auto port_name = scf->core_paragraph->name;
+                if (cache.find(port_name) == cache.end())
+                {
+                    SourceControlFileLocation scfl{ std::move(scf), ports_dir / port_name };
+                    auto it = cache.emplace(port_name, std::move(scfl));
+                    ret.emplace_back(&it.first->second);
+                }
+            }
+        }
+        return ret;
+    }
+
     std::vector<RemovePlanAction> create_remove_plan(const std::vector<PackageSpec>& specs,
                                                      const StatusParagraphs& status_db)
     {
@@ -495,10 +605,11 @@ namespace vcpkg::Dependencies
                 VCPKG_LINE_INFO, "Error: Cannot find definition for package `%s`.", cluster.spec.name());
         }
 
+        auto&& control_file = *p_source->scfl->source_control_file.get();
         if (feature.empty())
         {
             // Add default features for this package. This is an exact reference, so ignore prevent_default_features.
-            for (auto&& default_feature : p_source->scf->core_paragraph.get()->default_features)
+            for (auto&& default_feature : control_file.core_paragraph.get()->default_features)
             {
                 auto res = mark_plus(default_feature, cluster, graph, graph_plan, prevent_default_features);
                 if (res != MarkPlusResult::SUCCESS)
@@ -513,7 +624,7 @@ namespace vcpkg::Dependencies
 
         if (feature == "*")
         {
-            for (auto&& fpgh : p_source->scf->feature_paragraphs)
+            for (auto&& fpgh : control_file.feature_paragraphs)
             {
                 auto res = mark_plus(fpgh->name, cluster, graph, graph_plan, prevent_default_features);
 
@@ -590,7 +701,8 @@ namespace vcpkg::Dependencies
 
             // Check if any default features have been added
             auto& previous_df = p_installed->ipv.core->package.default_features;
-            for (auto&& default_feature : p_source->scf->core_paragraph->default_features)
+            auto&& control_file = *p_source->scfl->source_control_file.get();
+            for (auto&& default_feature : control_file.core_paragraph->default_features)
             {
                 if (std::find(previous_df.begin(), previous_df.end(), default_feature) == previous_df.end())
                 {
@@ -635,7 +747,7 @@ namespace vcpkg::Dependencies
     /// <param name="map">Map of all source control files in the current environment.</param>
     /// <param name="specs">Feature specifications to resolve dependencies for.</param>
     /// <param name="status_db">Status of installed packages in the current environment.</param>
-    std::vector<AnyAction> create_feature_install_plan(const std::unordered_map<std::string, SourceControlFile>& map,
+    std::vector<AnyAction> create_feature_install_plan(const std::unordered_map<std::string, SourceControlFileLocation>& map,
                                                        const std::vector<FeatureSpec>& specs,
                                                        const StatusParagraphs& status_db)
     {
@@ -698,7 +810,11 @@ namespace vcpkg::Dependencies
             if (p_cluster->transient_uninstalled)
             {
                 // If it will be transiently uninstalled, we need to issue a full installation command
-                auto pscf = p_cluster->source.value_or_exit(VCPKG_LINE_INFO).scf;
+                auto* pscfl = p_cluster->source.value_or_exit(VCPKG_LINE_INFO).scfl;
+                Checks::check_exit(VCPKG_LINE_INFO,
+                                   pscfl != nullptr,
+                                   "Error: Expected a SourceControlFileLocation to exist");
+                auto&& scfl = *pscfl;
 
                 auto dep_specs = Util::fmap(m_graph_plan->install_graph.adjacency_list(p_cluster),
                                             [](ClusterPtr const& p) { return p->spec; });
@@ -706,7 +822,7 @@ namespace vcpkg::Dependencies
 
                 plan.emplace_back(InstallPlanAction{
                     p_cluster->spec,
-                    *pscf,
+                    scfl,
                     p_cluster->to_install_features,
                     p_cluster->request_type,
                     std::move(dep_specs),

toolsrc/src/vcpkg/export.cpp

@@ -488,7 +488,10 @@ With a project open, go to Tools->NuGet Package Manager->Package Manager Console
 
         // create the plan
         const StatusParagraphs status_db = database_load_check(paths);
-        Dependencies::PathsPortFileProvider provider(paths);
+
+        // Load ports from ports dirs
+        Dependencies::PathsPortFileProvider provider(paths, args.overlay_ports.get());
+
         std::vector<ExportPlanAction> export_plan = Dependencies::create_export_plan(opts.specs, status_db);
         Checks::check_exit(VCPKG_LINE_INFO, !export_plan.empty(), "Export plan cannot be empty");
 

toolsrc/src/vcpkg/help.cpp

@@ -111,10 +111,12 @@ namespace vcpkg::Help
                        "  vcpkg contact                   Display contact information to send feedback\n"
                        "\n"
                        "Options:\n"
-                       "  --triplet <t>                   Specify the target architecture triplet.\n"
+                       "  --triplet <t>                   Specify the target architecture triplet\n"
                        "                                  (default: " ENVVAR(VCPKG_DEFAULT_TRIPLET) //
                        ", see 'vcpkg help triplet')\n"
                        "\n"
+                       "  --overlay-ports=<path>          Specify directories to be used when searching for ports\n"
+                       "\n"
                        "  --vcpkg-root <path>             Specify the vcpkg root "
                        "directory\n"
                        "                                  (default: " ENVVAR(VCPKG_ROOT) //

toolsrc/src/vcpkg/install.cpp

@@ -330,9 +330,10 @@ namespace vcpkg::Install
                 System::printf("Building package %s...\n", display_name_with_features);
 
             auto result = [&]() -> Build::ExtendedBuildResult {
-                const Build::BuildPackageConfig build_config{action.source_control_file.value_or_exit(VCPKG_LINE_INFO),
+                const auto& scfl = action.source_control_file_location.value_or_exit(VCPKG_LINE_INFO);
+                const Build::BuildPackageConfig build_config{*scfl.source_control_file,
                                                              action.spec.triplet(),
-                                                             paths.port_dir(action.spec),
+                                                             static_cast<fs::path>(scfl.source_location),
                                                              action.build_options,
                                                              action.feature_list};
                 return Build::build_package(paths, build_config, status_db);
@@ -652,13 +653,10 @@ namespace vcpkg::Install
             Build::FailOnTombstone::NO,
         };
 
-        auto all_ports = Paragraphs::load_all_ports(fs, paths.ports);
-        std::unordered_map<std::string, SourceControlFile> scf_map;
-        for (auto&& port : all_ports)
-            scf_map[port->core_paragraph->name] = std::move(*port);
-        MapPortFileProvider provider(scf_map);
+        //// Load ports from ports dirs
+        PathsPortFileProvider provider(paths, args.overlay_ports.get());
 
-        // Note: action_plan will hold raw pointers to SourceControlFiles from this map
+        // Note: action_plan will hold raw pointers to SourceControlFileLocations from this map
         std::vector<AnyAction> action_plan =
             create_feature_install_plan(provider, FullPackageSpec::to_feature_specs(specs), status_db);
 

toolsrc/src/vcpkg/postbuildlint.cpp

@@ -857,14 +857,15 @@ namespace vcpkg::PostBuildLint
     size_t perform_all_checks(const PackageSpec& spec,
                               const VcpkgPaths& paths,
                               const PreBuildInfo& pre_build_info,
-                              const BuildInfo& build_info)
+                              const BuildInfo& build_info,
+                              const fs::path& port_dir)
     {
         System::print2("-- Performing post-build validation\n");
         const size_t error_count = perform_all_checks_and_return_error_count(spec, paths, pre_build_info, build_info);
 
         if (error_count != 0)
         {
-            const fs::path portfile = paths.ports / spec.name() / "portfile.cmake";
+            const fs::path portfile = port_dir / "portfile.cmake";
             System::print2(System::Color::error,
                            "Found ",
                            error_count,

toolsrc/src/vcpkg/remove.cpp

@@ -229,7 +229,8 @@ namespace vcpkg::Remove
                 Checks::exit_fail(VCPKG_LINE_INFO);
             }
 
-            Dependencies::PathsPortFileProvider provider(paths);
+            // Load ports from ports dirs
+            Dependencies::PathsPortFileProvider provider(paths, args.overlay_ports.get());
 
             specs = Util::fmap(Update::find_outdated_packages(provider, status_db),
                                [](auto&& outdated) { return outdated.spec; });

toolsrc/src/vcpkg/update.cpp

@@ -23,10 +23,10 @@ namespace vcpkg::Update
         for (auto&& ipv : installed_packages)
         {
             const auto& pgh = ipv.core;
-            auto maybe_scf = provider.get_control_file(pgh->package.spec.name());
-            if (auto p_scf = maybe_scf.get())
+            auto maybe_scfl = provider.get_control_file(pgh->package.spec.name());
+            if (auto p_scfl = maybe_scfl.get())
             {
-                auto&& port_version = p_scf->core_paragraph->version;
+                auto&& port_version = p_scfl->source_control_file->core_paragraph->version;
                 auto&& installed_version = pgh->package.version;
                 if (installed_version != port_version)
                 {
@@ -57,7 +57,7 @@ namespace vcpkg::Update
 
         const StatusParagraphs status_db = database_load_check(paths);
 
-        Dependencies::PathsPortFileProvider provider(paths);
+        Dependencies::PathsPortFileProvider provider(paths, args.overlay_ports.get());
 
         const auto outdated_packages = SortedVector<OutdatedPackage>(find_outdated_packages(provider, status_db),
                                                                      &OutdatedPackage::compare_by_name);

toolsrc/src/vcpkg/vcpkgcmdarguments.cpp

@@ -45,6 +45,45 @@ namespace vcpkg
         option_field = new_setting;
     }
 
+    static void parse_multivalue(const std::string* arg_begin,
+                                 const std::string* arg_end,
+                                 const std::string& option_name,
+                                 std::unique_ptr<std::vector<std::string>>& option_field)
+    {
+        if (arg_begin == arg_end)
+        {
+            System::print2(System::Color::error, "Error: expected value after ", option_name, '\n');
+            Metrics::g_metrics.lock()->track_property("error", "error option name");
+            Help::print_usage();
+            Checks::exit_fail(VCPKG_LINE_INFO);
+        }
+
+        if (!option_field)
+        {
+            option_field = std::make_unique<std::vector<std::string>>();
+        }
+        option_field->emplace_back(*arg_begin);
+    }
+
+    static void parse_cojoined_multivalue(std::string new_value,
+                                          const std::string& option_name,
+                                          std::unique_ptr<std::vector<std::string>>& option_field)
+    {
+        if (new_value.empty())
+        {
+            System::print2(System::Color::error, "Error: expected value after ", option_name, '\n');
+            Metrics::g_metrics.lock()->track_property("error", "error option name");
+            Help::print_usage();
+            Checks::exit_fail(VCPKG_LINE_INFO);
+        }
+
+        if (!option_field)
+        {
+            option_field = std::make_unique<std::vector<std::string>>();
+        }
+        option_field->emplace_back(std::move(new_value));
+    }
+
     VcpkgCmdArguments VcpkgCmdArguments::create_from_command_line(const int argc,
                                                                   const CommandLineCharType* const* const argv)
     {
@@ -117,6 +156,13 @@ namespace vcpkg
                     parse_value(arg_begin, arg_end, "--triplet", args.triplet);
                     continue;
                 }
+                if (Strings::starts_with(arg, "--overlay-ports="))
+                {
+                    parse_cojoined_multivalue(arg.substr(sizeof("--overlay-ports=") - 1),
+                                              "--overlay-ports",
+                                              args.overlay_ports);
+                    continue;
+                }
                 if (arg == "--debug")
                 {
                     parse_switch(true, "debug", args.debug);
@@ -166,7 +212,21 @@ namespace vcpkg
                 const auto eq_pos = arg.find('=');
                 if (eq_pos != std::string::npos)
                 {
-                    args.optional_command_arguments.emplace(arg.substr(0, eq_pos), arg.substr(eq_pos + 1));
+                    const auto& key = arg.substr(0, eq_pos);
+                    const auto& value = arg.substr(eq_pos + 1);
+                    
+                    auto it = args.optional_command_arguments.find(key);
+                    if (args.optional_command_arguments.end() == it)
+                    {
+                        args.optional_command_arguments.emplace(key, std::vector<std::string> { value });
+                    }
+                    else
+                    {
+                        if (auto* maybe_values = it->second.get())
+                        {
+                            maybe_values->emplace_back(value);
+                        }
+                    }
                 }
                 else
                 {
@@ -264,7 +324,51 @@ namespace vcpkg
                 }
                 else
                 {
-                    output.settings.emplace(option.name, it->second.value_or_exit(VCPKG_LINE_INFO));
+                    const auto& value = it->second.value_or_exit(VCPKG_LINE_INFO);
+                    if (value.front().empty())
+                    {
+                        // Fail when not given a value, e.g.: "vcpkg install sqlite3 --additional-ports="
+                        System::printf(
+                            System::Color::error, "Error: The option '%s' must be passed an argument.\n", option.name);
+                        failed = true;
+                    }
+                    else
+                    {
+                        output.settings.emplace(option.name, value.front());
+                        options_copy.erase(it);
+                    }
+                }
+            }
+        }
+
+        for (auto&& option : command_structure.options.multisettings)
+        {
+            const auto it = options_copy.find(option.name);
+            if (it != options_copy.end())
+            {
+                if (!it->second.has_value())
+                {
+                    // Not having a string value indicates it was passed like '--a'
+                    System::printf(
+                        System::Color::error, "Error: The option '%s' must be passed an argument.\n", option.name);
+                    failed = true;
+                }
+                else
+                {
+                    const auto& value = it->second.value_or_exit(VCPKG_LINE_INFO);
+                    for (auto&& v : value)
+                    {
+                        if (v.empty())
+                        {
+                            System::printf(
+                                System::Color::error, "Error: The option '%s' must be passed an argument.\n", option.name);
+                            failed = true;
+                        }
+                        else
+                        {
+                            output.multisettings[option.name].emplace_back(v);
+                        }
+                    }
                     options_copy.erase(it);
                 }
             }
@@ -306,7 +410,14 @@ namespace vcpkg
         {
             System::printf("    %-40s %s\n", (option.name + "=..."), option.short_help_text);
         }
+        for (auto&& option : command_structure.options.multisettings)
+        {
+            System::printf("    %-40s %s\n", (option.name + "=..."), option.short_help_text);
+        }
         System::printf("    %-40s %s\n", "--triplet <t>", "Set the default triplet for unqualified packages");
+        System::printf("    %-40s %s\n", 
+                       "--overlay-ports=<path>", 
+                       "Specify directories to be used when searching for ports");
         System::printf("    %-40s %s\n",
                        "--vcpkg-root <path>",
                        "Specify the vcpkg directory to use instead of current directory or tool directory");

toolsrc/src/vcpkg/vcpkgpaths.cpp

@@ -81,9 +81,6 @@ namespace vcpkg
 
     fs::path VcpkgPaths::package_dir(const PackageSpec& spec) const { return this->packages / spec.dir(); }
 
-    fs::path VcpkgPaths::port_dir(const PackageSpec& spec) const { return this->ports / spec.name(); }
-    fs::path VcpkgPaths::port_dir(const std::string& name) const { return this->ports / name; }
-
     fs::path VcpkgPaths::build_info_file_path(const PackageSpec& spec) const
     {
         return this->package_dir(spec) / "BUILD_INFO";

toolsrc/vcpkg/vcpkg.vcxproj

@@ -21,8 +21,8 @@
   <PropertyGroup Label="Globals">
     <ProjectGuid>{34671B80-54F9-46F5-8310-AC429C11D4FB}</ProjectGuid>
     <RootNamespace>vcpkg</RootNamespace>
-    <WindowsTargetPlatformVersion>8.1</WindowsTargetPlatformVersion>
-    <PlatformToolset>v140</PlatformToolset>
+    <WindowsTargetPlatformVersion>10.0</WindowsTargetPlatformVersion>
+    <PlatformToolset>v142</PlatformToolset>
   </PropertyGroup>
   <Import Project="$(VCTargetsPath)\Microsoft.Cpp.Default.props" />
   <PropertyGroup Condition="'$(Configuration)|$(Platform)'=='Debug|Win32'" Label="Configuration">

toolsrc/vcpkglib/vcpkglib.vcxproj

@@ -21,8 +21,8 @@
   <PropertyGroup Label="Globals">
     <ProjectGuid>{B98C92B7-2874-4537-9D46-D14E5C237F04}</ProjectGuid>
     <RootNamespace>vcpkglib</RootNamespace>
-    <WindowsTargetPlatformVersion>8.1</WindowsTargetPlatformVersion>
-    <PlatformToolset>v140</PlatformToolset>
+    <WindowsTargetPlatformVersion>10.0</WindowsTargetPlatformVersion>
+    <PlatformToolset>v142</PlatformToolset>
   </PropertyGroup>
   <Import Project="$(VCTargetsPath)\Microsoft.Cpp.Default.props" />
   <PropertyGroup Condition="'$(Configuration)|$(Platform)'=='Debug|Win32'" Label="Configuration">

toolsrc/vcpkgmetricsuploader/vcpkgmetricsuploader.vcxproj

@@ -21,8 +21,8 @@
   <PropertyGroup Label="Globals">
     <ProjectGuid>{7D6FDEEB-B299-4A23-85EE-F67C4DED47BE}</ProjectGuid>
     <RootNamespace>vcpkgmetricsuploader</RootNamespace>
-    <WindowsTargetPlatformVersion>8.1</WindowsTargetPlatformVersion>
-    <PlatformToolset>v140</PlatformToolset>
+    <WindowsTargetPlatformVersion>10.0</WindowsTargetPlatformVersion>
+    <PlatformToolset>v142</PlatformToolset>
   </PropertyGroup>
   <Import Project="$(VCTargetsPath)\Microsoft.Cpp.Default.props" />
   <PropertyGroup Condition="'$(Configuration)|$(Platform)'=='Debug|Win32'" Label="Configuration">

toolsrc/vcpkgtest/vcpkgtest.vcxproj

@@ -48,8 +48,8 @@
     <ProjectGuid>{F27B8DB0-1279-4AF8-A2E3-1D49C4F0220D}</ProjectGuid>
     <Keyword>Win32Proj</Keyword>
     <RootNamespace>vcpkgtest</RootNamespace>
-    <WindowsTargetPlatformVersion>8.1</WindowsTargetPlatformVersion>
-    <PlatformToolset>v140</PlatformToolset>
+    <WindowsTargetPlatformVersion>10.0</WindowsTargetPlatformVersion>
+    <PlatformToolset>v142</PlatformToolset>
   </PropertyGroup>
   <Import Project="$(VCTargetsPath)\Microsoft.Cpp.Default.props" />
   <PropertyGroup Condition="'$(Configuration)|$(Platform)'=='Debug|Win32'" Label="Configuration">
@@ -84,7 +84,7 @@
   <ImportGroup Label="Shared">
   </ImportGroup>
   <PropertyGroup Label="UserMacros" />
-    <PropertyGroup Condition="'$(Configuration)|$(Platform)'=='Debug|Win32'">
+  <PropertyGroup Condition="'$(Configuration)|$(Platform)'=='Debug|Win32'">
     <IntDir>$(SolutionDir)msbuild.x86.debug\$(ProjectName)\</IntDir>
     <OutDir>$(SolutionDir)msbuild.x86.debug\</OutDir>
   </PropertyGroup>