Adds Experimental option to use PnP.Core.Provisioning

[Adam-it]

🎯 Aim

The aim is to add - Experimental option provisioning template commands so that when used it will use PnP.Core.Provisioning engine under the hood, and not PnP Framework.

Worth pointing out

Since PnP Core is more MS Graph and SP Rest based, additional scopes are needed than just AllSites.FullControl. So in order to use PnP.Core.Provisioning fully the app reg should have the following
Select Microsoft Graph > Delegated permissions > select Group.ReadWrite.All
Select Microsoft Graph > Delegated permissions > select openid
Select Microsoft Graph > Delegated permissions > select profile
Select Microsoft Graph > Delegated permissions > select Sites.Manage.All
Select Microsoft Graph > Delegated permissions > select User.Read.All
Select SharePoint > Delegated permissions > select AllSites.FullControl
Select SharePoint > Delegated permissions > select AllSites.Manage
Select SharePoint > Delegated permissions > select TermStore.ReadWrite.All
Select SharePoint > Delegated permissions > select User.Read.All

📸 Result

[gautamdsheth]

Thanks @Adam-it! I tested this against a dev tenant, running both engines side by side. The read and extract side works well: a full site extract with -Experimental took 118 s against 212 s with PnP Framework. Export-PnPListToSiteTemplate, Add-PnPDataRowsToSiteTemplate, Add-PnPListFoldersToSiteTemplate, Add/Remove-PnPFileToSiteTemplate, Set-PnPSiteTemplateMetadata, Convert-PnPSiteTemplate and the tenant template .pnp round trip all produce the same output as PnP Framework. A few things need fixing before merge, though.

Must fix

1. Invoke-PnPTenantTemplate no longer accepts the template by position or from the pipeline. This affects scripts that don't use -Experimental.
-Template is now TenantTemplateInstancePipeBind, so binding to it needs a type conversion. [string] Path also takes position 0 and pipeline input, and also needs a conversion, so PowerShell can't pick a parameter set:

$t = Read-PnPTenantTemplate .\tenant.xml
$t | Invoke-PnPTenantTemplate          # AmbiguousParameterSet (works in 3.4.1)
Invoke-PnPTenantTemplate $t            # AmbiguousParameterSet (works in 3.4.1)
Invoke-PnPTenantTemplate -Template $t  # works

In a quick simulation of the binder, adding DefaultParameterSetName = ParameterSet_OBJECT to the [Cmdlet] attribute fixed this, and strings still bound to -Path.

2. .pnp packages don't work across engines, in either direction.

• A PnP Framework package read with -Experimental fails with Unable to cast object of type 'PnP.Framework.Provisioning.Connectors.OpenXML.Model.PnPManifest' to type 'PnP.Core.Provisioning.Connectors.OpenXML.Model.PnPManifest'.
• The reverse fails the same way.
• 3.4.1 can't open a package saved with -Experimental: Could not load file or assembly 'PnP.Core.Provisioning'.

So Invoke-PnPSiteTemplate -Path .\template.pnp -Experimental (example 12) fails for every existing package. The root cause is in PnP.Core.Provisioning: the manifest is XAML with a clr-namespace:…;assembly=… reference, and PnPPackage.cs only rewrites the old OfficeDevPnP.Core namespace, not PnP.Framework. The real fix belongs in PnP Core, plus the reverse in PnP Framework. Until then, could this PR catch the cast error and tell the user what's going on?

3. -ExcludeHandlers doesn't work reliably with -Experimental.
PnP Framework has two page flags, Pages and PageContents, which both become Core's single Pages handler. InvertExcludedHandlers runs before that merge, so excluding one flag is undone by the other:

• Get-PnPSiteTemplate -ExcludeHandlers PageContents -IncludeAllPages -Experimental still extracts every client-side page. PnP Framework extracts none.
• Invoke-PnPSiteTemplate -ExcludeHandlers Pages -Experimental still creates or overwrites the template's pages. PnP Framework skips them.

Excluding every handler produces an empty Core handler list. Core treats that as "all handlers" and extracts everything, where PnP Framework extracts nothing.

Suggested fix: map the excludes to Core handlers first, remove them from the full list of Core handlers, and error if nothing is left.

4. Files inside a .pnp aren't found when applying through -Stream or -InputInstance.
-Path pkg.pnp -Experimental works. These two routes fail with The file 'x.txt' is not present in the template's files.:

• -Stream: CoreProvisioningHelper.LoadSiteTemplatesFromStream never sets template.Connector, so it keeps Core's default FileSystemConnector("."). This also affects -Path https://…/x.pnp and Read-PnPSiteTemplate -Stream -Experimental. The PnP Framework path sets provisioningTemplate.Connector = provider.Connector.
• -InputInstance: InvokeSiteTemplate.Experimental.cs always replaces the connector with one for the current directory. The PnP Framework path only does that when -ResourceFolder is passed.

Should fix

5. Ctrl+C doesn't stop an -Experimental apply. CoreProvisioningReporter.Run uses Task.Run with no cancellation. I stopped a run 3 s into applying a template with 6 lists. The cmdlet returned after 1.5 s with 0 lists created, then all 6 lists appeared within the next 10 s. PnP Framework waits until the engine stops and creates nothing afterwards. Could you connect a CancellationTokenSource to StopProcessing, or at least wait for the task before returning?

6. Document the extra permissions. The scopes listed in the PR description aren't in the cmdlet docs or the article.

7. CHANGELOG fixes:

• The two Improved connect-pnponline cert auth to work with persist login and also added cmdlet to list persist login cmdlets #5463 entries are duplicated under Changed.
• The new Changed entry has no PR link, and it says PnP Framework templates "keep binding to these parameters as before", which isn't true (see 1).
• There's a stray blank line.

Minor

• Has → HasFlag in the PnP Framework code paths, and the BOM removal in two PipeBinds, are unrelated churn. Behaviour is unchanged.
• InvokeSiteTemplate.Experimental and SetSiteTemplateMetadata.Experimental copy the provider and connector setup from CoreProvisioningHelper.
• The release build references PnP.Core.Provisioning 1.18.*-*, and only nightly versions exist on NuGet, so the next stable release would ship a nightly build of it.
• Engine Error messages are shown as warnings.

FYI: differences in engine output (from PnP.Core.Provisioning, not this PR)

Compared with PnP Framework, a full extract of the same site with -Experimental:

• leaves out per-list hidden taxonomy note fields and CheckoutUser (39 fields)
• leaves out site and web features (12)
• drops several system property bag keys
• adds 51 SupportedUILanguages
• adds SHAREPOINT\system as an additional owner

I haven't checked whether these matter when the template is applied.

[Adam-it]

@gautamdsheth all addressed... at least I hope I didn't miss anything 😁, and ready for another round

[gautamdsheth]

Awesome stuff @Adam-it , 4 small changes and we are good to go :

1. Save-PnPSiteTemplate -Experimental -Force fails when the existing .pnp was made by PnP Framework, and its error gives the wrong advice. Fix: delete the existing file first, as the tenant version does.
2. Has → HasFlag: in the existing PnP Framework code paths of Get-PnPSiteTemplate, Invoke-PnPSiteTemplate and Invoke-PnPTenantTemplate, the PR swaps PnP Framework's Has() extension for .NET's HasFlag(). Both do the same bit check, so the result is identical. It just touches non-experimental code for no reason.
3. ApplyConfigurationPipeBind.cs and ExtractConfigurationPipeBind.cs lost the invisible byte-order mark at the start of the file. That makes the first line show up as changed in the diff, with no effect on the code.
4. One more suggestion: at the moment, engine Error messages are only shown as warnings, so a run where some items fail still ends as a success, and try/catch, -ErrorAction Stop and $? never notice. Could CoreProvisioningReporter collect those errors as they come in, keep showing them live as warnings, and then call WriteError once per distinct error after the engine finishes? For example, use ErrorCategory.InvalidResult with the site URL as the target. The template would still be fully applied, but scripts could tell that part of it failed.

[Adam-it]

Awesome stuff @Adam-it , 4 small changes and we are good to go :

1. Save-PnPSiteTemplate -Experimental -Force fails when the existing .pnp was made by PnP Framework, and its error gives the wrong advice. Fix: delete the existing file first, as the tenant version does.
2. Has → HasFlag: in the existing PnP Framework code paths of Get-PnPSiteTemplate, Invoke-PnPSiteTemplate and Invoke-PnPTenantTemplate, the PR swaps PnP Framework's Has() extension for .NET's HasFlag(). Both do the same bit check, so the result is identical. It just touches non-experimental code for no reason.
3. ApplyConfigurationPipeBind.cs and ExtractConfigurationPipeBind.cs lost the invisible byte-order mark at the start of the file. That makes the first line show up as changed in the diff, with no effect on the code.
4. One more suggestion: at the moment, engine Error messages are only shown as warnings, so a run where some items fail still ends as a success, and try/catch, -ErrorAction Stop and $? never notice. Could CoreProvisioningReporter collect those errors as they come in, keep showing them live as warnings, and then call WriteError once per distinct error after the engine finishes? For example, use ErrorCategory.InvalidResult with the site URL as the target. The template would still be fully applied, but scripts could tell that part of it failed.

@gautamdsheth ok mostly addressed your feed: updated the error message, reverted the invisible strange changes...I hope it worked, and added an error write logic which is kinda tricky but should work 😉

As for the 2nd comment The Has → HasFlag change is needed: PnP Framework and PnP.Core.Provisioning both define a public System.EnumerationExtensions.Has<T>, so with both referenced every .Has(...) call is ambiguous (CS0121). HasFlag does the same bit check. What do you think?

[gautamdsheth]

Thanks @Adam-it , merged it . HasFlag is the right one to have , awesome stuff, will be available starting with tomorrow's nightly builds !!

[Adam-it]

Thanks @Adam-it , merged it . HasFlag is the right one to have , awesome stuff, will be available starting with tomorrow's nightly builds !!

Woooohooo! Party Hard 😎