Hi r/powershell,

I have a very large amount of object streaming in that I want to collect and export as CSV. First I collected all objects, and exported them to disk like this:

$result = 1..100 | Foreach {

    [PsCustomObject] @{
        'Id'   = $_
    }

}

$result | Export-Csv -Path out.csv -NoTypeInformation

This is quite fast, however, very memory intense. In reality there are a few 100k objects. So I thought I write them to disk immediately and don't keep them in memory:

1..100 | Foreach {

    [PsCustomObject] @{
        'Id'   = $_
    } | Export-Csv -Path out.csv -NoTypeInformation -Append

}

This works, but it's magnitudes slower than the other version.

So I thought I could somehow keep a part of the objects in memory, and write them to disk in batches. I thought I'm clever and do it like this:

$result = 1..100 | Foreach {

    [PsCustomObject] @{
        'Id'   = $_
    }

    if ($result.Count % 10 -eq 0) {

        $result | Export-Csv -Path out.csv -Append -NoTypeInformation
        $result = $null
    }

}

But $result is always $null inside the | foreach loop. Any way to get this working? Or any other way? foreach ($Item in $Collection) is not an option in my case.

Thanks