Read File Line by Line in Powershell
I Want to Read a File Line by Line in Powershell. Specifically, I Want to Loop Through the File, Store Each Line in a Variable in the Loop, and Do Some...
I want to read a file line by line in PowerShell. Specifically, I want to loop through the file, store each line in a variable in the loop, and do some processing on the line.
I know the Bash equivalent:
while read line do
if [[ $line =~ $regex ]]; then
# work here
fi
done < file.txt
Not much documentation on PowerShell loops.
4 Answers
Not much documentation on PowerShell loops.
Documentation on loops in PowerShell is plentiful, and you might want to check out the following help topics: about_For, about_ForEach, about_Do, about_While.
foreach($line in Get-Content .\file.txt) {
if($line -match $regex){
# Work here
}
}
Another idiomatic PowerShell solution to your problem is to pipe the lines of the text file to the ForEach-Object cmdlet:
Get-Content .\file.txt | ForEach-Object {
if($_ -match $regex){
# Work here
}
}
Instead of regex matching inside the loop, you could pipe the lines through Where-Object to filter just those you're interested in:
Get-Content .\file.txt | Where-Object {$_ -match $regex} | ForEach-Object {
# Work here
}
Get-Content has bad performance; it tries to read the file into memory all at once.
C# (.NET) file reader reads each line one by one
Best Performace
foreach($line in [System.IO.File]::ReadLines("C:\path\to\file.txt"))
{
$line
}
Or slightly less performant
[System.IO.File]::ReadLines("C:\path\to\file.txt") | ForEach-Object {
$_
}
The foreach statement will likely be slightly faster than ForEach-Object (see comments below for more information).
Reading Large Files Line by Line
Original Comment (1/2021) I was able to read a 4GB log file in about 50 seconds with the following. You may be able to make it faster by loading it as a C# assembly dynamically using PowerShell.
[System.IO.StreamReader]$sr = [System.IO.File]::Open($file, [System.IO.FileMode]::Open)
while (-not $sr.EndOfStream){
$line = $sr.ReadLine()
}
$sr.Close()
Addendum (3/2022) Processing the large file using C# embedded in PowerShell is even faster and has less "gotchas".
$code = @"
using System;
using System.IO;
namespace ProcessLargeFile
{
public class Program
{
static void ProcessLine(string line)
{
return;
}
public static void ProcessLogFile(string path) {
var start_time = DateTime.Now;
StreamReader sr = new StreamReader(File.Open(path, FileMode.Open));
try {
while (!sr.EndOfStream){
string line = sr.ReadLine();
ProcessLine(line);
}
} finally {
sr.Close();
}
var end_time = DateTime.Now;
var run_time = end_time - start_time;
string msg = "Completed in " + run_time.Minutes + ":" + run_time.Seconds + "." + run_time.Milliseconds;
Console.WriteLine(msg);
}
static void Main(string[] args)
{
ProcessLogFile("c:\\users\\tasaif\\fake.log");
Console.ReadLine();
}
}
}
"@
Add-Type -TypeDefinition $code -Language CSharp
PS C:\Users\tasaif> [ProcessLargeFile.Program]::ProcessLogFile("c:\\users\\tasaif\\fake.log")
Completed in 0:17.109
The almighty switch works well here:
'one
two
three' > file
$regex = '^t'
switch -regex -file file {
$regex { "line is $_" }
}
Output:
line is two
line is three
Chloe Bennett explores the intersection of pop culture, streaming entertainment, digital trends, and contemporary lifestyle. Her weekly commentary reaches thousands of culture enthusiasts.
