Golang buffio Complete Guide

Bufio is a package in the standard library of Go that provides buffered I/O. It implements a buffered reader and writer that both implement the io.Reader and io.Writer interfaces.

What is Buffered I/O?

Buffered I/O is a technique that allows a program to read or write data in chunks rather than one byte at a time. This is useful because it allows the program to read or write data more efficiently. It also allows the program to read or write data more predictably.

In Go, this is done by using the bufio package. This package provides buffered readers and writers.

How to Use Bufio

file.txt The file we’ll be using in this guide is:

...
...
...

Creating a Buffered Reader

To create a buffered reader, you can use the bufio.NewReader function. This function takes an io.Reader as an argument. This means that you can pass in any type that implements the io.Reader interface. This includes os.File , strings.Reader , and bytes.Buffer .

package main

import (
    "bufio"
    "fmt"
    "os"
)

func main() {
    file, err := os.Open("file.txt")
    if err != nil {
        fmt.Println(err)
        return
    }
    defer file.Close()

    reader := bufio.NewReader(file)
    fmt.Println(reader)
}

In the above example, we are creating a buffered reader from a file. We are then printing the buffered reader to the console. The buffered reader is a pointer to a bufio.Reader struct.

Reading from a Buffered Reader

To read from a buffered reader, you can use the bufio.Reader.Read function. This function takes a byte slice as an argument. This byte slice is where the data will be read into.

package main

import (
    "bufio"
    "fmt"
    "os"
)

func main() {
    file, err := os.Open("file.txt")
    if err != nil {
        fmt.Println(err)
        return
    }

    reader := bufio.NewReader(file)
    data := make([]byte, 100)
    _, err = reader.Read(data)
    if err != nil {
        fmt.Println(err)
        return
    }

    fmt.Println(string(data))
}

In the above example, we are creating a buffered reader from a file. We are then reading 100 bytes from the file into a byte slice. We are then converting the byte slice to a string and printing it to the console.

The output of the above program is:

Ex ad mollit laborum non esse nostrud excepteur. Mollit fugiat nisi magna est deserunt excepteur par

Creating a Buffered Writer

To create a buffered writer, you can use the bufio.NewWriter function. This function takes an io.Writer as an argument. This means that you can pass in any type that implements the io.Writer interface. This includes os.File , strings.Builder , and bytes.Buffer .

package main

import (
    "bufio"
    "fmt"
    "os"
)

func main() {
    file, err := os.Create("file.txt")
    if err != nil{
        fmt.Println(err)
        return
    }
    defer file.Close()

    writer := bufio.NewWriter(file)
    fmt.Println(writer)
}

In the above example, we are creating a buffered writer from a file. We are then printing the buffered writer to the console. The buffered writer is a pointer to a bufio.Writer struct.

Writing to a Buffered Writer

To write to a buffered writer, you can use the bufio.Writer.Write function. This function takes a byte slice as an argument. This byte slice is the data that will be written to the writer.

bufio.Writer.Flush is used to write the data to the writer. This function must be called before the program exits or the data will not be written to the writer.

package main

import (
    "bufio"
    "fmt"
    "os"
)

func main() {
    file, err := os.Create("file2.txt")
    if err != nil {
        fmt.Println(err)
        return
    }

    writer := bufio.NewWriter(file)
    _, err = writer.Write([]byte("Hello World!"))
    if err != nil {
        fmt.Println(err)
        return
    }

    err = writer.Flush()
    if err != nil {
        fmt.Println(err)
        return
    }
}

This will create a file called file.txt and write Hello World! to it. The bufio.Writer.Write function will not write the data to the file until the bufio.Writer.Flush function is called.

Change the Buffer Size

The default buffer size for a buffered writer is 4096 bytes. This means that the data will be written to the writer in chunks of 4096 bytes. If you want to change the buffer size, you can use the bufio.NewWriterSize function.

package main

import (
    "bufio"
    "fmt"
    "os"
)

func main() {
    file, err := os.Create("file2.txt")
    if err != nil {
        fmt.Println(err)
        return
    }

    writer := bufio.NewWriterSize(file, 100)
    _, err = writer.Write([]byte("Hello World!"))
    if err != nil {
        fmt.Println(err)
        return
    }

    err = writer.Flush()
    if err != nil {
        fmt.Println(err)
        return
    }
}

In the above program, we changed the buffer size to 100 bytes. This means that the data will be written to the writer in chunks of 100 bytes. This will make the writer slower but it will also use less memory.

Bufio vs. I/O

The main difference between buffered I/O and normal I/O is that buffered I/O reads or writes data in chunks rather than one byte at a time. While on the other side normal I/O reads or writes data one byte at a time. This might not seem like a big difference but it can make a big difference in performance.

In a case where you are reading or writing a lot of data, buffered I/O can be much faster than normal I/O. To see this, we can compare the performance of buffered I/O and normal I/O using benchmarks.

main.go


package main

import (
    "fmt"
    "bufio"
    "io"
    "os"
)

func funcToWithIO() {
    file, err := os.Open("file.txt")
    if err != nil {
        fmt.Println(err)
        return
    }
    defer file.Close()

    data := make([]byte, 100)
    for {
        _, err := file.Read(data)
        if err == io.EOF {
            break
        }
        if err != nil {
            fmt.Println(err)
            return
        }
    }
}

func funcToWithBufio() {
    file, err := os.Open("file.txt")
    if err != nil {
        fmt.Println(err)
        return
    }
    defer file.Close()

    reader := bufio.NewReader(file)
    data := make([]byte, 100)
    for {
        _, err := reader.Read(data)
        if err == io.EOF {
            break
        }
        if err != nil {
            fmt.Println(err)
            return
        }
    }
}

func createFile() {
    file, err := os.Create("file.txt")
    if err != nil {
        fmt.Println(err)
        return
    }
    defer file.Close()

    for i := 0; i < 1000000; i++ {
        file.Write([]byte("Hello World!"))
    }
}

func main() {
    createFile()
    funcToWithIO()
    funcToWithBufio()
}

main_test.go

package main

import "testing"

func BenchmarkFuncToWithIO(b *testing.B) {
    for i := 0; i < b.N; i++ {
        funcToWithIO()
    }
}

func BenchmarkFuncToWithBufio(b *testing.B) {
    for i := 0; i < b.N; i++ {
        funcToWithBufio()
    }
}

In the above example, we are creating a file called file.txt and writing Hello World! 1,000,000 times to it. We are then reading the file using normal I/O and buffered I/O. We are then benchmarking the two functions to see which one is faster.

The output of the above program is:

goos: linux
goarch: amd64
pkg: Rbufio
cpu: Intel(R) Core(TM) i7-10750H CPU @ 2.60GHz
BenchmarkFuncToWithIO-4                2         901096096 ns/op
BenchmarkFuncToWithBufio-4            33          39213737 ns/op
PASS
ok      Rbufio  4.194s

As you can see, the buffered I/O function is much faster with a runtime of 39.213737ms per iteration. The normal I/O function has a runtime of 901.096096ms per iteration. This is a huge difference in performance.

Other Bufio Functions

There are many other functions in the bufio package that can be used to read and write data. Here are some of the most commonly used functions:

`bufio.Reader.ReadString`

This function reads data until a specific delimiter is found. It returns a string containing the data up to and including the delimiter.

package main

import (
    "bufio"
    "fmt"
    "os"
)

func main() {
    file, err := os.Open("file.txt")
    if err != nil {
        fmt.Println(err)
        return
    }
    defer file.Close()

    reader := bufio.NewReader(file)
    data, err := reader.ReadString('\n')
    if err != nil {
        fmt.Println(err)
        return
    }
    fmt.Println(data)
}

In this case, the bufio.Reader.ReadString function will read data from the file until it finds a new line character. It will then return the data up to and including the newline character.

`bufio.Reader.ReadBytes`

This function is similar to the bufio.Reader.ReadString function. The only difference is that it returns a byte slice instead of a string.

package main

import (
    "bufio"
    "fmt"
    "os"
)

func main() {
    file, err := os.Open("file.txt")
    if err != nil {
        fmt.Println(err)
        return
    }
    defer file.Close()

    reader := bufio.NewReader(file)
    data, err := reader.ReadBytes('\n')
    if err != nil {
        fmt.Println(err)
        return
    }
    fmt.Println(data)
}

`bufio.Reader.ReadSlice`

This function is similar to the bufio.Reader.ReadString function. The only difference is that it returns a byte slice instead of a string. The difference between bufio.Reader.ReadSlice and bufio.Reader.ReadBytes is that bufio.Reader.ReadSlice will return an error if the delimiter is not found.

package main

import (
    "bufio"
    "fmt"
    "os"
)

func main() {
    file, err := os.Open("file.txt")
    if err != nil {
        fmt.Println(err)
        return
    }
    defer file.Close()

    reader := bufio.NewReader(file)
    data, err := reader.ReadSlice('\n')
    if err != nil {
        fmt.Println(err)
        return
    }
    fmt.Println(data)
}

`bufio.Reader.ReadLine`

package main

import (
    "bufio"
    "fmt"
    "os"
)

func main() {
    file, err := os.Open("file.txt")
    if err != nil {
        fmt.Println(err)
        return
    }
    defer file.Close()

    reader := bufio.NewReader(file)
    data, isPrefix, err := reader.ReadLine()
    if err != nil {
        fmt.Println(err)
        return
    }
    fmt.Println(data, isPrefix)
}

`bufio.Reader.Read`

The bufio.Reader.Read function reads data into a byte slice. It returns the number of bytes read and an error if any.

package main

import (
    "bufio"
    "fmt"
    "os"
)

func main() {
    file, err := os.Open("file.txt")
    if err != nil {
        fmt.Println(err)
        return
    }
    defer file.Close()

    reader := bufio.NewReader(file)
    data := make([]byte, 5)
    n, err := reader.Read(data)
    if err != nil {
        fmt.Println(err)
        return
    }
    fmt.Println(n, data)
}

Other Bufio Types

`bufio.Writer`

The bufio.Writer type implements a buffered writer. It wraps an io.Writer and provides buffering and some help for textual I/O. It is not safe for concurrent use by multiple goroutines.

It has the following methods:

bufio.NewWriter - creates a new buffered writer
bufio.Writer.Flush - flushes any buffered data to the underlying writer
bufio.Writer.Available - returns the number of bytes that can be written to the buffer without blocking
bufio.Writer.Buffered - returns the number of bytes that are currently buffered
bufio.Writer.Reset - resets the buffer to be empty
bufio.Writer.Write - writes a byte slice to the buffer
bufio.Writer.WriteString - writes a string to the buffer
bufio.Writer.WriteByte - writes a single byte to the buffer
bufio.Writer.WriteRune - writes a single UTF-8 encoded Unicode character to the buffer

`bufio.Scanner`

The bufio.Scanner type implements a simple scanner for reading data. It wraps an io.Reader and provides a simple interface for reading data, line by line. It is not safe for concurrent use by multiple goroutines.

It has the following methods:

bufio.NewScanner - creates a new scanner
bufio.Scanner.Scan - advances the scanner to the next token, which will then be available through the Text method. It returns false when the scan stops, either by reaching the end of the input or by an error.
bufio.Scanner.Text - returns the most recent token generated by a call to Scan as a newly allocated string holding its bytes.
bufio.Scanner.Bytes - returns the most recent token generated by a call to Scan as a slice of bytes that will be overwritten by the next call to Scan .
bufio.Scanner.Err - returns the first non- nil error that was encountered by the Scanner .

Conclusion

In this article, we learned about the bufio package and its types. We also learned about the different methods of the bufio.Reader type. We also learned about the other types of the bufio package.