Table of Contents
Methods to read different file types in GO
There are several ways to read a file in Go, some of the most common methods are:
- The
ioutil.ReadFilefunction reads the entire file into memory as a byte slice. - The
os.Openfunction opens a file, and theio.Readmethod reads the contents of the file. - The
bufio.NewScannerfunction returns a new Scanner to read from the file, allowing for reading line by line or token by token. - The
encoding/csvpackage provides aNewReaderfunction that reads a CSV file and returns a*csv.Readerwhich can be used to read the file as a series of records. - The
json.NewDecoderfunction can be used to read a JSON file, allowing for decoding the JSON data into a struct or map. - The
xml.NewDecoderfunction can be used to read an XML file, allowing for decoding the XML data into a struct or map. - The
gopkg.in/yaml.v3package is used to parse YAML file, allowing for decoding the XML data into a struct or map.
Method-1: Using os.Open() function to read text file
In this example we are using os.Open() function to open the "file.txt" file. This function takes the filename as an argument and returns the file object and an error. Then we define file.Close() method inside a defer statement, which makes sure that the file is closed after the function exits, regardless of whether an error occurs or not.
The make([]byte, 1024) function creates a byte slice to hold the file contents with the capacity of 1024 bytes.
The program then uses a for loop to read the file contents, where the file.Read() method reads the contents of the file into the data byte slice and returns the number of bytes read and an error.
Within the loop, the code checks if the error returned is io.EOF (end of file) and if it is, the loop breaks. If not, the code checks if there was an error and if there is an error, the program prints an error message and returns.
package main
import (
"fmt"
"io"
"os"
)
func main() {
file, err := os.Open("file.txt")
if err != nil {
fmt.Println("File reading error", err)
return
}
defer file.Close() // it's important to close the file after reading it
// create a byte slice to hold the file contents
data := make([]byte, 1024)
for {
n, err := file.Read(data)
if err == io.EOF {
break
}
if err != nil {
fmt.Println("File reading error", err)
return
}
fmt.Println("Read", n, "bytes:", string(data[:n]))
}
}In this example, we use the os.Open function to open the file and the file.Read method to read the contents of the file, this method reads a specified number of bytes from the file into a byte slice. This approach is useful when working with large files, as it allows you to read the file in chunks, rather than loading the entire file into memory at once.
Method-2: Using ioutil.Read() function to read text file
The ioutil.ReadFile function is used to read the contents of the "file.txt" file. This function takes the filename as an argument and returns the contents of the file as a byte slice and an error. We will need string() function to convert the byte slice to a string before printing it.
package main
import (
"fmt"
"io/ioutil"
)
func main() {
data, err := ioutil.ReadFile("file.txt")
if err != nil {
fmt.Println("File reading error", err)
return
}
fmt.Println("Contents of file:", string(data))
}os.Open() vs ioutil.ReadFile() - Comparison and Differences
os.Open()andioutil.ReadFile()are both used to read the contents of a file, but they are used for different purposes and have some key differences.os.Open()is used to open a file and return a file descriptor, which is an object that can be used to read, write, or manipulate the file. This function returns anos.Filetype, which implements theio.Readerandio.Writerinterfaces. This allows you to perform various operations on the file, such as reading or writing specific portions of the file, or seeking to a specific position in the file.ioutil.ReadFile, on the other hand, is a convenience function that reads the entire contents of a file into a byte slice. This function reads the file in one go, and returns the contents as a byte slice, so it's only useful when you want to read the entire contents of a file at once and don't need to perform any other operations on the file.
In short, os.Open() gives you more control over the file and allows you to perform various operations on it, while ioutil.ReadFile() is a simple function that reads the entire file in one go and returns the contents as a byte slice.
Method-3: Using bufio.NewScanner() to read text file
In this example we are using os.Open() to open the "file.txt" file. Next, the program uses the file.Close() method inside a defer statement, which makes sure that the file is closed after the function exits, regardless of whether an error occurs or not.
The bufio.NewScanner function is used to create a new scanner that reads from the file. Then the code uses a for loop to read the file contents, where the scanner.Scan() method reads the next line from the file and returns true if there is another line to read and false otherwise.
Within the loop, the code uses the scanner.Text() method to get the text of the current line and print it using the fmt.Println function.
The code uses the scanner.Err() method to check if there was an error while reading the file, if there is an error, the program prints an error message and returns.
package main
import (
"bufio"
"fmt"
"os"
)
func main() {
file, err := os.Open("file.txt")
if err != nil {
fmt.Println("File reading error", err)
return
}
defer file.Close()
scanner := bufio.NewScanner(file)
for scanner.Scan() {
fmt.Println(scanner.Text())
}
if err := scanner.Err(); err != nil {
fmt.Println("File reading error", err)
}
}The bufio.NewScanner is useful when you want to read a file line by line and it's performance is good even with large files.
Method-4: Using encoding/csv to read a CSV file
We again use os.Open function is used to open the "file.csv" file, which takes the filename as an argument and returns the file object and an error. Next, the program uses the file.Close() method inside a defer statement, which makes sure that the file is closed after the function exits, regardless of whether an error occurs or not.
The csv.NewReader(file) function creates a new CSV reader that reads from the file. The reader.ReadAll() method reads all the records in the CSV file and returns them as a slice of slices of strings and an error.
The code then checks if there was an error while reading the file using an if statement. If there is an error, the program prints an error message and returns. Then the program uses a for loop to iterate over the records, where record is a slice of strings representing a row in the CSV file.
package main
import (
"encoding/csv"
"fmt"
"os"
)
func main() {
file, err := os.Open("file.csv")
if err != nil {
fmt.Println("File reading error", err)
return
}
defer file.Close()
reader := csv.NewReader(file)
records, er := reader.ReadAll()
if err != nil {
fmt.Println("File reading error", err)
return
}
for _, record := range records {
fmt.Println(record)
}
}
This function assumes that the CSV file is using comma (,) separator. But if you want to provide a custom separator then you can define the same in the Reader struct as shown below where we are using semi-colon (;) as the separator:
reader := csv.NewReader(file)
reader.Comma = ';' // set the separator to ';'
records, err := reader.ReadAll()
if err != nil {
fmt.Println("File reading error", err)
return
}
Method-5: Parse JSON file using json.NewDecoder()
In this example we are using json.NewDecoder function to read a JSON file and decode it into a struct. We still have to rely on os.Open to open the file.
This is the data.json file content which we will parse using our golang code:
{
"Name": "John Doe",
"Age": 30
}
The json.NewDecoder(file) function creates a new JSON decoder that reads from the file. The decoder.Decode(&p) method reads the JSON data from the file and unmarshals it into the p variable of type Person. You can read more about parsing JSON at Golang Parse JSON (With or Without Struct)
package main
import (
"encoding/json"
"fmt"
"os"
)
type Person struct {
Name string
Age int
}
func main() {
file, err := os.Open("data.json")
if err != nil {
fmt.Println("File reading error", err)
return
}
defer file.Close()
var p Person
decoder := json.NewDecoder(file)
err = decoder.Decode(&p)
if err != nil {
fmt.Println("File reading error", err)
return
}
fmt.Println(p)
}Method-6: Using xml.NewDecoder() to parse XML Files
Here is our same file.xml file which we will parse using xml.NewDecoder():
<?xml version="1.0"?>
<person>
<name>Amit Kumar</name>
<age>30</age>
</person>Here the xml.NewDecoder function creates a new XML decoder that reads from an io.Reader interface, in this case the file object returned from os.Open. The Decode method reads the next XML token from the input and stores it in the value pointed to by the argument, in this case the struct p.
package main
import (
"encoding/xml"
"fmt"
"os"
)
type Person struct {
XMLName xml.Name `xml:"person"`
Name string `xml:"name"`
Age int `xml:"age"`
}
func main() {
file, err := os.Open("file.xml")
if err != nil {
fmt.Println("File reading error", err)
return
}
defer file.Close()
var p Person
decoder := xml.NewDecoder(file)
err = decoder.Decode(&p)
if err != nil {
fmt.Println("File reading error", err)
return
}
fmt.Println(p)
}
Method-7: Using gopkg.in/yaml.v3 to parse YAML files
Here is the same YAML file which we will parse in this section. It is a very simple YAML file, for more complex examples you can refer Golang Parse YAML file [With or Without Struct]
name: Amit Kumar
age: 35The code uses ioutil.ReadFile function to read the "config.yaml" file. Next the code uses yaml.Unmarshal function is used to unmarshal the file data into the variable p of type Person and lastly uses the fmt.Println function to print the struct p.
package main
import (
"fmt"
"io/ioutil"
"gopkg.in/yaml.v3"
)
type Person struct {
Name string
Age int
}
func main() {
data, err := ioutil.ReadFile("config.yaml")
if err != nil {
fmt.Println("File reading error", err)
return
}
var p Person
err = yaml.Unmarshal(data, &p)
if err != nil {
fmt.Println("File reading error", err)
return
}
fmt.Println(p)
}
Some Practical Examples
Follow is text.txt content which we will use through out this article:
Welcome to the GoLinuxCloud program.
We have robust systems and study programs underway.
Join us and study various languages starting with GoLang.
1. Read the entire file in GoLang
In Go, Reading an entire file content/text is to use ReadFile() function from the ioutil/os package. This function reads the entire content of the file into a byte slice. The ioutil package should not be used in reading a large file thus the function is quite sufficient for small files. Below is the implementation of how to use it.
package main
import (
"fmt"
"io/ioutil"
"os"
)
func main() {
// get file from terminal
inputFile := os.Args[1]
// read the whole content of file and pass it to file variable, in case of error pass it to err variable
file, err := ioutil.ReadFile(inputFile)
if err != nil {
fmt.Printf("Could not read the file due to this %s error \n", err)
}
// convert the file binary into a string using string
fileContent := string(file)
// print file content
fmt.Println(fileContent)
}
Explanation:-
inputFile;- We are getting file inputs from the terminal and using the os package Args[] function to get the file parameter as a string.
file, err := ioutil.ReadFile(inputFile) :- This ioutil package as ReadFile() function capable of reading the whole file content and map the results into file variable. It receives filename from inputFile variable. We have an error check in case the reading of file content wasn’t successful and print the nature of the error.
fileContent: = string(file):- Finally we need to convert the file content into a string using the string function. Print the whole fileContent into the user's terminal using the fmt package Println() function.
Output:-
$ go run main.go text.txt
Welcome to the GoLinuxCloud program.
We have robust systems and study programs underway.
Join us and study various languages starting with GoLang.
2. Read file in chunks in GoLang
In Go, Reading a file in chunks is the most efficient approach. Thus it doesn’t take the whole file content completely into the memory but is loaded in a specific chunk. It is essential because reading a whole file at once may lead to memory leakage or memory errors.
Below is the implementation of the reading file in a chunk in GoLang
package main
import (
"fmt"
"io"
"os"
"strconv"
)
func main() {
// get file from terminal
inputFile := os.Args[1]
// declare chunk size
maxSz, _ := strconv.Atoi(os.Args[2])
// read the whole content of file and pass it to file variable, in case of error pass it to err variable
file, err := os.Open(inputFile)
if err != nil {
fmt.Printf("Could not open the file due to this %s error \n", err)
}
defer file.Close()
// create buffer
b := make([]byte, maxSz)
for {
// read content to buffer
readTotal, err := file.Read(b)
if err != nil {
if err != io.EOF {
fmt.Println(err)
}
break
}
fileContent := string(b[:readTotal])
// print content from buffer
fmt.Println(fileContent)
}
}This produces output as shown below when we use a long string
Output:-
$ go run main.go text.txt 10
Welcome to
the GoLin
uxCloud pr
ogram.
We
have robus
t systems
and study
programs u
nderway.
J
oin us and
study var
ious langu
ages start
ing with G
oLang.
Explanation:-
The os package Args[] function is used to provide the inputFile and maxSz values respectively for the program. However maxSz values need to be converted into an integer thus we are using strconv package Atoi() function to convert string input value to integer before being stored into maxSz variable.
The os package open() function is used to open file for easy access and read of the content, thus allowing us to use defer keyword to close the file to avoid memory leakage.
b := make([]byte, maxSz) creating a buffer to hold bytes slice of specified length and of defined capacity into which the bytes of the file will be read.
Using the Read() method of the Filetype, it reads up to the specified length and returns the number of bytes read. We store the bytes retuned in a variable. Then the slice is read from index 0 to n-1, up to the number of bytes returned by the Read() method, and printed. The reading loop finishes when an io.EOF error occurs, indicating the end of the file.
Finally, we can print the file content to the terminal using the fmt package Println() function.
3. Read file line by line in GoLang
In Go, we can use the bufio package and os package for reading a file content line by line. The process to read a text file line by line include the following steps:
- Use
os.args[]function to read filePath - Use
os.open()function to open the file. - Use
bufio.NewScanner()function to create the file scanner. - Use
bufio.ScanLines()function with the scanner to split the file into lines. - Then use the scanner
Scan()function in a for loop to get each line and process it.
Here is an example of reading a file line by line in GoLang.
package main
import (
"bufio"
"fmt"
"os"
)
func main() {
// get file from terminal
filePath := os.Args[1]
// read the whole content of file and pass it to file variable, in case of error pass it to err variable
file, err := os.Open(filePath)
if err != nil {
fmt.Printf("Could not open the file due to this %s error \n", err)
}
fileScanner := bufio.NewScanner(file)
fileScanner.Split(bufio.ScanLines)
for fileScanner.Scan(){
fmt.Println(fileScanner.Text())
}
if err = file.Close(); err != nil {
fmt.Printf("Could not close the file due to this %s error \n", err)
}
}
Output:-
$ go run main.go text.txt
Welcome to the GoLinuxCloud program.
We have robust systems and study programs underway.
Join us and study various languages starting with GoLang.
Explanation:-
Using os.Args[] function accessed filePath, Then we opened a text.txt file. Then we create a new scanner using the file. The scan() method reads the next line of the file, available through the text() method. After Scan returns false, the Err() method will return any error that occurred during scanning. If the error is End of File, Err() will return nil.
NB// We are processing the file lines, which is keeping our resources open for a longer time. We can read the file data into a string array and then process it accordingly. This is not recommended for a very large file though.
4. Reading File line by line to String Array in GoLang
It is similar to reading file line by line however, we have to store file content into a slice of an array string. Using var fileLines []string, we can append the text from the file into the fileLines variable and then access line by line later.
Below is an example
package main
import (
"bufio"
"fmt"
"os"
)
func main() {
// get file from terminal
filePath := os.Args[1]
// read the whole content of file and pass it to file variable, in case of error pass it to err variable
file, err := os.Open(filePath)
if err != nil {
fmt.Printf("Could not open the file due to this %s error \n", err)
}
fileScanner := bufio.NewScanner(file)
fileScanner.Split(bufio.ScanLines)
var fileLines []string
for fileScanner.Scan() {
fileLines = append(fileLines, fileScanner.Text())
}
if err = file.Close(); err != nil {
fmt.Printf("Could not close the file due to this %s error \n", err)
}
for key, value := range fileLines {
fmt.Printf("line %v : %s \n", key, value)
}
}
Output:-
$ go run main.go text.txt
line 0 : Welcome to the GoLinuxCloud program.
line 1 : We have robust systems and study programs underway.
line 2 : Join us and study various languages starting with GoLang.
Explanation:-
Using os.Args[] function accessed filePath, Then we opened a text.txt file.Then we create a new scanner using the file. The scan() method reads the next line of the file, available through the text() method. Scans results are appended to an array of strings. After Scan returns false, the Err() method will return any error that occurred during scanning. If the error is End of File, Err() will return nil.
5. Read file word by word in GoLang
The implementation is similar to read file line by line in Go, we just add the ScanWords from bufio package can be used to read a file word by word.
package main
import (
"bufio"
"fmt"
"os"
)
func main() {
// get file from terminal
filePath := os.Args[1]
// read the whole content of file and pass it to file variable, in case of error pass it to err variable
file, err := os.Open(filePath)
if err != nil {
fmt.Printf("Could not open the file due to this %s error \n", err)
}
defer file.Close()
fileScanner := bufio.NewScanner(file)
fileScanner.Split(bufio.ScanWords)
var fileLines []string
for fileScanner.Scan() {
fileLines = append(fileLines, fileScanner.Text())
}
if err := fileScanner.Err(); err != nil {
fmt.Printf("Could not scan the file due to this %s error \n", err)
}
for key, value := range fileLines {
fmt.Printf("word_No: %v -> %s \n", key, value)
}
}
Output:-
$ go run main.go text.txt
word_No: 0 -> Welcome
word_No: 1 -> to
word_No: 2 -> the
word_No: 3 -> GoLinuxCloud
word_No: 4 -> program.
word_No: 5 -> We
word_No: 6 -> have
word_No: 7 -> robust
word_No: 8 -> systems
word_No: 9 -> and
word_No: 10 -> study
word_No: 11 -> programs
word_No: 12 -> underway.
word_No: 13 -> Join
word_No: 14 -> us
word_No: 15 -> and
word_No: 16 -> study
word_No: 17 -> various
word_No: 18 -> languages
word_No: 19 -> starting
word_No: 20 -> with
word_No: 21 -> GoLang.
Explanation:-
Using os.Args[] function accessed filePath, Then we opened a text.txt file. Then we create a new scanner using the file. The scan() method reads the next word of the file, available through the text() method. Scans results are appended to an array of strings. After Scan returns false, the Err() method will return any error that occurred during scanning. If the error is End of File, Err() will return nil.
Summary
It’s very easy to read a file in Go programming. We can read the file contents line by line, chunk, word by word, file content to string, process them or we can store into a string array and process them later on. We should always close the file resources as soon as our work is done.
References
io/ioutil - Go Packages
os - Go Packages
encoding/xml - Go Packages
bufio - Go Packages
