Read file with timeout in golang [Practical Examples]

Introduction to Read file with timeout in Go

Reading and writing files are basic tasks needed for many Go programs. Reading files is one of the most essential tasks in programming. It allows us to see content, modify and write it using programming. To read files in Go, we use the os, ioutil, io, and bufio packages.

 

Using Goroutines and channels

Goroutine is a lightweight thread of functions/methods which executes concurrently along with the main program flow. It's implemented using the Go keyword e.g go func(){}
Channels is a communication pipe in which goroutines can communicate with each other. Its main purpose is to transfer a block of data which data can be written into or read from.

 

How to declare a channel and use in Go

chan is the keyword used in a channel declaration in Golang
var keyword or := shorthand are used in declaration of a channel and make keyword is used to create the channel.

channelName:= make(chan type) or var channelName chan type

For Example

dataChannel:= make( chan int) or var dataChannel chan int

or

Declaration of a buffered channel which is a channel with a capacity e.g channelName:= make(chan Type, n)

For example

dataStream := make(chan string, 1)

How to add values to a channel, refers to sending a signal to the channel

channelName <- “data from” results of a function etc e.g channelName <- 5;

Retrieving values from channel

valueName := <- channelName

Closing channel

close(channelName)

 

Golang Timeout

Timeouts are important for programs that connect to external resources or that otherwise need to bound execution time. This is because too long server-side processing will cost too many resources, resulting in a decline in concurrency, and even service unavailability.

 

Example-1: Create a program with Timeout in GO

Create a timeout with the select statement. To use timeouts with concurrent goroutines, one must import time.
Then, create a channel with time.After() which as a parameter takes time. The call time.After(time.Second) would fill the channel after a second.

For example, on how to implement timeouts in Go with channel and select

package main
import (
    "fmt"
    "time"
)
func main() {
  dataChannel:= make(chan string, 1)
    go func() {
        time.Sleep(2 * time.Second)
        c1 <- "result 1"
    }()
select {
    case results := <- dataChannel:
        fmt.Println(results)
    case <-time.After(1 * time.Second):
        fmt.Println("timeout 1")
    }
}

 

Explanation: From the channel dataChannel. Its buffered channel returns results after 2 seconds from the non-blocking goroutine function. The select function is implementing the timeout. results:= <- dataChannel awaits the result and <-time.After awaits a value to be sent after the timeout of 1s. Since select proceeds with the first receive that’s ready, we’ll take the timeout case if the operation takes more than the allowed 1s.

In this post, we will see how to read file contents with timeout in Go:-

• Read an entire file
• Read file line by line
• Read file word by word
• Read file on a specific chunk

 

Method-1: Read an entire file with timeout in Go

In Go, Reading an entire file content/text is to use ReadFile() function from the ioutil/os package. This function reads the entire content of the file into a byte slice. Ioutil package should not be used to read a large file thus the function is quite sufficient for small files.

Example of reading an entire file content in Go.

package main

import (
	"fmt"
	"io/ioutil"
	"os"
	"strconv"
	"time"
)

func main() {

	filePath := os.Args[1]
	timeOut, _ := strconv.ParseFloat(os.Args[2], 64)
	// buffered channel of dataStream
	dataStream := make(chan string, 1)
	// Run ReadFileToString function in it's own goroutine and pass back it's
	// response into dataStream channel.
	go func() {
		data, _ := ReadFileToString(filePath)
		dataStream <- data
		close(dataStream)
	}()
	// Listen on dataStream channel AND a timeout channel - which ever happens first.
	select {
	case res := <-dataStream:
		fmt.Println(res)
	case <-time.After(time.Duration(timeOut) * time.Millisecond):
		fmt.Println("Program execution out of time ")
	}
}
func ReadFileToString(file string) (string, error) {
	content, err := ioutil.ReadFile(file)
	// error encountered during reading the data
	if err != nil {
		return "", err
	}
	// convert bytes to string
	return string(content), nil
}

Explanation:-

• In the main() function, we are using the os package to read filePath as the first argument and pass timeOut as the second argument value to be used in the ReadFileToString function which enhances the dynamic use of values instead of hardcoding the filename values as well as timeout values in the select statement.
• The ParseFloat() function is a strconv package inbuilt function that converts strings to a floating-point number with the precision defined by the bitSize.
• The value of bitSize must be 32 or 64. 32 is used for float32, and 64 is used for float64.
• Os.Args[2]:- This is the string value to parse into the floating-point number. 64 is the bitesize specified precision.
• dataStream:- is a buffered channel receiving data from ReadFileToString function.
• go func (){} function handles the ReadFileToString which passes the filePath of type string and returns the response data into the dataStream channel and closes the dataStream channel.
• The select statement will be executed multiple times. We also have time.After with a duration of a specified timeout second as one of the case statements. So, this setup will receive all the file content from the dataStream channel for the specified timeOut a second and then exit.
• func ReadFileToString(file string) (string, error){} :- It’s a function that receives file name of type string and uses ioutil.ReadFile() function from ioutil package to read file content and returns a response of type string and error. The err variable gets an error description if an error occurs while reading the file. In the case of any problem, the error gets displayed to the user.

Output:- Using timeout 1 seconds

$ go run main.go text.txt 1.0
Welcome to the Go Linux program.
We have robust systems and study programs underway.
Join us and study various languages starting with Golang.

Explanation:- we are executing the program by using go run main.go text.txt 1.0

• Main.go is a file holding all code for the program.
• text.txt is the first argument we are reading the filePath
• 1.0is timeOut value.

NB// Our program execution time is below 1 second thus the program won’t terminate.

Output:- Using timeout 0.9 seconds

$ go run main.go text.txt 0.9
Program execution out of time

This time our program is terminated because its execution time is beyond the specified timeout

 

Method-2: Read file line by line with timeout in Go

To read a file line by line, we can use a convenient bufio.Scanner structure. Its constructor, NewScanner(), takes an opened file (remember to close the file after the operation is done, for example, by using defer statement) and lets you read subsequent lines through Scan() and Text() methods. Using Err() method, you can check errors encountered during file reading.
The os package contains the Args array, a string array that contains all the command-line arguments passed. We can use GoLang “bufio” package along with the “os” package to read the contents of a file line by line.

The process to read a text file line by line with timeout include the following steps:

• Use os.Args[] function to get filepath and timeout values
• Create a buffered channel for datastream, readerr unbuffered channel.
• Use os.open() function to open the file.
• Use bufio.NewScanner() function to create the file scanner.
• Use bufio.ScanLines() function with the scanner to <a href="https://www.golinuxcloud.com/csplit-split-command-examples-linux-unix/" title="15 csplit & split command examples in Linux [Cheat Sheet]" target="_blank" rel="noopener noreferrer">split the file into lines</a>. Then use the scanner Scan() function in a for loop to get each line and process it.</li>
<li style="margin-bottom: 10px;">For loop, the statement is used to execute select cases one at a time.</li>
<li style="margin-bottom: 10px;"><code>time.After function that sends the current time to a channel after a specific duration has elapsed.
• select statement behavior, where it will block until one of its cases can run. And it will choose one at random if multiple cases can run.

Here is the complete code to read a file line by line with timeout in GoLang.

package main

import (
	"bufio"
	"fmt"
	"log"
	"os"
	"strconv"
	"time"
)

func main() {
	//get filepath and timeout on the terminal
	filePath := os.Args[1]
	timeOut, _ := strconv.ParseFloat(os.Args[2], 64)
	//creating  channels
	dataStream := make(chan string, 1)
	readerr := make(chan error)

	// Run ReadFileLineByLine function in its own goroutine and pass back it's
	// response into dataStream channel.
	go ReadFileLineByLine(filePath, dataStream, readerr)

loop:
	for {
		// select statement will block this thread until one of the three conditions below is met

		select {
		case data := <-dataStream:
			// Process each line
			fmt.Println(data)
		case <-time.After(time.Duration(timeOut) * time.Second):
			fmt.Println("Program execution out of time ")
			break loop
		case err := <-readerr:
			if err != nil {
				log.Fatal(err)
			}
			break loop
		}
	}
}

func ReadFileLineByLine(filePath string, data chan string, er chan error) {
	// open file
	file, err := os.Open(filePath)
	if err != nil {
		fmt.Println(err)
	}
	// close the file at the end of the program
	defer file.Close()

	// read the file line by line using scanner
	scanner := bufio.NewScanner(file)

	for scanner.Scan() {
		data <- scanner.Text()
	}
	close(data) // close causes the range on the channel to break out of the loop
	er <- scanner.Err()
}

 

Explanation:-In the above program:

• At the main() function we have created a two-channel named dataStream which is a buffered channel and readerr unbuffered channel, dataStream which will hold data of string type while readerr holds errors.
• Then we pass this channel to the ReadFileLineByLine() function which is pushing the content from the file line by line to this channel.
• In the select statement, we are receiving the file content from the dataStream channel, in case of an error, we receive it on the readerr channel.
• This select statement is inside an infinite for loop so the select statement will be executed multiple times until we exit out of for loop.
• We also have time.After with a duration of a specified timeout second as one of the case statements.
• So this setup will receive all the file content from the dataStream channel for the specified timeOut second and then exit.

Output

$ go run main.go text.txt 1.0
Welcome to GoLinux program.
We have robust systems and study programs underway.
Join us and study various languages starting with Golang.

Our program will read line by line and execute within the stipulated time.

If we give a smaller timeout window, then the program will exit with timeout:

$ go run main.go test.txt 0.1
Program execution out of time

 

Method-3: Read file word by word with timeout in Go

It's similar to the read file line-by-line function the only difference is that we use scansWord scanner.Split(bufio.ScanWords)

package main

import (
	"bufio"
	"fmt"
	"log"
	"os"
	"time"
)

// Wrapper function to implement the timeout which takes
// timeout value and the function to monitor as input
func timeoutFunc(timeout time.Duration, myFunc func()) {
	finished := make(chan bool)
	go func() {
		// Add your function
		myFunc()
		finished <- true
	}()
	select {
	case <-time.After(timeout):
		fmt.Println("Timeout reached")
		os.Exit(10)
	case <-finished:
		os.Exit(0)
	}
}

func main() {
	f, err := os.Open("files.txt")
	if err != nil {
		log.Fatal(err)
	}
	defer f.Close()
	// read a file word by word
	scanner := bufio.NewScanner(f)
	scanner.Split(bufio.ScanWords)

	// add a timeout of 3 seconds before exiting
	timeoutFunc(3*time.Second, func() {

		for scanner.Scan() {
			// do something with a word
			fmt.Println(scanner.Text())

			// give time of 0.5 seconds interval after reading every word
			time.Sleep(500 * time.Millisecond)
		}
	})

}

Output:

$ go run main.go 
In
the
main
()
function,
we
Timeout reached

As expected, the for loop was able to read some words from the file but after reaching 3 seconds, the code timed out.

 

Method-4: Read file on a specific chunk with timeout in Go

In Go, Reading file chunk by chunk is useful if you have a very large file or don’t want to store the entire file in memory, you can read the file in fixed-size chunks. In this case, in the readFileChunk() function you need to create a byte slice of the specified size (chunkSize in the example) as a buffer for storing the subsequent read bytes. Using the Read() method of the Filetype, we can load the next chunk of the file data. The reading loop finishes when an io.EOF error occurs, indicating the end of the file. The file chunk to be read is specified from the terminal alongside the filePath, timeout, and chunksize.

package main

import (
   "fmt"
   "io"
   "log"
   "os"
   "strconv"
   "time"
)

func main() {
   dataStream := make(chan string, 1)
   filePath := os.Args[1]
   timeOut, _ := strconv.ParseFloat(os.Args[2], 64)
   chunkSize, _ := strconv.Atoi(os.Args[3])
   go readFileChunk (filePath, dataStream, int64(chunkSize))
   select {
   case resultData := <- dataStream:
      fmt.Println(resultData)
   case <-time.After(time.Duration(timeOut) * time.Millisecond):
      fmt.Println("timeout")
   }

}

func readFileChunk(filePath string, data chan string, chunkSize int64) {
   // open file
   f, err := os.Open(filePath)
   if err != nil {
      log.Fatal(err)
   }
   // remember to close the file at the end of the program
   defer f.Close()

   buf := make([]byte, chunkSize)
   for {
      readTotal, err := f.Read(buf)
      if err != nil && err != io.EOF {
         log.Fatal(err)
      }

      if err == io.EOF {
         break
      }

      data <- string(buf[:readTotal])
   }
   close(data)
}

Output: timeout 1.0 seconds

$ go run main.go text.txt 1.0 20
Welcome to GoLinux p

 

Explanation:- go run main.go text.txt 1.0 20

• main.go is the main file containing code.
• text.txt holds the content to be read.
• 1.0 is the specified timeout by the user.
• 20 is the chunk to be read.

 

Summary

• Golang Timeout is essential when a resource-intensive program is being executed. This helps to terminate such prolonged programs.
• Reading a file is essential in Go, for it enables writing, updating, deleting, and retrieval/reading of information in a file.

 

References

• reading a file in Go
• Timeout in Go
• channel in Go