Golang FIFO Tutorial With Examples


Written by - Antony Shikubu
Reviewed by - Deepak Prasad

Introduction to Golang FIFO

Consider two or more programs running at the same time using different or same runtimes, and these programs need to exchange data/communicate. How will these programs communicate? Well these communication or data exchange can be made possible using a FIFO(First In First Out) file. In computing FIFO files also known as named pipes are being used to achieve inter process communication.

A FIFO is a form of redirection of communication from one process to another. The communication in FIFO or pipes is unidirectional and operates in a kind of a server and  client architecture or producer and consumer architecture.

A pipe requires at least one source and one destination for data to be transmitted between the client and server.

In this article we will learn about reading , writing into a pipe, and error handling while working with pipes. In order to proceed successfully, you need to have basic Go knowledge. In case this is your first time learning Go, I would recommend checking out Go Docs then come and learn about this topic. This article examples are based on Go version go1.18.3 linux/amd64.

 

Creating a FIFO File using GO

We use syscall.Mkfifo function to create a FIFO file using GO. The syntax to be used would be func Mkfifo(path string, mode uint32) (err error) where we provide the filename along with the path and the permission to be applied to the file.

For example, here we are creating a FIFO file with 640 permission:

syscall.Mkfifo(filePath, 0640)

 

RDONLY vs RDWR Modes in GO (It is Important!)

You may be familiar with the different read and write modes available with os package. Here I have captured the important ones from https://pkg.go.dev/os#pkg-constants

  • O_RDONLY = open the file read-only.
  • O_RDWR = open the file read-write.

So, why have I highlighted these 2 modes explicitly here?

That's because when we are reading a FIFO file, it is important to understand whether the file was opened as READ ONLY or with READ WRITE mode.

  • If the FIFO file was opened as READ ONLY then we are bound to get EOF at the end of the FIFO file. So the writer will exit the loop once it reaches end of line of the FIFO
  • If the FIFO file was opened as READ WRITE then there will be no EOF and writer will always continue to wait for reader to write into the FIFO File.

So depending upon your requirement, you must open a the FIFO File accordingly. Assuming you have a writer which will write into the FIFO file with some time gap and inconsistently then you can go for READ WRITE mode in which case your reader will continue to wait for incoming content. While if you have a one shot requirement to read a FIFO File and exit then you should read it as READ ONLY mode.

 

Perform simple read write operation using Golang FIFO

In this section I have created a simple reader and writer using Golang code. The writer function will be executed using goroutine in the background while reader will continue to read in the foreground.

package main

import (
	"bufio"
	"fmt"
	"os"
	"strings"
	"syscall"
	"time"
)

var fifoFile = "/tmp/myFifo"

func writer(filePath string) {
	f, err := os.OpenFile(filePath, os.O_WRONLY, 0600)

	fmt.Printf("WRITER << opened: %+v|%+v\n", f, err)
	if err != nil {
		panic(err)
	}

	fmt.Printf("WRITER << encoder created\n")

	for i := 0; i < 3; i++ {
		time.Sleep(1 * time.Second)
		_, err = f.WriteString(fmt.Sprint("line", i, "\n"))
		fmt.Printf("WRITER << written line%d, %+v\n", i, err)
	}

	time.Sleep(1 * time.Second)
	err = f.Close()
	fmt.Printf("WRITER << closed: %+v\n", err)
}

func reader(filePath string) {

	// Delete existing pipes
	fmt.Println("Cleanup existing FIFO file")
	os.Remove(filePath)

	// Create pipe
	fmt.Println("Creating " + filePath + " FIFO file")
	err := syscall.Mkfifo(filePath, 0640)
	if err != nil {
		fmt.Println("Failed to create pipe")
		panic(err)
	}

	// Open pipe for read only
	fmt.Println("Starting read operation")
	pipe, err := os.OpenFile(fifoFile, os.O_RDONLY, 0640)
	if err != nil {
		fmt.Println("Couldn't open pipe with error: ", err)
	}
	defer pipe.Close()

	// Read the content of named pipe
	reader := bufio.NewReader(pipe)
	fmt.Println("READER >> created")

	// Infinite loop
	for {
		line, err := reader.ReadBytes('\n')
		// Close the pipe once EOF is reached
		if err != nil {
			fmt.Println("FINISHED!")
			os.Exit(0)
		}

		// Remove new line char
		nline := string(line)
		nline = strings.TrimRight(nline, "\r\n")
		fmt.Printf("READER >> reading line: %+v\n", nline)

	}

}

func main() {
	fmt.Printf("STARTED %s\n", fifoFile)
	go writer(fifoFile)
	reader(fifoFile)

}

Here we have created a FIFO file /tmp/myFifo and then opened the file for reading as RDONLY mode so once EOF is reached then our for loop will return error. We are checking for that error and declaring FINISHED and exiting the loop.

# go run main.go 
STARTED /tmp/myFifo
Cleanup existing FIFO file
Creating /tmp/myFifo FIFO file
Starting read operation
READER >> created
WRITER << opened: &{file:0xc0000561e0}|<nil>
WRITER << encoder created
WRITER << written line0, <nil>
READER >> reading line: line0
WRITER << written line1, <nil>
READER >> reading line: line1
WRITER << written line2, <nil>
READER >> reading line: line2
WRITER << closed: <nil>
FINISHED!

 

Capture EOF in FIFO File

FIFO Reader open File as RDONLY

In the previous example we were exiting the reader as soon as we were getting error while reading the line from FIFO File. We can further improvise it to explicitly check for EOF using io.EOF and then exit. This code assumes that you have opened FIFO File as Read Only.

	// Infinite loop
	for {
		line, err := reader.ReadBytes('\n')
		// Close the pipe once EOF is reached
		if err != nil {
			if err == io.EOF {
				fmt.Println("EOF Reached")
				fmt.Println("FINISHED!")
				os.Exit(0)
			}

			panic(err)
		}

		// Remove new line char
		nline := string(line)
		nline = strings.TrimRight(nline, "\r\n")
		fmt.Printf("READER >> reading line: %+v\n", nline)

	}

Output:

# go run main.go 
STARTED /tmp/myFifo
Cleanup existing FIFO file
Creating /tmp/myFifo FIFO file
Starting read operation
READER >> created
WRITER << opened: &{file:0xc0000561e0}|<nil>
WRITER << encoder created
WRITER << written line0, <nil>
READER >> reading line: line0
WRITER << written line1, <nil>
READER >> reading line: line1
WRITER << written line2, <nil>
READER >> reading line: line2
WRITER << closed: <nil>
EOF Reached
FINISHED!

As you can see, now we added a proper check for EOF in the file and then exit accordingly.

 

FIFO Reader open File as RDWR

When we use RDWR mode to read the FIFO File then we will never get EOF so our reader will continue to wait for input from the writer. To overcome this I prefer to use timeout with goroutine so we will exit the loop based on timeout value.

// This will forcefully close the operation from func() after time.Duration value
func timeoutMyFunc(timeout time.Duration, myFunc func()) {
	finished := make(chan bool)
	go func() {
		myFunc()
		finished <- true
	}()
	select {
	case <-time.After(timeout):
		fmt.Println("Timeout reached")
		
	case <-finished:
		fmt.Println("Successfully finished the script execution")
		os.Exit(0)
	}
}

func main() {
	fmt.Printf("STARTED %s\n", fifoFile)
	go writer(fifoFile)

	timeoutMyFunc(10*time.Second, func() {
		reader(fifoFile)
	})

}

Output:

# go run main.go 
STARTED /tmp/myFifo
Cleanup existing FIFO file
Creating /tmp/myFifo FIFO file
Starting read operation
READER >> created
WRITER << opened: &{file:0xc0000562a0}|<nil>
WRITER << encoder created
WRITER << written line0, <nil>
READER >> reading line: line0
WRITER << written line1, <nil>
READER >> reading line: line1
WRITER << written line2, <nil>
READER >> reading line: line2
WRITER << closed: <nil>
Timeout reached

So here our timeoutMyFunc will continue to monitor the reader and after 10 second it will just forcefully exit.

Downside of this approach:

The timeout function will not check if the buffer is empty or reader is still processing data. In my personal code (not the one listed here) I had further enhanced it in my logic to make sure buffer is non empty before exiting but you can choose to enhance this code based on your requirement.

Moreover you also have to make sure you close your file properly to avoid leaking file descriptors as we are using os.Exit. Ideally os.Exit is not recommended because it will not entertain the defer close functions and there are possibilities of leaking FDs. This code was added just as a proof of concept and you should further think of improvising it based on your environment.

 

Summary

A FIFO file is a special kind of a file  that sends data from one process to another so that the receiving process reads the data in first-in-First-out (FIFO) fashion. In this article we have two processes: writer and reader. The writer sends data into a FIFO file and the reader reads data from the FIFO file. The writer can only send data to the reader only if the reader process is running. The writer process can only write data into the FIFO file only if the reader is ready to read from the FIFO file.

 

References

Read continuously from a named pipe
bufio.Reader not getting EOF on Darwin FIFO

 

Views: 8

Antony Shikubu

He is highly skilled software developer with expertise in Python, Golang, and AWS cloud services. Skilled in building scalable solutions, he specializes in Django, Flask, Pandas, and NumPy for web apps and data processing, ensuring robust and maintainable code for diverse projects. You can connect with him on Linkedin.

Categories GO

Can't find what you're searching for? Let us assist you.

Enter your query below, and we'll provide instant results tailored to your needs.

If my articles on GoLinuxCloud has helped you, kindly consider buying me a coffee as a token of appreciation.

Buy GoLinuxCloud a Coffee

For any other feedbacks or questions you can send mail to admin@golinuxcloud.com

Thank You for your support!!

Leave a Comment